TRAVEL: Tag-Aware Conversational FAQ Retrieval via Reinforcement Learning
Authors: Yue Chen
Dingnan Jin
Chen Huang
Jia Liu
Wenqiang Lei
Venue: EMNLP 2023 Main (2023)
Keywords: FAQ retrieval
Conversational FAQ Retrieval
Dialogue System
Human-machine interaction

Abstract: Efficiently retrieving FAQ questions that match users' intent is essential for online customer service. 
Existing methods aim to fully utilize the dynamic conversation context to enhance the semantic association between the user query and FAQ questions.
However, the conversation context contains noise, e.g., users may click questions they don't like, leading to inaccurate semantics modeling.
To tackle this, we introduce tags of FAQ questions, which can help us eliminate irrelevant information. 
We later integrate them into a reinforcement learning framework and minimize the negative impact of irrelevant information in the dynamic conversation context. 
We experimentally demonstrate our efficiency and effectiveness on conversational FAQ retrieval compared to other baselines.

TRAVEL: Tag-Aware Conversational FAQ Retrieval via Reinforcement Learning
Yue Chen
Dingnan Jin Chen Huang", Jia Liu Wenqiang Lei
College of Computer Science, Sichuan University
Ant Group, China
[chenyuecee24, huangc.scu, wenqianglei) @ gmmail.com [dingnan.jdn, jianiu. Ij] @ @antgroup.com
Abstract
adhere their intent. In fact, users may click ques- tions containing information that is irrelevant their intent due to domain unfamiliarity or misoper- ation (Keyvan and Huang, 2022). This information, which we call "tags"! following (Yu et al., 2019; Romero et al., 2013), brings noise to the conversa- tion context and disturbs the retrieval efficiency. Taking Figure.1 (a)
Efficiently retrieving FAQ questions that match users' intent is essential for online customer ser- vice, Existing methods aim fully utilize the dynamic conversation context to enhance the semantic association between the user query and FAQ questions. However, the conversation context contains noise, e.g., users may click questions they don't 't like, leading to inaccurate semantics modeling. To tackle this, intro- duce tags of FAQ questions, which can help us eliminate irrelevant information. We later inte- grate them into reinforcement learning frame- work and minimize the negative impact of irrel- evant information the dynamic conversation context. We experimentally demonstrate our efficiency and effectiveness on conversational FAQ retrieval compared to other baselines.
example, the user's target question is "How to cancel the automatic repayment of credit cards?". Due domain unfa- miliarity, in Figure. (c), the user clicks the ques- tion "How to cancel the automatic payment of bank cards", because it contains the same tag ("cancel") with the user's 's intent. Unfortunately, this intro- duces irrelevant tags such as "automatic payment" and "bank card". As result, the system retrieves irrelevant FAQ questions subsequent turns (e.g., "How to cancel the automatic payment of the credit cards?" in the next turn). Therefore, such irrele- vant information makes cxisting systems require more turns to hit the FAQ question that matches the user's intent.
1 Introduction
Retrieving FAQ questions that match users' intent during user-system conversations is critical for on- line customer service in large companies (e.g., Al- ibaba, Amazon). In this paper, we call this sce- nario Conversational FAQ retrieval. Normally, it employs an Al assistant to model user behaviors (e.g., queries and clicks) and iteratively retrieve FAQ questions until meeting users' intent (Rano- liya et al., 2017; Vishwanathan et al., 2023). To sat- isfy user experience (Gao et al., 2022), is needed to design an efficient retrieval strategy to find FAQ questions that match user intent in minimal turns. Current methods focus modeling the seman- tie information in the conversation context. They model the semantic similarity to perform FAQ re- trieval by concatenating user queries and clicking questions (Rosset et al., 2020; Vishwanathan et al., 2023) or applying attention mechanisms (Li et al., 2019). Although these methods show promising results, they assume that users' behaviors strictly
We believe that the key to finding the right FAQ question in minimal turns is to reduce the impact of irrelevant tags in clicked questions. Thus, we need to estimate whether tag is irrelevant user intent. Accordingly, as shown in Figure. (b), we assume that a tag has a high probability of being irrelevant if the user seldom clicks FAQ questions containing this tag. Besides, as shown in Figure. (e), the probability of a tag being irrelevant can be gradually estimated along with the dynamics of the conversation. This motivates us to utilize reinforce- ment learning (RL) to model the dynamic changes of the tags' irrelevance estimation. By maximizing the cumulative reward based on the estimation, the RL model learns an optimized strategy to reduce the impact of irrelevant tags and obtain the user intent in minimal turns. More specifically, we con- vert the conversation context into representation
Corresponding author.
TTgs take forms of keywords or segments in questions.

---

FAQ question that fits the user intent
Previous Unelicked Questions
Tag-Aware Modeling
FAQ: s-10 RCul $t1 Erto st
(+)
FAQ: te t6 + e 48 15.07
[lo caneel the automatic payment of swving carde?
The didilo Tag
cancet aucoonatic repayment credit curda??
FAQ: of & I 1o st #2?
unclicked
automatic payment
(0)
payment membersiip cmbe?
(b)
Previous System
Our System
71 di, F. iE it &s ib 5-000
A+f 2.6 t it it bod8y
may
mone in my card sntematic deductoo??
Why money in me antonatic doductal??
nutomatic repayment, cancel.
FAQ: in PS Bt-f 96-6 i s04j 887
FAQ: te 35.88 F shit R2
click
xf
(low cancel the antomatic pay nyymento of bank carde
x
Low to cancel antomatic repayment of bank carde??
Click
f. 2h A. menn the credit
10 the credit
sect may dielike Tags antommalic paymeni, bank
EAQ:
in se-f8 f& it tiil 0 ti
EAQ: io froe s8 N 45 shaaddr
How cancel automatic povment creditt cards?"
[Low cancel outomatic repayment credit cards""
The use may like Tay- automatic repayment cancel, credit card
A MEM
Not
(c)
That's Thrnss
(d)
Figure I: An illustration of conversation FAQ retrieval in the finance domain. The red and purple texts describe the relevant and irrelevant tags in the context, respectively. Figure (a) shows the user's 's desired FAQ question. Figure (b) shows that the user's past unclicked questions contain the tag automatic payment. Figure (c), the user clicks a question containing automatic payment in the first turn, leading the system to retrieve irrelevant questions containing automatic payment in the next turn, resulting in user undissatisfaction. In Figure (e), our tag-aware modeling infers that the user doesn't 't like automatic payment according to the previous unclicked questions Figure (b). As a result. in Figure (d), our system doesn't retrieve the irrelevant FAQ questions containing automatic payment.
that models the dynamic tags' irrelevance estima- tion. When taking the representation as the state, we pumish the RL system when it retrieves FAQ questions containing irrelevant tags, and reward it when it retrieves users' desired FAQ questions in minimal turns. By doing so, the RL system can dynamically adjust its strategy to avoid retriev- ing questions that contain tags with high estimated irrelevance and achieve a successful retrieval as quickly as possible. In this way, we can effectively decrease the negative impact of irrelevant informa- tion in the conversation context, and retrieve users' desired FAQ questions in minimal turns. We call our method the Tag-aware conveRsational FAQ VetrieVal via Einforcement Learning (TRAVEL).
where q is a FAQ question with 0g as its answer. Each question qi is categorized into a set of tags Pa: = {P11. ..mm. For user ug there is question Q: that matches his/her intent, which needs to be retrieved. At the turn t of conver- sation, the system performs retrieval based on the conversation context. The conversation context at the turn t consists of: H,, which records the user's ss queries; Q' which records the questions that user clicked; Qroj. which records the questions that user ignore; The conversation ends if the system retrieves the question q:- Otherwise, it continues to retrieve until reaching the maximum turns T. Framework Overview. As depicted in Fig- ure 2, We formulate TRAVEL as a multi- turn tag- aware reinforcement learning framework, which aims to learn promising policy T¹ r(s), arg max r ax) Here, the action a indicates which FAQ question to retrieve from the candidates, and 8 capture the conversation con- text at the turn t. With the help of RI., TRAVEL can learn an efficient strategy to perform successful retrieval in minimal turns. Our TRAVEL contains two key components, i.e., Tag-Level State Rep- resentation and Conversational Retrieval Strategy Optimization. The former estimates irrelevant tags in the context and converts the conversation con- text into the state; The latter optimizes a retrieval strategy by RL given the state. Overall, we begin by elaborating on the RL environment setting of
To sum up, we have the following contributions: (1) For the first time, we point out the significance of reducing the impact of irrelevant information introduced by noisy user behaviors in Conversa- tional FAQ Retrieval. (2) We propose tag-aware reinforcement learning strategy that models the dy- namic changes of the tags' irrelevance to achieve successful FAQ retrieval in the minimal turn. (3) By developing new FAQ data, we test our method with intensive empirical studies. The results sup- port the efficiency and effectiveness of our method.
2 Framework
Notation. We denote the collection of question- answer pairs as FAQ = {()), a1), (gn, o₂)

---

Preferase Priiing
Aciia Space
Represccntttoo
Erahodiiigs
Astin
Newwrk
Pertereer hineteon
The Creven dien Contes:
Fnndins Quarteen
Clickoi Guedizes
Taast [Schatm]]
deainow Cliqss [Croda
Anten (Raurment
Repirescnaaiion
Tog
Aware Mediaminu
Wartu doil ard sper
Hos thdk prggroso spereex
Conversational Retrieval
FAt
Dynamic User Behaviors
Level State Representation
Stralegy Optimication
Figure 2: The overview of TRAVEL, for conversational FAQ retrieval.
TRAVEL in section 2.1. Then, in section 2.2,
positive reward when the user clicks. However, if clicked questions contain irrelevant tags, the value of this reward is reduced. (2) Telick_ fail+ a nega- tive reward when the user does not click any ques- tion. (3) Tret_suc- a strong positive reward when the user successfully obtains his target question, (4) Textro_tee Textro_turn, negative reward when the number of turns increases, (5) Tquit, a strong negative reward when reaching the maximum turns.
delve into the two key components of TRAVEL.
2.1 RL Environment Setting
We elaborate on how to formulate the conversa- tional FAQ task into RL. involves informing the system about the state and actions (i.e., questions to retrieve), transitioning between states, and pro- viding feedback-based rewards.
State. The state, formulated
{H's Outickr Pt Paick Orejz Pt. Prey), containing the conversation context up to the turn L. Here, D'tick is set of user-clicked questions, and pt click set tags associated with questions Q'diitt More- over, Qrej and rej are the questions that the user did not click and their corresponding tags.
2.2 Components of TRAVEL.
TRAVEL consists of two components: Tag-Level State Representation and Conversational Retrieval Strategy Optimization. The Tag-Level State Repre- sentation component focuses on estimating irrele- vant tags within the conversation context and trans- forming the context into the state representation. The Conversational Retrieval Strategy Optimiza- tion utilizes the state determine strategy for FAQ retrieval using Q-Learning, aiming to achieve accurate retrieval in minimal turns. It further en- hances the RL process by pruning the action space, following previous work (Lei et al., 2020b).
Action. Given the state, the system takes ac- tion a, by finding out which FAQ question should be retrieved from the candidate set Qcand. In prac- tice, we retrieve five questions at each turn. Transition. When the user clicks/ignores ques- tions or launches query at turn L, our conversation context changes. Namely, the state is updated to specifically, by adding the clicked gi and its tags to elick and Plick appending them into the Q++1 and pf+1 if the question is clicked. If the user launches a query at turn we add the new query into conversation history Ht Aware Reward. To achieve successful re- trieval in minimal turns, we reward the model when the retrieval succeeds and punish it when it fails or the turn number exceeds the maximum turns. It is also crucial inform models when irrelevant tags are introduced by noisy user behavior, such click- ing questions with irrelevant tags. enables mod- cls to adapt the strategy during conversations. In this paper, we propose five rewards: (1) Fetick_svi soc-
new state
2.2.1 Tag-Level State Representation
This section explains the estimation of irrelevant tags in the context and the modeling of the conver- sation context at the tag level to obtain the state rep- resentation. The process involves two steps. First, graph representations of questions at the tag level are obtained. This eliminates irrelevant semantics within questions and captures correlations between tags and questions. Then, tag-aware mechanism is proposed model the conversation context. It estimates irrelevant tags in the context and obtains the state in a fine- men-ggrined way. This state records the dynamically estimated irrelevance of tags.

---

Tag-Level Question Representation. To elimi- nate irrelevant information and utilize correlations between questions and tags, a graph structed with questions and tags as nodes. The node representations of a question and a tag Pi are obtained using TransE (Bordes et al., 2013), denoted as eq, and Ep.-
need strategy to retrieve the right question in min- imal turns. Thus, utilize the Ducling Q-network following previous work (Zhou et al., 2020). The Dueling Q-network is formulated as:
con-
ac) = for (5x) + for ai).
(5)
where foy (.) and for (.) are two separate multi- layer perceptions with parameters Oy and 0s. This equation takes the state S; and an action (the graph representation of a FAQ question) as inputs and pro- vides a score for that question. The score Q(s,, ax) represents the expected reward when taking the action a based on the state St. To maximize the cu- mulative expected reward, we follow the Bellman equation (Bellman and Kalaba, 1957):
Tag-Aware Mechanism. We employ the tag- aware mechanism is employed transform the conversation context into the state. This mecha- nism estimates whether a clicked question in the context contains irrelevant tags and calculates a weight to reflect this information. By doing so, the model becomes aware of the presence of irrelevant tags in the click question and implicitly eliminates them. The mechanism is defined as follows:
W. B
g1 = Eays1 51 + max Q' (c11ss (6)
8m
10teg
Con
Opn
nEQ".;
where ye denotes the (* ar). Since the reward is associated with retrieval accuracy, irrelevant tags information, and the number of turns as mentioned in section 2.1, maximizing the expected reward promotes the strategy to achieve accurate retrieval in minimal turns while avoiding retrieving other questions that contain irrelevant tags. Ultimately, for each action a,, which corresponds to an FAQ question, the system selects the FAQ question with the highest Q-value to retrieve.
where W,, Rdxd is trainable parameters. The vt is derived from unclicked questions and their tags that are ignored by users. This information captures tags information that contradicts users' intent, referred to as negative embedding. Further- more. given the graph embedding of the clicked questions O'ticck [equ eq:2...q.... we obtain the state representation as follows:
(3) Two-Step Pruning Strategy. The performance of reinforcement learning is compromised by large action space (Lei al., 2020a). Thus, we propose two pruning strategies to shrink the action space. (1) We utilize semantic similarity pruning to shrink the action space following baselines (Vish- wanathan et al., 2023). It models the semantics association between queries and FAQ questions. We select top-k, FAQ questions with the highest se- mantic scores, forming the candidate set Q Qoand sim' (2) Then, we use tag preference pruning to fur- ther reduce the action space. We choose top-k, questions with the highest score form D'annd pre following the formula in Appendix A.5.
Oka Coa
eap (h (W (vál eg.)))
On
(4)
> eap ooo (w (ville,,)))
where hT and W are trainable metries.
The S, is calculated by combining the clicked question embeddings ey.,. using weights. Each weight On is determined based on the score be- tween v and clicked question embedding eg.. higher weight indicates the presence of more irrel- evant tags in the question within the context.
It's important to note that although this weight is applied to questions, is implicitly mapped onto the corresponding tags of questions through the graph representation. Using the weight contained in S; signal, our model can implicitly eliminate irrelevant information in the context.
Experiments
This paper focuses proposing strategy to trieve the appropriate FAQ question within min- imal number of turns. Consequently, our first research question investigates whether TRAVEL. can outperform the baselines (including ChatGPT) given the limited turns. Subsequently, we explore
2.2.2 Conversational Retrieval Strategy Optimization
Given the state st contains the information about the conversation context and irrelevant tags, we

---

whether TRAVEL can maintain stable and supe- rior performance in the face of different levels of noisy user behavior. Furthermore, we validate the efficiency of TRAVEL's individual components, en- suring their impact on performance. Finally, we verified if the tag-awe awgr mechanism can actually estimate whether and how many irrelevant tags clicked question contains in the context. The above research questions are follows:
click items relevant to their intent based on their interests (defined by their profiles). Therefore, the probability of user u, clicking a FAQ question Qk is calculated as follows:
r (1s, 9x) = =0*Tk: Pki + 0.) Chi.
Oki = f (qk, e.),
(8)
where: (1) Thi represents the probability of user "j clicking qh based on relevance to their target question 4i- We define relevance as the number of overlapping tags between O: and the user's tar- get 9:- The click probabilities for different levels of relevance are derived from online statistics, presented in Appendix A.4. (2) Cki represents the probability of user clicking d based on their inter- est profile e; It is modeled as Ci: - (gk, ei). We train the function f (-) using online data. Details of the hyperparameter a are in Appendix A.I.
RQI:
Can TRAVEL outperform baselines in achieving more accurate retrieval given the limited number of turns?
RQ2: How does our method perform in the presence of different levels of noisy user be- haviors compared to baselines?
RQ3: What the impact of each component of TRAVEL on its performance?
3.2.2 Baseline
Can the tag-aware mechanism effec- tively estimate the presence and quantity of irrelevant tags the clicked questions contain?
RQ4:
We compare the TRAVEL with two classes of base- lines methods (comparisons with ChatGPT are in Appendix A.2). The first is FAQ retrieval which represents the standard way to retrieve FAQ ques- tions. The second is Question Suggestion which is used web searches to predict the next ques- tion users may ask, which has a similarity to FAQ retrieval in the form of the task.
3.1 Dataset
We conduct experiments our proposed data, since existing FAQ datasets do not contain the ques- tion tags and conversation history. Our dataset con- tains 72,013 conversation sessions. Each session formulated as (u4, qt, Hi), where qi represented the user's intended question, H; contains the conversa- tion history between the user and system, and uj denoted the user along with their profile. There are 1449 FAQ questions and 1201 tags in the dataset. On average, each question contains 6 tags. See Appendix A.6 for more details of the dataset.
FAQ retrieval: 1) _ERT_TSUBAK TSUBAKI (Sakata et al., 2019)employs BERT to compute scores be- tween queries and FAQ answers, and uses BM25 to compute scores between queries and FAQ ques- tions; 2) SBERT [BERT_FAQ (Vishwanathan al., 2023) is a sune-tuned BERT model optimized with triplet loss using FAQ questions; 3) DoQA (Campos et al., 2020) is a baseline for conversation- tion answering that utilizes only the first turn of the query; 4) CombSum (Mass et al., 2020) is a state-of ffate-of-the-art FAQ retrieval method that calcu- lates scores between the query and the question us- ing both BM25 and BERT, and the score between the query and the answer using BERT; Question Suggestion: 5) CFAN (Li et al., 2019) is a multi- turn question suggestion method that takes queries and clicked questions as input; 6) KnowledgeSelect (Kim et al., 2020) is a multi-turn retrieval method in conversational settings that utilizes BERT to model the multi-turn user queries; DeepSuggest (Keyvan and Huang. 2022; Rosset et al., 2020) a standard question suggestion method for conversa- tions that incorporates multi-turn user queries and clicked questions using BERT.
ques-
3.2 Experimental Settings
3.2.1 User Simulator
Due to the interactive nature, online experiments where the system interacts with real users and learns from their behaviors would be ideal. How- ever, the trial-and-erros strategy for training RL. (Zhou et al., 2020) online would degrade the user's experience and the system's 's profit. Thus, following Lei et al. (2020c); Zhang et al. (2021), we simulate users' behaviors via a simulator.
Given a user u, and their target question @i we simulate their click behaviors. Users" clicking be- havior influenced by two main factors: their in- tent (target) (Zhang et al., 2021) and their profile (Zhou et al., 2020). Specifically, users are inclined

---

Table 1: The comparison among different conversation methods. Recall, SR and hDCG are the higher the better, while AT and AS are the lower the better. The comparison with ChatGPT is in the Appendix A.2.
Recall
SR@2
SR@3
SR@4
SR@5
AT
AS
hNDCG
BERT BERT_TSURAL TSUBAKI
0.3463
0.4461
0.5053
0.5496
0.5850
3.1526
15.7630
0.4304
SBERT FAQ
0.5298
0.5569
0.5696
0.5824
0.5940
2.7513
13.7565
0.5167
DoQA
0.5212
0.5545
0.5796
0.6051
0.6254
2.7395
13.6977
0.5204
CombSum
0,5179
0.5788
0.6141
0.6449
0.6722
2.6443
13.2213
0.5515
KnowledgeSeleet
0.4265
0.5252
0.5825
0.6274
0.6630
2.8385
14.1925
0.5043
CFAN
0.5213
0.5690
0.6145
0.6531
0.6864
2.6422
13.2109
0.5481
DeepSuggest
0.5398
0,6068
0.6711
0.7203
0.7392
2,4620
12.3101
0.5956
TRAVEL
0.5398
0.6179
0.6922
0.7586
0.8093
2.3915
11.9574
0.6177
3.2.3 Evaluation Metrics
ing more accurate retrieval within limited turns. The results are presented in Table 1.2 The result indicates that TRAVEL outperforms all baselines. Analyzing Table reveals that while FAQ re- trieval methods can deliver satisfactory results in the initial turn (Recall@5), their performance noticeably declines in subsequent retrieval turns (S R@k) compared to TRAVEL. TRAVEL, on the other hand, outperforms these methods by achiev- ing 13.71% higher SR@5 and 25. 25.28% lower AT compared CombSum. This highlights the limitations of existing FAQ retrieval methods in adequately capturing and utilizing the multi-turn conversation context. These methods require nu- merous turns for successful retrieval.
Firstly, we evaluate the retrieval performance in the first turn using Recall@5. To assess the sys- tem's ability to successfully retrieve users' target questions within the limited & turns, utilize the metric success rale (SR@K) (Lei et al., 2020c). To measure the average number of turns the model takes in conversations, we use AT(Average Turn) (Lei et al., 2020c), and to evaluate the retrieval ranking performance, we employ hNDCG@(1 T, K) (Deng et al., 2021). Furthermore, considering the importance of user experience, we limit the expo- sure of users to large number of questions, which could be burdensome. Hence, introduce the Av-
erage Shown (AS) quantify the average number of FAQ questions seen by users.
Furthermore, compared to existing question sug- gestion methods, TRAVEL also demonstrates supe- rior performance. For instance, when compared to CFAN and DeepSuggest, which leverage multi-turn user queries and click questions, TRAVEL achieves 12. 12.29% 22.% /5.1/ % higher S R and 25. 25.07% 77.0%% lower AT, respectively. Additionally, TRAVEL outperforms these methods in terms of ranking ef- fectiveness, with a 6.96% % higher hN DCG. These finds indicate that TRAVEL excels in rank- ing FAQ questions that align with the user's in- tent. The aforementioned experiments effectively demonstrate that TRAVEL outperforms state-of- the-art art question suggestion methods by effectively utilizing the conversation context for FAQ retrieval.
3.2.4 Implementation Details
We split the dataset by 4:1:1 for training, validation, and testing and set the number of retrieved ques- tions at each turn as 5. We set the maximum turn T as 10 during training. We set the k; of similarity- based as 100 and the k, of preference-based prun- ing 10. We set the graph embedding size 40. During the training procedure of DQN, the size of the experience buffer is 50000, and the sample size is 32. The learning rate is set to be le-4 with Adam optimizer. The discount factor 7 is set to be 0.99. We adopt the reward settings to train the proposed method: Telik_ suc 0.03 # m, Telic__ fast = -011, Tret_sue 1, Fextrr_fawr
-0.05, Tquit = 0.3. where denotes the number of tags that coincide between the clicked question and the user's tar- get question. When the user clicks question that introduces noise, the m is set to a minor value.
3.4 The Noise Robust Testing (RQ2)
In this section, evaluate the stable performance of TRAVEL in comparison to baselines across vary- ing levels of noisy user behaviors. The results in- dicate that TRAVEL consistently maintains stable
3.3
The overall comparison among different methods (RQ1)
'TRAVEL and DeepSuggest have consistent 5 This is because there are no clicks in the first turn, so TRAVEL employs the same model as DeepSuggest. It can also use other models that have higher recall, which is not our focus.
This section presents the superior performance of TRAVEL compared to baselines in terms of achiev-

---

Table 2: The ablation study. Recall, SR, and hDCG are the higher the better, while AT and AS are lower the better.
SR@2
SR@3
SR@4
SR@5
AT
AS
hNDCG
(a) w/o semantic similarity pruning
0,5802
0.6190
0.6555
0.6847
2.6054
13.0272
0,5589
(b) w/o tag preference pruning
0.5966
0.6503
0.6931
0.7271
2.5202
12.6012
0.5798
(c) w/o tag-aware mechanism
0.6113
0.6852
0.7525
0.8034
2.4113
12.0565
0.6140
(d) w/o tag-level question representation
0.5565
0.5700
0.5826
0.5941
2.7511
13.7556
0.5167
(e) w/o noise-aware reward
0.6149
0.6879
0.7555
0.8025
2,4018
12.0090
0.6140
TRAVEL
0.6179
0.6922
0.7586
0.8093
2.3915
11.9574
0.6177
and superior performance. To investigate this, we vary the weight a of the user simulator, setting it to 0.2, 0.5, 0.8, and I. Appendix A.I provides detailed experimental results that demonstrate how the user simulator deviates increasingly from real user be- havior as the weights shift. This process can be considered manually introducing noise. As illus- trated in Figure 3, TRAVEL consistently achieves superior performance compared to the baselines, with an improvement of 4% on SR@5, even when different levels of noise are introduced. In contrast, the performance of the baselines shows consider- able fluctuations as the noise values change. No- tably, DeepSuggest experiences decrease of 5% in @R@5 when the weight transitions from 0.8 to 0.5. In summary, this experiment provides com- pelling evidence of TRAVEL's capability to effec- tively handle conversation contexts in the presence of varying levels of user-introduced noise.
the tag-level question representation leads to a sig- nificant decrease performance across all metrics. For instance, the $ RR5 drops by 21.52%. This illustrates the importance of modeling the ques- tion representation at the tag level. Additionally, removing the tag-aware mechanism (Table 2 row (c)) results in 5.9% in SR@5. This highlights the benefit of incorporating the tag-aware mechanism. worth noting that the improvement brings by the tag-aware mechanism is smaller than that of the tag-level representation. This is because the tag-level representation has already captured the user's possible preferences for tags via the graph.
3.5.2 Noise-Aware Reward
Next, we evaluate the effectiveness of the noise- aware reward Tolick_swe (section 3.1). The experi- ment demonstrates that the reward setting enhances the performance of TRAVEL Specificaly, we per- form the ablation study by setting all clicked be- haviors to receive the same reward. Table 2 row (e) illustrates that replacing the noise-aware reward leads to 6.8% decrease in SR05. This demon- strates the significance of the noise-aware reward.
Travel
0.9
DeepSuggest
CFAN
3.5.3 Two-Step Pruning
The top part in Table 2 (rows(a-b)) presents the results when the proposed pruning strategics are omitted. Notably. all metrics experience notice- able decline when discarding them. Specifically, without semantic similarity pruning, the SR @5 decreases by 12.46%. Similarly, omitting the tag preference pruning leads to an 8.22% decrease in SR@5. These findings emphasize the critical role of utilizing semantic and tag information for ef- fective pruning. which significantly enhances the performance of reinforcement learning (RL).
0.7
0.6
0.2
0.5
0.8
Figure 3: We observed the SR05 (y-axis) for different a (x-axis), When is 0.8, the noise is the lowest.
3.5
Ablation Study (RQ3)
3.5.1
Tag-Level State Representation
We assess the effectiveness of modeling the conver- sation context the tag level, specifically through the tag-level question representation and tag-aware mechanism. The experiments show the important role of these two components the model perfor- mance, especially for the legel question repre- sentation. As shown in Table row (d), removing
3.6 Validation of the Mechanism (RQ4)
Attention
In this section, we aim to verify the effectiveness of the tag-aware mechanism in estimating the pres- ence and quantity of irrelevant tags in clicked ques-

---

tions. As explained in Section 2.2.1, the tag-aware attention mechanism is designed to assign higher weight to inform the model when a clicked question contains many irrelevant tags.
user experience in real-world applications. FAQ retrieval in conversation aims perform success- ful retrieval very few interactions. Technically, FAQ retrieval is achieved by modeling the semantic association between user queries and FAQ ques- tions. Existing works use convolution neural net- works (Karan and Snajder, 2016), long short-term memory (Gupta and Carvalho, 2019) pre-trained language models to model the semantic similarity (Sakata et al., 2019; Zhang et al., 2020). To fully utilize the conversation context, some web question suggestion methods (Li et al., 2019) incorporate the user-clicked questions with queries as the input of the semantic model using concatenation or at- tention. However, existing methods mainly model the conversation context from the semantic level, instead of modeling user behavior through policy network. Meanwhile, existing datasets (Karan and Snajder, 2016; Sakata et al., 2019; Zhang et al., 2020) do not contain tags or conversation history.
To verify this, we compared the weight value of the Tag-Awar Mechanism in formula (3) with the weight value obtained from a fine-tuned sen- tence BERT. The sentence BERT takes the sen- tences of the clicked questions and the user's target question as input and oulputs a score indicating the level of irrelevant information in the clicked questions. As depicted in Figure 4, the x-axis rep- resents the degree of irrelevant tags in clicked ques- tions of the context, where a larger value indicates fewer relevant tags and more irrelevant information. The comparison shows that tag-level modeling per- forms better in identifying the degree of irrelevant tags in the context. Specifically, as the degree in- creases from to 10, the weight value obtained from the tag-aware mechanism exhibits a change 24. 12%. In contrast, the weight value obtained from the sentence BERT model only changes by 7.37%. This clearly demonstrates that our mecha- nism can better estimate the number of irrelevant tags in clicked questions compared to previous methods. These findings highlight the superiority of the tag-aware mechanism in accurately identify- ing irrelevant tags within the conversation context. This mechanism provides explanatory power for our approach and underscores its effectiveness.
5 Conclusion
In this paper, we focus on proposing a strategy for retrieving FAQ questions that match user intent in minimal turns. To achieve this goal, we introduce a tag-aware conversational FAQ retrieval framework via reinforcement learning called TRAVEL. This framework is designed to eliminate the detrimental impact of irrelevant information in the conversation context. It contains two main components: Tag- Level State Representation and Conversational Re- trieval Strategy Optimization. TRAVEL mitigates the negative impact of irrelevant information in the context by estimating its degree of irrelevance and employs reinforcement learning strategy for per- forming FAQ retrieval. To verify ideas, cre- a dataset and develop user simulator. Through extensive experiments, we justify the effectiveness of the TRAVEL framework, offering valuable in- sights into achieving successful FAQ retrieval in the fewest possible turns.
Ing. Level Weight
0.6
Seutence- Level weight
Weight Value
0.4
10
Figure 4: We observed the weight value (y-axis) for different levels of irrelevant information (x-axis).
Limitation
Due to the interactive nature of our framework, on- line experiments where the system interacts with real users and learns from their behaviors would be ideal. However, the trial-and-error strategy for training RL online would degrade the user's experi- ence and the system's 's profit. Therefore, we propose a user simulator to conduct the experiments offline. Even if we have proven the authenticity of the user
4 Related Work
FAQ retrieval has broad applications in conversa- tion such conversational QA systems (Campos al., 2020) and dialogue systems (Vishwanathan et al., 2023; Zhu et al., 2019). It also widely used for online customer service in large companies such as Alibaba, Amazon, and Google. To satisfy

---

simulator, this still leaves us with a gap from the real scenario.
International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 929-932.
7 Ethics Statement
Mladen Karan and Jan Snajder. 2016. Faqir-a fre- quently asked questions retrieval test collection. In Text, Speech, and Dialogue: 19th International Con- ference, TSD 2016, Brno, Czech Republic, September 12-16, 2016, Proceedings 19, pages 74-81. Springer.
This paper presents a conversation FAQ retrieval framework with a new dataset and user simula- tor. Although our datasets are collected from an e-commerce company, they are designed for nor- mal users and have been widely used by the public for some time. We also have carefully checked our dataset to make sure they don't 't contain any person ally identifiable information or sensitive personally identifiable information (the user interest profile in Section 3.2.1 has been successfully desensitized). Thus, we believe there are no privacy concerns.
Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches. tools, and challenges. ACM Computing Surveys, 55(6):1- 40.
Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia. Yang Liu, and Dilek Hakkani- Tur. 2020. Beyond domain apis: Task-oriented con- versational modeling with unstructured knowledge access. arXiv preprint ar arxiv:0066.00 200.03300...3333
Acknowledgement
W. Lei, G. Zhang, X. He, Y. Miao, X. Wang, L. Chen, and T. S. Chua. 2020a. Interactive path reasoning graph for conversational recommendation. ACM.
This work was supported in part by the Na- tional Natural Science Foundation of China (No. 62272330); in part by the Fundamental Research Funds for the Central Universities (No. YJ202219).
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong. Min- Kan, and Tat-Seng Chua. 2020b. Estimation-actio aettaaactiiioneeeee reflection: Towards deep interaction between conversational and recom- mender systems. In Proceedings of the 13th Interna- rional Conference on Web Search and Data Mining, pages 304-312.
References
Richard Bellman and Robert Kalaba. 1957. On the role of dynamic programming in statistical communica- tion theory. IRE Transactions Information Theory. 3333:197-203.
Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang. Liang Chen, and Tat-Seng Chua. 2020c. Interactive path reasoning on graph for con- versational recommendation. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2073- 2083.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data, Advances in neural information pro- cessing systems, 26.
Ruirui Li, Liangda Li, Xian Wu, Yunhong Zhou, and Wei Wang. 2019. Click feedback-aware query recom- mendation using adversarial examples. In The World Wide Web Conference, pages 2978-2984.
Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan De- riu, Mark Cieliebak, and Eneko Agirre. 2020. Doqa- accessing Nomain-specinf faqs via conversational qa. arXiv preprint arxiv:2005.01328.
Yosi Mass. Boaz Carmeli, Haggai Roitman, and David Konopnicki. 2020. Unsupervised faq retrieval with question generation and bert. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 807-812.
Yang Deng, Yaliang Li, Fei Sun. Bolin Ding, and Wai Lam. 2021. Unified conversational recommendation policy learning via graph-based reinforcement learn- ing. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1431-1441.
Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(1 (10)2205.
Joseph L Fleiss and Jacob Cohen. 1973, The equiva-
Ience of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3)6613-619
Bhavika R Ranoliya, Nidhi Raghuwanshi, and Sanjay Singh. 2017. Chatbot for university related faqs. In 2017 International Conference on Advances in Com puting, Communications and Informatics (ICACCI). pages 1525-1530. IEEE.
J. Gao, C. Xiong, P. Bennett, and N. Craswell. 2022. Neural approaches to conversational information re- trieval.
M Romero, Alejandro Moreo, and Juan Luis Castro. 2013. A cloud of faq: A precie fag retrieval system for the web 2.0. Knowledge-Baud Based Systems. 49:81-96.
Sparsh Gupta and Vitor R Carvalho, 2019. Faq retrieval using attentive matching. In Proceedings of the 42nd

---

Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Cam- pos, Nick Craswell, Saurabh Tiwary, and Paul Ben- nett. 2020. Leading conversational search by sug- gesting useful questions. In Proceedings of the web conference 2020, pages 1160-1170.
(6). As shown in Figure 5, we observed that the highest level of consistency between the simulator and the users the test set was achieved when a was set to 0.8. Moreover, across different values of o the simulator consistently exhibited high lev- els of consistency (above 0.7). These experiments provide compelling evidence supporting the relia- bility of our simulator, indicating that it accurately emulates user behavior in the given context.
Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. Faq retrieval using query- question similarity and bert-based query-anse answer rele- vance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1113-1116.
Asha Vishwanathan, Rajeev Unnikrishnan Warrier, Gau- tham Vadakkekara Suresh, and Chandra Shekhar Kandpal. 2023. Multi-tenan optimization for few- shot task-oriented faq retrieval. arXiv preprint 1rXiv223011055/1
0.9
0.7
0.5
Lili Yu, Howard Chen, Sida Wang. Tao Lei, and Yoav Artzi. 2019. Interactive classification by asking infor- mative questions. arXiv preprint arXiv: 1XI1:1911.00590
0.3
0.8
Xinliang Frederick Zhang. Heming Sun, Xiang Yue, Si- mon Lin, and Huan Sun. 2020. Cough: A challenge dataset and models for covid-19 19 faq retrieval. arXiv preprint arXiv:2010.1 12800.
Figure 5: We observed the consistency (y-axis) between the simulator and the real user click behavior for differ- ent weights a (x-axis).
Yiming Zhang. Lingfei Wu, Qi Shen, Yitong Pang. Zhi- hua Wei, Fangli Xu, Bo Long, and Jian Pei. 2021. Multi-choice questions based mulli-interess policy learning for conversational recommendation. arXiv preprint 2xXiv:212.11177
A.2 The Comparison with ChatGPT
With ChatGPT's rapid development, we are curi- ous about how well it would work in our scenarios. Due to the time-consuning inference of ChatGPT and the large length of the conversation, we did not test it on the complete testing set. We compare the performance of our method and ChatGPT on 1000 random samples from the complete testing set. It is worth noting that due to the limited input length of ChatGPT, we cannot feed all candidate questions prompts to ChatGPT. Instead, only the 50 questions recalled by BM25 were used as input to ChatGPT. At the same time, we do an operation in favor of ChatGPT; that is, there must be the ground truth question in the recalled questions. This prevents ChatGPT from losing performance due to inaccu- rate BM25 recall. The experimental results are in Table 3. It shows that TRAVEL achieves better results (22.87% higher on Recall@5 and 12.83% higher on SR@5) compared to ChatGpt. This may be due to ChatGPT's lack of domain-specific a priori knowledge, which causes it to remain inade- quate in domain-specific information retrieval.
Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang. Xiugiang He, and Yong Yu. 2020. Interactive recommender system via knowl- edge graph-enter -nhannee reinforcement learning. In Pro- ceedings of the 43rd international ACM SIGIR con- ference on research and development in information retrieval, pages 179-188.
Lin Zhu, Xinnan Dai, Qihao Huang. Hai Xiang, and Jie Zheng. 2019. Topic judgment helps question similarity prediction in medical faq dialogue systems. In 2019 International Conference on Data Mining Workshops (ICDMW), pages 966-972. IEEE
A Appendices
A.1 Evaluation of the User Simulator
Through experiments, we validate the authenticity of the user simulator and determine the value of & that best aligns with real user behaviors in formula (6). In this section, we assessed the fidelity of our simulator by comparing its behavior with actual user behaviors. We built a test set using online user click data, which included information on users and their clicked and unclicked FAQ questions. We then examined whether the simulator exhibited con- sistent click behavior with the users in this test set while varying the weight values (a) in the formula
A.3 Parameter Sensitivity Analysis
The upper part of Table 4 summarizes the exper- imental results (SR) by varying the kg, which de- notes the size of the semantic similarity pruning. When the k2 is 50, the performance essentially de-

---

Table 3: The performance comparison between the Chat- GPT and TRAVEL.
Table 5: Corresponding click probabilities under differ- tag overlap degrees.
@
SR@5
hNDCG
Tag
0%
05--20%%
20%%
ChatGPT 0.3133
0.7207
0.4708
Probability
0.100
0,145
0.720
TRAVEL
0.5420
0.8490
0.6482
Tag 40%-60%
60%-80%
10%%-000%
Probability
0.900
0.960
0.990
Table 4: The effect of hyper-parameters.
A.5 Tag Preference Pruning
SR@2
@
SR@4
SR@5
0.5740
0.6204
0.6686
0.7178
The score of tag preference pruming is calculated by a (8:), where 8; is determined by:
k. 100
0.6179
0.6922
0.7586
0.8093
& 150
0.6014
0.6685
0.7329
0.7832
0.6179
0.6922
0.7586
0.8093
T 6ug On C.
Σ
Σ The Potcok
Σ Oen Ene
T Cipy Egm (9)
& 15
0.6098
0.6813
0.7430
0.7931
Ciyy
Cp.
& 20
0.6100
0.6762
0.7393
0.7820
5
0.6127
0.6880
0.7512
0.8025
Here, Cq. represents the graph embedding of the clicked question, Epr, represents the embedding of the corresponding tag of Cqu, and 6pm represents the embedding of the question the user didn't t click.
T' === 10
0.6179
0.6922
0.7586
0.8093
T 15
0.6136
0.6919
0.7578
0.8106
creases. This may be because the similarity model is limited by ambiguous queries, resulting in some ground-truth FAQ questions being over-filtered. When the Kg is 150, the performance also decreases. This is because the pruning might be ineffective when the pruning size is too large. The middle part of Table 4 summarizes the results by varying the ky, which denotes the size of the preference-bace pruning. The performance is better when we set the ky small value of 10. This result demonstrates the effectiveness of using tag preference pruning. The lower part shows the experimental results by varying the maximum turn T during training. Al- though the maximum turn T is set to 5 during test- ing, the performance is better when the T is set to 10 during training. This is probably because the DQN can learn a better strategy when trained in long conversation session. The performance decreases when the T is set to a more significant value, such as 15. This may be because of the large gap in the conversation environment between train- ing and testing when the length of the conversation session varies greatly.
A.6 Data Collection
Considering that existing FAQ datasets lack con- versation history and question tags, to verify our studies, we propose a new dataset with the support of large Chinese financial enterprise. We built this dataset in three steps. Firstly, we generated the FAQ questions. Next, we assigned tags to each FAQ question. Finally, we assembled conversa- tion sessions, including user profiles and conver- sation histories, to create the training and testing sets. This table shows the statistics of our dataset. Specifically, the dataset has 1449 questions and 1201 tags. On average, there are 6 tags per ques- tion. The dataset contains 65, 100 users and their conversation histories. In total, we collected 72,013 conversation histories. The average length of each conversation four turns.
Table 6: Summary statistics of datasets.
User
Conversation
Question
Tag
65100
72013
1449
1201
A.6.1 Questions Collection
A.4 The Click Probabilities Table
We collect massive user questions and cluster them using the HDBSCAN algorithm (McInnes et al., 2017). Each cluster was then reviewed by domain experts who selected 10-15 representative ques- tions form the set of FAQ questions. The final collection consists 1449 user questions, covering the majority of user intentions in the given domain.
This section shows the correlation between the probability of user clicking and the tag overlap. which is calculated from online data mentioned in Appendix A.4. The results are shown in Table 5. is important to note that this probability value only factors as part of the user simulator behavior.

---

A.6.2 Tags Labeling
For tag labeling, engaged three domain experis pre-define the ontology of tags, which included business objects and user actions (e.g.. business objects and user actions such as "Product" and "In- tent" in Table 7). Using this ontology. we were able to assign multiple detailed tag values to each user question as shown in Table 7. To annotate the tag values, we initially assigned 20 crowd-workers to independently label ten randomly chosen ques- tions. If the questions were simple had few tags, another set of questions was provided. Af- ter the initial annotation, the workers resolved any disagreements and revised the annotation scheme. Subsequently, they collaboratively annotated an additional 30 questions, resulting in a high Fleiss kappa score of 0.691 (Fleiss and Cohen, 1973). Finally, these annotators proceeded to label the remaining questions, resulting in a high-quality corpus of 1449 questions with 1201 tag values.
Table An example of labeling the tags of a question.
What should I do if my credit card is overdue?
Product
Movement
Intent
Credit Card
Overdue
Solution
A.6.3 Conversation Collection
Once the constructed FAQ questions were online, we collect 72013 conversation session data for training and testing. Each session is formulated (u4, 4;, H2), where qi represented the user's in- tended question, H, contains the conversation his- tory between the user and system, and uq denoted the user along with their profile. Specifically, to determine the target question d for each user, we gathered the FAQ questions they had clicked on from the online data, which we formulated as Qetick = {q,, J. Among these clicked questions Qetick. we obtain qi based on the follow- ing rules: (a) The user terminated the conversation after clicking Z- (b) The user refrained from conducting further searches for certain period of time following the click. (c) The user did not seek assistance from human customer service after the search session. Once we obtained q. we conducted a manual review to ensure its quality.


Reviews:
[{"review_id": "jKf8mnBQkQ", "reviewer": "Reviewer_M6qM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions. We will address your concerns point by point.\n\n$\\text{\\color{blue} 1. Paper is an application of reinforcement learning ... not strong on novelty aspects ....}$\n\n**Answer:** We suppose that the reviewer may have misunderstood our contribution and novelty. We want to emphasize that our main contribution is to minimize the impact of irrelevant information from noisy user behaviors, thereby enhancing retrieval efficiency, as explained in line 104. This is a crucial but poorly addressed challenge in the current FAQ retrieval community. RL is a suitable method to help us address an issue in this challenge and can be replaced by other frameworks.\n\n$\\text{\\color{blue} 2. This paper ... should conduct evaluation ... in a real-time application.}$ \n\n**Answer:** We follow the existing studies that employ a user simulator for evaluation, which is a well-established paradigm (Lei et al. (2020a); Zhang et al. (2021)). \nIt has good reproducibility and enables us to have fair comparisons with existing methods that use simulators. \nWe will try to conduct a real-time application in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel application of reinforcement learning framework in the relevant FAQ retrieval.\n\nAuthors carefully design state action and reward space.\nAuthors also conduct thorough evaluation of the proposed system.", "reasons_to_reject": "Paper is an application of reinforcement learning framework to the problem of FAQ retrieval.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents TRAVEL, a reinforcement learning based approach to retrieve FAQs most relevant to current conversational context of user.\nAuthors systematically study the impact of each component of TRAVEL.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8lub2Q5Ktf", "reviewer": "Reviewer_TqE8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and suggestions. We will address your concerns point by point.\n\n$\\text{\\color{blue} 1. Why introduce a tag ... The motivation for the tag is not well explained ...}$\n\n**Answer:** **(1)** Tags help estimate the irrelevant information in clicked questions, and **(2)** help identify whether a FAQ question matches the user intent. Related discussion can be found in lines 61-66.\n\nMore specifically, it is crucial to have an efficient strategy to find FAQ questions that match user intent in minimal turns, especially when user behaviors could be noisy (Line 40-55). The noisy behaviors may involve irrelevant information that disrupts the user intent modeling. The irrelevant information often takes the form of keywords in clicked questions, which we refer them as **\"tags\"** following (Yu et al., 2019; Romero et al., 2013). For example, due to domain unfamiliarity (Keyvan and Huang, 2022), a user might click the question \"How to freeze the credit card\" while their actual intent is \"How to close the credit card\". This mistakenly clicked question contains the irrelevant tag \"freeze\", which could **disrupt the subsequent retrieval**, as shown in line 50. Besides, we found that in **47\\%** of the conversations, users clicked questions that contained irrelevant tags. Consequently, there arises a need to estimate the relevance of tags within clicked questions and reduce the impact of those irrelevant tags.\n\nWe statistically analyzed the connection between online user click patterns and tags, as shown in the table below. We first counted the number of each tag contained in the user click question. Then, we calculated the probability of tags being relevant to user intent under different numbers. As shown in the Table, **the probability of the tag being relevant is directly proportional to the number of clicked questions that contain this tag.** \n\n| The Number of clicked questions contain this tag      | 1    | 2    | 3    | 4    | 5    | \n| --------------------- | ---- | ---- | ---- | ---- | ---- |\n| Relevance probability of this tag | 0.55 | 0.67 | 0.75 | 0.80 | 0.85 |\n\n\nThis observation grants us two advantages in utilizing tags:\n\n**(1) By combining tags and user behaviors, we can better estimate the irrelevant information in clicked questions.** In particular, if a user's historical clicked questions rarely contain a specific tag, this tag can be regard as irrelevant information in the current clicked question. \n\n**(2) Tags help identify whether a FAQ question matches the user intent.** A user who frequently clicks questions containing a specific tag is likely to have that tag relevant to his/her intent. Consequently, other questions that contain that tag could also match the user intent.\n\nBuilding upon this observation, we propose the tag-aware RL framework to model the dynamic changes of the tags' irrelevance estimation. We also experimentally demonstrated the effectiveness of our framework.\n\n$\\text{\\color{blue} 2. The method ... tested on one dataset ...not clear about its generality.}$ \n\n**Answer:** We focus on the scenario of conversational FAQ retrieval, i.e., retrieving FAQ questions that match users’ intent during user-system conversations. This is prevalent in online services like XiaoAi, Siri, and Alexa [2,3,4].\n\nAfter investigation and research, we noticed that existing FAQ datasets lack tag and conversation histories (Karan and Šnajder, 2016; Sakata et al., 2019; Zhang et al., 2020), which makes them unavailable for conversation FAQ retrieval research.\nHence, we took the first step to annotate a conversational FAQ dataset derived from the financial domain, as detailed in Appendices A.6. Notably, the task of data annotation is widely recognized as labor-intensive [1], which is why we had to test our method on one dataset. It's worth noting that our method was not specifically for the financial domain. It could be adapted to other domain datasets if available.\n\nIn addition, we have described the details of our method (cf.Section 2.2) and the dataset construction (cf. Appendices). Thus, our method is clear and reproducible. We will also provide more comprehensive descriptions of tables and figures directly in our captions.\n\n**Reference:**\n\n\n[1] Alexander Sorokin and David Forsyth. 2008. Utility data annotation with amazon mechanical turk. In 2008 IEEE computer society conference on computervision and pattern recognition workshops, pages 1–8.IEEE.\n\n[2] https://xiaoai.mi.com/\n\n[3] www.siri.com/\n\n[4] https://www.alexa.com/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of this paper is very interesting and promising. They want to better align the semantics between queries and retrieved answers.", "reasons_to_reject": "1. The main issue this paper wants to address is the utilization of conversation data when retrieved answers fail to satisfy the user's query. They introduce a tag for each retrieved answer. But the connection between the problem they want to solve and the solution is not very clear. Why introducing a tag can better align the user's query to retrieved answers? The motivation for the tag is not very well explained. The paper is written in a way that the tag is the only extra information they have, they use it and it performs better.\n\n2. The method is only tested on one dataset, which is not clear about its generality.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a tag-aware conversational FAQ retrieval method. It proposed to use tags of FAQ questions to eliminate irrelevant semantics.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be better to provide more comprehensive descriptions of tables and figures directly in their captions."}, "extra_text": {}}, {"review_id": "o0T45oE7Vb", "reviewer": "Reviewer_LGK8", "summary": "", "questions": "Q1) Could you discuss the prospects for extending this framework to more diverse FAQ datasets from different domains? What adaptations maybe needed?\n\nQ2) Can you provide more analysis of how the tag relevance estimates change over the course of the conversational episodes? Some examples could be insightful.\n\nQ3) Have you considered deploying this in a real interactive setting with humans for more rigorous evaluation? What would be the challenges?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We will address your questions point by point.\n\n$\\text{\\color{blue} Q1: Could you discuss ... extending this framework to FAQ datasets from different domains? What adaptations may be needed?}$\n\n**Answer:** It is not difficult to extend TRAVEL to different domains. These adaptations are at the data level and revolve around the construction of domain-specific datasets. This process involves collecting conversational data and annotating tags for FAQ questions. Obtaining the former data can be achieved through collaboration with various companies or online services, whereas the latter task can be efficiently completed using automated tools for keyword extraction such as [1], or human annotation as we did.\n\n$\\text{\\color{blue} Q2: ... provide analysis of how the tag relevance estimates change over ... conversational episodes?}$\n\n**Answer:** Thank you for your insightful suggestions! As suggested, we sampled 5000 conversational data. For each data, we randomly selected a tag that is irrelevant to the user intent and a relevant tag. We then analyze how the relevance score of these two tags changes over the conversational episode, as shown in the table below. \n\nTurn_count                               | 1    | 2    | 3    | 4    | 5    |\n| ---------------------------------------- | ---- | ---- | ---- | ---- | ---- |\n| Relevance Score for a **irrelevant** tag | 0.51 | 0.50 | 0.49 | 0.49 | 0.48 |\n| Relevance Score for a **relevant** tag   | 0.56 | 0.57 | 0.57 | 0.62 | 0.62 |\n\nThese results also support our statement in line 66 that \"the probability of a tag being irrelevant or not can be gradually estimated along with the dynamics of the conversation.\"\n\nMore analysis will be added in the revised version.\n\n$\\text{\\color{blue} Q3: ... deploying this in a real-interactive setting ... What would be the challenge?}$\n\n**Answer:** A potential challenge is the Concept Drift[2], which refers to the phenomenon where the statistical properties of a target variable change over time. It makes a previously trained model less accurate or even obsolete in the real-interactive setting. This is a common challenge in the conversational information retrieval (CIR) community. Our future work will try to bridge the gap between offline and online environments by building more realistic interactions using LLMs. \n\n**Reference:**\n\n[1] Matej Martinc, Blaž Škrlj, and Senja Pollak. 2022.Tnt-kid: Transformer-based neural tagger for keyword identification. Natural Language Engineering, 28(4):409–448. \n\n[2] Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., \\& Zhang, G. (2018). Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12), 2346-2363.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses an important real-world problem of improving conversational FAQ retrieval, which has many applications in conversational agents and customer service.\n\n2. Proposes a novel tag-aware reinforcement learning framework as a solution, leveraging fine-grained tag information to deal with noisy behaviors. This is a technically sound approach.\n\n3. Provides useful new insights on reducing the impact of irrelevant information during conversational retrieval through implicit modeling of tag relevance. Introduces a new FAQ dataset with conversation history and fine-grained tags, which could enable future research.\n\n4. Achieves significantly improved results over baselines on the dataset across metrics like success rate and turns. This demonstrates the effectiveness of the framework.", "reasons_to_reject": "1. The dataset is collected from a single commercial domain, which may limit the generalizability of the results. Testing on more diverse FAQ data could further validate the approach.\n\n2. There could be more analysis on the learned strategy to provide better understanding, e.g. how tag relevance estimates change over time.\n\n", "questions_to_authors": "Q1) Could you discuss the prospects for extending this framework to more diverse FAQ datasets from different domains? What adaptations maybe needed?\n\nQ2) Can you provide more analysis of how the tag relevance estimates change over the course of the conversational episodes? Some examples could be insightful.\n\nQ3) Have you considered deploying this in a real interactive setting with humans for more rigorous evaluation? What would be the challenges?", "paper_topic_and_main_contributions": "This paper focuses on conversational FAQ retrieval, where the goal is to retrieve the appropriate FAQ question that matches the user's intent within a minimal number of conversation turns. The main contributions are:\n\n1. Proposes a tag-aware conversational FAQ retrieval framework called TRAVEL that uses reinforcement learning to eliminate the negative impact of irrelevant information introduced through noisy user behaviors during the conversation.\n\n2. Introduces two key components in TRAVEL: Tag-Level State Representation and Conversational Retrieval Strategy Optimization. The first estimates irrelevant tags in the context and converts it into a state representation. The second uses Q-Learning to determine an optimal retrieval strategy given the state.\n\n3. Creates a new FAQ dataset with conversation histories and question tags to evaluate the framework.\n\n4. Shows through experiments that TRAVEL outperforms baselines in achieving more accurate retrieval within limited turns. It also maintains superior performance across varying levels of noisy user behavior.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 96: Missing space before TRAVEL\n\n\nLine 115: FAQ should be FAQs\n\nLine 270: Missing period at the end of sentence\n\nLine 515: it should be its\n\nThe writing is clear overall and the paper is well organized. The background and related work provide good context. More intuition could be provided when introducing the technical details."}}]

Author rebuttal:

Meta-review: The paper presents TRAVEL, a reinforcement learning-based approach to retrieve FAQs most relevant to the current conversational context of a user.

Overall, reviewers are satisfied with the contributions. The paper can be improved by adding results on more datasets, currently, only one dataset is experimented with.
Decision: Accept-Main