Domain Adaptation for Conversational Query Production with the RAG Model Feedback
Authors: Ante Wang
Linfeng Song
Ge Xu
Jinsong Su
Venue: EMNLP 2023 Findings (2023)
Keywords: conversational query production
knowledge-aided dialogue system
text generation

Abstract: Conversational query production is an emerging fundamental task for the dialogue system, where search queries are generated to explore the vast and continually updating knowledge from a search engine. To accelerate this line of research, previous studies have released several datasets with human-annotated search queries. However, the limited annotations still can not cover conversations of various domains. To solve this challenge, we propose a novel domain adaptation framework. It is inspired by a weakly supervised learning algorithm from previous work that guides a model using reinforcement learning with BM25 scores as feedback. Though effective, it is fragile facing noisy content on webpages from a commercial search engine and variance in conversations because of ignoring deep semantic information of dialogue contexts. Thus, we improve the algorithm by taking the advance of retrieval-augmented generation (RAG) and exploring several practical techniques such as knowledge distillation for stable training. We conduct experiments in multiple settings across different languages. Guided by the RAG model feedback, our model is more robust and performs significantly better especially in a more challenging setting over strong baselines.

Domain Adaptation for Conversational Query Production with the RAG Model Feedback
Ante Wang Linfeng Song', Ge Xu', .Jinsong Su
'School of Informatics, Xiamen University, China
KKey Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China TTencent AI Lab, Bellevue, WA
*College of Computer and Control Engineering, Minjiang University, China wangante@stu. xmu, edu. cn jssu@xmu. edu. cn
Abstract
Human
Chatbot
Conversational query production emerg- ing fundamental task for the dialogue system, where search queries are generated to explore the vast and continually updating knowledge from search engine. To accelcrate this line of research, previous studies have released several datasets with human-annotated search queries. However, the limited annotations still can not cover conversations of various domains. To solve this challenge, we propose novel do- main adaptation framework. It is inspired by weakly supervised learning algorithm from pre- vious work (Wang et al., 2023b) that guides model using reinforcement learning with BM25 scores feedback. Though effective, it is frag- ile facing noisy content on webpages from a commercial search engine and variance in con- versations because of ignoring deep semantic information of dialogue contexts. Thus, we of
I'm trying leam surfing at old age
Search: benelit surting
Not unly does surfing imprrve your candivvssclar health, but gives Vitamin D which regulating the arvount of calcium in your blood
importent for
Cardio that's I'm trying huild up. dider realize how hard it was just to puddle out. And my arm museles need get stronger.
Search: umn musele exercises
There's plenty of ways using equipment, arn circks, phans and puslups arv great for inereasing strength.
strength without
Figure I: A conversation with 4 dialogue turns, where the chatbot sends search queries to search engine for retrieving external knowledge (not shown) before gener- ating the responses. With the aid of the search engine, generated responses can be more engaging and helpful (underlined).
improve the algorithm by taking the advance
generation (RAG) and exploring several practical techniques such as knowledge distillation for stable training. We conduct experiments in multiple settings across different languages. Guided by the RAG model feedback, model is more robust and per- forms significantly better especially in more challenging setting over strong baselines.'
to generate search queries for interacting with a search engine. In this work, we focus on con- versational query production. It aims to generate search queries reflecting user intents so that better responses can be obtained with the help of exter- nal knowledge retrieved from search engine. A typical example is shown in Figure 1. This task is conceptually similar other knowledge-intense tasks such Question Answering (Nakano et al., 2021). However, it is more challenging because a conversational query producer has mine user intents from complex dialogue contexts involving multiple utterances.
1 Introduction
Leveraging external knowledge has been proven to be important for various text-generation tasks (Lewis et al., 2020b). Very recently, with the bur- geoning of large language models, it can effectively alleviate the hallucination issue and improve faith- fulness (Nakano et al., 2021: Glaese et al., 2022). Along this line, exploring the Internet for external knowledge is gaining popularity due to its con- tinually updated content and broad coverage on a variety of domains (Komeili et al., 2022). To ac- cess this type of knowledge, model is required
Naturally, the current efforts consider query pro- duction as text-to-text generation task and can be generally sorted into two categories: supervised and unsupervised learning. In this first category, previous studies (Komeili et al., 2022: Zhou et al., 2022) are dedicated to collecting conversations, where the query for each utterance is annotated. With the development of pretrained language mod-
*Conresponding author. ihttps://githhi com/ com//eeepeearnWWW/DN..

---

els (Raffel et al., 2020; Lewis et al.. 2020a), strong query producer can be obtained with only thousands of training instances. However, these models may still be suboptimal in an out-of out-of-domain conversation. As another category, Wang et al. (2023b) propose to train their query producer with a weak-supervised learning algorithm free of costly human efforts. They first extract keywords from the dialogue context candidate search queries, which are then fed Wikipedia search² for correspond- ing documents. Next, the quality of each candidate is evaluated by comparing its retrieved documents against the gold response using the BM25 algo- rithm (Robertson and Walker, 1994). Finally, the query producer is trained to select the best key- words as search queries by maximizing correspond- ing BM25 scores (rewards) using reinforcement learning.
trained RAG model. Compared with the BM25 al- gorithm only considering the surface word overlap, the RAG model can better capture the deep seman- tic information from the dialogue context and learn to evaluate each document based on its contribution to the target response. Then, we heuristically filter some potentially harmful instances where candi- date queries are scored too close or lower than a given threshold, indicating that these queries are in- distinguishable low-quality. Lastly, the standard REINFORCE, algorithm (Williams, 1992) adopted in (Wang et al., 2023b) is fragile when facing such noisy rewards. This makes the trained query pro- ducer always too far from the good initial policy that is trained on the well-labeled source-domain dataset. Therefore, we further introduce regular- ization objective based on knowledge distillation (Hinton et al., 2015), avoiding involving too large gradient updates (often leading to bad policy) away from the initial policy.
Inspired by these studies, we try to tackle the domain adaptation challenge by combining both worlds. As a tough combination, we first train a model on the labeled source-domain instances in the standard supervised training manner. Then, the model is adapted to the target domain based on the weak-supervised learning algorithm introduced above. Though working effectively, we notice that there cxist two under-explored issues, severely pre- venting our model from further improving.
We conduct experiments on two domain adap- tation settings: CLEAN and NOISY, according to whether model will be influenced by the two issues described above. In the CLEAN setting. carry out experiments Wizard-of-Internet (Komeili al., 2022) Hizard-of-Wikipedde (Di- nan et al.). We follow Wang et al. (2023b) use Wikipedia search as the search engine. In the NOISY setting, we set up DuSinc (Zhou et al., 2022) ~ KdConv (Zhou et al., 2020) and DuS- inc - DuConv (Wu et al., 2019) where Sogou search² is adopted, which is a Chinese commercial search engine. Experiment results show that our model significantly outperforms strong baselines, especially in the NOISY setting. Further analysis demonstrates that our designed model feedback is more accurate than BM25 scores and all proposed operations can benefit the final model.
First, commercial search engine always re- turns noisy webpages containing unrelated infor- mation such as advertisements. However, Wang al. (2023b) only study in toy setting by us- ing Wikipedia search to simulate a search engine, where the returned Wikipedia pages are much cleaner. Another problem attributes to the vari- ation of conversations. Wang et al. (2023b) con- duct experiments on an impractical scenario, where nearly all utterances involve external knowledge. Nevertheless, there often exist variances that some utterances can be generated without any external knowledge (e.g., greetings). As there are no cor- responding queries to these instances, training on them will inevitably hurt the model performance. Concretely. the above two problems result in in- accurate BM25 scores and unstable reinforcement learning, challenging the robustness of the adapta- tion algorithm.
2 Background
In this section, we first introduce how a dialogue system equipped with a query producer works. Formally, taking a concatenated dialogue history Xa Mi,.... 44-1 of t - 1 turns as inputs, query producer generates search query q. Then, & will be sent search engine for retrieving list of NA documents KC kQ.... A Finally, response uy generated by a response generator consuming both X and K"", this work, we mainly focus on the crucial query producer.
To tackle these problems, we first take advantage of aetrieval-augmentte generation (RAG, Lewis et al. 2020b) and replace the BM25 algorithm with
Phttps: h/tww: wi www.wwwwwiiiii..ommm pediia.oorr org//
thttps://www.sgguu.com sogou. com/

---

Xxt
amm museles
my arm museles need to get stronger
Query Producer
museles strengthening
aITO mossele exercises
S
umm muscles
0.1
GG
aEm muscles strengthening
0.3
amm musele excrcises
0.5
90
Mr
Health
The Best Exerciooss for Stronger Muscles
There's plenty of ways to increase arm strength...
RAG
The adaptation workflow for query producer well-trained in the source domain, where solid lines (-+) and dashed lines (++) indicate training and inference passes. At stage 1 (-), the query producer consumes input utterances Net to get set of search queries I1,... QN. Then (-+), we retrieve relevant documents K ...., K.. for these queries from a search engine. At stage 2 (-), RAG model is trained by maximizing the probability dialogue response ie, given both Xx and K ..... ,...,... KaN, During this process, the retriever from the RAG model learns the contributing score of each document to response generation. At stage 3 (-+), supervision signals $1,..., are calculated based on retrieval scores from the last step to update (-) the query producer.
Figure 2:
Following the common practice (Komeili et al., 2022), the query producer can be initialized with T5 model (Raffel et al., 2020) that is pretrained on a massive text corpus. On an annotated query generation benchmark, the query producer is then trained to generate the target query q in an autore- gressive manner. Formally, this model takes the standard cross-entropy (CE) loss as the training objective:
model scores for the queries in the offline cache are calculated as rewards. With the informative model feedback, our framework can be more effective than traditional algorithms such as Self-training (Kulshreshtha et al., 2021).
3.1 Offline Cache Creation
Since qucrying a search engine during the model training is time-consuming, we follow Wang et al. (2023b) to set up an offline cache storing docu- ments for all possible queries before the actual model training. However, a query producer will take large search space of candidate queries, which is exponential to the average query length, Thus, we only consider most N queries with high probabilities for each instance, which can be easily accessed via beam search (Sutskever et al., 2014)4.
CEE - log K)t;0)
(1)
where 0 represents the model parameters.
3 Domain Adaptation Framework
Figure 2 visualizes the workflow of our framework. Before adaptation, the query producer has been ini- tialized with a source-domain model (fwre), which is trained using LCE. Intuitively, the model can predict some good but not perfect queries for the target domain because of the domain shift.
Optionally, inspired by (Wang et al., 2023b), the candidate query set can be enriched with keywords extracted from the dialogue context using some public keyphrase extractors, e.g., TagMe (Ferrag- ina and Scaiella, 2010). This because the tar- get query sometimes is a keyword that the model- predicted query set does not cover due to the limi- tation of the model capability in the target domain.
To tackle this issue, we then adopt frame- work which can be decomposed into stages for better clarification. At stage ($3.1), we create an offline cache for all possible queries (generated by the source- cource-domain model) and their retrieved documents from a search engine for target-domain instances. They will be used to train the models in later stages. At stage 2 ($3.2), we train the RAG model the turget-domain dataset using docu- ments from the offline cache. This model will play the role of the reward model for reinforce- ment learning stage 3. At stage 3 ($3.3), the query producer (0grc) is transferred to the target do- main using reinforcement learning, where the RAG
3.2 Training a RAG Model
With the conversations and related documents from the offline cache, now are able to train RAG
"We also explore diverse sampling (Tan et al., 2018: Holtz- man et al.), but it does not make difference in our preliminary study possibly due to the limited length of a query (mostly less than 5 words).

---

model. RAG is first proposed in (Lewis et al., generation
the cross-encoder structure allows for richer inter- actions between the input context and candidate document (Humeau et al.). also saves compu- tation costs by directly feeding the intermediate hidden states the shared encoder to the generator decoder for predicting
2020b) for various knowledge-intense tasks. consists of retriever and a generator. During inference, the retriever will obtain several documents, each of which is treated as a latent variable and fed to the generator for predicting the target response. Then, the predictions of the gen- erator given different documents are marginalized for the final prediction.
Note that our implementation is generally based on RAGeToken (Lewis et al., 2020b), which jointly considers multiple documents during predictions. echoes the previous study (Shuster et al., 2021) that RAG- is better choice for this task than RAG- AGG-equence, which produces an output based on exactly one document.
Usually, the retriever is initialized from the Dense Passage Retriever (DPR, Karpukhin al. 2020), which is built upon Wikipedia and only serves in English. To satisfy wider use, we simplify the model by sharing the parameters of the retriever and generator encoder. Similar to the query pro- ducer, we initialize our implemented RAG model with T5. Formally, given the dialogue context Aes,, the response u and corresponding retrieved docu- ments K = K ...,, K K for this instance from the offline cache, we define the predicted probability to 4s as
3.3 Reinforcement Learning with the RAG Model Feedback
We use reinforcement learning to finetune the query producer, where output scores over candidate docu- ments of the trained RAG model from the last stage used as feedback.
Scoring function based on retrieval scores For- mally, for a search query and its retrieved docu- ment list KC with NA documents, the query score s is decided by scoring function S(CC, *). where can be some optional inputs (e.g., Aee and u.). Wang et al. (2023b) define
Jud p(u Xat)
Ket, ue,sis 23; gen))
"1, 2 top-k(p(- Ket Kretet));
(2)
where top-k(*) is function selecting & documents (e.g.. zj) with the highest retrieval scores fed to the generator for predicting and p(z,) = p(2 | Ne: K: fret). fret and ggen denote parameters for
f(K", u.) max( (BM2((B25(())) ue) be)eeee)))) (4)
As the BM25 algorithm is based on surface word overlap, its scores may be inaccurate without taking deep semantic information into account. In this work, we define
the retriever and the generator, respectively. Concretely, the retriever consists of the shared T5 encoder (T5-enc) and randomly initialized Multilayer Perceptron (MLP) head. It consumes the concatenation of Xet and each document kę K. and then produces the probability distribution over IC, indicating their importance for predicting uy:
f(KY, Xet, max(fc) (cG)...., "N, N,, }),
(5)
where ef is the retriever (Oret) output given kạ from K (Fquation 3). In this way, we can easily obtain the scores for all possible queries of each instance from the offline cache.
p(. 1et, 7) = Softmax(ci,,..... 9K); e, = (wL.LLTTSec((CLLsskkk...... (O))),
(3)
Quality control via filtering Because of the noisy webpages and variance conversations, it is risky to train the model on all instances with- out quality control. We propose two easy-to- implement operations to clean the dataset. For an instance with query scores $1,..., and 8maz / 8min denoting the maximum minimum within them, we only keep the instance where Smaz a and Smez Smin 3, a and are hyperparameters. Using the first operation (a-Filtering) we aim to filter the instances where no external knowledge is required. For these instances, the candidate query
where [CLS] is a special token prepended for pro- ducing the vector fed to the MLP head.
One drawback of our RAG model is the slower retrieving process compared with the traditional one. This is because our model adopts cross- encoder structure that takes concatenated dialogue context and document pairs as inputs. While previ- ous work uses bi-encoder structure, which sepa- rately encodes each part and calculates scores using Maximum Inner Product Search (MIPS). However,

---

scores are generally low as their retrieved docu- ments can benefit response generation. Via the latter operation (3-Filtering), we drop the in- stances where their query scores are quite similar, indicating that they are difficult to differentiate by the reward model and thus are less helpful for our query producer.
System
R@3
R@5
T5-base
54.78
69.07
73.51
QP-Extt
62.41
72.91
74.87
QP-Gent
56.77
66.08
68.22
WSMF w/ BM25
65,91
77.69
80.48
WSMF
66.20
76.77
79.27
Self-Train
52.31
65.97
70.50
Self-Train (MIX)
52.76
66.39
70.86
Reinforcement finetuning with regularization We adopt the REINFORCE algorithm (Williams. 1992) finetune the query producer using the query scores as the rewards. For a training in- stance, we have obtained a set of queries QI....,NN and their scores $1....,N via the above process. To reduce the training variance, we define the re- ward Ti for qi by normalizing the query scores Smin
DAMF w/ BM25
67.63
78.49
81.02
DAMF (ours)
68.34
78.32
80.96
Table 1: Main results in CLEAN setting (Wol WoW). where denotes the results from (Wang et al., 2023b).
evaluation. Both CLEAN and NOISY settings are investigated. We mainly focus on the latter one, which is more challenging.
r3 Smags -mman-8min. Besides, we subtract baseline value by averaging the rewards. Formally, the reinforcement learning loss is defined as
We take Sizzardoff-Inernee (Wol, Komeili et al. 2022) Wizard-of- Wiiki-eeiai (WoW, Dinan et al.) as the CLEAN setting. Following Wang al. (2023b), we adopt Wikipedia search and take the target Wikipedia page titles provided in the dataset for evaluation. As the Wikipedia page is well- constructed and nearly all instances require exter- nal knowledge, it is easier to train a query pro- ducer. The NOISY setting includes DuSine (Zhou et al., 2022) KdConv (Zhou et al., 2020) and DuSinc + DuConv (Wu et al., 2019). We manu- ally annotate 817 and 321 queries for KdConv and DuConv as test sets, respectively. We use Sogou search, which is a Chinese commercial search en- gine. Thus, the returned documents always con- tain unrelated information. Besides, both KdConv and DuConv include many instances that need not search queries, nearly covering 32% and 45% ac- cording to our annotations.
- (F, F) log Kec fegt)
(6)
where q is sampled from the model predictive prob- ability distribution over the candidate query set, and f is the corresponding reward value.
A common issue for domain adaptation is hurt- ing the model performance in the source domain. Besides, though we control the quality of training instances by filtering operations, some unrecog- nized harmful instances still make the training un- stable. Considering that we have obtained query producer far fitted to well-constructed source- domain dataset, we adopt this model to regularize the training process via knowledge distillation (KD, Hinton et al. 2015). Note that we also perform fil- tering operation by dropping the queries with their confidence lower than different threshold yy to construct small high-qualit, KD corpus. For each instance from the KD corpus, we ask the model to predict the pscudo query generated by Oxrci
Detailed introductions to these datasets and an- notation guidelines are provided in Appendix A and B.
Evaluation Metrics We report the average re- sults of 3 runs for all experiments. We mainly focus on query production, but also evaluate re- sponse generation to validate the positive effects of improving query quality.
Ckd = log Xet; Begt).
Thus the overall training loss for adaptation is
L Lrt XLnd-
(8)
Query production For WoW, we follow Wang et al. (2023b) to use recall, denoted as R @@ K (K (1,3,5)). which compares retrieved doc- uments of the top K predicted queries with ground-truth documents evaluate the perfor- mance of query producers. For KdConv and DuCony, we follow previous work (Zhou et al., 2022) to use Unigram Fl (Uni. FI), BLEU-1/2
where À is a hyperparameter used to balance the two loss terms.
4 Experiment
4.1 Setup
Dataset We prepare 3 domain udaptation bench- marks involving datasets across languages for

---

KdConv
DuConv
System
Uni. Fl
BLEU-1/2
ROUGE-1/2/L
Uni. FI
BLEU-1/2
ROUGE-1/2/L
TS-base
60.21
57.57/54.72
70.95/39.11/70.088
64.18
54.90/52.36
iext-davinci-003
54.93
45.29
67.53/54.23/66.244
61.44
57.62/55.79
0.82/60.16/68.09
WSMF w/ BM25
53,98
41.60/41.05
61.71/50.96/6 /6.55
50.97
30.82/30,54
(1.66/4..44/56.2
WSMF
60.54
45.87/45.41
05.85/57.18655.77
58.18
37.43/37.23
62.54/54.52/62.44
Self-Train
62,21
60.31/57.55
67.20
58.51/56.06
72.67 72.67/64.94/71.44
Self-Train (MIX)
62,42
60.31/57.55
72.71 (1.71/6.47771../
68.08
60.10/
73.45 66.04//2..7
DAMF w/ BM25
64.27
63.71 / 10
73.40 /7..27
68.82
61.34/58.80
74.52/ 2..52//66.666 66.66/6666///
DAMF (ours)
67.44
65.98/ / 62.81
75.66 / 66,29 /74.99
72.32
/5.99
76.29 / 70.12/75.18
Table 2: Main results in the NOISY setting (DuSinc KdConv & DuSinc - DuConv).
(Post, 2018) and also introduce ROUGE-1/2/L (Lin, 2004) for comparing generated queries against the annotated ones, Among these metrics, BLEU-1/2 mainly considers precision, while the others are more comprehensive by jointly consid- ering precision and recall.
model-predicted pseudo generations targets and Self-Train (MIX) additionally uses gold queries in the source domain. (6) DAMF and DAMF w/ BM25. They are trained using our framework. Again, we provide DAMF w/ BM25 for a fair comparison, which uses BM25 as the base of f. Our model configuration is shown in Appendix D.
Response generation As implemented in (Di- nan al.), use Perplexity (PPL) and Uni. F1 to evaluate the predicted responses against gold references.
For the CLEAN setting (Table 1), we observe that: (1) Weakly supervised learning approaches even give better performance than T5-base. It par- tially because the queries can usually be extracted from the contexts for WoW. Besides, QP-Ext and QP-Gen give inferior results than our implemented WSMF w/ BM25 because we adopt a stronger pre- trained model (T5-base). (2) For domain adaptation approaches, our framework can help train stronger models, while self soff-training even hurt model per- formance. It is because the source-domain model can not generate good pscudo targets due the large domain discrepancy and this leads to severe error propagation. In comparison, our models are better guided by pseudo targets with their quality scores decided by f. (3) Using different f gives competitive results, with DAMF / WSMF slightly better than DAMF w/ BM25 / WSMF w/ BM25 on still notice
4.2 Main Results
Table 1 and 2 show the main test results on query production in CLEAN and NOISY settings. We list our models and some typical baselines from previous efforts: (1) T5-base.
trained on the source-domain dataset using the CE loss (Eq. 1) and directly tested on the target-domain datasets. (2) text-davinci-003$ It is a strong large language model from the popular GPT3 family (Brown et al., 2020). We follow common practice (Zhao et al., 2021) using 8-shot in-context learning evaluate its query production performance. An input case is provided in Appendix C. (3) QP-Ext and QP-Gen. Both models are trained in the weakly supervised algorithm (Wang et al., 2023b). The former is fine- tumed from ELECTRA -base (Clark et al., 2020), aiming at extracting keywords from the context as queries. The latter is based on BART-base, generat- ing queries using typical sequence generation. (4) WSMF and WSMF w/ BM25. They are our imple- mented QP-Gen by optionally replacing its BM25 scores with RAG retrieval scores. (5) Self-Train and Self-Train (MIX). Self-training is a popular technique for domain-transte learning. We follow Kulshreshtha et al. (2021) and He et al. to imple- ment these models, where Self- Train only adopts
the main metric R @@L 1. However, that RAG feedback is better supervision given our later analysis ($4.4). Thus, we believe that the ad- vantage is not obvious due to the ideal scenario of the CLEAN setting and the robustness of the framework.
For the challenging NOISY setting (Table 2), we can draw the following conclusions: (1) text- -xx--dainnc-003 performs well using only 8-shot demonstrations, but still worse than the T5-base finetuned on full source-domain instances. (2) Comparing WSMF and WSMF w/ BM25, the for- mer performs significantly better thanks to the much more accurate model feedback, which is
"This model is tested by calling the official OpenAl api https://platfore openai, com cow/docs////ccooocccccccccccc /models.

---

System
Uni.
BLEU-1/2
ROUGE-1/2/L
DAMF w/o Lea
66.89 (-2.36)
66.21/62.75 (-0.39/-0.11)
BM25
63.12
60.82/56.79
73,40/ 62.05/72.27
keywords
66.81
62.60/60.53
73,21 02221/65.11//0 72.663
w/o --ilteriig
66. 08
66.07/61.60
74,80/ 64, 73.81
w/o B-Fittrring
66.58
65.82/62.32
74.84/ 65 /tt///6.29////// 73.997
w/ Lid
67.44 (-0,56)
65.98/
(-0.90/40.67)
75.66/ 75.66/66.29/7444 66.29/74.79 (40.90000000.00005---77777
w/ Laka
66.84 (-1.48)
62.86/62.67 (-0.87 (-.90)
75.21/63.71/74.4 (1235)777774..
w/ Cee
66.48 (-1.02)
65.44/62.30 (1.4//22.30011441/1.110 (1444/15577
74.78/65,255 25/74....2/.11 (4.77
(4097/-113/-99))
DAMFW/O L.
"3.49
63.62/60.70
64.57764.5077.777
Table 3: Ablation Study on DuSinc + KdConv. To study the negative impacts on the source domain, we also provide
performance decreases on the source-domain data (DuSine) which are shown brackets.
a strong proof validating our motivation. Both of these models give low BLEU scores, it is be- cause they usually only generate the main key- words without other details due to the unsuper- vised setup and thus have a large length penalty of BLEU metrics." 6 (3) Again, though Telf-Train and Self-Train (MIX) give better results than T5-base this time, model still significantly outperforms these baselines. Besides, DAMF' significantly sur- passes DAMF w/ BM25 in this challenging NOISY setting, convincingly demonstrating the effective- ness of our framework.
80
70
R@K
60
R&L (BM25)
50
RØl (RAG)
R83 (BM25)
40
R@M (Deth)
2
6
10
Turns
Figure 3: Evaluation of query scores calculated with different scoring functions (f) at various dialogue turns on WoW training set, where R M is the ceiling perfor- mance.
4.3 Ablation Study
Table 3 shows the ablation study for our frame- work. It is conducted on DuSinc KdConv, which is challenging and annotated more queries, thus can provide more convincing results. We can draw the following conclusions. First, we again observe that using RAG feedback works better than adopt- ing BM25 scores, when Lad is not implemented. We also explore enriching the query set with key- words appearing in the dialogue context as Wang et al. (2023b) suggested.? However, it instead hurts the results of BLEU and ROUGE, which is be- cause some unrelated keywords may disturb the model training. Besides, both filtering operations can slightly improve the final results and save some training costs by dropping some harmful instances. Second, we compare some choices of regulariza- tion loss terms. Lykd is variant of Lkds where the Cosine Similarity loss term is adopted to close the predicted distribution of source-domain and target-domain models the small KD corpus.
However, it gives inferior results Lid most metrics. it because the source-domain model can be unconfident on target-domain instances, leading to unreasonable probability distributions. Another choice is jointly training with instances from the source domain (CCE). However, it performs worst in the target domain and is also less effective than Led for regularization. Finally, we remove the En thus only keep the Lid term for finetuning, It also helps but lags behind our model, validating the effectiveness of reinforcement learning. Note that similar to Self- Teaii, but it is trained on our small KD corpus, verifying the effectiveness of using RAG model feedback from another aspect.
4.4 RAG Model Feedback is More Accurate
The selection of f decides the accuracy of the query scores, directly influencing the effectiveness of vi- tal reinforcement learning in stage 3. Thus, we provide comparison of f using BM25 or RAG scores in Figure 3, validating that the RAG scores can be more accurate than BM25 scores. Thanks to
"For example, these models prefer "Lionel Messi" when the user asks "the birthplace of Lionel Messi". "We use an open tool, TexSmart (https://ai. tencent. con/ wwwaaaalllltneeeeeeeeeeeaaae ties as keywords.
ww/hhh///x../// html). extract enti-
Details are introduced in Appendix E.

---

WoW
KdConv
query, For example, when the user asks about "the birthplace of Lionel Messi", the target knowledge can also be retrieved by sending "Lionel Messi" to a search engine. However, we believe the query "Lionel Messi" is not good, as does not faithfully reflect the user intent.
System
PPL4
Uni.
PPLJ
Uni.
Seq2seq
17.52
22.08
10.48
33,92
T5-base
15.97
22.68
9.91
37.60
Rank-Gen'
21.60
17.81
Merge-Genf
20.20
18.15
WSMF w/ BM25
15.53
22.96
10.50
36.39
WSMF
15.47
23.06
10.28
36.41
Related Work
Teaf-Train (MIX)
15.98
22.59
9.82
38.02
DAMF w/ BM25
15.51
22.91
9.57
38.36
5.1 Knowledge-aided Dialogue Model
DAMF (ours) 15.38
23.11
9.27
38,99
In recent years, effective dialogue agents have ex- plored various knowledge for facilitating dialogue understanding and response generation, such as sentiment (Song et al., 2022), speaker emotion (Po- ria et al., 2019), discourse structure (Wang et al., 202 la, 2023a), and external knowledge from dif- ferent knowledge sources. This work falls into the last category of the above research lines, which has shown effectiveness in alleviating the halluci- nation problem. Early studies (Sun et al., 2019; Zhou et al., 2020; Dinan et al.; Wang et al., 2021 explored static databases such as built knowledge graphs or collected documents (e.g., Wikipedia dump). Besides, as a typical dialogue model has to select correct knowledge and generate conversation simultaneously, some researchers also explore fus- ing the QA and dialogue systems together (Adolphs et al., 2021; Oh al., 2023). Recently, build stronger models especially based on large language models (LLMs), leveraging vast knowledge from the Internet is gaining popularity because of its dy- namically updating nature (Komeili et al., 2022; Zhou et al., 2022).
Table 4: Main test results for response generation, where t denotes the results from (Wang et al., 2023b).
the gold document annotations (page titles) from WoW, we can directly evaluate f using @@ K, rep- resenting the hitting of the gold document with top-K scored queries. Comparing the main metric R@ 1, using RAG feedback is always better than using BM25 and the conclusion holds across all di- alogue turns. But they give similar results on R @ @3, It shows that the RAG model mainly helps select the best queries. Besides, f perform worse when the conversation contin- ues. However, the ceiling performance only drops slightly. due to the much higher complexity of longer context, challenging the understanding capacity of the models. Another performance boost can be expected when using stronger RAG variants (Shuster et al., 2021; Paranjape et al.). We leave it future work.
observe that both
We also provide case from KdConv further validate our conclusions, which could be found in Appendix F.
5.2 Conversational Search Query Production
4.5 Response Generation
Leveraging vast knowledge from the Internet is an important topic for developing various NLP tasks. We focus on conversational query produc- tion, which is the key to building search-engine- aided dialogue system. Though it is recently pro- posed, this field has attracted wide attention. Fol- lowing common practice, Komeili et al. (2022) and Zhou et al. (2022) have proposed Wizard-of- Internet and DuSinc with collections of conversa- tions annotated with search queries for English and Chinese respectively to train their query producers. Wang et al. (2023b) proposed a weakly supervised learning algorithm for building query producer free of human annotations.
Table 4 shows the response generation results, where the models are equipped with corresponding query producers. All our models adopt the RAG model introduced in $3.2, except Seq2seq, which does not consider external knowledge inputs. Ad- ditionally, we provide Rank-Gen and Merge-Gen which are the two types of models proposed in Wang et al. (2023b) equipped with QP-Ext. Gener- ally, the performance of response generation grows when feeding external documents retrieved with better search queries. This validates the importance of developing stronger query producers. Some may concern that the improvement of response genera- tion is limited even though there is a large margin for query production. It is because sometimes the target knowledge can be covered by non-perfect
In this work, we extend these studies serve conversations with different distributions by proposing a novel domain adaptation framework.

---

We notice that (Chen al., 2022) shares some same spirits with our approach. They study several reinforcement learning rewards for enhancing the question answering performance by improving a question rewriting model, which works similarly to our query producer. However, their methods may not be well adapted to conversation query produc- tion because of the task discrepancy. Besides, their rewards either suffer from the noisy issues have discussed or are not interpretable compared with our used retrieval scores.
Acknowledgments
The project was supported by the National Natural Science Foundation of China (No. 62276219), and the Central Leading Local Project "Fujian Mental Health Human-Compute Interaction Technology Research Center" (No. 2020L3024). We thank reviewers for their insightful comments and also thank editors for their every effort.
Limitations
Our work is also remotely related to building large language models accessible to search en- gine for wewwledge-intensiig tasks, such as We- bGPT (Nakano et al., 2021) and Sparrow (Glaese al.. 2022). However, both of them developed their systems by adopting reinforcement learning with human feedback (RLHF). We explore enhanc- ing our query producer with RAG model feedback, which is free of human efforts thus costs much less than these approaches.
We only propose general adaptation framework in this paper, while there are still many variants worth exploring. First, there are several studies (Shuster et al., 2021; Paranjape al.) in these years pro- posed to improve the RAG model. Using these stronger models as the reward model, our model feedback can be more accurate intuitively. Second, our framework may work better combination with other adaptation approaches such as adopting domain pretraining (Karouzos et al., 2021). How- ever, they are both out of the scope of this paper.
6 Conclusion
In this work, we propose domain adaptation framework for conversational query production. Without any human annotation the target do- main, we train a query producer using reinforce- ment learning with feedback from a RAG model. Besides, we also enhance our framework with some masy-to-implement techniques. Compared to frag- ile combinations of previous approaches, frame- work is more robust against the noisy content on webpages and variations of conversations. Experi- ment results show that our model significantly out- performs strong baselines, especially in more chal- lenging settings.
References
Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2021. Reason first, then respond: Modular generation for knowledge-infusee dialogue. arXiv preprint arxiv:2111.05204
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Zhiyu Chen, Jie Zhao, Anjie Fang. Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. 2022. Reinforced question rewriting for conversational question an- swering. arXiv preprimi urXiv:2210.1577.
An important future direction can be exploring improving query producer with LLMss. Though scaling the model size of query producer has been demonstrated less effective (Zhou al., 2022), we can still manage to improve small query producer with the guidance of an LLM. Recently, researchers have greatly exploited the potential of LLMs by conducting prompt engineering, such as demon- stration selection (Liu et al., 2021) or automatic prompt generation (Yang et al., 2023). We believe these work can also benefit the query production task. For example, construct pseudo training in- stances using an L.M to enrich the training set, or design better training signals for improving a query producer with the LLM feedback.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint 2rXxiv2003.100 10555.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 889-898.

---

Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on on-the- fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM inter- national conference on Information and knowledge management, pages 1625-1628.
for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 1871-7880.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni. Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, al. 2020b. for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Amelia Glaese, Nat McAleese, Maja Trębacz. John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14377 14375.
generation
Chin- Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization
Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.
branches out, pages 74-81.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint 101.066044
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network, arXiv preprint arXiv: 553.03231..
Nlya I. Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degen- eration. In International Conference on Learning Representations.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:21
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux. and Jason Weston. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi- sentence scoring. In International Conference on Learning Representations.
Minsik Oh, Joosung Lee, Jiwei Li, and Guoyin Wang- 2023. Pk-icr: Persona- -ooona--nowwedee interactive con- text retrieval for grounded dialogue. arXiv preprint 2rX6v:2302.06674.
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. Udalm: Unsuper- vised domain adaptation through language modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2579-2590.
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. Hind- sight: Posterior Posterior-guided training of retrievers for im- proved open- open-ended generation. In International Con- ference on Learning Representations.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in con- versation: Research challenges, datasets, and recent advances. IEEE Access, 7:100943-100953.
Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmentedd dialogue generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 8460-8478.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 2 21(1):5885-5511
Devang Kulshreshtha, Robert Belfer, Tulian Vlad Ser- ban, and Siva Reddy. 2021. Back-training excels self- training at unsupervised domain adaptation of ques- tion generation and passage retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7064-7078.
Stephen E Robertson and Steve Walker. 1994, Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SI- GIR GIR"94: Proceedings of the Seventeenth Annual In- ternational ACM-SIGIR Conference on Research and Development Information Retrieval, organised by Dublin City University, pages 232-241. Springer.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.
Bart: Denoising sequence-to-sequeee pre-training

---

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet inge- nious pre-trained models for chinese. arXiv preprint 10.0666..
Linfeng Song, Chunlei Xin, Shaopeng Lai. Ante Wang, Jinsong Su, and Kun Xu. 2022. Casa: Conversational aspect sentiment analysis for dialogue understanding. Journal of Artificial Intelligence Research, 73:511- 533.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced represen- tation through knowledge integration. arXiv preprint arXiv:1904.099
Han Zhou, Xinchao Xu, Wenquan Wu, Zheng- Yu Niu, Hua Wu, Siqi Bao. Fan Wang, and Haifeng Wang. 2022. Link the world: Improving open-domain con- versation with dynamic spatiolemporal--ware knowl- edge.
Ilya Sutskever, Oriol Vinyals, and Quee V Lc. 2014. Sequence sequence learning with neural networks. Advances in neural information processing systems, 27.
Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang. and Xiaoyan Zhu. 2020. Kdconv: A chinese multi-domaii dialogue dataset towards multi-tur turn knowledge-dricg -divenn conversation. In Proceedings of the 58th Anmual Meeting of the Association for Com- putational Linguistics, pages 7098-7108.
Ante Wang, Linfeng Song, Hui Jiang, Shaopeng Lai, Junfeng Yao. Min Zhang, and Jinsong Su. 2021a. structure self-aware model for discourse parsing on muulti-party dialogues. In Proceedings of the Thirti- eth International Joint Conference on Artificial Intel- ligence, IJCAT-21, pages 3943-3949. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Ante Wang, Linfeng Song, Lifeng Jin, Junfeng Yao, Haitao Mi, Chen Lin, Jinsong Su, and Dong Yu. 2023a. D2psg: Multi-party dialogue discourse pars- ing as sequence generation. JEEF/ACM Transactions on Audio, Speech, and Language Processing, pages 1-10.
Ante Wang, Linfeng Song. Qi Liu, Haitao Mi, Longyue Wang, Zhaopeng Tu, Jinsong Su, and Dong Yu. 2023b. Search-engine-augmeette generation with cheaply supervised query production. Artificial Intelligence, page 103874.
dialogue response
Xiaoyang Wang, Chen Li, Jianqiao Zhao, and Dong Yu. 2021b. Naturalconv: A chinese dialogue dataset towards multi-tur turn topic-driven conversation. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 14006-14014.
Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement leamning. Reinforcement learning, pages 5-32.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang- 2019. Proactive human-nan human-machine machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1794-3804
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le. Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409.

---

A Details of Datasets
B Annotation Guidelines
For the source domain, we choose Wizard-of- Internet (Wol, Komeili et al. 2022) and DuSinc (Zhou et al., 2022), both of which provide conver- sations annotated with search queries.
We hire Chinese students with master's degrees and NLP backgrounds to annolate queries for KdConv and DuConv instances. Each annotator provided with the whole dialogue session. As a query is clearer when a dialogue response is provided (usu- ally the topic of the response), annotators are asked to label considering the information next utterance. A typical case is shown as follows:
Wizard-of-nttenet, is in English, built upon Bing² as the search engine, and is split into 35,201/2,471/2,,10 105 training instances for train- ing/developmenntess,, respectively.
"Context:
DuSine. It is shared task on Chinese conversa- tional query production containing 6,254/660 in- stances for training/developmennt respectively. 10
भTi E AI #144? (Do you know the
movie Chongqing Forest?)
#
(You
For the target domain, we choose Wizard-of- Wikipedia (WoW, Dinan et al.), KdConv (Zhou al., 2020), and DuConv (Wu et al., 2019). All of them commonly used knowledge-intensifn response generation datasets.
know, it's film directed by Wong Kar wai.)
Response:
M -
$ - (Faye Wong is one of the main actors. This
film is highly sought after as soon as it was re- leased.)"
Wizard-of-Wol Wikipedia. It is an English dataset, built using Wikipedia. Following Wang al. (2023b), remove the first turns for all con- versations, which are the keywords indicating the main topics of those sessions. We select to test on unseen sets, where the dialogue top- ics do not appear in the training set. Finally, we have 6/6611//5558335566 instances for train- ing/developmeentresiingg respectively. As it pro- vides ground-trute truth documents for each instance, measure the quality of generated queries by comparing the retrieval results with these ground-truth documents.
In this case, the query should be "MR# Tm Pu (actors in Chongqing Forest)" because the response discusses one main actor "HE (Faye Wong)" in it.
If a response does not include external knowl- edge, we ask annotators simply pass this case, making sure all queries are faithful to the conversa- tions. A passed example is shown follows:
"Context: i H. E
A + (Many people watch this form Leslie Che- ung.)
KdConv. It contains Chinese conversations from 3 domains (film, music, and travel). The av- erage turn number for each utterance is much longer than DuSine (19.0 vs. 10.2). The in- stances are split into 62.9988/8,81199,887 for train- ing/developneen/eesinngg Because of lacking query annotation, we manually annotate 817 in- stances from the test set for query evaluation. The annotation guideline is provided in Appendix B.
1NT.
- (When it comes to him, feel so sad. Since he passed away in 2003, there has been sixteen years.)
Response:
FA i he has been away from us for so many years, the works he left behind make us never forget him.) "
it. (Although
DuCony. It collects Chinese conversations about movies. The average character number per utter- ance is much shorter than that of DuSinc (10.6 vs. 21.7). We filter some unmeaningful utterances and the split for traning/devllopnenntesstgg is
C Input Format for texi-davinci-003
We follow Zhao et al. (2021) to design input format for text-davinci-003. Besides, thanks to the advice of a reviewer, we also enrich the prompt with task definition and speaker role information. A 2-shot case is shown as follows:
70.043/7.00 70.043/7.054/4.088 05844.08000 Similarly, we manually an- notate 321 queries for evaluation.
"nttps://www bing.ccom com/
Given the input dialogue, output a search query, which will be fed to a search engine for knowledge helpful for replying.
"We use the part that is publicly available at https: //aistudio. baidu. con/ wwwtaaisstdoooooooooo /tt.eteetttte 139431/1

---

Input: "usr": "I saw a solar eclipse when was 8 years old."
Dialogue
T S- No 9
T
S
#1 f22
L (Is the Miyun Rexervoir is good place to visit? My child needs to write an article about going out for fum, and I'm worried about where to take him.) 1 IR T X.
Output: solar eclipse
Input: "usr": "Hello there, are you enjoying the cold weather? just went for a run!"
BE ii z. k ThE Rf 19. R. A 2 ( if T 18 (As far I know,
Output: winter runs
Input: "usr": "I'm trying to learn surfing at old age."
Miyun Reservoir seems to be nature re- serve and is not open the public.)
R ?g? if 12 [=] f5s 1- Gt SI T (Fortumately, asked you, otherwise it would be troublesome.)
Output:
Using more demonstrations usually gives better performance. We use 8 shots, which are randomly sampled from the source-domain instances.
He 12 R. ts t iàj inl- (1 remem- ber it wax. You can consult elsewhere. fT. [ F. BE T- un ME ill Jj UE? (Okay, what about the place called Tianzhuang Reservoir around here?)
D Model Configuration
We adopt T5-base/ aaaaasemmenziiit--.... mengzi-t5-base (Zhang et al., 2021) for all experiments in English/ Chinese." II All models are trained using AdamW optimizer (Loshchilov and Hutter) with the linear scheduler and initial learning rate of 5e-5. For source-domain query producers and RAG models, the batch sizes are set as 64 when training, and we train these mod- els until convergence and select the checkpoints with the lowest loss values on the development set as final models. For the adaptation process, we set a larger batch size of 256 for the stability of gra- dient descent. Without query annotation, we stop the training until the model achieves stable average rewards of preferred queries on the development set. As for other hyperparameter selections, we empirically set NA. N, k; and as 10, 5, 5, and 0.1 for all experiments. For a, B, and 7, we set 0.4 / 20, 0.2 5, 0.6 / 30 when using RAG/ / BM25 scores as feedback.
Response
# _ f
E n| (f) . (I remember that the
EE
&
Tanzhuang Reservoir doesn't require fickets, it's quite magnificent, but don't think there's much to see either.)
f w/ BM25
T * B (Scenic Spots of Miyun Reservoir): 32.46
(Myyn #Foyyy reservoir): 33.61
f #E fr to Miyun
Reservoir: 36.06
MB EEiakk:TTTTTaan Reservoir); 34.16 Spots of Miyun
J w/ WAGG
Reservoir): 0.06
1 reservoir): 0.62
* - * ( to Miyun
Reservoir: 0.38
(Tiincchuag Reservoir): 1.44
Table 5: An example with their scored predictions using different f from KdConv development set, where [H HE* | (TianzJuan Reservoir)" should be the target query and the key clues are underlined.
E Knowledge Distillation Term Lakd
calibrated as it is not trained on the target domain thus may instead hurt model performance.
Both Lid and Lakd are adopted for asking our model to learn the behavior of Osrc. which has been well-trained on clean labeled corpus. Thus, it can ease the impact of unexpected updating because of some noisy instances when reinforcement learning. For each input Xet, we ask our model to approxi- mate the free output probability distribution instead of generated query à used in Lkdi
F Case Study for Different f
As shown in Table 5, We further demonstrate a case to help visualize the benefits of using RAG model feedback. The conversation first discusses about T / reservoir) However, in the last turn, they start a new topic about HE E TTanzhuang Reservoir). Among the 4 model predictions, only the 4-th one is correct and the others are still about the last topic. Using based on BM25, these queries are scored quite similar and the 3-rd one are scored highest. But our designed f based on RAG returns more reasonable scores. The correct one is scored much higher than other queries.
Lokd cos cos_coossnnn cos_sim(p(]] | Keii (aee) Xesi Orgt)),
where cos cos_sim the cosine similarity function. Intuitively, probability distribution provides richer information from gycc- However, it can be poorly
Il For all these pretrained models, we use the checkpoints from ht /ttpp://huggiin wtt.gaa.........elllll co/models


Reviews:
[{"review_id": "uEykyl4xrK", "reviewer": "Reviewer_VdLz", "summary": "", "questions": "How does the introduction of RAG instead of BM25 affect the computational cost? Is there any quantitative comparison available?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Novelty of the paper\n\nDifferent from [1], we are the first to explore domain adaptation on this task and set up the challenging NOISY settings suffering noisy information from a commercial search engine and variation of conversations. RAG scores and knowledge distillation are adopted to tackle the problems above, which have also not been explored before. We believe our work is more practical and solid than [1].\n\n2. More in-context learning baselines\n\nWe have tested ChatGPT but it does not give more improvement (ChatGPT vs. text-davinci-003 ROUGE scores 59.45 / 49.77 / 58.22 vs. 63.19 / 48.15 / 61.99 on KdConv). We notice that it tends to reply to the dialogue instead of completing the query, though 8 demonstrations are provided. This may be because of its nature of chatting.\n\nWe also test introducing speaker roles and task definition. They did not help the models based on T5-base in our early study but indeed improve the LLM peformance (ROUGE scores 67.53 / 54.23 / 66.24 on KdConv). However, it still does not outperform the basic T5-base baseline (70.95 / 59.11 / 70.08), thus our conclusions still hold. \n\nNevertheless, leveraging LLMs to complete this task is still under-exploring. We will update these results in the revision. \n\n3. Computational cost for the introduction of RAG instead of BM25\n\nOur framework only takes additional training costs at stage 2, where the RAG model is trained on the target domain data.  As both RAG and BM25 scores are only used for training, these models enjoy the same inference speed as the basic baseline, T5-base.\n\n[1] Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is complete and the motivation is clear.\n2. The experiments are sufficient and effective in supporting the views of the paper.\n3. The paper is well-organized and easy to read.", "reasons_to_reject": "1. My main concern is the novelty of the paper. The authors mainly follow the architecture of Wang et al., and improve it by introducing RAG and knowledge distillation loss. These components are not new, which makes the contribution of this paper limited.\n\n2. There is no sufficient comparison with the in-context learning method. Although the authors provided the results of text_davinci_003  in Table 2, it seems not optimal based on the prompt setting provided in Appendix C. For example, the system role was not set, that is, telling the model to complete a query generation task instead of directly giving Input-output pairs. Additionally, the authors did not use the better GPT_3.5_tubo on both CLEAN and NOISY settings. It would be interesting to test the performance of the latest LLMs on this task.", "questions_to_authors": "How does the introduction of RAG instead of BM25 affect the computational cost? Is there any quantitative comparison available?", "paper_topic_and_main_contributions": "The paper focuses on domain adaptation for the task of conversational query production, and the authors mainly follow the framework proposed by Wang et al. This study makes  improvements to two weaknesses. Firstly, they introduce a RAG model based on T5 to generate better reward socres. Secondly, a regularization term based on knowledge distillation is introduced to make the reinforcement learning training more stable. The authors have fully verified the effectiveness of the above improvements through experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nHLsaDev7y", "reviewer": "Reviewer_NqYR", "summary": "", "questions": "- CLEAN is only experimented with English and NOISY is only experimented with Chinese, perhaps the setting to experiment with a challenging setting in English should be suggested in the paper?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Figure, writings and missing citations\n\nWe will definitely improve these parts. Thank you for your suggestions.\n\n2. Challenging setting in English\n\nAs commercial search engines are actually not free and we only have free access to Sogou, a Chinese one, thus we experiment on Chinese benchmarks for NOISY settings. We believe the conclusions can be the same when using different languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper takes care to introduce the relatively new task.\n2. The paper experiments with both English and Chinese in different settings.\n3. Thorough experiments are performed in regard to reinforcement learning, loss ablations, and retrieval models.", "reasons_to_reject": "1. Figure 2 is too basic, might be good to write the model names and component names both in each case (i.e. Query Producer (T5)), and also explicitly label RL training and other processes in the figure.\n2. How Question Answering domain can be adapted to dialogue tasks has been studied previously, in regards to knowledge retrieval and generation (citations below). They should be mentioned in the paper (line 049).\n3.  skd-loss is only mentioned in the ablation study (section 4.3) without details. Experimental details should be included, at least in Appendix.\n4. The paper should discuss more future directions on this task, as it’s very briefly discussed in Limitations and only regarding RAG models.", "questions_to_authors": "- CLEAN is only experimented with English and NOISY is only experimented with Chinese, perhaps the setting to experiment with a challenging setting in English should be suggested in the paper?", "paper_topic_and_main_contributions": "This paper tackles the conversational query production task, which is a task to generate search queries. It uses reinforcement learning to train the query producer model using RAG model feedback. It also explores other techniques such as knowledge distillation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- PK-ICR: Persona-Knowledge Interactive Context Retrieval for Grounded Dialogue https://arxiv.org/abs/2302.06674\n- Reason first, then respond: Modular Generation for Knowledge-infused Dialogue https://arxiv.org/abs/2111.05204"}}, {"review_id": "ex8ahvLFj5", "reviewer": "Reviewer_HE7F", "summary": "", "questions": "**Question A:**\n\nHow does the reward normalization mentioned in Line 325 affect the results?\n\n**Question B:**\n\nWhat computational budget and computing infrastructure were used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Necessary steps for processing search results\n\nFor Wikipedia search, we follow [1] to use the summary of the wikipedia page. For Sogou search, we use the returned snippets from the search engine and clean the special tokens in them (such as email address and emoji). We will append these details and release the code.\n\n2. Pseudo query for knowledge distillation\n\nWe first follow common practice [2] to obtain pseudo query, then filter some bad ones when their query score s lower than the given threshold. We will improve this part in the revision.\n\n3. Annotations lack of reliability\n\nAs described in Appendix B, our annotators are Chinese native speakers and target queries are easier to recognize when the responses are provided. We will definitely check the annotations and also release them for future studies.\n\n4. Evaluation metrics\n\nUni. F1 has been adopted to evaluate query production in [3] and we additionally adopt BLEU and ROUGE to have better evaluation. Please note that we also provide the response generation results in Table 4, which also show the effectiveness of our framework.\n\n5. End-to-end RL baselines\n\nInteresting suggestions! However, better performance might not be expected from these baselines. As shown in [4], the end-to-end model (RL-F1) does not yield better performance than other methods (RL-QR, RL-C) for QA tasks. Besides, it can be more difficult to obtain good rewards for dialogue tasks because a dialogue response is much more diverse than a question answer span. In this work, we leverage the retrieval scores from an RAG model. It is more interpretable and we also conduct instance selection based on its scores for better training.\n\n6. Reward normalization\n\nIt aims to adjust the reward within [0, 1]. Otherwise, the reward values can vary a lot across different instances, making the training unstable.\n\n7. Computational budget\n\nAll experiments are conducted on a single P40 GPU (24G).\n\n8. Detailed related work\n\nWe will improve this part. Thank you!\n\n[1] Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production.\n\n[2] Understanding Knowledge Distillation in Non-autoregressive Machine Translation.\n\n[3] Link the World: Improving Open-domain Conversation with Dynamic Spatiotemporal-aware Knowledge.\n\n[4] Reinforced question rewriting for conversational question answering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a more accurate and effective RL feedback for query generation and make a step in developing domain adaptation methods. The method's effectiveness is encouraging because it lessens the reliance on human annotations.\n2. Given that the training stability of RL is crucial, the authors do propose two simple yet effective strategies with detailed ablation studies.", "reasons_to_reject": "**1. The proposed method needs to be clarified with more details.**\n\n- For example, this paper uses a search engine to obtain knowledge documents (Line 143). However, it fails to provide the necessary steps for processing search results. \n\n- In addition, this paper introduces knowledge distillation as an effective and important strategy but omits details on how to determine the pseudo query from the source generator in Line 345.\n\n**2. The query generation evaluation on the target-domain dataset could be improved.**\n\n- As the target-domain datasets used in this paper have no query annotation, this paper manually annotates some instances for evaluation. However, the annotations lack of reliability, which is briefly introduced without the quality control of collected data. And the word-overlap metrics between generated queries and human-written queries do not guarantee end-to-end performance (i.e., developing a better response) and therefore are not robust enough to show effectiveness.\n\n- It would be beneficial to compare with additional RL methods. One such method, as demonstrated by Chen et al. (2022) (citation provided below), utilizes end-to-end performance as the feedback for RL.", "questions_to_authors": "**Question A:**\n\nHow does the reward normalization mentioned in Line 325 affect the results?\n\n**Question B:**\n\nWhat computational budget and computing infrastructure were used?", "paper_topic_and_main_contributions": "This paper focuses on the problem setting of domain adaptation for conversational query generation. One of the prior works adopts reinforcement learning with nonparametric retrieval signals (i.e., BM25 score) as feedback to train query generation. This paper builds on that research by proposing a new reward function and implementing two strategies to improve stability in the RL training process. The adaptation pipeline contains three steps: \n1. train a query generator on the source domain; \n2. train a RAG model on the target domain; \n3. train the query generator on the target domain with RL. The reward depends on the parametric retrieval score from the previously trained RAG.\n\nThe main contribution as I see is the new reward function used in RL.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Chen Z, Zhao J, Fang A, et al. Reinforced question rewriting for conversational question answering[J]. arXiv preprint arXiv:2210.15777, 2022.", "Typos_Grammar_Style_and_Presentation_Improvements": "- The related work can be more detailed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for conversational query production, where search queries are generated from interlocutor utterances during a dialogue. The method builds on previous works by introducing RAG and knowledge distillation techniques.

**Pros**: After rebuttal, reviewers find the paper is well-written with most claims sufficiently defended by experiments. Most reviewers agree the paper is complete, with clear motivations and coverage of multiple languages.

**Cons**: While most concerns on soundness are addressed during rebuttal, the reviewers discussion reveals some disagreement on perceived novelty of the work. It is noted the work will be useful to the community, but in terms of methodology, the work is perceived as mostly incremental.

Given the disagreement, I took a look into these details myself. There is some lack in methodological novelty, and perhaps, more of the novelty comes from the new evaluation setup (domain adaptation) which highlights the need for the method. Still, this aspect of the paper appears somewhat underdeveloped (limited domains, no domain adaptation specific generation baselines, etc.) with most of the papers' focus on the methods.
Decision: Accept-Findings