Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems
Authors: Tianyuan Shi
Liangzhi Li
Zijian Lin
Tao Yang
Xiaojun Quan
Qifan Wang
Venue: EMNLP 2023 Main (2023)
Keywords: End-to-End Task-Oriented Dialogue System
Knowledge Retrieval
Retriever training

Abstract: Efficient knowledge retrieval plays a pivotal role in ensuring the success of end-to-end task-oriented dialogue systems by facilitating the selection of relevant information necessary to fulfill user requests. However, current approaches generally integrate knowledge retrieval and response generation, which poses scalability challenges when dealing with extensive knowledge bases. Taking inspiration from open-domain question answering, we propose a retriever-generator architecture that harnesses a retriever to retrieve pertinent knowledge and a generator to generate system responses. Due to the lack of retriever training labels, we propose relying on feedback from the generator as pseudo-labels to train the retriever. To achieve this, we introduce a dual-feedback mechanism that generates both positive and negative feedback based on the output of the generator. Our method demonstrates superior performance in task-oriented dialogue tasks, as evidenced by experimental results on three benchmark datasets.

Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems
Tianyuan Shi', Liangzhi Li Zijian Lin', Tao Yang', Xiaojun Quan'; Qifan Wang'
School of Computer Science and Engineering, Sun Yat-sen University, China
"Meetyou AI Lab, 'Meta Al (shity6, yang12255 @ maill.sysuu.du..cm quanxj3 @ mail.sysu.edu..nn liliangzhi @ iirinnghi@xiaoyyuuv.ccomm xiaoyouzi.com, wqfcr@ @gfcrr@f.com fb.com
Abstract
None
0.5
KB
Efficient knowledge retrieval plays a pivotal role in ensuring the success of end-to-end task- oriented dialogue systems by facilitating the se- lection of relevant information necessary to ful- fill user requests. However, current approaches generally integrate knowledge retrieval and re- sponse generation, which poses scalability chal- lenges when dealing with extensive knowledge bases. Taking inspiration from open- open-domain question answering, we propose retriever- generator architecture that harnesses retriever to retrieve pertinent knowledge and generator to generate system responses. Due to the lack of retriever training labels, propose relying feedback from the generator as pseudo-labele train the retriever. To achieve this, intro- duce a dual-feedback mechanism that generates both positive and negative feedback based on the output of the generator. Our method demon- strates superior performance in task-oriented dialogue tasks, as evidenced by experimental results three benchmark datasets. Our code is available at ht nttpss/ / /gi thub. com/ com/Stycoo/ Dual-Feedbace Fell-Feedbcc-TOOO
2 2,8
delees
ctrmz
oHl
Feedbock
Retriever
Generator
Retrdewed Enttiees
Plac
Fool
Ste
Hewr
rakin:
check
01:2-
Migha
2kk
6883
Chane
4ky
need a moderately priced Chinese restaurant located USER in the east of the city.
Chesteron may be what you're
looking for. Would you like know more about it?
Figure I: Visualization of erroneous feedback from the generator for retriever training. Entities with darker colors indicate higher relevance scores. The generator mistakenly identifies "Milton" as the most relevant en- tity, whereas the correct entity should be "Chesterton".
retrieval remains a critical component that necessi- tates long-term research in dialogue systems.
Traditional pipeline approaches in TOD systems involve multiple modules such as dialogue state tracking and dialogue policy learning, which heav- ily rely on annotated belief states for knowledge retrieval (Lei et al., 2018; Yang et al., 2021). In contrast, end-to-end task-oriented dialogue (E2E- TOD) systems aim to generate responses in sin- gle step without the need for intermediate retrieval annotations, thereby highlighting the importance of external knowledge retrieval. Existing E2E- TOD systems can be classified into three categories based on their utilization external knowledge. Firstly, memory networks are employed to store the knowledge, and multi-hop interactions are de- signed to aggregate relevant information (Madotto et al., 2018; Qin et al., 2020; Raghu et al., 2021), 1). Secondly, pre-trained language models encode lin- earized KB records, which are then used input for the response generator (Xie et al., 2022: Wu et al., 2022; Tian et al., 2022). Thirdly, the knowl- edge base can be embedded into model parame- ters through data augmentation, enabling implicit
1 Introduction
Task-oriented dialogue (TOD) systems (Eric et al., 2020) are designed to fulfill specific tasks, such as hotel bookings, through natural language conversa- tions with users. These systems can be integrated into applications such as chatbots and voice as- sistants, serving various industries like hospitality, e-commerce, and customer service. To generate in- formative system responses, TOD systems typically rely on an external knowledge base (KB) to retrieve relevant entity information. While large language models (LLMs) like ChatGPT have demonstrated impressive capabilities in understanding multi-turn dialogues and generating fluent responses, there still cases where they require access to localized KBs to handle specific tasks. Therefore, knowledge
Conresponding authors.

---

knowledge retrieval (Madotto et al., 2020; Huang al., 2022). These approaches typically integrate the processes of knowledge retrieval and response generation and train them under the supervision of reference responses. However, this approach suffers from two notable shortcomings. Firstly, the system response often comprises both pure lan- guage tokens and KB-related tokens (e.g.. hotel names and addresses), making it challenging to train an effective retriever using weak supervision signals from reference responses. Secondly, the efficiency of these systems may decrease as the knowledge base expands in size.
from the generator's hypothesis responses. Then, we construct negative feedback based on these neg- ative samples to calibrate the positive feedback.
We conducted evaluations of our system on three well-established TOD datasets: Multi-WOZ 2.1 (MWOZ) (Eric et al., 2020), Stanford Multi- Domain dataset (SMD) (Eric et al., 2017), and CamRest (Wen et al., 2017). The experimental results demonstrate that our model outperforms the baseline methods, particularly in scenarios in- volving large-scale KB, Additionally, through ex- tensive analysis, we have made several notable findings. Firstly, our retriever exhibits clear ad- vantages over the baseline methods as the size of KB increases. Secondly, our dual-feedback mecha- nism effectively mitigates the problem of incorrect knowledge learned solely with positive feedback from the generator. Lastly, our method exhibits good compatibility with LLMs like ChatGPT.
Unlike the aforementioned works, we employ a retriever-generator architecture that explicitly sep- arates the retrieval process from response genera- tion. The retriever is responsible for identifying relevant information from the KB, while the gen- erator utilizes the dialogue context and retrieved information generate the response. Although this architecture seems straightforward, constructing an effective retriever presents significant challenges, particularly in two key aspects. Firstly, TOD sys- tems inherently possess ground-truth responses to train the generator, but lack annotations for training the retriever. Thus, the retriever can only be trained using weak supervision signals derived from the response generator. Secondly, within specific domain, different entities often exhibit structural and content similarities, making challenging for the generator to learn which entities are truly rele- vant. Consequently, the weak supervision signals from the generator may not be reliable. Figure provides a visual illustration of these challenges.
2 Related Work
2.1 End-to-End Task-Oriented Dialogue
E2E-TOD systems employ different strategies incorporate KB information for response genera- tion. Firstly, memory networks are utilized store knowledge, using multi-hop interactions to aggre- gate relevant information. For instance, Mem2seq (Madotto et al., 2018) employs multi-hop attention over memory cells to select KB-related tokens dur- ing response generation. GLMP (Wu et al., 2019) introduces global-to-local memory pointer net- work retrieve relevant triplets and complete the response template. CD-NET (Raghu et al., 2021) retrieves relevant KB records by computing a dis- tillation distribution based on dialogue context.
To tackle these challenges, we propose a dual- feedback mechanism for the retriever, consisting of positive feedback and negative feedback. Positive feedback is constructed based the conditional generation probabilities of responses correspond- ing different retrieved entities. Intuitively, the relevance between an entity and the response is higher, the conditional generation probability of the response corresponding to this entity will also be higher. Utilizing positive feedback allows us to train the retriever based on the knowledge learned by the generator from reference responses. In or- der to prevent the retriever from being misled by inaccurate information, we contend that calibra- tion necessary. Calibration involves the initial identification and resolution of errors, which we accomplish by sampling negative samples derived
Secondly, the entire lincarized KB is encoded by pre-trained language models and taken as input to generator to generate the final system response. For instance, UnifiedSKG (Xie et al., 2022) uses unified text-to-text framework, while Q-TOD (Tian et al., 2022) rewrites the dialogue context into a natural language query for knowledge re- trieval. MAKER (Wan et al., 2023) introduces a multi-grained retrival with both entity and attribute selection.
Thirdly, knowledge bases stored in model pa- rameters for implicit retrieval during response gen- eration. GPT-KE (Madotto al., 2020) embeds the KB into pre-trained model parameters through data augmentation. Following GPT-KE, ECO (Huang et al., 2022) first generates the most relevant entity

---

with trie constraint before response generation to ensure entity consistency in the response.
respectively, we denote the dialogue context of the t-th turn as G {ui, yi...., 4 1t-1 3r An external KB is provided, represented as a set of entities, i.e., & = {el, e2...., where each entity 4 consists of attribute-value pairs, i.e., e; = = ,/,...,""" of }. E2E-TOD takes the dia- logue context e₁ and the KB as input and directly generates natural language response yt.
2.2 Knowledge Retrieval
Previous research has extensively explored meth- ods for knowledge retrieval across various tasks, including question answering (Chen et al., 2017; Kwiatkowski et al., 2019), fact checking (Thorne et al., 2018), and dialogue systems (Dinan et al., 2019). Recently, neural network- based dense re- trievers have become popular. These retrievers com- monly use dual-encoder architecture (Yih et al., 2011), where queries and passages are encoded as separate vectors, and relevance is determined through inner product or Euclidean distance.
3.2 Knowledge Retriever
Our knowledge retriever comprises encoder Ene, that maps any input to d-dimensional vec- tor. The user utterances and system responses the dialogue context et are first concatenated and encoded by Enc, as the query. To encode an en- tity, we concatenate the attribute-value pairs of the entity into sequence and pass it to Enc,. The similarity score between ce and is computed by taking the dot product of their respective vectors:
Supervised retrievers, such as DPR (Karpukhin et al., 2020), have been developed for open- -een-domain question answering. These retrievers trained us- ing labeled cnestion-documeed pairs. To overcome the need for costly query-document annotations, researchers have explored alternative approaches that leverage signals from the answer generator. REALM (Guu et al., 2020) and RAG (Lewis et al., 2020) propose joint training of the retriever and the generator by treating documents as latent variables, FiD-KD (Izacard and Grave, 202 a) employs cross- attention scores supervision through knowledge distillation. EMDR² (Sachan et al., 2021) mod- els retrieval decisions as latent variables and em- ploys an expectation maximization algorithm to approximate the computation. Unsupervised ap- proaches have also been explored. ICT (Lee et al., 2019) introduces the inverse cloze task for unsuper- vised pre-training of dense retrievers. Izacard et al. (2022) investigate contrastive learning methods for training retrievers, while Spider (Ram et al., 2022) utilizes recurring spans within document create pseudo-positive query-document pairs.
St = (nc,( Fan (c:)..
(1)
Based on these similarity scores, we identify the top- K entities from the entity set & the candi- date entities for response generation. This set of candidates is represented as & {ei, 62, ..., ex): We employ BERT (Devlin et al., 2019) the en- coder and extract the representations of the (CLS) token to represent e and 6, Prior research has highlighted that initializing the encoder directly with BERT weights can lead to collapsed represen- tations and impact retrieval performance. Conse- quently, we initialize the weights via pre-training with distant supervision (Qin et al., 2019).
3.3 Response Generator
Our generator is built upon the Fusion-in-Decoder (FiD) (Izacard and Grave, 2021b) model, which is based on the pre-trained T5 (Raffel et al., 2020). The FiD model consists of an encoder Enc, and a decoder Decy. The encoder is responsible for processing K different text inputs independently, where each input Tt,i is formed by concatenating the dialogue context ot and candidate entity ej. The output representations of the encoder then concatenated create global representation. X; [Ency (at,y((rr))):) (Ir,,),.... Ency (x,, for the current turn. The decoder takes X, as input and generates the response autoregressively. During this process, the decoder utilizes causal attention to incorporate information from previously generated tokens,
3 Methods
As illustrated in Figure 2, our system comprises a knowledge retriever and response generator. The retriever first retrieves top-K K relevant entities from knowledge base. These retrieved entities, along with the dialogue context, are then fed into the generator model to generate the response.
3.1 Notations
Given a dialogue = (u1, 3/1 ..., ur, 3r) con- sisting of T turns, where ue and yt represent the user utterance and system response at the f-th turn,
Details of the pre-training are available in Appendix D.

---

Positive Feedback
Apos
Xnowledge Base
a²
a²
Gr Entily Sporing
Retriever
Generator
of
Tap-K Eniiiies
1%
89
Enc,
a²
Encg Enng EnGol
vg
Xxs
DeCg
Mg
v
Inner Product
X
p(xllee)) )
98
LALL
Dialogue Context
Enc,
Negalive Semple
Entlyy Scoting
4r
Negativee Foolbbak
The overall architecture of our end-to-end task trientrde dialogue system, comprising knowledge retriever and a response generator. The retrieval component is trained using dual feedback from the response generator.
Figure 2:
well as cross-attention to the input tokens repre- sented by X₁- This enables the decoder to consider information from both the generated tokens and the tokens of retrieved entities. The probability of the response is defined follows:
between the retrieval scores S, - IeisK ob- tained from Eq. (1) and the pseudo-labels G2 using KL-divergence as follows:
I Tppos = DKi(GE,
1y
(4)
Σ ge. (log Ht.s log st,i),
Xi) II
p(y,,lue,ss, X1), i=1
where [Bel represents the length of the response ye-
where Ôt.; = softmax (G:) 8t, softmax (St)
The use of the generator's supervision train the retriever is referred to as positive feedback. How- ever, important note that the generator may assimilate incorrect knowledge from the entities, leading to inaccurate entity relevancy scores.
3.4 End-to-End Training
Entity scoring Given the retrieved entities & {ei, 62, Cx) we employ the aforementioned re- sponse generator to assign a score to each entity. Firstly, the dialogue context ς and each entity es are concatenated to form Tt,j. which is then fed into the generator. The conditional log probability Sength-normalizede of the response yt is utilized to score entity 6, follows:
Negative feedback To address the issue of positive feedback when training the retriever, we propose incorporating negative feedback from the generator to calibrate the pseudo-labels. To achieve this, we select a negative sample for the global represen- tation X, during the generation process. A neg- ative sample refers to response that exhibits a high generation probability but is of low quality. Specifically, we generate
9t.i = pi,)
Σ 1= log ((pp(( 1u, Xx.))
(3)
Isis of sponses using beam search, where M is the beam search size, and rank them based on their genera- tion probabilities. We use Rę (()) to represent the rank of candidate response ÿî- Note that a high generation probability does not guarantee high- quality response. To evaluate the quality, we define function o(0)- y1) that measures the overlap be- tween response and the reference response. We implement o(ūi, Me) using the BLEU metric. As result, we obtain new sorted list responses based on their true quality, where the rank can- didate response 0f is R2 Finally, we identify
y/
where X1, Ency (.),))
The rationale behind the scoring is straightfor- ward: if an entity is pertinent to the response, the probability of generating this response given the dialogue context and the entity should also be high. Positive feedback We utilize the entity scores obtained from the generator pseudo-labels train the retriever, with the ob- jective of transferring the knowledge acquired by the generator from reference responses to the re- triever. To achieve this, we enforce consistency
G,

---

the negative sample follows:
answering (Karpukhin et al., 2020). Re@ K mea- sures the effectiivneess of the retriever by determin- ing the percentage of gold attriibute values covered by the entities retrieved in the top- list. Secondly, we assess the overall performance of our TOD sys- tem from two aspects. To evaluate the system's capability to generate relevant entities, employ the micro Entity-F1 metric (Eric et al., 2017). Ad- ditionally,
= argming (R (())
(5)
Intuitively, it is desirable for the retriever to avoid incorporating the signal from the negative sample when updating its parameters. Therefore, we employ Eq. (3) to calculate the score of each entity in generating the negative sample, and uti- lize the scores G2 for. for all retrieved entities to calibrate the positive feedback by intro- ducing a margin loss:
utilize the BLEU metric to measure the N- N-gram overlap between the generated response and the reference response. With these metrics, we can comprchensively evaluate the performance of our model in both retrieval and TOD systems.
Inse max (0, (D 8.) (Go S.)) +
(6)
4.3 Experimental Settings
The training objective of the retriever comprises pos and neg. Additionally, we train the re- sponse generator by minimizing the negative log- likelihood loss:
We instantiate the knowledge retriever using the BERT-base model. As for the generator, we instan- tiate it with T5 of varying model sizes: T5-base and T5- T5-large. Both the retriever and generator models fine-tuned using the ADAM algorithm (Kingma and Ba, 2015) with different learning rate sched- ulers. The retriever model employs fixed learning rate scheduler, while the generator model uses a linear learning rate scheduler. The experiments are conducted on a single 24G NVIDIA RTX 3090 GPU. We select the checkpoint that yields the best results on the validation set. For more detailed in- formation regarding experimental setup, please refer to Appendix B.
loel
LNLL = P (tt. X²).
(7)
The final training objective is a combination of the objectives of the retriever and the generator:
= LNLL + Lpus Ling-
(8)
4 Experiments
this section, we provide a detailed description of the experiment and present the main results.
4.4 Baselines
In our comparison, we classify the existing ap- proaches for E2F-TOD systems into three cate- gorics based on their utilization of KB.
4.1 Datasets
We conduct our evaluation using three publicly available TOD datasets: Multi-WOZ, titii-WO.222 2.1 (MWOZ) (Eric ct al., 2020), Stanford Multi-Domain dataset (SMD) (Eric et al., 2017), and CamRest (Wen et al., 2017). These datasets contain dialogues that are associated with relevant KBs. This is referred to the session-level KB. To construct a compre- hensive and extensive knowledge base, merge the session-level KBs corresponding each dialog turn, resulting in dataset-level KB. The provided train, validation, and test splits for each dataset are utilized in our evaluation. The statistics of the three datasets are summarized in Appendix A.
Memory networks: These methods store ex- ternal knowledge in memory cells in the form of triplets and utilize multi-hop attention to retrieve relevant information for generating responses. Ex- amples include DSR (Wen et al., 2017), KB- Retriever (Qin et al., 2019), GLMP (Wu al., 2019), DF-Net (Qin et al., 2020), FG2Seq (He al., 2020), and CDNET (Raghu et al., 2021).
Linearized KB: These approaches leverage pre- trained language models to encode the entire lin- earized KB, along with the dialogue context, input for assisting response generation. Examples include DialoKG (Rony et al., 2022), UnifiedSKG (Xie al., 2022), and Q-TOD (Tian al., 2022).
4.2 Evaluation Metrics
We evaluate the performance of our model using two metrics. Firstly, we calculate the top-K re- trieval recall (Re@K) which is inspired by the widely used top-K K retrieval accuracy in question
Model paramefers: These approaches encode the KB into model parameters through data aug- mentation of the dialogues with KB records, en- abling implicit retrieval during response genera-

---

MWOZ
SMD
CamRest
Model
BLEU
Entity-Fi
BLEU
Entity-Fi
BLEU
Entity-F1
GLMP
6,901
32.40
13,90
60.707
15.10
58.90
DF-Net
9.40
35.10
14.40
62.70
GPT2-KE
9.40
35.10
14.40
62.70
FG2Seq
14.60*
36.50*
16.80*
61.10*
20.20*
66.40*
CDNET
11.90
38.70
17.80
62.90
21.80
68.60
DialogKG
12.60
43.50
20.00
65.90
23.40
75.60
UnifiedSKG (T5-large)
13.69
46.04
17.27
65.85
20.31
71.03
Q-TOD (T5-base)
20.14
68,22
Q-TOD (TS-large)
17.62
50.61
21.33
71.11
23.75
74.22
Ours (T5-base)
18.26
52.52
24.12
69.36
25.85
72.83
Ours (T5-large)
18.48
53.17
25.10
71.58
26.00
74.04
Table 1: Main results of E2E- TOD systems with session-level KBs on MWOZ. SMD, and CamRest, with the best scores highlighted in bold. 1, *, and & indicate that the results sourced from (Qin al., 2020), (Raghu al., 2021), and (Tian et al., 2022), respectively.
tion. Examples include GPT-2+KE (Madotto et al., 2020) and ECO (Huang et al., 2022).
method outperforms Q-TOD on all three datasets using T5-large. We believe this because Q-TOD trains the retriever independently, whereas our pro- posed dual-feedback method allows for joint train- ing of the retriever and generator. This facilitates better alignment between the retrieved entities and their relevance to the current dialogue context.
MWOZ
CamRest
Model
BLEU
Entity-F1
BLEU
Entity-F1
De-Nee
6.45
27.31
PG25eq
10.74
33.68
19.20
59.35
CDNET
10.90
31.40
16.50
63.60
Q-TOD (TS-
16.67
47.13
21.44
63.88
Ours (SS-base)
17.61
51.61
27.39
70.74
Dataset-level KB Table 2 presents the results of our system on the dataset-level KB and compares with the reimplementation several well-known E2E- BOD systems. Our system demonstrates sig- nificant advantages, particularly in the Entity-FFl metric. It achieves an improvement of 5.83 on the MWOZ dataset and 9.7 on the CamRest dataset over the strong Q-TOD baseline.
Ours (T5-large)
18.36
$2.96
26.61
73,58
Table Main results with dataset-level KBs on MWOZ
and CamRest. Best scores highlighted in bold.
4.5 Main Results
We conducted experiments in both session-level and dataset-level KB scenarios. In the following paragraphs, we diseuss the detailed results.
Furthermore, when comparing the experimental results in the two KB settings, we observe that our model exhibits more stable performance. As the KB size increases, our model experiences only a mi- nor decrease in performance, while other baseline models show noticeable decline. For instance, DF-Net exhibits a decrease of 7.79 in terms of Entity-F1 the MWOZ dataset, indicating that the method struggles to adapt to large-scale KBs. Similarly, Q-TOD experiences reduction of 3.48 in Entity-F1 on MWOZ, highlighting the less stable performance its independently trained retriever.
Session-level KB The results for the session- level KB setting are summarized Table 1. Our system, instantiated with T5-large, achieves state- of-the-art performance on the MWOZ and SMD datasets. Specifically, our method demonstrates an improvement of 2.56 in the Entity-FI metrie for MWOZ and 0.47 for SMD over Q-TOD. Moreover, our system achieves the highest BLEU scores, with an increase of 0.86 on MWOZ, 3.77 on SMD, and 2.25 on CamRest compared to Q-TOD. However, our system does not achieve the best performance in terms of Entity-Fl on the CamRest dataset. This can be attributed to the presence of session-leve KBs containing only 1-2 entities, which poses challenge for the retriever perform optimally.
Analysis and Discussion
To further investigate our method, conduct comprehensive analysis that includes an ablation study, an exploration of different retriever train- ing methods, an examination different negative sample selecting methods, and an assessment
Note that Q-TOD also employs Transformer- based response generator similar to ours and uti- lizes manually annotated queries. However, our

---

Dataset-level KB
Session-level KB Entity-F1
Method
BLEU Entity-F1
Re@7
BLEU
Re@3
Ablation
Ours
17.61
51.61
90.98
18.26
52.52
79.26
w/o negative feedback
17.54
50.32
87.97
17.13
51.49
72.93
w/o positive feedback
16.05
48.07
83,46
17.64
50.48
76.69
Retriever training methods
Attention-basee w/ negative feedbeak
16.42
49.44
85.71
16.61
51.16
69.17
16.83
50.23
89.47
16.73
52.08
77.78
MML-based
17.24
49.26
84.15
17.47
51.07
70.05
Table 3: Results of ablation study and different retriever training methods, where "w/" means "with" and "w/o" means "without". "Attention- and "MML-based" indicate the attention-basee (Izacard and Grave, 2021a) and the maximum marginal likelihood (MML)-based (Sachan al., 2021) retriever training methods, respectively.
compatibility with LLMs. base is used as the base model for the first three analyses.
5.2 Methods for Retriever Training
We conduct a comparative experiment with other retriever training methods commonly used in ques- tion answering (QA), including attention-baset (Izacard and Grave, 2021a) and maximum marginal likelihood (MML)-based (Sachan al., 2021) methods. To further demonstrate the necessity negative feedback, we perform additional exper- iments by incorporating negative feedback into the attention-basee method. Note that the underly- ing model architecture remains the same across all methods, with the difference lying solely in the em- ployed training methods. The results presented in the lower section of Table Due the similar performance trends observed for both dataset-level and session-level KB settings, the following analy- sis will focus solely on the dataset-level KB setting. We observe that our proposed method outper- forms the attention based method by 2.26 and the MML method by 3.82 in terms of the @ met- ric. This indicates that in the TOD task where KB-related tokens are sparse in the response, our utilization of conditional probability of response enables better correlations between the retrieved en- tities and the response. Furthermore, when compar- ing the performance of the attention-based method before and after incorporating negative feedback, we consistently observe improvements across all metries. This suggests that even in existing meth- ods, negative feedback can effectively enhance the performance of the retriever and the overall model.
5.1 Ablation Study
We conduct an ablation study the MWOZ dataset evaluate our method under both session- level KB and dataset-level KB settings. In the session-level KB setting, we adjust the retriever evaluation metric from 7 to Re@3 due to the smaller size of the KB. The results of this study are presented in the upper section of Table 3.
By removing negative feedback, we observe a decrease in retrieval metrics (Re@ 7/Re@3) 3) and TOD metrics (BLEU/Entiy-FI), Notably, the re- trieval metrics exhibit more pronounced decline. This suggests that the incorporation of negative feedback allows the retriever to more effectively learn from the generator, resulting in more relevant entities for response generation.
Furthermore, the further removal of positive feedback reduces the model to learnable gen- erator with a fixed retriever. In the datasct-level KB setting, all evaluated metrics exhibit a further decline. This indicates that our positive feedback successfully facilitates the transfer of knowledge learned by the generator to the retriever.
However, in the session-level KB setting, while the Entity-F1 metric decreases, the Re@3 metric increases. We attribute this to the small size of the KB, causing significant fluctuations in the top-3 entities as the retriever updates. Consequently, the generator tends to prioritize the entities that occur most frequently among the retrieved results. While this improves the model's performance in dialogue scenarios that only require a single entity, which constitutes the majority, it hampers performance in scenarios that require multiple entities.
5.3 Methods for Selecting Negative Sample
To understand how different negative sample lecting methods affect the results, we conduct ad- ditional experiments. As depicted in Figure 3, these methods can be categorized into two types: argmin,.. and rank.... The argmin,.. method re-

---

as the generator. We train the retriever using posi- tive and negative feedback provided by ChatGPT. The experimental results are presented Table 4, Notably, since the ChatGPT API does not directly provide the generation probabilities of responses, we make slight adaptations. The details of this experiment can be found in Appendix F. From Table 4, we observe that the best perfor- mance is achieved when incorporating negative feedback in both the zero-shot and few-shot scenar- ios. This highlights the effectiveness of the dual- feedback mechanism in training superior retriever, even when utilizing LLMss. Moreover, the perfor- mance in the few-shot scenario surpasses that in the zero-shot scenario, emphasizing the importance of having few demonstrating examples. These examples provide valuable guidance and enable the model to better understand the task.
34.34
15.0
16.0
9019
(a) BLEU
(b) Entity-FI F1
Figure 3: Results of different negative sample selecting methods on MWO7, where argmin,.. means selecting the response with the lowest score for negative feedback, and rank... combines the scoring of the oracle function with the generation probabilities to select the response.
lies solely on the scoring of the oracle function to select the response with the lowest score con- struct negative feedback. On the other hand, the rank,., method combines the scoring of the ora- cle function with the generation probabilities to select responses with high probabilities but low factual quality to construct negative feedback. Ad- ditionally, we also experiment with using BLEU and Entity-Fl as the oracle functions to assess their influence on the experiments. From Figure 3, have made three notable observations.
Task
BLEU
Entity-Fi
Re@7
Zero-shot
5.95
27.4
83,46
w/ positive feedback
6,21
28.97
85.76
w/ negative feedback
6.98
30.48
87.97
Few-shot
6,78
28.73
83,46
positive feedback
6.43
29.07
84.49
w/ negative feedback
7.14
31.46
88.03
Firstly, the results in Entity-F1 and Re@7 demonstrate consistent trends, indicating that im- provements in retrieval performance lead to en- hanced performance in TOD. This highlights the significance of enhancing the retrieval performance. Secondly, the rank... methods outperform the argmin... methods. Therefore, we conclude that the rank... methods are more accurate in select- ing appropriate negative samples that could reflect the generator's 's mistakes compared to methods that solely rely on the oracle function.
Table 4: Results of employing ChatGPT as the generator on the MWOZ dataset.
Conclusion
In this paper, we propose novel dual-feedback knowledge retriever for E2E-TOD systems. Our approach separates the knowledge retrieval pro- cess from response generation, and leverages the knowledge learned by the generator to create syn- thetic positive and negative feedback for retriever training, eliminating the need for retrieval annota- tions. Through empirical evaluations, we demon- strate that our system achieves state-of-the-an per- formance, regardless of whether a small or large- scale KB is used in each dialogue. Furthermore, ablation studies indicate that our dual-feedback mechanism effectively mitigates the problem of incorrect knowledge learned solely from positive feedback generated by the generator. Consequently, this improvement in retrieval performance directly translates enhanced performance in E2E-TOD systems. Lastly, our method exhibits good compat- ibility with LLMs like ChatGPT.
Lastly, it is observed that using Entity-Fl as the oracle function often results in higher Entity-FI and @ scores, but slightly lower BLEU scores. This suggests that Entity-F1 can provide more pre- cise feedback regarding entity selection compared to BLEU. However, in practical scenarios, obtain- ing gold entity annotations from responses is chal- lenging, despite their availability in the MWOZ dataset. This is why we employ BLEU as the ora- cle function in our experiments.
5.4 Compatibility with LLMss
To demonstrate the compatibility of our proposed dual-feedback mechanism with LLMs, we conduct zhro-shot and few-shot experiments using ChatGPT
Phtpps//openaii.om/api

---

Limitations
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar. Anuj Kumar Goyal, Peter Ku, and Dilek Hakkani- 2020. Multiwoz 2.1: consolidated multi-domain dialogue dataset with state corrections and state track- ing baselines. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 422 -22. European Language Resources Association.
There are three potential limitations to consider in our work. Firstly, the process of obtaining nega- tive feedback through response sampling can result in decreased training efficiency. Secondly, train- ing the retriever using feedback from ChatGPT can be costly. In each epoch, it is necessary to re-predict for every sample, and finding ways to reuse intermediate predictions is an area to explore. Thirdly, there still exists a noticeable gap between fine-tuning the generator and achieving few-shot learning with ChatGPT. Future research is needed to investigate methods for narrowing this gap.
Mihail Fric, Lakshmi Krishnan, François Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, August 15-17, 2017, pages 37-49. Association for Computational Linguistics.
Ethics Statement
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, I0, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
All experiments in this study were conducted using publicly available datasets that do not contain any private information. Our work does not involve the analysis or utilization of identity characteristics, and we do not engage in any form of gender or racial discrimination.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. 2020. Retrieval augmented language model pre-training, In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research. pages 3929-3938. PMIR.
Acknowledgements
This work was supported by the National Natu- ral Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832)
Zhenhao He. Yuhong He, Qingyao Wu, and Jian Chen. 2020. Fg2seq: Effectively encoding knowledge for end-to-end task-oriented dialog. In 2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 8029-8033 8033. IEEE.
References
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Guanhuan Huang, Xiaojun Quan, and Qifan Wang- 2022. Autoregressive entity generation for end-to- end task-oriented dialog. In Proceedings of the 29th International Conference on Computational Linguis- tics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 17, 2022, pages 323-332. International Committee on Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019. Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long und Short Papers), pages 4171 1171-4186. Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave 2022. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn, Res., 2022.
Gautier Izacard and Edouard Grave. 2021a. Distilling knowledge from reader to retriever for question an- swering. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume,

---

EACL 2021, Online, April 19 23, 2021, pages 874- 880. Association for Computational Linguistics.
Andrea Madotto, Chien- Sheng Wu, and Pascale Fung. 2018. Mem2seq: Effectively incorporating knowl- edge bases into end-to-end task-oriented dialog sys- tems. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Vol- ume 1: Long Papers. pages 1468- 1478. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. Associa- tion for Computational Linguistics.
Libo Qin, Yijia Liu, Wanxiang Che. Haoyang Wen, Yangming Li, and Ting Liu. 2019. Entity-consisten end-to-ens end mask-oriented dialogue system with KB retriever. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLL--IJCN) 133-142, 142, Hong Kong, China. Association for Com- putational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
pages
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Bing-Weii&o Chang. Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Libo Qin. Xiao Xu, Wanxiang Che, Yue Zhang, and Ting Liu. 2020. Dynamic fusion network for multi- domain end-to-end task-oriented dialog. In Proceed- ings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 6344 6354, On- line. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with a unified text-to- toxttt trans- former. J. Mach. Learn. Res., 21:140:1-140.67.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics. ACL 2019, Florence. Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 6086-6096. Association for Computational Linguistics.
Dinesh Raghu, Atishya Jain, Mausam, and Sachindra Joshi. 2021. Constraint hased knowledge base distil- lation in end-to-end task oriented dialogs. Find- ings of the Association for Computational Linguis- rics: ACL/ICNLP 2021, Online Event, August 1-6, 2021, volume ACI//ICNLP 2021 Findings of ACI., pages 5051-5061. Association for Computational Linguistics.
Wenqiang Lei. Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018 Sequicity: Sim- plifying task-oriented dialogue systems with single sequence-to-sequene architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1437-1447, Melbourne, Australia. Association for Computational Linguistics.
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to retrieve pas- sages without supervision. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2687- 2700. Association for Computational Linguistics.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval- Leurietal-ugmmenteed augmeeted generation for knowledge-intensive NIP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Md. Rashad Al Hasan Rony, Ricardo Usbeck, and Jens Lehmann. 2022. Dialokg: Knowledge-stnucture aware task-oriented dialogue generation. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2557-2571, Association for Com- putational Linguistics.
Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pas- cale Fung. 2020. Learning knowledge bases with parameters for task-oriented dialogue systems. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL, pages 2372-2394. Association for Computational Linguistics.
Devendra Singh Sachan, Siva Reddy, William L. Hamil- ton, Chris Dyer, and Dani Yogatama. 2021. End-to- end training of multi-document reader and retriever for open- ppmn-domain question answering. In Advances in Neural Information Processing Systems 34: An- nual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, viriual, pages 25968-25981.

---

James Thome. Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 602 -3116,1.444 Association for Computational Linguisties.
Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. Proceedings of the AAAI Conference Artificial Intelligence, 35(16):1423-- 11238-8-43.
Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In Proceed- ings of the Fifteenth Conference on Computational Natural Language Learning, pages 247-256, Port- land, Oregon, USA. Association for Computational Linguistics.
Xin Tian, Yingzhan Lin, Mengfei Song, Siqi Bao, Fan Wang, Huang He, Shuqi Sun, and Hua Wu. 2022. Q- TOD: A query-driven task oriented dialogue system. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7260-7271. Association for Com- putational Linguistics.
A Dataset Statistics
Fanqi Wan, Weizhou Shen, Ke Yang. Xiaojun Quan. and Wei Bi. 2023. Multi culti-grained knowledge retrieval for end-to-end task-oriented dialog. In Proceedings of the ō1st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 11 1000, Toronto, Canada. Association for Computational Linguistics.
In the case of MWOZ, each dialogue session in- cludes 7 candidate entities, while for SMD and CamRest, the sizes of the candidate KBs vary, rang- ing from 0 to 8 and 0 to 57, respectively. Table 5 shows the statistics of the three datasets: Multi- WOZ 2.1 (MWOZ) (Eric et al., 2020), Stanford Molti-Domman dataset (SMD) (Eric et al., 2017), and CamRest (Wen et al., 2017). When we include the entire knowledge base (KB) as input, the length of the input text becomes significantly long, pos- ing challenges for existing end-to-end task-oriented dialogue systems to handle.
Tsung-HHien Wen, David Vandyke, Nikola Mrksic, Mil- ica Gasic, Lina Maria Rojas-Barahona. Pei-Hao Su, Stefan Ultes, and Steve J. Young. 2017. A network- based end-to-end trainable trientree dialogue sys- tem. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Pupers, pages 438 438-449. Association for Computational Linguistics.
Statistics
Sequence Length
Dataset
#Dial
#Utt
Dial
w/ SessionKB
w/ DatasetKB
Chien-Sheme Sheng Wu, Richard Socher, and Caiming Xiong. 2019. Global-to-loce memory pointer networks for task-oriented dialogue. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenKeview.net.
MWOZ
2877
19870
730
996
23730
SMD
3031
15928
109
435
CamRest
2744
156
1356
Table 5: Dataset statistics. We count the maximum input lengths for different cases: dialogue only (Dial), dialogue with session-level KB (w/ SessionKB), and dialogue with dataset-level KB (w/ DatasetKB).
Jie Wu, Ian G. Harris, and Hongzhi Zhao. 2022. Graph- memdialog: Optimizing end-to-end task-oriented dialog systems using graph memory networks. In Thirty-Sixth AAAI Conference on Artificial Intelli- gence, AAAI 2022, Thirty-Fourts Conference on In- novative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Ad- vances in Artificial Intelligence, EAAl 2022 Virtual Event, February 22- March 1, 2022, pages 11,044 11512. AAAI Press.
B Hyperparameters
The hyperparameters of our system with session- level and dataset-level KB are shown in Table 6 and Table 7, respectively.
Tianbao Xie, Chen Henry WU, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin. Sida I. Wang. Vic- tor Zhong, Bailin Wang. Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong. Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022,
C Traing Process
We provide more comprehensive description of the training process below. During the initial train- ing phases, we incorporate warm-up period for the generator, facilitating dependable feedback for the retriever. Subsequent to this phase, both the retriever and generator are subjected to joint train- ing until reaching convergence. It's important to

---

MWOZ
SMD
CamRest
Parameters
T3-base
T5-large
TS-base
T5-large
TS-base
TS-large
Retriever LR
2e-5
2e-5
2e-5
2e-5
2e-5
2e-5
Retriever IR schedule
Fixed
Fixed
Fixed
Fixed
Fixed
Fixed
Retriever input length
128
128
128
128
128
128
Top-K retrieved entities
Retriever training start step
625
625
750
750
750
750
Generator LR
lc-4
lc-4
lc-4
lc-4
lc-4
lc-4
Generator LR schedule
Linear
Linear
Lineur
Linear
Linear
Lineur
Generator imput context max length
200
200
200
200
200
200
Generator input KB length
100
100
200
200
200
Response max length
64
64
Beam search size
5
5
5
5
Oracle function
BLEU
BLEU
BLEU
BLEU
BLEU
BLEU
Barch size
2
2
Training steps
1500
1500
1500
1500
1000
1500
Grad accumulation steps
32
Table Hyperparameter settings our system when session-level KBs are used on MWOZ, SMD and CamRest.
MWOZ
CamRest
Parameters
T5-base
T5-large
T5-base
T5-large
Retriever LR
2e-5
2c-5
2c-5
2e-5
Retriever schedule
Fixed
Fixed
Fixed
Retriever input max length
128
128
128
128
Top-K retrieved entities
10
10
10
7
Generator LR
le-4
lc-4
lc-4
lc-4
Generator schedule
Linear
Linear
Linear
Linear
Generator input context max length
200
200
200
200
Generator input KB max length
100
100
200
200
Response length
64
64
64
Beam search size
5
Retriever training start step
750
750
750
750
Oracle Function
BLEU
BLEU
BLEU
BLEU
Batch
2
Training steps
1500
1000
1500
Grad accumulation steps
32
64
32
32
Table 7: Hyperparameter settings of our system when the dataset-level KBs are used on MWOZ and CamRest.
emphasize that carry out negative sampling at each training step, ensuring consistent information updates from the generator to the retriever.
While the inclusion of sampling operations dur- ing training does indeed extend the overall training duration, it is important to note that these sampling operations exclusively carried out within the training phase and, therefore, do not impact the inference time.
Parameters
MWOZ
CamRest
Batch size
128
108
Epoch
10
15
LR schedule
Linear
Linear
LR
Se-5
Se-5
Max input length
128
128
Pooling type
CLS
CLS
D Retriever Pre-training
Weight decay
0.01
0.01
Given dialogue context and the system response, utilize the entity with the highest frequency of its attribute values in the dialogue context and system response as the label. To optimize this pro- cess, we employ supervised contrastive learning (Gao et al., 2021). Specifically, the positive ex- ample for dialogue context is the corresponding
Table 8: Hyperparameter setting for pre-training triever on the dataset-level KBs of MWOZ and CamRest datasets, respectively.

---

labeled entity, while the negative examples are the labeled entities from other examples in the same mini-batch. We utilize the InfoNCE loss as the training objective, aiming to bring the sentence rep- resentations of positive samples closer together and push away those of negative samples. This pre- training procedure is performed on the MWOZ and CamRest datasets. Since the knowledge base in the SMD dataset is specific to each dialogue and lacks global knowledge base, do not conduct pre- training on the SMD dataset. The hyperparameters for the pre-training are detailed Table 8.
culty of knowledge utilization in response genera- tion. Nevertheless, our model, empowered by the dual-feedback mechanism, still enables effective retriever training even in such circumstances.
F Train the Retriever Using ChatGPT
Regarding the zero-shot and few-shot experiments, our retriever underwent training on the complete MuldWWZ dataset, with total of 2877 dialogues. Furthermore, in the few-shot context, we create prompts to simulate three main knowledge retrieval scenarios, aiming to enhance the model's compre- hension of dialogue tasks. These scenarios involve knowledge base (KB) containing entities that align with user intent, KB containing similar entities for recommendations, and KB lacking similar entities and leading to query failures. Furthermore, we have also taken into account the potential inaccura- cies in self-reported ChatGPT scores, which could negatively impact the feedback. Our experimental results indicate that the proposed composite scor- ing strategy, combining self-reported scores and BLEU metrics, effectively alleviates this issue.
Entity11 Comparison: FID vs. Ours
FID
50
48
46
Recall@ Comparison: FID vs. Ours
80
racaligk
60-
F.1 Hyperparameters
The hyperparameters for our retriever when using ChatGPT as the generator are presented in Table 9,
40
Parameters
MWOZ
Batch size
8
Figure 4: Performance on MWOZ as the number of retrieved entities changes.
Epoch
LR
5e-5
LR schedule
Fixed
Max input length
128
E The Number of Retrieved Entities
Pooling type
CLS
Weight decay
0.01
In our experiments, the retriever retrieves the top- K relevant entities from the knowledge base for response generation. We conduct an analysis to assess the performance of our system and FiD as we varied the number of retrieved documents K, as shown in Figure 4. Notably, we observe that FiD achieves the best performance when K- 7. while our model exhibits optimal performance when K = 10. This suggests that a small number of K might be inadequate to cover necessary knowl- edge entities for response generation. However, as K further increases, FiD's performance is signifi- cantly affected, leading noticeable decline in performance. In contrast, our system demonstrates only slight decrease in performance. This finding suggests that a large value of would inevitably introduce more noisy entities and increase the diffi-
Max output tokens
150
Response Temperature
1.0
Rank Temperature
0.1
Oracle function
BLEU
Beam size
5
Table Hyperparameter setting for retriever training when using ChatGPT the generator, as well as the configuration of ChatGPT, on the dataset-level knowl- edge base MWOZ.
F.2 Accuracy of ChatGPT Scores
As indicated in the table below, relying solely on self-reported (sr) scores for constructing feedback leads performance degradation compared to the approach proposed in the paper. This, to some extent, validates the presence of inaccuracies

---

self-reported scores. However, quantifying the pre- cision is challenging due to the absence of ground truth labels for ChatGPT's outputs. Furthermore, our approach combines self-reported scores with BLEUUbased ranking for selecting negative sam- ples, thereby mitigating the unfavorable effects that may arise from these inaccuracies.
Scoring Type
BLEU
Entity-F1
Re@7
Mew
7.14
31.46
88.03
(ew-shot(rankar)
6.03
29.40
84.07
Table 10: The accuracy of the self-reported ChatGPT scores
F.3 Prompt for ChatGPT
Since the ChatGPT API does not directly provide the generation probabilities responses, which are required our method for constructing feedback and entity scoring, we make slight adaptations. For negative sample selection, we construct prompts to elicit ChatGPT to generate responses along with their corresponding confidence scores as approx- imations of generation probabilities. We further select hypothesis responses with low confidence scores but high actual quality as negative examples. Regarding entity scoring, we employ a similar ap- proach by constructing prompts to assess the rele- vance of each (entity, response) pair. The specific prompts used in this process are shown in Table 11 and Table 12. It should be noted that the prompts used in the zero-shot and few-shot scenarios only differ in the presence of examples, and therefore the shw-shot prompts are not separately displayed.

---

You are a task-oriented dialngue chatbot. Your initial priority to understand the user's intent within the current user input, taking into account the dialogue history. Subsequently, need to select the relevant information from the knowledge base aligns with this intent. Finally, generate concise response by incorporating the current user input and selected infor- mation from the knowledge base. Additionally, you need provide confidence score for each response to indicate the level of certainty associated with it. The confidence score falls within the range of 0.0 to 1.0, denoted as decimal, The output fo- rmat responses follows the structure:
Response: [Generated response]
Confidence: [Confidence score]
Knowledge base
1. name charlie chan, address regent street city centre, area centre, domain restaurant, food chinese, phone 01223361766
postcade cb, pricerange cheup, type restaurant.
name alexander bed and breakfast, address 56 saint barnabas road, area centre, domain hotel, internet yes. parking yes, phone 01 21225255225, postende 2h1 pricerange cheap, stars star, name restaurant seven, address vere university arms regent street city centre, area centre, food british, phone 01223337766, postcode cb21ab, pricerange moderate, type restaurant.
Dialogue history
[user]: are there any restaurants that serve proper british food in town ?
[sys]: yes quite fow which part will you he dining
User input
[user]: west, if possible.
Response:
Table 11: Response prompt for ChatGPT.
You are required to assign relevance scores to each entity-response pair input. These scores should range from 0.0 1.0 and maintain the order bused on the inpul sequence and the lotal number of entities in the knowledge base. The output format should follow the pattern:
Score: Ireeevance-scorel relevance-score2,...
Knowledge base
1. name charlic chan, address regent street city centre, area centre. domain restaurant, food chinese, phone 01223361763,
postcode ch21 pricerange cheap, type restaurant.
name alexander hed and breakfast, address 56 saint harnahas road, area centre, domain hotel, internet yes, parking yes, phone 01223525725, postcude cb12de, pricerange cheap, stars star, restaurant seven, uddress vere university arms regent street city centre, centre, food british. phone 01223337766, postcode ab, pricerange moderate, type restaurant.
4.
Response:
we have three: graffiti saint john s chop house, and traveller rest
Score:
Table 12: Entity scoring prompt for ChatGPT.


Reviews:
[{"review_id": "v3DukP2FkW", "reviewer": "Reviewer_ctRL", "summary": "", "questions": "A. Please provide more details regarding the  zero-shot and few-shot experiments, e.g. how many ChatGPT samples were used to train the retriever? Was any further study done to support the accuracy of the self-reported ChatGPT scores? Could useful scores be obtained by estimating the probability of the ChatGPT outputs through a different pretrained language model?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and valuable suggestions!\n\n**Q1: More details regarding the zero-shot and few-shot experiments.**\n\n**A1:** Regarding the zero-shot and few-shot experiments, our retriever underwent training on the complete MultiWOZ dataset, with a total of 2877 dialogues. Furthermore, in the few-shot context, we create prompts to simulate three main knowledge retrieval scenarios: knowledge base (KB) containing entities that align with user intent, KB containing similar entities for recommendations, and KB lacking similar entities and leading to query failures. We intend to incorporate these particulars during the revision of our paper.\n\n**Q2: The accuracy of the self-reported ChatGPT scores.**\n\n**A2:** This paper aims to introduce a retriever-generator architecture that combines a retriever for retrieving relevant knowledge and a generator for producing system responses. Given the absence of retriever training labels, our primary focus is on leveraging feedback from the generator as pseudo-labels to train the retriever. The experimentation with LLMs aims to showcase the compatibility of our proposed dual-feedback mechanism with these models. \n\nAs indicated in the table below, relying solely on self-reported (sr) scores for constructing feedback leads to performance degradation compared to the approach proposed in the paper. This, to some extent, validates the presence of inaccuracies in self-reported scores. However, quantifying the precision is challenging due to the absence of ground truth labels for ChatGPT's outputs. Furthermore, our approach combines self-reported scores with BLEU-based ranking for selecting negative samples, thereby mitigating the unfavorable effects that may arise from these inaccuracies. \n\n| Few-Shot                | BLEU | Entity-F1 | Re@7   |\n|-----------------------|------|-----------|--------|\n|$rank_{sr-bleu}$| 7.14 | 31.46     | 88.03  |\n| $rank_{sr}$      | 6.03 | 29.40     | 84.07  |\n\n\n**Q3: Suggested modifications.**\n\n**A3:** Thank you for the feedback. We will certainly take your suggestions into consideration during the paper revision process.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach shows good performance, outperforming previous work on three dataset across two settings.\n- The provided analysis and comparisons further support the proposed method.\n- The paper is for the most part easy to follow and clear to understand.\n", "reasons_to_reject": "- The zero-shot and few-shot experiments are somewhat underexplained.\n", "questions_to_authors": "A. Please provide more details regarding the  zero-shot and few-shot experiments, e.g. how many ChatGPT samples were used to train the retriever? Was any further study done to support the accuracy of the self-reported ChatGPT scores? Could useful scores be obtained by estimating the probability of the ChatGPT outputs through a different pretrained language model?\n", "paper_topic_and_main_contributions": "The paper explores how to train the retriever in a retrieve-and-generate approach for knowledge-grounded dialogue systems, assuming no access to explicitly annotated data. Specifically, the proposed approach first assigns to the retrieved entities a score calculated as the normalized conditional (on context and entities) log probability of the generated sequence according to the model (i.e. a pretrained LM). The retriever is then fitted to these scores via KL divergence. Furthermore, the scores of entities that result in generator samples with low BLEU score but high generator probability, is treated as negative feedback for the retriever.\n\nThe paper presents experimental results on MultiWoz, Stanford Multi-Domain Dataset, and CamRest, on two settings employing a session-level and dataset-level knowledge base correspondingly. The approach is shown to outperform recent work on BLEU and Entity-F1, and is accompanied by an ablation study,  a comparison to other training methods for the retriever, and an analysis on how to best select the negative samples.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In Figure 1, perhaps include the price value as its unclear alone why Chesterton should be preferred.\n- In Tables 1 and 2, please  indicate the size of the underlying models (in number of parameters) in each approach, to help the reader make a fair comparison between the methods. Specifically, in Table 2, one would assume that Q-TOD is using the base model, but this should be indicated.\n"}}, {"review_id": "ip8MedESYM", "reviewer": "Reviewer_iQGR", "summary": "", "questions": "1. How long does the training process take?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and insightful questions.\n\n**Q1: Detailed description of the training process.**\n\n**A1:** We provide a more comprehensive description of the training process below. During the initial training phases, we incorporate a warm-up period for the generator, facilitating dependable feedback for the retriever. Subsequent to this phase, both the retriever and generator are subjected to joint training until reaching convergence. It's important to emphasize that we carry out negative sampling at each training step, ensuring consistent information updates from the generator to the retriever. We intend to include this detailed training process description in the revised version of the paper.\n\n**Q2: Regarding the training efficiency.**\n\n**A2:** We acknowledge that the inclusion of sampling operations during training does indeed extend the overall training duration. Nonetheless, it's important to note that these sampling operations are exclusively carried out within the training phase and therefore do not impact the inference time. Moreover, through our evaluation on the validation set during training, we observe that our dual-feedback approach consistently outperforms the baseline in terms of results achieved within the same number of training steps. This indicates that our approach can attain favorable results earlier in the training process.\n\nThanks again for your review! Please let us know if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This method achieves good performance on three publicly available datasets. In addition, the authors conducted a variety of complementary experiments to demonstrate the validity of their method.\n2. Although the retriever-generator architectures have become a popular trend in the field of dialog systems, most of the current work still combines these two modules in a pipelined form. This work, on the other hand, designs a method to pass positive and negative feedback from the generator to the retriever, which is innovative and can be generalized to other approaches that employ retriever-generator architectures.\n3. The source code is provided.", "reasons_to_reject": "1. Generating negative samples during training may affect efficiency.\n2. There is a lack of detailed description of the training process, e.g., when end-to-end training starts and how often negative samples are sampled.", "questions_to_authors": "1. How long does the training process take?", "paper_topic_and_main_contributions": "This paper proposes a new retriever-generator training architecture for task-oriented dialog systems. This architecture is able to train the retriever using positive and negative feedback signals from the generator. The authors conducted experiments on three publicly available datasets and obtained good performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xJrHtodOAl", "reviewer": "Reviewer_gvAr", "summary": "", "questions": "1.\tDuring the training process, are the retriever and generator jointly optimized with the final loss function?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reviews and insightful suggestions.\n\n**Q1: Explanation of the effectiveness of the negative loss function.**\n\n**A1:** We appreciate your observation regarding the oversight in formulating the negative loss function. In particular, we inadvertently omitted the negative sign after subtracting the two $D_{KL}(\\cdot)$ terms. The correct negative loss function should be:\n\\begin{equation}\nL_{\\text{neg}} = \\max\\left(0, -(D_{\\text{KL}}(G_{t}^{-}, S_{t}) - D_{\\text{KL}}(G_{t}, S_{t})) + \\eta\\right).\n\\end{equation}\n\nIn the actual implementation (Source code: `Dual-Feedback-TOD/model/FiD_ToD_FullKB.py`: line 645), this function is instantiated using `torch.nn.MarginRankingLoss()` *(y=1)*. Thus, this minor error has no bearing on the experimental results. We will rectify this oversight when revising the paper.\n\n**Q2: Choice of metrics for evaluating natural language generation (NLG).**\n\n**A2:** We concur that BLEU is not an ideal metric for assessing tasks involving NLG, and semantic-oriented metrics like BERTSCORE could be considered as an alternative measure. However, we opted for BLEU due to its computational efficiency, whereas semantic-focused metrics such as BERTSCORE might require additional model training. This introduces a notable challenge in acquiring task-specific pairwise training data for dialogues. In our future research, we will delve into exploring semantic-based metrics such as BERTSCORE.\n\n**Q3: The compatibility of our proposed dual-feedback mechanism with LLMs.**\n\n**A3:** This paper aims to introduce a retriever-generator architecture that combines a retriever for retrieving relevant knowledge and a generator for producing system responses. Given the absence of retriever training labels, our primary focus is on leveraging feedback from the generator as pseudo-labels to train the retriever. The experimentation with LLMs aims to showcase the compatibility of our proposed dual-feedback mechanism with these models. We opted for the closed-source ChatGPT over other open-source language models due to its relatively lower demand for computing resources. However, in the revised version of the paper, we will expand our experimentation to include open-source LLMs as well.\n\n**Q4: Detailed description of the training process.**\n\n**A4:** We provide a more comprehensive description of the training process below. During the initial training phases, we incorporate a warm-up period for the generator, facilitating dependable feedback for the retriever. Subsequent to this phase, both the retriever and generator are subjected to joint training until reaching convergence. It's important to emphasize that we carry out negative sampling at each training step, ensuring consistent information updates from the generator to the retriever. We intend to include this detailed training process description in the revised version of the paper.\n\nThanks again for your helpful feedback and suggestions. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper utilized generator to produce positive labels as pseudo labels of retrieved entities for optimization, which addressed the lack of retriever training labels.\n2.\tThis paper further proposed negative feedback prevent the retriever from being misled by inaccurate information.\n3.\tThis paper conducted extensive experiments and analysis which demonstrated the effectiveness of proposed methods\n", "reasons_to_reject": "1.\tThe paper mentioned that the introduction of negative feedback is to solve the problem of positive feedback assimilating incorrect knowledge from the entities, but the negative loss function in Section 3.2 might not meet this purpose well. It would be better to explain it more fully.\n2.\tIt was insufficient that evaluating natural language generation (NLG) systems with only word-overlap based metrics such as BLEU. It would be better to use semantic-based metrics such as BERTSCORE.\n3.\tIn Section 5.4, paper prompted ChatGPT to obtain entity scores. it was different from the proposed method which utilizing conditional log probability as entity scores. It was not enough to verify the compatibility of proposed positive feedback on LLMs.\n", "questions_to_authors": "1.\tDuring the training process, are the retriever and generator jointly optimized with the final loss function?", "paper_topic_and_main_contributions": "This paper proposed a Task-Oriented Dialogue System with retriever-generator architecture. The retriever to retrieve pertinent knowledge and a generator to generate system responses. Due to the lack of retriever training labels, for the optimization of retriever, this paper introduced a dual-feedback mechanism where the generator produced positive and negative feedback as pseudo labels. The experiments on three public benchmarks demonstrated the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: An interesting coupling of knowledge retriever and dialog generator to improve the quality of the retrieved knowledge to be incorporated in the generated dialog response. The paper is sound and shows clear evidence of the efficiency of the proposed method.
Decision: Accept-Main