Retrieval-Generation Alignment for End-to-End Task-Oriented Dialogue System
Authors: Weizhou Shen
Yingqi Gao
Canbin Huang
Fanqi Wan
Xiaojun Quan
Wei Bi
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge Retrieval
End-to-End Task-Oriented Dialogue System

Abstract: Developing an efficient retriever to retrieve knowledge from a large-scale knowledge base (KB) is critical for task-oriented dialogue systems to effectively handle localized and specialized tasks. However, widely used generative models such as T5 and ChatGPT often struggle to differentiate subtle differences among the retrieved KB records when generating responses, resulting in suboptimal quality of generated responses. In this paper, we propose the application of maximal marginal likelihood to train a perceptive retriever by utilizing signals from response generation for supervision. In addition, our approach goes beyond considering solely retrieved entities and incorporates various meta knowledge to guide the generator, thus improving the utilization of knowledge. We evaluate our approach on three task-oriented dialogue datasets using T5 and ChatGPT as the backbone models. The results demonstrate that when combined with meta knowledge, the response generator can effectively leverage high-quality knowledge records from the retriever and enhance the quality of generated responses. The code of this work is available at https://github.com/shenwzh3/MK-TOD.

Retrieval-Generation Alignment for End-to-End Task-Oriented Dialogue System
Weizhou Shen', Yingqi Gao', Canbin Huang', Fanqi Wan', Xiaojun Quan'; Wei Bi² 'School of Computer Science and Engineering, Sun Yat-sen University, China
Tencent AI Lab
(shenwzh3, gaoyq28, huangcb3, wanfq) @ mail2.sysu.ccd.com
quanxj3 @ mail. wyl.llsssuuuuu... en, victoriabi @ tencent..com
Abstract
Can you help find a called Guesthouse?
Developing an efficient retriever to retrieve knowledge from large-scale knowledge base (KB) is critical for task-oriented dialogue sys- tems to effectively handle localized and special- ized tasks. However, widely used generative models such as T5 and ChatGPT often struggle differentiate subtle differences among the re- trieved KB records when generating responses, resulting in suboptimal quality of generated responses. In this paper, we propose the appli- cation of maximal marginal likelihood to train perceptive retriever by utilizing signals from re- sponse generation for supervision. In addition, our approach goes beyond considering solely retrieved entities and incorporates various meta knowledge to guide the generator, thus improv- ing the utilization of knowledge. We evalu- ate our approach on three task-oriented dia- logue datasets using T5 and ChatGPT as the backbone models. The results demonstrate that when combined with meta knowledge, the response generator can effectively lever- age high-quality knowledge records from the retriever and enhance the quality of generated responses. The code of this work is available httpp:://gtthhi thub. com/ http://hwww/////T000
It the enst, and maderately priced. Would you like
to book room?
I get the address and phone rumber, ploase?
Name
Area
Price
Addres
Phone
ani Gueslhouse
eas.
moderale
Terison Road
3315702
2
Carolina Bel
usst
modenste 138 Pame Road
3247015
Amttway House
north madento 52 Gilbert Rond
3575314
Watkort's Houze
cxst modente Wark Temace
3363802
Artury I odge
north madentt 82 Arhrry Road
3384319
The address is Work Terrace and the phone number 3363682.
X
Figure 1: A demonstrative case in E2E-TOD. The ta- ble displays retrieved entities sorted by retrieval order. The correct entity is highlighted in blue. However, the response generator mistakenly selects the false entity, highlighted in red, leading erroneous response.
to be more compatible with practical scenarios and large-scale language models, it imposes challenges in acquiring and utilizing external knowledge no belief state is provided for knowledge retrieval.
Retrieval-augmenned generation (Lewis et al., 2020; Ren et al., 2021; Singh et al., 2021) has demonstrated success in various knowledge- intensive tasks by employing held-out dense re- triever to retrieve knowledge and then taking the knowledge to generate results. Q-TOD (Tian et al.. 2022) applies this approach to E2E-TOD and signif- icantly outperforms previous methods that combine knowledge retrieval and response generation into a single model (Madotto et al., 2018; Qin et al., 2020; Raghu et al., 2021). However, our preliminary study (Section 5.3) shows that under this frame- work the correlation between the performance of knowledge retriever and that of response generator is relatively weak, meaning that simply improving the retriever may not lead to a better generator. We characterize this phenomenon as the misalignment between the retrieval and generation processes in E2E-TOD systems. This misalignment poses a bot- tleneck for current dialogue systems, as improve-
1 Introduction
Task-oriented dialogue systems (TOD) assist users accomplish daily tasks such as restaurants, scheduling appointments, and navigating traffic by leveraging external knowledge bases. Among them, pipeline systems (Henderson et al., 2014; Hosseini- Asl et al., 2020) involve several intermediate stages such as dialog state tracking and system policy learning for retrieving knowledge and generating responses. In contrast, end-to-end task-oriented dialog systems (E2E-TOD) (Wu et al., 2022; Tian et al., 2022) have gained increasing concentration for their ability to directly generate responses based on the knowledge hase without intermediate anno- tations. Although the end-to-end paradigm appears
"Corresponding authors.

---

ments in the retriever component do not necessarily translate enhanced generation quality.
empowers the generator to better differentiate be- tween similar entities during response generation.
Through qualitative analysis, we hypothesize that the misalignment between retrieval and gener- ation is attributed to the homogeneity of retrieved knowledge entities. As illustrated in Figure 1, the retrieved entities exhibit a high degree of similarity, with only minor variations in their values. Con- sequently, since the response generator is trained on reference responses that predominantly consist of language tokens rather than knowledge-related tokens, it struggles to differentiate between similar entities and may inadvertently select inappropriate entities for response generation.
2 Related Works
2.1 End-to-End Task-Oriented Dialogue
The existing work on the usage of external knowl- edge in end-to-end task-oriented dialogue systems can be divided into three categories. The first category takes the whole knowledge base as the model input, and conducts knowledge selection and response generation in one single model. For instance, Mem2seq (Madotto et al., 2018), KB- Retriever (Qin et al., 2019), GLMP (Wu et al., 2019) and CDNET (Raghu et al., 2021) employ memory networks for querying knowledge. Uni- fiedSKG (Xie et al., 2022) directly concatenates entities as the input of Transformers. The second category directly encodes knowledge into model parameters. GPT-KE (Madotto et al., 2020) pre- trains their model on augmented dialog data to em- bed the knowledge base, while ECO (Huang et al.. 2022) applies tri-constraints on top of GPT-KE to ensure entity consistency. The third category is to use individual retriever retrieve knowledge. For example, Q-TOD (Tian al., 2022) decouples the dialogue system into retriever and genera- tor and uses the generator to generate a query for knowledge retrieval. DialogKG (Rony et al., 2022) inputs the flattened records to graph neural net- work to select entities. MAKER (Wan et al., 2023) introduces multi-grained retriever with both en- tity and attribute selection. As mentioned earlier, although the retrieve-then-gennt generrtee framework has been successful paradigm to date, it leads mis- alignment between the retriever and the generator in end-to-end tusk-oriented dialogue systems.
In this paper, we introduce Meta Knowledge for end-to-end Misk-Oriented Dialogue system (MK-TOD) as a solution to address the retrieval- generation misalignment. MK-TOD aims to cor- relate the performance of the knowledge retriever and response generator for improved system per- formance. To enhance the knowledge retriever, we propose the application of maximum marginal like- lihood (Singh et al., 2021) for progressive retriever updating during the training of the response gen- erator. order enable the response generator to distinguish between entities, we explore several methods for utilizing retrieval-related meta knowl- edge. Here, meta knowledge refers to various infor- mation about the retrieved entities, such as retrieval order, retrieval confidence, and co-occurrence rate. We propose three approaches for incorporating the meta knowledge: adding special prefix tokens, us- ing prompts, and applying contrastive learning. Ad- ditionally, we investigate the introduction of neg- ative knowledge during the generator's 's training to enhance its discriminative ability.
We apply MK-TOD to several backbone models, including T5 (Raffel et al., 2020) and the large lan- guage model ChaIGPT (OpenAI, 2022). We com- pare MK-TOD with other E2E-TOD systems on three benchmark datasets, namely SMD, CamRest, and WoZ. (Eric et al., 2017; Wen et al., 2017; Eric et al., 2020). The empirical results demonstrate the superiority of our proposed system over the current state-of-the-art systems with similar model scales. Additionally, our system effectively enhances the performance of ChatGPT in E2E-TOD with in- context learning, Furthermore, through comprehen- sive analysis, we uncover that our meta-knowleege approach successfully alleviates the misalignment between the retriever and generator. This approach
2.2 Alrrivval-Augmennee Generation
With the success of the dual-encoder neural re- triever (Karpukhin al., 2020). the retrieval- augmented generation framework is widely applied to various knowledge-intensivy tasks. This frame- work uses a retriever to retrieve knowledge from knowledge base and inputs the retrieval results into a generator to generate the answer. Among them, RAG (Lewis et al., 2020) separately encodes each retrieved knowledge record with the query and marginalizes the probabilities of the answer based on each entity. FiD (Izacard and Grave, 2021) encodes each retrieved knowledge like RAG and fuses their hidden states in the decoder. FiD-

---

The retriever module comprises context encoder and an entity encoder. The context encoder trans- forms the current dialogue context or into a vector representation ha. the other hand, the entity encoder concatenates the attribute-value pairs each entity as plain text and encodes into a vec- tor representation he,- The matching score St,i is computed by taking the dot product between ha, and heg- Consequently, the top-K entities with the highest scores selected candidate enti- ties & for the current dialogue turn. Furthermore, if meta knowledge is utilized, each entity in & is augmented with its corresponding meta knowledge. Ranking onder: rop- recalled, sopel recalled,
The Teussa plane 3315702. Anything
regallive ensiy
Coocermare Newly mealled Fnsiese history
Confideace: hightndddo coniilanss
Response
Knowledge
POS 4)
Mots (nou)tdpr((t.)))
Annd
-werr Can
Albuyy Lodge
Mora knowleewwe
Caroliza
suser- Can you
Response Generator
Retriever
A B Guesthouse arce: cad prorr modente....
Caodlaa
Mara (eooleeeeed)))
Arbur...
you...
Retrieved Eniiiies (+g]
Marginal Likelihoed
<perrh Can help Can the
sodrres and phone number, Dialogue Context
The MK-TOD framework comprises knowledge retriever and response generator. Given the dialogue context, the retriever retrieves entities from the knowledge base. Each entity is concatenated with its corresponding meta knowledge and subsequently input into the generator to generate the response. The optimization process involves two likelihoods: the normal text generation likelihood and the marginal likelihood.
Figure 2:
KD (Izacard and Grave, 2022) and EMDR² (Singh et al., 2021) are both based on the FiD framework but with different retriever training methods: FID- KD uses knowledge distillation while EMDR uses marginal likelihood. REPLUG (Shi et al., 2023) ap- plies the method of RAG to large language models but only updates the retriever during training.
3.2 System Overview
3 Methodology
The framework of our proposed MK-TOD is de- picted in Figure 2. It consists of retriever and response generator. In each dialogue turn, the re- triever retrieves a set of relevant entities, which then combined with retrieval-related meta knowl- edge and the dialogue context. The generator uti- lizes this information to generate a response for the current turn. In the following section, we first introduce the notations and provide an overview of our method. Then, we delve into the detailed explanations of two crucial components: maximum marginal likelihood and meta knowledge.
The generator takes the retrieved entities & and dialogue context C as inputs to generate the final system response ry. The probability of generating the response rt given the entities & and dialogue context ct be caleulated follows:
Irel
p(relc,, En;) = II cn. (1)
3.1 Notations
where 0 denotes the parameters the generator. Similar to most text generation tasks, we incor- porate the negative log-likelihood (NLL) loss as a training objective to train the generator:
Given a dialogue D {u,, (11,71,". (T) T turns, where u, and r are the 1-th turn user utter- ance and system response, respectively. We use ot to represent the dialog context of the 1-th turn. where = {u1, Fi,..., .... #4-1, Tt-1, 3 An exter- nal knowledge base (KB) is provided in the form of set of entities, i.c., K.: {EI, Et.,,..... EB) where each entity 6 consists of several attribute-value pairs and B the size of knowledge base. End- to-end taik-trie dialog systems take dialogue context eN and knowledge base as input and gen- erate an informative natural language response rt.
LNLL Elog
(2)
3.3 Maximum Marginal Likelihood
Due to the absence of retrieval labels for training the retriever, depend supervision signals from the generator. However, since it is pos- sible to backpropagate gradients through the NLL loss in Eq. (2) to the retriever, we propose updat- ing the retriever's parameters by maximizing the

---

marginal likelihood (MML) of the response rt. The marginal likelihood offers Bayesian perspective to compute o(relet,A) (C) by integrating the likelihood over all the entities in the knowledge base:
Retrieval confidence: To provide more retrieval information, we categorize retrieved entities into low-confidence, middle-confidence, and high- confidence based on retrieval scores. The thresh- olds for categorizing entities are hyper-parameters! Retrieval confidence, in conjunction with retrieval order, enables the generator to disregard entities with low confidence but high retrieval order.
p(rlon0
Σ eEE
(3)
where @ denotes the parameters of the retriever and g(e/)e4;)) Φ) is the retrieval probability of entity 65- Note that computing q(eale;; σ) for all entities in the entire knowledge base would incur an unaf- fordable computational cost for Eq. (3). Therefore, following the approach of EMDR2 (Singh et al., 2021), we compute (ccale Φ) over the retrieved entities & instead of the entire knowledge base K::
Co-occurrence: Entities that have already ap- peared in the dialogue context are more likely to be relevant for future responses. Thus, we inform the generator about the occurrence of entities in the dialogue context through meta knowledge.
To implement the above meta knowledge our system, we design three approaches: prefix, prompt, and contrastive learning.
p(rlees000)
Σ eL.EEE
3.4.1 Prefix
(4) In this approach, we create a mapping function that assigns special tokens representing meta knowl- edge to each entity. For instance, an entity ranked second in retrieval order, with middle retrieval (5) confidence, and not yet mentioned in the context would be mapped to the set of <2nd-entity>, <mid-conff idence>, <new-entitys?, These pre- fix tokens then concatenated with the corre- sponding entity and input the generator during both training and inference stages.
where ((t,,143)) Φ) is implemented as follows:
exp($r,)
g( Ee exp($t.)
The loss function for the marginal likelihood defined as follows:
LMML llog Σ o,
()p(riee et,;; 0).
(6)
3.4.2 Prompt
By incorporating q(et.j cz; Φ), propagate gradients back to the retriever and update its pa- rameters. The ultimate training loss function for MK-TOD is defined as follows:
To fully leverage the generator's 's language model- ing capability, we explore using prompts to incorpo- rate meta knowledge. Here, we design mapping function that assigns each entity set of prompts, which are natural language sentences representing the meta knowledge. For example, a prompt can be "This is the top-1 recalled entity with low con- fidence Similar to the prefix approach, these prompts are concatenated with the corresponding entities and fed into the generator.
L - = BLMML.
where a and 3 are hyperparameters.
3.4 Meta Knowledge
We introduce the concept of rellaee meta knowledge, which encompasses various informa- tion about the retrieved entities to guide the gener- ator and enhance the alignment between retrieval and generation. Three key factors are considered in the meta knowledge: retrieval order, retrieval confidence, and co-occurrence.
3.4.3 Contrastive Learning
We can also train the generator to distinguish be- tween entities by employing contrastive learning. In this approach, we select subset of entities from the retrieved entities & based on their retrieval or-
Retrieval order: The retriever evaluates entities based on their matching with the dialogue context, prioritizing those that exhibit a higher degree of matching. We incorporate the retrieval order of each entity as a part of the meta knowledge.
WWe established the following ranges: (-infinity, 0.4) indi- cates low confidence, (0.4, 0.75] indicates medium confidence, and (0.75, +infinity) indicates high confidence. "The complete mapping is discussed in Appendix B.1 "The complete mapping is discussed in Appendix B.2

---

der, forming a positive entity set e For each entity e, in E we compute its length-normalized log-likelihood of generating the response:
It's important to highlight that, while we inten- tionally provide the generalor with meta knowl- edge, such as retrieval order and confidence scores, this does not compel the generator to exclusively rely on the highest-ranked entities. Rather, our ap- proach encourages the generators to autonomously distinguish between these entities.
dr. logr(riles,
(8)
Inal
where Irel is the length of r, Additionally, we cal- culate the log-likelihodd of generating the response without any entity as the baseline likelihood:
4
Experimental Settings
4.1 Datasets
logp(relen Iral
We evaluate our MK-TOD on three task-oriented dialogue datasets: MultiWOZ 2.1 (MWOZ) (Eric et al., 2020), CamRest (Wen et al., 2017), and Stan- ford
We employ a pairwise marginal ranking loss that ensures the likelihood of positive entities greater than the baseline likelihood by certain margin:
(SMD) (Eric et al., 2017). We compare the methods with two different settings about the knowledge bases: First, each dialogue has a small session-level knowledge base associ- ated with the user goal, which is the typical setting of previous work. Second, all conversations MWOZ/CamRest share dataset-level large-scale knowledge base by injecting all the session-leve knowledge bases. There are 223 and 1 112 entities in the large-scale knowledge base for MWOZ and CamRest, respectively. Other detailed statistics of these datasets are shown in Appendix A.
LCTR Σ m0, d, de. + A), (10) 60,
where À is hyperparameter. We then add this loss term to the loss function of MK-TOD:
C = OoNEL + 3LMML + LLTTR-
(11)
3.5 Negatíve Entity
Inspired by negative sampling in information re- trieval (Karpukhin et al., 2020), we also consider incorporating negative entities into the generator. The negative entity, denoted as e, &s, is chosen the entity with the lowest retrieval score from K. Special meta knowledge is designed for the nega- tive entity as well. Note that the negative entity is different from the baseline likelihood the above contrastive learning (Section 3.4.3).
For all three datasets, we employ BLEU (Pap- ineni et al., 2002) and Entity Fl (Eric et al., 2017) as the metrics to evaluate the quality of generated responses. Entity Fl assesses the presence of ac- curate knowledge in the responses by calculating the micro-averaged precision and recall scores of attribute values. Additionally, for experiments ducted on large-scale knowledge bases, we intro- duce Recall @K as the performance metrie for the retriever. Recall measures the percentage of gold entities appearing in the retrieved entities.
3.6 Model Inference
During inference, we first retrieve entities using the retriever. Then, prepend each entity with its corresponding meta knowledge. Finally, concatenate the entities with the dialogue context and input the resulting sequence to the generator to generate the final response. Notably, we do not include negative entities during inference.
4.2 Implementation Details
We utilize BERT (Devlin et al., 2019) as the con- text encoder and entity encoder for the retriever. As for the generator, we employ T5 (Raffel et al., 2020) and ChatGPT (OpenAI, 2022). Note that ChatGPT is not fine-tuned but instead undergoes in-context learning using our datasets." 6 The triever for ChatGPT is directly copied from the retriever trained with T5 using MML. All experi- ments are performed on single 24G NVIDIA RTX 3090 GPU, and the best checkpoints are selected based on Entity Fl scores on the validation set. Hyperparameter settings are listed in Appendix E.
3.7 Discussion
The motivation of meta knowledge is to empower the generator by enabling it utilize retrieved in- formation for improving the generation process.
'Retrieval confidence and co-occurrence are not consid- ered avoid sparsity in the positive entity set. Details are provided in Appendix 1.1 and B.2
Further details are provided in Appendix C

---

MWOZ
CamRest
Model
BLEU
Entity F1
Recall @07
BLEU
Entity F1
@
DF-Net (Qin et al., 2020)
6,45
27.31
EER (He al., 2020b)
11.60
31.86
20.61
57.59
FG2Seq (He et al., 2020a)
10.74
33.68
19.20
59.35
CDNET (Raghu al., 2021)
10.90
31.40
16.50
63.60
Q-TOD (TS-Large) (Tian al., 2022)
15.52
46.74
92.97
21.44
63.88
95.52
MAKER (T5-Base) (Wan et al., 2023)
16.25
50.87
26.19
72.09
MAKER (T5-Large) (Wan et al., 2023)
18.23
52.12
25.34
72.43
Poeff (Base)
16.39
50.35
92.51
25.23
71.15
94.35
OurSprompl (Base)
17,56
50.69
92.74
26.69
71.67
92.24
OurSetr (Base)
15.96
51.35
92.74
26.85
73.51
93.88
prefiz (Large)
16.69
53,59
92.93
27.32
72.77
91.08
prompe (larg))
17.15
52.99
94.42
26.88
72.92
95.41
Our (Large)
17.40
53.26
95.22
27.82
71.98
95.38
ChatGPT (OpenAl, 2022)
6.79
30.31
92.74
14.76
52.92
94.35"
OurSprefsz (L.LM)
7.01
30.69
92,74"
14.51
$2.38
94.35*
OurSprompe (LLM)
7.31
32.04
92.74
14.91
53.58
94.35*
Table I:
Overall results of E2E-TOD systems with large-scale knowledge bases on MWOZ. and CamRest, where means that we directly use the retriever co-trained with T5- Base using MML.
Consistent with previous studies (Qin et al., 2019), we initialize the retriever through pre- training with distant supervision to prevent col- lapsed representations. Additional details on the pre-training process can be found in Appendix D.
strate the phenomenon of retrieval-generaticn mis- alignment and conduct ablation studies.
5.1 Overall Results with Large-Scale KBs
Comparing method with others in the setting of retrieving knowledge from large-scale knowl- edge base aligns better with real-world TOD sce- narios. Therefore, we begin by comparing our pro- posed MK-TOD approach with baselines in the context of large-scale knowledge bases. The re- sults of this comparison are displayed in Table 1. The upper section of Table shows the results of methods employing fine-tuned response gener- ator. "OurSprefta". "OurSyromyd", and "Oursær" denote our method implementing meta knowledge using prefix, prompt, and contrastive learning tech- niques, respectively. "Base" and "Large" following the method names indicate the use T5- Base or TS-Large as the response generator.
4.3 Baseline Methods
We include several strong basclines for comparison. Implicit retrieval: These methods combine knowl- edge retrieval and response generation in a sin- gle model, including DI-NNet (Qin et al., 2020), EER (He et al., 2020b), FG2Seq (He al., 2020a), CDNET (Raghu et al., 2021) and GPT- KE (Madotto et al., 2020).
Explicit retrieval: These approaches decouple the TOD system into a knowledge retriever and a response generator, including Q-TOD (Tian et al., 2022), DialoKG (Rony et al., 2022) and MAKER (Wan et al.. 2023).
Overall, MK-TOD outperforms all previous methods with the same scale of generator model, in- dicating the effect of our proposed meta- meta-knowledi knowledge. Further more, Q-TOD's retriever can achieve com- parable and even higher performance than ours due to their utilization of an additional query generator. However, even when employing T5- Base and rel- atively weaker retriever, our method still surpasses Q-TOD in terms of BLEU and Entity Fl by sig- nificant margin. This indicates that our proposed method effectively utilizes the retrieved knowledge better than Q-TOD. is also noteworthy that the Entity Fl score of T5-Large on CamRest poorer than that of 25-Base, we attribute this to the small
Large language models: Large language models (LLMs), such as ChatGPT (OpenAI, 2022), have demonstrated remarkable capabilities in engaging dialogues with humans. We establish a baseline LLM utilizing ChatGPT as the response generator by leveraging the 3pt-3. 5.5tbbbb API. To enhance its performance in TOD tasks, we integrate our knowledge retriever with the system.
5 Results and Analysis
this section, we present the overall results ob- tained using both lcrge-scale knowledge bases and condensed knowledge bases. Besides, we demon-

---

MWOZ
CamRest
SMD
Model
BLEU
Entity F1
BLEU
Entity F1
BLEU
Entity F1
DF-Net (Qin et al., 2020)
9.40
35.10
14.40
62.70
GPT-2+KE (Madotto et al., 2020)
15.05
39.58
18.00
54.85
17.35
59.78
EER (Hee et al., 20206)
13.60
35.60
19.20
65.70
17.20
$9.00
FG2Seq (He ct al., 2020aa
14.60
36.50
20.20
66.40
16.80
61.10
CDNET (Raghu et al., 2021)
11.90
38.70
17.80
62.90
21.80
68.60
DialoKG (Rony et al., 2022)
12.60
43.50
23.40
75.60
20.00
65.90
Q-TOD (T5-Base) (Tian ct al., 2022)
20.14
68.22
T-T (TS-Large) (Tian al., 2022)
17.62
50.61
23.75
74.22
21.33
71.11
purfe (Base)
16.97
51.99
26.39
72.43
23.96
68.60
Our OprSprompe (Basc)
17.05
52.42
25.00
72.09
23.54
68.28
Ou (Base)
17.33
51.86
26.76
73.60
24.77
67.86
Ou (Large)
18.02
53.13
25.68
71.98
24.97
72.87
OurSprompe (Large)
16.66
52.96
26.40
72.80
25.21
73.04
Ours (Large)
17.55
$2.97
26.20
71.72
25,43
73.31
CharGPT (OpenA1, 2022)
7.47
32.87
15.29
54.71
14.60
58.11
Our (ChatGPT)
7.22
32.78
15.56
54.96
15.07
58.41
OurSprompe (ChatGPT)
7.58
35.84
16.07
56.83
15.24
59.72
Table Overall results of E2E--TOD systems with condensed knowledge bases on MWO7, SMD, CamRest. The best results highlighted in bold, and the second-bes results underlined.
size of CamRest's training data, which includes merely 406 dialogues, leading to overfitting partic- ularly for larger models.
For the evaluation with ChatGPT on the con- densed knowledge base, performance gain of ChatGPT when enhanced with our proposed meta-knowledge. Besides, the per- formance gain is more significant than that of the large-scale knowledge bases, suggesting that Chat- GPT has higher demand for retrieval quality.
still observe the
The bottom section of Table 2 presents the re- sults of methods employing ChatGPT. Since Chat- GPT is not fine-tunable, trastive learning for meta knowledge. According to the results, we note that relying solely on in-context learning does not enable ChatGPT to perform as well as the fine-tunable methods in the context of E2E-TOD. However, our proposed approach out- performs vanilla ChatGPT. Additionally, our pro- posed prefix method for implementing meta knowl- edge yields only marginal improvement performs worse than ChatGPT. This is attributed to ChatGPT's limited ability learn the special prefix tokens representing meta knowledge from a limited number of in in-context demonstrations and concise explanatory text. contrast, our proposed prompt method significantly enhances its performance.
do not apply
5.3 Retrieval-Generatton Misalignment
To investigate the influence of retrieval perfor- mance on the E2E-TOD generator, we select six retrievers on MWOZ with a large-scale knowledge base. The details of the retrievers can be found in Appendix G. We then use different generators to generate responses based the retrieval sults. As for the generators, choose Q-TOD, FiD (Izacard and Grave, 2021), and ChatGPT. The Entity Fl scores of these generators are depicted Figures 3(a) and 3(b) as the retrieval performance varies with different retrievers.
The solid lines in the figures show that the per- formance of generators does not consistently align with that of retrieval performance. Furthermore, the performance of Q-TOD and FiD with oracle en- tities is even worse than those with a weak retriever, We refer to this phenomenon misalignment. In contrast, the dashed lines, which depict the results of the generators with pro- posed meta knowledge, exhibit greater consistency between the retrieval performance and the gener- ators. This indicates that our proposed method mitigates the misalignment issue. The correlation
5.2 Overall Results with Condensed KBs
To make a comprehensive comparison with the previous methods, we also conduct evaluations on the condensed knowledge base. The results in Ta- ble 2 indicate that our proposed method surpasses the baselines on MWOZ and SMD with the same model scale, validating the efficacy of our approach. However, among the three meta knowledge imple- mentations, challenging to determine clear preference as the system with these implementa- tions performs differently on different datasets.

---

Condensed
Large-Scale Entity F1
Method
Type
Litt con
BLEU
Entity F1
BLEU
Recall
16.97
16.39
50.35
92.51
50
order
16.97
51.64
16.20
49.88
92.27
Oursgentti
cunf
16.15
51.70
13.96
47:35
89.11
conceur
16.70
51.14
15.61
49.66
91 91.39
all
17.05
52.42
17.56
50.60
92.74
order
16.15
49.88
15.60
49.47
91.15
11
67-5
63.55
100
11.5
95.5
100
Chrrgronue
cont
17.02
51.06
16.84
50.16
91.93
conccur
50.38
(a) 15-Based model
(b) ChatGPT
Table 4: Results of different types of meta knowledge on MWOZ with condensed and large- scale knowledge bases. "order", "conf". "cooccur". and "all" mean using only retrieval order, retrieval confidence, co-occurrence, and all types of meta knowledge, respectively.
Figure 3: Entity Fl scores of generators ((a) FiD&Q- TOD and (b) ChatGPT) as the retrieval performance varies with different retrievers. Bracketed numbers fol- lowing model names refer to the correlation coefficients between retrieval performance and Entity FI.
Condensed
Large-Scale Entity
Type
Large-Scale Entity Fl
BLEU
Entity F1
BLEU
Recall
Model
BLEU
Recall
Ourspeefa (Basc)
16.97
$1.99
16.39
92.51
weo neg
16.68
50:45
15.
49.46
90.24
16.39
50.35
92.51
OutSueonee (Base)
52.42
17.56
w/o MML
16.07
49.56
91.39
w/o neg
16.98
$1.35
15.99
49.85
91.15
17.56
50.69
92.74
Ours (Base)
17.33
51.86
92.74
w/o MML.
16.67
50.41
91.39
wño
17.11
50.34
15.79
48.32
92.29
Ourset
15.96
51.35
92.74
Ourspee fir (L.MM)
7.32
32.38
6,83
30.47
92.74
w/o MML.
14.78
50.54
91.39
Ourggrang (LLM)
neg
7.22
7.29
32.98
35.98
7.01
6.97
30.69
31,88
92.74
92.74
w/o neg
7.58
36.18
92.74
Table 3: Ablation study of the MML loss.
Table 5: Ablation study of negative entities.
coefficients shown in parentheses next to method names further confirm this observation.
knowledge. However, when using the prompt, the results are reversed. We attribute this phenomenon to the design of the prefix and prompt. Repre- senting meta knowledge with a prefix introduces a higher diversity in ranking order since distinet prefix is assigned to each ranking order. This in- creased diversity enables the generator to better distinguish the recalled entities.
5.4 Ablation Study
We assess the impact of maximum marginal like- lihood, various types of meta knowledge, and the inclusion of negative samples. Unless otherwise specified, the ablation study is performed the MWOZ dataset using T5-Base the generator, considering the computational resource constraints.
Furthermore, the distinction between retrieval confidence and co-occurrence in the prefix setting is less obvious. In contrast, when representing the meta knowledge with a prompt, the retrieval order becomes less diverse, since only the numbers representing the retrieval order are varied.
5.4.1 Maximum Marginal Likelihood
Table 3 presents the impact of the maximum marginal likelihood (MML) loss. The methods labeled as "w/o MML" utilize the warmed-up retriever described in Section 4.2, without the joint training with the response generator. The re- sults demonstrate that the inclusion of maximum marginal likelihood enables further enhancement of the retriever during training. Consequently, the retriever leads to enhanced final responses.
5,4.3 Negative Samples
We conduct an investigation into the impact of negative entities on the performance of dialogue systems. The results presented in Table demon- strate that the inclusion of negative entities sig- nificantly improves the performance of dialogue systems when applied to T5-Base- This perfor- mance enhancement can be attributed to two main factors. Firstly, the presence of negative entities facilitates easier entity distinction for the generator, enabling it learn more effectively. Secondly, the introduction of negative entities aids in training the retriever through the MML loss in Equation (6). This concept is somewhat analogous to the moti-
5.4.2 Types of Meta Knowledge
We compare different types of meta knowledge, and the results presented in Table 4. The find- ings indicate that using single type of meta knowl- edge yields inferior performance compared com- bining all three types. Furthermore, an interesting observation emerges when using the prefix: the retrieval order outperforms other types of meta

---

incorporating retrieval-related meta knowledge. We also propose incorporate negative entities to en- hance the discriminative capability. Experimen- tal results demonstrate that when combined with meta knowledge, the response generator effec- tively leverages high-quality retrieval knowledge, leading to enhanced quality in the generated re- sponses. Through analysis, we observe that pre- vious retrieval-augmenned generator models suf- fer from severe ettrieall-enneran misalignment, while our method mitigates this misalignment.
Cun Baodie
Clrr Blastinn
10
102
1033 10p-
2id Rentera ciefideecce
Restonal
(a) Retrieval order
(b) Retrieval confidence
Figure 4: The percentage of samples utilizing the enti- ties to generate responses with respect to (a) retrieval order and (b) retrieval preference.
vation behind incorporating negative samples in knowledge retrieval tasks (Karpukhin et al., 2020).
Limitations
However, when applied to ChatGPT, negative en- tities do not contribute to model performance. The reason is that ChatGPT cannot be fine-tuned, mean- ing that solely adding negative entities the in- context demonstrations does not effectively teach ChatGPT to differentiate between entities. Con- sequently, we opt not to include negative entities when employing our method with ChatGPT.
There are three potential limitations of the paper that warrant consideration. Firstly, the employ- ment of the marginal likelihood method necessi- tates computing the likelihood for each retrieved entity, resulting in higher computational resource requirements compared to solely using negative log-likelihood (NLL). Secondly, despite conduct- ing various comparisons and ablation studies this paper, there are certain aspects missing in our proposed meta knowledge, such as investigating the combined utilization of prompt and contrastive learning, well as exploring the utilization of re- trieval order alongside co-occurrence. Lastly, the theoretical rationale behind the contribution of our proposed meta knowledge to task-oriented dialogue (TOD) is thoroughly discussed.
5.5 Behavior of Generator
We examine how the generator utilizes the retrieved knowledge with the assistance of meta knowledge on the MWOZ test set. For our model and the bascline (T5-Large), we gather all their responses that contain entities and analyze the percentage of retrieved entities that appear in the responses ac- cording to retrieval order and confidence. As illus- trated in Figure 4, our generator exhibits a higher propensity than the baseline to utilize entities with both high retrieval order and high confidence. Furthermore, we have assessed the retrieval results on the MWOZ test set. The findings demonstrate that our retriever adeptly recalls 80.69% of the gold entities with a top-1 retrieval order, which di- rectly correlates with the system- wide performance enhancement. This observation suggests that our proposed meta knowledge aids the generator in de- veloping inductive bias to prioritize entities that are highlighted by the retriever.
Acknowledgements
This work was supported by the National Natu- ral Science Foundation of China (No. 62 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A 151 151001283)), and the Ten- cent AI Lab Rhino-Bird Focused Research Pro- gram. We thank Ke Yang for his efforts in the preliminary experiments.
References
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
6 Conclusion
This paper aims to address the retrieval-generatee -eneration misalignment in end-to-end task-oriented dialogue systems by introducing maximal marginal like- lihood train perceptive retriever that lever- ages signals from response generation. To en- able the response generator to better distinguish between entities, we explore several methods for
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi. Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj

---

Goyal, Peter Ku, and Dilek Hakkani- 2020. Mul- tiWOZ 2.1: A consolidated dulti-domain dialogue dataset with state corrections and state tracking base- lines. In Proceedings of the Tweffth Language Re- sources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association.
Patrick Lewis, Ethan Perez. Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmennt generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459- 9474. Curran Associates, Inc.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task- dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Sarrrüüken,, Germany. Association for Computational Linguistics.
Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pas- cale Fung. 2020. Learning knowledge bases with parameters for task-oriented dialogue systems, In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 2372-2394.
Zhenhao He, Yuhong He, Qingyao Wu, and Jian Chen. 2020a. Fg2seq: Effectively encoding knowledge for aud-to-end task-oriented dialog. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8029-
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowl- edge bases into end-to-end crientreeted dialog sys- tems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Pupers), pages 1468-1478, Melhourne, Australia. Association for Computational I Linguistics.
8033. IEEE.
Zhenhao He, Jiachun Wang, and Jian Chen. 2020b. Taik-oriented dialog generation with enhanced en- tity representation. In INTERSPEECH, pages 3905- 3909.
OpenAl. 2022. Chatgpt: Conversational ai language model. Website. https://openai. wtt. com//hatgpt.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recur- rent neural networks. In Pruceedings of the 15th Annual Meeting of the Special Interest Group Dis- course and Dialogue (SIGDIAL). pages 292-299.
Libo Qin, Yijia Liu, Wanxiang Che, Haoyang Wen, Yangming Li, and Ting Liu. 2019. Entity-consistent end-to-end task-oriented dialogue system with KB retriever. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 133-142, Hong Kong, China, Association for Com- putational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-orte oriented dialogue. In Ad- vances in Neural Information Processing Systems, volume 33, pages 20179-20191, Curran Associates, Inc.
Guanhuan Huang, Xiaojun Quan, and Qifan Wang. 2022. Autoregressive entity generation for end-to- end task-oriented dialog. In Proceedings of the 29th International Conference on Computational Linguis- tics, pages
Libo Qin, Xiao Xu, Wanxiang Che, Yue Zhang, and Ting Liu. 2020. Dynamic fusion network for multi- domain end-to-end task-oriented dialog. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6344 6354, On- line. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Gautier Izacard and Edouard Grave. 2022. Distilling knowledge from reader to retriever for question an- swering, arXiv preprint arXiv:2012.04584
Dinesh Raghu, Atishya Jain, Sachindra Joshi, et al. 2021. Constraint based knowledge base distillation in end- to-end task oriented dialogs. In Findings of the Asso- ciation for Computational Linguistics: ACL-IJCNLP 2021, pages 5051-5061.
Vladimir Karpukhin. Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao. QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong

---

Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2825-2835, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
A Dataset Statistics
We follow the previous work (Raghu et al., 2021) to split the datasets. The statistics of the three datasets are shown in Table 6.
Md Rashad Al Hasan Rony, Ricardo Usbeck, and Jens Lehmann. 2022. Dialokg: Knowledge-strieture aware orientree dialogue generation. arXiv preprint WxX:12204.09149.
Dataset
Dialogues Train/Val/Tes Test
# Turns
Train/Vall/esd
MWOZ (Eric et al., 2020)
1839/117/141
9943/576/71
SMD (Eric ct al., 2017)
2425/302/304
6291/777/808
CamRest (Wen et al., 2017)
406/135/135
2095/675/643
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen tau Yih. 2023. Replug: Retrieval- augmented block-box language models. arXiv preprint arXhv:2301.12652
Table Statistics of the three datasets.
B Mapping Rules of Meta Knowledge
Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end train- ing of multi- multi-document reader and retriever for open- domain question answering. In Advances in Neural Information Processing Systems, volume 34, pages 25968-25981. Curran Associates, Inc.
B.1 Prefix of Meta Knowledge
The mapping rules from different forms of meta knowledge to the prefix tokens are shown Ta- ble 10. We use the same set of prefix for T5 and ChatGPT. Particularly, for ChatGPT, we design a prompt paraphrase to explain the prefix to Chat- GPT. This explanation prompt is shown below: "Each record knowledge base is accompanied by three tags. The first tag indicates whether this entity appeared before. <new-entity> means this is a new entity, and <old-entity> means this entity appeared before. The second tag indicates the authenticity of the third tag. There are three types <low-confidence>, <mid-confidence> and <high-confidence> indicating low, middle, high retrieval confidence respectively. higher retrieval confidence means the entity potentially more related to the user goal. The third tag indicates its importance to the dialogue. <nth-entity> means it is the nth important entity in the knowledge base, for example, <1th-entity> is the top-1 important and <other-entity> means it is not important."
Xin Tian, Yingzhan Lin, Mengfei Song, Siqi Bao, Fan Wang, Huang He, Shuqi Sun, and Hua Wu. 2022. Q-tod: A query-driven task-oriented dialogue system. arXiv preprint arXiv:2210.07564.
Fanqi Wan, Weizhou Shen, Ke Yang. Xiaojun Quan, and Wei Bi. 2023. Multi-grained knowledge retrieval for end-to-end task-oriented dialog. In Proceedings of the 61st Annual Meeting the Association for Compustational Linguistics (Volume 1: Long Papers), pages 11 192-11210, Toronto, Canada. Association for Computational Linguistics.
Tsung- -siinn--nnnnn Wen, David Vandyke, Nikola Mrkšić, Mil- ica Gašić, Lina M. MojjasBaraahona Pei-Hao Su, Ste- fan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguisties: Volume 1, Long Papers, pages 438 449, Valencia, Spain. Association for Computational Lin- guistics.
Wu, Richard Socher, and Caiming Xiong. 2019. Global-to-local memory pointer networks for task-oriented dialogue. In International Conference on Learning Representations.
B.2 Prompt of Meta Knowledge
The mapping rules from meta knowledge the prompt are shown in Table 11. For ChatGPT, more intricate prompts are used, as presented Table 12.
Jie Wu, Ian Harris, and Hongzhi Zhao. 2022. Graph- memdialog: Optimizing end-to aad-to-end task-oriented di- alog systems using graph memory networks. In the AAAI Conference on Artifiiial Intelligence, pages 11502-11512.
B.3 Discussions
How does generator learn to distinguish entities with meta knowledge?
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text lan- guage models. arXiv preprint 2201 22.2006996..
Our method aims to enable the models to compre- hend meta knowledge. such as retrieval order and confidence scores, and then compare the retrieved entities. To achieve this, we employ distinct strate- gies based on whether the models are fine-tunable

---

not. For fine-tunable models like T3, we include the meta knowledge as a part of the model input, enabling its assimilation during the training pro- cess. In the case of non-fin non-fine-tun fine-tunable models like ChatGPT, our approach utilizes the model's 's existing knowledge to grasp the essence of meta knowledge. This is implemented by providing an explanation of meta knowledge within the system prompt, as elaborated in Appendix B. and Appendix B.2.
Condenud
Large-Scala
MWOZZ
CamRest
SMD
MWOZ
CamRest
Retreedd entties 15 Base Retneved enities T> Large
Table 7: The number of retrieved entities under different settings.
Model
MWOZ
RecalliS
Camfiest
Recallis
Outipeuttz (Large)
15.09
53.59
84.53
27. 52
72.77
$7.05
Ouflynoest (Large)
88.66
26.88
ange)
How to design the prompts of meta knowledge?
Table 8: Overall results of E2E- 22E-TOD systems with large- scale knowledge bases on MWOZ and CamRest.
When it comes the prompts for T5 models, given their fine-tunability, our emphasis lies in the ef- fectiveness of implementation with the pytorch code, and leave the comprehension of prompts to the model itself during the training phase. To achieve this, we craft the prompts in a manner that permits them to be tokenized into sequences of uniform length with the tokenizer, which fa- cilitates efficient mapping to entities through the torch. gather operation.
G Details of Retrievers for Section 5.3
In Section 5.3, we investigated the retrieval- generation misalignment by introducing several retrievers with different performances. The details of these retrievers are introduced as follows.
BM25: The BM25 retriever computes the BM25 score between the dialogue context and each entity. Frequency: This is a rule-based method. For each entity in the knowledge base, we compute the number of its attribute values that appear in the context. We then take the entities with the most attribute values appearing in the dialogue context as the recalled entities.
As for ChatGPT, the final prompts are obtained through evaluations on the dev set MWOZ.
C In-Context Learning Demonstration
The inputs for ChatGPT to include meta knowledge by the prefix and prompt approaches are shown in Figure and Figure 6, respectively.
Pre-train: This retriever is the pre-trained re- triever introduced in Appendix D.
D Pre-training for Retriever
Ours: This is the retriever introduced in our method (OurSprompa (Base)). (Base)))
To pre-train the retriever, we employ distant su- pervision method. This involves labeling the entity that exhibits the highest occurrence attribute values both the dialogue context and system re- sponse as the pscudo positive entity. Subsequently, we conduct pre-training of the retriever using in- batch contrastive learning, considering the positive entities from other examples within the same mini- batch negative entities.
Q-TOD: This is the retriever of Q-TOD.
Oracle: This method uses the condensed knowl- edge base as the retrieved entity set. The gold entity, which must appear the condensed knowl- edge base, is marked as the top-1 retrieved entity with high retrieval confidence, while other entities are marked with low retrieval confidence.
We show the performance (Recall @7) 7) of these retrievers in Table 9.
E Hyperparameter Settings
BM25
Frequency Pre-train
Ours 92.74
Q-TOD
Oracle 100
The hyperparameters of our system with both con- densed and large-scale knowledge bases are shown
75.51
88.66
91.39
92.97
in Table 13. We also retrieve different numbers of entities for different datasets in our experiments, as the details shown in Table
Table The performance (Recall @7) different re- trievers for the retrieval-generation misalignment study.
F Result of MK-TOD with T5-Large on Large-Scale Knowledge Bases
The results of our method built upon T5- L5-Large are shown Table 8.

---

Meta Knowledge
Prefix
Retrieval Order
Firslly recalled entity
<1th-entity>
Secondly recalled entity
<2th-entity>
Thirdly recalled entity
<3th-entity>
Fourthly recalled entity
<4th-entity>
Fifthly recalled entity
<5th-entity>
The entities recalled behind the 5th entity and the easy negative entity
<other-entity>
Retrieval Confidence
Entity with retricval score >= 0.75
<high-confidence>
Entity with retrieval score 0.75 and 0.25
<mi mnid-confidence
Entity with retricval score < 0.25 and the casy negative entity
<low-confidence
Co-occurrence Relation
Entity existed in the dialogue context Entity not existed in the dialoglue context and the casy negative entity
<old-entity> <new-entity>
Table 10: The mapping rules from different types of meta knowledge to the prefix token.
Meta Knowledge
Prompt
Retrieval Order
Firstly recalled entity
"The top-1 recalled:"
Secondly recalled entity
"The top-2 recalled:" "The top-3 recalled:" "The top-4 recalled:" "The top-5 recalled:"
Thirdly recalled entity
Fourthly recalled entity
Fifthly recalled entity
The entities recalled behind the 5th entity and the easy negative entity
"The negative entity recalled:"
Retrieval Confidence
Entity with retrieval score 0.75
"with high confidence:" "with middle confidence:" "with low confidence:"
Entity with retrieval score < 0.75 and >= 0.25
Entity with retrieval score 0.25 and the easy negative entity
Co-occurrence Relation
Entity existed in the dialogue context Entity not existed the dialoglue context and the easy negative entity
"existed in history:" "newly recalled:"
Table II: The mapping rules from different types of meta knowledge to the prompt for T5.
Meta Knowledge
Prompt
Retrieval Order
Firstly recalled entity
"this entity is top-1 important."
Secondly recalled entity
"this entity is top-2 important." "
Thirdly recalled entity
"this entity top-3 important."
Fourthly recalled entity
"this entity is top-4 important."
Fifthly recalled entity
"this entity top-5 important."
The entities recalled bchind the 5th entity and the casy negative entity
"this entity is not important."
Retrieval Confidence
Entity with retrieval score >= 0.75
"It has high possibility that" "It has medium possibility that" "It has low possibility that"
Entity with retrieval score < 0.75 and >= 0.25
Entity with retrieval score 0.25 and the easy negative entity
Co-oceurrence Relation
Entity existed in the dialogue context Entity not existed in the dialoglue context and the casy negative entity
"This entity has appeared before." "This is entity."
Table 12: The mapping rules from different types of meta knowledge to the prompt for ChatGPT.

---

Condensed KB
Large-Scale KB
Hyperparameters
TS-Base
T5-Large
TS-Base
T5-Large
Batch size
2
1
2
1
Gradient accumulation steps
32
64
32
64
Training gradient sieps
1500
Learning rate schedule
Linear
Retriever learning rate
le-4
Response generator learning rate
lc-4
Gradient weight decay
0.01
Gradient clipping Retriever max input length Generator max input length for context Generator max input length for entity Max output length
0.01
128
200
100
Loss weight for Eq. 7
Loss weight |9 for Eq. 7 Loss weight for Eq. I1
Margin À for contrastive learning in Eq. 10
0.01
Table 13: Hyperparameter settings of our system.
You questions Like custoner service. There knowledge base for each question.
Each record of knowledge base is accompanied by three tags.
first tag indicates entity, and sold-entity> means this entity appeared before.
this entity appeared before. <new-entity>
this is
The second tag indicates the authenticity of the third tag. There are three types <low- confidences, smid-confidence> and <high-confidence> indicating confidence respectively. Higher retrieval confidence nean the entity is potentially nore related to the user goal.
niddle, high retrieval
The third tag indicates its importance to the dialegue. <nth-entity> neans it is the nth important entity in the knowledge base, for example, <lth-entity> is the top-1 important and <other-entity> means
not important.
The nax length of your response is 5e words.
Next There are three special tokens [assistant], [user] and [answer]. [assistant] leads the response the customer service, [user] leads what user say and [answer] leads the Ground Truth answer the exanple.
demonstrations.
answer questions Like Knowledge base is in the form of address area internet name parking phone postcade pricerange stars type. Knowledge base is as follow.
reservation
knowledge base.
First one : 124 tenison road east yes and b guest house 01223315782 eb12dp moderate star guesthouse <1st-entity> <Lom-confidence> <new-entity>
Next ane
Next one
That's all of knowledge base. The dialogue is follow:
[user] yes i an looking far place stay tonight the hotel should Like guesthouse in looks and style ideally please
like in moderate price range
[answer] is there specific area you would like to stay in also you need internet and / Free parking That's all of example
[Add the test sample here}
Figure 5: The input prompt and demonstration for ChatGPT with meta knowledge as the prefix.

---

answer questions like custoner service. There knowledge base for each question. The length of your response is 50 words.
demonstrations.
There are three special tokens [assistant], [user] and [answer], [assistant] leads the response the customer service, [user] leads what user say and [answer] Leads the Ground Truth
the example.
answer questions like custoner service hotel reservation with knouledge base. Knowledge base is in the form of address area internet name parking phone postcode pricerange stars type. Knowledge base is as follow.
First one : 124 tenison road east yes and b guest house 01223315762 cb12dp moderate star guesthouse This is new entity. It has low possibility that this entity top-1 inportant. Next one Next one .......
That's all of knowledge base. The dialogue is as follow: [user] yes an Looking for place stay tonight hotel should Like guesthouse in looks and style ideally i like one in the moderate price range please
[answer] there specific you would like stay in also you need internet and / or free parking ? That's of exanple
(Add the sample here}
Figure 6: The input prompt and demonstration for ChatGPT with meta knowledge as prompt.


Reviews:
[{"review_id": "gJ8HAccsyO", "reviewer": "Reviewer_myc6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: The experiments do not demonstrate the effectiveness of the proposed MK-TOD. a) In Table 1, the MK-TOD fails to outperform baselines in Entity F1 on the two datasets (CamRest & SMD). b) In Table 1, the authors do not show the performance of the MK-TOD-Large on SMD.***\n\nA1: In Table 1, our focus is directed towards the comparison of methods with identical model sizes, such as QTOD (T5-base) v.s. ours (Base). Drawing from this criterion, we claim that our proposed method demonstrates superior performance over the baseline methods on both MWOZ and SMD datasets. \n\nRegarding the missing results for T5-large on the SMD dataset, we conducted experiments using only a single 24G NVIDIA RTX 3090 GPU. However, this fell short of what the SMD dataset needed in terms of longer model input. Now, we've included the experiments on the SMD dataset using T5-large with 40g A100 GPUs, and we've displayed the results below. As we update the paper, we'll also supply the missing results.\n\n| Methods          | BLEU  | Entity F1 |\n| ------------------- | ----- | --------- |\n| Ours_prefix (Large) | 24.97 | 72.87     |\n| Ours_prompt (Large) | 25.21 | 73.04     |\n| Ours_ctr (Large)    | 25.43 | 73.31     |\n\nFurthermore, it's important to acknowledge that the outcomes displayed in Table 1 provide an incomplete depiction of our method's performance. As elaborated in Lines 351-360 and detailed in Section 5.2, each dialogue within Table 1 corresponds to a restricted set of 7-8 entities, thereby diminishing the emphasis on retrieval. **In light of this, we contend that Table 2 deserves greater consideration for evaluating our results.** Responding to the feedback from Reviewer b5KD, we intend to emphasize Table 2 in stead of Table 1 in order to avert any potential misunderstandings in the upcoming revision.\n\n***Q2: The proposed MK-TOD is of limited innovation. The essence of MK-TOD is to simply make the generator focus on the top-ranked entities in the retrieval ones. The idea is not refreshing.***\n\nA2: Besides the ranking order, we also introduce confidence and co-occurrence as meta-knowledge to enhance the retrieval-generation alignment of E2E-TOD. At the same time, we introduce prefix, prompt, and contrastive learning to implement these components. We extensively evaluate the above components in Section 5.4.2. Besides, except for the meta-knowledge, we also introduce negative sample in training the response generator, which is novel and effective, as validated in Table 5. Notably, our work is the first attempt to include meta knowledge in E2E-TOD, providing valuable insights for future research in this field.\n\n***Q3: So the proposed method seems to use some tricks. For example, in Table 10, the loss weight in Equ.7 and Equ.11 lacks explanation and validation. In line 269, the authors do not explain and validate the thresholds for categorizing entities.***\n\nA3: Due to limitations in computational resources, we didn't extensively fine-tune hyperparameters. Instead, we opted for a balanced 1:1:1 ratio for the loss weights in Eq. 7 and Eq. 11. We recognize that we didn't thoroughly discuss the thresholds for categorizing entities. To provide clarity, we established the following ranges: (-infinity, 0.4] indicates low confidence, (0.4, 0.75] indicates medium confidence, and (0.75, +infinity) indicates high confidence. These thresholds were chosen based on an empirical evaluation of the retrieval scores. We will incorporate these clarifications when updating the paper.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem this paper focuses on is of interest.\nPrevious works mainly focus on improving retrieval and generation performance separately, while ignoring how to make the improvement of retrieval better serve the final generation performance.\n\n2. This paper clearly states the motivations of the authors and the implementation of the method, which brings a low reading burden.", "reasons_to_reject": "1. The experiments do not demonstrate the effectiveness of the proposed MK-TOD.\na)  In Table 1, the MK-TOD fails to outperform baselines in Entity F1 on the two datasets (CamRest & SMD).\nb)  In Table 1, the authors do not show the performance of the MK-TOD-Large on SMD.\n\n\n2. The proposed MK-TOD is of limited innovation.\nThe essence of MK-TOD is to simply make the generator focus on the top-ranked entities in the retrieval ones. The idea is not refreshing.\n\n3. The paper lacks validation for numerous hyperparameters. So the proposed method seems to use some tricks. For example, in Table 10, the loss weight in Equ.7 and Equ.11 lacks explanation and validation. In line 269, the authors do not explain and validate the thresholds for categorizing entities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In a retrieval-augmented TOD, the correlation between the performance of the retriever and the generator is weak. Thus, the improvements in the retriever may not translate to enhancing the generator.\n\nTo solve the above problem, the authors propose MK-TOD. It contains the maximum marginal likelihood to assist the retriever to train synchronously with the generator, and the meta-knowledge of entities to help the generator utilize the most appropriate entity among all the retrieved ones.\nThe experiments demonstrate that MK-TOD improves the correlation between the performance of the retriever and the generator, and makes a certain improvement in generation and retrieval.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VVgcAdv53q", "reviewer": "Reviewer_b5KD", "summary": "", "questions": "A: What are the missing results in Table 1 for T5-Large? Have you analyzed why the base model has better performance on CamRest?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Automatic evaluation is not enough. Human evaluation should be included in the generation task.***\n\nA1: Thanks for the suggestion. We are conducting a human evaluation and will report the result as soon as it is finished.\n\n***Q2: There are some missing results in Table1 for T5-Large on SMD dataset. It seems that the experiments have not been finished yet. Thus, the results do not support the claim that the proposed method surpasses the baselines on MWOZ and SMD (Lines 429-432). It is also strange to see that the large model provides poorer performance than the Base model on CamRest.***\n\nA2: There are no results for T5-large on the SMD dataset because we only used a single 24G NVIDIA RTX 3090 GPU. However, this didn't meet the SMD dataset's requirements, which needed a longer model input length. Currently, we have added to the experiments on the SMD dataset by using T5-large with 40g A100 GPUs, and we're presenting the results below. We'll include the missing results when we update the paper.\n|Methods           | BLEU  | Entity F1 |\n| ------------------- | ----- | --------- |\n| Ours_prefix (Large) | 24.97 | 72.87     |\n| Ours_prompt (Large) | 25.21 | 73.04     |\n| Ours_ctr (Large)    | 25.43 | 73.31     |\n\nThe reason behind the weaker performance of T5-large compared to T5-base on the Camrest dataset can be traced back to the small size of its training data. This dataset includes merely 406 dialogues, comprising a total of 2095 responses, which potentially leads to overfitting particularly for larger models such as T5-large.\n\n\n***Q3: In this work, Entity F1 is the key metric to evaluate the retrieval-generation misalignment. Table 2, which should be highlighted, shows good performance on large-scale knowledge bases. Table 1, the main table, cannot convince me that the proposed method improves the performance since only results on MWOZ provide better Entity F1.***\n\nA3: We agree that the emphasis should be placed on the results in Table 2. The findings depicted in Table 1 primarily showcase the application of a concise knowledge base (containing only 7-8 entities), which doesn't effectively reflect the impact of retrieval-generation alignment. We will revise this in the forthcoming revision.\n\nThank you once again for your review! We are more than happy to engage in further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Meta knowledge for retrieval is easy to use and the result shows marginal improvement in the ablation study. \n\n- The retriever and the generator are trained in an end-to-end manner and apply the response generation as the signal to update the retriever. \n\n- Authors explore different techniques further improve the performance including contrastive learning, negative entity", "reasons_to_reject": "- Automatic evaluation is not enough. Human evaluation should be included in the generation task. \n\n- There are some missing results in Table1 for T5-Large on SMD dataset. It seems that the experiments have not been finished yet. Thus, the results do not support the claim that the proposed method surpasses the baselines on MWOZ and SMD (Line429-432). It is also strange to see that the large model provides poorer performance than the Base model on CamRest. \n\n- In this work, Entity F1 is the key metric to evaluate the retrieval-generation misalignment. Table 2, which should be highlighted, shows good performance on large-scale knowledge bases. Table 1, the main table, cannot convince me that the proposed method improves the performance since only results on MWOZ provide better Entity F1.", "questions_to_authors": "A: What are the missing results in Table 1 for T5-Large? Have you analyzed why the base model has better performance on CamRest?", "paper_topic_and_main_contributions": "The authors propose MK-TOD to address the retrieval-generation misalignment in the end-to-end task-oriented dialogue system. The authors train the retriever and the generator in the end-to-end dialogue system and apply MML from the response generation as the signal. MK-TOD considers both retrieved entities and meta knowledge to guide the generator and the results show better performance when combined with meta knowledge.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Highlight Table 2 instead of Table 1"}, "extra_text": {}}, {"review_id": "Hlkxt94Dbc", "reviewer": "Reviewer_YzME", "summary": "", "questions": "A. Did the authors explore using the raw retrieval order and confidence scores? On the same topic, did the authors explore a more granular mapping of the confidence scores?\n\nB. On Table 1, why are results not presented for the Ours_x(Large) settings on SMD?\n\nC. In 5.4.1, the \"w/o MML\" settings mean that the pre-trained retriever is not further trained at all?\n\nD. Looking at tables 8 and 9, what is the motivation between different prompts being used in T5 vs. ChatGPT? Have the authors explored using other prompting language, and how has that affected performance?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: The mapping of retrieval order and confidence scores (both continuous numerical values) to discrete and independent string labels seems arbitrary and dismisses how the retrieved entities may compare to each other.***\n\nA1: Good question. Our method aims to enable the models to comprehend meta knowledge, such as retrieval order and confidence scores, and then compare the retrieved entities. To achieve this, we employ distinct strategies based on whether the models are fine-tunable or not. For fine-tunable models like T5, we include the meta knowledge as a part of the model input, enabling its assimilation during the training process. In the case of non-fine-tunable models like ChatGPT, our approach utilizes the model's existing knowledge to grasp the essence of meta knowledge. This is implemented by providing an explanation of meta knowledge within the system prompt, as elaborated in Appendix B.\n\n***Q2: The prompt based results may be particular to the specific language used to create the prompts. The paper includes no study on alternative phrasings of the prompt.***\n\nA2: We agree. We have experimented with various prompts to arrive at the final one, particularly for ChatGPT. Due to the limit of space, we were unable to thoroughly showcase the prompt design process. We will provide a detailed introduction of the prompt design process in our future revisions.\n\n***Q3: Did the authors explore using the raw retrieval order and confidence scores? On the same topic, did the authors explore a more granular mapping of the confidence scores?***\n\nA3: Good question. Regarding the ranking order, which involves discrete numbers, we incorporate the actual raw ranking number directly into the prompt. For instance, the number \"1\" is included in the prompt \"the top 1 recalled.\" However, as evidenced by the results in Table 4 (Ours_prompt (order)), T5 exhibits limitations in comprehending the order conveyed by these discrete numbers.\n\nIn terms of confidence scores, which entail continuous numbers, multiple studies have demonstrated that even powerful models like ChatGPT struggle with precise numerical problem-solving (https://arxiv.org/abs/2103.07191). Consequently, in this paper, we opt for a course-grained confidence measure rather than relying on exact confidence scores.\n\n***Q4: On Table 1, why are results not presented for the Ours_x(Large) settings on SMD?***\n\nA4: When conducting this research, our experiments were ran using a 24G 3090 GPU, which fell short of the requirements for the SMD dataset, with its demand for longer model input length. Currently, we have finished the experiments on the SMD dataset using T5-large with 40g A100 GPUs, and the results are presented below. We will provide the missing results when updating the paper.\n\n| Methods          | BLEU  | Entity F1 |\n| ------------------- | ----- | --------- |\n| Ours_prefix (Large) | 24.97 | 72.87     |\n| Ours_prompt (Large) | 25.21 | 73.04     |\n| Ours_ctr (Large)    | 25.43 | 73.31     |\n\n***Q5:. In 5.4.1, the \"w/o MML\" settings mean that the pre-trained retriever is not further trained at all?***\n\nA5: Yes, we will provide explicit clarification on this.\n\n***Q6: Looking at tables 8 and 9, what is the motivation between different prompts being used in T5 vs. ChatGPT? Have the authors explored using other prompting language, and how has that affected performance?***\n\nA6: When it comes to prompts for T5 models, given their fine-tuning capability, our emphasis lies in the effectiveness of implementation with the ```pytorch``` code, and we leave the comprehension of prompts to the model itself during the training phrase. To achieve this, we craft the prompts in a manner that permits them to be tokenized into sequences of uniform length with the ```tokenizer```, which facilitate the efficient mapping to entities through the ```torch.gather``` operation.\n\nFor prompts for ChatGPT, we have experimented with various prompts to arrive at the final one. Due to the limit of space, we were unable to thoroughly showcase the prompt design process. We will provide a detailed introduction of the prompt design process in our future revisions.\n\nThank you for the review! Feel free to let us know if you have more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is simple to understand and is shown to lead to consistent performance gains in a variety of settings.\n- The paper includes thorough analysis and properly supports the efficacy of its proposed approach.\n- Well motivated and written paper.\n", "reasons_to_reject": "- The mapping of retrieval order and confidence scores (both continuous numerical values) to discrete and independent string labels seems arbitrary and dismisses how the retrieved entities may compare to each other. \n- The prompt based results may be particular to the specific language used to create the prompts. The paper includes no study on alternative phrasings of the prompt.\n", "questions_to_authors": "A. Did the authors explore using the raw retrieval order and confidence scores? On the same topic, did the authors explore a more granular mapping of the confidence scores?\n\nB. On Table 1, why are results not presented for the Ours_x(Large) settings on SMD?\n\nC. In 5.4.1, the \"w/o MML\" settings mean that the pre-trained retriever is not further trained at all?\n\nD. Looking at tables 8 and 9, what is the motivation between different prompts being used in T5 vs. ChatGPT? Have the authors explored using other prompting language, and how has that affected performance?\n", "paper_topic_and_main_contributions": "This paper addresses shortcomings in retrieve-and-generate approaches for knowledge-grounded dialogue systems, and specifically aims to improve the information flow from the retriever output to the generator, as they observe that improvements in retrieval performance often do not correlate with improvements in generation. To that end, the paper proposes to include 'meta-knowledge' labels that denote the retrieval order, confidence, and entity co-occurrence, alongside the input of the generator. The retriever is trained by maximizing the marginal likelihood of the response and the generator is further trained using contrastive learning and negative entity sampling (with corresponding 'meta-knowledge' labels). \n\nThe paper presents results on three datasets (MultiWoz, CamRest, SMD) and on two settings: using session-level or dataset-level knowledge bases. The proposed additions lead to consistently better output quality (measured by BLEU) but the performance increase is not as prevalent on Entity-F1 when using session-level knowledge bases, and Recall@7 when using dataset-level  knowledge bases. The paper also includes results when employing ChatGPT as the generator and prompting it with the retriever output. The included meta-knowledge show consistent improvements across the board in this setting, but cannot be compared with previous work.\n\nThe paper also includes some analysis that shows that when using their additions, the correlation between retriever and generator performance increases, as well as ablation experiments over how to best represent the additional knowledge (prefix vs. prompt), the use of maximum marginal likelihood, the effect of including different types of \"meta-knowledge\" labels, and negative samples.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The conclusions of section 5.4.2 are a bit confusing ('on the other hand' followed by 'in contrast'), please rephrase.\n"}, "extra_text": {}}, {"review_id": "zfZpS7m14s", "reviewer": "Reviewer_ybw6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: The scores chatgpt reaches in the experiment are too low whether meta-knowledge is introduced, which means other metrics other than BLEU and Entity F1 might be necessary for the evaluation.***\n\nA1: Despite the application of in-context learning, ChatGPT continues to attain relatively lower Entity F1 scores, mainly because it hasn't undergone fine-tuning on the E2E-TOD datasets. However, this shouldn't be interpreted as ChatGPT performing inadequately on these datasets. Certain behaviors like generating overly lengthy responses or including unnecessary attributes may account for the lower scores. We will present the results of human evaluation once it's completed.\n\n***Q2: More types of meta-knowledge and more methods to leverage them might be applied in the framework.***\n\nA2: Indeed, our framework can accommodate a broader range of meta-knowledge types. We will further investigate these possibilities using widely adopted fine-tunable language models, including Llama-2-chat.\n\n\n***Q3: The consistence between the quality of meta-knowledge, the performance retriever and ground-truth lacks study. It’s better to include more analysis in the future stage.***\n\nA3: We appreciate your suggestions. We have assessed the retrieval results on the MWOZ test set. The findings demonstrate that our retriever adeptly recalls 80.69% of the gold entities with a top-1 retrieval order. This directly correlates with the system-wide performance enhancement, as shown in Section 5.5, where our generator prioritizes entities with higher retrieval orders. We will include the result when updating the paper.\n\nWe appreciate your positive feedback. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper puts forward the misaligned problem in the retrieval-generation process, which is of practical significance. The meta-knowledge they introduce in the process combines the retriever and generator more tightly and proves to work for both small and large models, which means the meta-knowledge might be introduced in various future works in similar paradigms.", "reasons_to_reject": "1. The scores chatgpt reaches in the experiment are too low whether meta-knowledge is introduced, which means other metrics other than BLEU and Entity F1 might be necessary for the evaluation.\n2. More types of meta-knowledge and more methods to leverage them might be applied in the framework.\n3. The consistence between the quality of meta-knowledge, the performance retriever and ground-truth lacks study. It’s better to include more analysis in the future stage.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper raised a \"misalignment\" problem in E2E-TOD systems, where the generator may sometimes mismatch with the retriever, so that better performance of the retriever may not lead to a better performance of the whole system. To alleviate the problem, the paper introduces a novel framework where their so-called meta-knowledge (the confidence of the retriever etc.) is taken into consideration. Multiple methods including special prefix tokens, prompts and contrastive learning are tried for the usage of the proposed framework, and meta-knowledge proves to work for various systems.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Meta knowledge is misspelled as mata knowledge in figure 2."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the work presented in the paper is well-motivated and paper is generally well-written. Empirically, the proposed method achieves marginal improvements. Questions around missing experiments and result analysis were largely addressed during rebuttal.
Decision: Accept-Main