Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation
Authors: Chenxu Yang
Zheng Lin
Lanrui Wang
Chong Tian
Liang Pang
Jiangnan Li
Qirong Ho
Yanan Cao
Weiping Wang
Venue: EMNLP 2023 Main (2023)
Keywords: knowledge-grounded dialogue generation
contrastive learning
text degeneration
pre-trained language model

Abstract: Knowledge-grounded dialogue generation aims to mitigate the issue of text degeneration by incorporating external knowledge to supplement the context. However, the model often fails to internalize this information into responses in a human-like manner. Instead, it simply inserts segments of the provided knowledge into generic responses. As a result, the generated responses tend to be tedious, incoherent, and in lack of interactivity which means the degeneration problem is still unsolved. In this work, we first find that such copying-style degeneration is primarily due to the weak likelihood objective, which allows the model to "cheat" the objective by merely duplicating knowledge segments in a superficial pattern matching based on overlap. To overcome this challenge, we then propose a Multi-level Adaptive Contrastive Learning (MACL) framework that dynamically samples negative examples and subsequently penalizes degeneration behaviors at both the token-level and sequence-level. Extensive experiments on the WoW dataset demonstrate the effectiveness of our approach across various pre-trained models and decoding strategies.

Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation
Chenxu Yang! Zheng Lanrul Wang Chong Tian', Liang Pang', Jiangnan Li Qirong Ho², Yanan Cao Weiping Wang 1.2
Lanrui
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 'Institute of Computing Technology, CAS *Mohamed bin Zayed University of AI {yangchenxu, wanglanrui, lijiangnan, linzheng, wangweiping ac cn (refrainkon, wwq.ooiiionng.o @gmmil com, pangliang@ict. ac com
Abstract
Blue aboups nioe. like roypal blue.
the
User
Chic Ruiroad
Showledge-groundde dialogue generation aims mitigate the issue of text degeneration by in- corporating external knowledge to supplement the context. However, the model often fails internalize this information into responses in a human-like manner. Instead, it simply in- serts snippets of the provided knowledge into generic responses. As result, the generated responses tend to be tedious, incoherent, and in lack of interactivity which means the degen- eration problem is still unsolved. In this work, we find that such copying-style degeneration is primarily due to the weak likelihood objec- tive, which allows the model to "cheat" the ob- jective by merely duplicating knowledge snip- pets in a superficial pattern matching manner based on overlap. To overcome this challenge, we propose Multi-level Adaptive Contrastive Learning (MACL) framework that dynamically samples negative examples and subsequently penalizes degeneration behaviors at both the token-level and sequence-level. Extensive ex- periments on the WoW dataset demonstrate the effectiveness of our approach across various pre-trained models and decoding strategies.
Royal
flophta gasin
Agent
York Woekkiggton D.C.
sourd really nice. bet
blus
Slue Scee: 2045 Amer musical direceed Stuart Henter and starting Rlng Cauday Frod fibd
speaking flue Skile, hove the 1945 boriing Crosby?
G1
Agent
tagrte Bur Steenis in American mustcal comedy film deocted by Stuars hibs and starr Sire coochr Frod
Salat and lorn
mode 1946. stares ling Ereeby Feed Autaine, Courhels
BART
Figure 1: An example shows how current models make rigid use of knowledge and cause the responses inco- herent with context. Both T5-large and BART-large are finetuned on WoW dataset with MLE.
as the Knowledge-Groundee Dialogue Generation (KGDG). Recent methods have put much empha- sis on the knowledge selection subtask, aiming to provide models with the most suitable knowledge (Kim et al., 2020; Chen et al.. 2020; Zheng et al., 2020; Yang et al., 2022).
1 Introduction
However, simply introducing golden knowledge as ground source does not indeed mitigate the problem of degeneration. We observe that exist- ing models often duplicate knowledge snippets construct responses to meet the informativeness requirement, which can lead to unnatural and con- textually incoherent responses. For example, the generated contents in Figure 1 simply duplicate the provided knowledge about the "blue skies" movie, resulting in tedious response incoherent with user's utterance. Consequently, we need to con- sider not only the introduction of knowledge, but also how to effectively integrate it into responses. We term the superficially knowledge duplicat- ing "Knowledge Regurgitation" as mirrors how the model absorbs an entire knowledge sentence
In recent years, pre-trained language models using transformer architectures have made remarkable strides in open-domain generation tasks (Lewis et al., 2020; Raffel al., 2020; Zhang et al., 2020; Roller et al., 2020). However, these models still struggle with dull and repetitive outputs, a prob- lem commonly termed as neural text degeneration (Holtzman et al., 2020; Welleck et al., 2019).
To address text degeneration in dialogue, Dinan al. (2018) proposed to equip interlocutors with external knowledge as additional support to enrich the informativeness of responses, which known
Zhheng Lin is the corresponding author. 'The code is available at https:// com/ sie-yCx/MACL.

---

and regurgitates it in the response without genuine comprehension. This issue can be seen as task- specific copying-style manifestation of text de- generation. We test some pre-trained models with different parameter scales, and discover that the Knowledge Regurgitation is common among them. To quantify its severity, we design two automated metries: PoD and KuD. Based on series of ex- periments, posit that degeneration is caused by the ineffecttul design of the training objective, which allows models "cheat" the MLE objective by merely duplicating knowledge snippets to con- struct responses. The model, therefore, improperly converges to superficial pattern matching based on overlap, given that knowledge sentences share spurious correlations (token overlaps) with ground- truth responses and exhibit high semantic fluency. In this paper, to tackle the aforementioned issues, we propose novel approach known as Multi-level Adaptive Contrastive Learning. This method effec- tively mitigates Knowledge Regurgitation through contrastive training at both the token and sequence levels. At the token level, enhance the negative training paradigm (Welleck et al., 2019) by em- ploying dynamic negative token sampling method and reweighting the unlikelihood loss according to model sensitivity. In this way, our approach penal- izes tokens that continuously appear the knowl- edge but are not the targets to cut off the potential shortcuts. At the sequence level, we first employ a group beam search strategy (Vijayakumar et al.. 2018) sample negative responses from the de- generator's predictions, then use novel metric as an oracle function to score them. Finally, samples demonstrating obvious degeneration are selected as hard negatives for the InfoNCE loss to distance them from their prefix in the representation space. It can expose the model to potential degeneration mistakes that happen at the inference stage and help the model learn to avoid them.
We conduct extensive experiments and pro- vide detailed analysis to validate the effec- tiveness of our method, showing a substantial improvement on both automatic and human evaluation metrics.
Preliminaries
In this section, we provide a brief introduction to the KGDG task and the Unlikelihood Training (UT) loss, which serves as the foundation for our token- level contrastive learning loss.
2.1 Task Formulation
The knowledge-grounde dialogue generation task encompasses two relatively independent sub-tasks: knowledge selection and knowledge- aware response generation. Knowledge selec- tion aims at selecting the most appropriate piece of knowledge, denoted k, from a given knowl- edge pool KP k2 .2....................... .kkk). Assuming that knowledge k has been selected by an effi- cient knowledge selector, we mainly focus on its utilization. Given the dialogue context u {ui, u2 ......, Um um and the related knowledge k [ki
), the goal is to generate an engag- ing and informative knowledge-infused response
y {y), 32,,......... .......... ..yl)
Given KGDG dataset (Dinan et al., 2018) D {(()), k()), ,)))(()))) derived from a collec- tion of multi-turn human interactions, the standard method for training a tooueeece----seencee model involves concatenating the context with the knowl- edge as the model's input sequence and applying maximum likelihood estimation (MLE) to mini- mize:
ly] log p(vn u, k, Vee),
(MLE(u.,y)) y)
(1)
where u is the context (golden dialogue history and user utterance at the current turn), k is the golden knowledge, and ys is the i-th token of
Our contributions are summarized as follows:
We explore unique degeneration phe- nomenon in KGDG, termed "Knowledge Re- gurgitation", which is confirmed by a series of preliminary experiments in popular pre- trained language models.
2.2 Unlikelihood Training
To address the problem of neural text degenera- tion, Welleck et al. (2019) proposed the unlike- lihood training method, combining a token-level unlikelihood objective with MLE. The core idea in- volves selecting a of negative token candidates. denoted as C,, at each training step and reducing their prediction probability p(ye), while concur- rently increasing the probability of ground-truth
We propose Multi-level Adaptive Con- trastive Learning (MACL) framework, which is designed to effectively internalize knowl- edge at both token and sequence levels.

---

tokens p(t:). Negative candidates typically consist of tokens that have already been generated, alleviat- ing the degeneration phenomenon where generated texts contain undesirable repetitions at various lev- els and high frequency tokens appear excessively.
Dup-16
Dup-32
PLCCS
mKP-1
T5- base
72.82%
61.84%
73.68%
82.25%
T5-large
66.33%
57.55%
69.88%
80.04%
BART-buse
70.55%
61.76%
73.20%
81.67%
BART-large
54.93%
46.34%
63.05%
75.91%
GPT2-large
45.52%
40.53%
53.85%
71.10%
GPT2-xl
54.45%
48.18%
60.07%
74.76%
LLaMa-7B
34.21%
37.34%
44.09%
60.56%
CUL(C,
log(1 p(ucl Mcr)).
human 6.53%
2.37%
24.47%
47 47.83%
McECc
Ct {x, ,1, 12. ................................ .011
Table 1: Part of the preliminary experiment results the WoW dataset.
(2)
where Cj is subset of the vocabulary and is the prefix sequence.
For PLCS, is calculated as ratio of the length of longest common sub-sequence (LCS) of response and knowledge to the length of response as follows:
The MLE loss aims to model the groud-trum truud sequence probability distribution, while the unlike- lihood loss corrects undesired patterns. The overall objective in unlikelihood training is mixed by them
LCS k. y)
follows:
PLCS
D
(5)
(k.y)D
LUT LILE + aLui,
(3)
The aforementioned metrics upproximately mea- sure the frequency of a degenerated knowledge uti- lization pattern where only snippets of the provided knowledge are inserted into a generic response without substantial integration.
where a is hyper-parameter varying from differ- ent tasks and datasets.
3 Knowledge Regurgitation
To highlight the gap in knowledge utilization patterns between humans and models, we further employed metries related the precision of knowl- edge grams (mKP-n), which calculates the mean ratio of knowledge tokens in response to all the response tokens. is performed as follows:
Although current pre-trained language models have demonstrated robust performance in generating fluent dialogue responses, they fail to align with human-like patterns of knowledge integration.
We conduct series of preliminary experiments to demonstrate the presence of text degeneration problems in some pre-traine trainra language models of various sizes. Specifically, we quantify the sever- ity of this issue by comparing human and model performance based on our newly proposed metrics. The results are presented in Table and Figure 2, which demonstrates the model's severe knowledge regurgitation and singular knowledge utilization patterns.
mKP-n =
KP-n.
(6)
D
(k.y)ED
KP-n In-gram n-gram(())
(7)
n-gram(y))
A comparison of the performance of each MLE- finetuned PLM with human dialogue, presented in Table 1, makes it evident that the duplication frequency of models surpasses that of humans. Al- though humans occasionally replicate professional knowledge and famous quotes to construct knowl- edgeable responses, such usage is less prevalent in casual chitchat, as shown in the last row of Table 1. Models, however, tend to misinterpret the du- plication pattern superficially and apply it across various contexts.
Dup-n and Proportion of Longest Common Sub- sequence tokens (PLCS) are mainly used to mea- sure knowledge snippets duplication. Dup-n rep- resents the proportion of samples with n-grams co-occurring in knowledge and response and is computed as follows:
Dup-n-
Σ
1(n-gram(k) O n-gram(y))).
D
In chit-chat scenarios. is important to strike a balance with the number of knowledge tokens used in responses. Excessive knowledge tokens can re- duce the interactivity of the response, potentially
(
where D denotes the dataset, denotes the golden knowledge, and y denotes the generated response.

---

diminishing users' desire continue the conversa- tion. From the data presented in Table 1, we can observe that knowledge tokens account for approx- imately 50% of human responses, whereas the pro- portion in model-generated responses is too high. Additionally, the knowledge precision distributions between humans and models are illustrated in Fig- ure 2. It highlights that humans exhibit more diverse usage of given knowledge across various contexts, as evidenced by their uniform knowledge precision distribution. In contrast, the distributions of pre-trained language models exhibit a distinct pattern, with the probability mass predominantly concentrated on the higher percentage side.
We empirically investigate the selection strategy of negative examples in our study. Given the objec- tive of the knowledge-grounded dialogue task to effectively integrate knowledge into the response, and that these knowledge tokens are essential po- tential candidates, punishing all knowledge with- out distinction is not appropriate. Our goal is to eliminate shortcuts leading to knowledge duplica- tion while keep the knowledge integrating ability. To achieve this, we propose to dynamically select negative tokens based on the model's sensitivity, thereby reducing the probability of tokens where the model is more likely to err. Specifically, we adopt a strategy of selecting the knowledge token with the highest prediction probability as the nega- tive token. We also explored other strategies, such as sampling by probability, random selection, but the adopted strategy outperforms the others. We conjecture that this is due to our strategy target- ing the prediction chains of the knowledge snippet (shortcuts), effectively disrupting it.
60.1 gax
As for the dynamic reweighting strategy, we con- sidered two reasons. Firstly, Jiang et al. (2020) highlight the importance of applying differentiable weights to individual token losses by proposing To- ken Focal Loss inspired by Lin al. (2018)'s work. In light of this, we enhance the unlikelihood loss by introducing an additional control parameter B(ye) to dynamically reweight the punishment strength for different tokens. By doing so, we suppress the gradients of easy tokens while amplifying the gradi- ents of hard tokens, leading to faster and improved convergence during training.
Figure 2: Knowledge Precision Distribution of human responses (red) and machine- generraed responses (blue) the WoW dataset.
4
Our Approach
We propose a novel multi-level contrastive learn- ing method that dynamically samples negative ex- amples and subsequently penalizes degeneration behaviors at both the token and sequence levels. Token-level Contrastive Learning The motiva- tion behind the token-level loss is to break the gen- eration inertia more effectively (refer to the Figure 3 for a detailed explanation). Our improvements over basic unlikelihood training method are mainly twofold: dynamic negative sampling and dynamic unlikelihood loss reweighting. The detailed loss function is presented in the following formula:
Secondly, Lin et al. (2021) expressed concern that the vanilla negative training method might cause the model to decrease the probability of the target token p. to reduce the gradient norm /aal (10) during the final stages of training, particularly when is excessively large (refer to the Appendix B for detailed explanation). Our approach miti- gates the problem of that inverse optimization and allows the loss function converge to lower value as the introduction of ((ye) facilitates the gradual decrease of the weight of LUi. (2).
Ctoken (Ct, y) log p(vil u, k, Mes))
Σ B(yc) log(1 p(uclu, k. Ver)),
(8)
CC
Ground-Truth Token (i = 8²):
where Cs and B(y:) are calculated as follows:
OC Op: 1 - pi(1 - a:3(3c) 1 Po Dp: Da;
Pc
V C.
)
C. arg max
A(ye) = cos( - pm)=) +
(10)
Sequence-level Contrastive Learning Regard- ing the sequence-level contrastive loss, An et al. (2023) point out that contrastive learning provides
where Pe is predictive probability of ye- and it is a scalar detached from the computational graph.

---

(A1)
Token-level
Sequence-leve
1...
vecabulary
rxpresestation spece
verable
Decoder
Decoder
Decoder
T...
T
(ROS] Yo-2
(Ria))
knowldop bedge
Response
pull
Responsel Y
prech
UrSs nac mm
vocabulary
Encoder
emerggon
E Eneader
Degenerator
9:1
Cooleec kekweleee
Contasti Knaolloggi
[oontect)
laowwedge)
knnnledge ledge
Figure 3: An overview the MACL, framework. The mark the top left corner of figure indicates that certain response tokens identical the knowledge tokens enclosed in brackets. Owing the generation inertia, that is, the shortcut, the model predicts an exceptionally high probability for the next knowledge token, which surpasses the probability of the ground-trute token. MACL breaks the knowledge snippet chain effectively and outputs more reasonable distributions. represents the source sequence (context and knowledge), and by stands for its target sequence (response). The feature representations are derived from pooling the outputs of the encoder (source sequence) or decoder (target sequence), which are both differentiable this context.
a novel solution to alleviate the exposure bias prob- lem. Arora ct al. (2022) suggest that the degener- ation problem is a result of exposure bias, which motivates us to address this issue by leveraging the sequence-level contrastive learning method during the training phase. By exposing the model nega- tive targets exhibiting degeneration, we aim help the model learn to avoid predicting them.
the model with a negative supervision signal on the shortcut paths that generated these sequences. Both hard negative samples and from-batch neg- ative samples are retained, but we assign higher weights the former to emphasize their impor- tance.
The final objective function defined as:
( Lroken (0) + XLseq (0).
(12)
Loog
See the Appendix E for the pseudo-code of the entire training procedure.
(11)
log
Evee ccaa(R,. Ryr Σ x cH coolaz.
5 Experiments
where B from-batch negative samples, H are generated hard negative samples and y is used to reweight the hard negatives.
5.1 Experimental Setup
Dataset. Three common datasets have been typ- ically used to evaluate the KGDG task: Holl-E (Moghe et al., 2018), CMU_Dot (Zhou et al.. 2018), 8), and WoW (Dinan et al., 2018). However, the topics in the first two datasets are limited to movies, with much of the knowledge being com- posed of movie reviews in the form of dialogues. This not in line with our goal of exploring the internalization of retrieved world knowledge. The CMU_ DoG dataset did not label golden knowl- edge, so there is no way to tell if the knowledge introduced is correct. Through observations and experiments, we find that the Holl-E dataset is also not suitable for conducting the evaluation of knowl- edge regurgitation, and put the detailed reasons and experimental results in the Appendix F. Conse- quently, we choose the WoW dataset for our experi- ments. See the Appendix A for a brief introduction to the chosen WoW dataset.
We begin by collecting a diverse set of utter- ances that exhibit significant degeneration prob- lems using the following negative sampling method. Firstly, we train the PLM with MLE to obtain de- generator, which generates responses exhibiting degeneration phenomena. We then employ group beam search strategy (beam size b) acquire nega- tive responses from this degenerator. Finally, these responses are evaluated using a metric that mea- sures the degree of duplication, serving as an oracle function for scoring. We utilize the length of the Longest Common Sub-sequence (LCS) with knowl- edge sequence as the oracle function. We retain the top degenerated response samples.
Following the collection procedure, drive the distance between the source sequence representa- tion and the negative target sequence representa- tion in a contrastive way. The Lweg loss provides

---

Baseline Methods. We compare our MACL frame- work with vanilla MLE and several state-of-the-ar (SOTA) methods that address the issue of text de- generation.
propose considering samples with Proportion of Longest Common Sub-sequence (PLCS) greater than 70% as degenerated samples. We manually annotate a subset of samples from the test set and calculate the precision of the automated metric, PoD. The annotators are given the following in- structions: A generated response will be classified as degenerated if it 1) evidently replicates external knowledge and 2) produces content that is visibly unnatural and contextually incoherent. The results in Table demonstrate effectiveness of the PoD metric. To compare the gap between different meth- ods and humans in knowledge utilization patterns, we introduce the metric KUD, which is defined as:
NT: Welleck et al. (2019) proposed the concept of unlikelihood loss, combining it with MLE loss. Scalegrad: Lin et al. (2021) modified the gradient of the MLE, encouraging the model to use novel to- kens. we designate knowledge tokens as non-novel tokens.
ND: Li al. (2022) developed novel training paradigm known as negative distillation, designed to steer the model away from undesirable degen- erated responses. We utilize the MLE-finetuned model as the negative teacher.
CTloss: Jiang et al. (2022) put forward new con- trastive token learning objective. This objective aims promote label tokens in the ranking each step while demoting negative tokens, leaving other irrelevant tokens unaffected.
KUD = (KPP
(13)
where Pn is the distribution of human response. and g is the distribution of generated response. For human evaluation, We randomly select 50 responses in test seen set and 50 responses in test unseen set. We conducted an aspect-based pairwise preference test. Specifically, for a given context, paired model's response with response from the baselines and asked five well- educated annotators choose the superior re- sponse based on the following four aspects: (1) Coherence: which model generates more contextu- ally coherent responses; (2) Engagingness: which model generates more interesting responses; (3) Informativeness: which response contains more knowledge; (4) Interactiveness: which model gen- erates more interactive responses that make the user want continue the conversation. We compute Fleiss' kappa value (Fleiss, 1971) among different annotators measure their agreement.
SimCTG: Su et al. (2022) introduced a contrastive objective designed to learn discriminative and isotropic token representations by increasing the distances between distinct tokens' representations.
Implementation Details. We use PyTorch (Paszke et al., 2019) framework to implement our work. For the implementation of PLMs BART, T5 and GPT-2, we utilize the open-source Hugging Face transformers (Wolf et al., 2020). The whole model is optimized with Adam (Kingma and Ba, 2014) algorithm. We set the learning rate to le-5 and training batch size to 16, train up to 15 epochs, and select the best checkpoints based on performance on the validation set. Some set as follows:
are 4, A 1, 2, b 32, m 16. At the inference stage, we utilize decent knowledge selector designed by Yang et al. (2022) to select knowledge first and utilize it to generate response to fit the KGDG. The decoding strategy is set to beam search with beam size of 3.
5.2 Experimental Results
Table 2 presents the automatic evaluation results on the WoW dataset. Our method, MACL, signifi- cantly mitigates knowledge regurgitation, reducing the proportion of degeneration by 13.42% on the test seen set and 16% on the test unseen set com- pared to the strongest baseline, Scalegrad. In terms of KUD metric, the enhancements of MACL are impressive, exhibiting stark reduction in discrep- ancics from humans. It is about ten times better than the best performing baseline NT on the test seen dataset and five times superior on the test unseen dataset. The distribution of 1-gram and 2-gram knowledge, as shown in Figure 4, closely aligns with that of humans, indicating that MACL
Evaluation Metrics. We choose perplexity (PPL) of the ground-truth responses, BOW Embedding (Avg., Ext.) (Liu et al., 2016), (Painnna et al., 2002), Knowledge Utilization Difference (KUD), and Porportion of Degenerated samples (PoD) as automatic metrics for evaluation. The latter two are the metrics we propose to measure knowledge regurgitation and the difference from humans knowledge utilization pattern.
Given that the primary characteristic of degen- eration is the duplication of knowledge fragments,

---

Test Seen
Test Unseen
PPL
PoD(%)
KUD
Avg.
Ext.
BLEU-1
PPL
PoD(%)
KUD
Avg.
Ext.
BLEU-1
MLE
23.784
52.31
7.28
0.842
0.428
23.7
26.146
54.78
7.62
0.839
0.422
NT
22.858
25.16
4.85
0.848
0.436
23.3
25,008
26.97
5.89
0.845
0.430
22.2
ND
53,653
29.68
5.34
0.852
0,428
22.0
63.907
25,40
5,28
0,847
0.419
20.4
CTloss
27.599
37.56
5.29
0.839
0.431
23.3
32.704
35.92
5.12
0,838
0.425
22.2
SimCTG
24.466
52.56
7.25
0.842
0.428
23.8
28.679
54.07
7.65
0.838
0.422
22.6
Scalegrad
25.204
21.35
5.02
0.843
0.436
22.3
29.461
23.82
5.88
0.840
0.428
21.7
MACI.
20.897
7.93
0.52
0.852
0.438
22.5
23.973
7.82*
1.11*
0.848
0.435
22.0
Table 2: Automatic Evaluation results on the WoW dataset (BART-lrgee, The best results are highlighted with bold. "*" denotes that the improvement to the best baseline statistically significant (t-test with p-value 0.01).
Coherence
Engagingness
Informativeness
Interactiveness
Compurisons
Lose
Tic
8
Lose
Tie
Tic
MACL NT
22.8
6.8
70.4
0.596
12.2
58.6
0.535
66.8
0.472
49.8
0.561
MACL vs. SimCTG
35.2
5.0
59.8
0.586
42.2
6.6
51.2
0.522
22.0
25.0
53.0
0.445
45.2
8,4
46.4
0.647
MACL Scalegrad
22.0
4.0
74.0
0.481
29.2
7.6
63.2
0.618
23,4
9.0
67.6
0.474
32.2
6.8
61.0
0.481
Table 3: Human Evaluation results on the WoW dataset (%). 0.05.
The result is statistically significant with y-value <
Method
PoD(aotu)
PoD(human)
PoD Precision
gitation, contain a higher number of knowledge tokens. As a result, the hit rate of knowledge to- kens in the response is higher, leading to inflated scores for degenerated responses, despite their con- textual incoherence. In the example provided in Table 10, we observed that the BLEU-1 score of the MLE-generated responses is higher than that of the MACL- MACL-ggeeertedd responses (30.77 17.65).
MLE 52%
55%
94.2%
MACL
8%
9%
87.5%
Table 4: The proportion of samples with degeneration phenomenon annotated by humans on the 100 samples extracted, and the precision of our proposed metric PoD.
successfully internalizes knowledge into responses and achieves similar ability utilize knowledge as humans.
More experimental outcomes based on other pre- trained models and decoding strategies are detailed in the Appendix C.
While the baseline methods effectively address traditional text degeneration, their impact on alle- viating knowledge regurgitation is not as signifi- cant. This suggests that knowledge regurgitation is distinct from conventional repetition-style degen- eration. We believe this is because KGDG ensures that knowledge is integrated into the response, and simply reducing the prediction probability of all tokens in the prefix insufficient. This highlights the effectiveness of MACL's dynamic sampling and dynamic penalty mechanisms in adapting to the KGDG task.
The human-based evaluation results are shown Table 3. Notably, MACL consistently outperforms all the compared methods. MACL effectively in- ternalizes knowledge into its generated responses. As result, the responses produced by MACL are more natural and human-like, avoiding direct nar- ration. Regarding the Context Coherence metric, MACL maintains a relatively better focus on the user, carefully balaneing the attention between the user's utterance and the knowledge during the gen- eration stage. While MACL may lose some in- formation compared to SimCTG's responses, it is important to note that excessive knowledge is inap- propriate in chit-chat scenarios. The kappa results indicate moderate level of agreement among the annotators.
In terms of conventional metrics that evaluate content quality, MACL achieves state-of-the-art (SOTA) performance in terms of perplexity, av- erage (Avg.), and extrema(Fxtt.). This indicates that the quality of the generated responses is high and comparable human performance. However, for the BLEU metric, which is based on gram overlap, MACL performs slightly worse than the baselines. We attribute this to the fact that the longer responses, which reveal knowledge regur-
5.3 Ablation Study
To analyze the sources of improvement achieved by MACL,, we conducted an ablation study. The ablation results, shown in Table 5, indicate that all of these design components contribute to the

---

Test Seen
Test Unseen
ppl
PoD(S)
KUD
Avg.
Ext.
BLEU-1
ppl
PoD(3)
KUD
Avg.
Ext.
BLEU-1
MACL
20.897
7.93
0.52
0.852
0.438
22.5
23,973
7.82
1.11
0.848
0.435
22.0
Reweighting
23.663
12.14
2.71
0.850
0.437
22.9
26.260
14.09
3.30
0.847
0.432
22.3
-TCL
23.441
37.76
6.49
0.844
0.431
23.5
26.473
38.05
7.01
0.840
0.423
22.4
NaiveSCL
21.592
22.37
1.57
0.849
0.436
23.0
24.136
21.19
1.92
0.846
0.431
22.3
-SCL
21.702
22.45
1.55
0.848
0.436
23.1
24.424
20.92
1.93
0.844
0.429
22.2
Table 5: Ablation study on the WoW dataset. RReeeighting denotes removing the dynamic reweighting method compared to MACL. -TCL. denotes removing the token-level contrastive learning compared to MACL. NaiveSCL denotes using only from-batch negative samples in InfoNCE compared to MACL. -SCL denotes removing the sequence-level contrastive learning method compared to MACL..
resulting in contextually incoherent and unengag- ing responses. MACL, on the other hand, not only expresses its own opinion on the football player but also supplements relevant knowledge about him. The examples in Table 11 demonstrate that MACL's dynamic negative sampling mechanism reasonably selects negative candidates and penal- izes them without disrupting complete quotes. See the Appendix D for more cases and analyses.
Figure 4: Knowledge Precision Distribution of human responses (red) and MACL-generated responses (or- ange) on the WoW dataset.
Related Work
Knowledge-grounded dialogue generation With the increased functional demands for open-domain dialogue robots, many source-grounded dialogue generation tasks appear on researchers' radar (Wang et al., 2022; Lim et al., 2023; Zhang et al., 2018), 8), especially tions. The hot spot of research is mainly concen- trated on how to improve the performance of knowl- edge selection (Sun al., 2023; Xu al., 2022). Zhan al. (2021) proposed collaborative latent variable (CoLV) model to integrate the two sub- tasks simultaneously in collaborative latent spaces; Zhao et al. (2022) established a multi-reference KGDG dataset and devised a span-based varia- tional model; Yang et al. (2022) introduced the topic shift information into knowledge selection subtask.
effectiveness of MACL. Removing the dynamic reweighting factor results in a significant increase the perplexity metric, demonstrating that dy- namic weighting is indeed beneficial for mitigat- ing the inverse optimization problem. Token-level contrastive learning plays a crucial role in align- ing the model's 's ability to utilize knowledge with that of humans, as removing this loss function leads to significant degradation in the KUD met- ric. Both the token-level and sequence-level con- trastive learning functions are key in suppressing knowledge regurgitation, removing either of them results in an increase in the PoD metric. The per- formance improvement brought by sequence-level contrastive learning mainly stems from the selec- tion of hard negative examples. When only from- batch negatives used, the negative examples are easily distinguishable from the ground truth, and the model cannot effectively learn additional capabilities through the loss.
conversa-
Neural Text Degeneration Neural text degenera- tion refers the problem that the generated texts from the language model tend to be dull and in- coherent, contain undesirable repetitions at differ- ent levels (Holtzman et al., 2020; Li et al., 2020). The existing methods mainly alleviates this prob- lem from two aspects: decoding strategy and train- ing strategy (Su et al., 2022; Lagutin et al., 2021). Holtzman et al. (2020) found that the distribution of candidate tokens obtained by existing language models had unreliable long-tails, so they cut off them to reduce the occurrence incoherent se-
5.4
Case Study
To better evaluate the performance of response gen- eration, we selected examples generated by MLE, NT. Scalegrad, and MACL for comparison. In the example provided in Table 10, the baselines are unable to avoid the pattern of inserting snippets the provided knowledge into generic responses,

---

quences; Li et al. (2020) adjusted the negative train- ing method alleviate some generation problems open domain dialogue tasks like inconsistency and contradiction. Su et al. (2022) attributed degen- eration to the anisotropy of token representation in vector space, and designed a solution from both the training and decoding stage.
Ethics Statement
The benchmark dataset we used in our experiments. WoW (Dinan al., 2018), a well-regarded, open- source dataset collected by crowdsourced workers. was compiled with rigorous adherence to user pri- vacy protection protocols, ensuring the exclusion of any personal information. Furthermore, our pro- posed approach consciously upholds ethical stan- dards and societal fairness, ensuring that no preju- dice is introduced. For human evaluation com- ponent, all participants were volunteers who were provided with comprehensive information about the research's purpose, ensuring informed consent. In addition, all participants received fair and rea- sonable compensation for their contributions.
7 Conclusion
In this paper, we investigate a distinctive degener- ation phenomenon in Knowledge- Howwlege-Grounded logue Generation referred to Knowledge Regur- gitation, which prevalent in pre-trained language models. To address this challenge, we present novel solution called Multi-level Adaptive Con- trastive Learning (MACL). Our approach tackles the problem by dynamically sampling negative ex- amples and penalizing degeneration behaviors at both the token-level and sequence-level. Experi- ments on the WoW dataset demonstrate that our approach significantly mitigates knowledge regur- gitation.
Grounded Dia-
Acknowledgments
This work was supported by the National Natu- ral Science Foundation of China (No. 61976207), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Na- tional Social Science Foundation of China (No. 21AZD145).
Limitations
References
MACL effectively addresses the issue of Knowl- edge Regurgitation. However, we acknowledge certain limitations in our work:
Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. 2023. Cont: Con- trastive neural text generation.
(I) Due to limited computational resources, we have focused on demonstrating the effectiveness of our method on pre-trained language models with Encoder- structures that have less than 1 billion parameters. However, our method indeed address the degeneration problem in lightweight chit-chat models, and plan to explore degeneration in larger language models (LLMs) future work.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters: An imitation learning perspective of error accumu- lation language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 700-710, Dublin, Ireland. Association for Computational Linguistics.
Xiuyi Chen, Fandong Meng, Peng Li, Feilong Chen, Shuang Xu, Bo Xu, and Jie Zhou. 2020. Bridging the gap between prior and posterior knowledge lection for nowwledge-roundeed dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3426-3437, Online. Association for Computa- tional Linguistics.
(2) As existing datasets did not align with the specific scenario we wanted to explore, we solely evaluated our method on the WoW dataset. Al- though the dataset size may not be large enough, it provided valuable insights for our research.
(3) Our sequence-level contrastive loss involves generating negative examples during the training stage. which requires multiple calls the pre- trained language model calculate the hidden states of negative targets. We have not yet opti- mized our algorithm code to run in parallel, result- ing in a decrease in training speed.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge powered conversational agents. CoRR, abs/1811.01241.
Joseph L. Fleiss. 1971. Measuring nominal scale agrec- ment among many raters. Psychological Bulletin. 76:378-382.

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Xiang Lin, Simeng Han, and Shafiq Joty. 2021. Straight to the gradient: Learning to use novel tokens for neural text generation.
Shaojie Jiang, Thomas Wolf, Christof Monz, and Maarten de Rijke. 2020. Tldr: Token loss dynamic reweighting for reducing repetitive utterance genera- tion.
Chia- Wei Liu, Ryan Lowe, Tulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2122- 2132,, Austin, Texas, Association for Computational Linguistics.
Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. 2022. A simple contrastive learn- ing objective for alleviating neural text degeneration.
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential latent knowledge selec- tion for knowledge-grounde dialogue. CoRR, abs/2002.07510.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting back- ground knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2322-2332, Brussels, Belgium. Association for Com- putational Linguistics.
Diederik Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. International Conference Learning Representations.
Evgeny Lagutin, Daniil Gavrilov, and Pavel Kalaidin. 2021. Implicit unlikelihood training: Improving neu- ral text generation with reinforcement learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1432-1441, Online. Association for Computational Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Demoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward 7. Yang, Zach Vito, Martin Raison, Alykhan Tejami, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun- jie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high seht-pefforaece deep learning li- brary. CORR, abs/1912.01703.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open- domain chatbot.
Yiwei Li, Shaoxiong Feng, Bin Sun, and Kan Li. 2022. Diversifying neural dialogue generation via negative distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 407-418, Seattle, United States. Association for Computational Linguistics.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation.
Jung Hoon Lim, Myunghoon Kang, Yuna Hur, Seung-Ju Jung, Jinsung Kim. Yoonna Jang, Dongyub Lee, Hye- sung Ji, Donghoon Shin, Seung Wook Kim, and Heu- Jeoung Lim. 2023. You truly understand what need: Intellectual and friendly dialogue agents grounding knowledge and persona. ArXiv, abs/2301.02401.
Weiwei Sun, Pengjie Ren, and Zhaochun Ren. 2023. Generative knowledge selection for knowledge- grounded dialogues. In Findings of the Associa- tion for Computational Linguistics: EACL 2023, pages 2077-2088, Dubrovnik, Croatia. Association for Computational Linguistics.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, 2018. Focal loss for dense object detection.
Ashwin K Vijayakumar, Michael Cogswell, Ram- prasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruy Batra. 2018.
Diverse beam

---

Decoding diverse solutions from neural se- quence models.
search:
Xueliang Zhao, Tingchen Fu, Chongyang Tao, and Rui Yan. 2022. There is no standard answer: Knowledge- grounded dialogue generation with adversarial acti- vated multi-reference learning. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Lunguage Processing, pages 1878-1891, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Lanrui Wang, JiangNan Li. Zheng Lin, Fandong Meng. Chenxu Yang, Weiping Wang. and Jie Zhou. 2022. Empathetic dialogue generation via sensitive emo- tion recognition and sensible knowledge selection. In Conference on Empirical Methods Natural Lan- guage Processing.
Chujic Zheng, Yumbo Cao, Daxin Jiang, and Minlic Huang. 2020. Difference-awar knowledge selection for knowledge-grounded conversation generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 115-125, Online. As- sociation for Computational Linguisties.
Sean Welleck, Mia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2019. Neu- ral text generation with unlikelihood training.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018, dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Process- ing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
A Details of Dataset
The WoW data sourced from a crowdsoureing website. During data collection, the user side plays the role of the apprentice, and the agent side plays the role of the wizard. The wizard has access to the knowledge retrieved from Wikipedia as ground- source to generate informative responses, while the apprentice prefers speaking common utterances, The WoW dataset consists of 22,311 dialogues with 201, 199, turns, which are divided into training set, validation set, test seen set, and test unseen set. The topics in the test unseen set are ones that never appeared in the training set.
Lin Xu, Qixian Zhou, Jinlan Fu, Min-Yen Kan, and See-Kiong Ng. 2022. CoreiDiffs: Co-referential and differential knowledge flow in document grounded conversations. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 47 1-484, Gyeongju, Republic of Korea, Inter- national Committee on Computational Linguistics.
Chenxu Yang, Zheng Lin, Jiangnan Li, Fandong Meng, Weiping Wang. Lanrui Wang, and Jie Zhou. 2022. TAKE: Topic-shift aware knowledge sElection for dialogue generation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 253-265, Gyeongju, Republic of Korea. Inter- national Committee on Computational Linguistics.
B Problem of Negative Training (NT)
Haolan Zhan, Lei Shen, Hongshen Chen, and Hainan Zhang. 2021. CoLV: A collaborative latent variable model for knowledge- grouudded dialogue generation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2250-2261, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
As the gradient-based optimization progresses, it is expected that the gradient of the loss approaches zero around minimum. Therefore, the probabil- ity of the ground-truth token p; should increase towards to decrease the gradient norm Ln and achieve convergence to a value close to during the training stage.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
However, in equation (14), when P: > T the value of the gradient norm VL becomes larger than 1. Consequently, the training procedure re- duces Pt to minimize the gradient norm, which contradicts the optimization principle. This issue prevents the loss function from converging during the later stages of training.
Yizhe Zhang, Siqi Sun. Michel Galley, Ven-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre- pre-training for conversational response generation.
Ground- Truth Token (i=1")=
0L Op:
Pc
(14)
-p a V.,, Op, Oa.
#1 Pe

---

C More Experimental Results
PoD
Dup-16
Dup-32
PL.CS
mKP-1
WoW
5.54%
6.53%
2.37%
24.47%
47.83%
As observed in Table 8, MACL works on the T5- large as well, effectively reducing the knowledge regurgitation. Furthermore, in addition to the beam search decoding strategy with beam size of also experimented with beam search using beam size 5, greedy decoding, and Nucleus Sampling. The results in Table 9 show that larger beam sizes are more prone to degeneration. Although nucleus sampling could alleviate knowledge regurgitation, it is far less effective than MACL. It also illus- trates that our approach is compatible with these decoding strategies it mitigates the degeneration further.
Holl-E
73.43%
75.68%
70.86%
78.91
82.36%
Table 6: Comparison between the WoW dataset and the Holl-E dataset.
PPL
PoD(%)
KUD
Avg.
Ext.
BLEU-1
MLE
2.151
67.34%
0.64
0.890
0.599
69.2
NT
2.214
61.77%
0.89
0.887
0.593
70.4
ND
3.576
58.21%
1.12
0.889
0.595
68.8
CTloss
2,653
66.34%
0.70
0.887
0.594
69.1
SimCTG
2.218
68.622%
0.61
0.888
0.595
69.5
Scalegrad
2.334
59. 98%
0.96
0.891
0.598
70.7
MACL
2.117
49.63%
1.50
0.889
0.595
71.3
Table Automatic Evaluation results the Holl-E dataset (BART-arge), The best results are highlighted with bold.
More Cases
replication phenomenon (-17.71%). However, the dataset remains unsuitable for evaluating the knowl- edge regurgitation problem in terms of the results of the other metrics. All the baseline methods achieved notably low perplexity and remarkably high BLEU-1 score. It indicates the excessive simi- larity between knowledge input and generation tar- gets. Besides, the generated response with less se- vere knowledge replication results in broader dis- crepancy in knowledge utilization capacity (higher KUD value). The phenomenon illustrates problem- atic ground-truth responses.
In the example provided in Table 12, MACL effec- tively incorporates the knowledge about the movie "Blue Skies" into response. It starts by ac- knowledging that there are indeed beautiful views and blue skies along the route of the Royal Blue train, and then proceeds to mention relevant content about the movie with the same name. On the other hand, the baselines fail to internalize the knowl- edge and generate responses that are incoherent and lack interactivity.
E Details of the Training Algorithm
See the pseudo-code below for details.
F
More results on the Holl-E dataset
The WoW dataset well collected, with a spe- cific focus on engagingness and interactiveness. The collectors crafted responses by integrating grounded knowledge naturally, and they were for- bidden to duplicate knowledge snippets for saving time. Compared to that, the collection of Holl- E dataset is relatively rough. There apparent replication between the ground-truth response and the introduced knowledge in it (which means the ground-trute response is not a ideal response with knowledge internaiization, it is unsuitable for eval- uating the prevention of knowledge regurgitation. sometimes takes movie comments as both external knowledge (input) and ground-truth responses (tar- get) during data collection, leading problematic ground-truth responses. The comparison between the datasets are shown in Table 6.
The experimental results on the PoD metric show that MACL still effectively mitigate knowledge

---

Algorithm MACL Training algorithm: Given a trained language model M; return a finetuned model M
dialogue dataset (X,)), pre-
I: procedure TRAINDEGENEEATORRRE
2: Update the parameters of M with VICMLE until convergence and Get the degenerator M
3: return Mee
4: procedure NEGATIVESAMPLINGLA (x, y), m)
group beam search algorithm 6: y (1:m) f- Do oracle mcasurement o(y(1:b), xx) for each element and get the top m samples
5: ((L:b) f- Do BeamSearch to get samples
7: return y (1:m)
8: procedure TRAIN(M, (2))
9:
+ Parameters M,, 6 + beam size
10:
Ma = -TainnDegenerato((MM))
> the degenerator frozen since now
11:
while not convergence do
(x(1:)), (lik)) <- A minibatch of k: datapoints from () Ltoken t Get token-level contrastive loss for (x(1:)), (Lik))
12:
13:
y (1:k;11m)) NegativeSampling (Mae. (x(1=k) (1(kk))), b)
y (1:8:11b+m) Append degeneration samples y (1:b)
Exeg + Get sequence-level contrastive loss for (x(11kk) y (ldd:)kt)))
17:
update parameters using (O(tkeen + >Leeq)
18:
return .M'
Test Seen
Test Unseen
ppl
PoD(%)
KUD
Avg.
Ext.
BLEU-1
ppl
PoD(%)
KUD
Avg.
Ext.
BLEU-1
MLE
15.018
69.54
8.05
0.843
0.426
23.1
16.773
71.00
8.19
0.836
0.419
21.9
NT
14,890
36.94
5.95
0.851
0.434
22.7
16.569
38.38
6.13
0.844
0.428
21.7
ND
33.653
32.58
5.34
0.842
0.428
21.0
47,931
34.55
5.28
0.841
0.419
20.4
CTloss
17.063
37.56
5.29
0.839
0.431
23.3
18.788
35.92
5.12
0.838
0.425
22.2
SimCTG
15.420
66.68
7.71
0.843
0.426
23.2
17.251
68.6
8.05
0,836
0.419
22.1
Scalegrad
15.089
36.97
4.72
0.841
0.427
22.4
16.863
36.62
8.28
0.834
0.420
21.8
MACI.
13.534
10.93
0.95
0.855
0.439
22.5
14.565
11.82
1.36
0.848
0.435
22.0
Table 8: Automatic Evaluation results on the WoW dataset (T5-large).
decoding
Test Seen
Test Unseen
method
strategy
Pol)(S)
KUD)
Avg.
Ext.
BLEU-1
ppl
Pol3(1))
KUD
Avg.
Exi.
BLEU-T
Beam
MLE
23.784
62.86
7.40
0.839
0.425
22.9
26.146
64.11
7.76
0.835
0.418
21.7
Search
MACI.
20.897
11.74
1.22
0.848
0.437
22.5
23.973
13.01
1.76
0.846
0.434
22.0
Greedy
MLE
23.784
43.03
5.92
0.850
0.437
23.8
26.146
45.21
6.12
0.844
0.430
23.4
Search
MACI.
20.897
3.32
0.851
0.437
23.973
4.41
1.71
0.841
0.433
20.8
Nucleus
20.03
0.427
26.146
3.08
0.845
Sampling
20.897
1.54
0.75
0.847
0.423
21.3
23.973
2.37
0.59
0.846
0.425
20.9
Table 9: Automatic Evaluation results on the WoW dataset across various decoding strategies (BART-large). The beam size of the beam search set and the pre-chosen threshold p for nucleus sampling is set to 0.9.

---

Context
Thierry Henry is one of my all time favorite players. What about you?
Thierry Daniel Henry (bom August 1977) is retired French professional foothaller who played as forward and the second assistant manager of the Belgium national team. He was good. He is retired French professional foothaller.
Knowledge
GT
Yes, retired French professional footballer who played as forward and
MLE
now he is the second ussistant manager of the Belgium national leam.
Yes, he great player. He is now the second assistant manager of the Belgium national team.
NT
Scalegrad MACL
Yes, he is a retired French professional footballer.
I love him too! He was great forward born August 17th, 1977!
Table 10: An example for case study.
(1) I am retired now. Are you familiar with the work of Mehr and Meyer, well known psychologists?
Context
(2) I am not. Could you tell me about them?
According Maehr and Meyer, "Motivation is word that part
Knowledge
the popular culture as few other psychological concepts are."
They said that "Motivation a word that is part of the popular culture few other psychological concepts are." Do you agree with what they said? Well, according to Maehr and Meyer, "Motivation the popular culture as few other psychological concepts are."
GT
word that is part of
MLE
Well according to Machr and Meyer, "Motivation is a word that is part of the popular culture as few other psychological concepts are."
NT
Machr and Meyer are psychologists. They argue that motivation is a word that part of the popular culture as few other psychological concepts are. They say that "Motivation word that is part of the popular culture few other psychological concepts are."
Scalegrad
MACL
Table 11: An example for case study.
(1) Blue always nice. I like royal blue.
Context
(2) once road on The Royal Blue train from New York to D.C.
(3) Oh that sounds really nice. bet there was a lot of scenery and blue skics.
Blue Skies is a 1946 American musical comedy film directed by
Knowledge
Stuart Heisler and starring Bing Crosby, Fred Astaire, and Joan Caulfield.
GT
Yes, speaking of Blue Skies, have you seen the 1946 movic staring Bing Crosby? It was a 1946 American musical comedy film directed by Stuart Heisler and starring Bing Crosby, Fred Astaire, and Joan Caulfield.
MLE
NT
bet there were lots of blue skies in that movie with Bing Crosby and Fred Astaire. Yes, there was a lot of scenery and blue skies in the movic Blue Skies which was released in 1946.
Scalegrad
MACL
Yes it was! And Blue Skies was also a comedy film directed by Stuart Heisler!
Table 12: example for case study.


Reviews:
[{"review_id": "W9ftkE4mJm", "reviewer": "Reviewer_Q1NT", "summary": "", "questions": "1) Can you provide more details on the implementation of the MACL framework, including the specific architecture, hyperparameters, and training process? \n\n2) How knowledge selection is handled in experiments with MACL and other baseline models.\n\n3) Experience has found that large language models do not suffer from such severe replication problems; is the knowledge echo problem mentioned by the authors only a problem with the WoW dataset?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review! We are glad that you believe our approach is \"a comprehensive solution\", that you thought the problem we explored is \"an important problem\", etc. Your thoughtful and valuable suggestions will be incorporated in the future version of our paper. The questions and problems will be properly addressed as we reply.\n\n**Q1: Clarifying our choice of Dataset**\n\n**A1:** Thank you for your constructive comment. We have implemented experiments on the Holl-e dataset to validate the problem we addressed is a generic problem instead of a flaw in the WOW dataset. We will include them in the appendix of a subsequent version of the paper to showcase the extensibility and generalization of the knowledge regurgitation issue as you suggested.\n\n| Holl-e (Bart-large) | PPL   | PoD    | KUD  | Avg.  | Ext.  | BLEU-1 |\n| ------------------- | ----- | ------ | ---- | ----- | ----- | ------ |\n| MLE                 | 2.151 | 67.34% | 0.64 | 0.890 | 0.599 | 69.2   |\n| NT                  | 2.214 | 61.77% | 0.89 | 0.887 | 0.593 | 70.4   |\n| ND                  | 3.576 | 58.21% | 1.12 | 0.889 | 0.595 | 68.8   |\n| CTloss              | 2.653 | 66.34% | 0.70 | 0.887 | 0.594 | 69.1   |\n| SimCTG              | 2.218 | 68.62% | 0.61 | 0.888 | 0.595 | 69.5   |\n| Scalegrad           | 2.334 | 59.98% | 0.96 | 0.891 | 0.598 | 70.7   |\n| MACL                | 2.117 | 49.63% | 1.50 | 0.889 | 0.595 | 71.3   |\n\nThe experimental results on the PoD metric show that MACL still effectively mitigate knowledge replication phenomenon (-17.71%). However, the dataset remains unsuitable for evaluating the knowledge regurgitation problem in terms of the results of the other metrics.\nAll the baseline methods achieved notably low perplexity and remarkably high BLEU-1 score. It indicates the excessive similarity between knowledge input and generation targets. Besides, the generated response with less severe knowledge replication results in a broader discrepancy in knowledge utilization capacity (higher KUD value). **The phenomenon illustrates problematic ground-truth responses.**\n\nNow we explain our choices in detail. Basically, there are three widely used KGDG datasets: WoW[1], CMU_DoG[2], and Holl-e[3]. The reason we only present the WoW dataset results is not only this dataset \"meet our requirements\". **Instead, we chose this dataset due to its superior quality and knowledge diversity. The WoW dataset is well collected, with a specific focus on engagingness and interactiveness**. The collectors crafted responses by integrating grounded knowledge naturally, and they were forbidden to duplicate knowledge snippets for saving time.\n\nWe did not adopt the Holl-E and CMU_DoG datasets  for evaluation because **there is apparent replication between the ground-truth response and the introduced knowledge in them (which means the ground-truth response is not a ideal response with knowledge internalization), they are unsuitable for evaluating the prevention of knowledge regurgitation.**\nThe two datasets take movie comments as both external knowledge (input) and ground-truth responses (target) during data collection, leading to problematic ground-truth responses.\n\nRegarding CMU_DoG, it is even worse. The dataset did not label golden knowledge, so there is no way to tell if the knowledge introduced is correct. We investigate how to prevent knowledge regurgitation under the assumption that knowledge has been correctly selected. Therefore, we were unable to complete the experiment on this dataset.\n\nDue to page limits, we did not elaborate on the above reasons supporting our dataset selection within the paper. We plan to supplement them in the forthcoming edition. \n\nSome of the low-quality data:\n\n```\ntr_3_0 -> tr_3_1\n\nQuery: Which scene did you like the most in the movie?\nKnowledge: I liked the one which the old man has a heart attack during a KY jelly wrestling match..\nResponse: I liked the one which the old man has a heart attack during a KY jelly wrestling match\n\ntr_4_0 -> tr_4_1\n\nQuery: What was your favorite scene in this movie?\nKnowledge: I liked the one which Jack is trolley cart while the bus was speeding so he could diffuse the bomb. It was exciting..\nResponse: I liked the one which Jack is trolley cart while the bus was speeding so he could diffuse the bomb. It was exciting\n```\n\nThe following table compares the quality of ground-truth responses in both the Holl-e and WoW datasets, evaluated on metrics concerning knowledge regurgitation.\n\n|        | PoD    | Dup-16 | Dup-32 | PLCS   | mKP-1  |\n| ------ | ------ | ------ | ------ | ------ | ------ |\n| WoW    | 5.54%  | 6.53%  | 2.37%  | 24.47% | 47.83% |\n| Holl-e | 73.43% | 75.68% | 70.86% | 78.91% | 82.36% |\n\n**Q2: Clarifying the Fairness of comparison**\n\n**A2:** Your next concern is about whether the knowledge is fairly provided. For all approaches, golden knowledge is used in the training stage and a SoTA knowledge selector is used in the inference stage to select the knowledge for generating response. **The knowledge selection and other settings are the same, so we believe our comparison is fair.**\n\n**Q3: Does LLM suffers from knowledge regurgitation issue as well?**\n\n**A3:** large language models also suffer from replication problems in the KGDG scenario. Regarding the open-source fine-tuned LLM, we carried out fine-tuning of the llama[4] model with lora[5]. In future endeavors, we intend to undertake more extensive experiments across various open-source LLMs.\n\n| WoW Test Seen | PoD    | Dup-16 | Dup-32 | PLCS  | KUD  | mKP-1 |\n| ------------- | ------ | ------ | ------ | ----- | ---- | ----- |\n| llama-MLE     | 26.45% | 34.21  | 37.34  | 44.09 | 3.97 | 60.56 |\n| human     | - | 6.53  | 2.37  | 24.47 | - | 47.83 |\n\n**The experimental results indicate that a noticeable gap still exists between the llama-generated responses and human-annotated responses.** (Almost **5 times worse** on the Dup-16 metric and **15 times worse** on the Dup-32 metric)\n\nIn the case of closed-source LLMs such as ChatGPT, we conducted tests using zero-shot prompting with provided instructions. Though the replication issue is largely mitigated, its outputs are excessively lengthy, which doesn't align with casual chitchat scenarios. Consequently, we assert that the problem of knowledge regurgitation remains a challenge.\n\n**Q4: Some implementation details missing in the paper**\n\n**A4:** Regarding model architecture, we embrace the original BART and T5 (encoder-decoder architecture). Our primary emphasis lies in training loss design and negative sample collection, without introducing new modules.\n\nWe have provided all the hyperparameter configurations in L383 of the paper and the Supplementary Materials (our codes). We are committed to releasing our code on GitHub upon publication. Please let me know what other hyperparameters you'd like to know about. We will supplement them then.\n\n```\nlearning rate: 1e-5, inference batch size: 32, training scheduler: linear_schedule_with_warmup, warmup_ratio: 0.2, max length of dialogue context: 64, max length of knowledge sentence: 64, max length of response during training stage: 64. Some generation configs: max_new_tokens: 60, min_length: 10.\n```\n\nAs for the training process, we have provided detailed steps in algorithm 1 in the Appendix. You can learn more about it there.\n\n\n[1] ICLR2019 Wizard of Wikipedia: Knowledge-powered conversational agents.\n\n[2] EMNLP2018 A dataset for document grounded conversations. \n\n[3] EMNLP2018 Towards exploiting background knowledge for building conversation systems.\n\n[4] LLaMA: Open and Efficient Foundation Language Models. 2023.\n\n[5] LoRA: Low-Rank Adaptation of Large Language Models. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper introduces an important problem in knowledge-grounded dialogue generation, which is the issue of knowledge regurgitation, and proposes a approach to mitigate this problem.\n\n2) The proposed MACL framework includes both token-level and sequence-level contrastive learning, providing a comprehensive solution to address degeneration behaviors at different levels.\n\n3) The paper provides a thorough evaluation of the proposed approach, comparing it against several baselines and conducting both automatic and human evaluations. The results show significant improvements in various metrics, demonstrating the effectiveness of MACL.", "reasons_to_reject": "1) The paper lacks implementation details, which may hinder the reproducibility of the proposed approach. More information on the specific architecture, hyperparameters, and training process would be helpful.\n\n2) The proposed method was validated on only one dataset (WoW) and the authors also mention in the data and limitations that only one dataset meets their requirements, does it prove that this is not a generic problem but a flaw in the WOW dataset？\n\n3) The authors did not focus on knowledge selection and did not mention if golden knowledge was used for training. This may be unfair to compare with other baseline models.", "questions_to_authors": "1) Can you provide more details on the implementation of the MACL framework, including the specific architecture, hyperparameters, and training process? \n\n2) How knowledge selection is handled in experiments with MACL and other baseline models.\n\n3) Experience has found that large language models do not suffer from such severe replication problems; is the knowledge echo problem mentioned by the authors only a problem with the WoW dataset?", "paper_topic_and_main_contributions": "The paper proposes a Multi-level Adaptive Contrastive Learning (MACL) to address the problem of knowledge regurgitation in knowledge-grounded dialogue generation. The authors first identify the issue of copying-style degeneration in current models, where the model simply duplicates knowledge segments without genuine comprehension. They then propose MACL, which includes both token-level and sequence-level contrastive learning to penalize degeneration behaviors and improve knowledge utilization in the generated responses. The proposed approach is evaluated on the WoW dataset and compared against several baselines, showing significant improvements in terms of knowledge regurgitation, knowledge utilization, and content quality.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQVXzb5zii", "reviewer": "Reviewer_XHDH", "summary": "", "questions": "A. Why do you mention Eq.(9)? Seems that it is not used in the MACL method. According to my understanding, it is for different weighting on the loss of different tokens. But such an idea is very straightforward which somehow makes this part redundant. Besides, what does p_t mean in Eq.(9)?\n\nB. How do you obtain z_x and z_y and make them differentiable in the whole procedure? Are they the representations of some special tokens in pre-trained LMs?\n\nC. In L254-257, do C^t and C_t represent the same thing? \"where p_c is prediction probability of C_t”. So what is p_c? According to my understanding, C_t is a set and p_c is a scalar. Maybe p_c is the probability of y_t?\n\nD. In Table 4, how do human annotators make these judgments? Could you provide more details like the instruction and definition?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the useful and constructive feedback! We are glad that you liked many aspects of our paper (thinking that it is well-motivated, that our method MACL is reasonable, etc.). We have fixed the typos and grammatical errors you mentioned in the updated version of the paper following your valuable advice. We hope the following responses will adequately address your concerns.\n\n**Q1: Clarifying our choice of Dataset**\n\n**A1:** \nThank you for your constructive comment. We have implemented experiments on the Holl-e dataset to validate the extensibility and generalization of the problem we addressed. We will include them in the appendix of a subsequent version of the paper to showcase the extensibility and generalization of the knowledge regurgitation issue as you suggested.\n\n| Holl-e (Bart-large) | PPL   | PoD    | KUD  | Avg.  | Ext.  | BLEU-1 |\n| ------------------- | ----- | ------ | ---- | ----- | ----- | ------ |\n| MLE                 | 2.151 | 67.34% | 0.64 | 0.890 | 0.599 | 69.2   |\n| NT                  | 2.214 | 61.77% | 0.89 | 0.887 | 0.593 | 70.4   |\n| ND                  | 3.576 | 58.21% | 1.12 | 0.889 | 0.595 | 68.8   |\n| CTloss              | 2.653 | 66.34% | 0.70 | 0.887 | 0.594 | 69.1   |\n| SimCTG              | 2.218 | 68.62% | 0.61 | 0.888 | 0.595 | 69.5   |\n| Scalegrad           | 2.334 | 59.98% | 0.96 | 0.891 | 0.598 | 70.7   |\n| MACL                | 2.117 | 49.63% | 1.50 | 0.889 | 0.595 | 71.3   |\n\n**The experimental results on the PoD metric show that MACL still effectively mitigate knowledge replication phenomenon (-17.71%)**. However, the dataset remains unsuitable for evaluating the knowledge regurgitation problem in terms of the results of the other metrics. All the baseline methods achieved notably low perplexity and remarkably high BLEU-1 score. It indicates the excessive similarity between knowledge input and generation targets. Besides, the generated response with less severe knowledge replication results in a broader discrepancy in knowledge utilization capacity (higher KUD value). The phenomenon illustrates problematic ground-truth responses.\n\nNow we explain our choices in detail. Basically, there are three widely used KGDG datasets: WoW[1], CMU_DoG[2], and Holl-e[3]. The reason we only present the WoW dataset results is not only this dataset \"meet our requirements\". Instead, we chose this dataset due to its superior quality and knowledge diversity. The WoW dataset is well collected, with a specific focus on engagingness and interactiveness. The collectors crafted responses by integrating grounded knowledge naturally, and they were forbidden to duplicate knowledge snippets for saving time.\n\nWe did not adopt the Holl-E and CMU_DoG datasets for evaluation because **there is apparent replication between the ground-truth response and the introduced knowledge in Holl-E and CMU_DoG datasets (which means the ground-truth response is not a ideal response with knowledge internalization), they are unsuitable for evaluating the prevention of knowledge regurgitation**. The two datasets take movie comments as both external knowledge (input) and ground-truth responses (target) during data collection, leading to problematic ground-truth responses.\n\nRegarding CMU_DoG, it is even worse. The dataset did not label golden knowledge, so there is no way to tell if the knowledge introduced is correct. We investigate how to prevent knowledge regurgitation under the assumption that knowledge has been correctly selected. Therefore, we were unable to complete the experiment on this dataset.\n\nDue to page limits, we did not elaborate on the above reasons supporting our dataset selection within the paper. We plan to supplement them in the forthcoming edition.\n\n**Q2: Partial LLM results**\n\n**A2:** We only carried out fine-tuning of the llama[4] model with lora[5] subject to rebuttal time limits. **The experimental results indicate that LLMs suffer from the knowledge regurgitation issue similarly, with a noticeable gap persists between the generated response and human-annotated responses** (Almost **5 times worse** on the Dup-16 metric and **15 times worse** on the Dup-32 metric). In future endeavors, we intend to undertake more extensive experiments across various open-source LLMs.\n\n| WoW Test Seen | PoD    | Dup-16 | Dup-32 | PLCS  | KUD  | mKP-1 |\n| ------------- | ------ | ------ | ------ | ----- | ---- | ----- |\n| llama-MLE     | 26.45% | 34.21  | 37.34  | 44.09 | 3.97 | 60.56 |\n| human     | - | 6.53  | 2.37  | 24.47 | - | 47.83 |\n\n**Q3: personalized dialogue generation**\n\n**A3:** Regarding personalized dialogue generation, we concur with your observation that replication issues could also be present. We intend to carry out experiments using the **Persona-Chat** dataset and incorporate their results in the forthcoming version of the paper.\n\n**Q4: Responses for \"Questions For The Authors\"** \n\n**A4:**  \n\n**A**. Eq(9) is introduced to illustrate that our approach draws inspiration from focal loss. As you rightly noted, this might lead to some redundancy in processing. Therefore, we plan to remove Eq(9) in subsequent versions. We appreciate your suggestion. Here, p_t signifies the predicted probability of the next target token.\n\n**B**. The caption of Figure 3 explicitly states that z_x and z_y are derived through mean pooling from sentence representations, and they retain their differentiability in this context.\n\n**C**. Indeed, C^t and C_t refer to the same entity. Additionally, p_c denotes the probability of y_t.\n\n**D**. Instructions: A generated response will be classified as degenerated if it **1) evidently replicates external knowledge and 2) produces content that is visibly unnatural and contextually incoherent**.\n\nGiven that automated metrics can only assess condition 1), we augmented the evaluation with human judgement. The results indicate the merit of the automated PoD metric.\n\nWe sincerely appreciate your opinion on Typos Grammar Style And Presentation Improvements.\n\n\n[1] ICLR2019 Wizard of Wikipedia: Knowledge-powered conversational agents.\n\n[2] EMNLP2018 A dataset for document grounded conversations. \n\n[3] EMNLP2018 Towards exploiting background knowledge for building conversation systems.\n\n[4] LLaMA: Open and Efficient Foundation Language Models. 2023.\n\n[5] LoRA: Low-Rank Adaptation of Large Language Models. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-motivated. Directly copying text pieces from the original knowledge, rather than making genuine comprehension and proper paraphrasing, is a common shortcut learned by LMs which degenerates the generated responses. This paper not only mentions this issue but also verifies its existence using experiments on multiple general pre-trained LMs, along with pre-defined metrics.\n\n2. The proposed method, MACL, is also reasonable as it is inspired by the observed phenomena. The additional token-level unlikelihood loss can prevent it from directly copying text pieces from the original knowledge context, while the dynamic weights can pose heavier weights on unexpected tokens with higher possibilities. On the other hand, sentence-level contrastive learning loss can further make the generated responses more dissimilar to the bad cases.", "reasons_to_reject": "1. The experiments can be further extended to make the paper more solid. 1) Only one dataset is included, which means its extensibility and generalization remain unverified. Maybe authors can add the experimental results on the rest two datasets mentioned in Section 5.1. 2) Only relatively smaller pre-trained LMs are involved. Since LLMs are hot topics in recent research, including LLMs and illustrating their performance can promote the value of this paper. 3) I think such a degeneration also appears in personalized dialogue generation, it would be better if related content can be included. ", "questions_to_authors": "A. Why do you mention Eq.(9)? Seems that it is not used in the MACL method. According to my understanding, it is for different weighting on the loss of different tokens. But such an idea is very straightforward which somehow makes this part redundant. Besides, what does p_t mean in Eq.(9)?\n\nB. How do you obtain z_x and z_y and make them differentiable in the whole procedure? Are they the representations of some special tokens in pre-trained LMs?\n\nC. In L254-257, do C^t and C_t represent the same thing? \"where p_c is prediction probability of C_t”. So what is p_c? According to my understanding, C_t is a set and p_c is a scalar. Maybe p_c is the probability of y_t?\n\nD. In Table 4, how do human annotators make these judgments? Could you provide more details like the instruction and definition?", "paper_topic_and_main_contributions": "This paper focuses on knowledge-grounded dialogue generation scenarios. \n\nIt first examines the \"Knowledge Regurgitation\" phenomena that exist in many LMs, where the generation results usually have the same substrings as the given knowledge. Such a copy mechanism makes responses more dull and more unnatural. \n\nTo tackle this issue, authors propose Multi-level Adaptive Contrastive Learning (MACL). It introduces negative samples that contain noticeable overlaps with the knowledge texts, then applies both token-level and sentence-level contrastive learning along with dynamic loss weights to fine-tune the pre-trained LM. \n\nExperiments are conducted on the WoW dataset, where MACL achieves similar n-gram accuracy as the baselines, while a significant improvement on the degeneration of knowledge text pieces .", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I did not carefully check the typos and may miss some of them.\n\nL68: \"mDup and KPDD-n\" only appear here and never appear later, maybe typos.\n\nL569: alleviates -> alleviate\n\nMissing articles, like L257 \"prediction\" and L595 \"degeneration\" and so on.\n\n"}}, {"review_id": "JQdu1NXQAJ", "reviewer": "Reviewer_pQyx", "summary": "", "questions": "(a) Do the same knowledge regurgitation problems occur if the model is trained on predicted knowledge from a knowledge selection module, rather than the golden knowledge?\n\n(b) Can you motivate why no knowledge-grounded dialogue baselines are included (or clarify which baselines count as knowledge-grounded dialogue baselines)?  \n\nPost-rebuttal edit:\nIn light of the significant additional work conducted by the authors in response to my questions/concerns, I have moved my soundness rating to a 4. To clarify my point around end-to-end methods being less likely to regurgitate knowledge, I had in mind the architectures that treat the knowledge as a latent variable to be marginalised over [1] [2]--here, the generator is trained on the retriever's distribution, rather than the golden knowledge.   \n\n1) Shuster, Kurt et al. “Retrieval Augmentation Reduces Hallucination in Conversation.” Conference on Empirical Methods in Natural Language Processing (2021).\n\n2) Zhang, Yizhe et al. “RetGen: A Joint Framework for Retrieval and Grounded Text Generation Modeling.” AAAI Conference on Artificial Intelligence (2021).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful review! We are glad that you believe our approach \"is quite novel\", that you thought the problem we explored was \"an interesting problem\", etc. We hope our response could alleviate your concerns and your valuable advice will be fully\nincorporated in the final version.\n\n**Q1: Additional experiment on training with predicted knowledge**\n\n**A1:** Thank you for your insightful questions. Your suggestion of employing predictive knowledge for training the response generator is very interesting. We conducted the experiment as you proposed.  But here we must correct your misconception about end-to-end training. Indeed, \"many SoTA architecture train both models jointly end-to-end\". This does not indicate that the RG module was trained on predicted knowledge if you check their source codes. **Basically, the RG module was also trained on golden knowledge in the end-to-end scenerio**[1][2][3][4][5].\n\n| Bart-large      | PPL    | PoD    | KUD  | Avg.  | Ext.  | BLEU-1 |\n| --------------- | ------ | ------ | ---- | ----- | ----- | ------ |\n| WoW test seen   | 24.871 | 27.61% | 5.13 | 0.842 | 0.428 | 22.3   |\n| WoW test unseen | 27.208 | 29.34% | 5.45 | 0.838 | 0.423 | 21.4   |\n\nAlthough knowledge regurgitation issue was somewhat mitigated, it didn't work out so well. Besides, **other metrics (including BLEU, perplexity, Avg., and Ext.) exhibited a corresponding decline**. By comparison, the MACL approach proved more effectual in mitigating knowledge regurgitation, without abandoning the model's capacity for knowledge integration. We do not think training with predicted knowledge is a solution for knowledge regurgitation for two reasons.\n\n**1)** By introducing some noises, the model avoids excessive reliance on external knowledge. However, this approach comes with drawbacks too: **the model's capacity to integrate knowledge is compromised, resulting in a notable reduction in the generated responses' knowledgeability**.\n\n**2)** As you rightly pointed out, the accuracy of the knowledge selector is far from 100%. However, it may approach 100% when better methods are proposed, and the issue of knowledge regurgitation will persist. Thus, we maintain that our work retains its intrinsic value.\n\n**We believe that the intrinsic reason for knowledge regurgitation is the ineffective design of the MLE training objective**. The model continued to converge towards superficial pattern matching based on overlap, as the selected knowledge still shares spurious correlations with ground-truth responses.\n\nNevertheless, we sincerely appreciate your question(ii) as lacking corresponding experiments and explanations, our paper is not solid enough.\n\n| WoW Test Seen(Bart-large) | PPL    | PoD    | KUD  | Avg.  | Ext.  | BLEU-1 |\n| ------------------------- | ------ | ------ | ---- | ----- | ----- | ------ |\n| TAKE-Bart                 | 24.871 | 27.61% | 5.13 | 0.842 | 0.428 | 22.3   |\n\n\n| WoW Test Unseen(Bart-large) | PPL    | PoD    | KUD  | Avg.  | Ext.  | BLEU-1 |\n| --------------------------- | ------ | ------ | ---- | ----- | ----- | ------ |\n| TAKE-Bart                   | 27.208 | 29.34% | 5.45 | 0.838 | 0.423 | 21.4   |\n\n\n\n**Q2: Clarifying our choice of Baselines**\n\n**A2:** **Although the baselines are all chosen from text degeneration, we concatenated user's query with golden knowledge in the input before response generation. We believe this treatment is consistent with the KGDG generation process.** In this setting, we denoted the MLE approach as the KGDG baselines because the optimization for response generators of the KGDG baselines relies on MLE loss functions.\n\n**Q3: Clarity improvement**\n\n**A3:** Thanks for your valuable suggestions aimed at enhancing the clarity of Section 4. The β in these places refers to the same parameter. While Eq(9) has been introduced to underscore the inspiration drawn from focal loss in our method, it does introduce a degree of redundancy. Therefore, we plan to remove Eq(9) in subsequent versions. Additionally, $L_{seq}$ is derived through the improvement of InfoNCE loss, a widely employed contrastive loss. We are committed to modifying them in the upcoming version of the manuscript.\n\n[1] Initiative-Aware Self-Supervised Learning for Knowledge-Grounded Conversations. 2021.\n\n[2] Sequential latent knowledge selection for knowledge-grounded dialogue. 2020.\n\n[3] Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation. 2020\n\n[4] DukeNet: A Dual Knowledge Interaction Network for Knowledge-Grounded Conversation. 2020\n\n[5] TAKE: Topic-shift Aware Knowledge sElection for Dialogue Generation. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(i) The paper identifies an interesting problem--namely, models learning to trivially copy knowledge into the response. This strategy can lead to models performing well in most automated metrics (e.g. PPL, bleu etc.) which can make it hard to identify with current metrics. The authors present a range of diversity metrics which show significant differences between the level of copying used by humans versus models.\n\n(ii) The authors support their claims across a range of automated and human metrics and ablations.\n\n(iii) The application of text degeneration techniques to knowledge-grounded dialogue is quite novel, and both the token-level and sequence-level contrastive learning each seem to provide a sizeable improvement to performance according to the ablation.", "reasons_to_reject": "(i) Section 4 could benefit from additional clarity, especially the part on token-level contrastive learning. It isn't clear whether the β(y_c) in equation 8 is the same β as in line 285 or equation 10. It also isn't clear how TFL fits into the loss function, as the only equation it appears in is equation 9. Also, InfoNCE loss is never fully explained--I assume it refers to Lseq, but then I'm not sure of the motivation behind having a separate term for it.\n\n(ii) The paper treats knowledge selection (KS) and knowledge-aware response generation (RG) as relatively independent (line 124). I think however that RG is highly depdendent on KS, given many SoTA architecture train both models jointly end-to-end. In that case, the RG module might be less likely to just regurgitate knowledge, as it is knows the KS module is not 100% accurate. So far as I can tell, all of the experiments involve only training RG on the golden knowledge, in which case it is less surprising that the model learns to regurgitate. I think it would be a stronger paper therefore if you could show that RG had the same problem when trained on the distribution of KS, rather than the golden distribution.\n   \n(iii) The baselines are all chosen from text degeneration literature, rather than knowledge-grounded dialogue. I think this is an issue because the relationship between the context and response is very different from the relationship between the knowledge and the response. There is typically little term-level overlap between context and response, so penalising terms in the context can work well. However, there is significant overlap between knowledge and response, as is the premise of the paper, so it is not clear at all that these techniques will perform well in that setting. From the results it is not clear that the baselines surpass even the basic MLE method, other than in diversity metrics.", "questions_to_authors": "(a) Do the same knowledge regurgitation problems occur if the model is trained on predicted knowledge from a knowledge selection module, rather than the golden knowledge?\n\n(b) Can you motivate why no knowledge-grounded dialogue baselines are included (or clarify which baselines count as knowledge-grounded dialogue baselines)?  \n\nPost-rebuttal edit:\nIn light of the significant additional work conducted by the authors in response to my questions/concerns, I have moved my soundness rating to a 4. To clarify my point around end-to-end methods being less likely to regurgitate knowledge, I had in mind the architectures that treat the knowledge as a latent variable to be marginalised over [1] [2]--here, the generator is trained on the retriever's distribution, rather than the golden knowledge.   \n\n1) Shuster, Kurt et al. “Retrieval Augmentation Reduces Hallucination in Conversation.” Conference on Empirical Methods in Natural Language Processing (2021).\n\n2) Zhang, Yizhe et al. “RetGen: A Joint Framework for Retrieval and Grounded Text Generation Modeling.” AAAI Conference on Artificial Intelligence (2021).", "paper_topic_and_main_contributions": "The paper applies text degeneration techniques to knowledge-grounded dialogue generation. Specifically, they tackle the problem whereby models learn to simply regurgitate knowledge wholesale, without appropriately integrating it into a natural human response. They approach this via a Multi-level Adaptive Contrastive Learning framework, which penalises degeneration both at the token and sequence level. They show SoTA results across both human and automated evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new degeneration penalizing objective to improve knowledge internalization (or grounding). Some reviewers expressed their uncertainty about the results, however minor, and they think the paper can be benefitted by incorporating them in the final version.
Decision: Accept-Main