Post-hoc Utterance Refining Method by Entity Mining for Faithful Knowledge Grounded Conversations
Authors: Yoonna Jang
Suhyune Son
Jeongwoo Lee
Junyoung Son
Yuna Hur
Jungwoo Lim
Hyeonseok Moon
Kisu Yang
Heuiseok Lim
Venue: EMNLP 2023 Main (2023)
Keywords: Utterance Refining
Knowledge Grounded Conversation
Entity Mining
Entity-level Hallucination
TL;DR: To address the entity-level hallucination, we propose a post-hoc utterance refining method by entity mining, called REM, for more desirable and source-faithful knowledge grounded conversation.

Abstract: Despite the striking advances in recent language generation performance, model-generated responses have suffered from the chronic problem of hallucinations that are either untrue or unfaithful to a given source. Especially in the task of knowledge grounded conversation, the models are required to generate informative responses, but hallucinated utterances lead to miscommunication. In particular, entity-level hallucination that causes critical misinformation and undesirable conversation is one of the major concerns. To address this issue, we propose a post-hoc refinement method called REM. It aims to enhance the quality and faithfulness of hallucinated utterances by refining them based on the source knowledge. If the generated utterance has a low source-faithfulness score with the given knowledge, REM mines the key entities in the knowledge and implicitly uses them for refining the utterances. We verify that our method reduces entity hallucination in the utterance. Also, we show the adaptability and efficacy of REM with extensive experiments and generative results. Our code is available at https://github.com/YOONNAJANG/REM.

Post-hoc Utterance Refining Method by Entity Mining for Faithful Knowledge Grounded Conversations
Yoonna Jang", Suhyune Son", Jeongwoo Lee", Junyoung Son, Yuna Hur, Jungwoo Lim, Hyeonseok Moon, Kisu Yang, and Heuiseok Lim Department of Computer Science and Engineering. Korea University (morelychee, ssh5131, time79779, sony, yj72722, wjddn803, glee889, wi willow4, 1 Srinseokeekorea ac
Abstract
Previous Turn
Hey Snopp, nice to meel you. I'm Jazzzzy. Riaal name Jasmine... Do you know about Snoop Dogg?
Despite the striking advances in recent language generation performance, model- generated responses have suffered from the chronic problem of hallucinations that are ei- ther untrue unfaithful pecially in the task of knowledge grounded conversation, the models are required to gen- erate informative responses, but hallucinated utterances lead to miscommunication. In par- ticular. entity-level hallucination that causes critical misinformation and undesirable con- versation is one the major concerns. To address this issue, we propose a post-hoc finement method called REM. It aims to en- hance quality and faithfulness of halluci- nated utterances refining them based the source knowledge. the generated utter- ance has low source-fie faarree-fathhulnesss score with the given knowledge, REM mines the key enti- ties in the knowledge and implicitly uses them for refining the utterances. We verify that our method reduces entity hallucination in the ut- lerance. Also, we show the adaptability and efficacy of REM with extensive experiments and generative results. Our code is available at https://gi thub. com/ Comm/YOONNAJJAG/EEEE
X I'm not sure, but know that he was born in New York.
Case 2
given source. Es-
Yes, he my favorite rapper.
Source-faithful Utterance
talking SSoopp Dogglt artist and performer that came out 1992 with Dr. Dre on(DEEP COVER anveRanddan THE CHRONIC
Source Knom ledier: Snoop Dogs
Calvin Cordozar Broadus Jr. (bomn October 20, 1971).
grofessionaly Snoop Dogg
reper,, singor songwrilee. prslucer, media personality.
entrepreneur, and actor. His music career began in 1992
by Li and featured Dre's solo dehul, "hesp Cuver", and album, Thee Chronic" He has sinoe sold over 23 million million albums
when he
solo debut
albums United
Figure An illustrated example of the utterances with entity-level hallucination (i.e., case and 2) and desir- able utterances in knowledge grounded conversation. the source-faithuu utterance, entities of the given source knowledge are included, making the utterance informa- tive.
2020; Yu al., 2022: Shuster al., 2022a; Liu et al., 2022; Sun et al., 2023; Xiao et al., 2023).
1 Introduction
A colossal number of training corpora and pa- rameters have brought the recent explosion language generation performance pre-trained language models (PL.ss) (Brown et al., 2020; Raffel et al., 2020) and large language models (LLMs) (Thoppilan et al., 2022; Touvron et al., 2023; OpenAI, 2023). However, hallucination. which has been chronic problem in natural lan- guage generation, still cited one of the biggest challenges remaining unsolved (Ji et al., 2023). the KGC task, even though the ground truth knowl- edge is given, the models make the error of gen- erating hallucinated utterances not faithful to the source knowledge (Li et al., 2022a: Ji et al., 2023). Especially, entity-level hallucination, generating
The knowledge grounded conversation (KGC; also called knowledge grounded conversation, KGD), which is subfield of the dialogue systems, is task of generating human-like utterances by referring to specialized knowledge such as Wikipedia' (Zhao et al., 2022: Li et al., 2022c). The KGC task re- quires the ability to generate fluent and informative utterances based on source knowledge. Consider- ing this ability, there has been numerous concurrent research on the KGC to build models that can have conversations with human-like expertise (Ma et al.,
"These authors contributed equally to this work. "Coreesponding author. 'https://vee /wwww wike ki pedia. .rrg//

---

names of entities that are incorrect or not present in the source document (Nan et al., 2021), causes crit- ical misinformation and jeopardizes the flow of the conversation (Das et al., 2023). As shown in Case I of Figure 1, the previous KGC models generate the utterance containing information about the wrong entity, which is not given in the knowledge. Further, the generated utterance is excessively general while not considering sufficient entities that align with the context of the conversation, as shown in Case 2. These deficiencies can undermine the helpfulness of AI conversational models. Though previous re- search has tried to mitigate them in general domain conversation (Shuster et al.. 2021), research to ad- dress the entity-level hallucination in KGC remains in dark (Zhang et al., 2022).
fective method is adaptable to the existing models, including large language models, in plug-and- play manner; (3) We substantiate that our method reduces entity-level hallucination and accomplish more desirable knowledge grounded conversation with diverse experiments.
Related Work
2.1 Knowledge Grounded Conversation
In the task of knowledge grounded conversation (KGC), the systems aim to generate an informa- tive conversation based on specialized knowledge, To support research in this area, publicly avail- able datasets for the KGC task have been devel- oped (Gopalakrishnan et al., 2019; Moon et al., 2019; Shuster al., 2022b). These datasets fo- cus particularly on generating informative con- versations on specific topics (Dinan et al., 2018; Zhou et al., 2018; Jang et al., 2022). Building upon these KGC datasets, there has been active research to generate contextually consistent utter- ances while utilizing the source knowledge. Kim et al. (2020); Adolphs et al. (2021); Wang et al. (2021); )); Zhang et al. (2023); Feng al. (2023) in- corporate knowledge and context selectors filter out irrelevant knowledge sentences and redundant dialogue history, respectively. Additionally, (Niu et al., 2023) propose the history-adapted knowledge copy (HAKC) network, which selectively chooses context-aware knowledge to maintain dialogue co- herence. Likewise, diverse research is widely con- ducted in KGC tasks.
To address these entity-level hallucination prob- lems, we propose a post-hoc utterance refining method by entity mining, called REM, for more desirable and source-faithful KGC. REM can be used to refine the unfaithful utterances generated by previous models in plug-and-play manner. To refine utterances, we leverage the entity mining method, which extracts the named entities to im- plicitly utilize key information in the knowledge in multi-tasking manner. With this simple but ef- fective method, REM aims to mitigate entity-level hallucination and lead to a more successful conver- sation.
In order to measure the effectiveness of REM, we conduct an extensive empirical evaluation. First, demonstrate the post-hoc refining ability of REM, we experiment with refining the utter- ances generated by baseline models on three KGC datasets. We investigate the flcxibility and valid- ity of REM with cross-data experiments and ad- versarial data refining experiments, respectively. In addition, we conduct an ablation study and hu- man evaluation to verify the effectiveness of our method, showing the improvement of the source- faithfulness score and entity coverage of refined utterances. We also demonstrate the scalability REM by applying it to large language models. From a case study comparing the refined results of REM and bascline utterance, we demonstrate that our REM model improves the source-faiee faitculfit in the utterances.
2.2 Knowledge Hallucination in KGC
Despite the remarkable advancements, KGC sys- tems are known to suffer from knowledge hal- lucination generating unfaithful utterances to source knowledge (Dziri et al., 2021b). To ad- dress hallucination, Shuster al. (2021) pro- pose -nurra--ettrievall----ee- thhe--ooo architectures improve knowledge-abilin consisting of retrievers. rankers, and encoder-decoders. Additionally, Li et al. (2022b) proposes a method that utilizes entity and relation information from a knowledge graph to generate more faithful utterances. However, in KGC, there is lack of studies on level hal- lucination, which directly identifies and controls the generation of unfaithful utterances (Das et al., 2023). Although Dzári et al. (2021a) explore the study on entity-level hallucination, this study only considers cases where the source of knowledge is
Our contributions are threefold: (1) We propose a post-hoc refining method by implicitly mining key entities in the knowledge for more cource-faithful conversation; (2) We show that our simple but ef-

---

knowledge graph. Therefore, there is need for research on entity-level hallucination in KGC tasks where the source knowledge provided.
k
2.3 Utterance Refining Methods
Refining methods have been studied to improve the generation results when they are not satisfactory or not in the desired intention. (He, 2021; Geng et al.. 2021; Sun et al., 2022; Bao and Zhang, 2023) Espe- cially in dialogue systems, the researches aim im- prove the quality of response by applying the refin- ing method have been proposed (Tran and Nguyen, 2018; Posokhov et al., 2022; Wang et al., 2022). To tackle the limitation of generating uninforma- tive or not engaging utterances (Shen et al., 2018 10) Weston et al. (2018) retrieves the utterance and refines it by regenerating the retrieved utterance. Moreover, for fascinating dialogue systems, the ability to generate personalized responses accord- ing the user's s dialogue history is required (Cao et al., 2022). However, as history usually long and noisy, the problem missing personalized in- formation exists (Kasahara et al., 2022). To address this problem, Zhong et al. (2022) refines the user dialogue history to extract valuable information. Furthermore, they generate personalized dialogue by utilizing refined history information. Also, Song al. (2020) adopts rewriting method for persona- consistent dialogue generation. In this study, our goal to reduce entity-level hallucination, which is critical in informative conversation (Corbelle et al., 2022), with the refining method.
(a)
(b)
Figure 2: The graphical model of (a) the knowledge grounded conversation (KGC) and (b) our proposed REM. The black line is modeled by the KGC model parameter B. The yellow lines represent the entity miner, which is modeled implicitly, and the blue lines indicates the utterance generator, respectively, The colored lines are modeled by the REM parameter vb.
3.1 Knowledge Grounded Conversation
As depicted in Figure 2 (a), the KGC models gen- erate the informative utterance á considering both the conversational context (or dialogue history) c and the corresponding source knowledge k fol- lows:
po(ūlk, c)
II pa po c), (1)
where indicates the max sequence length of the utterance, and f denotes the KGC model parameter.
3.2 Post-hoc Utterance Refining
While KGC models aim generate informative utterances, gener--geerrttdd utterance á may not reflect the input source knowledge faithfully, as shown in Figure 1. In this context, we first filter out the utterance which is not faithful to the source. Then we renovate the given utterance into the re- fined utterance à which represents the intended knowledge better by utilizing the model-generated utterance â and its corresponding knowledge k from KGC models follows:
3 Proposed Method
Our proposed REM aims to refine the model- generated utterance to be more faithful to the source knowledge, as shown in Figure 3. The ut- terance filtered by the faithfulness scoring func- tion, whether be refined or not. We train the pre- trained language model with an encoder-decoder structure in the REM method. In the REM model, the entity miner extracts the named entities from the source knowledge and learns them implicitly. The utterance generator makes more faithful utter- ances based the key information extracted by the entity miner. In this section, formulate KGC task and our proposed utterance refining method, and describe the REM model and its training pro- cess.
Py P k, ④)
II P uet, k u),
(2)
where v denotes the REM model parameter.
Faithfulness Filtering To filter the model- generated utterance é, we quantify the source- faithfulness score, which indicates how â is con- sistent with the source knowledge k. To this end, we adopt the DAE (Goyal and Durrett, 2020) as

---

Entity Miner
Utterance Generator
Source Knowledge: Tears for Fears
Extracted Entities e
Refined Utterance i
Torm for Fears ane an Faglish pop rock band formed Bath, England in 1981 by Rohand Onzabal and Cirt Smith. Founded after the disoolation of their first handl the mad-infllencal Gradutte,
Tears for Fears are an English pop rock band Formed in Hoth, England by Roland Orzabal Smith: Founded after the dissolution their first band, mod- influcneed Graduate.
Tears for Fears were pop rock band formed in Bathy England in (Roland Orzabal) Curl Smithy They were intiialy associeted with new waae synthesiser bends of the early 1980s
Curt
love 1980s music, One of my favorite artists is Michael Jackson.
You mean all of the major events and trends music in the 80s? love music!
NER Head
GEN Head
have never heard of teors for fears before. What were their singles?
ENCODER
DECODER
that wave was influenzed by blues rock psychedelic rock.
not
i
Figure 3: The architecture of REM model, which consists of entity miner and utterance generator. The utterance û with a lower source fuure--fithhulness. score than threshold enters as input to REM with source knowledge k. The entity miner extracts entities from k, in white circles, while the utterance generator refines the à with the extracted entity e.
scoring function estimate the entailment be- tween the source knowledge and û considering the dependency are-level consistency of the utterance. For efficiency, we only refine the utterances that have lower scores than the threshold according to the source-faithfulness score, as follows:
reducing entity-level hallucinations and producing more fource-faithful utterances.
3.3 Training Objectives
The training objective of REM mainly falls into named entity recognition (NER) by entity miner and utterance re-generation (GEN) by utterance generator.
REM(K, ù) if score(k, i < û
(3)
To predict the named entities, the NER head is at- tached on top of the encoder and mines the entities inside the knowledge k. It makes the model learn the essential entity information, enhancing the inter- pretation of the source knowledge. To train NER, we tag named entities with one of the following entity types { ('LOC', 'PER', 'ORG', 'MISC') } using the off-the-shed entity tagging module? for the source knowledge in the data.
otherwise,
where score(.) denotes the scoring function (i.e., DAE) and 7 is threshold value between 0 and I. When 5 is I, all utterances are re-generated as ū; otherwise, when is O, all utterances are not refined. We utilize this filtering module only in the inference step, in the training step.
Refining Utterance by Entity Mining (REM)
The NER loss NNER is only defined for the NER label tagged for each token kū in k and minimized during training as follows:
Our refining method, REM, is decomposed into two sub-modules that are (i) entity miner and (ii)
utterance generator, as shown in Figure 2. The re-
No.
fining task is formulated follows:
CNER = No
Σ log p(k)())
(5)
Pv ( i) x Po a)
Pv (u à)
(4)
where N& denotes the token length of the source knowledge.
Entity miner
Ulterance generator
The language modeling head (GEN) learns to refine the given model-ggeerated utterance â along with k in an auto-regressive manner while extract- ing entities e. The model learns to compose the in- formation of the inputs and extracted entities from
These two modules are trained in multi-tasking manner with parameter th The former mines en- tities from source knowledge and learns entity- level knowledge implicitly. The latter learns to gen- crate the utterance u with the implicitly mined enti- ties, which is key information of source knowledge,
Pht / thub. com/flairNLP/flairr

---

the encoder hidden states. It trained by minimiz- ing the following loss function:
chrl' (Popović, 2015), ROUGE-L (Lin, 2004) and SacreBLEU (Post, 2018) evaluate how close the generated hypothesis is to the ground-truth answer in the test set. For diversity evaluation, Distinct- N (Li et al., 2015) is adopted.
Nu
1
LGEN
log plgp(u K, éz e),
(6)
Na
Entity Coverage and Entity Type Coverage We compute the entity coverage (EC) and entity type coverage (TC) to quantify the extent of correct en- tities and context coherence, respectively. To com- pute EC and TC, we extract the named entities from the generated (model-predicted) utterance up (in- cluding 7 and û), source knowledge k, and ground truth utterance и, respectively, using the off-the- shelf NER model (Schweter and Akbik, 2020) (The distribution of named entity types are described in Appendix B). First, we evaluate the ratio of the named entities residing in the generated utterance up by comparing it with the ground truth utterance u as follows:
where e is implicitly mined in the encoder during training.
The final training objective for a REM trained on both tasks is the following, where An is the training hyperparameter:
CREM ALEE + AeLGEN
(7)
4 Experimental Settings
In this section, we introduce the dataset, the auto- matic evaluation metric, and the baseline models used in the experiments. The implementation de- tails are attached Appendix C.
4.1 Datasets
EC = ((EE O EE) Exp) N EE))
We utilize three datasets as our testbed, but we only use instances for training and testing where ground truth knowledge given in the data.
(8)
where (I) is the number of values in list I and Ex is the set of named entities in source knowledge. Fry and Eu, indicate the set of named entities ground truth utterance and generated utterance, re- spectively.
FoCus (Jang et al., 2022), which considers both knowledge and persona in conversation, has been released. In FoCus, machine gener- ates utterances with customized and knowledge- able utterances about world landmarks. Wizard of Wikipedia (WoW) (Dinan al., 2018), which is a widely used benchmark, consists of Wizard and Ap- prentice talking to each other on various topics in Wikipedia. Another dataset, CMUDoG (Zhou et al.. 2018) includes conversations between two speak- ers discussing different aspects of specific movie, such information, plot, etc. The data statistics are presented in Appendix A.
In addition, we evaluate TC, as existing models produce fluent text but demonstrate low context coherence. To identify the intent of context and the information that should be included in the response, leverage the named entity types in the ground truth utterance u. For example, if the entity type 'LOC" has the highest proportion among the named entities in u. it indicates that the conversation is focused on location-related information. In such cases, the model should generate responses that are relevant to the location to ensure coherence and relevance in the conversation.
4.2 Automated Evaluation Metrics
We evaluate the refining performance of REM with automated metrics in three criteria: source- faithfulness, reference matching, and diversity. To evaluate source- faithfllatt,,,,ness, we adopt DAE, en- tity coverage (EC), and entity type coverage (TC). DAE (Goyal and Durrett, 2020) is used to assess whether the system accurately reflects the facts from the given knowledge when generating ut- terances the level of dependency arcs. Further- more. we utilize two entity-level metrics (EC and TC), following subsequent paragraphs, evaluate the cource-faithfulnes in terms of the entity. For the reference matching n-gram score, we utilize
To this end, we compute the ratio of name entity types extracted from the generated utterance up compared to the ground truth utterance u each named entity type.
T TC Σ
NO
(9)
where T is the set of named entity types ('LOC', 'PER', 'ORG', 'MISC").
htttps: thub. com/neural al wwweeeuralliillooo dialogue metgguuemmtrcc//// Distinct-N

---

4.3 KGC Baselines
datasets are presented at the top of Table 1. We fine-tune the BART-base model with each dataset and refer to it as BARTbasse To the generated out- puts of BART base, we evaluate the refining perfor- mance with REMbase- The results demonstrate that the REM leads to improvements not only in the scores of fourue-faithfulnees metrics but also in the reference matching metrics. We also show the ability of REM that refines the utterances of the larger model Appendix F.
BART BART (Lewis et al., 2020a) is Transformer-based (Vaswani et al., 2017) pre-trained model with the encoder-decoder structure. It utilizes various noising methods and in-filling schemes during pre-training. We fine-tune BART three KGC datasets and use its predicted utterance for train, validation, and test set.
INFO INFO (Lim et al., 2022) leverages RAG (Lewis et al., 2020b) to enhance the factu- ality of the generation results in FoCus dataset. It retrieves a large number of knowledge documents and generates informative conversations based on them.
In contrast, REM shows a tendency to decrease the performance of diversity metrics. This indicates that in BART - utterances, there are tokens that contribute to increased diversity but also lead to ballucination. These ballucinated tokens are subsequently refined through the REM process. In particular, the significant improvements in EC and TC metrics suggest that entities that cause entity- level hallucination are refined during the refining process with REM.
EDMem We utilize and refine the utterances gen- erated by EDMem (Zhang et al., 2022). EDMem is pre-trained on Wikipedia documents to learn entity embeddings and incorporates entity knowledge for entity-intensive dialogue generation.
Existing Baselines To investigate the adaptabil- ity of REM, we compare the refining performance with previous studies, and the results are annotated with 1 symbol in Table 1. We re-implement ED- Mem, ITDD, and INFO for fair comparison with REMaagge The results reveal that REMaagg ex- hibits improvement in fatthe-aaithfulness per- formance. Additionally, it demonstrates enhanced generation performance in the reference match- ing score, cxcluding INFO. The reason is that INFO utilizes a large number of knowledge doc- uments for generation, whereas REM generates utterances based on the given ground truth knowl- edge within the dataset. Furthermore, while the Source-faithfulnese score increases, the diversity score decreases. This result discusses that REM removes tokens that enhance diversity but also con- tribute to entity hallucination, similar to the results observed with vanilla PLM.
ITDD ITDD (Li et al., 2019) is the best- performing method on the CMUDoG dataset. With two-pass decoder inspired by the human cogni- tive process, improves context coherence and knowledge correctness.
5 Results and Analysis
In this section, we report the experimental results and analysis. We first discuss the post-hoc refining ability of our method in $5.1. Then, evaluate the flexibility of REM through cross-data refining experiments in $5.2. We also conduct experiments on refining adversarial utterances in $5.3, and per- form an ablation study in $5.4. Furthermore, we provide the results of human evaluation in $5.5, and present case study in $5.7. Finally, we analyze the application of REM to large language models in $5.6.
5.2 Cross-data Utterance Refining
5.1 Post-hoc Refining Ability
To inspect the flexibility of REM, we conduct cross- data experiments, and the results are shown in Ta- ble 2.
To demonstrate the post-hoe refining ability, we evaluate the refining performance of REM with vanilla PLM fine-tuned on three datasets and exist- ing models from previous studies. The results are in Table where REM and REMtarge refer to the models trained using the REM method on the BART-base and BART-large (Lewis et al., 2020a).
REM trained on CMUDoG consistently un- derperforms in foirce-fittfullesss and reference matching scores on the other two datasets. When refining FoCus test set with REM trained on CMU- DoG, the performance significantly decreases in all metrics. This tendency is similarly shown in the results of WoW test set. Likewise, the REM trained
Fine-tuned Baselines The comparison results of REM with the vanilla PLM fine-tuned on three

---

Source-faithfalness
Reference Matching
Diversity
Data
Model
EC (%)
TC (%)
DAE
chrF
ROUGE-L
BLEU
Dist-1
Dist-2
BART base
7.54
13.85
0.60
19.47
28.04
2.51
0.37
0.80
FoCus
REMbase (Ours)
23.40
20.32
0.83
30.38
33.14
9.47
0.28
0.73
BART
14.23
8.80
0.84
31.70
34.02
12.72
0.31
0.78
WoW
REM (Ours)
14.57
9.03
0.87
32.44
34.17
12.83
0.29
0.77
BART baue
3.40
3.90
0.26
12.92
13.14
3.25
0.40
0.83
CMUDoG
REM (Ours)
4.95
7.55
0.40
15.06
11.92
2.49
0.25
0.70
INFO
0.05
24.40
0.79
50.35
54.27
32.55
0.26
0.70
FoCus
REMMarge (Ours)
0.11
25.48
0.86
49.98
52.30
30.17
0.24
0.68
FDMem
0.00
3.50
0.50
14.30
15.21
2.90
0.28
0.71
Wow'
REMiarge (Ours)
4.51
6.51
0.77
19.46
18,30
4.25
0.25
0.68
ITDD
0.00
1.42
0.12
9.30
9.39
1,71
0.48
0.82
CMUDoG
REMMerge (Ours)
1.01
3.96
0.30
13.64
10.44
1.74
0.24
0.67
Table 1: Experimental results of baseline utterance refining with REM. The top half shows the results of refining (1) the tine-tuned baselines, and the bottom half shows the results of refining (2) existing baselines. Numbers in boldface indicate higher scores. denotes the evaluation settings of the existing models, and the vanilla setting (without !) denotes the evaluation settings of our fine-tuned baseline. The utterances are filtered with T = 0.5.
Soaree-fatthtlless
Ref. Matching
Source-faithfulnes
Ref. Matching
Train
Test
EC (%)
(C.)
DAE
chrik
R-1.
Model
(%)
(C)
chrlf'
R-L
FoCus
50.64
29.88
0.84
50.47
46.55
ADV
27.77
19.28
0.63
27.09
31.10
WOW
RoCus
0.03
8.35
0.03
9:24
8.65
FoCos
REM
46.68
35.41
0.85
42,49
37.00
CMUDoG
17.02
16.20
0.29
21.32
15.72
RiMMarge
48.04
34.19
0.87
41.91
36.87
WoW
20.84
13.59
0.77
37.18
31.55
ADV
5.82
16.53
16.78
FoCus
WoW
21.21
14.51
0.78
36.76
30.12
CMUDoG
35.23
29.16
REM
18.93
12.71
0.75
33.03
29.33
RicMbarge
17.58
12.38
0.75
34.09
28.77
CMUDOG
6.90
9.50
0.43
16.42
12.44
FoCus
CMUDEG
12.51
0.83
13.37
7.08
ADV
3.33
4.98
0.23
12.07
12.04
WoW
3:40
3,90
0.26
12.92
13.14
CMUDoG
REMMss
5.21
10.08
RRM,
9.24
Table 2: The result of cross-data experiments with
REMiarge: All utterances are refined with r = 1.0.
Table 3: Experimental results of adversarial data refining. 'ADV' denotes the adversarially edited utterances by ChatGPT. All utterances are refined with r = 1.0.
on WoW exhibits decrease in performance on FoCus and CMUDoG test sets.
utterances that provide responses without knowl- edge, FoCus ensures that all utterances are gener- ated based on knowledge in the dataset. Therefore, the REM trained on FoCus has facilitated the train- ing of a more faithful KGC model.
According to the previous comprehensive human study (Dziri al., 2022b) that analyzes the portion of hallucination in KGC dataset, the reason for the performance decrease is hallucinated utterances in WoW and CMUDoG datasets. The results revealed that only 24.15% of utterances in wow and 16.2% in CMUDoG exhibited entailment. These propor- tions indicate that significant portion of utterances contains knowledge hallucination. Consequently, hallucination in the dataset has an impact on model training and evaluation.
5.3 Adversarial Data Refining
To investigate the validity of REM, we have prompted CharGPT (OpenAl-Blog, 2022) to gener- ate the synthetic data by adversarially changing the ground truth utterance We demonstrate the perfor- mance of REM by refining the utterances that are distorted to be unfaithful to the source knowledge; all data are refined with (r = 1.0). As presented in Table 3, compared to the adversarially synthesized
On the other hand, the model trained on FoCus dataset demonstrates high foithu-faitffulness per- formance across all datasets. We assume that the reason for performance differences lies in dataset construction. While WoW and CMUDoG contain
"The prompts are shown in Appendix E.

---

Souroe-fattfullnes
Ref. Matching
Min
Tie
Model
EC (%)
TC (%)
DAE
chri'
R-L
FoC.05
Lose
BARTiore
28.04
REMisa WO NER
30.24
33.08
23.28
20.22
0.82
REMsase
30.38
33.14
23.40
20.32
0.83
REMiorge WO
24.17
20.26
REMiurge
31.52
34.56
20.58
20.58
0.84
CNUDsG F
CMUDoG
Table 4:
The result of ablation study on training objec- tives. The utterances are filtered with 0.5
CMIDaG
100
data (ADV), both source-faithfulness and reference matching scores increased significantly after refin- ing, ROUGE-L score of the ADV slightly higher in CMUDoG, which likely due to the characteris- tic that the utterances of CMUDoG have much shorter lengths than other datasets (Dziri et al., 2022a).
Figure 4: Results of human evaluation. F, S and P. tached after the dataset name, denote fluency, source faithfulness and paraphrasing, respectively.
second criterion measures source- fairce-fittfullesss by assessing whether the sentence is factually consis- tent with the given knowledge. In addition, the third criterion evaluates whether the given knowledge has been appropriately reorganized and incorpo- rated into the utterance rather than just copied and pasted. We provide the questionnaire used in Ap- pendix G
5.4 Ablation Study
To explore the effect of entity miner, we conduct an ablation study of our proposed method. In Table 4, we present the results on the FoCus dataset, which has the most informative conversations in $5.2. We compare the REM model trained with multi-task learning and only trained with utterance refining (w/o NER).
The results of the comparative human evalua- tion of baseline utterances with scores below the threshold and utterances after refining are shown in Figure 4 (r = 0.5). Across all data and crite- ria, the refined utterances win in most cases. Es- pecially in FoCus, refined utterances do not lose to the utterance before refining. REM performs the worst in CMUDoG, which is similar to the experiment in Section 5.1. The dissimilarity be- tween the CMUDoG and the other datasets can be attributed its predominant resemblance to a general utterance instead of the knowledge grounded conversations. Nevertheless, we qualita- tively demonstrate the effectiveness of REM with a significantly higher number of winning cases.
In case the model is trained without entity miner, the results show a performance decrease across the board except in one case. Our proposed REM method, which learns essential information with en- tity miner, can be demonstrated to be effective for source-faithfulnes: and reference matching scores, The DAE score of the large models shows bet- ter performance without entity miner. The large model shows unstable results $5.3. However, significant improvement in the performance of the REM model is shown, with or without entity miner, compared to the baseline utterance. This suggests that refining helps increase the quality and source- faithfulness of fatal utterances.
5.6 REM-LLM
To show the scalability of our method to large lan- guage models (LLMs), we apply REM method to the prompt of not-tunable LLM (i.e., ChatGPT) be- yond fine-tuning the models. In this experiment, large language models with (LLMREM) and with- out (LLM) the REM method refine the utterance of the KGC bascline model (BART and Chat- GPT (LLMxGc). LLM required refine the utterance considering the given source knowledge. while L.M REM is made modify the utterance by extracting key entities from the knowledge and utilizing them. We attach the prompts used in Ap-
5.5 Human Evaluation
To qualitatively evaluate the results before and after refining, we perform a human evaluation on 100 randomly sampled utterance examples, each from three datascts. One example contains utterance produced by the baseline and the refined utterance by REM, and we evaluate them with three crite- ria: 1) fluency, 2) foirfe-,ait and 3) para- phrasing. The first criterion assesses naturalness by measuring the fluency of the generated results. The

---

faith
Mutching
FoCus
Data
Model
EC
TC
DAE
chriF
R-L
[Knowledge]
BART
7.56
13.85
0.60
19.47
28.03
Designed by French architect Paul Andreu, the NCPA opened in 2007 and the largest theatre complex in Asia.
FoCus
LLM
23.54
20.14
0,86
26.96
33.57
[Persona Knowledge]
LLMREM
22.18
28.27
33.66
love art. I'm interested in architecture. love lakes, I want go on trip. I want to visit China. [Utterance]
BART
14.23
8.79
0.84
31.73
34.02
14.34
8.89
31.95
33.87
-LLM
14.57
9.12
0.86
32.27
33.84
SARThaxe National for Per Arts in opened 2007.
REM : National Center for the Performing Arts
BART
3,40
3.90
12.92
(NCPA) opened 2007 and the largest theatre complex
CMUDoG
+ LLM
4.93
6.07
0.36
13.26
12.43
Asia. If you love art, you have
it with your
LLM
7.36
11.39
0.57
14.70
11.23
eyes!
LLMRGG
42.74
26.72
0.84
38.
40.64
FoCus
L.M
44.95
27.45
0.89
39.61
39.61
WoW
LLM
45.83
28.46
0.91
40.01
41.98
[Knowledge]
LLMKGG
6.91
5.87
0.36
18.68
18.65
Among other honors. he has won nine Goyas, two European Film Awards and an Oscar.
WoW
L.L.
9.27
6.73
0.47
21.24
20.17
[Utterance]
LLMRREM
13.28
9.25
0.63
25.17
21.19
BART beaT I've seen few times. It was directed by Alejandro
LLMRGc
4.77
7.24
0.29
14.36
13.44
Amená
CMUDoG
LLM
5.61
8.53
0.38
14.36
12:21
REMause course! He has won nine Goyas, two Euro- pean Film Awards, and Oscar! That's a lot of awards for director!
R.L.
7.88
14.26
0.54
14.36
10.04
Table 5: The result of REM with the large language model (denoted with +) refining the KGC utterances of BART base and large language model (LLMkac). The utterances are filtered with 7 = 0.5
CMUDoG
[Knowledge]
The Asgardian Loki encounters the Other, the leader of an ex- traterrestrial race known as the Chitauri. ... Loki takes the Tesser- and his scepter enslave Selvig and couple of other agents, including Clint Barton, to aid him in his getaway. [Utterance]
pendix H.
BART you should watch the avengers and
As shown Table 5, both LLM and LLMREM show improved faithullist performance, especially in CMUDoG and FoCus. REM is able to achieve the most significant improvement by entity mining and refining the utterance with key entities. This demonstrates that when prompting with LLMs, more source-faithful utterances can be produced with REM by asking LLMs to find the important parts and use them for making responses.
REM you should watch the avengers 2 and 3, they are really good as well. what did you think of the scene where Loki takes the Tesseract?
Table 6: Examples of refined utterances by REM on three datasets.
Conclusion
In this work, we proposed REM, a post-hoc refin- ing method for improving the source-faithfulie of the utterances in the knowledge grounded con- versation (KGC), REM enabled simple but effec- tive refining by extracting entities from the source knowledge given in the KGC task. It makes the model learn the key information implicitly and use it for refining the utterance more faithful to the source knowledge. We presented extensive exper- iments applying REM in a plug-in-play method to various model-generated outputs, showing in- creased nurce-faithfulless. and entity coverage af- ter refining. Also, qualitative analysis and human evaluation proved the refining efficacy of REM. We verified that our proposed method could be utilized with the prompt of the large language models.
5.7 Case Study
To analyze the generative results of REM, present the examples of three datasets in Table 6. In FoCus example, the utterance of baseline answers only provides the information of year, but with the imperfect entity mention 'Per'. However, after be- ing refined with REM, the utterance provides more specific knowledge with correct entity mentions, even utilizing the persona knowledge. In WoW ex- ample, the bascline utterance provides knowledge that is not given in the knowledge, but REM gives source-faithful information. In CMUDoG example, REM provides more detailed factual knowledge in the utterance than in the baseline utterance. We show more examples also in Appendix I.

---

Limitations
support program(ITP-002-2211--680
supervised by the IInt(tnt for Information & Communications Technology Planning & Evaluation). This work was supported by Institute of Information & communications Technology Planning & EValu)tion(IITP) grant funded by the Korea governmen((MSII) (No. 2020-0-00368, A Neural-Symbolis Model for Knowledge Acquisi- tion and Inference Techniques). Also, this work was supported by NCSOFT NLP Center.
Our research proposed a method that aims to refine the non-source-faithful utterances in the knowledge grounded conversation. Though we address the problem of models not reflecting even the ground truth knowledge given, it is difficult solve if the retriever does not give accurate knowledge from the in-the-wild setting. Therefore, the performance of the refiner may depend on the retriever's per- formance. As it is beyond the scope of this pa- per, we leave it as future work. When filtering the utterances by source-faithfulness score, we only utilized DAE, but other scores or classifiers (Dou al., 2022; Manakul et al., 2023) can be adopted for filtering according to the data or domain. There is an area of research in detecting critical errors generated results in neural machine translation. Similarly, in the dialogue tasks, the better the clas- sifier performs in detecting hallucinated utterances, the better the performance of the refiner will be. In addition, expect that existing baseline mod- els could be further improved if trained with REM in an end-to-end manner, we leave it as future work.
References
Leonard Adolphs. Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2021. Reason first, then respond: Modular generation for knowledge-infusee dialogue. arXiv preprint 2r1 11.020044.
Guangsheng Bao and Yue Zhang. 2023. A general contextualized rewriting framework for text summa- rization. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Ethics Statement
Yu Cao, Wei Bi, Meng Fang. Shuming Shi, and Dacheng Tao. 2022. aggostic data manipulation method for persona-based dialogue generation. arXiv preprint arXi::220..09867.
The datasets used in our work are from previ- ously published papers, so we do not attach pri- vacy or ethical issues to the dataset. At inference, we will make the model not generate tokens in a list of several stopwords utilizing Hatebase", Silva et al. (2016), etc., to avoid generating harmful ut- terances that the model may have learned during aware that excessive
Javier González Corbelle, Alberto Bugarin Diz, Jose Alonso-Moral, and Juan Taboada. 2022. Dealing with hallucination and omission in neural natural language generation: A use case meteorology. In Proceedings of the 15th International Conference on Natural Language Generation, pages 121-130.
pre-training. As putational energy used to train models stimulates environmental problems, we adopt the multi-task learning method for training entity miner and utter- ance generator instcad having a separate entity miner model to improve efficiency. Also, we dis- tribute all codes and model checkpoints so they do not have to be trained again. We believe that our refining method can contribute to mitigating the entity-level hallucination of model-generated utter- ances and preventing the misuse of Al systems.
Souvik Das, Sougata Saha, and Rohini K Srihari. 2023. Diving deep into modes of fact hallucinations in dia- logue systems. arXiv preprint urXiv:2301.044
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wiz- ard wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv: 1871 1871,01241.
Yao Dou, Maxwell Forbes, Rik
Noah Smith, and Yejin Choi. 2022. Is gpt-3 text indistinguishable from human text? scarecrow; A framework for scrutinizing machine text. In Pro- ceedings the 60th Annual Meeting the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7270-7274.
Acknowledgements
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center)
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Os- mar Zaiane, Mo Yu, Edoardo Ponti, and Siva Reddy. 2022a. Faithdial: A faithful benchmark for
"http://hatebase.org

---

ceeemattion-sekiingg dialogue. Transactions of the Association for Computational Linguistics, 10:1473- 1490.
Sihyung Kim, Oh- Oh-Woog Kwon, and Harksoo Kim. 2020. Knowledge-grow grounded chatbot based on dual wasser- stein generative adversarial networks with effective attention mechanisms, Applied Sciences, 10(9):3335.
Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021a. Neural path hunter: Re- ducing hallucination in dialogue systems via path grounding. arXiv preprint arXiv:2104.08455. 104.08455.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdeirahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. Bart: Denoising sequence-to-sequeni pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880.
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy, 2022b. On the origin of hallucinations in conversational models: Is it the datasets or the models? arXiv preprint 2r0i 07931.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021b. Evaluating groundedness in dia- logue systems: The begin benchmark. arXiv preprint arXiv:2105.00071.
Patrick Lewis, Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020b. Retrieval- -uueell-uggmeteed -aggmettdd generation for knowledge-intensiee nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Jiazhan Feng, Chongyang Tao, Xueliang Zhao, and Dongyan Zhao. 2023. Learning multi-turn response selection in grounded dialogues with reinforced knowledge and context distillation. ACM Transac- tions on Information Systems, 41 1124)
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2013. A Dnversity-promotte objec- tive function for neural conversation models. arXiv preprint urXiv:1570.0300
Xinwei Geng, Xiaocheng Feng, and Bing Qin. 2021. Learning to rewrite for non-autoregressive neural ma- chine translation. In Proceedings of the 2021 Con- ference on Empiricul Methods in Natural Language Processing, pages 3297-3308.
Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022a. Faithfulness in natural language generation: A systematic survey of analysis. evaluation and optimization methods. arXiv preprint arXiv:2203.05227.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilck Hakkani-Tie Tür. 2019. Topical-chat: Towards knowledge groundedd open-domain conversations. Proc. Interspeech 2019, pages 1891-
Yanyang Li, Jianqiao Zhao, Michael Lyu, and Liwei Wang. 2022b. Bliciting knowledge from large pre-trained models for unsupervised knowledge-groundee conversation. arXiv preprint arXiv:2211.0158)
Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment. arXiv preprint arX1v:2010.05478.
Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2022c. Knowledge- grounded dialogue generation with a unified knowl- edge representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 206-218.
Xingwei He. 2021. Parallel refinements for lexically constrained text generation with bart. arXiv preprint arXiv:2109.1P 12487.
Yoonna Jang, Jungwoo Lim, Yuna Hur, Dongsuk Oh, Suhyune Son, Yeonsoo Lec. Donghoon Shin, Se- ungryong Kim, and Hcuiseok Lim. 2022. Call for customized conversation: Customized conversation grounding persona and knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 10803-10812.
Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou. 2019. Incremental transformer with deliberation decoder for document grounded conversations. arXiv preprint urXiv:1907.000
Jungwoo Lim, Myugnhoon Kang, Yuna Hur, Se- ung Won Jeong, Jinsung Kim, Yoonna Jang. Dongyub Lee, Hyesung Ji. DongHoon Shin, Seun- gryong Kim, et al. 2022. You truly understand what need: Intellectual and friendly dialog agents ground- ing persona and knowledge. In Findings of the Asso- ciation for Computational Linguistics: EMNIP 2022, pages 1053-1066.
Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su. Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38
Tomohito Kasahara, Daisuke Kawahara, Nguyen Tung, Shengzhe Li, Kenta Shinzato, and Toshinori Sato. 2022. Building personalized dialogue system with prompt-tuning. arXiv preprint arXiv:2206.05399
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.

---

Qi Liu, Dani Yogatama, and Phil Blunsom. 2022. Rela- tional memory-augmennt language models. Trans- actions of the Association for Computational Linguis- tics, 10:555-572.
Pavel Posokhov, Kirill Apanasovich, Anastasia Matveeva, Olesia Makhaytkina, and Anton Matveev. 2022. Personalizing dialogue agents for russian: Retrieve and refine. In 2022 31st Conference of Open Innovations Association (FRUCT), pages 245-252. IEEE.
llya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
ar1.v0171..0510..
Matt Post. 2018. A call for clarity in reporting bleu scores, arXiv preprint arXiv: 1804.0877..
Longxuan Ma, Wei-Nan Zhang. Mingda Li, and Ting Liu. 2020. A survey of document grounded dialogue systems (dgds). arXiv preprint arXiv:2004. 13818.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint 2rXiv:2303.08890
Stefan Schweter and Alan Akbik. 2020. Document-level features for named entity recogni- tion.
FLERT:
Seungwhan Moon, Pararth Shah. Anuj Kumar, and Ra- jen Subba. 2019. Opendialkg: Explainable conver- sational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 845-854)
Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg, 2018. Improving variational encoder-decoders in dialogue generation. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Feng Nan, Ramesh Nallapati, Zhiguo Wang. Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen Meke- own, and Bing Xiang. 2021. Entity-level 2023tency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-2733.
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022a. Language models that seek for knowl- edge: Modular search & generation for dialogue and prompt completion. Findings of the Association for Computational Linguistics: EMNIP 2022.
Chenxu Niu, Yue Hu, Wei Peng, and Yuqiang Xie. 2023. Learning to balance the global coherence and infor- mativeness in knowledge- srowledge-grounded grounded dialogue genera- tion. In ICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint 2rXiv:2104.07567.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, et al. 2022b. Blenderbot 3: deployed conversational agent that continually learns to responsibly engage. arXiv preprint arxiv:2208.03/88.
OpenAL. 2023. 4pt-4 technical report.
OpenAI-Blog. 2022. Chatgpt: Optimizing language models for dialogue.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Leandro Silva, Mainack Mondal, Denzil Correa, Fabrí- cio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Tenth international AAAI conference on web and social media.
Haoyu Song, Yan Wang, Wei-Nan Zhang, Xiaojiang Liu, and Ting Liu. 2020. Generate, delete and rewrite: A three-stage framework for improving persona con- sistency of dialogue generation. arXiv preprint arXiv:2004.0667..
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, al. 2021. Kilt: benchmark for knowledge in- tensive language tasks. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2544.
Qingfeng Sun, Can Xu, Huang Hu, Yujing Wang. Jian Miao, Xiubo Geng, Yining Chen, Fei Xu, and Daxin Jiang. 2022. Stylized generation via disentangled template rewriting. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3304-3318.
grounded dialogue
Maja Popović. 2015. chrf: character n- n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392-395.

---

Weiwei Sun, Pengjie Ren, and Zhaochun Ren. 2023. Generative knowledge selection for knowledge- grounded dialogues. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2032-2043.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge--ehaaeeeddeee text generation. ACM Computing Surveys, 54(11s):1-3
Yogin Zhang, Haomin Fu, Cheng Fu, Haiyang Yu, Yong- bin Li, and Cam-Tu Nguyen. 2023. Coarse-to-fine knowledge selection for document grounded dialogs. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng. Ali- cia Jin, Taylor Bos, Leslie Baker, Yu Du, al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.022.
Zhihan Zhang, Wenhao Yu, Chenguang Zhu, and unified encoder-decoder
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971 139771
Meng Jiang, 2022. framework with entity memory. arXiv preprint arXiv:2210.0227.
Xueliang Zhao, Tingchen Fu, Chongyang Tao, Wei Wu, Dongyan Zhao, and Rui Yan. 2022. Learning to ex- press in soowledgegrooudde ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2273..
conversation. In Pro-
Van-Khanh Tran and Le-Minh Nguyen. 2018. Semantic refinement gru-based neural language generation for spoken dialogue systems. In Computational Linguis- tics: 15th International Conference of the Pacific As- sociation for Computational Linguistics, PACLING 2017. Yangon, Myanmar, August 16-18, 2017, Re- vised Selected Papers 15, pages 63-75. Springer.
Hanxun Zhong, Zhicheng Dou, Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. 2022. Less more: Learn- ing to refine dialogue history for personalized dia- logue generation. arXiv preprint arXiv:2204.08/28.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processsin systems, 30.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 708-713.
Weichao Wang, Wei Gao, Shi Feng, Ling Chen, and Dal- ing Wang. 2021. Adaptive posterior knowledge se- lection for improving generation. In Proceedings of the 30th ACM Inter- national Conference on Information & Knowledge Management, pages 1989-1998.
wwwpwwlddgggrounddee dialogue
Yihe Wang, Yitong Li, Yasheng Wang, Fei Mi, Pingyi Zhou, Xin Wang. Jin Liu, Xin Jiang, and Qun Liu. 2022. Pan more gold from the sand: Refining open- domain dialogue training with noisy self-retrieval generation. arXiv preprint 220v 11367.
Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint TrXiv:1808.04777
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. 2019, Huggingface's 's transformers: State-of- the-art natural language processing, arXiv preprint arX.v:1910.03771.
Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao Cao, and Maosong Sun. 2023. Plug-and-play document modules for pre-trained models. arXiv preprint 2305v 17660.

---

A Dataset Details
C Implementation Details
FoCus (Jang et al., 2022) is a dataset where a hu- man and machine take turns having conversation about a specific landmark. This dataset consists of 14,452 conversations, with 173,424 utterances. We use the validation set, which has 1,445 conversa- tions and 17,340 utterances, for the experiment, as the official test set does not provide ground truth knowledge and utterances.
When implementing REM, we adopt BART (Lewis et al., 2020a), which has an encoder-decodes struc- ture, and train the entity mining and utterance gen- eration tasks. We experiment with BART base of 140M parameters and BARTiorge of 406M parame- ters. We train REM on the pairs of model-predicted utterances, generated by the fine-tuned BART base- line, and reference utterance. We implement the models by exploiting Pytorch (Paszke al., 2019) and HuggingFace (Wolf et al., 2019) with a fixed seed for reimplementation. The models are trained with a learning rate of 6.25e-5 for 10 epochs with early stopping. AdamW (Loshchilov and Hutter, 2017) is used as the optimizer. We set the train batch size to with gradient accumulation of 32. The time required for training is about 2 hours per epoch on a single RTX-6000 GPU. For de- coding, we used a beam size of 5, min length of 32, max length of 512, top k of 50, and no-repeat n-gram size of 2. NER loss weight λι and GEN loss weights X2 are set as 0.5:1, 0.3:1, and 0.7:1 for WoW, CMUDoG, and FoCus, respectively, ac- cording to the preliminary study. We will open the source codes which used for the experiments after the review process.
Wizard of Wikipedia (WoW) (Dinan al., 2018) is a conversational dataset based on Wikipedia ur- ticles on various topics. The dataset covers 1,365 topics and consists of 22,311 conversations with a total of 201,999 utterances, which are divided into 166,787 for train, 17,715 for validation, and 17,497 for test. We used random split version of the validation and test set. Only for evaluating ED- Mem (Zhang et al., 2022), follow its test setting, KILT (Petroni et al., 2021), variant of the WoW dataset.
CMUDoG (Zhou et al., 2018) is a knowledge grounded conversation dataset with two speakers conversing based on movic Wikipedia articles. Un- like the WoW and FoCus, where only one of the two speakers has access to the knowledge, both speakers of this dataset have access the content of the article. It has resemblance to generic dialogue compared to the other two datasets. It is composed total of 4,112 conversations with an average 21.43 turns.
Data
Model
ACC (NER)
FI (ACC)
PPL (GEN)
FoCas
REMMose
0.991
0.944
3,778
FoCas
REMaage
0.989
0.925
3,323
WoW
REM
0.998
0.938
12.742
Distribution of Named Entity-Type
Wow
REMaagge
CMUDoG
REMous
1.000
1.000
29.775
We show the entity type distribution of all datasets, the train, development, and test sets used for fine- tuming the baseline model BART bases in Figure 5. WoW has the evenest distribution, and FoCus has the second most even distribution. CMUDoG has the most "PER" entity class among the other three classes.
CMUDoG
REMarge
0.991
27.247
Table 7: The result on validation set used training and evaluating the models with both tasks, NER and GEN. Larger models show slightly higher scores in GEN, but lower scores in ACC.
1009
D Threshold
To utilize the off-the-shelf metric as our scoring function for filtering, we manually find the thresh- old both for effectiveness and efficiency. The high threshold is intended for evaluating the source- faithfulness of given utterance strictly. A low threshold, on the other hand, targets only those terances that are not faithful to the knowledge and aims to regenerate only fatal cases. As shown in Figure 6, we use a threshold of 0.5 to consider all datasets and 1.0 we need to refine utterances.
1229%
10%
focus
conalog
Figure 5: Entity type distribution three datasets.

---

G
Human Evaluation Questionnaire
Questionnaire
27
The given sentences from model A and model the gener- uted result of cach machine consulting its knowledge to produce informative utterances. Considering the sentences produced by model A and model B, name the model that better on the following thrce criteria.
30
Fluent The response is linguistically fluent and not awkward). Factually correct: The response is consistent with the given knowledge and does not generate the facts that are not given. Well-pare well-paraphrased The response is well- rell-paraphrased rather than just copying and pasting the given knowledge.
3.4
WeN
33
CHUDeG
FoCus
Table 10: An example of the questionnaire for human evaluation.
Figure 6: Manual search the threshold filtering module. DAE score in the y-axis indicates the scource- faithfulness score after refining the utterances under the threshold.
H Prompts for LLM and LLMREM
The prompts for refining utterances with LLM and LLMReM are in Table 11.
E Prompts for Adversarial Data Generation with ChatGPT
L.L.
"You are tasked with refining the response given the knowl- edge, question, and response. First, you determine whether the response factual response given tion. the response is factual response considering the given knowledge and question, oulpul as il is, and not, regenerate it factually refining the response considering given knowl- edge and question. You should only write "Output" the format given without saying why. Knowledge: (knowledge) Response: (response)."
The prompt used for generating adversarial data with ChatGPT is Table 8:
knowledge and ques-
Prompt
"Considering the knowledge, persona, and dialogue history be- low and make the unswer incorrect terms factuality. Knowl- edge: (knowledge), dialogue history: answer: [answer]."
LLMass
"You are tasked with refining the response given the knowledge. question, resuonse. You determine whether response factual response considering the given knowledge and question, and if not, extract the necessary entities to refine the resporse more factually from knowledge. Then, refine and regener- response into factual response considering the given knowledge and question using the extracted entities. You must write both "Entities" and "Outpul" and only "Entities" and "Out- put" according the given format. Knowledge: (knowledge) Response: (response)"
Table 8: The prompt used for generating adversarial data.
F Refining Utterances of Larger Models
Sourte-aitt
Ref. Matching
Table 11: The prompts for refining utterance by LLM and LLMREM-
Data
Model
TC
DAR
chri
R-I.
FoCus
LLMg
26.72
38
40.64
BART HAMM
46.06
28.76
0.91
40.95
42.36
WoW
LLM
6.91
5.87
0.36
18.68
18.65
BART nEMicee
15397
11.15
0.77
31.48
27.44
CMUDaG I.L.MoC
4.77
7.24
0.29
14.36
13.44
BART geMoees
6.27
9.63
0.43
16.10
12.38
Table The result of BART REMorgg (denoted with +)
refining the KGC utterances of large language model (LLMxgc). The utterances are filtered with r = 0.5

---

I REM Generation Examples
FoCus
[Knowledge]
The Václav Havel I Fihrary, located in Prugue, is charitable organizution founded by Dugmar Havlovi, Karel Schwarzenherg and Miloslav Petrusek on 26 July 2004.
[Persona Knowledge]
1 love going to libraries. love reading. want to visit Prague. want to visit New York. love books. [Utterance]
BARThwe Library founded 2004.
REM Václav Havel Library founded July 2004 Dugmar Havlová, Karel Schwarzenberg and Miloslav
Petrusck.
[Knowledge]
The design of the house is attributed John Verge and John Bibb and built from 1835 to 18399 Jumes Hume.
[Persona Knowledge]
love colonial architecture. like Australia. I would like to visit muscum. would love to see grotto. have read about the
Codigal people.
[Ulterance]
BART was from35 18 to39
1835 by James Hume. got information on internet. was designed by John Verge and John Bibb.
REMoase was
WoW
[Knowledge]
A tattoo is a form of body modification where a design made by inserting ink, dyes and pigments, either indelible or temporazy, into the dermis layer of the skin to change the pigment. [Utterance]
BART Muse I'mm REM of course, form of body mudification where design is made by inverting ink, dyes und pigments into the dermis layer of the skin to change the pigment.
it really tattoo artist, I do know that
[K [Knowledge]
It was also applied to the pit-style cooking techniques now frequently used in the Southeastern United States. [Utterance]
BARTise: :. I'll have to try it sometime! I've heard it's used in the + REM :, don't know if I'd ever try it. know that pit-style cooking techniques are now frequently used in the Southeastern United States.
CMUDoG
[Knowledge]
Leonardo DiCaprio as Dom Cobb, professional thief who specializes in conning secrets from his victims by infiltrating their dreams. Joseph Gordon-Levitt as Arthur, Cobb's 's partner who manages and researches the missions. ... Tom Hardy as Tames, sharp-tongued ussociate of Cobh. wildly ingenious chess game, [Utterance]
BARTT buse have you the movie inception?
+ REM ;, I'm good, how you? Have you seen Inception? It's science fiction movie starring Leonardo DiCaprio, Joseph Gordon-Levitt, and Tom Hurdy.
[Knowledge]
Angelina Jobie Maleficent Sharlto Copley Stefan Fille Fanning Aurora Sam Riley Diaval Imeldu Staunton Knolgrass Maleficent demolishes stereotypes... Robert Stromberg SuperHero Malefieent is a 2014 American dark fantasy film directed by Robert Stromberg from screcuplay by Linda Woolverton, and starring Angelina Jolie as Malefieent with Sharlto Copley, Elle Fanning, Sam Riley. Imelda Staunton, Juno Temple, and Lesley Manville in supporting roles...
[Utterance]
BART have you seen Maleficent before?
Maleficent. Have you seen It's fantasy from It Angelinu Jolie, Sharito Copley, Elle Fanning, Sam Riley, and Imelda Staunton.
REMaas: we're talking
Table 12: Examples of the fine-tuned baseline utterances refining by REM KGC datasets.

---

FoCus
[Knowledge]
Elizabeth Bay House is a heritage-listed Colonial Regency style house and now museam and grotto, located at 7 Onslow Avenue in the inner eastern Sydney suburb of Elizabeth Bay in the City of Sydney local government area of New South Wales, Australia. [Persona Knowledge]
love colonial unchitecture, like Australia. I would like to visit museum. would love to see grotto. I have read about the Codigal people.
[Utterance]
BART base Elizabeth House located Australia which like. is place you like
+ REEMury This place is called Elizabeth Bay House and it is located in Australia, which you like. It is a heritage-listed Colonial Regency style house now a museum and grotio.
[Knowledge]
The Memphis Zoo, located Midiown, Memphis, Tennessee, United States, home to more than 3,500 animals representing over 500 different species.
[Persona Knowledde]]
hope visit Tennessce. like to ride on tram. I have never seen wetland. I would like to see Giant Panda. I wish I could see penguin.
[Utterance]
BART iis Memphis in, is place hope visit.
+ REMarge It's the Memphis Zeo in Tennessce, a place you hope visit. It home more than 3,500 animals, 500 different species.
WoW
[Knowledge]
Competing theories state that Hindu identity developed in the British colonial era, or that it developed post-8th century CE after the Islamic invasion and medicval Hindo-Muslim wars. [Utterance]
BARTiwe :. I.I'm fluent English, understand Hindu identity developed in
RREMuege No, I'm not fluent in English, but understand that Hindu identity developed in the British colonial era, that it developed post-8th century CE.
[Knowledge]
Human cccupation of Qatar dates back to 50,000 years ago, and Stone Age cacampments and tools have been uncarthed in the
peninsula.
[Utterance]
BART 's "s pretty husy place, I've been to Qatar few times, It's
+ REMirrge : That's true. It's pretly huxy place, but it's been uround for a long time. Human occupation Qutar dates back 50,000 years ago, and Stone Age eneampments and tools have been found in the peninsula.
CMUDoG
[Knowledge]
Sally Haw kins as Elisa Esposito, mute cleaner who works at secret government laboratory.... ... The Shape of Water Rotten Tomatoes: 92% and avernge: 8.4/10 Metacritic Score: 87/100 CinemuScore: A 2017 [Utterance]
BART have you seen The Shape of Water? + R.EM Hello, Have you seen The Shape Water? It's really good movie. It has 92% Rotten Tomatoes and Metacritic Score of 87/100.
[Knowledge]
Lindsay Lohan Cady Heron Rachel McAdams Regina George Lacey Chabert Gretchen Wieners, Amanda Seyfried as Kuren Smith Lixeyy Caplan Janis lan Boasts one tw three punch in star Lindsay Lohan, screenwriter Tima and directer Mark Waters, and, indeed, delivers knockout The screenplay was highly praised by crities with Peter Travers of Rolling Stone calling it ccoicc gold'... [Utterance]
BART :. what did you think of it?
+ REMarrge I really liked it, I thought Lindsay Lohan did great job as Cady Heron. Tima Fey did an amaziny, joh wriling the screemplay as well.
Table 13: Examples of the fine-tuned baseline utterances refining by REMarge on KGC datasets.

---

FoCus
[Knowledge]
hesselberg pronounced [ about this soundlisten ) 689 above sea level the highest point middle franconia and the franconian jura and situated 60 km south west of nuremberg, germany.
[Persona Knowledge]
love to enjoy nature. live near Nurembeng, I Tam from Germany. like heights. I stndy geology. [Utterance]
INFO: The name of this place is Hesselberg,
+ IEE The name of this place is Hesselberg, and it is located 60 km south west of nuremberg, germany, the country you are from.
[Knowledge]
st. george's 's united methodisa church, located the commer and new streets, old city neighborhoood of philadelphia, oldest methodist church in continuous use united states, beginning in 1769.
[Persona Knowledge]
will stay in Philadelphia. am going to Ponnsylovania. I like history. I seligious. I am interested in the American Revolation. [Utterance]
INFO : This is St. George's United Methodist Church you may know of it because you are religious.
4 REMarge : This St. George's United Methodist Church, you may know of because you are religions. It is the oldest methodist church in continuous use in the united states. WoW (KILT)
[Knowledge]
Red Ketchup is cult Quchec comic book series featuring FBI's crazed rogue agent, Steve RedKetchup. The series was created by Pierre Fournier and Réal Godhout, and will soon be adupted into a live action feeature film by Martin Villeneuve for GO Films in Montreal.
[Utterance]
EDMem: : ketchup red is a sweet sauce made from tomatoes.
+ REM love red ketchup, which is a cult Quebee comic book series featuring FBI's crazed rogue agent, Steve kedKKtthhu.. The series created by Pierre Fournier and Réal Godbout
[Knowledge]
body negatively affects child's health well-being, As methods to deternine hody fat directly difficult, the diagnosis of ohesity often hased on BMI. Due the rising prevalence obesity its many adverse health effects it is being recognized as serious public health concern. The term overweight rather than obese is often used when discussing childhood obesity. especially in open discussion, as it is less stigmatizing. [Utterance]
Childhood obesity condition where
in children and
EDMem:: : i'm not sare, but i do know that in the united states, overweight people are more likely to he obese than healthy people. + REM : Yea i'm not sure, but the prevalence of obesity in children and its many adverse health effects is being recugnized serious public health concern.
CMUDoG
[Knowledge]
Leonardo DiCaprio us Frank Abagnale, Jr. Tom Hanks as Carl Hanratty Christopher Walken as Frank Abugnale, Sr. Nathalie Baye as Paula Abaguale.... ... Catch me if you can Rotton Tomatoes: 96% and average: 7,9/10 Metacritic Score: 76/100 CinemaScore: A- 2002 [Utterance]
ITDD: yes, has 96 rating rotten tomatoes.
REMaage yes, has 96 percent rating rotten tomatoes and 7.9/10 on metacritic and a cinema score of an A- which pretty good.
[Knowledge]
Mike and Sulley are led away by the CDA, and expelled from the university as result of their actions, hut the other members Oozma Kuppu accepted into the scure program the next semester, as Hardserabble was impressed their performances the Games, As Mike leaves on the bus, Sulley runs after him to raise his spirits. Hardscrabble then appears and wishes the two luck, claiming they were the first students to have surprised ber. The two take jobs in the mail room of Monsters, Inc., eventually working their way up to join the scarer team. [Utterance]
ITDD: they do
REMarge Yes, they do. Sulley good friend Mike's and tries to cheer him on when he leaves on the bus to try to raise his spirits.
Table 14: Examples of the existing model utterances refining by REMiarge on KGC datasets.


Reviews:
[{"review_id": "CODJKfXQRB", "reviewer": "Reviewer_fxJP", "summary": "", "questions": "\nA question that was haunting me throughout the paper is whether we would see a comparable increase in truthfulness to the knowledge source if the REM model was directly implemented as KGC model instead of a refinement model that is used for post-hoc improvement on top of the actual KGC model.\n\nFurthermore, what influence does the quality of the utterance to be refined (û) have on the REM output quality? is there a relation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions. We sincerely appreciate your time reading the paper, and our responses to your comments are below.\n\n&nbsp;\n\n    Reasons To Reject: The choice of putting the related work and the implementational detail into the appendix is irritating.\n\nRe: Due to the page limit, we had to put the more important sections (e.g., Proposed Methods, Experiments) on the main pages. However, we agree that the Related Work and Implementation Details will help improve the understanding of the readers. We will place it on the main pages with the extra page given if our manuscript is accepted.\n\n&nbsp;\n\n    Questions For The Authors: A question that was haunting me throughout the paper is whether we would see a comparable increase in truthfulness to the knowledge source if the REM model was directly implemented as KGC model instead of a refinement model that is used for post-hoc improvement on top of the actual KGC model.\n\nRe: In general, it is known that direct training helps to increase performance [1]. However, in this case, based on experimental results that show better performance with refinement only when needed than with refinement even when not needed, direct training may decrease performance and efficiency because it forces the model to refine every case. As a result, we aim to refine only the failed utterances, which are not source-faithful, in a post-hoc manner. Additionally, this method ultimately reduces the number of training parameters because it does not require additional training of existing models, and it can be applied to models with undisclosed parameters, such as ChatGPT, in a plug-and-play manner.\n\n&nbsp;\n\n    Questions For The Authors: Furthermore, what influence does the quality of the utterance to be refined (û) have on the REM output quality? is there a relation?\n\n\nRe: We observed that the length of the utterance to be refined $\\hat{u}$ influences the refined output; inputs with shorter lengths tended to have shorter outputs. We will add this observation to improve our paper.\n\n&nbsp;\n\n    Typos Grammar Style And Presentation Improvements:\n\nRe: Regarding the typos, we appreciate you for letting us know. We will modify it and improve our manuscript.\n\n&nbsp;\n\nReferences\n\n[1] Ramos, A. G. C. P., Mehrotra, A., Lane, N. D., & Bhattacharya, S. (2021, October). Conditioning sequence-to-sequence networks with learned activations. In *International Conference on Learning Representations*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They take an interesting approach to refine utterances that were already produced, making the system quite versatile. The performance improvement in regard to knowledge source truthfulness is good. The paper is well written and mostly easy to understand.", "reasons_to_reject": "The choice of putting the related work and the implementational detail into the appendix is irritating.", "questions_to_authors": "\nA question that was haunting me throughout the paper is whether we would see a comparable increase in truthfulness to the knowledge source if the REM model was directly implemented as KGC model instead of a refinement model that is used for post-hoc improvement on top of the actual KGC model.\n\nFurthermore, what influence does the quality of the utterance to be refined (û) have on the REM output quality? is there a relation?", "paper_topic_and_main_contributions": "The authors implement and test a refinement model for utterances generated by Knowledge grounded conversation (KGC) models.\nIN KGC, hallucination, especially entity-level hallucination, is a major issue that makes conversation fail completely. The authors design a REM model for refining non-truthful utterances produced by some KGC model using a bipartite training strategy, making use of a NER head and a language generation head, both trained with a separate log likelihood loss. The overall loss being the sum of the two. They define their refinement model probabilty P(u_refine|k, û) as proportional to the product P(e|k,û) P(u_refine|e,k,û). \n\nSo, the REM model uses a mining module to extract information, takes the non-optimal output of the KGC model, scores this for unfaithfulness against the knowledge source and then conditions on both in order to produce the revised utterance.\n\nNumerous different metrics were used for evaluation, specialized for knowledge truthfulness, on numerous corpora.\nREM decreases diversity metrics, since refinement removes hallucinations that may increase those metrics.\nBUT REM strongly improves source-faithfulness! \nThe ablation study shows that without the entity miner, performance decreases (though still better as baseline), therefore entity mining is of essential importance for REM performance and source faithfulness of KGC.\n\nHuman evaluation is performed as well, showing that in majority, the refined utterances are rated best. Concepts of evaluative criteria are opaque, but this is an inherent problem of metrics for human evaluation and not an issue of this paper.\n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "p. 2 line 122: incomplete sentence, I think one of the authors accidentially deleted a sentence segment here."}, "extra_text": {}}, {"review_id": "3Mmwd836ns", "reviewer": "Reviewer_uiSe", "summary": "", "questions": "The methodology for REM-LLM is not very clear, it would benefit from adding implementation details. Will the plug-and-lay system be shared along with trained models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank the reviewer for their _insightful comment_ and detailed feedback on _missing references_, _typos_, and _presentation improvements_. They are exceedingly helpful for us to improve our paper. Please find our responses in the following.\n\n&nbsp;\n\n    Reasons To Reject: Human evaluation is done for a few samples per dataset and can be improved.\n\nRe: While it would be most objective to perform a human evaluation on all test data samples, given our financial situation, it was our best to hire three annotators to evaluate 100 samples. In particular, we believe that a sample of 300 completely randomly selected test data, without any manipulation, is sufficiently representative of our overall experimental results, given other studies that have conducted human evaluations in the dialogue domain and model generation [1, 2].\n\n&nbsp;\n\n    Reasons To Reject: Important sections are in the appendix, particularly the literature review, which situates this work.\n\nRe: Due to the page limit, we had to put the more important sections (e.g., Proposed Methods, Experiments) on the main pages, but we will put a literature review on the main pages for the completeness of the manuscript following your comments, when we are given one extra page.\n\n&nbsp;\n\n    Questions For The Authors: The methodology for REM-LLM is not very clear, it would benefit from adding implementation details.\n\nRe: We attached the prompts used for REM-LLM in Appendix H, but we will add more detailed implementations regarding the information of the prompts and the models.\n\n&nbsp;\n\n    Questions For The Authors: Will the plug-and-lay system be shared along with trained models?\n\nRe: Our plug-and-play method can be used to refine utterances generated by any other models that have already been trained, or it can be trained with existing KGC models from scratch by jointly minimizing the training loss (KGC loss and REM loss).\n\n&nbsp;\n\nReferences\n\n[1] Li, Y., Peng, B., Shen, Y., Mao, Y., Liden, L., Yu, Z., & Gao, J. (2021). Knowledge-grounded dialogue generation with a unified knowledge representation. *arXiv preprint arXiv:2112.07924*.\n\n[2] Adolphs, L., Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2021). Reason first, then respond: Modular generation for knowledge-infused dialogue. *arXiv preprint arXiv:2111.05204*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a model to improve source faithfulness of generated utterances using the knowledge source, which can be used on top of LLMs. \n\nThe results show an improvement from using REM over baselines, the paper builds a strong case using multiple evaluation metrics. \n\nThe work also explains the reason for performance differences, including dataset compositions.\n\nThe paper lays out limitations well.\n\nThe models will be shared to reproduce results. \n\nThe work considers the carbon footprint of the training. ", "reasons_to_reject": "Human evaluation is done for a few samples per dataset and can be improved. \n\nImportant sections are in the appendix, particularly the literature review, which situates this work. ", "questions_to_authors": "The methodology for REM-LLM is not very clear, it would benefit from adding implementation details. Will the plug-and-lay system be shared along with trained models?", "paper_topic_and_main_contributions": "The work aims to address the challenge of hallucinations in language generation systems that are not faithful to the source information through an entity refinement method applied to generated utterances. The proposed system, Refining Method by Entitiy Mining (REM), seeks to enhance the quality of generated utterances. REM achieves this by leveraging source knowledge, i.e., mining key entities from the original source and implicitly using these mined entities to refine utterances, aiming to increase faithfulness to the original source of information. \n\nThe first step is identifying the utterances for refinement, done via a scoring method, DEA, which generates a score between 0 and 1 between the utterance and knowledge source. The candidates for refinement are filtered based on a score threshold, which is manually tuned and set to 0.5 for most settings for the paper. Post the filtering process, REM consists of two modules, an entity miner and an utterance generator, both trained in a multi-task setting with a joint training function. \n\nThe paper presents results on FoCus, WoW, and CMUDoG datasets. Four knowledge-grounded conversation (KGC) baselines are used, BART, EDMem, ITDD, and INFO, presented compared to REM variants. Results are shown for cross-data refinement, adversarial refining, and ablation studies. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "It will be helpful to discuss recent work focusing on knowledge-grounding, including the following, in relation to the proposed work: \n\nKim, S., Kwon, O. W., & Kim, H. (2020). Knowledge-grounded chatbot based on dual Wasserstein generative adversarial networks with effective attention mechanisms. Applied Sciences, 10(9), 3335.\n\nZhang, Y., Fu, H., Fu, C., Yu, H., Li, Y., & Nguyen, C. T. (2023, June). Coarse-To-Fine Knowledge Selection for Document Grounded Dialogs. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.\n\nWang, W., Gao, W., Feng, S., Chen, L., & Wang, D. (2021, October). Adaptive posterior knowledge selection for improving knowledge-grounded dialogue generation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 1989-1998).\n\nWu, T. W., & Juang, B. H. (2022, May). Knowledge Augmented Bert Mutual Network in Multi-Turn Spoken Dialogues. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7487-7491). IEEE.\n\nGrassi, L., Recchiuto, C. T., & Sgorbissa, A. (2022). Knowledge-grounded dialogue flow management for social robots and conversational agents. International Journal of Social Robotics, 14(5), 1273-1293.", "Typos_Grammar_Style_and_Presentation_Improvements": "General comments: \n\nSections of the paper can be hard to parse, particularly sections 2, 3, and 4. \n\nAcronyms are introduced without the full form, e.g., DEA, PLM, etc. \n\nKey sections on related work and implementation are in the appendix. \n\n\nTypos: \n\n038: a large number of training corpora\n\n122: incomplete sentence\n\n162-164: typo in the value of tau\n\n187-189: suggest rewriting the sentence for clarity \n\n205: missing punctuation or incomplete sentence\n\n211: consider omitting either both or two\n\n263: add space between parentheses\n\n275: produce fluent “text”\n\n379: table 2 (not 9)\n"}}, {"review_id": "zh8bdzOX3c", "reviewer": "Reviewer_9ska", "summary": "", "questions": "A: For the LLM experiment (Section 4.6), are the LLMs just used to refine the BART responses? What happens when the LLMs generate their own responses from scratch, and REM is used to refine LLM responses? E.g. it would be interesting (but may not be the case) if REM (BART-large, 400M parameters) is still able to improve the responses of a larger model.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We sincerely appreciate your time in reading the paper. Please find the responses below.\n\n&nbsp;\n\n    Reasons To Reject: It seems that the approach is only tested on refining BART-large model responses (400M parameters). It is unclear whether the approach would still be effective for refining LLM responses (e.g., open-source GPT-style models with >1-10B parameters, or responses from API calls to larger LLMs).\n\nRe: We believe that the reason for the rejection is outside the scope of our paper, as our goal was not to use the smaller model to correct the larger model's responses or to correct all the answers, but to refine only the erroneous utterances in a post-hoc manner to reduce the entity hallucination during the knowledge grounded conversation.\n\nNevertheless, following your suggestion, we have conducted a refining experiment of LLM (GPT-4; > 1 trillion parameters) responses. We first generated the KGC response $\\hat{u}$ with LLM. Then, we made LLM mine the key entities from the knowledge source and refine the given utterance. The prompts used in this experiment are the same as in Appendix H.\n\nAs a result, all samples in the test set are refined with $\\tau$=0.5. In the FoCus dataset, the refined utterances (marked with '+' in Table) show about a 2-3% increment in EC and TC and a 5-7% increment in DAE. Also, in WoW and CMUDoG, the result shows a similar tendency with the experiments in the paper, and our refining method helps improve the source-faithfulness scores (e.g., EC, TC and DAE)as shown below. \n\n\n\n|Data|Model|EC|TC|DAE|chrF|R-L|\n|--- |--- |--- |--- |--- |--- |--- |\n|FoCus|LLM_KG|42.74|26.72|0.84|38.16|40.64|\n||+LLM|44.95|27.45|0.89|39.61|42.20|\n||+LLM_REM|**45.83**|**28.46**|**0.91**|**40.01**|**41.98**|\n|WoW|LLM_KGC|6.91|5.87|0.36|18.68|18.65|\n||+LLM|9.27|6.73|0.47|21.24|20.17|\n||+LLM_REM|**13.28**|**9.25**|**0.63**|**25.17**|**21.19**|\n|CMUDoG|LLM_KGC|4.77|7.24|0.29|**14.36**|**13.44**|\n||+LLM|5.61|8.53|0.38|**14.36**|12.21|\n||+LLM_REM|**7.88**|**14.26**|**0.54**|**14.36**|10.04|\n\nThis experiment confirms that LLM utterance suffers from entity hallucination in generating answers as in [1] and it shows the need for post-hoc refining. This result will be further included in Appendix of the paepr.\n\n&nbsp;\n\n    Questions For The Authors: For the LLM experiment (Section 4.6), are the LLMs just used to refine the BART responses? \n\nRe: Yes, LLM was used to refine the BART responses in Section 4.6. Also, in Section 4.3, we show the case that BART was used to refine the adversarial utterances of LLM. \n\n&nbsp;\n\n    Questions For The Authors: What happens when the LLMs generate their own responses from scratch, and REM is used to refine LLM responses? E.g. it would be interesting (but may not be the case) if REM (BART-large, 400M parameters) is still able to improve the responses of a larger model.\n\nRe: While the performance of a generative model is largely affected by its size, in terms of source-faithfulness, we believe that even smaller models can contribute because they are used only when refining the *erroneous* utterances of larger models, not refining all cases. However, as you mentioned, it would be interesting to see how the smaller model improves the responses of the larger models. Along with the experiments of LLM-to-LLM refining above, we plan to conduct and report additional experiments where the smaller model refines the larger models’ response to understand the performance and limitations of our method when we are given an extra page. We thank you for the insights and for helping us improve our paper.\n\n&nbsp;\n\nReferences\n\n[1] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.Chicago", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "REM improves the entity coverage of model responses relative to source knowledge. The ablation study demonstrates that the NER sub-task is helpful for REM.", "reasons_to_reject": "It seems that the approach is only tested on refining BART-large model responses (400M parameters). It is unclear whether the approach would still be effective for refining LLM responses (e.g. open-source GPT-style models with >1-10B parameters, or responses from API calls to larger LLMs).\n\nDiscussion period update: the authors have presented results for GPT-4, demonstrating that their REM method can still improve the factuality of responses. This is useful given the widespread usage of these larger models for chat systems (excitement score increased to 3).\n\nI agree with other reviewers that much of the related work and implementation sections in the appendix should appear in the main text.", "questions_to_authors": "A: For the LLM experiment (Section 4.6), are the LLMs just used to refine the BART responses? What happens when the LLMs generate their own responses from scratch, and REM is used to refine LLM responses? E.g. it would be interesting (but may not be the case) if REM (BART-large, 400M parameters) is still able to improve the responses of a larger model.", "paper_topic_and_main_contributions": "This paper proposes REM, which uses an encoder-decoder model with named entity recognition (NER) to refine a language model response for increased faithfulness to provided source knowledge. The encoder is trained to extract named entities from the source knowledge; jointly, the decoder is trained to refine the original response given the source knowledge and extracted entities from the encoder. REM improves the entity coverage of model responses.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Interesting post-hoc refinement technique, potentially extendable to other knowledge-grounded generative tasks. It also contains important checks for source-faithfulness or provenance.
Decision: Accept-Main