Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment
Authors: Boyang XUE
Weichao Wang
Hongru WANG
Fei Mi
Rui Wang
Yasheng Wang
Lifeng Shang
Xin Jiang
Qun Liu
Kam-Fai Wong
Venue: EMNLP 2023 Findings (2023)
Keywords: knowledge-grounded dialogue system
factual consistency
knowledge enhancement

Abstract: Pretrained language models (PLMs) based knowledge-grounded dialogue systems are prone to generate responses that are factually inconsistent with the provided knowledge source. In such inconsistent responses, the dialogue models fail to accurately express the external factual knowledge they rely upon. Inspired by previous work which identified that feedforward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability of FFNs by knowledge enhancement and alignment respectively. We first propose K-Dial, which explicitly introduces extended FFNs in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs. Additionally, we apply the reinforcement learning for factual consistency (RLFC) method to implicitly adjust FFNs’ expressions in responses by aligning with gold knowledge for the factual consistency preference. To comprehensively assess the factual consistency and dialogue quality of responses, we employ extensive automatic measures and human evaluations including sophisticated fine-grained NLI-based metrics. Experimental results on WoW and CMU\_DoG datasets demonstrate that our methods efficiently enhance the ability of the FFN module to convey factual knowledge, validating the efficacy of improving factual consistency for knowledge-grounded dialogue systems.

Improving Factual Consistency for Showledge-Groundee Dialogue Systems via Knowledge Enhancement and Alignment
Boyang Xue Weichao Wang",, Hongru Wang', Fei Mi², Rui Wang', Yasheng Wang', Lifeng Shang', Xin Jiang', Qun Liu', and Kam-Fai Wong".
The Chinese University of Hong Kong HHuawei Noah's s Ark Lab
Harrbin Institute of Technology, Shenzhen, China (byxue, kfwong)@se cuhk. edu. wangwei manggeicchao@hhuwwi...mm com
Abstract
The champion is france
The champion is Angentina
Pretrained language models (PLMs) based mnowledge-grounder dialogue systems are prone generate respouses that are factually in- consistent with the provided knowledge source. such inconsistent responses, models fail to accurately express the externl factual knowledge they rely upon. Inspired by previous work which identified that feed- forward networks (FFNs) within Transformers are responsible for factual knowledge expres- sions, investigate methods efficiently improve the factual expression capability of FFNs by knowledge enhancement and align- ment respectively. We first propose K- DIAL, which explicitly introduces extended FFNs in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs. Addi- tionally, we apply the reinforcement learning for factual consistency (RLFC) method to im- plicitly adjust FFNs' expressions responses by aligning with gold knowledge for the fac- tual consistency preference. To comprehen- sively assess the factual consistency and dia- employ exten-
①L K-DIAL
Enhancing [Argentina)
Dialogue Model
Dialogue Model
RLFC
Aligning with (Argentina)
dialogue
[Knowledge]:
Argentina won the 2022 World Cup champion. Which team won the 2022 World Cup champion?
[Contexts]:
Figure 1: An illustration of enhancing the factual knowl- edge expression to tackle the inconsistency problem for knowledge-grounded dialogue system in this work.
and reliable responses in many conversational ap- plications (Dinan et al., 2019; Moghe al., 2018; Ghazvininejad et al., 2018; Gopalakrishman et al., 2019). However, these knowledge-grond logue systems (KDS) are always hampered by fac- tual inconsistency or even "hallucination" problem (Santhanam et al., 2021; Ji al., 2023), which has been widely investigated in many natural language generation (NLG) tasks such as abstractive sum- marization (Zhu et al., 2021; Nan et al., 2021: Xic et al., 2021; She et al., 2023) and machine transla- tion (Lee et al., 2019). The factually inconsistent responses produced by dialogue models are linguis- tically fluent and contextually coherent but deviate from the grounding factual knowledge, exempli- fied in the left hand of Figure 1, potentially leading to misinformation to the users and restricting the applicability of dialogue agents.
dia-
logue quality of responses, sive automatic measures and human evalua- tions including sophisticated fine-grained NLI- based metrics. Experimental results on WoW and CMU DoG datasets demonstrate that our methods efficiently enhance the ability of the FFN module to convey factual knowledge, val- idating the efficacy of improving factual con- sistency for knowledge-grounde dialogue sys- tems.'
The factual consistency in KDS indicates "ac- curately portrays the input knowledge (assuming the provided knowledge is correct)" as defined in prior research (Santhanam et al., 2021). Identi- fying the intrinsic causes of factual inconsistency in KDS remains persistently challenging, as the generated responses are jointly affected by con- versational history, grounding knowledge, and dia- logue PLMs. Generally, the dialogue context and
1 Introduction
Pretrained dialogue models with the assistance of external knowledge sources have demonstrated re- markable performance to generate knowledgeable
"Equal contributions. Corresponning author.
'Our code has boen released on http://gituub..o//Amoo Faltz/FactDial..

---

grounding knowledge are assumed to be accurate and considered as ground truth, the factual inconsis- tency thus can naturally attributed to the innate limitations of PLMs. The prior knowledge in PLMs learned from large-scale unlabeled corpus might be incomplete, outdated, or incorrect (Elazar et al., 2021; Cao et al., 2021) thus inevitably affecting the provided factual knowledge expressions, and consequently results in untrustworthy responses as shown in Figure 1, where the knowledge stored in the model is likely to be "France won the World Cup champion." ". Therefore, it is essential to fig- ure out the mechanism by which language mod- els express factual knowledge. Previous research (Geva et al., 2021a; Dai et al., 2022a) observed that feed-forward networks (FFNs) in Transformers can be viewed key-value memories that store and activate specific knowledge representations given certain input patterns. Accordingly, we propose two promising solutions to bolster FFNs ability to produce factual knowledge and enhance factual consistency for KDS.
'hee obtained consistency score of the reward model is utilized for RLFC training to induce factual ex- pressions within FFNs.
To assess the factuality and conversationality of dialogue generations, we conduct a compre- hensive evaluation employing both automated and human evaluations, including our carefully de- fined finely-grained NLI metrics based on recent human-annotated datasets released by Labadic al. (2021); Dziri et al. (2022); Gupta et al. (2022). Significant performance improvements across the aforementioned metrics are obtained on both WoW (Dinan et al., 2019) and CMU DoG (Zhou et al.. 2018) datasets using K- K-DIAL and RLFC, demon- strating their efficiency in improving the expres- sions of factual knowledge within FFNs and miti- gating the risk of factual inconsistency for KDS. Our contributions are summarized as follows: (1) We propose K-DIAL, which explicitly ex- tends FFN modules in Transformers for knowledge enhancement to express factual knowledge in sponses to improve factual consistency while main- taining conversational properties.
First, we propose K-DIAL, a knowledge- enhanced dialogue generation method that explic- itly incorporates extended FFN modules in Trans- formers for knowledge enhancement, which im- proves the model's ability to express the gold knowledge given specific knowledge snippets and dialogue contexts. As illustrated in Figure 1, the factual knowledge "Argentina won the 2022 World Cup champion." " with the contexts is directly used to enhance the expression of "Argentina" in the re- sponse. Notably, the parameters in extended FFNs are updated solely over the mnoleedgerelltted to- kens occurring in both grounding knowledge and
(2) We propose RLFC, which implicitly pro- motes FFNs' ability of factual expressions by align- ing generated responses with the gold knowledge for factual consistency preference of KDS.
(3) We obtain significant improvements across range of sophisticated automatic and human eval- uation metrics, demonstrating the efficacy of our two proposed methods in achieving superior per- formance in terms of both the factual consistency and dialogue quality of KDS.
Methodology
responses, ensuring the efficiency of improved fac- tual consistency while maintaining the dialogue quality of KDS.
In this section, we first introduce the KDS models in this work and pose the view of key-value mem- ories of FFNs in Transformer models. We then present our knowledge-enhanee tion method K-DIAL and reinforcement learning for factual consistency (RLFC) technique respec- tively.
Second, we propose the reinforcement learning for factual consistency (RLFC) technique, which leverages alignment with the factual consistency preference to implicitly enable FFNs to express fac- tual knowledge in responses. As shown in Figure I, the response is aligned with factual knowledge "Argentina won the 2022 World Cup champion." " to implicitly adjust FFNs to express accurately. The reward model is utilized for alignment which is binary NLI model as a factual consistency classi- fier trained on publicly available benchmarks (San- thanam et al., 2021; Gupta et al., 2022; Dziri et al., 2021) for factual consistency evaluations of KDS.
dialogue genera-
2.1 Knowledge-Groundee Dialogue Model As depicted in Figure 2, the causal graph illustrates the procedure of KDS generation where response y is jointly determined by dialogue history X. re- trieved knowledge K. and PLM M. In this study, we employ GPT2 (Radford et al., 2019) as PLMs and fine-tune these models on grounded dialogue datasets and obtain the dialogue model. The in-

---

K-DIAL
Dialogae Contexts,
[Respetse] (P)
External Knowledge.
Dialogue Model.
Feed- forward Network
Genersted Responses.
Nx
Multi-head Self-Attention
3
4
Word Embeeding Layer
K
Aggentin was dampio..
0e3022 Cup derpoee
denpppo snlugostau
X
[Knowldgee] (K)
[Contex]]
[Response] (Y)
(a)
(b)
Figure 2: An illustration of (a) a causal graph denoting the process of knowledge-grounded dialogue generations. where factual inconsistency issue this work is attributed in M and procedure while others are assumed correct; (b) our proposed K-DIAL framework in dialogue model with sample.
put to the model concatenates a piece of knowl- edge K and a dialogue history X consisting of utterances that are segmented by the speaker types <bot> and <user>. Distinct special token-type em- beddings employed to delineate each part of the input for all GPT-2 models. For simplicity, directly leverage the gold knowledge in this work thus the input knowledge naturally correct. The dialogue model trained to generate the response 1V, V/,--..... ym given the input via minimiz- ing the cross-entropy loss:
activation function. The bias terms are omitted.
Geva al. (2021b) pointed that O in FFNs cor- responding to keys multiplied with h' to yicld d memory coefficients. Each individual key k O can capture a textual pattern across the input prefix [h1. ... 1a/ and only be triggered upon the occur- rence of specific input patterns. The coefficients are then employed to compute the weighted sum with values O to induce a distribution over the vocabulary of the next token prediction. Dai et al. (2022a) further proposes the concept of knowledge neurons in FFNs that store and activate the factual knowledge prediction. The observations provide insight to improve factual consistency for KDS by augmenting PLMs to recall and output factual knowledge in responses.
m LCE
lo X, K) (1)
m
2.2 Key-Value Memories in FFNs
Prior studies have exhibited PLMs are knowledge base (Petroni al., 2019) and the knowledge is implicitly preserved in the parameters of FFNS in Transformers (Dai et al., 2022a). Generative PLMs, such as GPT-3 or GPT-2 (Brown et al., 2020; Rad- ford et al., 2019), feature a deep stacking of mul- tiple Transformer decoder blocks (Vaswani et al., 2017). As shown in Figure 2 (b), each feed- ferw-ffrwwrr network (FFN) module in a Transformer block con- tains two-layer linear model with activation function between. Assume that h dm rep- resents the i-th hidden input of the FFN module in the I-th Transformer layer with dm dimdeimnnoonn word embedding... The normalized h is then fed into FFN as:
2.3 K-DIAL: Knowledge-Enhanced Dialogue Generations
KDS is supposed to generate more reliable and knowledgeable responses for knowledge-intente intensive situations leveraging the wealth of information in external knowledge. Even though gold knowledge is provided, the models still encounter challenges related to fictional expressions of gold knowledge they rely on, and resulting in factual inconsistency. for example, manifesting in the responses that are factually incorrect or uninformative. As shown Figure 2 (a) where X' and K. are considered always correct, we can naturally infer that the inconsis- tency arises from M.
FFN = O', Act(O)/ 1)))
(2)
As the knowledge in PLMs inevitably contains inaccurate, outdated, incomplete redundant infor- mation (Elazar et al., 2021; Cao et al., 2021), which may influence factual knowledge predictions of
where O E dxddm O E dh xd denote the weight matrices of the FFNs and Act(.) is the

---

Reward
KL-div
Binay
Knowledge- Chooledde-Respoosee Pairs Set
Clasilirr Toining
Factual Ccmsistency
deacke
Posibire Samples
Reward Model
[Reupune] (P1)
[Roperec] (Y')
(Knoowedgg)) [Cousatert Respanse)
Policy Tniniing
Policy
Reference
Negetice Samples 8 ( [Knowtedge] [ncousster Response))
Feedforward Network
Perdforward Netwark
Multi-head Self-Attention
Multi Heati-head Self- Attention
High-Quality Human-Amootteed
Factual Consistency Datasets
[Knowligd]]] (K)
[Contacc]
Figure 3: Reward model training and workflow of RLFC training for KDS.
FFNs. Factual knowledge, or world knowledge, is generally represented among entities in languages (i.e., dates or locations) (Roberts et al., 2020). As illustrated in Figure 2(b), we propose K-DIAL. which directly extended an additional FFN module with d' key-value pairs and further concatenated with the original FFNs of PLMs, to maximize the activation of each entity token y3 over certain showledge-groundee dialogue input pattern of the sequence KK, X, cel. The loss function of K- DIAL framework is formulated as follows:
been improved while maintaining the general lan- guage ability.
2.4 RLFC: Reinforcement Learning for Factual Consistency
For KDS, we prefer knowledgeable responses that are faithful to the gold knowledge. However, since PLMs are trained on the massive unlabeled cor- pus, KDS models do not inherently prioritize fol- lowing the preferences constantly output factual knowledge and consistent responses. Aligning with the factual consistency preference can implicitly encourage FFNS of Transformers to convey fac- tual knowledge and ultimately reinforce the fac- tual consistency of responses. Inspired by the re- cent progress of reinforcement learning from hu- man feedback (RLHF) technique to align human preferences (Ouyang et al., 2022: Ziegler et al., 2019; Christiano al., 2017) like mitigating toxic- ity (Faal al., 2023), we regard factual consistency as one type of preferences and thus propose rein- forcement learning for factual consistency (RLFC) method in this work.
m LKCE
1
Σ n
in (v.) log log X,K)
(3)
where ní is the number of entities in Y and 1 i (y.)
denotes whether y; belongs to the entity set ÿk--
The training process of K- DIAL framework is specified in two steps. First, all the parameters of the original PLM are frozen, and the loss LKCE is only calculated over the parameters of the extended FFNs. Afterward, we further adapt the knowledge- enhanced model on the KDS datasets using super- vised fine-tuning of Equation (I) while keeping the parameters in extended FFNs fixed. The word embedding dimension and hidden size of extended FFN module are set equal to the corresponding Transformer FFN layers. Note that K-DIAL frame- work is only applied the top 3 layers of the model our experiments.
Specifically, we first design reward model us- ing factual consistency classifier. There are some recent publicly human-annotated benchmarks and datasets containing information on the preference for factual consistency (Santhanam et al., 2021; Gupta et al., 2022; Dziri et al., 2021), where the similar definitions to factual consistency as "at- tributed" and "supported" are used in KDS indi- cating whether the response utilizes and follows the provided knowledge. We thus take advantage of these well-aligned data to train binary NLI model, serving reward model to provide infor- mative reward signals for RL training. The reward model R(-) is optimized using the following binary cross-entropy loss function:
As illustrated in Figure 2, the extended FFNs are supposed to predict Argentina as the next to- ken given the specific knowledge and context. The K-DIAL framework takes advantage of FFNs' abil- ity learn the complex dependency between the knowledge snippet and dialogue via activating re- lated entity tokens. In this way, factual consistent entity words can be triggered in the response. The ability of PLMs to express factual knowledge has

---

responses for dialogue modeling. Note that the initial configuration of CMU DoG entails the pro- vision of a gold knowledge paragraph to the mod- els alongside the dialogue. In this work, we split these knowledge documents into sentence pieces and select the most relevant one as the grounding knowledge, preserving the average token number of knowledge snippets comparable to those on WoW.
BBC
g log R(K ((K()) )())
+ (1 / log (1 R(K )()
(4)
where the knowledge-response pair (K(1), (() is the input to the reward model and g( is the label.
As illustrated in Figure 3, the dialogue model be optimized for RLFC training is used as the policy model. The response Y generated by the policy model and gold knowledge snippet K are fed into the reward model obtain the consistency reward score rı as F1 = R K, Y) which is mainly used to align the preferences for FFNS' factual ex- pression. The reward model will return a higher score for the factually consistent pairs to facilitate the factual expressions of the policy model. Fur- thermore, reference model generating response Y' is also introduced, which usually the dialogue model before RLFC training. The KL divergence 12 = KL Y'] between the outputs of the ref- erence model and the policy model is used as an extra reward signal to make sure the generated re- sponses don't diverge too far from the originals. The optimization objective for RLFC training via the Proximal Policy Opti- mization (PPO) algorithm (Schulman al., 2017).
More data processing details can be referred in Appendix A.
3.2 Implementation Details
For the dialogue generation models, we leverage GPT2 series (GPT2-Mcdium(M)) GPT2-Large(L), GPT2-XL) models (Radford et al., 2019) imple- mented using HugginglFace library (Wolf et al.. 2020) based on PyTorch (Paszke al., 2017). All the PLMs are further fine-tuned on the above WoW and CMU CMU_DoG dialogue datasets by minimizing the cross-entropy loss Equation (1), ADAM parameter update is used in a mini-batch mode for all models. In the decoding stage, we use the beam search algorithm and set the number of beams 5. During K-DTAL training, all the knowl- edge entities in gold knowledge and responses recognized using spaCy$. The RLFC is imple- mented by tr1 in this work, and all the hyper- parameters related to PPO algorithm are default values by the tr1 PPOConfig recipe except the epoch, learning rate and batch size. The reward model is obtained by training a BERT-Large (De- vlin et al., 2019) based NLI model as a factual consistency classifier trained on three public, high- quality human-annotated benchmarks and datasets Santhanam et al. (2021)8, Dziri al. (2021) 9, Gupta al. (2022) 10.
rı r2 is utilized
3
Experimental Setup
3.1 Datasets
WoW The Wizard of Wikipedia (WoW) is large-scale multi-turn knowledge-grounded dia- logue dataset (Dinan et al., 2019) collected through human-human worker chats, where a "wiz- ard" as the <bot> can respond to an "apprentice" as a <user> in a Knowledgeable way given evidence from external Wikipedia documents. We only fo- cus modeling the responses by the "wizard" provided the selected gold-label evidence and the previous dialogue contexts.
Further model setting information can be found in Appendix B.
3.3 Metrics
We exhibit range of comprehensive metrics gauge the factuality and conversational ability of
DoG The CMU Document Grounded Conversations Dataset (CMU_DoG) 3 (Zhou et al., 2018) refers to a collection of conversations that encompass two users discussing various famous movies given related Wikipedia documents. Ut- terances by the user who can access the movie evidence in the documents
"nttss//tuggrinffaccoo//g
2 /https://spacy io/
"https://ithhbb https///gthhu..oc///eeeeeee com/Iweera/tll
treated <bot>
http:///huulbooooaaees///ccclsao of oof-dialogs
hhttp::/poullaipppeecsswwwww of wikipedia/ httts:///gthuhh.ooffettroorrddd
"https///ithhb..omm//ee
10 httpps///gitub..oo//aeeeeefe

---

KDS, entailing both a series of automated tech- niques as well as human evaluations.
edge that the response should be grounded to eval- uate the factual consistency of KDS.
Lexical and Semantic Metrics In this work, we adopted token-level Fl uni-gram overlap, BLEU, and ROUGE-L metrics to measure the lexical simi- larity for dialogue quality evaluation between gen- erated and ground-truth responses. Additionally, Knowledge F1 (KF1) (Shuster et al., 2021) and BERTScore (Zhang* et al., 2020) (BERT.) are used capture such lexical overlap and semantic simi- larity of response and grounding knowledge.
Human Evaluation We exhibit human evalua- tions as means of assessing performance across various dimensions of generation quality. Annota- tors were requested to answer: 1) whether the re- sponse is fluent and understandable (Flue.) and 2) whether the response is contexiually coherent given previous conversations (Cohe.) and 3) whether the response is factually correct (Fact.).
All the annotators were asked to rate each quality on a two-level Likert scale from (Not Flunet, Not Coherent, Inconsistent) to I (Flunet, Coherent, Consistent) to evaluate the fluency, coherence, and factual consistency of generated responses. The averaged results by the human evaluation scores reported.
Fine-Grained NLI-based Metrics NLI-based metries are more robust and widely used to de- tect factual inconsistency or hallucinations in knowledge-inlensive NLP tasks (Dušck and Kas- ner, 2020; Mishra et al., 2021; Falke et al., 2019; Welleck et al., 2019; Chen et al., 2023). Therefore, we developed synthetic dataset for BERT-based (Devlin et al., 2019) NLI model pre-training. The synthetic dataset adopts factual consistent samples that are derived from the ground-truth response and gold knowledge pairs in WoW. Inconsistent responses are generated by random pairing, nega- tion, and entity swapping (Kryscinski et al., 2020a; Gupta et al., 2022).
4 Results and Analysis
4.1 Main Experiments of K-DIAL and RLFC
Results of Automatic Evaluations In Table 1, we present the experimental results of various GPT2-based dialogue PLMs using K-DIAL and RLFC on WoW and CMU_DoC test sets. Several trends can be found below:
Following (Santhanam al., 2021), we develope three metries to finely-grained evaluate factual con- sistency and fine-tune the pre-trained NLI model on the datasets released by (Santhanam et al., 2021; Gupta et al., 2022; Dziri et al., 2021), which are also used for reward model training of RLFC. The three fine-grained NLI metrics are designed to in- spect 1) Verification (Verif.): whether a response contains verifiable information: 2) Hallucination (Hallu.)): whether a response does NOT comprises hallucinated content; and 3) Factual Correctness (Fact.): whether a response is factually consistent with grounding knowledge.
I) The effects of K-DIAL: GPT2 series models using K-DIAL outperform all standard dialogue models in both factual consistency and dialogue quality for KDS on both WoW and CMU IMMU_DoG test sets. Significant factual consistency improvements of GPT2-L+K-DAA DIAL in Fact. and Q² in terms of 5.36% and 7.63% absolutely over GPT2-L on WoW indicate that K-DIAL effectively enhances factual expressions. On CMU LAUU, supreme fac- tual consistency improvements 3.84% and 4.11% absolutely on Fact. and Q² are obtained on GPT2- M after using K-DIAL.
Improvements in the KFI measure suggest that the responses equipped by K-DIAL are more knowledgeable and faithful to the supported knowl- edge. The performance improvements obtained on the conversationality metrics of BLEU, F1, and ROUGE-L present that through enhancing factual expressions for responses, the dialogue quality can be also marginally improved.
Although there may be slight variations across the definitions in the aforementioned benchmarks, their shared objective is to enhance the faithfulness and reliability of responses to the provided gold knowledge. The data processing and alignment details are presented in Appendix C.
Q² Metric Honovich et al. (2021) proposed Q2 metrie 11 employed a question generation system, question answering system, and an NLI model to find the corresponding answer span in the knowl-
2) The effects of RLFC: Comparable per- formance improvements are also acquired using RLFC on GPT2 dialogue models, demonstrating that RLFC can proficiently improve the factual con- sistency of KDS on both WoW and CMU_DoG;

---

Model
Dataset
KF1
BERT.
Verif.
Hallu.
Fact.
Q
BLEU
ROUGE
GPT2-M
46.51
38.25
13.67
10.94
5.74
55.35
27.09
60.83
7.83
K-DIAL
48.36
41.97
16.67
11.54
9.36
62.47
28.33
61.48
8.69
RLFC
47.23
40.34
18.84
11.15
8.44
59.14
25.43
59.26
6.24
RLFC K-DIAL
50.27
41.65
18.74
12.37
11.56
63.26
27.45
61.34
7.98
GPT2-L
68.46
47.44
43,96
32.35
40.19
76.08
53.32
76.18
30.82
K-DIAL
70.59
50.73
45,38
34.12
45,55
83.71
55.78
77.64
32.45
WoW
RLFC
70.44
52.27
47.73
35.16
44.41
80.25
55.42
75.25
31.24
RLFC+ K-DIAL
72.38
53.25
48.81
37.98
46.37
82.72
56.78
77-57
33.34
GPT2-XL
73.67
51.40
50.15
34.44
48.40
79.38
54.45
79.72
36.90
+ K-DIAL
75.48
53.38
50.07
36.45
49.12
83.38
53.96
80.23
36.84
RLFC
75.21
52.26
51.33
36.17
49.31
82.01
54.60
80.11
37.02
RRFC K-DIAL
76.35
53.60
50.94
37.08
50.13
83.87
54.77
79.99
37.19
GPT2-M
16.
32.35
10.27
7.54
4.26
37.51
36.68
61.33
19.12
K-DIAL
29.21
37.44
12.08
41.62
37.64
62.19
19.39
RLFC
30.54
35.20
8.21
40.43
38.24
62.13
20.13
RL.C+ +K-DIAL
31.66
38.12
14.09
42.36
38.56
62.49
21.07
GPT2-L
51.35
39.42
31.54
27.65
23.38
57.82
45.17
68.55
31.27
K-DIAL
53.16
42.23
33.64
30.06
25.45
59,40
45.74
69.16
32.57
CMU
RLFC
52.87
42.36
33.95
28.71
25.31
59.12
46.72
69.88
33.37
RLFC K-DIAL
54.02
44.89
34.70
31.43
28.67
61.34
46.65
70.18
34.09
GPT2-XL
65.14
45.09
42.88
35.17
33,97
66.13
48.25
75.39
34.35
K-DIAL
66.48
45.26
44.08
37.24
34.53
68.50
48.79
75.22
35.52
RLFC
66.
51.33
44.06
37.16
34.83
69.34
50.02
75.64
35.86
RLFC+ K-DIAL
67.41
46.21
44.97
38.03
35.05
70.14
51.14
75.49
36.10
Table I: Experiments of GPT2 series models fine-tuned on KDS datasets employed with K-DIAL and RLFC methods on WoW and CMU_DoG test set.
where performance improvements of 3.77% in Verif. on WoW and 2.41% % CMU _Do are ob- tained by GPT2-L+RLFC over GPT2-L models.
GPT2-M and GPT2-L models using either K-DIAL RLFC than on larger-scale GPT2-XL models both WoW and CMU_DDG test sets, the GPT2- XL models finetuned on KDS datasets are more robust to generate factual consistent contents.
RLFC performs better on Verif. measure over standard baseline models than K K-DIAL, suggest- ing that RLFC is better promoting the model's ability generate verifiable responses by aligning with factual knowledge. The side-effect of degra- dation on dialogue quality metrics implies that ap- plying RLFC uniquely cannot effectively maintain the original conversationality of the standard GPT2 dialogue model.
Model
Dataset
Flue.
Cohe.
Fact.
GPT2-L
1.00
0.88
0.65
K-DIAL
WoW
RLFC
1.00
0.91
0.69
RIFC K-DIAL
1.00
0.91
0.74
GPT2-L
1.00
0.86
0.69
+ K-DIAL
1.00
0.86
0.73
CMU_DoC
RLFC
1.00
0.87
0.74
RLFC + K-DIAL
1.00
0.76
3) The effects of RLFC+K-DIAL: The opti- mal training strategy to obtain the final models is specialized in two stages. We first train the GPT2 models using RLFC and then apply K-DIAL on the obtained model. The supreme performance im- provements are obtained on the setting of GPT2-L using the combination of RLFC and K-DIAL meth- ods in Q² and Fact. in respective of 6.18% and 8.64% absolutely over standard GPT2-L dialogue model on WoW. The combination of RLFC and K-DIAL can implicitly and explicitly improve the models' ability to express factual knowledge as a complementary, demonstrating the best perfor- mance in respect of factual consistency and dia- logue quality.
Table 2: Human evaluation results of GPT2-L model
using respective and combination of proposed K- K-DIAL and RLFC on 100 samples selected from WoW and CMU_DoC test sets respectively.
Results of Human Evaluations To accurately gauge the performance of the proposed methods, we perform human evaluations in Table 2. We select a subset of examples from the WoW and test sets, using 100 examples from each dataset per model variant with 3 human raters. Results indicate that responses generated by all the models are fluent and coherent appropriately make sense in engaging the context of the conver- sations. Furthermore, the assessment of factual consistency by human evaluators demonstrates a
4) The effects of model size: We observe that better performance improvements are attained on

---

Model
Dataset
Para
KF1
BERT.
Verif.
Hallu.
Fact.
Q
BLEU
F1
R.L.
GPT2-M
35SM
46.51
38.25
13.67
10.54
5.74
55.55
27.09
60.83
7.83
K-ADAPTER
377M
46.96
39.32
14,66
10.35
7.64
60.36
26.03
59.17
7.06
K-Former
361M
47.43
39.91
15.51
11,50
8.37
57.87
26.28
59.08
8.14
WoW
+ NKB
361M
46.60
38.74
13.49
9.51
6.35
58.
25.30
60.55
6.84
+ K-DIAL (ours)
361M
48.36
41.97
16.67
11.54
9.36
64.47
28.33
61,48
8.69
K-DIAL-O (ours)
361M
49.03
41.66
16.50
11.23
62.34
29.36
8.40
GPT2-M
355M
26.13
32.35
10.27
7.54
4.26
37.51
36.68
61.33
19.12
K-ADAPTER
377M
27.69
34.44
11.32
8.63
6.64
39.26
37.26
61.19
18.84
K-Former
361M
28.90
35.89
12.61
9.06
7.37
40.35
37.18
61.48
19.24
DMU
+ NKB
361M
27.63
32.97
10.59
9.51
5.23
36.86
36.45
61,30
18.55
+ K-DIAL (ours)
361M
29.21
37.44
12.08
8.94
8.10
41.62
37.64
62,19
19.39
+ K-DIAL-O (ours)
361M
28.83
37.25
12.20
9.23
7.68
40.28
37.77
62.10
20.84
Table 3: Experiments and parameter use of several knowledge enhancement baseline and K-DIAL variants compar-
isons on GPT2-M on WoW and CMU DoG test set.
Grounding Documents
Dialogne History
Generated Responses
1980s in Music
Music in 1980s was so much what have heard?
Baseline: popular type of music coming in the 1980s was disco.
As disco fell of fashion in the decade's 's early years, genres such post-disco, Italo disco, Buro disco, and dance pop became more popular the 1980s.
1980's was emergence of dance music and wave music.
K-DIAL: Dance- pop, post
<bot>
Euro disco became more popular in the 1980s.
What exactly was the typo of music coming the 1980x? Pop?
RLECC As know price to the 1980s, disco was popular fashion of music.
<usero
<boD
Figure 4: An example of conversation on WoW valid set before and after using K-DIAI. and RLFC the dialogue model (denoted as <bot>). Incongruous content is highlighted in Red, while the counterpart gold knowledge in the Wikipedia document is in Blue.
strong correlation with the Fact. and Q² metrics in Table 1, which confirms that our proposed methods exactly improve factual consistency for KDS. The raters' agreements for each quality are measured separately using Fleiss' Kappa of statsmodels 12, All the results (Flue.:0.99, Cohe.:0.95, Fact.:0.77 respectively) demonstrate substantial and almost perfect agreement levels.
eural Knowledge Bank (NKB) (Dai et al., 2022b). which first integrate substantial exogenous knowl- edge and are further adapted on KDS tasks base- lines presented in Table Experimental results on GPT2-M dialogue models generally show that all the knowledge injection methods can marginally improve the factual consistency but slightly de- grade the dialogue quality on BLEU and ROUGE- L. K-DIAL outperforms the three baseline knowl- edge injection methods in both factuality and con- versationality, demonstrating superior performance to improve factual consistency without sacrificing the dialogue qualities for KDS tasks.
4.2 Baseline Methods and K-DIAL Variants Comparisons
Experiments of Baseline Comparisons To the best of our knowledge, this work is the first to propose to improve factual consistency for KDS. Previous related works that investigate the factual consistency of KDS only focus on evaluation meth- ods (Honovich al., 2021) or datasets (Labadie et al., 2021; Dziri al., 2022; Gupta al., 2022). Therefore, there is no direct improvement method to be compared.
More details regarding the baseline configura- tions and implementations are available in Ap- pendix D.
Experiments of K-DIAL Variants We also con- duct variant comparison experiments of a K-DIAL- which updates the extended FFN modules using LCB as Equation (1) and calculate the loss on all tokens rather than LKCE of Equation (3) for just knowledge entities. Experimental results suggest that optimizing K-DIAL by either Lee or LKCE has comparable performance, as the knowledge formation mainly represented by the knowledge entities and learned by the FFN modules. For ef-
Nevertheless, for the method K-DIAL,, we still carry experiments on several knowledge injection and enhancement methods for NLG tasks, including K-Adapter (Wang et al., 2021), Kformer (Yao et al., 2022), and
#https//gttuu..co/sttassmm

---

ficiency, we adopt only updating extended FFN parameters on knowledge entities for the K-DIAL method.
Enhancing Knowledge in PLMs Previous works have explored ways to incorporate exter- nal knowledge into pre-trained language models (PLMs). ERNIE (Zhang et al., 2019) and Know- BERT (Zhang et al., 2019; Peters et al., 2019) en- hance the word representations by incorporating external knowledge graphs. K-ADAPTER intro- duces two adapters to inject factual and linguis- tic knowledge into PL.M respectively (Wang al., 2021). Inspired by Dai al. (2022a), recent works focused to add extended FFNs Transformer-like K-former (Yao et al., 2022) or Neural Knowledge Bank (NKB) (Dai et al., 2022b) to inject and update extra knowledge while keeping the original model parameters frozen. Dong et al. (2022) investigated to detect the incorrect knowledge stored in PLMs and proposed CALINET for knowledge calibration.
4.3 Case Analysis of K-DIAL and RLFC
We further present representative case in Figure 4 to analyze the practical performance of proposed K-DIAL and RLFC methods in comparison with the standard GPT2-L model respectively. The fol- lowing trends are found:
1) Both K-DIAL and RLFC can effectively cor- rect the inconsistent response generated by the standard GPT2-L dialogue model that contradicts the factual knowledge in the Wikipedia document, demonstrating their efficacy in improving factual consistency for KDS.
2) The K-DIAL method is more likely learn the important knowledge snippet and directly ex- press in responses, which is achieved by extended FFNs' explicit ability to enhance factual knowledge expressions.
Reinforecment Learning in NLG With growing interest in RL technique, learning enhanced LMs from human feedback has been explored in Ouyang et al. (2022); Bahdanau al. (2019); Ramamurthy et al. (2022). Language models are optimized to align with human preferences such as harmless and non-toxic outputs (Faal et al., 2022; Bai et al., 2022), which means following human instructions better. RLHF technique was also widely applied in downstream tasks such as dialogue system (Jaques et al., 2020; Lu al., 2022: Kwan et al., 2023; Wang et al., 2022) and abstractive summarization (Böhm et al., 2019; Wu al., 2021).
3) RLFC implicitly aligns the FFNS' expressions with external gold knowledge for the factual con- sistency preference, which is more semantically natural and human-like than K-DIAL.
5 Related Works
Factual Consistency in NLG The issue of fac- tual inconsistency in NLG tasks has attracted in- creasing attention in many fields such as abstrac- tive summarization, with studies focusing on both improving and evaluating the factual consistency (Kryscinski et al., 2020b; Maynez et al., 2020; Xie et al., 2021; Zhu et al., 2021; Nan et al., 2021). Related works were also conducted on data-to-text generation (Dušek and Kasner, 2020; Thomson and Reiter, 2020). In the context of dialogue sys- tems, Dziri et al. (2021); Gupta al. (2022) in- troduced the benchmarks for measuring the attri- bution and fact-checking of dialogue generations with grounding knowledge. In the context of di- alogue systems, Rashkin et al. (2021) added con- trollable tokens on the input of the dialogue model to generate responses that are more faithful to the source knowledge. Shuster et al. (2021) investi- gated the Aetrivval-Augeente approach to reduce knowledge hallucination in con- versational agents. Peng al. (2023) introduced LLM AUGMENT, a framework for augmenting black-bul box LLMs with external knowledge and au- tomated feedback to reduce hallucinations.
6 Conclusion
In this work, we investigate the inadequacy of KDS that often produces factually inconsistent responses unsupported by grounding knowledge. We pro- pose two strategies to tackle this issue. K-DIAL introduces extended FFN modules to explicitly hance factual knowledge expressions given specific input patterns of the knowledge snippets and di- alogue contexts. RLFC technique used to im- plicitly adjust FFNs to augment factual expressions in responses by aligning with the gold knowledge for factual consistency preferences. Experiments present that both K-DIAL. and RLFC can promote the knowledgcabilitty factual consistency and con- versational ability of responses, demonstrating the efficacy of our proposed methods improve the ability of FFNs to express factual knowledge
Generation (RAG)
generate more informative and reliable responses in dialogue applications.

---

Limitations
(EMNLP-IJCNLP), pages 3110-3120, Hong Kong. China. Association for Computational Linguistics.
The limitations of this work are summarized below:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
1) As shown in Figure 2 (a) and described be- fore, this paper assumes that factual inconsistency comes along with dialogue model M and proce- dure 4 in Figure 2 (a), which deviated from the reality that the knowledge retrieval process ① and hallucinations in knowledge K. and contexts A' are not always correct. A more challenging problem lies in locating the inconsistency cause of genera- tion processes. In future work, we will make a sys- tematic investigation of the factual inconsistency and hallucination problems in KDS.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun. Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting lan- guage models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1860- 1874, Online. Association for Computational Linguisties.
2) Recently, large-scale language models (LLMs) such as GPT3 and ChatGPT have demon- strated state- state-of-the-art the-art performance across range of NLP tasks. This work was only conducted on the GPT2 series PLMs with a maximum of 1.26B pa- rameters, which is extremely small in comparison with such LLMs containing hundreds of billions of parameters. However, since the proposed methods involve plenty of model parameter updating, it is difficult to employ on LLMs due to the limitations of GPU resources in the initial work. Next, we will continue to explore the transferabilitty of the framework using the parameter-efficient method to the employment of open-source LLMs.
Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, and Kam-Fai Wong. 2023. Be- yond factuality; comprehensive evaluation of large language models as knowledge generators.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022a. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 8493- 8502, Dublin, Ireland. Association for Computational Linguistics.
A Acknowledgements
This research work is partially supported by CUHK direct grant no. 4055209.
Damai Dai, Wenbin Jiang, Qingxiu Dong. Yajuan Lyu. Qiaoqiao She, and Zhifang Sui. 2022b. Neural knowledge bank for pretrained transformers.
References
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 417 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019, Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08075.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge- poweeed conversational agents. In International Conference on Learning Representations.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira. Ido Dagan, and Iryna Gurevych. 2019. Bet- ter rewards yield better summaries: Learning to sum- marise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing
Qingxiu Dong, Damai Dai, Yifan Song, Jingiing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing (EMNLP), On- line, Association for Computational Linguistics.

---

Ondiej Dušek and Zdeněk Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with nat- ural language inference. In Proceedings of the 13th International Conference on Natural Language Gen- eration, pages 131-137, Dublin, Ireland. Association for Computational Linguistics.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tüür 2019. Topical-Chat: Towards snowledge-Groundee Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021. Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark. arXiv e-prints. page urXiv:2105.000 000000.........
Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A benchmark for fact-checking in dialogue. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3785-3801, Dublin, Ireland. Association for Compu- tational Linguistics.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue sys- tems: The BEGIN benchmark. Transactions of the Association for Computational Linguistics, 10: 10:1066- 1083.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander. Eduard Hovy, Hinrich Schütze. and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- tions of the Association for Computational Linguis- tics, 1:1022-1031.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q": Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. 2022. Reward modeling for mitigating toxicity in transformer- based language models. arXiv preprint arXiv:2202.09662.
Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. 2023. Re- ward modeling for mitigating toxicity in transformer- based language models. Applied Intelligence, 53(7):8421-8335
Natasha Jaques, Asma Ghandebarioun, Judy Hanwen Shen. Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020. Way off- policy batch deep reinforcement learning of human preferences in dialog.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy, Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 20 Transformer feed-forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wojeiech Kryscinski. Bryan McCann, Caiming Xiong, and Richard Socher. 2020a. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9332-9346, Online. Association for Computa- tional Linguistics.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 202 Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 3484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020b. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Online. Association for Computa- tional Linguistics.
Marjan Ghazvininejad, Chris Brockett, Ming- ding-Weii Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounde neu- ral conversation model. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Wai- Chung Kwan, Hong-Ru Wang, Hui-Min Wang, and Kam-Fai Wong. 2023. A survey on recent advances and challenges in reinforcement learning methods for trie-trd dialogue policy learning. Machine Intelligence Research, 20(3):318-33)

---

Roberto Labadie. Mariano Jason Rodriguez, Reynier Or- tega, and Paolo Rosso. 2021. RoMa at SemEval--221 task 7: A transformer-basee approach for detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-202) ). pages 297-305, Online. Association for Computational Linguistics.
human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-277 27744. Curran Associates, Inc.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Katherine Lee. Orhan Firat, Ashish Agarwal, Clara Fan- njjang, and David Sussillo. 2019. Hallucinations in neural machine translation.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv: 12811333...
Hongyuan Lu, Wai Lam, Hong Cheng, and Helen Meng. 2022. Partner personas generation for dialogue re- sponse generation. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics; Human Language Technologies, pages 5200-5212, 5212, Seattle, United States. Association for Computational Lin- guistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019, Knowledge enhanced contextual word representations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 1906-1919, On- line. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayaku- mar, Xiang Lorraine Li, Pavan Kapanipathi, and Kar- tik Talamadupula. 2021. Looking beyond sentence- level natural language inference for question answer- ing and text summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1322-1336, On- line. Association for Computational Linguisties.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting back- ground knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages Brussels. Belgium. Association for Com- putational Linguistics.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu. Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforeement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. arXiv preprint arx:v:22 10.00124.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng. Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang. Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6881-6894, Online. Association for Computational Linguistics.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grouneee grounded dialogue with controllable features. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 704-718, Online. Association for Computa- tional Linguistics.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1416-6426, Online. Association for Computational Linguistics.

---

Sashank Santhanam, Behnam Hedayatmia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study on factual correctness in knowledge-gnnn -ordddddggrddddddeoooo response generation. arXiv preprint
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguisties.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109 10862.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 100-1 10, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Shuaijie She, Xiang Geng, Shujian Huang, and Jiajun Chen. 2023. Cop: Factual inconsistency detection by controlling the preference. Proceedings of the AAAI Conference on Artificial Intelligence.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Yunzhi Yao, Shaohan Huang, Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang, 2022. Kformer: Knowledge injection in transformer feed-forwar lay- ers. In Natural Language Processing and Chinese Computing, pages 13 131443,, Cham. Springer Interna- tional Publishing.
Craig Thomson and Ehud Reiter. 2020. gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, pages 158-168, Dublin, Ireland. Association for Computa- tional Linguisties.
Tianyi Zhang*. : Varsha Kishore* Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Reprexentations.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proceedings of the 57th Annual Meeting of the A Association for Computational Linguistics, pages 1441-1451 1451, Florence, Italy. Association for Compu- tational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in neural information pro- cessing systems, volume 30.
Hongru Wang, Huimin Wang, Zezhong Wang, and Kam- Fai Wong. 2022. Integrating pretrained language model for dialogue policy evaluation. In ICASSP 2022-2022 IEEE International Conference Acous- tics, Speech and Signal Processing (ICASSP), pages 6692-6696. IEEE.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018, dataset for document grounded con- versations. In Proceedings of the 2018 Conference Empirical Methods Natural Language Process- ing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-adapter: Infusing knowledge into pre-trained models with adapters. In ACIJIJCNIP (Findings), pages 1405-1418.
Chenguang Zhu, William Hinthorn, Ruochen Xu. Qingkai Zeng. Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies, pages 718-733, Online. Association for Computational Linguisties.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Daniel M Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint 1rx.v:190..08593.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing.

---

A Dataset Details
with respect to Verification, Factual Consistency, and Hallucination exemplified in Figure 6. Sim- ilar taxonomy also adopted in Gupta al. (2022) (Generic/Atriiutaaleeoo Attributable) and Dziri et al. (2021) VeriffattoonSupporeddReejjlllll Enough Information (NEI)).
WoW As listed below in Table 4, the Wow dataset contains 18,430, 1,948, and 1,933 conver- sations in Train/Valid/Test sets respectively. Both "seen" and "unseen" topic portions of test sets have been merged. Each conversation data spans 4-5 turns of utterances between Wizard and Appren- tice. In each turn, the response by the Wizard is grounded on gold knowledge of one or two checked sentences in the Wikipedia documents as eviden- tiary support.
D Baseline Method Details
K-Adapter Wang al. (2021) proposed K- ADAPTER, a neural adapter architecture specifying one kind of knowledge (c.g. Factual Knowledge or Liguistic Knowledge), as plug-in in connections into different Transformer layers of PLMs. Following Wang et al. (2021), we set each adapter model con- sisting of three adapter layers plug-in among the highest, middle, and lowest Transformer layers of PLMs (e.g. for the 36-layer GPT2-Large, adapter layers plug-in is configured as {1,18,36)) and pa- rameters are not shared across different adapter lay- ers. Each adapter layer comprises two Transformer layers and two projection layers illustrated in Fig- ure 5. The Transformer block of the adapter layer has been established to be of equal size to that of the PLMs. Additionally, the hidden dimensions of the down-projection and up-projection layers have been correspond to the word embedding and hidden dimension of the PLMs in different scales respectively.
The CMU_Dod (Zhou et al., 2018) dataset contains 3,373, 229, 619 conversations with an average of 21.43 turns per conversation. In CMU_DoG, the grounding knowledge at the doc- ument level, which is a more difficult (hut realistic) setting rather than WoW.
EMU_DoC
The number of processed training samples are presented in 4 below.
WoW
CMU_DoG
Conv.
# Samp.
# Conv.
# Samp.
Train Valid
18,430
73,571
3,373
78,406
1,948
7,903
229
5,474
Test
1,933
7,844
619
14,700
Table 4: Data statistics of two datasets.
B Model Details
Kformer Kformer (Yao et al., 2022) is knowl- edge fusion model that converts the knowledge into dense embedding vectors and then injects them into the parameters of expanded FFNS of the Trans- former layers followed by Dai et al. (2022a). In accordance with (Yao al., 2022), we encode the external knowledge via embedding layer which is initialized the same word embedding matrix of GPTs. Then we map the obtained knowledge representations into the corresponding vector space of FFN weights. Only the top 3 layers of all PLMs were adopted for knowledge enhancement.
The hyper-parameters of GPT2 series models on both WoW and CMU tasks are listed in Ta- ble 5 including training epochs, batch size, learn- ing rate, warm-up steps, and maximum sequence length.
Metries Details
NLIIbased Metrics For our annotation of fac- tual consistency, we categorize responses into three types as shown in Figure 6: Non-verifiable Re- sponse does not include any information that needs to be verified and cannot be evaluated as consis- tent or not consistent. A factually consistent re- sponse is informative and highly relevant to the provided knowledge. Hallucinated responses may not be always consistent with the knowledge but could still be correct. Despite the exorbitant ex- penses associated with human annotations, there are still publicly precious gold-label datasets. Fol- lowing Santhanam et al. (2021), we defined finc- grained metrics on factual consistency evaluation
Neural Knowledge Bank Neural Knowledge Bank (Dai et al., 2022b) have put forth the Neural Knowledge Bank (NKB) which is an extended FFN module as the memory slots for knowledge infu- sion using Salient Span Masking (SSM) (Guu al., 2020) strategy to preserve the general language modeling competency. The NKB architectures are expanded onto the top three FFN layers of GPTs. The quantity of supplementary memory slots is es- tablished to match the dimension of intermediate

---

Model
Dataset
Epoch
Batch
L.R.
Warm.
Seq.
Len.
GPT2-M
6e-5
GPT2-M+K- -IAL (FFNs)
256
GPT2-M-K-DIAL (GPT2 Model)
3
6c-5
GPT2-M+RIFC
3
16
256
GPT2-I.
4
60-5
GPT2-L.*K- ----IILL (FFNs)
2
60-5
256
WoW
GPT2-L+K ---ALLL (GPT2 Model)
3
6e-5
4k
256
GPT2-L+RLFC
3
le-5
256
GPT2-XL
4
6e-3
8k
256
GPT2-XL+K3 --K--LAALLLLLLL (FFNs)
2
16
6e-5
8k
256
(012-XL+K-DILLE (GPT2 Model)
3
4
6e-5
8k
256
GPT2-XLARLFC
3
4
le-5
256
GPT2-M
4
16
le-5
4k
256
GPT2-M+K DIAL (FFNs)
2
32
le-5
4k
256
1172-M+K-DIAA (GPT2 Model)
3
16
le-5
4k
256
GPT2-M+RLFC
3
16
le-5
256
GPT2-1
4
le-5
8k
256
GPT2-L+K- K-DIAL (FFNs)
2
24
le-5
8k
256
CMU_Dot DoG
GPT2-1+K-DIM (GPT2 Model)
3
8
le-5
8k
256
GPT2-L+RL
3
le-5
256
GPT2-XI
le-5
8k
256
GPT2-XLL T K-DIAL (FFNs)
2
16
le-5
8k
256
W-72-XL+K-DILE (GPT2 Model)
3
le-5
8k
256
GPT2-XL+RLFC
3
le-5
256
Table 5: Training hyper-parameters of baseline GPT2 models as well K-DIAL and RLFC based models different sizes WoW and CMU_Dot MMG datasets respectively. Epoch, Batch, L.R., Warm., Seq. Len. denote training epochs, batch size, learning rate, warm-up steps, and maximum sequence length respectively.
K-Foreer Layer
Neural Knowledge Bank
Up Proesion Layer
Trandformer
Transfemer
Transtomorr
Linear Ex-Linear
Embodding
Tranaformer
Linear Layer
Layer
Layer
Key
Projeten Lapar
Ksovledgs
Transforeer
Transforer
Multhead Attention
Transtomer
Mutli haad Adantion
[Rooweddes]
Ierowel eilan] (5SM)
(n)
(b)
(c)
Figure Illustrations of (a) K-ADAPTER; (b) Kformer; (c) Neural Knowledge Bank (NKB) architectures.
Grounding Documents
Dialogue History
Generated Responses
1980s in Music
Music was so much what have heard?
whether pop the papalar type of music in the 1980s.
Non-Perjfiable:
As disco fell of fashion in the docade's early years, geares such as post-disco, Italo disso, Faro disco, and dance-pop became more popular 1980s.
1980's emergence dance music and now wave music.
Hollscinotion: I think 1980s music influcuced lot by the movies and TV shows from the
What exactly was the type of music coming 1980s? Pop?
Factual Incorret: The popalar type of music coming the disco.
<bocs
Figure
hidden states within the FFN module.
by only updating the parameters in the knowledge module (e.g. K-Adapter, Kformer, and NKB). Then, we fine-tune the knowledge-enhanced mod- els for the two knowledge- emowledge-groundedd dialogue tasks respectively.
The training procedure for all the knowledge- enhanced models is divided into stages. First, When injecting knowledge, we froze all the param- eters of GPT-2 models and injected factual knowl- edge (passages of WoW or CMU DoG datasets)

---

Model
Subset
KF1
BERT.
Verif.
Hallu.
Fact.
BL.UU
F1
RI..
GPT2-M
Seen
47.25
38.44
13.70
11.01
5.81
28.11
60.99
8.16
+ K-DIAL
Seen
50.07
42.23
16.77
11.62
9.39
28.26
61.63
8.97
RLFC
Scen
48.02
40.56
18.91
11.25
8.57
25.82
59.49
6.37
GPT2-M
Unseen
45.78
38.01
13.64
10.87
5.67
26.07
60.67
7.50
+ K-DIAL
Unseen
48.66
41.71
16.57
11.47
9.33
28.20
61.33
8,41
+ RLFC
Unseen
46.45
40.12
18.77
11.06
8.31
25.04
59.03
6.11
Table 6: Experiments of standard GPT2-M models before and after using K-DIAL and RLFC methods on respective seen and unseen WoW test subsets.
E Experiments on WoW Seen and Unseen Sets
this work, whether a topic in the test set has been seen or unseen during training is irrelevant to the effect of knowledge enhancement and alignment. There are no distinct performance improvement dif- ferences between seen and unseen test sets of WoW using K-DIAL and RLFC in early verification experiments as Table 6, which doesn't 't affect the conclusions. Therefore, we merge the subsets for all models.


Reviews:
[{"review_id": "mk9C3V6JYz", "reviewer": "Reviewer_waXu", "summary": "", "questions": "* Could you provide the number of parameters used for each approach in Table 4? \n* Is the result for CMU_DoG still consistent, similar to that of Table 1 (RLFC+K-DIAL)? Could you also report that result?\n* Additional clarification is needed regarding the model's training process (Lines 241-244). The current information lacks specific training details, such as the number of epochs/steps dedicated to training FFNs and the full model. Moreover, it is unclear whether lambda is utilized for KL-penalty. Furthermore, during the training of RLFC, could you specify whether the training only focuses on the FFNs or LM?\n* Was there a performance difference between the seen and unseen of WoW?\n* What specific scale was employed for the Human evaluation? (For instance, a 3-point Likert scale)\n* How are the final models for each row determined in Table 1? Did you use early stopping or fixed training epoch?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable questions.\nWe apologize for the ambiguities in the submitted manuscript.\nThe questions are addressed as follows.\n\n**Questions For The Authors:**\n\n1. Could you provide the number of parameters used for each approach in Table 4?\n\nWe apologize for the missing additional parameter information in the submitted paper.\nThe updated Table 4 with the number of parameters is as follows\n\n| Model                       | # Param | KF1 | BERTScore | Verif. | Hallu. | Fact. | ChatGPT | $Q^2$ | BLEU | F1 | RL. |\n|--------------------------------------------------|----------------|-----------|-----------------|--------------|--------------|-------------|---------------|--------------------|------------|----------|-----------|\n| GPT2-M                                           | 355M           | 46.51     | 38.25           | 13.67        | 10.94        | 5.74        | 26.25         | 55.55              | 27.09      | 60.83    | 7.83      |\n| + K-Adapter                            | 377M           | 46.96     | 39.32           | 14.66        | 10.35        | 7.64        | 28.75         | 60.36              | 26.03      | 59.17    | 7.06      |\n| + K-Former                                     | 361M           | 47.43     | 39.91           | 15.51        | 11.50        | 8.37        | 26.25         | 57.87              | 26.28      | 59.08    | 8.14      |\n| + NKB                                          | 361M           | 46.60     | 38.74           | 13.49        | 9.51         | 6.35        | 25.00         | 58.88              | 25.30      | 60.55    | 6.84      |\n| + K-Dial                              | 361M           | 48.36     | 41.97           | 16.67        | 11.54        | 9.36        | 32.50         | 64.47              | 28.33      | 61.48    | 8.69      |\n| + K-Dial ($\\mathcal L_{\\mathrm{CE}}$) | 361M           | 49.03     | 41.66           | 16.50        | 11.23        | 9.48        | 32.50         | 62.34              | 29.36      | 61.06    | 8.40      |\n\nwhere the **# Param** denotes the number of total model parameters.\nNote that the details of model designs of the knowledge-enhanced methods are presented in Appendix D in the paper.\n\n\n2. Is the result for CMU\\_DoG still consistent, similar to that of Table 1 (RLFC+ K-Dial)? Could you also report that result?\n\nFor RLFC-related experiments, the reward model is trained on NLI datasets, but we lack high-quality human-annotated factual consistency NLI data on CMU\\_DoG.\nThe high-quality fine-grained NLI datasets to evaluate factual consistency including Verification, Hallucination, and Factual Correctness, are only collected and annotated on WoW dataset [[1]](https://arxiv.org/abs/2110.05456), [[2]](https://aclanthology.org/2022.tacl-1.62/), [[3]](https://aclanthology.org/2022.tacl-1.62/).\n\nDirectly transferring the NLI model on WoW to CMU\\_DoG for RLFC training cannot obtain comparable performance improvements.\nPresumably these fine-grained factual consistency metrics to evaluate KDS are highly dependent on specific datasets.\nWe are trying to explore a more general NLI model for factual consistency evaluation that can be adapted to different datasets in future work.\n\n3. Additional clarification is needed regarding the model's training process (Lines 241-244). The current information lacks specific training details, such as the number of epochs/steps dedicated to training FFNs and the full model. Moreover, it is unclear whether lambda is utilized for KL-penalty. Furthermore, during the training of RLFC, could you specify whether the training only focuses on the FFNs or LM?\n\n- For additional information on the training process of K-Dial (Lines 241-244), the related hyper-parameters of two stages to train extended FFNs and the full parameters of GPT2 models on both WoW and CMU\\_DoG datasets are listed below.\n    All the models are trained to converge in the fixed training epochs.\n\n| Model                           | Dataset  | Epoch | Batch Size | Learning Rate | Warm-up Steps | Sequence Length |\n|-------------------------------------|--------------|-----------|----------------|-------------------|-------------------|---------------------|\n| GPT2-M                              | WoW      | 4         | 16             | 6e-5              | 2k                | 256                 |\n| GPT2-M+ K-Dial (FFNs)        | WoW      | 2         | 32             | 6e-5              | 2k                | 256                 |\n| GPT2-M+ K-Dial (Full GPT2)  | WoW      | 3         | 16             | 6e-5              | 2k                | 256                 |\n| GPT2-M+RLFC                         | WoW      | 3         | 16             | 1e-5              | -                 | 256                 |\n| GPT2-L                              | WoW      | 4         | 8              | 6e-5              | 4k                | 256                 |\n| GPT2-L+K-Dial (FFNs)       | WoW      | 2         | 24             | 6e-5              | 4k                | 256                 |\n| GPT2-L+K-Dial (Full GPT2)  | WoW      | 3         | 8              | 6e-5              | 4k                | 256                 |\n| GPT2-L+RLFC                         | WoW      | 3         | 8              | 1e-5              | -                 | 256                 |\n| GPT2-XL                             | WoW      | 4         | 4              | 6e-5              | 8k                | 256                 |\n| GPT2-XL+K-Dial (FFNs)      | WoW      | 2         | 16             | 6e-5              | 8k                | 256                 |\n| GPT2-XL+K-Dial (Full GPT2) | WoW      | 3         | 4              | 6e-5              | 8k                | 256                 |\n| GPT2-XL+RLFC                        | WoW      | 3         | 4              | 1e-5              | -                 | 256                 |\n| GPT2-M                              | CMU\\_DoG | 4         | 8              | 1e-5              | 4k                | 512                 |\n| GPT2-M+K-Dial (FFNs)       | CMU\\_DoG | 2         | 32             | 1e-5              | 4k                | 512                 |\n| GPT2-M+K-Dial (Full GPT2)  | CMU\\_DoG | 3         | 8              | 1e-5              | 4k                | 512                 |\n| GPT2-L                              | CMU\\_DoG | 4         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-L+K-Dial (FFNs)       | CMU\\_DoG | 2         | 16             | 1e-5              | 8k                | 512                 |\n| GPT2-L+K-Dial (Full GPT2)  | CMU\\_DoG | 3         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-XL                             | CMU\\_DoG | 4         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-XL+K-Dial (FFNs)      | CMU\\_DoG | 2         | 16             | 1e-5              | 8k                | 512                 |\n| GPT2-XL+K-Dial (Full GPT2) | CMU\\_DoG | 3         | 4              | 1e-5              | 8k                | 512                 |\n\n- For the training details of RLFC, lambda is set to 0.95 for KL-penalty.\n    All the hyper-parameters related to PPO are provided default values by the trl PPOConfig recipe [[4]](https://github.com/huggingface/trl/blob/main/trl/trainer/ppo\\_config.py) except the epoch, learning rate, and batch size which are provided as above.\n- During the training of RLFC, all the parameters of GPT2 are updated.\n    We also experimented by applying RLFC on FFNs only but got poor performance results.\n    Since the FFNs are mainly responsible for the factual knowledge expressions [[5]](https://aclanthology.org/2021.emnlp-main.446), only updating parameters in FFN cannot improve the knowledge-grounded dialogue ability compared with fully fine-tuned baseline models.\n    Therefore, RLFC must be used on full model parameters which can implicitly enhance the FFNs' ability to express the factual knowledge in responses.\n\n4. Was there a performance difference between the seen and unseen of WoW?\nWhat specific scale was employed for the Human evaluation? (For instance, a 3-point Likert scale)\n\n- In this work, whether a topic in the test set has been seen or unseen during training is irrelevant to the effect of knowledge enhancement and alignment.\n    There are no distinct performance improvement differences between seen and unseen test sets of WoW using K-Dial and RLFC in our early verification experiments as below, which doesn't affect the conclusions. \n    Therefore, we merge the results of two subsets for all models.\n\n| Model | Subset | KF1 | BERTScore | Verif. | Hallu. | Fact. | BLEU | F1 | RL. |\n|----------------------------|--------------|-----------|-----------------|--------------|--------------|-------------|------------|----------|-----------|\n| GPT2-M                     | Seen         | 47.25     | 38.44           | 13.70        | 11.01        | 5.81        | 28.11      | 60.99    | 8.16      |\n| + K-Dial         | Seen         | 50.07     | 42.23           | 16.77        | 11.62        | 9.39        | 28.26      | 61.63    | 8.97      |\n| + RLFC                   | Seen         | 48.02     | 40.56           | 18.91        | 11.25        | 8.57        | 25.82      | 59.49    | 6.37      |\n| GPT2-M                     | Unseen       | 45.78     | 38.06           | 13.64        | 10.87        | 5.67        | 26.07      | 60.67    | 7.50      |\n| + K-Dial        | Unseen       | 48.66     | 41.71           | 16.57        | 11.47        | 9.33        | 28.20      | 61.33    | 8.41      |\n| + RLFC                   | Unseen       | 46.45     | 40.12           | 18.77        | 11.06        | 8.31        | 25.04      | 59.03    | 6.11      |\n\n- The Human Evaluations are asked to rate each quality on a two-level Likert scale from 0 (Not Flunet, Not Coherent, Inconsistent) to 1 (Flunet, Coherent, Consistent) to evaluate the Fluency, Coherence, and Factual Consistency of generated responses as described in line 479-491.\nThe averaged results of the human evaluations are reported in Table 2 in the paper.\n\n5. How are the final models for each row determined in Table 1? Did you use early stopping or fixed training epoch?\n\nWe use the fixed training epoch to determine the final models in Table 1.\nThe optimal training strategy to obtain the final models is specialized in two stages.\nFirst, we train the GPT2 models using RLFC.\nThen K-Dial method is applied to the obtained model.\nThe fixed training epoch strategy is adopted on both RLFC and K-Dial stages as presented in the Table in Question 3 in this rebuttal.\n\n**Reference**\n\n- [1] [Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation](https://arxiv.org/abs/2110.05456)\n- [2] [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://aclanthology.org/2022.tacl-1.62/)\n- [3] [DialFact: A benchmark for fact-checking in dialogue](https://aclanthology.org/2022.tacl-1.62/)\n- [4] [trl](https://github.com/huggingface/trl/blob/main/trl/trainer/ppo\\_config.py)\n- [5] [Transformer feed-forward layers are key-value memories](https://aclanthology.org/2021.emnlp-main.446)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results demonstrate the effectiveness of the combination of FFNs and RL in the generation of knowledge-based dialogues.", "reasons_to_reject": "There are performance gains associated with the proposed components; however, I am concerned about whether these gains arise from an increase in training steps or the use of additional parameters. K-DIAL and K-DIAL($L_{CE}$) were comparable in most metrics, indicating that Equation 3 does not appear to be advantageous. By addressing the following questions, you might be able to resolve my concerns.", "questions_to_authors": "* Could you provide the number of parameters used for each approach in Table 4? \n* Is the result for CMU_DoG still consistent, similar to that of Table 1 (RLFC+K-DIAL)? Could you also report that result?\n* Additional clarification is needed regarding the model's training process (Lines 241-244). The current information lacks specific training details, such as the number of epochs/steps dedicated to training FFNs and the full model. Moreover, it is unclear whether lambda is utilized for KL-penalty. Furthermore, during the training of RLFC, could you specify whether the training only focuses on the FFNs or LM?\n* Was there a performance difference between the seen and unseen of WoW?\n* What specific scale was employed for the Human evaluation? (For instance, a 3-point Likert scale)\n* How are the final models for each row determined in Table 1? Did you use early stopping or fixed training epoch?", "paper_topic_and_main_contributions": "This paper proposes extended FFN modules in Transformer and reinforcement learning for knowledge-based dialogue generation. The FFN modules aim to ground knowledge-related token to response. The reinforcement learning is inspired by RLHF, reinforcing the model to prefer factually consistent responses using NLI metrics. To validate the effectiveness of the proposed approach, the study conducts experiments on the WoW and CMU_DoG datasets.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "THg3qkcnH2", "reviewer": "Reviewer_BZCd", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable questions. We apologize for the ambiguities in the submitted manuscript. The comments are clarified as follows.   \n   \n   \n\n1. The motivation of this paper is not exciting, and the paper does not provide valuable insights. The authors claim that they focus on promoting the FFN modules in Transformers to improve factual consistency. To my best knowledge, there are lots of related works, such as knowledge editing. Why is there no explicit comparison of their method (i.e. K-Dial) with existing related works? I think the author should discuss the difference between the proposed method and other relevant works in the Related work and experiment, which can provide valuable insights into the subfield of factual consistency.\n\nWe thank the reviewer for the insightful comment and for providing valuable related works.\nThe reviewer has raised concerns about the lack of baseline comparisons with some knowledge editing methods for our proposed K-Dial.\nWe would like to clarify that \n\n- First, we have presented several baseline comparisons of several knowledge injection or enhancement methods including K-Adapter [[1]](https://doi.org/10.18653/v1/2021.findings-acl.121), Kformer [[2]](https://arxiv.org/abs/2201.05742), and Neural Knowledge Bank [[3]](https://arxiv.org/abs/2208.00399) which are adapted on KDS tasks in Table 4 in the submitted paper.\n    Experimental results demonstrate our proposed K-Dial outperforms other baseline methods on both factual consistency and dialogue qualities, suggesting that K-Dial can take advantage of FFN's ability to express factual knowledge given specific input patterns (or prompts) of the concatenated external knowledge and dialogue contexts.\n\n- Second, the reviewer's provided works are knowledge-editing methods that seek and edit alternative parameters to calibrate the incorrect knowledge and make new predictions on revised instances such as [[4]](https://aclanthology.org/2022.findings-emnlp.438/).\n    For KDS, we focus more on integrating the external knowledge base to generate responses in a knowledgeable way by dialogue models.\n    Therefore, the knowledge enhancement method which incorporates external knowledge to PLMs, and mostly into additional modules like adapters [[1]](https://doi.org/10.18653/v1/2021.findings-acl.121) is more appropriate for KDS tasks.\n\n  \n    \n\n2. Some important information is missing. a) The training details of the proposed RLFC. In Section 2.4, although the authors show the overall workflow of RLFC, they do not provide information about reward design and optimization objectives. This omission makes it difficult for researchers to reproduce their work. b) The rating details about human evaluations in the experiments section. In line 479, the authors do not tell the rating details about human evaluations, such as score range and criteria. This makes it difficult for readers to assess the credibility of human evaluations.\n\nWe thank the reviewer for the comment.\nWe would like to clarify that\n\n- The reward design and optimization objectives are all provided in Section 2.4 in the submitted manuscript.\n    The reward model $R(\\cdot)$ is a BERT-based factual consistency NLI classifier described in line 273-288 and line 339-342.\n    The reward score $r_1$ is calculated by $r_1 = R(\\boldsymbol K, \\boldsymbol Y^{(1)})$ where $\\boldsymbol K$ and $\\boldsymbol Y^{(1)}$ denote the grounding knowledge and the response generated by the policy model.\nThe optimization objective $r$ is determined by $r=r_1+ r_2$.\n    $r_1$ is the reward score and $r_2$ is the KL divergence between policy model and reference model generated responses $\\boldsymbol Y^{(1)}$ and $\\boldsymbol Y^{(2)}$, which is introduced in line 289-308 and Figure 3.\n    The hyper-parameters of PPO algorithm are referred to the trl PPOConfig recipe [[5]](https://github.com/huggingface/trl/blob/main/trl/trainer/ppo\\_config.py).\n\n- The Human Evaluations are asked to rate each quality on a two-level Likert scale from 0 (Not Flunet, Not Coherent, Inconsistent) to 1 (Flunet, Coherent, Consistent) to evaluate the Fluency, Coherence, and Factual Consistency of generated responses as described in line 479-491. The averaged results by the human evaluations are reported in Table 2 in paper. \n\n\n3. The experiments are not comprehensive and there is negligence. a) Experiments do not appear to be sufficient to convince of the superiority of the proposed method. There are no reasonable baselines for comparison. Why do not the authors compare their proposed method with any relevant baselines? b) The experiments on the CMU\\_DoG dataset are not comprehensive, lacking RLFC, RLFC+K-Dial, ChatGPT evaluation, and human evaluation on GPT-2 models. Moreover, the authors do not conduct the comparisons and Ablation study shown in Section 4.2. c) The experimental results are not consistent. In Table 4, the GPT2-M + K-Dial achieves 49.36 score in KF1 metric. However, in Table 1, the same model (i.e., GPT2-M+K-Dial) achieves 48.36 score in KF1 metric.\n\n\nWe thank the reviewer for the comment.\nWe would like to clarify that\n\n- For baseline comparisons, to the best of our knowledge, this work is the first to propose to improve factual consistency for KDS.\n    Previous related works that investigate the factual consistency of KDS only focus on evaluation method [[6]](https://aclanthology.org/2021.emnlp-main.619) or datasets [[7]](https://arxiv.org/abs/2110.05456), [[8]](https://aclanthology.org/2022.tacl-1.62/), [[9]](https://aclanthology.org/2022.acl-long.263).\n    Therefore, there is no direct improvement method to be compared.\n    \n    Nevertheless, for the knowledge-enhancing method K-Dial, we still conduct experiments on several knowledge injection and enhancement methods for NLG tasks, including K-Adapter [[1]](https://doi.org/10.18653/v1/2021.findings-acl.121), Kformer [[2]](https://arxiv.org/abs/2201.05742), and Neural Knowledge Bank [[3]](https://arxiv.org/abs/2208.00399), which are adapted on KDS tasks as baselines presented in Table 4.\n    The results suggest that our proposed K-Dial outperforms other baseline methods with improvements in factual consistency while without sacrificing the dialogue qualities.\n -  For RLFC related experiments, the reward model is trained on NLI datasets, but we lack of high-quality human-annotated factual consistency NLI data on CMU\\_DoG.\n    The high-quality fine-grained NLI datasets to evaluate factual consistency including Verification, Hallucination, and Factual Correctness are only collected and annotated on WoW dataset [[7]](https://arxiv.org/abs/2110.05456), [[8]](https://aclanthology.org/2022.tacl-1.62/), [[9]](https://aclanthology.org/2022.acl-long.263).\n    Directly transferring the NLI model on WoW to CMU\\_DoG for RLFC training does not obtain comparable performance improvements.\n    Presumably these fine-grained factual consistency metrics to evaluate KDS are highly dependent on specific datasets.\n    We are trying to explore a more general NLI model for factual consistency evaluation that can adapt to different datasets in the revised manuscript.\n\n- For ChatGPT and Human Evaluations, as well as the comparisons and ablation studies on CMU\\_DoG, due to the page limitation, they cannot be presented in the submitted version. We will supplement all the experiments in future work.\n\n- We apologize for the typo. The correct result is 48.36 of GPT2-M+K-Dial in Table 1 in KF1 metric.\n\n\n**Reference**\n- [1] [K-adapter: Infusing knowledge into pre-trained models with adapters](https://doi.org/10.18653/v1/2021.findings-acl.121)\n- [2] [Kformer: Knowledge injection in transformer feed-forward layers](https://arxiv.org/abs/2201.05742)\n- [3] [Neural knowledge bank for pretrained transformers](https://arxiv.org/abs/2208.00399)\n- [4] [Calibrating factual knowledge in pretrained language models](https://aclanthology.org/2022.findings-emnlp.438/)\n- [5] [trl](https://github.com/huggingface/trl/blob/main/trl/trainer/ppo\\_config.py)\n- [6] [q 2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering](https://aclanthology.org/2021.emnlp-main.619)\n- [7] [Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation](https://arxiv.org/abs/2110.05456)\n- [8] [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://aclanthology.org/2022.tacl-1.62/)\n- [9] [DialFact: A benchmark for fact-checking in dialogue](https://aclanthology.org/2022.acl-long.263)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors attribute the factual inconsistency to the innate limitations of PLMs and propose two solutions to enhance the factual consistency of PLMs, which is reasonable.\n2. The writing is clear and easy to follow.", "reasons_to_reject": "1.\tThe motivation of this paper is not exciting, and the paper does not provide valuable insights. The authors claim that they focus on promoting the FFN modules in Transformers to improve factual consistency. To my best knowledge, there are lots of related works, such as knowledge editing (Refs [1]~[4]). Why is there no explicit comparison of their method (i.e. K-Dial) with existing related works? I think the author should discuss the difference between the proposed method and other relevant works in the Related work and experiment, which can provide valuable insights into the subfield of factual consistency.\n2.\tSome important information is missing. \na)\tThe training details of the proposed RLFC. In Section 2.4, although the authors show the overall workflow of RLFC, they do not provide information about reward design and optimization objectives. This omission makes it difficult for researchers to reproduce their work.\nb)\tThe rating details about human evaluations in the experiments section. In line 479, the authors do not tell the rating details about human evaluations, such as score range and criteria. This makes it difficult for readers to assess the credibility of human evaluations.\n3.\tThe experiments are not comprehensive and there is negligence.\na)\tExperiments do not appear to be sufficient to convince of the superiority of the proposed method. There is no reasonable baselines for comparison. Why do not the authors compare their proposed method with any relevant baselines?\nb)\tThe experiments on CMU_DoG dataset are not comprehensive, lacking RLFC, RLFC+K-DIAL, ChatGPT evaluation and human evaluation on GPT-2 models. Moreover, the authors do not conduct the comparisons and Ablation study shown in Section 4.2.\nc)\tThe experimental results are not consistency. In Table 4, the GPT2-M + K-DIAL achieves 49.36 score in KF1 metric. However, in Table 1, the same model (i.e., GPT2-M+K-DIAL) achieves 48.36 score in KF1 metric.\n\n\nRefs:\n[1]: https://openreview.net/pdf?id=Plr5l7r0jY6\n[2]: https://arxiv.org/pdf/2012.00363.pdf\n[3]: https://arxiv.org/pdf/2205.11482v3.pdf\n[4]: https://openreview.net/pdf?id=ngCT1EelZk\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on improving the factual consistency of PLM-based dialogue systems given the grounding factual knowledge. The authors attribute the factual inconsistency to the innate limitations of PLMs and propose two solutions to enhance the factual consistency of PLMs: 1) They first explicitly extend FFN modules in Transformers to express factual knowledge. 2) They use Reinforcement Learning to implicitly align generated responses with the grounding knowledge for factual consistency. And the authors conduct experiments on WoW and CMU_DoG two datasets to demonstrate the effectiveness of their proposed methods in improving the factual consistency of PLMs.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iFur9bEbMM", "reviewer": "Reviewer_j1xH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comment. Due to the constraints of computation resources, experiments were limited to be conducted on Large-scale LMs which presently entail expensive costs to fully fine-tune the parameters. \n\nRegarding the significant improvements demonstrated by our proposed methodologies which prove the efficiency, we are actively committed to an ensuing endeavor aimed at the extension of K-Dial and RLFC on widely used LLMs like LLaMA and Alpaca in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. By introducing extended FFNs which can be viewed as key-value memories, K-DIAL can effectively update the knowledge into the network parameters.\n\n2. Inspired by RLHF, RLFC effectively encourages the model to output knowledge-consistent responses.\n\n3. Experimental results show the effectiveness of the proposed method.", "reasons_to_reject": "1. As mentioned in Section 4.1.1, the enhancement achieved by the (GPT2-XL) method is relatively limited when applied to larger models. As stated in Section Limitation, methods have not yet been validated on larger scale language models.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed K-DIAL (knowledge-enhanced dialogue generation) and RLFC (reinforcement learning for factual consistency) to enhance the factual knowledge expression capability of FFNs in Transformers by introducing extended FFNs and aligning the responses with gold knowledge respectively.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1IKA5Sd81w", "reviewer": "Reviewer_RZpp", "summary": "", "questions": "see my Reasons To Reject", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable questions. We apologize for the ambiguities in the submitted manuscript. The comments are clarified as follows.\n\n1. The first K-Dial can not improve factual consistency. It's just a technique that makes the model more active toward generating entity words. To cover up this issue, the authors made a strong but not practical statement that the given knowledge is always correct. In fact, it is hard to retrieve the totally correct knowledge. Meanwhile, even if we do not consider this statement, I think the copy mechanism will be more powerful in leveraging the given knowledge compared to using extended FNNs. Using extended FNNs can be regarded as a simplified copy net.\n\nWe thank the reviewer for the insightful comment!\nThe reviewer has raised specific questions regarding the efficacy of the proposed K-Dial to improve factual consistency for knowledge-grounded dialogue systems (KDS). We would like to clarify that\n\n- First, the factual consistency in KDS indicates \"accurately portrays the input knowledge (assuming the provided knowledge is correct)\" as defined in prior research [[1]](https://arxiv.org/abs/2110.05456) which makes a distinction between \"factual consistency\" and \"factual correctness\".\n    Factual consistency denotes whether the response utilizes and follows the provided knowledge, where the similar definitions of \"attributed\" and \"supported\" are used in [[2]](https://aclanthology.org/2022.tacl-1.62/) and [[3]](https://aclanthology.org/2022.acl-long.263) respectively.\n    For simplicity, we directly leverage the gold knowledge of WoW and CMU\\_DoG thus the input knowledge is naturally correct in this work.\n    Therefore, activating the entity words expressions of given knowledge in responses is an effective way to improve factual consistency.\n- Second, The proposed K-Dial is not just a copy net.\n    The K-Dial can better learn and understand on semantics of the knowledge snippet and dialogue contexts by taking advantage of FFN's ability to activate the factual knowledge outputs in provided evidence that are highly related to the dialogue context.\n    The copy mechanism cannot learn to understand the semantic relations between the provided knowledge and dialogue contexts.\n\n2. The second RLFC can improve factual consistency, but the idea is a little incremental.\n\nWe thank the reviewer for the comment.\nWe would like to clarify that the proposed RLFC for KDS is specialized in two aspects:\n\n- We collect several released high-quality human-annotated datasets for reward model training.\n    The datasets collect a large number of instances of evaluating the factual consistency of KDS in systematically fine-grained metrics including Verification, Hallucination, and Factual Correctness [[1]](https://arxiv.org/abs/2110.05456).\n    The reward model benefits from such high-quality data to better align with the factual consistency preference of human beings. \n    Therefore, RLFC demonstrates significant performance improvements in factual consistency on KDS tasks.\n- The RLFC implicitly adjusts the FFN's expression, which is a complementarity of K-Dial which explicitly enhances the factual knowledge expressions of FFNs.\n    The supreme performance improvements over baseline GPT2 models are obtained using the combination of RLFC and K-Dial methods.\n\n3. This work is a little hard to follow, and thus this work should improve the organization. For example, 1) the description heavily relies on the given figure, but the figure is ambiguous; 2) the detail of K-Dial is not clear.\n\nWe thank the reviewer for the comment.\nAs the training procedure of the proposed method contains several stages that are a bit complex, all the information is difficult to present in one compressed figure.\nWe would like to try to revise the figure details and make it clear in the following version.\nWe will also supplement the additional implementation details of K-Dial in lines 237-244 in the submitted manuscript for assistance as the following quoted texts.\n\n\"The training process of the K-Dial framework is specified in two steps. First, all the parameters of the original PLM are frozen, and the loss $\\mathcal L_{K-Dial}$ is only calculated over the knowledge entities in $\\boldsymbol Y$ to optimize the parameters of the extended FFNs.\nAfterward, we further adapt the knowledge-enhanced model on the knowledge-grounded dialogue datasets using supervised fine-tuning of Equation (1) while keeping the parameters in extended FFNs fixed.\nThe word embedding dimension and hidden size of the extended FFN module are set equal to the corresponding Transformer FFN layers.\nNoted that the K-Dial framework is only applied on the top 3 layers of the model in our experiments.\"\n\nWe present the related hyper-parameters of the baseline model, two stages of K-Dial to train extended FFNs and the full model, and RLFC on both WoW and CMU\\_DoG datasets as below.\n\n| Model                           | Dataset  | Epoch | Batch Size | Learning Rate | Warm-up Steps | Sequence Length |\n|-------------------------------------|--------------|-----------|----------------|-------------------|-------------------|---------------------|\n| GPT2-M                              | WoW      | 4         | 16             | 6e-5              | 2k                | 256                 |\n| GPT2-M+ K-Dial (FFNs)        | WoW      | 2         | 32             | 6e-5              | 2k                | 256                 |\n| GPT2-M+ K-Dial (Full GPT2)  | WoW      | 3         | 16             | 6e-5              | 2k                | 256                 |\n| GPT2-M+RLFC                         | WoW      | 3         | 16             | 1e-5              | -                 | 256                 |\n| GPT2-L                              | WoW      | 4         | 8              | 6e-5              | 4k                | 256                 |\n| GPT2-L+K-Dial (FFNs)       | WoW      | 2         | 24             | 6e-5              | 4k                | 256                 |\n| GPT2-L+K-Dial (Full GPT2)  | WoW      | 3         | 8              | 6e-5              | 4k                | 256                 |\n| GPT2-L+RLFC                         | WoW      | 3         | 8              | 1e-5              | -                 | 256                 |\n| GPT2-XL                             | WoW      | 4         | 4              | 6e-5              | 8k                | 256                 |\n| GPT2-XL+K-Dial (FFNs)      | WoW      | 2         | 16             | 6e-5              | 8k                | 256                 |\n| GPT2-XL+K-Dial (Full GPT2) | WoW      | 3         | 4              | 6e-5              | 8k                | 256                 |\n| GPT2-XL+RLFC                        | WoW      | 3         | 4              | 1e-5              | -                 | 256                 |\n| GPT2-M                              | CMU\\_DoG | 4         | 8              | 1e-5              | 4k                | 512                 |\n| GPT2-M+K-Dial (FFNs)       | CMU\\_DoG | 2         | 32             | 1e-5              | 4k                | 512                 |\n| GPT2-M+K-Dial (Full GPT2)  | CMU\\_DoG | 3         | 8              | 1e-5              | 4k                | 512                 |\n| GPT2-L                              | CMU\\_DoG | 4         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-L+K-Dial (FFNs)       | CMU\\_DoG | 2         | 16             | 1e-5              | 8k                | 512                 |\n| GPT2-L+K-Dial (Full GPT2)  | CMU\\_DoG | 3         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-XL                             | CMU\\_DoG | 4         | 4              | 1e-5              | 8k                | 512                 |\n| GPT2-XL+K-Dial (FFNs)      | CMU\\_DoG | 2         | 16             | 1e-5              | 8k                | 512                 |\n| GPT2-XL+K-Dial (Full GPT2) | CMU\\_DoG | 3         | 4              | 1e-5              | 8k                | 512                 |\n\n\n4. Although both K-Dial and +RLFC have shown performance improvements, this work did not compare to any other related solutions besides the naive GPT-2.\n\nWe thank the reviewer for the comment.\nFor other related solutions, to the best of our knowledge, this work is the first to propose to improve factual consistency for knowledge-grounded dialogue systems (KDS).\nPrevious related work that investigates the factual consistency of KDS focuses on evaluation method [[7]](https://aclanthology.org/2021.emnlp-main.619) or datasets [[1]](https://arxiv.org/abs/2110.05456), [[2]](https://aclanthology.org/2022.tacl-1.62/), [[3]](https://aclanthology.org/2022.acl-long.263).\nTherefore, there is no direct research to compare with our proposed improvement method except baseline supervised fine-tuned GPT-2 models.\n\nNevertheless, for the knowledge-enhancing method K-Dial, we still conduct experiments on several knowledge injection and enhancement methods for PLMs including K-Adapter [[4]](https://doi.org/10.18653/v1/2021.findings-acl.121), Kformer [[5]](https://arxiv.org/abs/2201.05742), and Neural Knowledge Bank [[6]](https://arxiv.org/abs/2208.00399) and adapt on KDS tasks as presented in Table 4 in the submitted manuscript.\nExperimental results suggest that our proposed K-Dial outperforms other baseline methods with improvements in factual consistency with grounding knowledge without sacrificing the dialogue qualities.\n\n**Reference**\n- [1] [Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation](https://arxiv.org/abs/2110.05456)\n- [2] [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://aclanthology.org/2022.tacl-1.62/)\n- [3] [DialFact: A benchmark for fact-checking in dialogue](https://aclanthology.org/2022.acl-long.263)\n- [4] [K-adapter: Infusing knowledge into pre-trained models with adapters](https://doi.org/10.18653/v1/2021.findings-acl.121)\n- [5] [Kformer: Knowledge injection in transformer feed-forward layers](https://arxiv.org/abs/2201.05742)\n- [6] [Neural knowledge bank for pretrained transformers](https://arxiv.org/abs/2208.00399)\n- [7] [q 2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering](https://aclanthology.org/2021.emnlp-main.619)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work has identified an urgent issue in the research of knowledge-grounded dialogue generation.\n\n2. The evaluation design of this work is comprehensive and strong because massive metrics are introduced and adopted in this work. This evaluation design could provide a demonstration for relevant researchers.", "reasons_to_reject": "1. The first KDial can not improve factual consistency.  It's just a technique that makes the model more active toward generating entity words. To cover up this issue, the authors made a strong but not practical statement that the given knowledge is always correct. In fact, it is hard to retrieve the totally correct knowledge. Meanwhile, even if we do not consider this statement, I think the copy mechanism will be more powerful in leveraging the given knowledge compared to using extended FNNs.  Using extended FNNs can be regarded as a simplified copy net.\n\n2. The second RLFC can improve factual consistency, but the idea is a little incremental.   \n\n3. This work is a little hard to follow, and thus this work should improve the organization. For example, 1) the description heavily relies on the given figure, but the figure is ambiguous; 2) the detail of K-DIAL is not clear.\n\n4. Although both KDial and +RLFC have shown performance improvements, this work did not compare to any other related solutions besides the naive GPT-2.", "questions_to_authors": "see my Reasons To Reject", "paper_topic_and_main_contributions": "This work studies the knowledge-grounded dialogue response generation and focuses on improving factual consistency.\n\nIn knowledge-grounded dialogue response generation,  the response generation is guided by the context and the retrieved knowledge. The authors have assumed that such two inputs are always correct and then attributed the reason to the backbone PLMs because the prior knowledge in PLMs learned from large-scale unlabeled corpus might be incomplete.\n\nConsequently, this work first explicitly adds a new FFN unit to the Transformer to boost the prediction of factual knowledge and then proposes a reinforcement learning process to improve the factual consistency with a binary classifier.\n\nUpdate：I have updated my soundness score to 3 due to the additional experiments.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents K-DIAL which introduces extensions to the FFN in Transformer neural architecture, and apply a reinforcement learning for factual consistency method to align the responses with ground-truth knowledge. The presentation clarity in the paper leaves room for further improvement. While the authors did clarify  some of the important details missing from the original paper during rebuttal, these should be incorporated back into the manuscript.
Decision: Accept-Findings