Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialogue
Authors: Lang Qin
YAO ZHANG
Hongru Liang
Jun Wang
Zhenglu Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge-grounded Dialogue
Knowledge Selection
Generator-agnostic

Abstract: Accurate knowledge selection is critical in knowledge-grounded dialogue systems. 
Towards a closer look at it, we offer a novel perspective to organize existing literature, i.e., knowledge selection coupled with, after, and before generation. 
We focus on the third under-explored category of study, which can not only select knowledge accurately in advance, but has the advantage to reduce the learning, adjustment, and interpretation burden of subsequent response generation models, especially LLMs.
We propose $\tt{GATE}$, a generator-agnostic knowledge selection method, to prepare knowledge for subsequent response generation models by selecting context-related knowledge among different knowledge structures and variable knowledge requirements.
Experimental results demonstrate the superiority of $\tt{GATE}$, and indicate that knowledge selection before generation is a lightweight yet effective way to facilitate LLMs (e.g., ChatGPT) to generate more informative responses.

Well Begun is Half Done: Generator-agnostie Knowledge Pre-Selection for Ghoollegg-Grrunddrr Dialogue
Lang Qin Yao Zhang *3 Hongru Liang* Jun Wang and Zhenglu Yang*
TKLNDST, CS, Nankai University *Key Laboratory of DISSec, Ministry of Education, China SSchool of Statistics and Data Science, LPMC, KLMDASR & LEBPS, Nankai University *College Computer Science, Sichuan University
CCollege of Mathematics and Statistics Science, Shandong Key Laboratory of Language Resource Development and Application, Ludong University
"Education Field Integrated & Publishing Knowledge Mining & Service Key Laboratory,
National Press and Publication Administration, China
ginlang14@mail..om (emaall.annaii. www.aa.aauuu....mmemommoo en, yaozhangenankai. edu. cn, Lianghongru@scu.cm wwwiccuedd..co cn, junwangemail 1. nankai. edu. en, yangzlenankai. edu. cn
Abstract
into three categories based on the occasion when knowledge selection is performed. The first cate- gory co-selection (Zhao et al., 2020; Tuan et al., 2022; Bai et al., 2023) wherein knowledge selec- tion and response generation are executed in a cou- pled manner (c.f. Figure 1 (b)-O). Although this category of approach is efficient and has been ex- tensively researched, it is costly to learn and is difficult to interpret and adjust when errors arise in the generated responses. The second category is post-selection (Dziri al., 2021; Xue et al., 2022), namely, the knowledge is selected after the genera- tion and used to correct the knowledge error in the generated response (c.f. Figure 1 (b)-2). This category is skilled in adjusting local errors yet has minimal impact on enhancing the informativeness of the response.
Accurate knowledge selection is critical in knowledge-grounded dialogue systems. To- wards closer look at it, we offer novel perspective to organize existing literature, i... knowledge selection coupled with, after, and before generation. We focus on the third under- explored category of study, which can not only select knowledge accurately in advance, but has the advantage to reduce the learning, adjust- ment, and interpretation burden of subsequent response generation models, especially LLMs. We propose GATE, generator-agnosti knowl- edge selection method, to prepare knowledge for subsequent response generation models by selecting context-related knowledge among different knowledge structures and variable knowledge requirements. Experimental results demonstrate the superiority of GATE, and indi- cate that knowledge selection before generation is lightweight yet effective way facilitate LLMs (e.g., ChatGPT) to generate more infor- mative responses.
Surprisingly, the third category, pre-selection, has been under-explored by previous research. This category performs knowledge selection as an in- dependent model before response generation (c.f. Figure (b)-@). In this paper, we pay attention to this study, not only because of the overlook by current work but also because selecting knowledge accurately advance can provide the potential reduce the learning, adjustment, and interpretation burden of subsequent response generation models, especially for large language models (LLMs). To select knowledge for preparation, in addition to the accuracy required for knowledge selection, there are two key issues that need to be addressed:
1 Introduction
Knowledge-grounde dialogue systems generate informative responses by incorporating external knowledge, such unstructured documents and structured knowledge graphs (Ghazvininejad et al., 2018; Lian et al., 2019). This generation process re- quires an agent to select context-related knowledge to support high user engagement. Taking Figure 1 (a) as an example, the knowledge "Mark Boal wrote Zero Dark Thirty" contributes to high-quality re- sponse compared with the knowledge "Zero Dark Thirty has genre War film", when the user focuses on "who wrote Zero Dark Thirty".
Different knowledge structure. An ideal knowl- edge selector should be able to tackle differ- ent knowledge structures, such as unstructured knowledge represented by text and structured knowledge represented by knowledge graphs.
Many efforts have been devoted to selecting context-relevant knowledge, and we classify them
*Corresponding author.
Variable knowledge requirement. Typically, the

---

Know ledge selection I
Knowledge selection
Know ledge selec
Do have anything by direcor Kathrya Higelow?
coupled with Generation
after Generation
before Generation
Dislogue Cantest
Knowledge Rasa
Dislogue
Knowledge
Dialogue
Knowledge
Contest
Base
Contect
Base
Alwalitdy. She dincctd Zem Dark Thirty and The Hurt Locker fetiliar with them?
Response Generation
Knowledge
Knowledge Selection
That's
Thirsy
Zero Dark Thitt was writion Eric Rod
Mark Boal wnote Zer Dark Thiry, Mark Beal wrose lriple Fruntier.
Jf
Mack Boal wrote Zaro Thiry.
Regganee Generution
Thity has geore film
Knowledge Selection
Mack Bost woote Triple Frontier.
Mark Boal wrotc Zen Dark Thitt.
Response Generation
Kathryn Bigslow has directed war fimss
Zero
Dark
Thisty
was
was writien by Mark Boal, who setually also Triple Frontler.
Willles by Mark Boal
(a) Example kaooledge-goounded dialogue
(b) Three categories kuowledge selection
Figure I: a) An example of knowledge-groundee dialogue. each turn, the dialogue agent generates response by selecting knowledge from the knowledge base. b) Three categories of knowledge selection used in the knowledge- grounded dialogue system: 1 knowledge selection coupled with generation cio-selection). @ knowledge selection after generation (post-solection), and @ knowledge selection before generation (pre-selection). Notice that the dashed rectangular box refers to an independent model.
number of knowledge required to generate an in- formative response is neither one nor fixed num- ber but a variable number. For example, when the dialogue about an wrining film rather than a little-known one, there is much more rel- evant knowledge involved. An ideal knowledge selector should be able to dynamically adapt the number of selected knowledge.
literature of knowledge selection in knowledge- grounded dialogue, i.e., knowledge selection cou- pled with, after, and before generation. Besides, we point out that the third category of study, though under-explored, has advantages to reduce the learning, adjustment, and interpretation bur- den of subsequent response generation models.
We propose GATE, a gemerator-agnostie knowl- edge selection method, to prepare knowledge for subsequent response generators by select- ing context-related knowledge among different knowledge structures and variable knowledge re- quirements.
To resolve the above issues, we propose GATE, a Generator-AgnosTic knowledgE selection method prepare knowledge for subsequent various re- sponse generation models, e.g., BART (Lewis et al., 2020) and T5 (Raffel al.. 2020). GATE has the ability to confront both unstructured and structured knowledge and to adapt the number of desired knowledge according to different dialogue contexts. Specifically, GATE consists of: knowledge struc- ture unifying, knowledge scoring, and knowledge pool size adapting module. Further, we employ reinforcement learning (RL) framework to train GATE, optimizing the reward of selecting appropri- ate knowledge in both quality and quantity. The experimental results on two datasets demonstrate the superiority of GATE on knowledge selection, and that GATE can facilitate the response genera- tion model (including ChatGPT) to generate more informative responses. We believe that GATE pro- vides lightweight and efficient solution, which is potentially viable way for reducing the learning, adjustment, and interpretation burden of LLMs. In summary, the main contributions of this work are as follows.
We conduct experiments to demonstrate the su- periority of GATE, and find that knowledge pre- selection a lightweight and effective way to facilitate ChatGPT to generate more informative responses.
2 Related Work
Knowledge selection is a crucial step in the knowledge-groundee dialogue system. Our work provides new perspective to review the literature of knowledge-groundee dialogue- dialogue- based on dif- ferent time points of knowledge selection in the response generation process (c.f., Figure 1(b)). Knowledge selection coupled with Generation This knowledge selection category denotes that the knowledge selection and response generation pro- cesses are modeled to be executed concurrently in a single model, as shown in Figure (b)-O. For unstructured knowledge, the Co-Selection pro- cess is an interactive matching process between the
We introduce a novel perspective to organize the

---

dialogue and the documents (Meng al., 2020, 2021). Dinan et al. (2019) utilizes dot product attention to select the most relevant knowledge. Bai et al. (2023) improves selection by enhancing knowledge's s dense representation. For structured knowledge, the selection process can be viewed as multi-hop reasoning procedure on a graph, which is subsequently followed by two-stage architec- ture for response generation (Liu et al., 2019; Zhou al., 2021; Tuan al., 2022). However, the con- struction or training of the above methods is tied to the generation model. In contrast, GATE is "plug- and-play" and can enhance response generation for various generation models.
to specific knowledge types or generation models, significantly limiting the generalization ability, and rely on fixed-size knowledge pool, which could undermine the performance of response (Shuster et al., 2021). In contrast, GATE operates in a Pre- Selection category that can handle diverse knowl- edge types and adapt the knowledge pool size to enhance various generation models.
3 Methodology
3.1 Overview
We first formulate the knowledge grouuddd dia- logue as follows: at t-turn conversation, given the dialogue history X, and currently accessible knowledge base K. the system generates response V based on the dialogue-relevant knowledge set Kr C K. Then, the knowledge pre-selection task that we focus refers to how to select more ful knowledge set K. from the knowledge set KG, before the response generation process. "Useful" here means that KC can help the agent generate high-quality responses.
Knowledge selection after Generation This knowledge selection category denotes that knowl- edge selection is executed as an independent model after response generation, as shown in Figure I (b)-2) Post-selection dedicated to correcting po- tential knowledge errors in the response. Dziri et al. (2021) address hallucinations in responses by re- placing them with correct knowledge obtained from the knowledge graph. Xue et al. (2022) employ text infilling incorporate retrieved knowledge into incomplete responses. However, these meth- ods would diminish the fluency and naturalness of responses, whereas GATE does not compromise generation models.
To select a more useful knowledge set K2, GATE performs three efforts:
Unifying knowledge of diverse structure types (unstructured and structured), improve the generalization and flexibility of GATE.
Knowledge selection before Generation This knowledge selection category denotes that knowl- edge selection is executed as independent model before response generation, as shown in Figure I (b)-O) Pre-selection dedicated improving the accuracy of knowledge selection and further en- hancing the quality of response generation. A few works have actually employed Pre-Selection without emphasizing or providing a formal defi- nition. Jung et al. (2020) implement graph-based reasoning through attention flows to select knowl- edge (i.e., paths in the graph). Eric et al. (2021) collected an augmented dataset and proposed pipeline to evaluate it. Li et al. (2022) constructs semantic graphs of textual knowl- edge and performs selection based on node similar- ity. Yang et al. (2022) proposed a topic-shift aware knowledge selector to utilize the role-initiative in- formation to help select knowledge. However, the above methods are designed to address specific knowledge types, while GATE can select knowledge across different knowledge bases.
Scoring knowledge to help select the knowledge that is more relevant to the desired response.
Adapting knowledge pool size to provide appro- priately sized knowledge sets for subsequent sponse generation models.
We will introduce the details of GATE in Sec- tion 3.2, the reinforcement learning framework for GATE in Section 3.3, and the optimization and train- ing details of GATE Scction 3.4.
3.2 GATE
Knowledge Structure Unifying GATE first uni- fies knowledge of diverse structure types to im- prove its generalization and flexibility. In general, there are two types of knowledge structures used by the knowledge- knowledge-rrounded grounded dialogue system: un- structured and structured knowledge. Unstructured knowledge generally exists in the form of docu- ments, and structured knowledge typically uses knowledge graphs. Graph structures are lengthy in modeling the association information between
In summary, existing methods are constrained

---

(4) Unstructured Knowledger Document
transition:
Topie
Article
Article
ScoTe, GAT ([St)e)) for in Adj:) (1)
Sentence
Senteace
Sentence
The knowledge nodes of sorted process nodes form KG. Considering the guiding role of process node scores knowledge selection, we calculate node attention weights based on the score distri- bution using MLP. These weights are utilized dot-product calculations with the Agent state and knowledge encoding to determine the score of each knowledge in Kg--
(b) Structured Knowledge: Knowledge Graph
Yiz
Yas
Figure Knowledge Structure Unifying
Knowledge Pool Size Adapting GATE determines the knowledge pool's s appropriate size by analyz- ing the node score distribution variance. The top- ranked knowledge selected to constitute K where (.) maps the input to the interval [0,1]:
knowledge and help select useful knowl- edge more accurately. Moreover, graph structures can enhance the interpretability of the knowledge selection process. Therefore, GATE uniformly trans- forms all the diverse types of knowledge into a graph structure, which has two types of nodes, i.e.. process node and knowledge node.
)
|K = [K2]] M
Var (score.))
(2)
For document-based unstructured knowledge, naturally has a topic >> article title > sentence hierarchy. GATE presents this hierarchy as graph, where topics and article titles used as process nodes in the graph, sentences used as knowl- edge nodes, and nodes are connected nodes by edges if there is a containment relation, as shown in Figure (a). For the structured knowledge graph, GATE keeps its original structure unchanged, and all the original nodes are used as process nodes. GATE will additionally add knowledge nodes! For the triple (ei, Tajs Ej) formed by two entity nodes (e; and ej) and the relation edge (ri;)) between them, GATE merges them into single knowledge node and connects them to the head entity (ej) in this triple, as shown in Figure? (b).
3.3 RL Framework for GATE
Graph-based reasoning improves the interpretabil- ity of knowledge selection. Due to the extensive use and success of reinforcement learning in this field, we formulate the knowledge selection pro- cess as Markov Decision Process and employ reinforcement learning for graph-based reasoning. State We employ SentenceBert (Reimers and Gurevych, 2019) to perform static encoding of the nodes and knowledge within the graph structure outlined in Section 3.2. We encode each piece of 10,.1
knowledge attached node as {cx,) and use the mean-pont paeaann of these encodings as the node's
encoding: en. McanPool {(e) |s,,
We
leverage KeyBert (Grootendorst, 2020) to extract keywords W; from the conversation history X1=-11 and user statement X,. Then, we use modified multi-head hierarchical modality attention mecha- nism (Moon et al., 2018) process all input infor- mation: I Attention (Xi=-1, X,, W.).
Knowledge Scoring After unifying the knowledge structure, GATE next scores each process node in the graph to help select the knowledge that is more relevant to the desired response.
Firstly, a subgraph Adj. is obtained based on the valid state transition target of the Agent. As shown in Figure 3, the encoding of the Agent's state is concatenated with the encoding of each process node in the subgraph. We utilize Graph Attention Network (GAT) (Brody al., 2022) to score each node and then sample to obtain the target for state
After initializing and encoding the graph, we employ GAT to update the encoding of the entire graph at the beginning of the Agent's traversal:
{en, As the Agent traverses the graph, we update its state considering the node en, that the Agent is located at time t:
)em,)ee i=
|en,
GAT
has been demonstrated that using fact triples can help dialogue systems generate high-quality responses hetter than separate entities and relations. (Dziri et al., 2021; Sarkar et al., 2022).
S, = Attention (z; S em.)
(3)

---

Dialogue Context
Process Node Scoring
Knowledge Node
Scoring
Knowledge Base
Process Node
Knowlodge Node
Figure 3: Knowledge scoring in GATE
Action The Agent's 8 traversal on the graph is Markov Decision Process with maximum step count of T. The Agent's action encompasses mov- ing to the next node and constructing the corre- sponding knowledge pool. The action space the set of one-hop neighbors of current node. Reward To accomplish our objective of enhanc- ing the quality and quantity of knowledge selection, we optimize the model in three ways:
We use the following stochastic gradient based the REINFORCE algorithm (Williams, 1992) to optimize A:
VgJ = V8 Σ Rs. log Tee (at 81) (7)
In order effectively utilize the available pervised information inherent in the graph, we integrate node loss Nade and knowledge loss
L Knowledge using standard cross-entropy. We de- rive LWatk by taking the negative value of J(0). Then, the complete loss function is as follows:
Enhancing the Agent's ability traverse the graph and reach the correct process node, we have RNde as positive if the Agent halts at the correct process node and negative otherwise.
L = LWalk + L + Knoutedgee
(8)
Improving the ability select appropriate knowl- edge from the knowledge pool, we design RGol:
The introduced model uniquely incorporates the aforementioned selection methods while maintain- ing its independence from any specific generator.
Roold Max (1 a (kigtta) :-1) (4)
where r refers to the rank of ground-truth knowl- edge kgold in Kit and Roold is negative if not select kgold.
4 Experiments
We conduct experiments in knowledge selection and response generation to investigate the effec- tiveness of knowledge Pre-Selection in GATE. We also analyze the auxiliary capability of knowledge Pre-Selection on the generation model and the ad- vantages of GATE adaptive knowledge pool size.
Adaptively determining an appropriate knowl- edge pool size. If the model selects the cor- rect knowledge with high probability, excessive knowledge can be deemed redundant. Otherwise, expanding the knowledge pool is necessary to enhance knowledge effectiveness. Hence, we design Poot as R.oaa [-
4.1 Datasets
We conduct experiments on Wizard of Wikipedia (WoW) (Dinan et al., 2019) and OpenDialKG (Moon et al., 2019) datasets for unstructured and structured knowledge bases. The statistics of the two datasets are presented in Appendix A.
The complete reward function is follows:
R2: RNode + RGold + Rool
(5)
Policy Network We design a policy network no (at 9t) = (at s;;0) 0) to obtain the proba- bility distribution of actions, where A represents the model parameters utilized in the Knowledge Scoring process and RL Framework.
In WoW, two participants take roles as wiz- ard and an apprentice. The wizard selects proper knowledge (sentence) from Wikipedia for the re- sponse. WoW split test set into Seen and Unseen based on topics. OpenDialKG is parallel corpus comprising open-domain dialogues and knowl- edge graph. The reasoning path of each turn is annotated, enabling participants to utilize graph information during the conversation.
3.4 Optimization and Training
The optimization objective of our policy network is maximize the expected cumulative reward:
Due to the absence of an official split in Open- DialKG, we follow WoW by dividing the dataset into Seen/Unseen categories based on topics.
J(0) Ee-n((al:0))
Σ Ris
(6)

---

Test Seen
Test Unseen
Method
PLM
Rouge
Meteor
BLEU
Rouge
Metcor
BLEU
F1
WoW
Random
BART BART
16.06
15.21
1.43
19.16
16.
15.33
1.44
19.15
Semantic
18.26
17.35
2.46
21.09
18.64
17.76
2.55
21.35
TMNet
16.34
13.87
0.85
17.49
14.98
12.25
0.51
13.20
SKT++
18.68
17.33
2.43
19.83
17.09
14.75
1.68
17.46
MIKE
19.13
18.07
2.75
19.70
17.25
15.64
1.91
17.12
KnowledGPT
GPT2
20.85
21.18
3.62
22.03
19.60
19.53
3.09
20.48
KINET
BART
21.47
21.63
3.84
22.45
20.54
20.48
3.47
21.45
20.17
20.36
3.75
22.34
19.05
18.98
3.15
21.28
GATE
T5
20.19
+0.14
=0.13
+0.08
#0.12
24.15
23.14
5,81
26.23
23.59
22.69
5,41
25.67
GATE
BART
20.04
#0.02
+0.04
20.09
=0.03
#0.13
40.06
OpenDialKG
Random
BART
24.52
24.31
4.97
28.45
24.29
24.02
4.60
28.43
Semantic
BART
25.84
25.64
5.67
29.59
25.55
25.30
5.29
29.58
D
BART
25,91
25,86
5.97
29.71
25.32
25.01
5.19
29.43
A
BART
27.04
26.86
6.37
30.94
26.65
26.44
5.60
30.65
DifIKG
T5
18.27
16.87
1.23
21.20
17.57
16.06
0.94
20.58
NPH
GPT2
27.53
27.56
6.71
31.34
26.97
27.01
5.76
30.51
23.88
23,09
4.26
26.36
23.58
22.63
3.70
26.17
GATE
T5
+0.05
+0.04
28.57
28.46
7.20
32.21
27.93
27.78
6.41
31.57
GATE
BART
20.01
20.03
20.05
20.05
20.08
+0.05
10.07
Table 1: The evaluation results of response generation on WoW and OpenDialKG datasets. The baseline model
results for WoW are reported from Bai et al. (2023). denotes our re-implementation.
4.2
Baselines
3) Methods using Post Post-Selection
We choose the following four types of baselines: I) Trivial baselines
NPH (Dziri et al., 2021): it retrieves correct enti- ties by crafting query signal propagated over graph to refine hallucination in response.
Random: it randomly selects knowledge from the candidate pool;
4) Methods using Pre-Selection
Semantic: selects knowledge based on seman- tic similarity between dialogue and knowledge.
DialKG (Moon et al., 2019): it models the sym- bolic transitions as structured traversal on KG and predicts entities with a graph path decoder;
2) Methods using Co-Selection
TMNet (Dinan et al., 2019): it combines memory network architecture and Transformer to encode dialogue and knowledge for response generation;
AttnIO (Jung et al., 2020): it flexibly adjusts the nodes and edges focus based on dialogue context via attention flow.
SKT++ (Chen al., 2020): it utilizes Posterior Information Prediction Module and Knowledge Distillation-Base Training Strategy:
Among the aforementioned baselines, TMNet, SKT++, MIKE, KnowledGPT, and KINET are uti- lized for unstructured knowledge, whereas DialKG, AttnIO, DiffKG, and NPH are utilized for struc- tured knowledge.
MIKE (Meng et al., 2021): it introduces an ini- tiative discriminator for knowledge selection;
KnowledGPT (Zhao al., 2020): it utilizes pre-trained language models and optimizes via unsupervised learning;
4.3 Metries
GATE strives to improve the accuracy of knowledge selection and further enhance the quality of re- sponse generation. Following previous works (Jung al., 2020; Meng et al., 2021: Bai al., 2023), we employ 1) ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), Meteor (Denkowski and Lavie, 201 4), and unigram overlap (F1) (Dinan et al., 2019) to evaluate the quality of the generated responses; and
KINET (Bai et al., 2023): it introduces negative-enhanced knowledge approximator and curriculum knowledge sampler;
DiffKG (Tuan et al., 2022): it employs Trans- former to generate relation sequences on KG and generates responses based on retrieved entities.

---

Method
PLM
Test Seen
Test Unseen
Engaging
Informative
Method
WoW
OpenDialKG
WoW
OpenDialKG
+5 Pieces Knowledge
Random
BART
1.52
1.47
ChatGiT
44.0%
46.0%
33.0%
31.5%
Semantic
BART
6.57
6.87
+ GATE
42.0%
47.0%
44.5%
42.0%
TMNet
Transformer
22.50
12.20
+10 Pieces of Knowledge
TMNet
BERT
23.86
16.33
ChatGP
46.5%
37.0%
35 5%
29.5%
SKT++
27.62
20.20
GATE
42.5%
55.0%
48,0%
49.5%
MIKE
28.41
21.47
Table 3:
The evaluation results of ChatGPT on 200
KnowledGPT
GPT2
28.00
25.40
response pairs. The percentages represent the proportion of responses that outperform another.
KINET
BERT
29.38
27.05
KINET
BART
28.90
27.14
GATE
T5
31.81
27.60
AIRTO-AS
DiNKG
GATE
BART
0125
61/6
59.51
OpenDialKG
54.33
Random
BART
0.93
0.66
49:51
Semantic
BART
13.13
13.44
16
D
BART
19.92
17.22
21-72
26-8
AttuIO*
BART
24.96
22.67
22.6
13:2
DiffKG
T5
29.31
23.65
T5
30.21
27.72
GATE
BART
+0.15
+0.28
1:600
Table The evaluation results (R@1) 1) of knowledge selection the WoW and OpenDialKG datasets. "sp" denotes our re-implementation. TMNet results are re- ported from Meng al. (2021).
Figure 4: The evaluation results on the OpenDialKG origin data. The baseline model results are reported from Tuan et al. (2022).
knowledge selection as measured by R@ 1. Our reinforcement learning bearning-based graph reasoning method for unified knowledge types effectively selects relevant knowledge.
2) Recall@k, which calculates the percentage of ground-truth knowledge included in the top-k selec- tions, to evaluate the knowledge selection accuracy.
4.4 Implementation Details
GATE outperforms models utilizing the same gen- eration module (e.g., Bart, T5) across multiple metrics. The knowledge selected by GATE signifi- cantly enhances generation modules.
We employ AdamW optimizer (Loshchilov and Hutter, 2017) with weight decay 0.12 and OneCy- cle policy (Smith and Topin, 2019). The activation function LeakyReLU with a negative slope of 0.21. GATE is trained on a single RTX A5000. We use the same pre-trained checkpoints as the state-of- the-art approaches to ensure fairess. Specifically, we employ B and 3 The code is available at https::t//.. / thub. cow/ qinlangl 4/GATE.
Specifically, in Table 1, the NPH method utiliz- ing Post-Selcction achieves considerable results through entity replacement but still falls short compared to using GATE's Pre-selection. Table 2 demonstrates that Co-Selection methods such TMNet and KINET are constrained by the pre- trained models, whereas GATE be generator- agnostic that selects relevant knowledge without being reliant on the generation model.
4.5 Results and Observations
Overall Performance Table and Table show that GATE markedly outperforms all baselines in terms of accuracy of knowledge selection and qual- ity of the generated responses. To ensure the relia- bility the results, we conduct additional experi- ments using the original data from OpenDialKG as Figure 4, We make the following observations:
Performance w.r.t. ChatGPT+GATE To evaluate the auxiliary capability of our model, we input the knowledge selected by GATE into ChatGPT sup- plementary information. We provide conversation history and instruct ChatGPT to generate responses using the selected knowledge or relying solely on its internal knowledge. Considering the cost of human evaluation, we follow recent works (Wang et al., 2023; Luo et al., 2023) to assess responses by ChatGPT. Detailed prompts are in Appendix B. The
GATE outperforms previous SOTA methods in 'https: h htth///hgggggccccee/ee ctace..c/llleaaalaaaa
unifiedga- t5- base

---

(User) heard about a movie called The Fault in Our Stars. Who stars in it?
Dialogue Context
(Agent) It's a good one! That was written by John Green and stars Shailene Woodley. (User) Oh, didn't it know that. What else has John Green Wrote?
Ground-Iruth Response
He also wrole Paper Towns.
Gold Knowlcdge
Paper Towns is written by John Green.
K: (A Confederacy of Dunces; written_ wriuten_b;; Walker Perey)
DifKKG
R: Sure! Confederacy Dunces written by John Green. Do you like his work?
K: (Meg Crosbie; karred_actors: Paper Towns)
NPH
R: Paper Towns is another one he wrote. It was released in Meg Crosbie.
K: (Paper Towns; written_ by; John Green), (Paper Towns; release year, 2008)... ...
GATE
R: Paper Towns is another one he wrote. It was released in 2008.
Table 4: Case Study on OpenDialKG. "K"denotes selected knowledge and "e"deessss the generated response.
WoW
OpenDialKG
(1) Wizard Of Wilipedin
(b) OpenDialKG
Method
20
Test Seen
Test UnSeen
Test Seen
Test UnSeen
15
GATE
31.81
27.60
30.21
27.72
PpL
w/n Lw
30.42
26.19
29.26
26.90
10
w/o NN
31.23
27.09
26.68
24.53
Livondelgge
2.366
2.273
0.765
0.454
w/ Lwakk 0.894
0.491
0.573
0.717
10
20
30
40
500
10
20
30
40
50
w/ Loode
2.471
2.247
0,526
0.669
35
w/ Lnnosledge
31.62
26.42
27.88
24.77
30
Table The ablation results (R@) 1) for loss function.
25
20
edge have higher informativeness, resulting in a relatively smaller number of required knowledge pieces. In OpenDialKG, triplets have less informa- tion, requiring a larger knowledge pool to obtain satisfactory results. "GATE(Fixed)) demonstrates that certain amount of knowledge is necessary for better responses. "GATE" demonstrates that our method can select an appropriate number of knowl- edge pieces, resulting in high-quality responses. GATE offers improved flexibility and effectiveness compared to previous works that solely utilize a fixed and inadequate selection of top-k knowledge.
15
10
20
30
500
10
20
30
40
50
Knowledge Pool Size
Random
Holdd
GATE (Fixed)
Figure 5: GATE adapting knowledge pool size at the sam- ple level. "GATE(Fixed)" employs a fixed-size knowl- edge pool for comparison, "Random" randomly selects knowledge as lower bound, and "Hold" ensures the correct knowledge is in the pool as an upper bound.
results in Table 3 demonstrate that, although similar the findings in Shuster al. (2021) that utilizing redundant knowledge may impair engagement, the knowledge selected by GATE significantly enhances the information quality of responses. This improve- ment persists even when employing ChatGPT, al- ready known for its powerful performance and vast implicit knowledge. Moreover, the inconsistency in engaging may stem from different knowledge quan- tities: the WoW dataset with complete sentences and overly detailed knowledge, causing rigid re- sponses, and the OpenDialKG dataset with concise triplets, causing engaging responses.
4.6 Ablation Study
As outlined in Section 3.4, the loss function com- prises three components: walk, node, and knowl- edge. Table 5 presents the respective influence of each component on knowledge selection. Our model attains optimal performance by leveraging the synergistic effect of three components within the loss function. In particular, Knowwedeee serves as the universal optimization objective for this task, acting as the comnessonn for knowledge selection accuracy. LWalk corresponds to the loss in the re- inforcement learning of our model, while N represents the loss in node selection after unifying the knowledge types. The combined influence of these two specialized components empower our model to achieve significantly higher accuracy than using L alone.
Performance w.r.t. Adaptive Pool Size Figure 5 shows the response performance under different knowledge pool sizes and illustrates that GATE de- termines the appropriate size for dialogues. The range of knowledge pool sizes up to 50 due to the limitation of input tokens that the generation model can accept. In WoW, sentences as knowl-

---

4.7 Case Study
in part by the China Postdoctoral Science Foun- dation (No.202ITQ0222 and No.2021M700004) in part by the Natural Science Foundation of Sichuan (No.2023NSFSCO0433) in part by the Shandong Provincial Natural Science Foundation NN..22221MMFOS and in part by the Funda- mental Research Funds for the Central Universi- ties, Sichuan University (N0.2033SCU1208) and Nankai University (No.63231183)
As shown in Table 4, we conduct case study on a sample from OpenDialKG. The dialogue topic shifted from movies to books, requiring the agent provide recommendations of works by speci- fied author. DiffKG provides appealing response but incorrectly "concatenates" the triples in its Co- Selection process, as the truc author of "Confed- eracy of Dunces" is John Kennedy Toole. NPH utilizes Post-Selection based on the response from GATE, resulting in the disruption of sentence se- mantics, which is an inherent drawback of entity substitution methods. In contrast, GATE acquires relevant knowledge through rel-Selecionn and pro- duces an accurate and fluent response.
References
Jiaqi Bai, Ze Yang. Jian Yang, Hongcheng Guo, and Zhoujun Li. 2023. Kinet: Incorporating relevant facts into knowledge-groundel dialog generation. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 31:1213-1222.
5 Conclusion
Shaked Brody, Uri Alon, and Eran Yahav. 2022. How attentive are graph attention networks? In Interna- tional Conference on Learning Representations.
This paper offers a novel perspective to organize the literature on knowledge selection in knowledge- grounded dialogue systems, i.e., knowledge selec- tion coupled with, after, and before generation. This paper focuses on the third category and pro- poses GATE, a pnveerator-agnostie knowledge selec- tion method, which prepares knowledge for sub- sequent response generation models by selecting context-related knowledge among different knowl- edge structures and variable knowledge require- ments. The experimental results demonstrate the superiority of GATE on knowledge selection, and that GATE can facilitate the response generation model (including ChatGPT) generate more in- formative responses.
Xiuyi Chen, Fandong Meng. Peng Li, Feilong Chen, Shuang Xu, Xu, and Jie Zhou. 2020. Bridging the gap between prior and posterior knowledge se- lection for www.wldgeggrouddeee dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 3426-3437, Online. Association for Computa- tional Linguisties.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Limitation
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard conversational
Despite we have conducted experiments demon- strating the remarkable ability of GATE in improv- ing the performance of ChatGPT and producing more informative responses, there is still ample scope for further exploration regarding the contri- bution of our model LLMss. For example, We can try to combine GATE with more advanced prompts techniques developed recently (Wei et al., 2023; Tan et al., 2023; Lu et al., 2023) to facilitate more LLMs. We believe this will be an effective way to amplify the capability of GATE.
of Wikipedia: Knowledge-powent agents. In Proceedings of the International Confer- ence on Learning Representations (ICLR).
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Jocy Bose. 2021. Neural path hunter: Re- ducing hallucination in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 2197-2214, Online and Punta Cana. Do- minican Republic. Association for Computational Linguistics.
Mihail Eric, Nicole Chartier, Behnam Hedayatnia, Karthik Gopalakrishnan, Pankaj Rajan, Yang Liu, and Dilek Hakkani- Tur. 2021. Multi-sentence knowl- edge selection in open-domain dialogue. In Proceed- ings of the 14th International Conference Natural Language Generation, pages 76-86, Aberdeen, Scot- land, UK. Association for Computational Linguistics.
Acknowledgements
This work was supported in part by the Na- tional Natural Science Foundation of China (No.62206191, No.623061 156 and No.62106091);

---

Marjan Ghazvininejad, Chris Brockett, Ming-Wai Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounde neu- ral conversation model. Proceedings of the AAAI Conference on Artificial Intelligence, 32.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization.
Chuan Meng, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tengxiao Xi, and Maarten de Rijke. 2021. Initiative-aware self-supervised learning for shooledge-grounddee conversations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 522-532, New York, NY, USA. Association for Computing Machinery.
Maarten Grootendorst. 2020. Keybert: Minimal key- word extraction with bert.
Jachun Jung, Bokyung Son, and Sungwon Lyu. 2020. AttnIO: Knowledge Graph Exploration with In-and- Out Attention Flow for Knowledge-Grouun Grounded Dia- logue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3484-3497, Online. Association for Computational Linguistics.
Chuan Meng. Pengje Ren, Zhumin Chen, Weiwei Sun, Zhaochun Ren, Zhaopeng Tu, and Maarten de Ri- jke. 2020. Dukenet: A dual knowledge interaction network for knowledge -ereedgg-grounded conversation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '20, page 1151-1160, New York, NY, USA. Association for Computing Machin- ery.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising eqquenceto--squenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long Papers), pages 852-860, New Orleans, Louisiana. Association for Computational Linguistics.
Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, and Dilek Hakkani- Tur. 2022. Enhancing knowledge selection for grounded dialogues via doc- ument semantic graphs. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823, Seattle, United States. Association for Computational Lin- guistics.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. OpenDialKG: Explainable conver- sational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 845-88 345-854, Florence, Italy. Associa- tion for Computational Linguistics.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for re- sponse generation in dialog systems. Proceedings of the Twenty-Eigtth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5081-5088 ijcai.org.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Zhibin Liu, Zheng-Yo Yu Niu, Hua Wu, and Haifeng Wang. 2019. Knowledge aware conversation generation with explainable reasoning over augmented graphs. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing DEMNLP-IJCNIL pages 1782- 1792, Hong Kong, China. Association for Computa- tional Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- to-tttt trans- former. J. Mach. Learn. Res., 21(1).
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNN) pages 3982-390 3992, Hong Kong, China. Association for Com- putational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
1..000171..05101.
Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error amalysis prompting en- ables human-like translation evaluation large lan- guage models: A case study on chatgpt.
Rajdeep Sarkar, Mihael Arcan, and John McCrae, 2022. KG-CRuSE: Recurrent walks over knowledge graph

---

for explainable conversation reasoning using seman- tic embeddings. In Proceedings of the 4th Workshop on NL.P for Conversational AI, pages 98-107, Dublin, Ireland. Association for Computational Linguistics.
Processing (EMNLP), pages 3377-3390, Online. As- sociation for Computational Linguistics.
Hao 7hou, Minlie Huang, Yong Liu, Wei Chen, and Xiaoyan Zhu. 2021. EARL: Informative wewwledge-grounder conversation generation with entity-agmosie representation learning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392 2395. Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Leslie N. Smith and Nicholay Topin. 2019. Super- convergence: very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 1006, page 1100612. Interna- tional Society for Optics and Photonics, SPIE.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of chatgpt as question answering system for answering complex questions.
Yi-Lin Tuan, Sajjad Beygi, Maryam Pazel-Zarande Qiaozi Gao, Alessandra Cervone, and William Yang Wang. 2022. Towards large-scale interpretable knowledge graph reasoning for dialogue systems. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 383-395, Dublin, Ireland. Association for Computational Linguisties.
Jiaan Wang. Yunlong Liang, Fandong Meng. Zengkui Sun, Haoxiang Shi, Zhixu Li. Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt good nlg evaluator? a preliminary study.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thouglh prompting elic- its reasoning in large language models.
Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Mach. Learn.. 8(3 (3:344)2292225 256.
Qiang Xue, Tetsuya Takiguchi, and Yasuo Ariki. 2022. Building knowledge- based dialogue system with text infilling. In Proceedings the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. pages 237-243, Edinburgh, UK. As- sociation for Computational Linguistics.
Chenxu Yang, Zheng Lin, Jiangnan Li, Fandong Meng. Weiping Wang. Lanrui Wang, and Jie Zhou. 2022. TAKE: Topic-shift aware knowledge sElection for dialogue generation. In Proceedings of the 29th Inter- national Conference Computational Linguistics, pages 253-265, Gyeongju, Republic Korea. Inter- national Committee on Computational Linguistics.
Xueliang Zhao, Wei Wu, Can Xu. Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge- grounded dialogue generation with pre-trained lan- guage models. In Proceedings of 2020 Confer- ence on Empirical Methods in Natural Language

---

A Dataset Statisties
In the ground-truth data, the Wizard responds with "wanting to help animals" based on the description of the veterinary role in selected knowledge. contrast, MIKE's generated response incorrectly assumes that the Wizard is already a veterinarian, disregarding the dialogue context. KnowledGPT's response lacks confidence and fails to provide ad- equate reasoning. While KINET offers reasons for pursuing career veterinarian, its empha- sis on the "clinical environment" does not align with the preceding context and relevant knowledge. Conversely, GATE provides a comprehensive and contextually appropriate answer regarding "becom- ing veterinarian," demonstrating its capability to effectively select suitable knowledge for generating coherent and engaging responses.
As shown in Table 6, WoW has 22,311 conver- sations and test set split into Seen and Unseen based on topics. Consistent with previous research, we employ the knowledge retrieved for the last turns the knowledge pool. OpenDialKG has 15,673 conversations and knowledge graph that contains 100K entities and 1.1M facts. The reasoning path of each turn is annotated, enabling participants to utilize graph information during the conversation. We maintain the train/valid/test sets in 70%/15%/15% ratio, consistent with previous works.
Prompts for ChatGPT
Tables 7 and Tables 8 contain the prompts for gener- ation and evaluation derive the results presented in Table 3.
C Supplementary Ablation Study
Table presents the extended results of Table measuring the performance on R@ 1/5/10. The three components have demonstrated effectiveness in improving the overall selection performance rather than solely focusing on enhancing R @@1 1.
As demonstrated in Table 10, proposed node attention and the corresponding dot product caleu- lation effectively enhance R @ 1/5. This suggests that the node representations obtained through the aggregation of knowledge embeddings possess ro- bust expressive capabilities. The scoring mecha- nism during the agent's 's traversal process and the corresponding selection strategy are effective, par- ticularly evident in the case of WoW Test Unseen, which suggests that the guiding role of node scores in knowledge selection remains effective even for zero-shot topics. The decrease in R 10 perfor- mance could be attributed to the amplification of knowledge score variance by node attention. More- over, the enhancement in knowledge selection ac- curacy contributes to improving the quality of re- sponse generation.
Case Study for WoW
To ensure the credibility of the comparisons, conducted case study using samples previously discussed in the relevant literature (Bai et al., 2023). As shown in Table 4, the dialogue topic "Veteri- nary physician" and the Wizard is asked the ques- tion, "What makes you want to be a veterinarian??"

---

WoW
OpenDialKG
Train
Valid-S
Valid-U
Test-S
Test-U
Tocal
Thain
Valid-S
Vali-L""
Test-S
Test-U
Total
Conversations
18,430
981
967
965
968
22,311
10,969
1,176
1,176
1,176
1,176
15,673
Utterances
166,787
8,921
8,794
8,715
8,782
201,999
63,832
6,845
6,839
6,842
6,851
91,209
Avg. Knowledge
75.48
75.27
76.90
75.24
74.59
600.0
590.2
600.7
603.5
610.4
Table 6: The statistics of WoW and OpenDialKG datasets.
Prompt- Response Generation
Assuming there seeker of knowledge who engages conversation (named "apprentice" "user") with wise person who has access to knowledge (named "wizard"" "assistant"), will provide the history of their conversation and the available reference knowledge as follows:
History of conversation:
History]
Reference knowledge:
Knowledge
["K") "K""]CChtGPT'+ + GATE):
As the wizard/assistant, please continue the dialogue with the apprentice/user, keeping in mind the history of their conversation
and the available reference knowledge.
("K""(Cha'PPT" Only):
["K"] Provide a response less than 20 words.
and leveraging your knowledge.
Table Prompts used for response generation. Different ["K") represent different response settings, as mentioned in Section 4.5.
Prompt Evaluation
Please evaluate the engagemment/ifformativvnss level of the following responses and rank them.
History of conversation:
History ]
Response A:
Response
Response B:
[
Reesponse
Please provide detailed explanation of your thought process, outlining each step you take arrive at your conclusion, before providing your answer to the question.
Table 8:
Prompts used for evaluation of engagement informativeness.
OpenDialKG
Test Secu
Test UnSeen
Test Seen
UnSeen
R@1 1
R@5
R 10
R@B
R 10
R@
R@9
R@0
R@1
R@5
R:00 10
GATE
31.81
60.02
69.05
27.60
57.84
72.25
30.21
54.96
62.33
27.72
56.85
61.01
w/o Laatk
30.42
56.65
67.32
26.19
58.19
70.79
29.26
56.16
62.80
26.90
53.27
59.51
w/o CNode
31.23
57.05
67.11
27.09
59.45
71.49
26.68
49.59
53.54
24.53
46.66
51.43
w/o Kinwobhdge
2.366
7.624
12.36
2.273
7.231
12.14
0.765
3.201
6.163
0.454
2.508
4.873
w/ LWakk
0.894
5.573
13.91
0.491
5.062
15.11
0.573
3.798
6.976
0.717
3.153
6,307
CNode
7.282
12.64
2.247
7.180
12.78
0.526
3.345
6.761
0.669
2.437
5.471
31.62
55.76
64.64
26.42
57.64
68.80
27.88
51.43
55.92
24.77
48.42
$2.87
Table 9:
The ablation results for the loss function.
OpenDialKG
UnSeen Knowledge Selection
Test
R
R@0
R
R
R
R 10
R
R
R60 10
R
R
R
GATE
31.81
60.02
69.05
27.60
$7.84
72.24
30.21
$4.96
62.33
27.72
56.85
61.01
Node Attention
29.86
$7.81
71.45
18.62
50.00
68.88
30.03
52.99
65.03
25.63
$7.29
61.40
Response Generation
Ronge
Meteor
Rouge
Meteor
F1
Rouge
Meteor
F1
Rouge
Meteur
GATE
24.20
23.12
26.23
23.81
22.78
25.75
28.57
28.46
32.21
27.93
27.78
31.57
Node Attention
21.11
24.29
Table 10:
The ablation results for node attention.

---

Wizard:
want to be a veterinary physician when I grow up.
History
That's really ambitious. started school to be vet, but I ended up being teacher.
Apprentice:
What makes you want to be a veterinarian?
Ground-truth
Wizard:
want to treat animal illness and perform surgery help them.
Response
many cases, the activities that may be undertaken by veterinarian (such as treatment of illness surgery in animals) are restricted only to those professionals who are registered as veterinarian.
Ground-truth
"Veterinary
Knowledge
physician":
am a veterinarian.
MIKE:
know that most veterinary physicians work in clinical settings, treating animals dircetly. I'm not sure how I'd like to be a veterinarian. I think would like to treat animals directly. love animals and want work in clinical settings treating them directly.
KnowleGPT:
KINET:
GATE
want to treat diseases, disorders and injuries in animals.
Table 11: Case Study on WoW dataset, baselines' responses are reported from Bai al. (2023).


Reviews:
[{"review_id": "uz3IiuMzUW", "reviewer": "Reviewer_CwT6", "summary": "", "questions": "In Table 4, the keyword 2008 is not in the knowledge sentence which would fall under the category of hallucination. How does your method compare against others in terms of this factor.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We will address your concerns point by point. \n\n**Q1:** *\"... there has been no work on selecting knowledge before response generation is not correct ...\"* \n\n**A1:**\nThere are indeed pieces of work that select for knowledge before response generation.\nWe have not overlooked this type of work and have mentioned it several times in our paper (cf. Lines 168-184 and 417-423).\nWe thank the reviewer for the two relevant works and we will include them in our revised paper.\n\nIn addition, we need to restate the core contributions of our work, i.e., **the introduction of a novel perspective to organize the literature of knowledge selection in knowledge-grounded dialogue**, instead of proposing a pre-selection method for the first time (cf. Lines 003-006, 037-039, and 107-110).\nWe hope that this perspective can offer valuable insights to this community.\n\n **Q2:** *\"There are a few items in the paper that need to be clarified\"* \n\n**A2:**\nWe provide clarifications for them as follows:\n\na. *\"How are the keywords extracted from the conversation history.\"*\n\nWe leverage the KeyBert toolkit to extract up to five keywords from the conversation history by calculating the cosine similarity between document-level representation and word embeddings.\n\nb. *\"The reward signals $R_{Node}$ and $R_{Gold}$ ... the difference ...\"*\n\n$R_{Node}$ refers to the reward for the Agent when halting at the correct process node,\nwhile $R_{Gold}$ indicates the reward for GATE when eventually choosing the gold knowledge (i.e., knowledge node).\n\nc. *\"... variable number of knowledge sentences but Wizard of Wikipedia only has one gold knowledge sentence so how did you train it.\"*\n\nIn accordance with the objective of knowledge pre-selection, GATE is trained to create suitably sized knowledge pools rather than learning to select single Gold knowledge.\n    Given the assumption: \"If the model selects the correct knowledge with a high probability, excessive knowledge can be deemed redundant. Otherwise, expanding the knowledge pool is necessary to enhance knowledge effectiveness'',\n    the pool size is determined by the variance of knowledge-scoring results, indicated by Equation (2): larger variance results in a smaller knowledge pool, and vice versa (cf. Line 279).\n\nd. *\"... the evaluation process on line 481 ...\"*\n\nWe provide the conversation history to ChatGPT, present different responses (ChatGPT/ChatGPT+GATE), and randomly order them in the prompt. ChatGPT compares responses based on engagement and informativeness, offering justifications and a comparative analysis.\nDue to space constraints, the complete prompts are included in Appendix B.\n\n **Q3:** *\"... discuss why is pre-selection better than post selection ...\"* \n\n**A3:**\nThank you for pointing out this worthwhile discussion.\n\nPost-selection is dedicated to correcting potential knowledge errors in the generated response, and pre-selection performs knowledge selection before generation to potentially reduce the learning, adjustment, and interpretation burden of subsequent response generation models.\nThese two categories of knowledge selection (pre- and post-) share the common pursuit of improving the quality of generated response, but they are executed in different ways at different stages of the process.\nTherefore, the two categories are not mutually exclusive and can be executed as a pipeline during the response generation process.\n\n **Q4:** *\"... the response ... should be more accurate at selecting the knowledge\"* \n\n**A4:**\n\nIn the post-selection category, the generated response can help in selecting knowledge to some extent.\nHowever, since this category is in an entity-replacement manner, there may be a mismatch between the selected knowledge and the generated response, leading to semantic and syntactic confusion.\n\nFor example: an adjusted response in Table 4 with confusion --\"It was released in __Meg Crosbie__\",  where the bold entity \"Meg Crosbie\" is a mistake adjustment because a personal name does not make sense in the context of \"released in\".\n\n**Q5:** *\"... Table 4, the keyword 2008 is ... hallucination. How does your method compare against others in terms of this factor.\"* \n\n   **A5:**\n   Thanks for your careful review.\n   We regret that the knowledge details selected by GATE and the two baselines are omitted in Table 4.\n   Please check the completed Table 4 below. K denotes selected knowledge, and R denotes the generated response.\n   We need to clarify that the \"2008\" keyword appearing in the response is not a hallucination, as it was selected from the knowledge base and does not contradict existing knowledge or reality.\n   Hallucination is a highly worthwhile topic but falls under the study of the GATE downstream task (i.e., response generation).\n\n| Dialogue Context       | (User) I heard about a movie called The Fault in Our Stars. Who stars in it?   |\n|------------------------|----------------------------------------------------------|\n|                        | (Agent) It's a good one! That was written by John Green and stars Shailene Woodley. |\n|                        | (User) Oh, I didn't know that. What else has John Green Written?                     |\n| **Ground-truth Response**  | He also wrote Paper Towns.                                 |\n| **Gold Knowledge**         | Paper Towns is written by John Green.                      |\n| **DiffKG**       | **K:** (A Confederacy of Dunces; written_by; Walker Percy)   |\n|                        | **R:** Sure! A Confederacy of Dunces is written by John Green. Do you like his work?|\n| **NPH**      | **K:** (Meg Crosbie; ~starred_actors; Paper Towns)   |\n|                        | **R:** Paper Towns is another one he wrote. It was released in Meg Crosbie.|\n| **GATE**       | **K:** (Paper Towns; written_by; John Green), (Paper Towns; release_year; 2008), ...   |\n|                        | **R:** Paper Towns is another one he wrote. It was released in 2008.|\n\n **Q6:** *\"``Paper towns'' ... should be highlighted green\"* \n\n   **A6:**\n   We will modify this typo in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The knowledge selection method that is supposed to handle both structured / unstructured knowledge is a good proposal and it seems results are sufficient to back that up.\n\n(2) Nice to see results using ChatGPT and that knowledge selection can further improve it's performance and that it is still critical even for non-current events.", "reasons_to_reject": "(1) The claim that there has been no work on selecting knowledge before response generation is not correct. Some relevant works are.\n\nhttps://arxiv.org/pdf/2203.00763.pdf\nhttps://arxiv.org/pdf/2104.07567.pdf\n\nWith that being said I do think that the method is able to retrieve knowledge in both a structured/unstructured setting is good.\n\n(2) There are a few items in the paper that need to be clarified such as: \n\nHow are the keywords extracted from the conversation history?\nThe reward signals R_node and R_gold seem the same and it is not clear what the difference is between them.\nYour method can choose variable number of knowledge sentences but Wizard of Wikipedia only has one gold knowledge sentence so how did you train it?\nExplain the evaluation process on line 481 in more detail.\n\n(3) It would be good to discuss why is pre-selection better than post selection. In post selection you already have the response so ideally it should be more accurate at selecting the knowledge.", "questions_to_authors": "In Table 4, the keyword 2008 is not in the knowledge sentence which would fall under the category of hallucination. How does your method compare against others in terms of this factor.", "paper_topic_and_main_contributions": "The paper focuses on the topic of knowledge selection. The authors compare various types of knowledge selection methods (pre-response generation, during response generation, post response generation). They find selecting knowledge before response generation performs the best.\n\nThe main contribution as I see it is developing a knowledge selection method that can access different types of knowledge bases (structured / unstructured).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Table 4 \"Paper towns\" is highlighted red for the GATE response indicating it is incorrect but it should be highlighted green I believe."}, "extra_text": {}}, {"review_id": "Md1OhrbYzt", "reviewer": "Reviewer_7Yzk", "summary": "", "questions": "1. Can you provide more implementation details, such as the hyperparameters and training process, to facilitate reproducibility?\n\n2. Why do the two datasets in Table 3 have inconsistent Engaging performance with the addition of GATE?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We will address your concerns point by point.\n\n **Q1:** *\"... lacks implementation details ...\"* [Reasons to reject 1]\n\n   **A1:**\n   The activation function used in the model is LeakyReLU with a negative slope of 0.21. The dropout layer probability applied across the model is uniformly set at 0.25.\n   GATE is trained on a single RTX A5000, employing a learning rate 1e-4.\n   We will add the above supplementary details in the revised paper.\n\n **Q2:** *\"... would benefit from a more detailed discussion of the limitations and future directions of the proposed\nmethod\"* [Reasons to reject 2]\n\n   **A2:**\n   There is room for improvement in the reward setting of our proposed GATE.\n   For example, considering the agent's walk through the graph, the variation of distance between the agent and the correct process node/knowledge node can be incorporated into the existing reward setting.\n   In addition, we will further explore the role of GATE as an aid to more LLM-based models.\n   For example, constructing scenario-specialized prompts and coupled with the CoT technique can guide LLMs more meticulously in utilizing the highly relevant explicit knowledge provided by GATE to generate higher-quality responses.\n\n **Q3:** *\"but there is actually a lot of work focused on knowledge selection\"* [Reasons to reject 3]\n\n   **A3:**\n   We have not overlooked the category of knowledge pre-selection and have mentioned it several times in our paper (cf. Lines 168-184 and 417-423).\n   We thank the reviewer for the two relevant works and we will include them in our revised paper.\n   It needs to be clarified that the SKT model in [1] incorporates posterior information from responses during training, thus belongs to the \"knowledge selection\" category rather than the \"pre-selection\" category.\n\n **Q4:** *\"... provide more implementation details ...\"* [Questions for the authors 1]\n\n   **A4:**\n   Please refer to A1 for more information.\n\n **Q5:** *\"inconsistent Engaging performance\"* [Questions for the authors 2]\n\n   **A5:**\n   This inconsistency stems from varying knowledge quantities in datasets. The WoW dataset contains complete sentences, and 5/10 knowledge pieces are overly detailed, causing rigid responses and reduced engagement. Conversely, the OpenDialKG dataset has concise triplets. Appropriate information from 5/10 knowledge pieces leads to captivating responses and heightened engagement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. GATE is a generator-agnostic method that can select context-related knowledge and improve the quality of response generation.\n\n2. The experimental results demonstrate the superiority of GATE in knowledge selection and its ability to facilitate response generation models.", "reasons_to_reject": "1. The paper lacks implementation details, making it difficult to reproduce the study.\n\n2. The paper would benefit from a more detailed discussion of the limitations and future directions of the proposed method.\n\n3. The author emphasizes several times that the pre-selection has been largely ignored by previous work, but there is actually a lot of work focused on knowledge selection:\n\n[1] Kim, Byeongchang ,  J. Ahn , and  G. Kim . \"Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue.\" arXiv (2020).\n\n[2] Yang, Chenxu, et al. \"TAKE: topic-shift aware knowledge selection for dialogue generation.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.", "questions_to_authors": "1. Can you provide more implementation details, such as the hyperparameters and training process, to facilitate reproducibility?\n\n2. Why do the two datasets in Table 3 have inconsistent Engaging performance with the addition of GATE?", "paper_topic_and_main_contributions": "The paper proposes a generator-agnostic knowledge pre-selection method called GATE for knowledge-grounded dialogue systems. The paper focuses on the  knowledge selection before generation and demonstrates the superiority of GATE in selecting context-related knowledge and facilitating response generation models to generate more informative responses. The experimental results on two datasets show that GATE outperforms other baselines in terms of knowledge selection accuracy and response quality.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dpLKgtwLd8", "reviewer": "Reviewer_AqSZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments.\n\nAccording to your suggestion *\"experiment should be more focused on LLM-based methods\"*, we believe that the CoT technology can be used to create prompts tailored to this scenario, enabling precise guidance for LLMs to utilize pre-selected knowledge for higher-quality responses. This enhances the effectiveness of the Knowledge Pre-Selection approach as an aid to LLMs. Furthermore, exploring the assessment of engagement and informativeness by leveraging LLMs is also a valuable avenue for future research. Your suggestions have greatly inspired our forthcoming research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed knowledge pre-selection method is an essential topic especially in the context of LLM. As language model can hardly encode all knowledge accurately in its storage, using a knowledge base to augment its generation is a progressing direction.\n2. The taxonomy of different manners to perform knowledge augmentation is inspirational, and the experiment design and analysis is in-detail and well structured.\n3. Both unification of different knowledge structures and the adaptation of knowledge pool size are interesting.", "reasons_to_reject": "1. As LLM is taking the dialogue generation area by storm, the experiment should be more focused on LLM-based methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a generator-agnostic knowledge pre-selection method for knowledge-grounded dialogue. Their method firstly unify different knowledge structures, and then knowledge is selected and adapted to appropriate size based on GAT and reinforcement learning. The authors apply their method on both pre-trained models and large language models for knowledge-grounded dialogues and make significant improvement.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A pipeline for the topic of knowledge selection. However, some reviewers think the author still has to address some of the points in the final version.
Decision: Accept-Main