Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue
Authors: Yizhe Yang
Heyan Huang
Yuhang Liu
Yang Gao
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge-Grounded Dialogue
Empirical Study

Abstract: Knowledge-grounded dialogue is a task of gener- ating an informative response based on both the dialogue history and external knowledge source. In general, there are two forms of knowledge: manu- ally annotated knowledge graphs and knowledge text from website. From various evaluation viewpoints, each type of knowledge has advantages and downsides. To further distinguish the principles and determinants from the intricate factors, we conduct a thorough experiment and study on the task to answer three essential questions. The ques- tions involve the choice of appropriate knowledge form, the degree of mutual effects between knowl- edge and the model selection, and the few-shot performance of knowledge. Supported by statistical shreds of evidence, we offer conclusive solutions and sensible suggestions for directions and standards of future research.

Consistent human knowledge is compulsory to achieve an intelligent AI, except for massive amount of data provided, whether a well-formed structural knowledge (Moon et al., 2019) or hu- man instruction (Ouyang et al., 2022). However, many dialogue systems are not proficient in inte- grating knowledge and cannot engage humans in thorough discussion about particular subjects. In or- der incorporate knowledge into the conversation, knowledge-groundee dialogue systems have gained popularity (Li et al., 2018; Dinan et al., 2019; Moon et al., 2019; Zhou et al., 2018). The goal is to gen- erate informative and meaningful responses based on dialogue history and external knowledge. So far, researchers have gathered datasets with various knowledge forms, such the recommendation dia- logues with knowledge graph (Moon et al., 2019; Graph vs. Sequence: An Empirical Study on Knowledge Forms for Groouddde-Grounde Dialogue
Yizhe Yang, Heyan Huang, Yuhang Liu, Yang Gao"
School of Computer Science and Technology, Beijing Institute of Technology
Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications
Beijing Institute of Technology Southeast Academy of Information Technology (yizheyang, hhy63, codelyh, goyeelyyggaang@bbb.. tbieeucccit edu.cn
Abstract
Li al., 2018) and the open-domain dialogues with Wikipedia (Dinan et al., 2019; Zhou al., 2018).
Generally, there are two forms of knowledge: knowledge graphs and knowledge text. Knowledge graphs are annotated by human experts and saved as triples like <head entiry, relationship, tail en- tity>. Because the knowledge is structured, the system can retrieve and use it to facilitate reason- ing (Moon et al., 2019). However, high-quality knowledge graph is quite labor expensive. Instead, the text naturally serves as medium of knowl- edge. knowledge text derived from knowledge- intensive texts and casily parameterized by large language models. Textual knowledge is unstruc- tured, easy to collect, yet challenging to reason and explain directly.
wwwwllldg-groundee dialogue is task of gener- ating informative response based both the dialogue history and external knowledge source. In general, there are two forms of knowledge: manu- ally annotated knowledge graphs and knowledge text from website. From various evaluation view- points, each type of knowledge has advantages and downsides. To further distinguish the principles and determinants from the intricate factors, we conduct thorough experiment and study the task answer three essential questions. The ques- tions involve the choice of appropriate knowledge form, the degree of mutual effects between knowl- edge and the model selection, and the few-shot performance of knowledge. Supported by statisti- cal shreds of evidence, we offer conclusive solu- tions and sensible suggestions for directions and standards of future research.
Different forms of knowledge tend to serve different knowledge-intensiv. language tasks (KILT) (Petroni et al., 2021; Yu et al., 2022; Mi- alon et al., 2023; Chowdhury al., 2023; Pan et al., 2023). Specifically, knowledge groounddeg dialogue systems can receive either forms of knowl- edge graph or knowledge text to facilitate response generation; nevertheless, their performances are quite different regarding response quality and fac- tual consistency, etc. The reason is closely tied to knowledge's characteristics. For instance, Open- DialKG (Moon et al., 2019) is typical knowl- edge graph dataset, which annotates the reference triples based on large knowledge graph for rec- ommended conversations. However, the knowl- edge in OienDia comes from Freebase (Bast et al., 2014). 4), which requires large-scale data an- notation, and the knowledge is sparse (1-2 triples) for generation. On the contrary, WoW dataset (Di- nan al., 2019) retrieves Wikipedia passages for open-domain dialogue assist intelligent agents in generating informative utterances. The information retrieved in WoW too dense (i.e., passages) and noisy for models to figure out the accurate knowledge. Thus, an extra selection module is re-
1 Introduction
Corresponding author

---

quired to polish the pertinent knowledge. (Zhao al., 2020b). Therefore, determining the forms of knowledge is crucial and merits extensive discus- sion in light of the task's s features.
Extensive experimental results and analysis demonstrate that: (1) Different forms of knowl- edge have their advantages in different situations of KILT. Specifically, the knowledge graph out- performs generation quality and exhibits stronger generalizability, while the knowledge text outper- forms factual consistency in generations. Perfor- mance can be effectively improved by denoising the knowledge, for example, by selecting the suc- cinct sequence or extracting structured knowl- edge graph. (2) Performance could be universally improved further by advanced Dual-Encoders struc- ture or by employing domain adaption pre- tre-training. However, the impact of model size is highly re- lated to the knowledge's own characteristics. Fu- ture work should choose the model size according to the situation and explore the broader impact of knowledge's s characteristics. (3) The number of samples affects the model selection. When the samples are extremely small (100-200 our stud- ies), it is preferable for the input form and model architecture to resemble the pre-trained model as much as possible. However, the model architecture selection criteria tend to favour the task itself as the amount of data increases (500+), there is suffi- cient data for the model to do task-level adaptive learning.
With a particular form of knowledge in hand, we have to choose appropriate approach to in- corporate the knowledge to support relevant tasks. Though knowledge-groune studied, the majority aiming for advanced per- formance, few of them have exhaustively com- pared the determinants in the model and knowl- edge. Prabhumoye et al. (2021) proposed large- size Dual-Encoders model on WoW. However, ac- cording to our experiments, the base-size Dual- Encoders model may perform better WoW data. Li et al. (2022) experimented with Encoder- Decoder model on multiple datasets but ignored the Dual-Encoders are more compatible with the task. Dziri et al. (2022a) experimented on sentence- level knowledge datasets, neglecting the effect of knowledge size and granularity. Nevertheless, com- parisons from multiple perspectives are needed to support the superiority and contribution of the model theoretically. However, considering all fac- tors every time in each specific research is not fea- sible. Therefore, to aid future research, we com- prehensively analyze the influencing factors, such as model architecture, model size, pretrained mod- els, etc., and then summarize several determinant conclusions. Concretely, we leverage the scientific strategy of controlling variables to investigate how intricate factors affect and to what extent the model and knowledge should be mutually adaptive.
tasks are extensively
The contribution of this paper is that it is, to the best of our knowledge, the first work to thoroughly investigate different forms of knowledge, knowl- edge graph and knowledge text, and their respec- tive advantages and disadvantages in knowledge- grounded dialogues. Furthermore, we comprehen- sively analyze implications from model selection, knowledge representation, and how to adapt the ca- pacity, highlighting feasible solutions and shedding light on future research and development.
Furthermore, we also explore how various knowledge performs in few-shot settings as few- shot learning with the knowledge-auxiliary is an important application (Chen et al., 2021). To an- swer the above question comprehensively, we em- ploy unified Transformer (Vaswani et al., 2017) framework to observe the effects of different knowl- edge and serialize the knowledge graph to adapt to the input. In this paper. our investigation mainly focuses on three points:
2 Literature Review
Knowledge-grounet dialogue is the task of gen- erating informative response R based both dialogue history and external knowledge K. It becoming an increasingly important topic in our community. The involved knowledge can be in dif- ferent forms, such as documents for open-domain conversation (Dinan et al.. 2019), movie database for movie recommendation conversation (Li et al., 2018), and knowledge graph for recommendation conversation (Liu et al., 2021b).
1. Graph v.s. Sequence, which form of knowl- edge is better?
2. To what extent the model and knowledge should be mutually adaptive?
3. How does various knowledge perform in few- shot settings?
Recent research investigates several techniques for improving knowledge representation to fuse

---

knowledge in response generation. Ghazvinine- jad al. (2018) encoded the dialogue history and knowledge documents separately to infuse the re- sponse with external world facts. Wang et al. (2022) added a particular knowledge graph rep- resentation in the response generation module.
3 Experimental Settings
In order to investigate the effect of various forms of knowledge, we comprehensively analyze the per- formance of different knowledge. Further, we also consider the factors of models identify the in- terrelation between models and knowledge. This section introduces the detailed experimental set- tings for a clearer illustration'.
Another line of work investigates pre-training methods for the knowledge grrungdeg dialogue to improve the system's performance on unseen top- ies and train in low-resource setting. Zhao et al. (2020a) pre-trained the dialogue generation mod- ule and the knowledge encoder with ungrounded dialogues and the Wikipedia dump separately. Li al. (2020) devised variational network that can effectively estimate a generation model from di- alogue corpus and an independent knowledge cor- pus. Liu et al. (2021a) proposed a three-stage learn- ing framework to separately pre-train the dialogue module and knowledge module.
3.1 Datasets
Our experiments were carried out on three pub- lic datasets, WoW (Dinan al., 2019), Faith- Dial (Dziri et al., 2022a) and OpenDialKG (Moon et al.. 2019). Table shows the data statistics.
WoW The dataset consists of human-human con- versations collected over Amazon Mechanical Turk and is grounded in Wikipedia passage (Dinan al., 2019). These conversations are grounded in a di- verse range of topics which are further split into seen and unseen topics during training and vali- dation. The dialogues between two humans, re- ferred to as "Wizard" and "Apprentice", engage in a conversation about given knowledge. The dialogue is designed to be coherent and informa- tive, with the Wizard providing information about the topic and the Apprentice asking questions and seeking clarification. The original knowledge WoW is passages retrieved from Wikipedia, and we extracted triples by Open Information Extraction (OpenlE) annotator as a knowledge graph. Since consider constructing knowledge-groundt agent, only the utterances from Wizard are used.
Inspired by the accomplishments of pre-trained language models (PLM) for range of natural lan- guage processing (NLP) tasks, researchers inves- tigate learning knowledge through PLM' param- eters. Zhao et al. (2020b) equipped a pre-trained language model with knowledge selection mod- ule and presented an unsupervised jointly optimiz- ing technique. Prabhumoye et al. (2021) focused on the attention mechanism in Transformers and proposed a strong baseline by a pre- pre-trained model. Rashkin et al. (2021) increased the faithfulness in knowledge- grounded dialogue with controllable features of pre-trained models.
However, the above studies only considered a single form of knowledge, neglecting the impact of knowledge forms. Chen et al. (2020) proposed a pre-trained model for data-to-text tasks by unifying the knowledge format in the pre-trat pre-training data and downstream tasks. Li al. (2022) used unified essential knowledge triples instead of documents pre-training, making their model more general- izable. Motivated by these methods, we conduct a thorough investigation of knowledge-grounded dialogue in various knowledge forms to explore the impact of knowledge. We focus on how to choose a knowledge form (sequence or graph) for NLG and investigate questions about which knowledge form is better and how it affects performance, Our study will facilitate the subsequent data collection and model construction and even influence the direction of knowledge-intene intensive language tasks.
Dziri et al. (2022b) analyzed the hal- lucination phenomenon and found that more than 60% of the response are hallucinated in the three datasets (including WoW). To mitigate this be- havior, Dziri et al. (2022a) create FaithDial for ballucination-free dialogues by editing hallucinated responses in the Wow. As the FaithDial is an edited version of WoW, the settings of FaithDial are the same as WoW. The original knowledge in FaithDial is one sentence from Wikipedia, which is shorter and more accurate, and we also construct knowl- edge graph by OpenlE in WoW.
FaithDial
Op dataset contains con- versations between two crowdsourcing agents en- gaging in dialogue about given topic. Open-
OpenDialKG
'we will release our code data the future. Phttps: / wttp///.....s.o.d..//////// stanford, tffu/ootwrrr twortw/ppnpeieeeeee html

---

Dataset
Form
Avg. Turns
Train Uttrs
Dev Uttrs
Test Uttrs
Avg. Tokens
Avg. Nodes
WoW
Sequence
9.0
30.5K
3.5K
3.5K
141.63
48.55
DaithDial Sequence
9.0
11.2K
2.2K
2.3K
26.36
17.92
OpenDialKG
Graph
5.8
14.8K
3.2K
3.2K
12.52
3.56
Table I: Data statistics of Wow, FaithDial and OpenDialKG. Form denotes the original forms of knowledge in datasets. Avg. Turns indicates the average number of turns involved in the dialogue. Avg. Tokens and Avg. Nodes indicate the size different types of knowledge. The Uttes denotes number total used utterances datasets.
D annotates knowledge graph path label for each dialogue and triple label for each dia- logue turn. The response is grounded on the labeled triples during data collection. The original knowl- edge in OpenDialKG one or some triples in the knowledge graph, and the sequence reasoning path is used as the source text. For example, the origi- nal triple is < Ryan Reynolds", starred_actons "Turbo": >> and sequence reasoning path is "Ryan Reynolds starred in Turbo".
Dual-Encoders Dual-Encoders is remarkable architecture for knowledge-grounded dialogue gen- eration which encodes knowledge and dialogue history by two encoders and fuse them with at- tention mechanism in decoder (Prabhumoye et al., 2021; Liu et al., 2021a; Yang et al., 2022). In- spired by these works, we adopt Dual-Encoders architecture in our experiment. The knowledge en- coder encodes the knowledge K,, and the dialogue encoder encodes dialogue history C. Each layer of the decoder contains a self-attention block, context cross-attention attending to dialogue con- text C' and a knowledge cross-attention attending knowledge K., i.e., p p(R,)K, C, R.<i).
3.2 Architectures
We utilize three typical architectures in NLG for thorough comparison.
There maybe many other variants to improve these basic architectures. However, the most sim- plest configuration demonstrate the most funda- mental abilities of the model. This kind of analysis provides the most important reference for other researchers. Adding too many components may distract from the main focus of the paper and make difficult for readers to understand the key contri- butions and findings. Besides, as the parameters different architecture is much different, we only fo- cus on the impact from knowledge form and avoid influences from the number of model parameters as much possible.
Decoder-Only Decoder-Only architecture is typical language model characterized by single autoregressive decoder. The decoder relies on pre- vious tokens within a sequence as input to predict the subsequent token, following the conditional probability distribution (p(z)(=<1)).
generates text from left to right and is, therefore, able to gen- erate human-like text by predicting the most likely next token at each time step. we employ GPT-2 the Decoder-Onie model. We concatenate the knowledge, the dialogue history, and the response the input, i.e., p(Re) [K; where [:] de- notes the concatenation of strings.
3.3 Serialize Knowledge Graph
The above architectures based the Trans- formers (Vaswani et al., 2017), which has become the dominant model in the field of NLP. Such mod- els can effectively handle discrete sequences of tokens rather than graphs. Hence, we serialize the knowledge graph as input to the model. Specially, we add three special tokens, "[triple]", "[entityy" and "[relation]" J"", to identify the triple, entity, and relation, respectively. For example, "<guy ritchie writlen by, snatch>, <snatch, starred actors, ewen bremner>" a knowledge graph with two triples. will be serialized as "[triple] [entity] guy ritchie [relation] written by [entity] snatch [triple] [en- tiry] snatch [relation] starred actors [entity] ewen
Encoder-Decoder Encoder-Decoue is another representative paradigm for NLG (Lewis et al., 2019; Raffel et al., 2020). Unlike Decoder-Only models, the els sequence-to-sequeecc generation by combining a bidirectional encoder and an autoregressive de- coder. In our experiments, to individually model the different semantic spaces of knowledge and dialogue, we only pass the knowledge IC to the encoder. The dialogue history C is considered as the prompt of the decoder, i.e., |K,, [C; R.) where [:] denotes the concatenation of strings. We use BART (Lewis et al., 2019) as our backbone.
architecture mod-

---

Architecture
Knowledge
BLEU-2
BLEU-4
ROUGE-2
ROUGE-L
DIST-1
DIST-2
OpenDialKG
Sequence
26.60
15.09
20.09
33.13
8.92
33.92
Decoder-Only
Graph
27.13
15.62
21.07
34.18
9.28
35.01
Sequence
25.08
14.37
19.34
31.30
10.00
37.42
Encoder-Decoder
Graph
25.57
14.28
19.14
31.41
9.68
35.85
Sequence
27.35
15.12
20.55
34.12
9.65
37.66
Dual-Encoders
Graph
27.70
15.57
20.45
34.34
10.46
35.55
FaithDial Seen
Sequence
25.07
12.61
16.50
32.18
10.43
40.26
Decoder-Only
Graph
18.98
8.15
11.60
27.40
10.63
42.23
Sequence
24.47
12.21
18.25
34.41
9.64
40.07
Encoder-Decoder
Graph 20.60
9.23
13.71
28.97
10.40
43.89
Sequence
27.59
14.51
18.29
34.43
9.62
33.29
Mnal-Encoders
Graph
21.16
9.68
14.51
30.57
10.59
38.42
FaithDial Unseen
Sequence
23.60
10.69
15.21
31.41
15.88
54.25
Decoder-Only
Graph
18.96
7.91
12.01
27.83
14.62
51.46
Sequence
24.02
10.88
17.35
33.57
14.52
54.36
Encoder-Decoder
Graph
21.42
9.06
13.58
28.62
14.05
54.40
Sequence
24.17
10.28
15.31
31.89
16.60
59.41
Dual-Encoders
Graph
20.38
8.72
14.55
30.55
15,92
52.16
WoW Seen
Sequence
14.37
7.12
8.21
20.89
13.13
54.29
Decoder-Oni Only
Graph
15.68
7.16
9.48
22.57
12.34
53.19
Sequence
13.73
6.89
8.76
20.35
11.53
50.55
Encoder-Decoder
Graph
17.31
9.03
11.00
22.27
11.62
51.87
Sequence
16.00
8.20
9.78
22.88
14.21
58.70
Dual-Encoders
Graph
20.31
11.02
14.75
28.83
12.91
53.78
Wo Unseen
Sequence
13.69
6.92
7.86
20.72
8.93
41.44
Decoder-Only
Graph
13.85
5.48
7.67
21.39
9.13
44.44
Sequence
13.00
6.67
8.37
19.90
7.61
36.27
Encoder-Decoder
Graph
15.00
7.14
9.09
20.46
8.68
43.53
Sequence 15.39
7.81
9.19
22.17
9.22
39.16
Dual-Encoders
Graph 18.60
9.47
12.63
26.80
9.18
42.41
Table 2: The response quality performance of large pre-trained models.
bremner". Considering the knowledge graph is dis- ordered, we sort the triples according to the token overlap between the triples and the dialogue history, which keeps the potential knowledge triples from being truncated. According to our experiments, as long as the valuable triples can be within the maxi- mum length, the effect of the order is not significant. In this way, bridge the information granularity gap between graph nodes and tokens, i.c.. node is composed of many tokens. Furthermore, we can leverage a pre-traine rre-traieed model for the graph by considering the serialized knowledge graph as
natural language.
3.4 Metries
Knowledge-groundee dialogue generation involves two-fold evaluations: first, it needs to generate coherent and diverse responses as dialogue gener- ation task. Second, as a knowledge-intensing task, the generation needs to satisfy factual consistency with external knowledge. To evaluate the quality of the response, we adopt BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) to evaluate the generated response against the reference. Higher scores indicate that the generated results are closer

---

the reference. We also evaluate the diversity of generated response by n-grams distinct(Li al., 2016). Following previous works in evaluating faithfulness (Dziri et al., 2022a), we take the NLI, Q and QL (Honovich et al., 2021) as met- rics. The NLI evaluates the text entailment, and the evalluates the factual consistency by ques- tion generation and question answering.
Architecture
Knowledge
NLI
QF
Q NLI
OpenDialKG
Sequence
50.69
42.20
41.56
Decoder-Only
Graph
51.85
44.13
43.56
Sequence
53.41
43.89
41.91
Encoder-Decoder
Graph
51,16
40.17
38.82
Sequence
52.39
46.28
45.89
Dual-Encoders
Graph
55.42
44.14
42.05
Faithdial Seen
Sequence
62.27
59.15
54.31
Decoder-Only
Graph
54.82
41.01
36.42
Sequence 64.28
67.03
61.44
4 Experimental Analyses
Encnder-Decoder
Graph 53.43
44,42
39.51
Sequence 63,49
70.88
66.67
Dual-Encoders
This section presents our detailed investigation of how the knowledge forms affect the knowledge- grounded dialogue, focusing the better forms of knowledge, the mutual adaption of model and knowledge, and the performance under few-shot settings. In this section, large pre-trained models (GPT2-Large and BART-Larre) are leveraged for initialization unless otherwise highlighted. We use sequence- or graph-based models to refer to models that use knowledge text or knowledge graph as knowledge, respectively.
Gruph 56.70
49.34
46.04
Faithdial Unseen
Sequence 64.85
61.31
56.35
Decoder-Onip
Graph 58.70
45.89
41.33
Sequence 67.55
65.21
59.67
Encoder-Decooder
Gruph 60.85
49.37
44.31
Sequence
69.34
68.14
61.39
Dual-Encoders
Graph
61.17
51.07
46.55
WoW Seen
Sequence
54.07
55.08
49.37
Decoder-Only
Graph
53.42
41.24
36.35
Sequence
53.49
59.67
53.22
Encodar-Decoder
Graph
52.62
42.60
37.08
Sequence
54.
66.65
58.94
Dual-Encoders
To facilitate the following analyses, we high- light the knowledge's characteristics of different datasets. OpenDialKG provides manually anno- tated knowledge graphs and knowledge text, which are short and accurate knowledge. FaithDial and WoW both provide knowledge text and the corre- sponding knowledge graphs are extracted by au- tomated tools. For knowledge text, FaithDial's knowledge sentence-leve knowledge annotated by humans, while the knowledge in WoW ranges over long passages retrieved from Wikipedia. In comparison, the extracted knowledge graph in both FaithDial and WoW is not as good quality manu- ally annotated OpenDialKG. Therefore, the results of graph-basee models on OpenDialKG can be re- garded as the upper-bound of knowledge graph grounded dialoguc and the results of sequence- based models on FaithDial and OpenDialKG can be regarded as the bound of knowledge text grounded dialogue. However, as retrieving the pre- cise knowledge sentence triples is difficult in real-world practice, the results WoW is much closer to reality.
Graph
53.68
45,34
40.32
WoW Unseen
Sequence
53.82
55.55
50.46
Decoder
Gruph
52.89
36.37
32.33
Sequence
53.22
60.87
55.17
Encoder-Decodet
Gruph
53.13
42.19
38.22
Sequence
54.90
68.20
62.07
Dual-Encoders
Gruph
53.99
43.73
39.31
Table 3: The faithful consistency performance of large pre-trained models.
Response Quality Table 2 illustrates the results of response quality. The results on OpenDialKG are numerically significantly higher than the other two data sets which may indicate that whether the form of knowledge is sequence graph, more precise information will be more advantageous the results. On the other hand, the performance of graph- models slightly higher than the sequence-based which shows the upper-bound graph-basee based models is better. While on FailhDial, the sequence-based models show clear advan- tage as the knowledge graph is much noisier or sparse. The results of WoW present the opposite conclusion from FaithDial, where the graph-based models' performance is significantly better than that of the sequence-based model. It may indicate that when the knowledge text complex and noisy, the extracted graph can act as an information filter to reduce the noise. This result provides us with
4.1
Q1: Graph v.s. Sequence, which form of knowledge form is better?
Our analyses examine the validity of the knowledge text and graph woallldde-ggrunnded dialogue and attempt to determine the better form.

---

paths to improve generation quality, is to simplify the text, such annotating fine-grained sentences, and the other is to extract the knowledge graph to reduce the redundancy.
Our observations indicate that in the FaithDial dataset, knowledge graphs improve response di- versity in the "seen" setting, while knowledge text enhances diversity the "unseen" setting. Con- versely, in the WoW dataset, knowledge text im- proves diversity in the "seen" setting, whereas knowledge graph improves it in the "unseen" set- ting. These findings suggest that lower-quality knowledge can lead to greater diversity when the domain/topic is familiar during training, while higher-quality knowledge may be more beneficial when the domain/topic is unfamiliar, Furthermore, we speculate that the variation in diversity across datasets and knowledge formats may be attributed to differences in modeling difficulty and general- ization capabilities. For instance, Recoder-Only ar- chitecture can readily model fine-grained sentences and generalize to unseen examples in FaithDial but may struggle with complex knowledge passages in WoW.
(a) BLEU-4
(b) ROUGE-L
Figure I: Generalization performance in terms of train- ing on FailhDial and testing on WoW.
of graph-based data.
Human Evalution In addition reference-based metrics, comprehensive human evaluation is cru- cial for dialogue-related generation tasks. this end, we randomly selected 100 samples with two responses (from sequence- or graph-based mod- els) from each dataset for each architecture, and instructed annotators to choose the preferred re- sponse based dialogue history and given knowl- edge. The results, presented in Table 4, indi- cate that sequence- models outperform graph- based models, with a significantly higher win rate on FaithDial and WoW. These findings suggest that sequence-based models, particularly those using Dual-Encoders architecture, have a distinct advan- tage in interacting with humans. Furthermore, the results reveal discrepancy between the reference- based metries such as BLEU and ROUGE and hu- man preferences.
Factual Consistency The results of factual con- sistency shown in Table 3. Unlike the response quality, the knowledge text shows an absolute ad- vantage in factual consistency. It may be a bias that the metrics are calculated based on the text, which closer to the sequence than the graph. However, we lack unified metrics to evaluate the factual consistency of the knowledge graph and sequence. Overall, the Dual-Encoders architecture outperforms both metrics and is superior model for knowledge-intensive tasks.
Win Rate
Dataset
Architecture
Sequence
Graph
Decoder-Only
51%
49%
OpenDialKG
Encoder-Decoder
50%
50%
Dual-Encoders
69%
31%
Decoder-Only
66%
34%
FailhDial Seen
Encoder-Decnder
75%
25%
Dual- Mucl-Fnnoders
78%
22%
Generalizability Dziri et al. (2022a) examines the usefulness of FaithDial in out-of-domain set- ting by testing the performance on other datasets of models trained on FaithDial. Following the ting, we examine the generalizability of models with different types of knowledge by transferring the results of response generation from FaithDial to WoW. As shown in Figure 1, the knowledge graph results in response quality metrics are much higher than the knowledge text in all three architectures. The results on the unseen test set of FaithDial and WoW also support the observation. demonstrates that grspd models have stronger generaliz- ability than sequence-basee models, and when our data limited, it is helpful to construct knowledge
Decoder-Only
65%
35%
Dai Unseen
Encoder-Decoder
84%
16%
mual-Encoders
76%
24%
Decoder-Oniy
71%
29%
WoW Seen
Lncodder-Decoder
75%
25%
Dual-Encoders
75%
25%
Decoder-One
60%
40%
WoW Unseen
Encoder-Decaler
74%
26%
Dual- Lncl-Lns
71%
29%
Table 4:
Human comparison evaluation.
Based on the above analyses, for knowledge- grounded dialogue, the knowledge graph has bet- ter response quality and generalizability: however, the knowledge text better the factual consis- tency of generating responses. Moreover, manual

---

CpmDannn
PannCial
Woll
Opereenn
Seuoder -DalOrroom
KDrown Dry
OpHDaIG
Far
Contee
FXnCCa
alorge
alorge
(a) Sequencc
Graph
(a) Decoder-Onis
(b) Decoder-Only
Figure 2: BLEU-4 score of different architectures.
subgraphs selection in OpenDialKG and manually grounded sentence annotation in FaithDial inspire us that denoising the source knowledge can im- prove the generation. Specifically, we can finely retrieve an accurate subgraph when the source is the knowledge graph. While the source is the knowl- edge text, we can extract knowledge graphs to re- duce redundancy like WoW rank fine-grained semantic units to select precise knowledge.
OprCced
Fartiia
adeer
(c) Encoder- Encoder-Decoder
(d) Encoder-Decoder
FarChw
Fand
Aboe adage
+boas addge
(e) Dual-Encoders
(I) Dual-Encoders
4.2
Q2: To what extent the model and knowledge should be mutually adaptive?
Figure 3: BLEU-4 score of various model size. The figures on the left side are based on result grounded in knowledge text, while the figures on the right side are based on result grounded in graph.
Following the selection of the knowledge form, problems arise like as what kind of architecture is suitable for encoding the particular knowledge, whether a larger model size is preferable, whether pre-trained model is required, etc. To facilitate the subsequent research, we comprehensively compare detailed factors of the model and investigate to what extent the model and knowledge should be mutually adaptive.
knowledge in OpenDialKG is of high quality; even though it has limited data, the larger model per- forms better. We infer that this is because some knowledge's characteristics, such as the standard knowledge graph in OpenDialKG, are more con- ducive to exploring the potential of pre-trained models, like prompt learning. Furthermore, knowl- edge in WoW has higher noise for both sequence and graph forms, with varying effects model size. The above statistics indicate it is not that the larger the model size is, the better performance is. The optimal model size is related to training data size and is highly associated with knowledge's sophisticated characteristics.
Model Architecture Figure compares the BLEU-4 score of different architectures. Besides OpenDialKG, the Decoder-Only architecture out- performs the Encoder-Decoder architecture for knowledge text, while the opposite is observed for the knowledge graph. On the contrary, the Dual- Encoders outperform others in most cases whatever the knowledge form is. indicates that the Dual- Encoders have a stably excellent performance over varying forms knowledge. Therefore, we con- clude that the Dual-Encoders is a wise choice to eliminate the impact of different knowledge.
Pretrained or not We experimented with inves- tigating the impact of pre-trained initialization, fo- cusing on the effect of different knowledge forms and architectures. Figure 5 shows the effect of pre- trained across different architectures and knowl- edge forms, with red indicating the results without pre-trained and blue indicating the improved results achieved with pre-trained. The positive impact of pre-trained initialization is universal. However, pre- trained can lead to a more remarkable performance improvement when the knowledge is represented in sequence form, compared to the serialization
Model Size Figure compares the BLEU-4 score of various model sizes of Dual-Encoders for the two forms of knowledge. Hoffmann et al. (2022) stated that the model size of large language mod- els should be matched to the size of the training data. In addition, our results further indicate that the model size should mutually adaptive to the knowledge's 's characteristics. For instance, the

---

form of the knowledge graph. The reason may be that the knowledge graph, a form of structured data, lacks the natural language pattern after serializa- tion, making it less intuitive for pre-trained mod- els designed to understand and generate language fluently. By comparing the results without pre- trained, we can see that the architectures exhibit different performances under different knowledge forms. For instance, the Dual-Encoders architec- ture yields the best performance the sequence form, whereas the Decoder-Oni) architecture at- tains the optimal results in the graph form.
(a) OpenDialKG
In terms of overall results, the nual-Encoders ar- chitecture with pre- pre-trained initialization positively impact the results. It suggests that performance could be improved further by advanced Dual- Encoders structure or by employing domain adap- tion pre-training, and the improvement is slightly related to knowledge's characteristics. There- fore, we suggest further research to choose Dual- Encoders with pre-trained initialization line and design experiments of various model sizes for different knowledge indicate the contribution.
(b) FaithDial
(c) WoW
base-
Figure BLEU-4 score in few-shot settings
4.3
Q3: How does various knowledge perform in few-shot settings?
only a few samples (100-200), the input form and model architecture should resemble the pre-trained model, such as Encoder-Decodee Then, when the data is larger (500+) or better (e.g.. detailed manual annotations), we can train a task-adapted model, such as mnal-Encoderr.
Since the knowledge-auxilian is an important di- rection of few-shot learning (Chen et al., 2021), we would like to explore the performance of various wowledge--bssed models under few-shot settings. Unlike related work dedicated to few-shot learn- ing (Li et al., 2022, 2020), we do not consider par- ticular methods such domain pre-training with pseudo-data optimize models, but only the models mentioned in the above paper.
Conclusion
Figure 4 shows the results from zero-shot to 1000-shot. Because the pre-trained model is trained natural language, the sequence- cequence-based model out- performs the graph-based model. However, it may be due the gap between serialized graphs and nat- ural language for the pre-trained language model. The graph-based models would be better if the model employed a graph-based pre-training stage. Similarly, the Deciddrr-Deoderrrree is closest to the pre- training phase, while the Dual-Encoders have enor- mous variation. Thus, the Sicodde-Decodde per- forms best, while the Dual-Encoders require more training data to fine-tune. However, as the data in- creases, the Dual-Encoders' performance improves significantly, surpassing other models, such as 500- shot on OienDialKG. Therefore, when we have
In the paper, we analyze the strength and limita- tions of two forms of knowledge for knowledge- grounded dialogue. We conduct comprehensive study to determine which form of knowledge su- perior and to what extent the model and knowledge should be mutually adaptable. Additionally, we in- vestigate the performance of various knowledge in few-shot settings. Our experimental results indicate that different knowledge forms have their advan- tages, but task-oriented instructions, such as proper knowledge and model selection, essential. We also summarize determinant conclusions about the influencing factors for subsequent research. In fu- ture work, we will analyze the influencing factors more quantitatively and explore general conclu- sions for other mnolledgeinntensin tasks.

---

Limitations
Natural Language Processing (EMNLP), pages 8635- 8648, Online. Association for Computational Lin- guistics.
This work may contribute to the development of knowledge-intensive language tasks. Nevertheless, there are two main limitations of the works. First, limited by the experimental equipment, we can not access the larger pre-trained language mod- els, such as T5-XXL. for the scaling experiments. As analyzed in Section 4.2, the model's size and knowledge are correlated. Therefore, all the conclu- sions in this paper could have changed if the larger models had been used. However, it is essential to realize that the training cost of large models is vast, so it may be necessary to compare effectiveness and efficiency. Secondly, We only focus on the mhowledge-grounded dialogue generation as a rep- resentative task for knowledge-intensive language tasks. Besides, other tasks, such as fact-checking and
Tanmoy Chowdhury, Chen Ling, Xuchao Zhang, Xu- jiang Zhao, Guangji Bai, Jian Pei, Haifeng Chen, and Liang Zhao. 2023. Knowledge-enhanced neu- ral machine reasoning: A review. arKiv preprint wwXiv:2302.02092
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Confer- ence on Learning Representations (ICLR).
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Za- iane, Mo Yu, Edoardo Ponti, and Siva Reddy. 2022a. Faithdial: A faithful benchmark for information- seeking dialogue. arXiv preprint, arXiv:2204. 10757.
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022b. On the origin hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 5271-5285 Scattle, United States. Association for Computational Linguisties.
should be considered. The larger models will be considered in future work, and more extensive research will be introduced in other knowledge-intensive language tasks.
A Acknowledgements
Marjan Ghazvininejad, Chris Brockett, Ming-Wai Chang, Bill Dolan, Jianfeng Gao. Wen-tau Yih, and neu-
We extend our sincere gratitude to the anonymous reviewers for their helpful feedback, the diligent efforts of the conference committees, and the col- laboration of the members of the Beijing Engineer- ing Research Center of High- Volume Language Information Processing and Cloud Computing Ap- plications for their valuable insights. This work has received partial funding from the Joint Funds of the National Natural Science Foundation of China (Grant No. U21B2009) and the Major Research Plan of the National Natural Science Foundation of China (Grant 90.
Michel Galley. 2018. A knowledge-groundee conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks. Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican. George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. CORR, abs/2203.
Or Honovich, Leshem Choshen, Roce Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q²: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
References
Hannah Bast, Florian Bäurle, Björn Buchhold, and EI- mar Haußmann. 2014. Easy access the freebase dataset. In Proceedings of the 23rd International Conference on World Wide Web, pages 95-98.
Jiaoyan Chen, Yuxia Geng, Zhuo Chen, lan Horrocks, Jeff Z Pan, and Huajun Chen. 2021. Knowledge- aware zero-shot learning: Survey and perspective. In the 30th International Joint Conference on Artificial Intelligence.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.69880
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeulleoyer. 2019. Bart: De- noising pequence-to-sequenee pre-training for natural language generation, translation. and comprehension. arXiv preprint arXiv:1910. 1910:1990.1461 13461.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-groundee pre-
training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in

---

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao. and Bill Dolan. 2016. A diversity-promoting ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies. pages I 10, San Diego, California. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow in- structions with human feedback. arXiv preprint arXiv:220..01555.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Ji- apu Wang, and Xindong Wu. 2023. Unifying large language models and knowledge graphss A roadmap. arXiv preprint 2xX6v:2306.08333
Linxiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020. Zero-resource soowledgg-grounder dialogue generation. Advances in Neural Information Processing Systems, 33:8475- 8485.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. Advances in neural information processing systems, 31.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544 Online. Association for Computational Linguistics.
Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Li- den, Zhou Yu, and Jianfeng Gao. 2022. Knowledge- grounded dialogue generation with unified knowl- edge representation. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 206-218, Seattle, United States. Association for Computational Linguisties.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdi- nov. 2021. Focused attention improves document grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics. Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 84-81.
Shilei Liu, Xiaofeng Zhao, Bochao Li, Feiliang Ren, Longhui Zhang. and Shujuan Yin. 2021a. A Three-Stage Learning Framework for Low-Resource Gnowledge-Groundee Dialogue Generation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 2262- 2272, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text- to-text transformer. Journal of Machine Learning Research. 21(140):1-67.
Zeming Liu, Haifeng Wang, Zheng- Yu Niu, Hua Wu, and Wanxiang Che. 2021b. DuRleDial 2.0: A bilin- gual parallel corpus for conversational recommen- dation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4335-4347, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in wowleddee-roundeee dialogue with controllable features. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 704-718, Online. Association for Computa- tional Linguistics.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. 2023. Augmented language models: a survey, arXiv preprint arXiv:2302.00 02842.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. Opendialkg: Explainable conver- sational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.

---

Petar Veličković. Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lid, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Architecture
Sequenceize
Params
base
237.35M
Decoder-Only
Medium
676.77M
Large
1476.35M
Lingzhi Wang, Huang Hu. Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022. RecInDial: A uni- fied framework for conversational recommendation with pretrained language models. In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 1 2thh International Joint Conference on Natural Lan- guage Processing ( (Volume 1: Long Papers), pages 489 189-500,000 Online only. Association for Computational Linguistics.
Base
265.92M
Encoder-Decodee
Large
774.94M
Base 375.60M
Cual-Encoders
Large
1161.39M
Table 5: The number of parameters of models in differ- ent size.
same weights for the Dual-Encoders model, and the two cross-attention blocks were initialized with the same weights in the decoder layers. Hence, the layer size of the cross-attention is the same as in BART, We trained all models for 10k steps, using a batch size of 32 and the Adam optimizer (Kingma and Ba, 2014) with learning rate of 5e-5. We warmed up the learning rate for 0.5k steps followed by a linear decay. The max length of dialogue history and response is 128 and 64 tokens. The max length of the knowledge text in OpenDialKG, FaithDial, and WoW are 64, 64, and 256. The knowledge graph's max length is 64, 512, and 512, respectively. The models were evaluated on the val- idation set per 0.5k steps, and the best-performing model was saved for testing. We early stopped the training with patience of 5. Training for all models was done on an Nvidia GeForce RTX 3090GPU 24GB, and for inference, we used nucleus sam- pling with p=0.6. Table 5 shows the statistics of parameters of models in different sizes.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:/910.03771.
Yizhe Yang. Yang Gao, Jiawei Li, and Heyan Huang. 2022. g": Enhance knowledge grounded dialogue via ground graph. arXiv preprint arXiv:2204. 12681.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of nowledge--nhhaneee text generation. ACM Computing Surveys, 54(11s):1-38
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020a. Low-resource gegeldddggggggeeedd dialogue generation. arXiv preprint arXiv:2002.10344
Xueliang Zhao, Wei Wu, Can Xu. Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020b. Knowledge- grounded dialogue generation with pre-trained lan- guage models. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLPP), pages 3377-3390, Online. As- sociation for Computational Linguistics.
For few-shot settings, inspired by the experi- ments on model size, we only explore the perfor- mance of base-size models [BART-base and GPT2- base) with pre-trained initialization. We also ex- periment with the large-size model, but the results show that the large-size model could not fit the few-shot data. To prevent overfitting at the few- shot setting, we reduce the learning rate to 1e-5, set the warmup steps to 100, and set the total training steps to 5K. We also early stop the training with a patience of 3.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
A Implementation Details
Models in the paper were implemented by Py- Torch and Transformers (Wolf et al., 2019). The Dual-Encoders and Encoder-Decoder architecture are based on BART (Lewis et al., 2019), and the Decoder-Onle Only architecture is based on GPT-2 (Rad- ford et al., 2019).
A.1 Impact of Pretrained B Encode Graph by GAT
We used the transformer toolkit (Wolf et al., 2019) implement the models. We initialized the external parameters for the Dual-Encoders fol- lowing DoHA (Prabhumoye et al., 2021). Specif- ically, We initialized the two encoders with the
In addition serialization, we also considered leveraging GAT (Veličković et al., 2018) to encode the knowledge graph, and the results are shown in Table 6. Specifically, we pool the token representa-

---

Architecture
dataset
Graph Encoder
Pre trained
BLEU-2
BLEU
ROUGE 2
R
DISTINCT
DISTINCT-2
GAT Encoder
No
12.41
5.03
8.43
19.93
4.40
21.95
OpenDialKG
13.33
28.55
Serialization
25.57
14.28
31.41
9.68
35.85
GAT Encoder
No
6.90
1.22
3.30
14.10
7.38
38.17
13.64
Serialization
Yes
20.60
9.23
13.71
28.97
10.40
43.89
No
9.03
2.57
4,85
16.70
7.97
41,48
Fncoder-Decoder
Serialization
Yes
21.42
9.06
13.58
28.62
14.05
$4,40
GAT-Enevder
No
6.04
1.17
2.71
13.89
5.98
35,60
Serialization
Yes
17.31
9.03
11.00
22.27
11. 62
$1.87
GAT-Fncoder
No
4.52
0.63
1.75
11.99
5.68
33.65
Serialization
Yes
15.39
7,81
9.19
22.17
9.22
39.16
GAI-Encoder
No
15.04
5.89
9.10
22.61
6.21
27.02
OpenDialKG
22.00
Serialization
Yes
27.70
15.57
20.45
34.34
10,46
35.55
GAl-Encoder
No
8.70
2.21
3,80
17.09
10.80
46.41
FaithDial
Serialization
Yes
21.16
9.68
14,51
30.57
10.59
38.42
GAN-Encoder
No
11.74
3.87
6.24
20.28
11.89
50,50
Dail
Serialization
Yes
20.38
8.72
14,55
30.55
15.92
52,16
GAT Encoder
No
8.74
2.35
3.80
16.64
8.73
44.39
44,96
Serialization
Yes
20.31
11.02
14.75
28.83
12.91
53,78
GAT Encoder
No
5.87
1.01
2.09
13.87
8.47
42.51
WoW
Serialization
Yes
18.00
9.47
12.63
26.80
9.18
42,41
Table 6: The comparison results of graph-based models with GAT encoder and serialization.
tion as a representation of the nodes and leverage GAT instead of the encoder in the architecture. As we do not apply pre-trained initialization on GAT, we only compare with the no pre-trained models. The table shows that the graph encoder results better than those leveraging the serialization ap- proach. However, the graph encoder models also underperform the models with pre-trained initial- ization. Therefore, when pre-training with Graph- to oo-Sequence is unavailable, it is an excellent way to serialize the knowledge graph into sequence.
40
20
10
Open.
WoN Falth.
Falth.
Pretronned
No pretroined
No pretrained
Pretroined
Decoder-Only
(b) Decoder-Only
40
40
20
20
Pretriined
No pretroined
ho pretrained
Pretrained
(c) Sncoder-Decoderr
(d) Enender-Decoder
40
40
20
88 Pretrained No pretroined
No pretraaed
Pretrened
(c) Ducl-ennoders
( Dual- Sanl--Enoddes
Figure 5: The effect of pretrained. The figures on the left side based on result grounded in knowledge text, while the figures on the right side are based on result grounded in graph.


Reviews:
[{"review_id": "JhcEHBYXgq", "reviewer": "Reviewer_Ez2J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and suggestions.\n\n- In addressing the critique of limited dataset use, it is important to underscore the representativeness and real-world applicability of the three datasets adopted in our study. As detailed in Section 3.1, OpenDialKG and FaithDial incorporate human-annotated knowledge graph or text, respectively. These datasets embody pure, high-quality knowledge that set an ideal experimental scenario, delineating the upper limit of similar knowledge forms. Conversely, the WoW dataset, with knowledge retrieved from Wikipedia or constructed via OpenIE, mirrors more closely a real-world scenario. While there are indeed other available datasets like CMU_DoG, they do not extend beyond the characteristics intrinsic to our chosen datasets. Consequently, we believe our selection of these three datasets for the study is justifiable and robust.\n\n- In response to the concern about under-representation of text and graphs structures, it should be noted that our study is predicated on employing the most straightforward knowledge augmentation methods for validation, as its primary goal is to compare graph and text sequences fairly. We acknowledge certain optimization approaches specific to each knowledge type—like graph-based reasoning and text-based retrieval—are not explicitly discussed. Although these methods are bespoke solutions well-suited to their respective knowledge types and may influence specific results, they do not impede our capacity to draw relevant general conclusions.\n\n- In response to the critique about dated dialog evaluation metrics, we acknowledge that concern. As stated in Section 4.1, we have indeed noted the limitations of automated metrics and hence incorporated human evaluation into our methodology for a more comprehensive understanding. Our analysis and conclusions incorporate both types of evaluation.\n\n- In response to the comment on results is dataset dependent, it is essential to highlight that the datasets we employed, as mentioned in Answer 1 and Section 4.1, are representative and encompass diverse aspects. The distinct characteristics of these datasets naturally yield varied conclusions. Consequently, our findings are not prescriptive, suggesting one approach over another, but rather they offer optimization methods tailored to individual datasets. Thus, our focus is on enhancing text quality, retrieving fine-grained knowledge, among other improvements, rather than asserting the superiority of one method over the other.\n\n- In responding to your observation about potential bias in factual consistency results due to text-based metrics, we appreciate the feedback. As you've duly noted, we have highlighted this limitation in our work. We plan to innovate knowledge-independent factual consistency evaluations in future studies to address this concern. Nonetheless, while this caveat is important, it does not invalidate our current findings or serve as a sufficient basis to dismiss this study.\n\n- Responding to your query about graph embedding methods and their interplay with response generation systems; indeed, a variety of such methods exist. However, these mostly focus on modeling the entire graph and do not readily adapt to knowledge-grounded conversations. The advent of pre-trained language models has led to the use of knowledge graphs as sequential text inputs becoming predominant in knowledge-based text generation. We conducted experiments using typical graph encoders (refer to Appendix C.8) and determined that it is not equitable to compare graph embeddings with large-scale pre-trained language models. Consequently, we opted for the serialized knowledge graph approach for comparison in our study.\n\n- In response to the comment about the absence of studying the role of prompting, while it's recognized that different prompts could potentially influence the generated responses, it is important to note that prompt engineering is highly individual and somewhat constrained. The central aim of our study is to compare knowledge across different forms, thus enabling us to set aside such engineered optimization strategies. Instead, we adopt a naive approach that promotes drawing broad generalizations without being overly influenced by the specificities of prompting.\n\n- We appreciate your observation regarding the omission of recent surveys in our work. We will diligently review these documents and incorporate necessary references to enhance the depth and relevance of our study in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper suggests some good findings as follows:\n- the knowledge graph outperforms generation quality and exhibits stronger generalizability, while the knowledge sequence outperforms factual consistency in generations.\n- Performance can be effectively improved by denoising the knowledge, for example, by selecting the succinct sequence or extracting a structured knowledge graph.\n- Performance could be universally improved further by advanced Dual-Encoders structure or by employing domain adaption pre-training; however, impact of model size and knowledge size is understudied\n- (and more) \nThe findings should be reported in a tabular form for quick access to the results and findings. \n- Exhaustive set of experiments are reported in the appendix to study the 3 major questions authors are trying to study in this work.", "reasons_to_reject": "- This seems like a limited study that uses 3 datasets for studying knowledge augmented response generation. The goal of the study is to compare graphs and text sequences. How one represents the text and graphs structures for knowledge augmentation including the reasoning methods utilized with the graphs is understudied (to make general claims about which paradigm is better)   \n- Dialog evaluation metrics are very dated and do not correlate well with human judgment.  \n- Response quality results seem to be inconclusive and dataset dependent \n- As the authors note, factual consistency results may be biased due to the text based metrics used in the study \n- Various graph embedding approaches exist, how would these interplay with the response generation system and compare to the sequence based representations.\n- Role of prompting that is very common presently in response generation is not studied at all in the paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the strength and limitations of graph and sequence based representations of knowledge for knowledge grounded dialogue. They use 3 datasets and conduct studies to test effect of different representations on response quality, factual consistency, generalizability and response scores in few shot settings. The paper tries to compare graphs and sequences head to head and answer which for may be better under various circumstances. \n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Search suggests several recent surveys exist on the topic; these are not mentioned in the paper. \n\nYu, Wenhao, et al. “A Survey of Knowledge-Enhanced Text Generation.” ACM Computing Surveys, vol. 54, no. 11s, Jan. 2022, pp. 1–38. Crossref, https://doi.org/10.1145/3512467.\n\nMialon, Grégoire, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière et al. \"Augmented language models: a survey.\" arXiv preprint arXiv:2302.07842 (2023).\n\nChowdhury, T., Ling, C., Zhang, X., Zhao, X., Bai, G., Pei, J., Chen, H. and Zhao, L., 2023. Knowledge-enhanced Neural Machine Reasoning: A Review. arXiv preprint arXiv:2302.02093.\n\nPan, S., Luo, L., Wang, Y., Chen, C., Wang, J. and Wu, X., 2023. Unifying Large Language Models and Knowledge Graphs: A Roadmap. arXiv preprint arXiv:2306.08302.\n\nGroth, P., Simperl, E., van Erp, M. and Vrandečić, D., 2023. Knowledge Graphs and their Role in the Knowledge Engineering of the 21st Century (Dagstuhl Seminar 22372). In Dagstuhl Reports (Vol. 12, No. 9). Schloss Dagstuhl-Leibniz-Zentrum für Informatik.\n"}}, {"review_id": "NAmxVekJJD", "reviewer": "Reviewer_eXkB", "summary": "", "questions": "1-\tMy question and suggestion concern noise since we know this can impact model structure. What you should do to minimize noise impact on model size?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and Suggestions.\n\n- We apologize if the extensive use of abbreviations in our text made it challenging to follow. In our revised version, we will endeavor to minimize the use of abbreviations and provide clearer explanations where necessary, thereby enhancing readability and comprehension..\n\n- As analyzed in Section 4.2, high noise knowledge can indeed confuse larger models, particularly those based on Dual-Encoder Architectures. That is an important and interesting future work involved from our analyses. We hypothesize that this issue arises because such models aren't sufficiently pre-trained and thus are more sensitive to noisy knowledge input. To mitigate this, we suggest a post-training stage for larger pre-trained models to better tailor them to the task at hand. Moreover, as larger models inherently have higher capability, they are more sensitive to fitting or overfitting to noise in data. Therefore, another possible solution is to 'denoise' the training data or modify it to closely resemble the format of pre-training inputs. This could involve retrieving fine-grained text and improving information extraction technology, which may prove beneficial for larger model training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1-\tThe paper is weel structured and present clearly the focos and main scientific goals;\n2-\tExperiments use three different architectures that produce good insigths for further research\n", "reasons_to_reject": "1-\tThe text as too much abbreviations. Some times is hard to follow;", "questions_to_authors": "1-\tMy question and suggestion concern noise since we know this can impact model structure. What you should do to minimize noise impact on model size?", "paper_topic_and_main_contributions": "The contribution of this paper is that it investigate different forms of knowledge, KG and KS, and their respective advantages and disadvantages in knowledge-grounded dialogues.\nThe paper aims to answer three main questions:\n1. Graph v.s. Sequence, which form of knowledge is better?\n2. To what extent the model and knowledge should be mutually adaptive?\n3. How does various knowledge perform in few shot settings?\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8nqnlaCJLx", "reviewer": "Reviewer_E5Fq", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and Suggestions. We apologize if the current writing style has made our paper difficult to follow. We take your feedback seriously and will strive to enhance the clarity and flow of our writing in the revised version, ensuring our message is effectively conveyed.\n\n- We apologize if our choice of terminologies has caused confusion. Indeed, \"decoder-only\" is a more accurate term than \"auto-regressive,\" and \"text sequence\" is simpler and easier to understand than \"knowledge sequence.\" Your corrections are well-received and will be carefully incorporated in our revisions. Thank you for your insightful suggestions.\n\n- We appreciate your insightful suggestions. In response, we have conducted a p-value significance test which confirms that the dual-encoder architecture statistically outperforms other architectures (p-value < 0.05). We concur that performance metrics alone are insufficient for model selection. Our study also explores other aspects like generalizability across different architectures and knowledge forms (refer to Section 4.1).\nRegarding scalability and adaptability, we performed experiments on large language models under zero-shot instruction, using Alpaca (13B) as a Decoder-Only architecture and Flan-T5-XXL (11B) as an Encoder-Decoder architecture. We validated the coherence and faithfulness of these large-scale models via human evaluations using Likert scores.\n\n|  Dataset   | Model  |  Knowledge Form | Coherence | Faithfulness |\n|  :----:  | :----:  |  :----:  |  ----:  |  ----:  |\n| OpenDialKG | Alpaca | text | 4.42 | 3.66 |\n| OpenDialKG | Alpaca | graph | 4.50 | 3.61 |\n| OpenDialKG | Flan-T5-xxl | text | 3.91 | 3.33 |\n| OpenDialKG | Flan-T5-xxl | graph | 3.60 | 3.48 |\n| FaithDial Seen | Alpaca | text | 4.24 | 3.25 |\n| FaithDial Seen | Alpaca | graph | 4.14 | 3.37 |\n| FaithDial Seen | Flan-T5-xxl | text | 3.41 | 3.43 |\n| FaithDial Seen | Flan-T5-xxl | graph | 2.99 | 3.47 |\n| FaithDial Unseen | Alpaca | text | 4.48 | 3.74 |\n| FaithDial Unseen | Alpaca | graph | 4.24 | 3.47 |\n| FaithDial Unseen | Flan-T5-xxl | text | 3.53 | 3.55 |\n| FaithDial Unseen | Flan-T5-xxl | graph | 3.14 | 3.36 |\n| WoW Seen | Alpaca | text | 3.93 | 4.32 |\n| WoW Seen | Alpaca | graph | 3.86 | 3.69 |\n| WoW Seen | Flan-T5-xxl | text | 3.61 | 3.75 |\n| WoW Seen | Flan-T5-xxl | graph | 3.28 | 3.20 |\n| WoW Unseen | Alpaca | text | 4.32 | 4.34 |\n| WoW Unseen | Alpaca | graph | 3.86 | 3.37 |\n| WoW Unseen | Flan-T5-xxl | text | 4.05 | 3.53 |\n| WoW Unseen | Flan-T5-xxl | graph | 3.88 | 2.80 |\n\nAs evident from the results, larger models demonstrate a preference for text due to the extensive text data used during pre-training and instruction tuning. However, the conclusions stated in our paper, such as the impact of knowledge quality and architecture, still hold true. These findings suggest valuable insights for building knowledge-grounded conversation applications and exploring the potential of other architectures (like Dual-Encoders) when scaled up and adapted for tuning. Nonetheless, our analysis is mainly based on small-scale models. \n\n- Thank you for your constructive feedback. We acknowledge that a significant portion of our results is presented in the appendix due to the large number of experiments we conducted. Given the paper's space constraints, it was challenging to include all these results and analyses in the main body. Some key findings from the appendix include:\n\n    (1)The robust performance of Dual-Encoders architecture across varying forms of knowledge.\n\n    (2)The strong generalizability of graph-based models as indicated by our transfer experiments.\n\n    (3)Insights on the interplay among model size, knowledge form, and knowledge size, which can guide practical application.\n\n    (4)The superiority of pre-training initialized sequence-based models over models with graph-encoder for knowledge graph. This suggests that when Graph-to-Sequence pre-training is unavailable, serializing the knowledge graph into sequence for sequence-based pre-trained models is a good alternative.\n\n    Taking into account your suggestion, we will strive to optimize the presentation in our revised version. We'll balance the constraints of space while ensuring the most valuable and interesting results are included in the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments performed to conclude the model architecture trade-offs are impressive. (Appendix is more interesting than the paper!)\n", "reasons_to_reject": "- The main drawback of the paper is the writing. It is very difficult to follow the message. \n\nFor instance, \n1. Some of the terminologies of choice are confusing. Some examples: \n\"Autoregressive\": Are the authors talking about decoder-only archictures? BART is autoregressive as well. \n\"Knowledge Sequence\": is there are any difference with the text sequence (Which is simpler and easier to discern). \n\n2. The authors show the promise of dual encoder models but the numbers and conclusions strongly recommend dual encoder models. A statistical significance test for instance could help make a stronger case. From the numbers and metrics chosen, it is hard to conclude the superiority of the model recommended. The performance metrics are not the only choice for model selection often times. Scalability and the ability to be adapted to different applications (for instance, prompting & Instruction tuning enable development of various other applications).\n\n3. The papers are supposed to not rely on Appendix to convey the message. A lot of the results from the appendix conveys the findings better. The writing needs to be significantly improved to get a lot of the interesting results from the Appendix into the main paper.  \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work studies how knowledge graphs (knowledge tuples) compare with knowledge sequences (unstructured textual contents). The author's posit that the two forms of knowledge are to be studied \"on their own\" and their impact on performance in models of choice. The authors perform experiments on four popular and relevant datasets. To compare the performance on two types of knowledge representations, the authors convert the knowledge graph into sequence of tokens and leverage decoder-only, encoder-decoder and dual encoder models for their experiments. The authors list model-architecture recommendations based on the experiments performed. \n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work analyzes the strength and limitations of knowledge graphs (knowledge tuples) compare with knowledge sequences (unstructured textual contents) for knowledge grounded dialogue.

Contributions:
 - The analysis provides valuable insights for building knowledge-grounded conversation applications

Weakness:
 - Missing references and the paper's message is sometime hard to follow.
Decision: Accept-Main