Multi-Source Multi-Type Knowledge Exploration and Exploitation for Dialogue Generation
Authors: Xuanfan Ni
Hongliang Dai
Zhaochun Ren
Piji Li
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue Generation
Natural Language Processing
Dialogue Knowledge

Abstract: Open-domain multi-turn dialogue generation encounters the significant challenge of lacking various types of knowledge from diverse sources. Existing models typically focus on identifying specific types of dialogue knowledge and utilize corresponding datasets for training. However, this approach often leads to limited generalization capabilities and increased computational resource requirements. Recently, large language models (LLMs) have shown impressive performance on natural language processing tasks. To harness the knowledge storage of LLMs, we propose a framework named KnowEE that explores multi-source multi-type knowledge from LLMs by leveraging diverse datasets and then exploits the obtained knowledge for response generation. Our framework comprises two phases: First, we leverage five external datasets encompassing various types of knowledge to extract the most relevant samples to the dialogue context which are served as prompts to generate corresponding type of knowledge; Second, we inject the acquired knowledge into the ongoing dialogue context in fine-grained and coarse-grained manners, which is then fed into LLMs to generate the final dialogue response. Both automatic and manual evaluation results validate the effectiveness of our framework in exploring and exploiting multi-source multi-type knowledge to generate coherent, informative, and fluent responses.

Multi-Source Multi-Type Knowledge Exploration and Exploitation for Dialogue Generation
Xuanfan N Hongliang Dail.², Zhaochun Ren', Piji Li
'College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China Shandong University, Qingdao, China xxannannii hongl dai, pjli) } @nuaa. edu. zzaochun, ren@sdu, edu cn
Abstract
am interested in the cultunl history the Republic of China period. Are and Zhou two people?)
Open-domain multi-turn dialogue generation encounters the significant challenge of lack- ing various types of knowledge from diverse sources. Existing models typically focus identifying specific types of dialogue knowl- edge and utilize corresponding datasets for training. However, this approach often leads to limited generalization capabilities and in- creased computational resource requirements. Recently, large language models (LLMs) have shown impressive performance on natural lan- guage processing tasks. To harness the knowl- edge storage of LLMs, we propose framework named KnowEE that explores multi-source multi-type knowledge from LLMs by leverag- ing diverse datasets and then exploits the ob- tained knowledge for response generation. Our framework comprises two phases: First, leverage five external datasets encompassing various types of knowledge to extract the most relevant samples to the dialogue context which are served as prompts to generate correspond- ing type knowledge; Second, we inject the acquired knowledge into the ongoing dialogue context in fine-ge gre-gra and coarse-gras coarse-grained man- ners, which is then fed into I.LMs to generate the final dialogue response. Both automatic and manual evaluation results validate ef- fectiveness of our framework in exploring and exploiting multi-source multi-type knowleds ledge generate coherent, informative, and fluent responses.!
Knuwledge Storuge Homan ChatGPT
Key Knowledge
Direct InpetIupu Input
Rilppeter Zhou Shrnen (1881 1990), better known by pet
sun: Lu Xan, was Chasse witte liteaay ctili, and lecturer
Omettoo: Cunous Ryoc: La Nan and Zheu Zureen
Fivee Spenier anks for abour Xun Zhon Shane
Activate
Premosa:: Speaker aahtni histry ofChin Chins
Response Respouse
are some person, Xun the pen name of Zhou Sburee.
La Xun (1881-1930) was one of the importunt pinneers modern Chinese literature.. Zhou Shuren (1897- 1975), also known Zhou Zacren, was another signiffoont modern Chinese sniter and intellectual..
Can introduce Lu Nun 7: Do you know La Nun
Retrivee
ChatGPT Knowledge
Zhou Shuren (25 September 1881 19 October 1936). known by his pen name writer, literary critic, lecturer, and stare servant.
Figure 1: Two cases of dialogue generation for human and ChatGPT. When generating responses, humans em- ploy their knowledge related the dialogue history to ensure coherency and accuracy. Conversely, solely in- putting the dialogue history into ChatGPT lead ballucination, despite its possession of corresponding knowledge.
topic, event, persona, general world knowledge. etc. to facilitate smooth and high-quality communi- cation. And this knowledge is gradually acquired through various sources in the process of learn- ing after birth. For example, as depicted in Fig- ure when asked if Lu Xun and Zhou Shuren are the same person, humans draw upon various knowledge sources to comprehend different types of knowledge related to the dialogue history, in- cluding the speaker's emotion and intent, the over- all topic of the dialogue, and relevant background knowledge about Lu Xun and Zhou Shuren. Finally, can provide a appropriate and correct response. Therefore, enabling a model to explore and ag- gregate various types of knowledge from multiple sources, similar to how humans do, and exploit- ing them effectively will play a crucial role in im-
1 Introduction
Open-domain multi-turn dialogue generation neces- sitates models capable of producing high-quality responses that are coherent, consistent, and infor- mative (Li et al., 6b; Roller et al., 2021; Wu et al., 2022). In human conversations, we naturally connect various types of knowledge such emotion,
"Corresponding author. 'Code: http://gttaah.com/Paattttttiiiie

---

proving the quality of intelligent human-machine conversations (Wu et al., 2022).
ChatGPT system. We analyze that relying solely on the dialogue historical context cannot effectively stimulate and explore the rich knowledge stored within the LLMs when generating responses.
Previous studies have utilized external knowl- edge sources to bridge the knowledge gap between machines and humans in conversation by inject- ing knowledge that is difficult to learn purely from the given dialogue training dataset (Ghazvininejad et al., 2018; Wu et al., 2021; Xu et al., 2022). How- ever, these existing snowledge-enhanced dialogue generation approaches have several notable prob- lems: (1) limited knowledge types, (2) monotonous knowledge sources, and (3) complicated and ineffi- cient knowledge mining and fusion strategies. For example, incorporating external knowledge of spe- cific type into the training process is a common strategy, such as commonsense and emotion (Li et al., 2022), topic (Xu et al., 2021), persona (Yin et al., 2023), Wikipedia knowledge (Zhao et al., 2020), etc. Nevertheless, these approach often suf- fers from poor generalization due to constraints in the knowledge types and sources. Furthermore, the process of modeling knowledge for training or inference introduces udditional computational resource overhead. Another approach is to use search engines to generate responses based on in- ternet search results (Komeili et al., 2022). While this reduces training and inference costs, it fails to enable the model to fully understand the dialogue history as the knowledge is not stored internally.
To maximize the utilization of the rich knowl- edge stored in LLMs for dialogue genera- tion system, we propose a framework named KnowEE, which can conduct multi-source multi- type Knowledge Exploration from LLMs and then conduct knowledge Exploitation for better dialogue understanding and response generation. As men- tioned above, in our work we identify the dialogue knowledge into five types: Emotion, Topic, Per- sona, Event, and General World knowledge. Our framework KnowEE includes two phases: multi- source multi-type -ype knowledge exploration via in- contexl learning and fine/course- -raiined knowl- edge exploitation for response generation. For the first phase, given the dialogue context, we lever- age five external source datasets with knowledge labels corresponding to the above five types of knowledge, to extract the most relevant samples to the given context. These samples will serve as prompts to explore the LLMs to generate knowl- edge for the corresponding type. Next, we inject the generated knowledge into ongoing dialogue context using two different approaches: utterance- level fine-grained knowledge injection (FgKI) and dialogue-level coarse-grained knowledge injection (CgKI). The combined information then fed into LLMs to obtain the final response. We evaluated our proposed framework KnowEE on four dialogue generation tasks: EMPATHETICDIALOGUEE (ED), DAILYDIALOG (DD), PERSONA- CHAA (PC) and OPENDIAL.KG (ODKG), and the experimental sults demonstrate that it outperforms the baseline methods in terms of both automatic and manual evaluation metries.
In addition, several researchers have endeavored utilize large language models (LLMs) for di- alogue generation tasks. These models, such as GPT-4 (OpenAI, 2023), have cxhibited exceptional performance across various natural language pro- cessing tasks (Yasunaga et al., 2021; Zhang et al., 2023; Ni et al., 2023), and show two significant capabilities to (1) store and access various types of knowledge from extensive training sources (Zhao et al., 2023), and (2) perform different downstream tasks without fine-tuning. Intuitively, the knowl- edge storage and inference capacity as well as the text understanding and generation capabilities of LLMs can help generate better conversations, even for the zero-shot or few-shot scenarios. Neverthe- less, despite these superior capabilities, even the most advanced LLMs such ChatGPT, often en- counter challenges in dialogue generation tasks, as depicted in Figure Although ChatGPT "knows" that Lu Xun a pseudonym for Zhou Shuren, fails to accurately distinguish between the two persons when directly input the dialogue context into the
Our contributions can be summarized as follows:
We propose a framework named KnowEB that explores multi multi-source multi-type knowledge from large language models using diverse source datasets, and exploits the knowledge in fine- tne-grained and barse-grained manners for dialogue generation.
We leverage five extrenal datasets and extract the most relevant samples to the dialogue context, and prompt the LLMs to generate dialogue knowledge via in-context learning. Additionally, we propose a fine- ane-ggrined and a coarse knowledge injection approaches to

---

combine the generated knowledge with the dialogue context.
crucial challenge (Yu al., 2022). A common and popular strategy for further training LLMs on downstream tasks is fine-tuning (Sun et al., 2022). However, with the rapid growth of model scale, LLMs require an abundance of high-quality corpus and expensive computational resources for single fine-tuning, rendering them impractical to use. To address these challenges, novel techniques such prompt learning (Liu al., 2023) and prefix-tuning (Li and Liang, 2021) have been proposed. These techniques fine-tune only a few parameters instead of the whole, with the expectation of achieving comparable performance to fully fine-tumed mod- els. Besides, in-context learning (Brown et al.. 2020), which involves inputting exemplars related to downstream tasks to models rather than addi- tional training, has proven to be effective in few- shot settings (Chan al., 2022; Rubin al., 2022).
The automatic and manual evaluation on four datasets shows that our framework is superior to the strong zero-shot and few-shot baselines in terms of perplexity and diversity, and capa- ble of generating more fluent, coherent, and informative responses.
2
Related Work
2.1
Open-domain Dialogue Generation
The task aims to establish long-term connections and provide communication satisfying human need (Ghazarian et al., 2019; Pan et al., 2019; Huang al., 2020; Aliannejadi et al., 2021). Previous works often leverage neural network model and identify one type of dialogue, such as emotion (Li et al., 2020; Hu et al., 2021; Li et al., 2022), topic (Shi et al., 2016; Zhu et al., 2021; Xu et al., 2021) and persona (Liu et al.. 2020; Yin et al.. 2023). which have achieved impressive performance on corresponding datasets and arouse widespread in- terest. In addition to the widely used supervised learning, researchers introduce other effective algorithms or means, like Reinforcement Learning (Saleh et al., 2020) and Contrastive Learning (Cai et al., 2020). Besides, the general LLMs, such as GPT-3 (Brown et al., 2020), GLM (Zeng et al., 2022) and OPT (Zhang et al., 2022), are found to provide competitive responses with more effi- cient solutions (Zheng and Huang, 2021). These observations motivate some researchers explore the prompt engineering in LLMs for dialogue gen- eration. Yu al. (2022) proposes a knowledge- grounded dialogue system that equipped with the prompt-aware tuning-free LLMs exploitation and supported by the ready-to-use open-domain external knowledge bases and search engines. In contrast, our work focuses on wider variety of dialogue knowledge and leverages richer external knowledge resources.
Methodology
3.1 Overview
Figure 2 depicts the workflow of our framework KnowLE. The input comprises set of conversa- tional utterances, formally represented Ui {U1, Si...., U- 1 Ur}. Here, Us and S de- note the 7th utterances from different speakers. The objective of the dialogue generation task is to gen- erate a response S for the thh round of the dialogue. Recognizing the fact that humans utilize various types of knowledge from diverse sources to formu- late responses, adopt two-step generation pro- cess: multi-source multi-rype dialogue exploration and knowledge exploitation for response genera- tion. First, we employ external datasets to explore the LLMs to generate five types of knowledge rel- evant to the dialogue historical context: Emotion, Topic, Persona, Event, and General World Knowl- edge, denoted as Kemo, Kipe- Kpen- Keets and Kwor, respectively. Subsequently, we inject the generated knowledge into ongoing dialogue con- text and feed it into LLMs to generate high-quality responses. We also design two approaches with varying levels of granularity to optimize knowl- edge injection while adhering to the constraint of input length.
2.2 Large Language Models
Language models have significantly enhanced the performance of various NLP tasks owing to the ben- struc-
cfits of pre-training and transformer-bast ture (Jing and Xu, 2019; Qiu et al., 2020; Han al., 2021; Li et al., 2021; Ni et al., 2023). With the groundwork of L..Ms- algorithms, re- searchers now focus on unfolding the capabilities LLMs effectively and efficiently, which a
3.2 Multi-Source Multi-Type Knowledge Exploration
Given the dialogue historical context U4 the tar- get of knowledge exploration to produce all types of knowledge required for the response gen-

---

Dialogue System
000
Dialogue Historical Context
External Dulasets
Hellet Nice chat with you!
Knowledge Sources Dialogues or Utterances
He
Injecting Knowledge into Dialogue History
Coonient, Knonlelge
Emotion
Utterance- evel Knowledge Injection
Madi-turx Diabegur History
Select Most Relesant Sumples Prompie
Topic
Evant
Dialogue-Leve Knowledge Injection
Temphate Constructing Prompts
Persona
Knowledge Exploration
Knowledge
Exploitation
Do you kaow
Ves. Ido
Large-scale Language Model
The overall architecture of our proposed framework, KnowEE, The direction of the arrows represent the process of dialogue response generation. The same content is marked with the same color.
Figure 2:
eration procedure. We analyze numerous dia- logue examples and identify the dialogue knowl- edge into five types: Emotion, Topic, Persona, Event, and General World knowledge. For each type, we select one labelled dataset corre- spondingly: GOEMOTIONS (GE) (Demszky et al., 2020) for emotion information, DAILYDIALOG (DD) (Li et al., 2017) for topic information, PER- SONACHAT (PC) (Zhang al., 2018) for persona, ATOMIC (AT) (Sap et al., 2019) for event, and WIZARD OF WIKIPEDIA (WOW) (Dinan al., 2019) for general world knowledge. Since these datasets gather a significant amount of human- labeled data, we can view them as knowledge sources DDmoo Dipe, Dpsn: Dest, Dwor) contain- ing large number of content-knowledge pairs (C, K), where "content" refers to the textual data in the dataset, such as Reddit comments in GE or dialogue context in DD; "knowledge" represents the specific type of knowledge corresponding to the "content"", such as dialogue topics in DD or Wikipedia knowledge in WOW. Table gives the statistics of these datasets.
Hence, employ query- query-based sample selec- tion method, wherein we utilize either the entire dialogue context or a single utterance as a query Q. to sample content-knowledn -nooent- pairs (C, K)) from all the source datasets, based on the desired level knowledge granularity. Consequently, we obtain Qeeoo Qipes Qpen. Qent, @ corresponding to the above five knowledge sources. To ensure the relevance of the selected examples to the query, utilize SentenceTransformeee (ST) (Reimers and Gurevych, 2019) to encode the query and the pairs {(C}, Ki): (Ci, Ký),.... ...,. (C K;n) in D; (i E [emo, tpc, psn, eut, wor)). Subsequently, we calculate the cosine similarity between the query Qi and jth pair (j E m]):
S =
For each source dataset, we select samples with the highest similarity scores to the query and use them to create in-context prompts. Specifically, the prompt PA for the ,4hh sample (r [1, in D, is "[C T, Kj]", and the prompt P Piuerry for the query Qi is "[Q;; =]"1, where [:] denotes concatenation and T; represents the name of one of the five knowledge types, such as "emotion", "topic", etc., depends on the corresponding dataset D,. In order to indicate the current task, also include instruction I, with the template "Please try to understand the input dialogue history and generate corresponding T; knowledge."
Then how to acquire those five types of knowl- edge based on the five source datasets for dialogue context 24, Considering the powerful knowledge storage capacity of large language models, we propose a prompt-based approach via in-context learning (Brown et al., 2020) to effectively stimu- late and explore the rich knowledge stored within them. Our hypothesis posits that the selection of appropriate samples from source datasets as prompts paramount in producing high-quality knowledge labels. Intuitively, incorporating knowl- edge from similar dialogue contexts can assist mod- els generating relatively accurate knowledge.
Finally, we concatenate the instruction and con- structed prompts, and feed them into LLMs to gen- erate the dialogue knowledge K₃;
K; - LCM(M, Pl:P pi. queryl)
For example: wanted to downvote this, hut it's not your fault, 
 Emotion = Disappointment

---

Type
Dataset
Num
Avg. CT
Avg. KT
Gran.
Dialogue-level Coarse-Grained Knowledge In- jection We hypothesize that preceding sentences in the dialogue history have lesser impact on re- sponse generation, as speakers tend to focus more on the most recent information. Therefore, in this approach, we specifically select the emotion, event, and general world knowledge from the last round to construct the background knowledge BK", de- noted "[BK; Komos Ke " We combine BK' with the dialogue historical context 2. Other settings, including conjunctions and instructions, remain consistent with the aforementioned method. Therefore, the model St output at this time is:
Emotion
GE
211K
20.1
1.0
Fine
Topic
DB
10K
144.2
1.4
Coarse
Persona
PC
17K
117.9
43.2
Coarse
Event
AT
24K
8.9
5.4
Fine
World Klg.
WOW
166K
22.9
35.9
Fine
Table Statisties of datasets corresponding to five types of knowledge. Each content-knowiedd pair is viewed as one data sample. Avg. CT and Avg. KT represent average content tokens and average knowledge tokens. Gran represents the granularity of textual data in the dataset and query from dialogue history.
Thus, we obtain K Kenor Kipes Kysns Keet, and Kwor, which is related to dialogue historical con- text.
S: = (Zg: BKC; S1: .. Up 1: S; 1: Ut))
3.3 Knowledge Exploitation for Response Generation
Experimental Settings
We inject the generated five type of knowledge into dialogue historical context for better response generation. The integration of knowledge with the dialogue context is critical factor, particularly considering the limited input length of language models. To address this, we adopt two approaches: utterance-level fine-grained knowledge injection (FgKI) for emotion, event, and world knowledge: dialogue-level caasee-raiied knowledge injection (Ig for topic and persona.
4.1 Datasets
Since most of the current dialogue datasets fo- cus
or two kinds of dialogue knowledge, variety of different types of datasets: EMPATHETICDIALOGUSS (ED) (Rashkin et al., 2019), DAILYDIALOG (DD) (Li et al., 2017), PERSONACHAT (PC) (Zhang et al., 2018), and OPENDIAL.KG (ODKG) (Moon et al., 2019). The datasets used for the external knowl- edge sources and experiments share some common parts. However, please NOTE that: 1) the datasets use for experiments do not contain any informa- tion besides the dialogue context; 2) the original train/dev/test split follows the original dataset, and the training set is utilized as external knowledge source, while the test set is utilized for experiments.
conduct experiments
Utterance-Level Fine-Grained Knowledge In- jection We regard emotion, event, and general world knowledge as utterance-level fine-grained knowledge. And we use topic and persona knowl- edge dialogue-level coarse grained background knowledge BK, denoted "[Kepei Kpsn]". We integrate emotion, event, and general world knowl- edge into the corresponding utterances of the dialogue. Hence, for the 2thh utterance (a E (1, the formulation of Uh is represented as "[Kžmo: K K : Uz]"" Each knowledge type is associated with a specific conjunction to prevent confusion. To connect different rounds in the di- alogue history, use the labels "System:" and "User:". Additionally, we provide an instruction Ig for dialogue generation: "Please consider the dialogue history. encompassing various types of knowledge such as emotion, topic, event, persona, and general world knowledge, to generate a re- sponse that exhibits rich diversity and coherency." We only apply this operation to the last two rounds of the dialogue. Therefore, the model output St for this approach is:
4.2 Baseline Methods
General Large Language Models: OPT (Zhang et al., 2022) is a suite of decoder-only pre-trained transformers with varying numbers of parameters, ranging from 125M to 175B. We use the 13B- parameter model of OPT. ChatGLM-6B (Zeng et al., 2022) is an open bilingual language model based on General Language Model (GLM) frame- work, with 6.2 billion parameters. ChatGLM uses technology similar to ChatGPT, and is optimized for Chinese QA and dialogue. Flan-T5-XXL, is the backbone model of KnowEF. To demonstrate the effectiveness of our approach in enhancing re- sponse quality, we utilize it for direct generation.
Pre-trained Dialogue Models: DialoGPT (Zhang et al., 2019) is finc-tuned GPT-2 (Radford et al., 2019) with Reddit comment data. We select
S, = CCC UU

---

ED
PC
ODKG
Method
PPLL
D-1
D-2
PPL1
D-1
D-2
PPLL
D-1
D-2
PPL1
D-1
D-2
OPT-13B
34.86
9.01
46.16
13.39
50.02
24.31
10.12
15.66
49.47
ChatGLM-66 66
30.12
3.85
23.61
19.30
8.10
38.77
22.92
4.21
22.52
33.75
8.87
32.42
Flan-T5-XXL
39.53
9.56
46.25
38,42
16.74
58.58
24.00
13.97
53.30
15.89
16.01
51.52
DialoGPT
21.54
6.69
24.03
17.91
10.84
34.56
27.71
9.54
30.91
16.01
25.05
BlenderBot
20.12
7.16
38.73
19.35
50.78
17.78
12.06
50.31
21.60
12.83
47.16
FWP
22.43
5.38
17.83
26.73
10.25
31.95
22.79
5.99
18.32
22.51
4.65
12.32
FSB
22.83
4.30
18.02
18.65
13.63
44.29
38.40
10.18
34.55
22.69
13.36
35.43
MSDP
24.41
20.71
63.69
KnowHE-FgKI
24.62
10.06
48.37
16.11
16.96
59.53
22.57
14.24
55.60
14.25
17.88
57.33
KnowHE-Cgki
19.04
11.95
52.13
16.68
17.48
61.36
22.50
14.13
54.74
15.83
17.74
56.18
Table 2: Automatic evaluation results of different methods and models four datasets. The bold numbers in the results represent the best scores, whereas the underlined numbers indicate the second- scores.
the 345M (best performance) for comparison. Blenderbot (Shuster al., 2022) is a pre-trained conversational model which combines multiple models and techniques including GPT-2, BERT (Devlin et al., 2018), and Seq2Seq. We select Blenderbot-3B for comparison.
unigrams/ / bigrams in the generated outputs, indi- cating diversity. However, ensure the integrity of experimental results, still represent the BLEU and ROUGE scores of various baselines on the ED dataset in Appendix
We conduct a human eval- uation on open-domain dialogue generation. We recruit university students to evaluate the quality of conversations. Considering the costs associated with human evaluation, 500 samples have been in- cluded for each baseline on each dataset. We follow up previous dialogue generation efforts (Yu et al., 2022; Li et al., 2022) and employ several general metrics to evaluate the dialogue quality Coher- ence measures relevance to the dialogue context, Informativeness evaluates information provided, and Fluency checks grammatical accuracy.
Human Evaluation
Few-shot Learning Methods: FWP (Zheng and Huang, 2021) is approach to improve the dia- logue generation task by learning continuous to- kens to query the language model more efficiently. FSB (Madotto al., 2021) chatbot which au- tomatically selects the most appropriate conversa- tional skill, queries different knowledge bases and uses the retrieved knowledge to generate a human- like response, all using only few dialogue examples. MSDP (Liu et al., 2022) is a few-shot framework that extracts general world knowledge from doc- uments and employs it generate responses via in-context learning. In MSDP, knowledge are ob- tained from Wikipedia (WoW) or Internet (Wol), which corresponds to general world knowledge in our work. Therefore, we only compare the perfor- mance of MSDP on the ODKG dataset.
Additionally, we use different metric for differ- dataset. For ED, we evaluate Empathy, measur- ing the match between the generated response and the speaker's emotion. For PC, we assess Personal- ity consistency. For DD, we determine Theme ad- herence. For ODKG, we check for Hallucination.) and factual errors. Note that the Coherence, Infor- mativeness, and Fluency scale is [0, 1, 3, 4), and Empathy, Personality and Theme scale is [o, 1, 2). whose higher score indicates a better performance. Moreover, the scale of Hallucination is [0, 1, 2]. whose lower score indicates a better performance.
4.3 Evaluation Metrics
Automatic Metrics Previous empirical studies reveal a significant gap between automatic metrics (e.g., BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004)) and human judgments in evaluating di- alogue generation (Liu et al., 2016). Nevertheless, there are reference-free metrics available that par- tially capture the quality of generated dialogues. Therefore, utilize Perplexity (Jelinek et al., 1977) and DISTINCT (Li et al., 2016a) evaluate the models. Perplexity (PPI.) assesses the overall quality the generation model, while DISTINCT (D-1 / D-2) measures the proportion of distinct
4.4 Implementation Details
We select Flan-T5-XXL (13B) backbonc model. Flan-T5 (Chung al., 2022) is fine- tuned version model class of T5 (Raffel al., 2019) that has been trained on variety of datasets phrased as instructions. It has shown impres- sive performance on several benchmarks, demon-

---

DD
ODKG
Method
Cobe.
Info.
Flu.
Emp.
Cohe.
Info.
Flu.
The.
Cohe.
Info.
Flu.
Hall.
OPT-13B
1.56
1.94
1.13
0.69
1.34
1.07
1.11
0.56
2.01
1.42
1.69
1.03
ChatGL.M-6B
2.11
2.08
2.14
1.02
2.49
2.61
2.09
1.23
1.38
1.50
1.33
1.28
Flan-TS-XXI
1.96
1.87
2.39
0.91
2.89
2.78
3.13
1.01
2.25
2.14
2.48
1.29
DialoGPT
1.92
1.19
1.81
0.77
2.20
1.79
2.39
1.28
2.73
2.27
2.76
BlenderBot
2.36
2.27
2.94
0.93
0.95
2.27
2,01
0.73
1.87
2.65
2,87
0.99
FWP
0.81
1.31
2.11
0.78
1.02
0.52
1.82
0.86
1.16
1.62
0.50
1.23
FSB
2.14
2.05
2.38
1.33
2.12
2.91
2.87
1.04
2.67
2.33
2.28
1.01
MSDP
2.92
3.01
1.54
0.88
KnowEE-FgKT 2.57
2.86
2.71
1.43
3.09
3.14
2.63
1.41
3.09
2.83
2,43
0.96
KnowEE- CE
2.67
2.75
2.74
1.57
3.19
2.94
3.59
1.44
3.01
3.13
2.74
0.83
Table 3: Human evaluation results different methods and models on ED, DD and ODKG, where Cohe., Info.. Flu., Emp., The., and H are the abbreviations corresponding to Coherence, Informativeness, Fluency, Empathy, Theme and Hallucination.
Method
PPL.
D-2
Cohe,
Info.
Flu.
Emp.
performance. As KnowEE capable of securing the Top positions all automatic metric scores across all datasets. This shows that our method can achieve better diversity while ensuring lower per- plexity.
KnowLE
19.04
52.13
2.67
2.75
2.74
1.57
w/o Emo
19.25
45.23
2.56
2.31
2.68
1.05
w/o Tpe
18.67
45.28
2.51
2.33
2.69
1.35
w/o Psn
19.34
45.10
2.60
2.62
2.73
1,36
w/o Evt
18.53
44.35
2.59
2.66
2.62
1.39
w/o Wor
18.10
42.07
2.61
2.68
2.72
1.46
Human Evaluation Results The results of all models or methods for ED, DD, and ODKG are presented in Table 3. For brevity, the remaining results can be found in Appendix B. These tables re- veal that KnowEE achieves prominent position in human evaluation scores, indicating the following trends:
Ablation study of our proposed framework. The abbreviation w/o is used to indicate that the LLM generates responses without using the corresponding type of knowledge.
Table 4:
strating strong zero-shot, few-shot, and Chain-of- Thought (CoT) (Wei et al., 2022) abilities. Flan- T5-XXL is the largest released checkpoint of this model, boasting parameter volume of 13B. We leave LLaMA (Touvron et al., 2023) based models such as Alpaca (Taori et al., 2023) and Vieuna (Chi- ang et al., 2023) in the future research work.
First, KnowEE outperforms existing approaches in human evaluation metrics, showcasing competi- tive performance in coherence, fluency, and infor- mativeness. The results emphasize the effective- ness of leveraging LLMs for open-domain dialogue tasks via pre-response knowledge generation using well-designed prompt patterns.
Second, compared to large-scale dialogue mod- els like ChatGLM and pre-trained dialogue models such as DialoGPT, KnowEE exhibits superior herence and informativeness, highlighting the effec- tiveness of its knowledge generation and injection mechanism. Additionally, dialogue models with specific architectures and adequate training con- sistently excel in terms of fluency, supporting the benefits of pre-trained dialogue models in generat- ing dialogue responses, as confirmed by empirical analysis of BlenderBot and DialoGPT.
5 Results and Analysis
5.1 Open-domain Dialogue Generation
We conduct automatic and human evaluations to compare our proposed framework, KnowEE, with the baselines mentioned earlier, using four datasets.
As shown in Ta- ble 2, the experimental results show that KnowEE overall outperforms the baselines in perplexity and diversity across all datasets. Only Blenderbot slightly outperforms our framework with different knowledge injection method in perplexity for ED and PC. Besides, MSDP outperforms our frame- work in terms of diversity for ODKG. However, KnowEE surpasses all other baselines in overall
Automatic Evaluation Results
Besides general human evaluation metrics, our main focus on whether the model incorporates implicit knowledge, such as emotions and themes, from the dialogue history during generation. To achieve this, we develop specific metrics (Empathy,

---

ED
ODKG
Method
PPL.
D-2
Cohe.
Info.
Emp.
PPL.
D-2
Cohe.
Info.
Flu.
Hall
ChatGLM
14.13
6.03
19.34
2.11
2.08
2.14
1.02
15.98
8.89
33.21
1.38
1.50
1.33
1.28
w FgK1
21.85
6.74
27.52
2.09
2.34
2.22
1.16
16.02
5.82
20.53
2.25
2.94
1.54
0.65
W CgK1
13.16
7.47
31.48
2.26
2.18
2.14
1.29
14.49
13.89
43.48
2.20
3.18
1.88
0.89
ChatGPT
16.43
16.85
50.04
3.51
3,50
4.00
1.63
20.38
22.08
50.58
3.57
3.69
4,00
0.44
FgK1
10.63
19.13
58.48
3.63
3,25
4.00
1.66
13.96
26.56
60.90
3.57
3.80
4,00
0.43
CgKi
10.37
19.15
61.02
3.65
3.60
4.00
1,68
13.86
26.46
62.23
3.69
3.79
4.00
0.40
D)
PC
Method
PPL.
D-1
D-2
Cobe.
Info.
Flu.
The.
PPI.
D3-1
D-2
Cohe.
Info.
Flu.
Per.
ChatGLM
19.30
8.10
38.77
2.49
2.61
2.09
1,23
22.92
4.21
22.52
2.11
2.71
1.29
1.30
PgK1
20.51
5.13
20.58
2.63
2.71
2.11
1.33
24.79
4.75
20.37
1.38
2.11
0.94
0.71
CgK1
19.09
10.84
40.50
2.69
3.01
2.60
1.39
15.61
7.34
32.21
2.94
2.90
1.50
1.31
ChatGPT
20,30
19.37
49.70
3.86
3.72
4.00
1.57
18.74
48.69
3.17
4.00
1.22
FgK1
4.00
1.90
61.32
3.19
CgKI
63.49
3.90
4.00
1.93
64.57
3.23
4.00
1.38
Table Generalization ability study of our proposed framework. FgKl and CgKI represent LLM generates responses using KnowEE-FgKl and KnowEB-CgKI respectively.
Theme, Hallucination, and Personality) for each dataset. Table presents the results. Surprisingly, models like BlenderBot, which excel coherence, informativeness, and fluency, struggle with under- standing dialogue knowledge accurately, leading to lower scores relevant metrics. Case analysis shows that they often deviate from the conversation theme, character personality, or express incorrect emotions, occasionally containing factual errors. contrast, our framework consistently ranks high- est across all relevant indicators for all datasets. Notably, KnowEE with coarse-grained injection generally outperforms fine-graine fine-grained method, possi- bly due to the superior comprehension abilities of F5an-T5-XX XXXXXL in processing input.
generation. Ablation experiment on ED is con- ducted, removing one knowledge type at a time (w/o Emo, w/o Tpc, w/o Psn, w/o Evt, w/o Wor). Evaluation results in Table 4 reveal that removing any knowledge type reduces perplexity, diversity, relevance, fluency, informativeness, and emotional matching in responses. Emotional knowledge (w/o Emo) has the greatest impact, indicating its sig- nificant role understanding dialogue history and affecting response quality. Results and analysis demonstrate that the dialogue knowledge obtained from LLMs contributes significantly to generating responses.
5.3 Generalization Ability Analysis
Last but not least, compared to MSDP, our ap- proach excels in terms of coherence and informa- tiveness. Besides, MSDP even scores lower in fluency compared to Flan-T5-XXL in zero-shot scenarios. Through case analysis, we find that Flan- T5-XXL with MSDP has a probability of copying content from generated knowledge during text gen- eration, which significantly reduces the fluency of results. In contrast, our method considers various types of knowledge during text generation, which avoids the scenario where the model is overly influ- enced by single type of knowledge, and enables to mitigate this issue.
We compare the performance of KnowEE with different backbone models evaluate the gen- eralization ability our framework. We select two instruction-tune -uunedd models with larger scale and parameters, specifically ChatGPT and ChatGLM. ChatGPT not open-source and can only be ac- cessed through API from OpenAl 2 for inference. From Table 5, observe that although Chat- GPT and ChatGLM exhibit strong dialogue sponse generation ability, the scores for all auto- matic and human evaluation metrics of responses generated by these models using KnowEE frame- work are consistently higher or equal to those di- rectly using them for inference. This demonstrates that our proposed framework is still effective for models with larger scale, stronger knowledge
5.2 Ablation Study
The aim of this study is to investigate the impact different dialogue knowledge types on response
Phtps://opeaai.com/

---

serve, and understanding ability, and is agnostic the type of base model.
responses. The interaction and assistance process do not involve any bias towards the participants. All datasets used in this work are from previously published works, and in our view, do not have any attached privacy or ethical issues.
5.4 Case Analysis
We conduct a case study to further prove the advan- tages of our proposed framework. We select several test generation results from the four datasets, with the expectation of observing improvements in the performance of LLMs in dialogue response tasks with the use of our framework. We present details in Appendix C due space limitation.
Acknowledgements
This research is supported by the National Natural Science Foundation of China (No.62100) 106105, No.62272274), the CCF-Baidu Open Fund the CCF-Tencent
(No. NVo.CCF-Baidu2023077, Open Research Fund NN..RAGR20200122) the CCF-Zhipu AI Large Model Fund (No.CCF- Zhipu202315), the Fundamental Research Funds for the Central Universities (No.N12023332), the Scientific Research Starting Foundation of Nanjing University of Aeronautics and Astronautics (No. YQR21022), and the High Performance Computing Platform of Nanjing University of Aeronautics and Astronautics.
6 Conclusion
In this paper, we study the task of open- open-domain knowledge generation and identifies challenges of lacking various types of knowledge from diverse sources. To address these challenges, we identify five key type of dialogue knowledge, and propose a framework called KnowEE that explores multi- source multi-type knowledge from large language models using external datasets, and exploits the knowledge in fine-grained and coarse-grained man- ners for response generation. It explores knowledge from LLMs and injects knowledge into dialogue context generate final responses. Experiments four dialogue datasets show that KnowEE enhances LLM's understanding dialogue context and im- proves generated responses terms of coherence, informativeness, and fluency. Moreover, KnowEE exhibits robust generalization capabilities, making it applicable to multiple LLMs.
References
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue cor- pora with clarifying questions. In Proceedingy of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021. pages 4473 4484. Association for Computa- tional Linguistics.
In our future endeavors, we intend to refine the dialogue knowledge further and explore the tapped potential of large language models in tasks that extend beyond dialogue response.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models few-shot learners. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020. virtual.
Limitations
The limitations of our framework mainly come from the disadvantages of using large language models. First of all, most of the large language models that work well not open source free. This makes difficult to conduct batch experiments or daily use on Next, a small number of open- source models require a lot of GPU resources when used, which is a difficult problem for quite many researchers, such as students.
Hengyi Cai, Hongshen Chen, Yonghao Song, Zhuoye Ding. Yongjun Bao, Weipeng Yan, and Xiaofang Zhao. 2020. Group-wise contrastive learning for neural dialogue generation. In Findings the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume
Ethics Statement
We honor and support the ACL code of Ethics. Our proposed framework KnowEE aims to generate friendly, high-quality, informative, and coherent
EMNLP 2020 Findings ACL, pages 793-802. Association for Computational Linguistics.

---

Stephanic C. Chan, Adam Santoro. Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. 2022. Data distributional properties drive emer- gent in-context learning in transformers. CoRR, abs/2205.05055.
Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song. Jie Tang, Jing Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. AI Open, 2:225-250.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. Dialoguecrn: Contextual reasoning networks for emotion recognition in conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACI/IJCNIP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7042-7052. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction anstrcction-iiettueed finetuned language models. CoRR, abs/2210.11441
Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain di- alog systems. ACM Trans. Inf. Syst.. 38(3):21:1- 21:32.
Fred Jelinek, Robert L. Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity- measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):5633566
Dorottya Demszky, Dana Movshovitz-Atte Attias, Jeongwoo Ko, Alan S. Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine- mna-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5- 10, 2020, pages 4054. As- sociation for Computational Linguistics.
Kun Jing and Jungang Xu. 2019. A survey on neural network language models. CoRR, (bs//900.03599..
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmenteed dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume T: Long Papers), ACI. 2022, Dublin, Treland, May 22-27, 2022, pages 8460 8478. Association for Com- putational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arviv preprint
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: snowledge-powereeg conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA,, USA, May 6-9, 2019. OpenReview.net.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A primosig objective function for neural conversation mod- els. In NAACL HIT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego California, USA, June 12-17, 2016, pages 119. The Association for Computational Linguistics.
Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Gal- styan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. CoRR, 001194440035.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016b. persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Lin- guistics.
Marjan Ghazvininejad, Chris Brockett, Ming-Wi Weig-Wei Chang. Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge- knowledge-rrounded grounded neural conversation model. In Proceedings of Artificial the Thirty-Second AAAI Conference Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (18. and the Sth AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18) New Orleans, Louisiana, USA, February 2-7, 2018, pages 5110-5117. AAAI Press.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. 2021, Pretrained language models for text gen- eration: A survey. CoRR, abs/2105.10311.
Qintong Li, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. 2020. Em- pdg: Multi-resolution interactive empathetic dialogue
Xu Han, Zhengyan Zhang, Ning Ding. Yuxian Gu, Xiao Liu, Yugi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang,

---

generation. In Proceedings the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4454-4466. International Committee on Computational Linguistics.
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Multi-stage prompting for knowledgeable dialogue generation. arXiv preprint arXiv:2203008745.
Qintong Li, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. 2022. Knowledge bridging for empathetic dialogue generation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAT 2022, Thirty-Fourth Conference on Innovative Applications Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Eddcaaiiona Advances in Artificial Intelligence. EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 11999-11000 AAAI Press.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. CoRR, abs/2110.08118.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. Opendialkg: Explainable con- versational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence. Italy, July 28- August 2019, Volume Long Papers, pages 845- 854. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix- tuning: Optimizing continuous prompts for gener- ation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582 4597. Association for Computational Linguisties.
Xuanfan Ni, Piji Li, and Huayang Li. 2023. Unified text structuralization with instruction-tuned language models. CoRR, 0bs/23003119555
OpenAL. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
7hu Feng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn in- complete utterance restoration. In Proceedings of the 2019 Conference on Empiriaa Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-UCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1824-1833, As- sociation for Computational Linguistics.
Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li, Ziqiang
Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December I, 2017 Volume
1: Long Papers, pages 986 995. Asian Federation of Natural Language Processing.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. July 6-12, 2002, Philadelphia, PA, USA, pages 31 1-18.. ACI.
Chia- Liu, Ryan Lowe, Iulian Serban, Michael D. Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue sys- tem: An empirical study of unsupervised evalua- tion metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2122-2132. The Association for Computa- tional Linguistics.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 195:
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text trans- former. CoRR, abs/1910. 10683.
Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang 2020. You impress me: Dialogue generation via mutual persona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10. 2020, pages 1417-1427, Association for Com- putational Linguistics.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Conference of the Association for Computational Linguistics.

---

ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5370-5381 Associa- tion for Computational Linguisties.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju. Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns respon- sibly engage. CoRR, abs/2208.03188
Nils Reimers and Tryna Gurevych, 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural SAILL--JCNPPP0 Hong Kong, China, November 3-7, 2019, pages 3980 3990. Association for Computational Linguistics.
Language Processing,
Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, and Xuan- Jing Huang. 2022. Paradigm shift natural language processing. Int. Autom. Comput. 19(3):169-11)
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason We- open-domain
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang and Tatsunori B. Hashimoto. 2023. Stan-
ston. 2021. Recipes for building chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, FACL 2021, Online, April 19 - 23, 2021, pages 300-325. Association for Computational Linguistics.
ford alpaca: An instruction-follone model. https:/ /tta:///giti com/tttttmomm/tttuuuu stanford_alpace
llama
cwm/tatsu lab/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR. 1ts/2332..1971
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational I. Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022. pages 2655-2671. Asso- ciation for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Lc, and Denny Zhou. 2022, Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201. 1903.
Abdelrhman Saleh. Natasha Jaques, Asma Ghande- harioun, Judy Hanwen Shen, and Rosalind W. Pi- card. 2020. Hierarchical reinforcement learning for open-domain dialog. In The Thirty-Fouth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence. EAAI 2020. New York. NY, USA. February 7-12. 2020, pages 8741-8748. AAAI Press.
Sixing Wu, Ying Li, Minghui Wang. Dawei Zhang. Yang Zhou, and Zhonghai Wu. 2021. More is better: Enhancing open-domain dialogue genera- tion via multi-source heterogeneous knowledge. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021. Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 2286-2300. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: an atlas of machine common- sense for if-then reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence. AAAI 2019, The Thirty-Firr Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Jamuary 27 February I, 2019, pages 2027-3035. AAAI Press.
Sixing Wu, Minghui Wang. Ying Li, Dawei Zhang. and Zhonghai Wu. 2022. Improving the applicabil- ity of knowledge-enhaneee dialogue generation sys- tems by using heterogeneous knowledge from mul- tiple sources. In WSDM '22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 25, 2022, pages 1149-11551 ACM.
Chen Xu, Piji Li, Wei Wang. Haoran Yang, Siyun Wang, and Chuangbai Xiao. 2022. COSPLAY: con- cept set guided personalized dialogue generation across both party personas. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July lI 15, 2022, pages 201-211. ACM.
Hongjie Shi, Takashi Ushio, Mitsuru Endo, Katsuyoshi Yamagami, and Noriaki Horii. 2016. Convolutional neural networks for multi-topic dialog state track- ing, In Dialogues with Social Robots- Enablements, Analyses. and Evaluation. Seventh International Workshop on Spoken Dialogue Systems, IWSDS 2016, Saariselkä, Finland, January 13-16, 2016, vol- ume 427 of Lecture Notes in Electrical Engineering. pages 45 1-463. Springer.
Yi Xu, Hai Zhao, and Zhuosheng Zhang. 2021. Topic- aware multi-turn dialogue modeling. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021. Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021,

---

The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 14176-14184 AAAT Press.
Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang. and Jure Leskovec. 2021. QA-GNN: reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11. 2021, pages 535-546. As- sociation for Computational Linguistics.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge- grounded dialogue generation with pre-trained lan- guage models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3377-3390, Associ- ation for Computational Linguistics.
Congehi Yin, Piji Li, and Zhaochun Ren. 2023, Ctrl- struct: Dialogue structure learning for open-domain response generation. CoRR, abs/2303.01094.
Chujie Zheng and Minlie Huang. 2021. Exploring prompt-based few-shot learning for grounded dialog generation. CoRR, abs/2109.065/ 13.
Jifan Yu. Xiaohan Zhang. Yifan Xu, Xuanyu Lei. Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang. 2022. In KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 18, 2022, pages 4422 4432. ACM. [link].
Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-driven and knowledge- aware transformer for dialogue emotion detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IICNIP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 1571-1582. Association for Computational Linguisties.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai. Ming Ding. Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang. Yuxiao Dong, and Jie Tang. 2022. GLM- 130B: an open bilingual pre-trained model. CoRR. abs/2210.02414.
A Automatic Evaluation Results of BLEU and ROUGE
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine transla- tion: A case study. CoRR, hts/2301.07069
The metrics based on N-gram content match, such as BLEU and ROUGE have already gained con- sensus as inappropriate for evaluating text genera- tion, particularly dialogue generation. However, In order to further illustrate the effectiveness of our framework, we employ these metrics to evaluate certain methods from Table 2 and Table 3 that ob- tain higher scores in both human and automatic evaluations. Taking the ED dataset as an exam- ple, the BLEU (BLEU-1, 2, 3, 4) and ROUGE (ROUGE-1, 1, 2, L) scores for various methods are presented in Table 6.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACI 2018, Melbourne, Australia, July 15-20, 2018. Volume Long Papers, pages 2204 2213. Association for Computational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab. Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ou, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01068.
The results indicate that even considering only the n-grams match of response content, all met- ric scores of KnowEE-FgKi and KnowEE-CgKI exceed all baselines. This further supports the con- clusion drawn from our previous experimental anal- ysis, which is the effectiveness our framework.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing
Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. CoRR, abs/1911,00536.
B Human Evaluation Results of DD
As shown in Table 7, our proposed framework can achieve an overall leading position, compared strong baselines on DD, and coarse searse-grained method has a slight advantage over fine -nniindd method.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du,

---

Method
BLEU-1
BLEU-2
BLEU-3
BLEU-
ROUGE-1
ROUGE-2
ROUGE-L
ChatGLM-6B
5.45
1.69
0.63
0.25
12.98
1.27
10.34
Flan-TS-XXL
7.61
2.92
1.30
0.62
11.58
1.77
10.31
DialoGPT
5.16
2.07
0.99
0.54
13.23
2.01
12.31
BlenderBot
7.51
2.03
0.71
0.30
13.43
1.35
10.98
FSB
7.24
2,15
0.92
0.49
12.75
1.43
11.39
SnowEE-FgKt
10.78
3.81
1.68
0.93
12.46
1.66
10.73
KnowEE-CgkI
11.84
4.40
2.13
1.27
13,40
2.02
11.47
Table BLEU and ROUGE results of KnowEB and several strong baselines on ED.
Models
Cohe.
Info.
Flu.
Per.
knowledge about the user's intention, the model combines the dialogue history and dialogue knowl- edge. producing different results. Notably, Flan- T5-XXL with the coarse-grained method provided the appropriate book recommendation.
OPT-13B
1.37
1.41
1.02
1.05
ChatGLM-6B
2.11
2.71
1.29
1.30
Flan-T5-XXL.
2.53
2.56
2.77
1.21
DialoGPT
2.03
2.16
2.24
1.22
BlenderBot
2.03
2.90
2.96
0.63
As shown in Table 9. In the first case DD, the user inquires about the price of map. Flan-T5- XXL fails to comprehend the dialogue and provides a meaningless generic response. However, by ex- plicitly incorporating event and topic knowledge to discern the user's s intent, our framework generates relevant responses. Among them, only the response from KnowEE-FgKI comes closest to the ground truth, whereas the response from KnowEli-CgKi deemed unreasonable. In the second case of PC, the user assumes the role of the author, while the system acts as a secretary. The response generated by M5---T5XXLL. is unhelpful and highly irrelevant. In contrast, our framework is capable of producing responses that align with the user's personality and are contextually relevant.
FWP
0.53
1,86
1.94
0.53
FSB
2.27
2.93
2.79
1.30
KnowEE-Pggi
2.36
3.01
1.53
KnowEE-CgKI
2.41
2.99
3.27
1.56
Table 7: Human evaluation results of DD, where Per. are the abbreviations corresponding Personality.
These results match our previous analysis in section 5.1.
Case Analysis
We conduct case study to further prove the advan- tages of our proposed framework. We select several test examples from four datusets and use Flan-T5- XXL (the backbone model of KnowÉl)) KnowEE- FgK and KnowEF-CgKI for response generation, with the expectation of observing improvements in the performance LLMs in dialogue response tasks with the use of our framework.
The results and analysis of the case study demon- strate the importance of injecting dialogue knowl- edge into the input of L.LMs before generating sponses. Our proposed framework effectively stim- ulates the storage knowledge of LLMs about dia- logue history, resulting in higher-quality dialogue responses.
As shown in Table 8, in the first case, the dia- logue history involves the user's experience of visit- ing an orphan and learning from the kid. However, Flan-T5-XXL. fails to recognize the user's emotions and gives a bland response. While with the help of pre-generated emotion and event knowledge, the model finally produces response expressing won- der and curiosity about the user's surreal moment, which dirccts the user to continue the conversation. Similarly, in the second case, the dialogue history focuses on romance novel, and the user secking recommendations,
1onn-T5XXIL, however, recom- mends a historical novel. By generating general world knowledge about romance novels and event

---

Empathetic Dialogues
User: I visited an orphan once and it was such surreal moment. It made
realise how thankful should be for family. (Thankful)
Dialogue History
System: That must have been and intense and emotional moment for you.
User: Yes it definitely was. went with intentions to teach the kid a few things,
but I ended up learning lots more from him. (Grateful)
Key Knowledge
Emotion: grateful. Event: The user described surreal moment; The ended learning lots more from the kids.
Ground Truth
What surreal moment. always believe kids have lot to teach us.
Flan-T5-XXL
Well thats what happens lot of times. (Neutral)
KnowEE-FgKI
Amazing! How old was he when you met him? (Wonder and curiosity)
KnowEE-CgKi
What an amazing experience that must have been. (Wonder and curiosity)
Open Dialog KG
User: I like the Author Jane Austen. Do you have any recommendations of books like hers?
System: Sense and Sensibility
Dialogue History
User: Can you tell me more about that book? What genre it?
System: Its a Romance novel
User: Oh! Sounds great. like Romance novels. Can you recommend any others?
Key Knowledge
General World: Romance novel is fictional genre of book centered around love story. Event: The user wants get some recommendations.
Ground Truth
From prada to nada
Flan-T5-XXL
The Kingmaker's Daughter is another one can recommend. (Historical novel)
KnowEF-FFgK
Three Men in Boat, I'd say that's another good one. (Humor novel)
KnowEE-CgKI
have book in my hands right now. It is called Twilight. (Romance novel)
Table Two cases of generated responses by Flan-T5- 1onnTT--XXLL and our framework. In the case of ED, we annotate the user's utterances with emotional labels. In the case of ODKG, we bold the key words that decisive for generating the response. Specifically, we highlight words related to dialogue knowledge in blue and words relevant to the predicted content in red.

---

DailyDialog
User: Do you have maps of downtown area?
Dialogue History
System: Yes, here you are.
User: How much is it?
Key Knowledge
Topic: Ask for directions. Event: The user asks for maps of downtown area.;
The user wants to buy something.
Ground Truth
It's free of charge. (Free gift)
Flan-T5-XXL
Thank you.
KnowF.FFFgKT
It's ninery dollars. (Charge ninety dollars)
KnowEE-CgKI
It's free, that ok? (Free gift)
Persona Chat
User: What do you do for a living?
Dialogue History
System: I am a secretary like my mom hard living, and you?
User: am an author. write short stories.
Key Knowledge
Persona: The user is an author; The system is an secretary.
Ground Truth
Do you want to write a story about major meat cating bodybuilder?
Flan-T5-XXL
is hard living, its very lonely.
Know EnogEEFFFKI
write short stories too. Do you have an agent?
KnowFF-CCgKT
write short stories too. Do you mind if get an agent to represent me?
Table 9: Two cases of generated responses by Flan-TS-XXL and our framework. We bold the key words that are decisive for generating the response. We highlight words related to dialogue knowledge in blue and words relevant to the predicted content red.


Reviews:
[{"review_id": "Aib5isLv2K", "reviewer": "Reviewer_Q3qA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our paper. Your insights are of utmost importance to us, and we would like to provide comprehensive explanations addressing the concerns you have raised.\n\n**Question 1: This proposed method seems to be the multi-source adoption of an existing method MSDP.**\n\nWe have thoroughly read the paper you mentioned: \"MSDP\"[1]. In this work, although a similar framework of generating knowledge first and then generating responses is employed, there exist distinctions between our approach and theirs:\n\n- In their experiments, knowledge are obtained from Wikipedia (WoW) or Internet (WoI), which corresponds to general world knowledge in our work. Conversely, our work delves into the more types of dialogue knowledge from diverse sources, such as topic, emotion, etc. The applicability of our framework is notably more extensive, as it can be employed in real-world open-domain dialogue systems due to its rich knowledge sources and types. By inputting dialogue history, our method explores and utilizes LLMs' knowledge to generate high-quality responses. In contrast, MSDP is confined to question answering and dialogues rooted primarily in sources like Wikipedia, lacking the capability to comprehensively capture various types of knowledge within dialogues. Further comparison of MSDP will be included in our revised paper.\n- Despite both methods involving the extraction of examples from external database to prompt LLMs for knowledge generation and employing In-context learning, our contribution serves as a highly intuitive yet effective extension to MSDP. Besides, our approach also investigates the utilization of generated knowledge. Based on varying levels of knowledge granularity, we propose two distinct approaches for injecting knowledge into dialogues and subsequently compare their respective effects.\n\n**Question 2: No experiment comparing MSDP's performance with in-context learning.** \n\nThe MSDP approach has been experimented on the WoW dataset. We will adopt it as a baseline on ODKG dataset. However, since MSDP's backbone model is trained via Megatron[2], which is not open-source, and we want to ensure fairness, so we utilize Flan-T5-XXL, same as KnowEE, to establish a comparison against our proposed framework. We adhere to the experimental settings outlined in the paper and conduct both automated and manual evaluations of MSDP on ODKG. The results are presented as follows:\n\n| Method      | PPL↓      | D-1       | D-2       | Cohe.    | Info.    | Flu.     | Hall.↓   |\n| ----------- | --------- | --------- | --------- | -------- | -------- | -------- | -------- |\n| Flan-T5-XXL | 15.89     | 16.01     | 51.52     | 2.25     | 2.14     | 2.48     | 1.29     |\n| MSDP        | 24.41     | **20.71** | **63.69** | 2.92     | 3.01     | 1.54     | 0.88     |\n| KnowEE-FgKI | **14.25** | 17.88     | **57.33** | **3.09** | 2.83     | 2.43     | 0.96     |\n| KnowEE-CgKI | 15.83     | 17.74     | 56.18     | 3.01     | **3.13** | **2.74** | **0.83** |\n\nFrom the table, we can observe that MSDP outperforms our framework in terms of diversity (D-1, D-2). However, our approach excels in terms of *coherence* and *informativeness*. Furthermore, MSDP even scores lower in fluency compared to Flan-T5-XXL in zero-shot scenarios. Through case analysis, we find that Flan-T5-XXL with MSDP has a probability of copying content from generated Wikipedia knowledge during text generation, which significantly reduces the fluency of the generated text. In contrast, our method considers various types of knowledge during text generation, which enables us to mitigate this issue more effectively.\n\n**Question 3: No experimental support for the claim that in-context learning actually helps LLMs in inferring emotion, topic, and event knowledge.**\n\nAn evaluation of the knowledge generated, such as emotion, topic, and event in the first phase will be conducted to demonstrate the effectiveness of our approach.  We evaluate the accuracy of emotion knowledge generated in the first phase of KnowEE on ED. We select two previous SOTA emotion predict model on this dataset, namely EmpDG[3] and KEMP[4]. The purpose of the experiment is to demonstrate the effectiveness of in-context learning and the similarity-based sampling method. Results are presented as follows. \n\nIn the table, \"*Prompt Num*\" represents the number of examples used for in-context learning (ICL). \"*Calculate Similarity*\" indicates whether the sampling is done based on the highest sentence similarity score. \"*Use NRC-VAD*\" indicates whether revised judgment is applied to the outcomes. NRC-VAD[5] is a lexicon comprising VAD vectors with three dimensions (Valence, Arousal, Dominance) for 20,000 English words. Given that numerous different words share similar emotion meanings, such as \"happy\" and \"joyful\", if \"Use NRC-VAD\" is set to \"yes,\" it signifies that we compute the cosine similarity between the predicted emotion and the ground truth emotion's vector in the NRC-VAD dataset. If the computed result exceeds 0.95, the predicted emotion is considered accurate.\n\n| Model  | Prompt Num | Calculate Similarity | Use NRC-VAD | Accuracy（%） |\n| ------ | ---------- | -------------------- | ----------- | ------------- |\n| EmpDG  | --         | --                   | No          | 34.31         |\n| KEMP   | --         | --                   | No          | **39.31**     |\n| KnowEE | 0          | --                   | No          | 3.16          |\n|        | 5          | No                   | No          | 7.76          |\n|        | 8          | No                   | No          | 18.50         |\n|        | 10         | No                   | No          | 19.44         |\n|        | 16         | No                   | No          | 14.53         |\n|        | 5          | Yes                  | No          | 16.96         |\n|        | 8          | Yes                  | No          | 26.14         |\n|        | 10         | Yes                  | No          | 36.29  |\n|        | 16         | Yes                  | No          | 33.17         |\n|        | 5          | Yes                  | Yes         | 75.01         |\n|        | **8**      | Yes                  | Yes         | **84.64**     |\n|        | 10         | Yes                  | Yes         | 79.22         |\n|        | 16         | Yes                  | Yes         | 82.12         |\n\nFrom the table, it is evident that without utilizing ICL (*Prompt Num = 0*), the model's accuracy is extremely low. However, with the integration of ICL, as shown in rows 5 to 8 of the table, the performance improves to some extent, yet it still falls significantly short of SOTA. By selecting examples with the highest similarity to the current text during the sampling process for ICL, when prompt num is set to 8, the prediction accuracy can approach nearly SOTA (36.29 compared to 39.31). This outcome serves to validate the effectiveness of ICL and the similarity-based sampling method within our approach. Furthermore, following the application of NRC-VAD for revised judgment, the highest accuracy reaches 84.64%. This illustrates that our method ensures the effectiveness of knowledge prediction in practical conversation scenarios.\n\nWe are committed to expanding and refining our research based on your suggestions. Your feedback will be carefully considered, with the aim of further enhancing the significance and value of our study.\n\n**References:**\n\n[1] Liu, Zihan, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. \"Multi-stage prompting for knowledgeable dialogue generation.\" *arXiv preprint arXiv:2203.08745* (2022).\n\n[2] Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. \"Megatron-lm: Training multi-billion parameter language models using model parallelism.\" *arXiv preprint arXiv:1909.08053* (2019).\n\n[3] Li, Qintong, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. \"EmpDG: Multiresolution interactive empathetic dialogue generation.\" *arXiv preprint arXiv:1911.08698* (2019).\n\n[4] Li, Qintong, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. \"Knowledge bridging for empathetic dialogue generation.\" In *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 36, no. 10, pp. 10993-11001. 2022.\n\n[5] Mohammad, Saif. \"Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words.\" In *Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers)*, pp. 174-184. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-organized, contains enough information in a limited number of pages, and is easy to understand.\n2. The paper evaluates different variants of the method KnowEE-FgKI and KnowEE-CgKI, and they generally achieve better results than baselines in both automatic and human evaluation. The experiments in the paper are extensive and convincing. The chosen evaluation metrics are presentative, and ablation experiments show the efficacy of the KnowEE's design choices.", "reasons_to_reject": "1. This proposed method seems to be the multi-source adoption of an existing method MSDP (https://arxiv.org/abs/2203.08745).  It seems to be incremental that the new method just adds multiple knowledge sources and in-context learning with demonstration retrieval (such as https://arxiv.org/pdf/2305.04320.pdf). Thus, the work lacks core novelty.\n2. There is no ablation experiment comparing MSDP'performance with in-context learning and that only with zero-shot. There is no experimental support for the claim that in-context learning actually helps LLMs in inferring emotion, topic, and event knowledge.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new method of KnowEE to flexibly integrate multiple knowledge sources into dialogue generation. Specifically, the new method includes two steps: multi-source in-context learning and fine/coarse-grained knowledge injection for response generation. At last, the method is tested on different dialogue datasets and proven to achieve noticeable improvements.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. ACL2022 Multi-Stage Prompting for Knowledgeable Dialogue Generation"}, "extra_text": {}}, {"review_id": "aLoY56NZiy", "reviewer": "Reviewer_eFw1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our paper. Your insights are of utmost importance to us, and we would like to provide comprehensive explanations addressing the concerns you have raised.\n\n**Question: Lack a discussion on how well KnowEE would generalize to real-world scenarios and diverse user interactions.**\n\nFirst and foremost, we greatly appreciate your recognition of our work. One of the central objectives of this study is to apply LLMs to downstream generation tasks in a concise and effective manner. In our future endeavors, we will employ more powerful and innovative LLMs such as LLaMA and Vicuna. Additionally, we will investigate the application of KnowEE in the context of smaller-scale models.\n\nSecondly, we acknowledge the absence of generalization analysis to real-world scenarios in our work. Therefore, we are establishing an authentic dialogue system. We will select 500 dialogue utterances randomly from four datasets to serve as initial sentences. For each of these sentences, we will engage university students for real conversational interactions. Specifically, after users input the start sentence, a three-round dialogue will ensue, and users will give the rating about the reponses. Due to time constraints, we are unable to report this result during the rebuttal period. Consequently, we have deferred this aspect of the work to a later phase. We will incorporate this experiment part in the next version of the paper.\n\nWe are committed to expanding and refining our research based on your suggestions. Your feedback will be carefully considered, with the aim of further enhancing the significance and value of our study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel approach for multi-turn dialogue generation. By leveraging large-scale language models and multiple external datasets, the proposed framework extracts and injects relevant knowledge into dialogues. This approach goes beyond existing methods that focus on specific datasets, showcasing innovation in utilizing LLMs for knowledge integration in dialogues.\n2. Effective validation through evaluation. The paper presents both automatic and manual evaluation results that validate the effectiveness of the proposed framework.", "reasons_to_reject": "Lack the generalization analysis to real-world scenarios. While the proposed framework shows promise in generating coherent responses, the paper lacks a discussion on how well it would generalize to real-world scenarios and diverse user interactions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the challenge in generating multi-turn dialogues by leveraging diverse knowledge from various sources. Existing methods focus on specific dialogue knowledge and corresponding datasets, limiting generalization and resource requirements. The proposed \"KnowEE\" framework employs large-scale language models (LLMs) to overcome this. It consists of two phases: First, it extracts relevant samples from five knowledge-rich datasets to create prompts for generating knowledge. Second, it integrates this knowledge into ongoing dialogues at different levels of detail and feeds it into LLMs to produce coherent, informative responses. Both automated and manual evaluations confirm the framework's effectiveness in utilizing multi-source, multi-type knowledge for generating fluent and meaningful dialogue responses.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dcYtc9fpSq", "reviewer": "Reviewer_keZg", "summary": "", "questions": "The use of \"multi-source\" for this approach is slightly confusing. Multi-source would usually refer to the five different knowledge datasets, but the proposed approach does not actually retrieve knowledge from these sources. There is only one source, the LLM, and the five different datasets are used for in-context learning prompts to retrieve the LLM knowledge.\n\nCan you explain what the difference is between Fine-Grained Knowledge Injection and Coarse-Grained Knowledge Injection? From the descriptions in Section 3.3, it seems that both approaches have access to K^t_emo, K^t_evt, and K^t_wor, although this knowledge appears in a different linear position in the prompt. Does the position matter? There is nothing in the prompt that instructs the LLM to treat the knowledge differently based on where it appears. Section 5.1 mentions that coarse-grained outperforms fine-trained \"possibly due to the superior comprehension abilities of Flan-T5-XXL in processing input,\" which I do not understand because Flan-T5-XXL is used as the backbone for both approaches; is this about the positioning?\n\nAs far as I can tell, only the K^t-1 knowledge is missing from the coarse-grained approach. If this is the case, then \"fine-grained\" and \"coarse-grained\" are confusing names for these two approaches, since they both use utterance-level knowledge. From the name, I would have expected the coarse-grained approach to use the entire dialogue context as the query for all knowledge types.\n\nHow many samples are included in the human evaluation? I am concerned because the metrics used for automatic evaluation, perplexity and DISTINCT, evaluate the generation quality, not the content; BLEU and ROUGE have their limitations, but they would have evaluated the response content, which I would argue is more important than text quality for a knowledge-based approach. We must depend entirely on the human evaluation's Coherence, Informativeness, and knowledge-specific categories to evaluate the response content, but human evaluations are usually small compared to the overall test set size, especially with as many comparison methods as are used in these experiments.\nUpdate: This concern was addressed in the author rebuttal.\n\nIt would also be helpful to see the evaluation results for a traditional, trained knowledge-retrieval dialogue system as a point of comparison. While the proposed approach is shown to outperform non-knowledge-based approaches and other few-shot learning-based approaches, the introduction section motivates the current work as an alternative to a trained knowledge retriever.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our paper. Your insights are of utmost importance to us, and we would like to provide comprehensive explanations addressing the concerns you have raised.\n\n**Question 1: The use of \"multi-source\" for this approach is slightly confusing.**\n\nThe term \"multi-source\" here indeed refers to the utilization of five external datasets for in-context learning. Our rationale for this choice stems from the belief that LLMs have already assimilated substantial knowledge from diverse sources during pre-training, effectively rendering them aggregations of various knowledge sources. Our concept revolves around leveraging multiple datasets to stimulate the pertinent knowledge stored in LLMs. We apologize for any confusion arising from this expression and acknowledge the possibility of amending this term.\n\n**Question 2: Explain the difference between Fine-Grained Knowledge Injection and Coarse-Grained Knowledge Injection.**\n\nDue to the input token length limitations of Flan-T5-XXL (2048), as dialogues extend over multiple rounds, they frequently surpass this limit. Consequently, our endeavor aims to reduce input length while circumventing knowledge attrition. Initially, we adopted a coarse-grained approach: the concatenation of knowledge corresponding to each utterance in dialogue history. Evidently, this strategy precipitated rapid inflation of the input token length. This prompted us to opt exclusively for the last round's emotion, event, and general world knowledge. We observed marginal disparities in the results between these two methods, with the latter even outperforming the former in most instances. This rationale underpins our exposition of the two knowledge injection methods in paper. As to why the latter method displayed superior performance, we attribute it to the inclusion of unadulterated dialogue history. This holistic dialogue contextual comprehension potentially enhances response generation.\n\nWe regret any confusion stemming from our nomenclature of the two knowledge injection methods. Concurrently, we appreciate your interpretation of the coarse-grained approach: utilizing the entire context of the dialogue as a query for all types of knowledge, which closely aligns with the concept of \"coarse-grained\". We are inclined to refine the description of Coarse-Grained Knowledge Injection method and intend to employ the entire contextual history as a query for knowledge generation in subsequent work.\n\n**Question 3: Num of samples that are included in the human evaluation. Missing metrics like BLEU and ROUGE to evaluate the response content.**\n\nConsidering the costs associated with human evaluation, **500** samples have been included for each method across each dataset. The metrics based on N-gram content matching, such as Bleu and Rouge that you mentioned, have already gained consensus as inappropriate for evaluating text generation, particularly dialogue generation. As a result, we did not initially consider incorporating such metrics. In order to address your concerns, we employ these metrics to evaluate certain methods from Table 2 and Table 3 of the paper that obtain higher scores in both human and automatic evaluations. Taking the ED dataset as an example, the BLEU (*BLEU-1, 2, 3, 4*) and ROUGE (*ROUGE-1, 2, L*)  scores for various methods are presented below:\n\n| Method      | BLEU-1    | BLEU-2   | BLEU-3   | BELU-4   | ROUGE-1   | ROUGE-2  | ROUGE-L   |\n| ----------- | --------- | -------- | -------- | -------- | --------- | -------- | --------- |\n| ChatGLM-6B  | 5.45      | 1.69     | 0.63     | 0.25     | 12.98     | 1.27     | 10.34     |\n| Flan-T5-XXL | 7.61      | 2.92     | 1.30     | 0.62     | 11.58     | 1.77     | 10.31     |\n| DialoGPT    | 5.16      | 2.07     | 0.99     | 0.54     | 13.23     | 2.01     | **12.31** |\n| BlenderBot  | 7.51      | 2.03     | 0.71     | 0.30     | **13.43** | 1.35     | 10.98     |\n| FSB         | 7.24      | 2.15     | 0.92     | 0.49     | 12.75     | 1.43     | 11.39     |\n| KnowEE-FgKI | 10.78     | 3.81     | 1.68     | 0.93     | 12.46     | 1.66     | 10.73     |\n| KnowEE-CgKI | **11.84** | **4.40** | **2.13** | **1.27** | 13.40     | **2.02** | 11.47     |\n\nThe results indicate that our approach continues to maintain a overall leading position in terms of the response content. We will incorporate these findings into the paper to enhance the robustness of our experiments. Thank you once again for your invaluable suggestions.\n\n**Question 4: It would also be helpful to see the evaluation results for a traditional, trained knowledge-retrieval dialogue system as a point of comparison.** \n\nWe extend our sincere gratitude for your valuable recommendations concerning our experimental setup. Due to time constraints, we are unable to report the result of an appropriate traditional, trained knowledge-retrieval dialogue system as a baseline during the rebuttal period. Consequently, we have deferred this aspect of the work to a later phase. We are committed to incorporating this experiment part in the next version of the paper.\n\n**Question 5: Typos Grammar Style And Presentation Improvements**\n\nWe apologize for any spelling and grammatical errors that may have detracted from your reading experience. We will rectify the issues you've raised and meticulously review the entire paper to prevent the existence of similar problems.\n\nFinally, we are committed to expanding and refining our research based on your suggestions. Your feedback will be carefully considered, with the aim of further enhancing the significance and value of our study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach is well-motivated, clearly explained, and outperforms almost all comparison systems on almost all metrics. The framework is agnostic to the choice of underlying LLM, and the authors demonstrate that it can be used to improve dialogue response generation performance on several LLMs.", "reasons_to_reject": "The two proposed knowledge injection approaches, utterance-based fine-grained and dialogue-based coarse-grained, seem very similar to me. From the description in the text, they seem to differ only in whether or not the penultimate user utterance is used. The experimental results show that coarse-grained outperforms fine-grained, but I am not sure what the takeaway lesson is because they are so similar to each other.\n\nAdditionally, I think the evaluation would be stronger with the inclusion of content-based automatic metrics like BLEU and ROUGE, as well as additional comparison systems representing the trained knowledge retrieval approach.\nUpdate: This concern was addressed in the author rebuttal.\n\n(See questions below for more deatils.)", "questions_to_authors": "The use of \"multi-source\" for this approach is slightly confusing. Multi-source would usually refer to the five different knowledge datasets, but the proposed approach does not actually retrieve knowledge from these sources. There is only one source, the LLM, and the five different datasets are used for in-context learning prompts to retrieve the LLM knowledge.\n\nCan you explain what the difference is between Fine-Grained Knowledge Injection and Coarse-Grained Knowledge Injection? From the descriptions in Section 3.3, it seems that both approaches have access to K^t_emo, K^t_evt, and K^t_wor, although this knowledge appears in a different linear position in the prompt. Does the position matter? There is nothing in the prompt that instructs the LLM to treat the knowledge differently based on where it appears. Section 5.1 mentions that coarse-grained outperforms fine-trained \"possibly due to the superior comprehension abilities of Flan-T5-XXL in processing input,\" which I do not understand because Flan-T5-XXL is used as the backbone for both approaches; is this about the positioning?\n\nAs far as I can tell, only the K^t-1 knowledge is missing from the coarse-grained approach. If this is the case, then \"fine-grained\" and \"coarse-grained\" are confusing names for these two approaches, since they both use utterance-level knowledge. From the name, I would have expected the coarse-grained approach to use the entire dialogue context as the query for all knowledge types.\n\nHow many samples are included in the human evaluation? I am concerned because the metrics used for automatic evaluation, perplexity and DISTINCT, evaluate the generation quality, not the content; BLEU and ROUGE have their limitations, but they would have evaluated the response content, which I would argue is more important than text quality for a knowledge-based approach. We must depend entirely on the human evaluation's Coherence, Informativeness, and knowledge-specific categories to evaluate the response content, but human evaluations are usually small compared to the overall test set size, especially with as many comparison methods as are used in these experiments.\nUpdate: This concern was addressed in the author rebuttal.\n\nIt would also be helpful to see the evaluation results for a traditional, trained knowledge-retrieval dialogue system as a point of comparison. While the proposed approach is shown to outperform non-knowledge-based approaches and other few-shot learning-based approaches, the introduction section motivates the current work as an alternative to a trained knowledge retriever.", "paper_topic_and_main_contributions": "The paper presents a framework, KnowEE, for retrieving knowledge for open-domain dialogue generation from LLMs and injecting that knowledge into LLM prompts to generate dialogue responses. The authors argue that training knowledge retrievers is both computationally expensive and brittle, with poor generalization to different knowledge sources. They also note that LLMs contain a large amount of knowledge from their pretraining but are often unable to retrieve and use that knowledge when prompted with only the dialogue history.\n\nThe authors propose to retrieve LLM knowledge by identifying the top n most similar content-knowledge pairs from one of five knowledge datasets representing different knowledge types (eg. emotion, personal, world knowledge); the pairs are used as in-context learning examples to prompt the LLM to retrieve knowledge for the current utterance or dialogue. The knowledge is then injected into the response generation prompt in one of two ways: at the utterance level, where knowledge is retrieved for the last two user utterances in the dialogue history, or at the dialogue level, where the last user utterance is used as a proxy to retrieve knowledge for the whole dialogue.\n\nThe experimental results include two automatic metrics, perplexity and DISTINCT, and a human evaluation including custom categories specific to each knowledge type. The authors find that the proposed KnowEE approach outperforms existing non-knowledge-based systems, including both LLMs and pretrained dialogue models, as well as two other few-shot learning, knowledge-based models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\nIntroduction, Line 100: person -> persons\nIntroduciton, Line 134: propsoed -> proposed\nExperimental Settings, Datasets, Line 388: \"... is followed the original dataset\" -> \"...is followed\"\nExperimental Settings, Datasets, Line 389: trainset -> training set\nExperimental Settings, Datasets, Line 390: testset -> test set\nResults and Analysis, Human Evaluation Results, Line 533: fin-grained -> fine-grained"}}]

Author rebuttal:

Meta-review: Overall this is a solid paper with extensive experiments and convincing results for the proposed framework KnowEE in different knowledge injection settings.

The novelty of he work was somewhat challenged by one of the reviewers, due to missing reference and comparison with MSDP in the original submission. While the authors provided further experiments with MSDP during author response, the reviewer was not fully convinced of positioning of the work as completely novel instead of incremental. 

Another criticism around the lack of generalization analysis to real-world scenarios is acknowledged by the authors, but will only be addressed in future work.
Decision: Accept-Main