Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators
Authors: Liang CHEN
Yang Deng
Yatao Bian
Zeyu Qin
Bingzhe Wu
Tat-Seng Chua
Kam-Fai Wong
Venue: EMNLP 2023 Main (2023)
Keywords: Evaluation framework;Knowledge generation;Large language model;
TL;DR: A Comprehensive evaluation for knowledge generation with Large language models

Abstract: Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives -- Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLM-generated knowledge with human annotations will be released to facilitate future research.

Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators
Liang Chen' | Yang Deng', Yatao Bian Zeyu Qin', Bingzhe Wu', Tat-Seng Chua², Kam-Fai Wong":
'The Chinese University of Hong Kong, Teencett AI Lab
National University of Singapore, "The Hong Kong University of Science and Technology lchenêse. cuhkn@se.cuhk. edu.
Abstract
Intrinsic Evaluation
Extrinsic Evaluation
Large language models (LLMs) outperform information retrieval techniques for down- stream knowledge-intenss
tasks when being prompted to generate world knowledge. Yet, community concerns abound regarding the fac- tuality and potential implications of using this uncensored knowledge. In light of this, we in- troduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to system- atically and automatically evaluate generated knowledge from six important perspectives Factuality, Relevance, Coherence, Informative- nexs, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely-studies knowledge-intense tasks, ie.. open-domain question answering and mowledge-groundee dialogue. Surpris- ingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence the outputs are more important than small factual mistakes. Further, we show how to use CONNER to im- prove knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowl- edge Selection. Our evaluation code and L.M-- generated knowledge with human annotations will be released! to facilitate future research.
IR
<Rew reve knowledge
<hndwer>
<quatiy>
<Arpwerr
Reader
LLM
<Generated knowledgges
knowledge's internal properties
downstream impact
Figure I:
The CONNER Framework: Intrinsic evaluations probe the internal properties of acquired knowledge, while extrinsic evaluations assess its downstream im- pacts. This framework applies universally to two-stage processes in knowledge-intensive tasks.
LLMs have demonstrated superior performance than information retrieval (IR) models (Karpukhin et al., 2020) when it comes to generating world knowledge (Yu et al., 2023; Liu et al., 2022) for the downstream tasks. However, the knowledge gener- ated may contain inherent issues, such as false state- ments off-topic information. Therefore, the lack of extensive evaluation of this knowledge raises concerns about its use in downstream tasks.
To this end, four lines of research emerge. Firstly, human evaluations are conducted to as- sess the generated knowledge from diverse per- spectives (Li et al., 2022; Yu et al., 2023; Liu al., 2023a). However, their conseminni nature and subjectivity often encounter issues of scalabil- ity and reproducibility. Secondly, datasets have been constructed to evaluate open-domain gener- ation with the aid of references (Honovich et al., 2021; Glover et al., 2022a; Lee et al., 2023; Li et al., 2023). These methods. while more objec- tive, are limited by their dependence on human- labelled references, impacting their real-world ap- plicability and generalizability to dynamically gen- crated content. Thirdly, self-evaluation methods (Kadavath et al., 2022b; Manakul et al., 2023) esti-
1 Introduction
The exceptional success of large language models (LLMs) like ChatGPT and GPT4 (Ouyang et al., 2022; OpenAI, 2023) has fueled a growing interest in substituting traditional models with LLMs to attain superior performance across various NLP tasks (Liu et al., 2023b; Jagerman et al., 2023; Wang et al., 2023). In open-domain question an- swering (QA) and knowledge-groundee dialogue,
Partial work was done his Tencent Al Lab internship. Corresponding author. /tttps://github.. wttpe/cccommcc/aee wwwm/haaniingg/CO/eeee

---

Evaluation Taxonomy
Definition
Factuality
whether the information in the knowledge can be verified by external evidence.
Relevance
whether the knowledge is relevant to the user query.
Intrinsic
Coherence
whether the knowledge is coherent the sentence and paragraph levels.
Informativeness
whether the knowledge is or unexpected against the model's existing knowledge.
Helpfulness
whether the knowledge can improve the downstream tasks, whether the results of downstream tasks using the knowledge are factually accurate.
Extrinsic
Validity
Table I: Taxonomy of evaluation metrics of acquired knowledge.
mate a model's uncertainty in its generated content. Despite simplicity, they lack interpretability and are less effective for long-form answers. Lastly, contemporary studies (Pan et al., 2023; Min et al., 2023) apply fact-checking principles spot factual inaccuracies. However, these evaluation methods mainly assess single aspect of the intrinsic quality of generated knowledge, overlooking other facets and their extrinsic impact on downstream lasks, thereby limiting comprehensive understanding of LLM-generated content.
necessarily guarantee higher quality and reliability (8 4.4). 3) In addition to assessing and analyzing the generated knowledge from different LLMs, the evaluation outcome of CONNER can be exploited to enhance knowledge generation and further improve the performance of downstream tasks (5 5).
Our main contributions are as follows:
We conduct the first empirical analysis focusing on both intrinsic quality and extrinsic reliability of the generated knowledge from LLMs.
We propose CONNER, COmpreheNsive kNowl- edge Evaluation fRamework that enables the au- tomatic evaluation of LLMs as knowledge gener- ators from diverse perspectives, climinating the need for labelela references.
In light of these limitations, we propose CONNER, a COmprcheNsive kNowledge Evaluation [Rame- work, illustrated in Figure 1. CONNER is designed be reference- framework that can system- atically and automatically evaluate the generated knowledge from six fine-grained perspectives, in- cluding diverse intrinsic evaluation of its internal properties, as well as uniform extrinsic evaluation of its impact on specific downstream tasks. The taxonomy of evaluation metrics is presented in Ta- ble 1. Based CONNER, we conduct empirical eval- uations on three different types of LLMs, including LLaMA (Wei al., 2022) base LLM), FLAN-T5 LLM), Chat-
The extensive evaluation and analysis yield pro- found insights and valuable practical experience for leveraging LLMs as knowledge generators. We collect a new set of multi-perspective hu- man judgments of LLM-generated knowledge for two mhowledge-intensing generation datasets. We demonstrate that CONNER aligns well with hu- man judgments. The human annotations will be released to facilitate future research.
(Wei et al., 2022) (an instruction-tune GPT (Ouyang et al., 2022) (a commercial LLM trained with human feedbacks). We evaluate them on two widely-studied knowledge-intensivy tasks: open-domain QA (Kwiatkowski et al., 2019) and knowlcdge-groume grounded dialogue (Dinan et al., 2018).
Related Work
Knowledge-intensinn tasks rely heavily on ac- cess to external knowledge sources, such as open- domain dialogue and QA (Dinan et al.. 2018; Kwiatkowski et al., 2019; Petroni et al., 2021). The main-streamed methods (Karpukhin et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021) typi- cally employ IR techniques to first retrieve the rele- vant knowledge from Wikipedia and then produce the answer or response conditioned on the knowl- edge. Nowadays, with the powerful capabilities of LLMs (OpenAT, 2023; Kadavath et al., 2022a), trending approach is to leverage LLMs di- rectly generate the relevant knowledge for a given query and then apply the model-generated knowl- edge to complete the downstream tasks (Liiu et al., 2022; Li et al., 2022; Yu et al., 2023). Despite out- performing retrieval based methods, these knowl-
Our detailed investigations yield several valu- able insights about the LLM-generated knowledge: 1) LLM-generated knowledge surpasses retrieved knowledge in most evaluation perspectives, while it actually suffers from the factuality issue as ex- pected. Notably, the factuality of downstream tasks is found to be less affected by this issue, when compared to the impact of lower relevancy and coherency observed in the retrieved knowledge (5 4.3). Several critical factors are identified to in- fluence the factuality of the generated knowledge, such as their frequency and length, while few-shot in-context learning and larger size of models do not

---

edge generation techniques lack rigorous evalua- tion of their quality and reliability, which may con- tain misleading or even plausible false information, e.g., hallucination and factual inconsistency.
orous evaluation of the quality and dependability of knowledge used in knowledge-intensiee tasks. CONNER rooted in in-depth error analysis, paving the way for the construction of an evaluation tax- onomy, which integrates six unique perspectives into two coherent categories, as delincated in Ta- ble 1. Capitalizing on the advantages of unsuper- vised metrics, our framework eliminates the need for human-labeled reference knowledge and stan- dardizes scores within an intuitive range of [0, 1], simplifying comparison and interpretation.
These issues are prevalent across various NLP tasks (Ji et al., 2023). However, most studies target specific downstream tasks, such as text summa- rization (Maynez al., 2020; Wang et al., 2020; Kryseinski et al., 2020a; Pagnoni et al., 2021), di- alogue generation (Dziri et al., 2022; Chen et al., 2023; Xue et al., 2023; Deng et al., 2023), and fact verification (Thorne et al., 2018; Wadden et al.. 2020; Schuster et al., 2021; Pan et al., 2023). These tasks are designed to examine consistency either between the imput and output or between the input and human-labeled reference, e.g,, the source doc- ument and its summary, the grounded knowledge and the generated response, or human-written claim and pre-annotated references.
The subsequent subsections provide detailed examination of the framework's 's design, commenc- ing with the formulation of knowledge-intensive tasks and the identification of associated error pat- terns. These insights direct the design of our met- rics. Through comprehensive intrinsic and extrinsic evaluations, we aim to gain a holistic understanding of the L.LMs-generatee knowledge.
The success of LLMs and generative search en- gines (Zhao et al., 2023; Zhu et al., 2023) have brought hallucinations in LLM outputs (Rawte et al., 2023; Zhang et al., 2023) into focus. Re- search typically falls into four categories. (Lee et al., 2023; Li et al., 2023) aim to assess the fac- tuality of open-domain generation automatically using specially designed datasets, but their reliance on references may limit real-world applicability. Another stream of work (Li et al., 2022; Yu et al.. 2023; Liu et al., 2023a) uses human evaluation to measure output quality, which difficult scale. A third approach (Kadavath et al., 2022b; Manakul et al., 2023) detects hallucinations by examining the model's uncertainty or confidence, which can be inaccurate for long answers. Lastly, recent stud- ies (Peng et al., 2023; Min et al., 2023) apply fact- checking principles to spot factual inaccuracies.
3.1 Tasks Formulation
Formally, we define the knowledge-intensive task as follows: given a user query q. the goal is to produce an answer with access to knowledge re- sources as illustrated in Figure. 1. Specifically, the system first obtains the relevant knowledge k that can help answer the query q from knowledge resources K., then the reader generates an answer a using the acquired knowledge k. Specifically, the knowledge resource IC can be either a knowledge base for knowledge retrieval or language models for knowledge generation. Detailed formulations of these two settings are presented in Appendix A.
3.2 From Error Patterns to Metries Design
To identify common errors by LLMs in knowledge- intensive tasks and create more targeted evalua- tion framework, used thematic analysis (Braun and Clarke, 2012). We began by extracting and consolidating patterns from subtle errors in knowl- edge and answers responses from LLaMA to 160 samples from NQ (Kwiatkowski et al., 2019) and WoW (Dinan et al., 2018) datasets. To ensure the breadth of the error spectrum was adequately represented, we further substantiated these patterns using additional questions from NQ and WoW. As result, we discerned four primary error categories in knowledge generation and two in answer genera- tion. In response, we devised four intrinsic metries for knowledge evaluation and two extrinsic metrics for answer evaluation, as outlined in Table I.
Different from previous studies, we propose a comprehensive framework for evaluating knowl- edge generated by LLMs. Our goal is to automati- cally test the intrinsic quality and extrinsic impact of generated information in wewwledge-intnsive tasks, without requiring knowledge labelling hu- man involvement. Through extensive testing with this framework, aim to deepen and broaden our understanding of LLM -.eeereeddt knowledge and provide valuable insights for future research.
3 The Evaluation Framework
We introduce CONNER, a comprehensive and inno- vative framework, specifically designed for the rig-

---

3.3 Intrinsic Evaluation
Relevance To assess the relevance between a given query 9 and the acquired knowledge we compute the relevance score as follows:
Intrinsic evaluation refers to the assessment of the acquired knowledge based on its internal properties and performance, without considering its impact on downstream tasks or applications. In specific, we implement four model-based metrics for evalu- ating the acquired knowledge in terms of factuality, relevance, informativeness, and coherence.
Srel (k, g) Matching
The Matching(+) function denotes a fine-grained matching model specifically designed for assessing the relevance between the query and knowledge. In our study, we employ the BERT ranking model (Nogueira et al., 2019) for this purpose.
Factuality The core of factuality assessment is validating the acquired knowledge by external ev- idence 2. Given an acquired knowledge
This methodology addresses the limitations that arise when traditional relevance metrics are applied within knowledge generation scenarios. Traditional relevance metries (Karpukhin et al., 2020; Shuster et al., 2021; Komeili et al., 2021), which typically rely on word overlap or similarity with human- written references, face two significant challenges. First, these traditional metrics do not correspond well with scenarios where LLMs serve as genera- tive search engines, as evidenced by the unsatisfac- tory results in Table 10. Second, the reliance on reference knowledge constitutes substantial chal- lenge, especially when such references scarce or absent in real-world applications. Contrarily, our BERT ranking model, trained on manually an- notated Bing search data, excels at comparing the relevance of different knowledge to a given query.
{si,..., _m composed of m sentences, we can use a dense retrieval model (Santhanam et al., 2021) or search engine API to recall the I most relevant evidence E, = {ei. ................................ Ci,i, for each sentence 8; from the expert knowledge base or the internet. Af- ter collecting all the evidence E (E....., J.
the factuality score is computed as follows:
Stact
min [st Pi)
(1)
= min max (1J(8,,6.)) (=1..m 1=1.
where f()) is function compute sentence-level factuality, NLI(-) is natural language inference model processing a premise-hypothesis pair to out- put a R vector, indicating whether a hypothesis (s,) is entailed by, neutral to or refuted by the given premise (eig). Following these computa- tions, sentence-level results are aggregated along the entailment dimension using one of three oper- ations: min, mean, or max to match the desired error tolerance level. In this instance, we exem- plify the process using min. Finally, we obtain a three-dimensional factuality score Sract (k, E). From each dimension of this vector, we can de- rive three fine-grained scores. We denote those non-verified
Coherence As the acquired knowledge is typi- cally long-form texts composed of multiple sen- tences, we propose to measure sentence-level co- hesion and paragraph-Ievel coherence: the for- mer measures the cohesion of individual sentences, and the latter measures the coherence between sentences. The sentence-level cohesion score Seoh_sent(A (k) is computed as follows:
scores as Pactual-consistern and factual-inconsistene respectively.
Scoh_sent (k) = m Σ>> 1/PPL(s:)
(3)
This strategy seeks to address the shortcomings of traditional factuality metrics (Wang al., 2020; Honovich al., 2021; Glover al., 2022a; Lee et al., 2023) that mainly depend on consistency references. These metrics often fail in emerging knowledge generation sce- narios (Table 10), as they struggle with model- generated content beyond reference knowledge scope and face difficulties when references un- available in real-word weal-world applications. Our method of evidence collection and results aggregation ef- fectively tackles these issues.
where PPL(-) is computed by GPT-based model (Radford et al., 2019; Black et al., 2021), measuring the perplexity for each sentence.
with
On the other hand, the paragraph-leved coherence score is determined by the normalized score a discourse coherence model (Jwalapuram et al., 2021), denoted as Scoh
Scoh_para = Scorerpaa 8mn)
(4)
By considering both sentence-level cohesion and paragraph-level coherence, we gain insights into the overall coherence of the acquired knowledge.
WW empirically demonstrate ground-truth knowledge is dispensable for the factuality evaluation in Appendix B.

---

Informativeness To assess the informativeness of the procured knowledge-dcfingg defined as the degree to which the knowledge is novel or unexpected in relation to the model's 's existing knowledge about the query-we calculate the informativeness score of the acquired knowledge k given q as follows:
where L'q, k, a) and L(q, K a) are cross entropy losses of answer generation using i and kj re- spectively. Ideally, the generated knowledge k can provide enough information and reduce the [(g, k; a) to zero, and then the helpfulness score equals one. The worst case is the generated knowledge is no better than random knowledge (C(q, k, a) 2 1 E L(q, KG ,a),, and the help- fulness score is naturally zero.
(isto(k; 9) = 1- exp Σ ln (oo (5)
a
Assuming the unbiased benchmark model encap- sulates world knowledge from general pretraining data, we thus select the GPT-2 series models.
Validity To measure how the reliability of the ac- quired knowledge affects the factuality of the gener- ated answer a on downstream tasks, we define the validity metric for two types of downstream tasks: span-based answers (e.g., open-domain QA) and open-ended answers (e.g., geguulddeggr di- alogue). As for span-based answers, the generated answers cannot form complete sentence for fac- tuality measurement. To this end, we concatenate (g, a²) as the premise and (q. a) as the hypothesis for deriving the factual-cons stent score of the NLI(-) model as the validity score:
To grasp the expected behaviour of this metric, consider a simple query: "What is the capital of the United States?" The knowledge acquired here is "Washington". :. In this situation, the model's 's aver- age probability of generating "Washington" is high, it already knows this fact. Consequently, our informativeness score for this knowledge would be low. Conversely, the acquired knowledge was "Chicugo", : the model's probability of generating would be low. This knowledge is surprising com- pared to its existing knowledge, resulting in a high informativeness score. On the other hand, for a tough query where the model is clueless, any pro- vided knowledge would score high on informative- ness due the model's low output probabilities.
Sval(g, a" a) NN.lacs ((alacc((ggg)) a) (q.n"))
(7)
where a" denotes the ground-truth answer for downstream tasks and the NLI((.) model is the same as that of Eq. (1).
We demonstrate this measure outperforms tra- ditional metries like Exact Match and F1 score shown in Table 10, which rely on literal matches, and often yield low recall. For instance, an entity pair like 'PRC' and 'China' would receive zero score due to their differing literal presentations. we measure how well the acquired knowledge con- tributes to the downstream task on two types of metrics (helpfulness and validity). Extrinsic evalua- tion provides more comprehensive understanding of the practical value of the acquired knowledge.
3.4 Extrinsic Evaluation
Extrinsic evaluation, in contrast to intrinsic evalu- ation, focuses on uniformly assessing the perfor- mance of the acquired knowledge within the con- text of different downstream tasks. Specifically,
As for open-ended answers, we collect evi- dence E = {ei,... 6t) and adjust Eq. (1) to be: (a, E) f(a, E) max (a,naa(a,e)) (8)
4 Evaluation
Helpfulness Given a query and answer pair (g. a), we assess to what extent the acquired knowl- edge i can help answer the query. As we assume no pre-annotated ground-truth knowledge, we use irrelevant knowledge as the baseline. Specifically, we randomly sampled knowledge . kg to reduce the variance of baseline estimation. Then the helpfulness score is computed as follows:
In this section, we will first validate our proposed metrics, and then leverage them to comprehen- sively evaluate three different types of LLMs across two knowledge-intene inteesvvete tasks, followed by an in- depth analysis of the results.
4.1 Metries Eflicacy Validation
To validate the effectiveness of our proposed met- rics, conducted manual evaluations and com- pared the results with baseline metrics. Specifi- cally, we developed specific annotation guidelines for each metric, detailed in Appendix J, and per- formed manual annotations accordingly. These
Swilgg, a. ht kT .... AT)
C(g, k.a)
=max(0,1
Σm= L(g, .a)))
(6)
log P(al q. k)
max
>1 log P(alg.

---

Factuality Non-verif.
Coherence
Model
Setting
Relevance
Inform.
Helpful.
Validity
Fact-cons.
Fact-ineon.
Coh-sent, sent. Cor-para.
DPR
Supervised
97.78%
2.23%
0.00%
0.7514
0.0301
0.7194
0.8965
0.1236
36.86%
FLAN-T5
58.40%
27.80%
13.80%
0.6848
0.1249
0.7776
0.6727
0.0000
32.47%
LLAMA
7cro-shot
94.20%
4.80%
1.00%
0.7316
0.1183
0.8240
0.7572
0,2191
42.00%
CHATGPT
83.63%
13.6%
2.77%
0.8491
0.0909
0.9033
0.7330
0.1461
4
FLAN-T5
20.75%
62.40%
25.40%
0,6787
0.0416
0,8110
0.6899
0.0000
34.65%
ILAMA Few-shot
89. 19.000%%
9.200%
1.80%
0.6966
0.0776
0,8550
0.8545
0.2528
40.49%
CHATGPT
86.07%
10.97%
2.96%
0.9205
0.0653
0.8837
0.7700
0.1966
42.36%
Table 2: Automatic evaluation results of different LLMs in the Natural Question test set. Underlined and Bold results denote the best results among each setting and among all settings, respectively.
Factuality
Coherence Coh-sent. Coh-para.
Model
Setting
Relevance
Inform. Helpful. Validity
Fact-cons.
Non-verif.
Fact-incon.
DPR
Supervised
91.96%
5.18%
2.87%
0.0907
0.0223
0.6569
0.9357
0.0000
61.52%
FLAN-TS
77.90%
17.28%
4.82%
0.3776
0.1203
0.8331
0.7239
0.0904
56.97%
LLAMA
Zcro-shot
89.46%
8.89%
1.65%
0,5041
0.0548
0.8389
0.7889
0.1178
63.50%
CHATGPT
88.51 %
10.38%
1.11%
0,5283
0.1028
0.9250
0.7448
0.1023
59.76%
FLAN-TS
76.50%
17.20%
6.30%
0,4463
0.1523
0.7988
0.6983
0.0934
57.18%
LLAMA
Few-shot
85.07%
12.05%
2.88%
0.3930
0. 10188
0.7947
0.7855
0.1132
63.79%
CHATGPT
85.75%
12.01 %
2.24%
0.4618
0.0979
0.8632
0.7922
0.1164
60.27%
Table 3: Automatic evaluation results of different LLMs in the Wizard of Wikipedia test set.
annotations allowed us calculate the correlation between each metric and human evaluations. Sub- scquently, we compared these correlations with baseline metrics (Table 10). Our metrics demon- strated a strong correlation with human evaluations, significantly outperforming the baseline metries. Details are presented in Chapter 6 and Appendix J.
Evaluation Setting Following (Yu et al., 2023),
we evaluate the knowledge generation of LLMs under both zero-shot and few-shot settings. Af- ter the knowledge acquisition, we perform QA or dialogue generation under the few-shot setting to further investigate the impact of different knowl- edge acquisition methods downstream tasks.
1) Zero-shot Evaluation: We test with varied prompts and report peak performance. A prompt could be "Generate Wikipedia knowledge for the query. (query)". Prompts tried are in Appendix F, 2) Few-shot Evaluation: We construct the prompt with k randomly chosen samples from the training set. The example templates used for knowl- edge generation are listed in Appendix F and G.
4.2 Experimental Setups
Baselines Compared with a popular retrieval- based model, DPR (Karpukhin et al., 2020), we evaluate knowledge generation with the three differ- ent types of LLMs, including FLAN-T5 (Wei et al.. 2022), LLaMA (Touvron et al., 2023), and Chat- GPT (Ouyang et al., 2022). By default, we report the results with the largest size of each LLM and adopt greedy decoding in our experiments for re- producibility. Details are presented in Appendix C.
4.3 Overall Evaluation
Table 2 and Table 3 summarize the evaluation sults of DPR and three LLM--based knowledge generators on NQ and WoW datasets, respectively. There are several notable observations as follows:
Datasets We evaluate the generated knowledge on two widely-studied benchmark datasets, includ- ing 1) Natural Questions (NQ) (K (Kwiatkowski et al., 2019), an open-domain QA dataset; and 2) Wizard of Wikipedia (WoW) (Dinan et al., 2018) a knowledge-grounded dialogue dataset. During experiments, we randomly sample 500 examples from the NQ and WoW test sets respectively for evaluation. Details are presented in Appendix D.
Generated knowledge exceeds retrieved knowl- edge in most evaluation perspectives, except the factuality and informativeness. In both NQ and WoW scenarios, LLMs show remarkable capabil- ities in generating highly relevant and coherent knowledge. Moreover, the knowledge generated by L.MMs also proves to be more beneficial for downstream tasks, regarding both helpfulness and validity. These results highlight the significant ad-
Implementation Details All the adopted models in CONNER are introduced in Appendix E.

---

Model
Query
Acquired Knowledge
Answer Billy Hill v
LLaMA
Who wrote the song the glory of love?
"The Glory of Love" a song written by Billy Hill, recorded by American singer Benny Goodman in 1936. The song has been covered by many other artists, including a version by American singer Peggy
Lee with Bing Crosby and the Johnny Mann Singers 1959.
FLAN-TS
Who write the song the glory of love?
The Glory Love is a song written by the American composer and lyricist Irving Berlin.
Trving Berlin x
DPR
Rain sleet or snow that contains a high concentration of
lce pellets ure form of precipitation consisting of small, translucent balls of icc. This form of precipitation is also referred to as ""sloct"" by the United States National Weather Service. (In British English ""sleet"" refers to mixture of rain and snow)...
icy ruin X
acids is called?
Table 4: Factuality of acquired knowledge may not influence the validity of the answer. Red words represent factual errors in critical information, while blue words represent factual errors in non-critical information.
Instrinsic
ble 4, which intuitively shows that the presence of factual errors in non-critical information has mini- mal impact on downstream tasks, while it is highly impossible derive the correct answer from the irrelevant retrieved knowledge. While LLaMA and
Model Extrinsic
Fact.
Rel.
Coh-sent. Coh-para.
Info.
DPR helpful.
0,10
0.24
0.07
-0.03
-0.14
validity
0.04
0.191
0.04
-0.06
40.09
helpful.
0.14
-0.05
0.10
-0.09
-0.05
L.MSS
validity
0.151
-0.02
0.07
+0.03
-0.03
ChatGPT generate knowledge with slightly lower factuality than DPR, it is shown to be adequate for downstream tasks. At this point, the relevance of the acquired knowledge is more critical. Hence, relying solely on the factuality of the knowledge itself is unreliable means of assessing its impact on the factuality of downstream tasks. Motivated by this finding, we investigate approaches to guid- ing the generated knowledge selection with the multi-perspective evaluation outcome of CONNER for improving the downstream performance in f 5.
Table 5: The Somers' correlation between intrinsic and extrinsic metrics on NQ. Scores with p-catue < 0.05 are marked with 1. Bold results denote the most corre- lated intrinsic metric to the concerned extrinsic metric. The breakdowns of all correlations are in Appendix H.
Instrinsic
Model Extrinsic
Fact. Rel.
Coh-sent. Coh-para.
Info
DPR helpful..
0.01
0
0.10t
-0.03
-0.14"
validity
-0.01 -0.06
0.131
-0.12
-0.13
helpful.
0.06
0.05
0.10
0.00
-0.16
DPR falls short of retrieving relevant and help- ful knowledge for knowledge-groundee dia- logues. As the DPR model is finetuned QA datasets match question to Wikipedia knowl- edge, the DPR model struggles to match dialogue utterances with the necessary knowledge. Also, the candidate Wikipedia passages in DPR (100 to- kens) are much longer than the knowledge needed in WoW, containing much redundant information.
L.MSS
validity
0.24 0.09
0.05
-0.02
-0.07
Table 6:
The Somers' correlation between intrinsic and
extrinsic metrics on WoW.
vantages of utilizing LLMs as knowledge genera- tors in terms of knowledge quality and applicability, rendering them valuable knowledge resource for various
applications.
Despite obtaining lower factuality than retrieved knowledge, generated knowledge contributes more the factuality of downstream tasks (i.e., higher validity). To investigate the underlying reason, we analyze the correlation between differ- ent intrinsic metries and extrinsic metrics on two tasks. As shown in Tables 5 and 6, the perfor- mance of downstream tasks is indeed hindered by the issue of factuality in the generated knowledge from LLMs. However, for retrieval models (e.g.. DPR), limitations may arise from the relevance and coherence of the retrieved knowledge, while its high factuality fails to ensure the performance downstream tasks. We present a case study in Ta-
This reveals the shortcomings of supervised dense retrieval models, such as limited transferability and being constrained by knowledge bases.
Few-shot in-context learning for LLMs gener- ally harms the factuality of generated knowl- edge. We observe that the length of knowledge generated by few-shot ICL is generally longer than that of zero-shot prompting since the ground-truth knowledge for demonstrations is relatively long. Consequently, LLM is more error-prone (see the analysis of long-form generation in 5 4.4). This in- dicates that few-shot ICL is not always better than zero-shot ICL in knowledge generation, and the selected demonstrations attach great importance.

---

(a) Model scale of FLAN-TS (b) Model scale of LLaMA
Fact.
Pact.
16-
Rel
Val.
Rel
Val.
Factuskcon.
0.1
Non-varifled
Factualincon.
65B
118
338
3B
07-
7B
0.88
Con
Coll.
Help.
Pegesteu crowtedge
Inowedge
Figure The impact of knowledge frequency and length on the factuality of the generated knowledge.
Figure 3: Performance on NQ with different sizes of FLAN-T5 and LLaMA as the knowledge generator (Help. and Val. scores are linearly scaled).
Inspired by this, we investigate approaches to guid- ing the few-shot demonstration selection with the evaluation outcome of CONNER for improving the performance of few-shot ICL in $ 5.
Figure 2(b) displays the factuality performance based on the number of sentences in the gener- ated knowledge. The results show that LLaMA exhibits higher error rates when generating long- form knowledge. Therefore, prompting the LLMs to generate the required knowledge in a concise rather than lengthy manner can benefit factuality.
FLAN-T5 fails to be a qualified knowledge gen- erator since its generated knowledge is poorly factual and rarely helpful to downstream tasks. Although FLAN-T5 (1 (IIB) significantly surpasses many models of the same scale through instruction tuning on numerous tasks, falls short of being a qualified knowledge generator. As shown in Ta- ble 4, such a low factuality leads to frequent oc- currences of factual errors in critical information, thereby harming downstream tasks. To this end, we study the scaling of performance w.r.t different perspectives by varying the model size in & 4.4.
Impact of Model Size Figures depicts the per- formance scaling with the model size, including (AMMA-65833BB 7B and LLAN-T5-11B//4 BB/BB/////0 780M. The results are reported on the NQ dataset using zero-shot prompting. We observe that larger mod- els do not necessarily outperform smaller models in terms of intrinsic evaluation (particularly when pa- rameter magnitudes are similar). However, larger models consistently outperform smaller models in terms of extrinsic evaluation(hellfuuness and valid- ity). Detailed tables are presented in Appendix I.
4.4 Further Analysis
We further analyze how different factors affect the quality and reliability of the generated knowledge and discuss our findings below.
Two Use Cases of CONNER
Long-tail Knowledge We investigate the impact the knowledge frequency on the factuality perfor- mance of LLaMA on the WoW dataset. Each data entry in WoW comprises a topic, query, knowledge, and answer. The topic indicates the corresponding Wikipedia page linked to the knowledge. We as- sess this knowledge's frequency using Wikipedia pageviews from 2015 to 20213. This enables to differentiate between common and long-tail knowl- edge in WoW. Our findings reveal that LLaMA exhibits lower reliability when it is expected to gen- crate rarc/long-tai tail knowledge compared to com- mon knowledge, as depicted in Figure 2(a).
To explore how framework can guide the future design of utilizing LLMs as a knowledge generator, we design two strategies to employ CONNER as a measurement for guiding the Prompt Engineering and Knowledge Selection for knowledge-intensive tasks. We define the overall quality of knowledge k given the query g follows:
Quaalg, k) 7T Sint 7 R
Sintr = [Sract, Sres, Soon, Santo]
where Qknow is the linear combination of four in- stinct metrics Sintr and y is the coefficient vector.
Prompt Engineering We show how to use CONNER to improve knowledge generation by per- forming prompt engineering for few-shot ICL. We random sample small set of m samples from the training set, then use Q k) as the scoring function to select the top n samples to compose the few-shot prompt. As shown in Table 7, the
Long-form Generation We investigate the im- pact of generation length on the factuality of the generated knowledge. Specifically, we consider knowledge over 40 tokens and take sentences as evaluation units aligned with factuality evaluation.
Chttps://wikineer http://wwkkmmmmaaa.rrrrgaraesss/ wrg//pi/rest..v.

---

Model
Fact.
Rel.
Coh.
Info.
Metric
DPR
FLAN-TS
LLaMA
ChaiGPT
ChaiGPT
85 85.%% 87.7% %
0.462
0.863
0.792
Factuality
0.65t
0.66
0.661
0.631
ChatGPT select prompe
0.503
0.899
0.775
Relevance
0.69t
0.37
0.55t
0.541
Table 7: CONNER-guided demonstration selection im- proves the intrinsic quality generated knowledge.
Coherence
0.531
0.58
0.44
0.491
Informative
0.301
0.17
0.35
0.321
Model
Helpfulness
Validity
Helpfulness 0.75t
0.45*
0.81
0.69t
ChatGPT
0.1461
43.45% 44.28%
Validity
0.831
0.731
0.85t
0.821
ChatGPT select knowledge
0,2090
Table 9:
Somer's D correlation of metrics with the
Table CONNER-guided knowledge selection improves extrinsic (downstream) performance.
human annotation on NQ (The results on WoW are presented in Appendix J.2). Correlation scores with p-value 0.05 are marked with '
knowledge generated by CONNER-enhanced few- shot prompting outperforms that with random demonstrations on 3 out of perspectives, under the setting of m 30 and n 8.
Metric
DPR
FLAN-T5
LLaMA
ChatGPT
Factuality
0.651
0,66
0.66
0.63
HE
-0,24
0.15
-0.03
0.29t
NLI
0.23
0.47
0.27
0.38t
Knowledge Selection We employ CONNER to im- prove downstream tasks by selecting high-quality generated knowledge. Specifically, we generate different knowledge H.- {ki, ..., } from LLMs with top-p sampling, then select the generated knowledge for the downstream task, according to k = argmax (g, (g, R). As shown in Table 8, we achieve a relative improvement of 43.15% in helpfulness on ChatGPT with p = 0.9 and 5.
NLI-Multitask
0,18
0.51
0.26
0.32
N.I-Decompose,
0.23!
0.47
0.27
0.381
Relevance
0.69t
0.37
0.55
0.54
F1
0.45
0.21
0.41
0.47t
Validity
0.831
0.73
0.85
0.82t
EM
0.591
0.51
0.54
0.61
F1
0.74
0.67
0.76
0.771
Table 10:
Comparing CONNER with referrence-reliant baseline metrics on the NQ dataset. Details of baseline metrics are presented in Appendix J.3.
6 Human Evaluation
knowledge and human knowledge. (2) CONNER met- rics consistently outperform all other reference- reliant metrics, indicating the effectiveness of our framework in the knowledge evaluation scenarios.
We conducted a human evaluation by randomly se- lecting 400 samples from the NQ and WoW test sets. Our three annotators provided ratings for the intrinsic and extrinsic metrics for the four mod- cls. Additionally, for FLAN-T5 and LLaMA, we annotated the specific locations of factual errors in the generated knowledge, aiming to facilitate future research on fine-grained fallacy detection. Detailed annotation instructions and the statistics of our labelled data can be found in Appendix J.1.
Conclusion
In this work, we introduce CONNER, a comprehen- sive evaluation framework designed to automati- cally assess both the intrinsic quality and extrinsic reliability of the knowledge generated by LLMs. Notably, CONNER is reference-free but demonstrates better correlation with human judgement com- pared with previous reference-relin metrics.
To evaluate how well CONNER matches human evaluation of knowledge and compares with several baseline metrics, we measure the Somers' D cor- relation (Somers. 1962) between the human rating 0, 1, 2 of the knowledge quality and corresponding metric scores. Table 9 and Table 10 illustrate the re- sults of four models on the NQ dataset. We observe that: (1) CONNER yields consistently good correla- tions with human evaluation w.r.t different eval- uation perspectives (except for informativeness). which indicates that the quality of knowledge can be more effectively evaluated with CONNER. The inconsistency between informativeness and human judgment is attributed to the differences in model
Through extensive evaluation and in-depth analy- sis, we identify several key factors affecting the fac- tuality of generated knowledge. We find although the generated knowledge is less factual than the retrieved knowledge, it remarkably enhances the factuality of downstream tasks over the retrieved ones. Furthermore, we propose two approaches to improve knowledge generation and downstream task performance with the guidance of CONNER. We believe our framework and findings will facilitate the future research trustworthy AIGC.

---

Limitations
data that targets fine-grained factual errors. Despite these limitations, we believe our work serves as significant catalyst for the automated evaluation of knowledge generated by large lan- guage models, contributing positively to the ad- vancement of more trustworthy AI systems.
In this section, discuss the limitations in this work from three perspectives.
Firstly, the knowledge we evaluate primarily re- lies on information sourced from Wikipedia. This choice is driven by two considerations: (1) Large language models (LLMs) trained on diverse corpora, which may include undisclosed domain- specific or task-specific data. To ensure fairness in our evaluations and enable meaningful compar- isons, we focus on the common data sources that all models have learned from, with Wikipedia be- ing prevalent pre-training corpus for different LLMs. (2) Wikipedia is renowned for its high- quality knowledge, providing us with authoritative evidence to validate the generated knowledge. Ad- ditionally, leveraging such authoritative evidence enhances the interpretability of our factual judg- ments. In future work, we aim to expand our evaluations to include a broader range of world knowledge, thus further enhancing the scope and generalizability of our findings.
A Acknowledgements
We extend our sincerest gratitude to Professor Jing Ma, whose insightful discussions and suggestions on factuality evaluation have significantly inspired our design. We are particularly grateful to our three anonymous reviewers, whose thorough and meticulous reviews have considerably improved the quality of our work. Their constructive discus- sions and insights have undoubtedly enhanced our revisions. This research work is partially supported by CUHK under Project No. 3230377 (Ref. No. KPF23GW20).
References
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autore- gressive language modeling with Mesh- Tens-Teenoo..ow.
Secondly, while work primarily aims pro- pose a general framework that can be applied to any language, our evaluation framework presents potential generalization challenges for Lon-English languages. This due to its reliance on several common NLP components, a limitation echoed across many NLP methodologies. Encouragingly, the development of model variants in other lan- guages, such as Chinese (Hu et al., 2020; Xie et al., 2023; Huang et al., 2017), indicates the potential for broader applications. Nonetheless, the reality remains that for very low-resource languages with- out existing NLP models, these components may need to be developed from scratch. This issue rep- resents a challenge that the community needs to address in the future.
Virginia Braun and Victoria Clarke. 2012. Thematic analysis. pages 57-71.
I Liang Chen, Hongru Wang, Yang Deng, Wai Chung Kwan, Zezhong Wang, and Kam-Fai Wong. 2023. Towards robust personalized dialogue generation via order-insensitive representation regularization. In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 7337-7345, Toronto, Canada. Association for Computational Linguistics.
Yang Deng. Wengiang Lei, Minlie Huang, and Vat-Seng Chua. 2023. Goal awareness for conversational Al: proactivity, non-collaborativity. and beyond. In Pro- ceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguisticx: Tutorial Ab- stracts, ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1-10. Association for Computational Linguis- tics.
A third limitation is that our assessment of factuality is limited to sentence-leve granularity. Through analysis and manual annotation, we have identified that large language models (LLMs) tend to exhibit errors at a more detailed level, particu- larly concerning numbers, time, and the generation of misleading or fabricated concepts (e.g., key char- acters, identities, and locations), particularly within parallel structures. To address this limitation, fu- ture research will concentrate on developing more fine- ine-gnained methods for detecting hallucinations and assessing factual accuracy. To facilitate such research, we have annotated a specific subset of
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-poowe poweeed conversational agents. arXiv preprint 1811v:
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Os- Zaiane. Mo Yu, Edoardo M. Ponti, and Siva Reddy, 2022. Faithdial: faithful benchmark for information-seekknng dialogue.
John Glover, Federico Fancellu, Vasudevan Jagan- nathan, Matthew R. Gormley, and Thomas Schaaf. 2022a. Revisiting text decomposition methods for

---

nli-based factuality scoring of summaries. CoRR. abs/2211.16853.
Kaplan. 2022b. Language models (mostly) know what they know. CoRR, abs/2207.0522 222.0221....
John Glover, Federico Fancellu, Vasudevan Jagan- nathan, Matthew R. Gormley, and Thomas Schaaf. 2022b. Revisiting text decomposition methods for nli-based factuality scoring of summaries.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Tidunox, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-678 6781.
Or Honovich, Leshem Choshen, Roee Aharoni, Ellaa Neeman, Idan Szpektor, and Omri Abend. 2021. q²: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. Intereetauugmente dialogue generation.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020a. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9332-9346, Online. Association for Computa- tional Linguistics.
Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. OCNLI: Orig- inal Chinese Natural Language Inference. In Find- ings the Association for Computational Linguistics: EMNLP 2020. pages 3512-3526, Online. Association For Computational Linguistics.
Guimin Huang. Min Tan, Sirui Huang, Ruyu Mo, and Ya Zhou. 2017. A discourse coherence model for analyzing chinese students' essay. In 2017 Interna- tional Conference on Progress in Informatics and Computing (PIC), pages 430-434.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020b. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 9332-9346.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL 2021, pages 874-880.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quae Le, and Slay Petrov. 2019. Natu- questions: benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452 466.
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky, 2023. Query expan- sion by prompting large language models. CoRR, abs/2305.03653.
Ziwei Ji, Nayeon Lee, Rita Frieske Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv, 55(12).
Nayeon Lee. Wei Ping. Peng Xu, Mostofa Patwary, Pas- cale Fung, Mohammad Shoeybi, and Bryan Catan- zaro. 2023, Factuality enhanced language models for open-ended text generation.
Prathyusha Jwalapuram, Shafiq R. Joty, and Xiang Lin. 2021. Rethinking self-supervision objectives for generalizable coherence modeling. CoRR, abs/211
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäsche. Sebastian Riedel, and Douwe Kiela. 2020.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022a. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522.
generation for coowledge-nntenstee NLP tasks. Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian- Nie, and Ji- Ji-Rong Wen. 2023. Halueval: A large- scale hallucination evaluation benchmark for large language models.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hoddield-Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai. Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jack- son Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared
Yanyang Li, Jianqiao Zhao, Michael R. Lyu, and Li- wei Wang. 2022. Eliciting knowledge from large pre-tran pre-trained models for unsupervised knowledge- grounded conversation. In Proceedings of the 2022 Conference Empirical Methods in Natural Lan- guage Processing. EMNLP 2022, pages 10551- 10564.

---

Nelson Liu, Tianyi Zhang, and Percy Liang. 2023a. Evaluating verifiability in generative search engines.
Fabio Petroni, Aleksandra Piktus. Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021. Kilt: benchmark for knowledge in- tensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-22 2544.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan, 2023b. On learning to summarize with large language models as references. CoRR, abs/2305. 14239.
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Multi-stage prompting for knowledgeable dialogue generation.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Potsawee Manakul, Adian Liusie, and Mark J. Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. CoRR, abs/2303.08896
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. On faithfulness and factu- ality abstractive summarization. the 58th Annual Meeting of the Association for Com- putational Linguistics, ACI. 2020, pages 1906-1919.
Proceedings of
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension text.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer. Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.
Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models.
Factscore: Fine-grained atomic evaluation of factual precision long form text generation.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/21 12.01488.
Rodrigo Frassetto Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. CoRR, abs/1910. 14424.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624 224-643, Online. As- sociation for Computational Linguistics.
OpenAL. 2023, GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll I.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 7784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: benchmark for factuality metrics. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 481 3812-40 12-29929, Online. As- sociation for Computational Linguistics.
Robert Somers. 1962, A new asymmetric measure of association for ordinal variables. American sociolog- ical review, pages 799- 811.
James Thorne, Andreas Vlachos, Christos Christodoollppulls, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang. Min-Yen Kan, and Preslav Nakov. 2023. Fact-checking complex claims with program- guided reasoning.
Baolin Peng. Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard

---

Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Prob of factual consistent (%6)
70
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics.
60-
50-
80
30
20
10
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational L Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics.
knowledge
Num of retrieved evidence
Figure 4: The influence of reference knowledge (e.g.- the annotated Wiki. document in WoW datasct) in fac- tuality evaluation weakens as the amount of retrieved evidence increases.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. GPT-NER: named entity recognition via large language models. CoRR, 01222304404228
Appendix
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners.
A Details of Problem Formulation
We provide formulation of the two-step process for knowledge-intensive tasks, illustrated in Fig.1. Formally, the knowledge-intensive gener- ation problem can be formulated the following chain rule:
Xiaohui Xie. Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yigun Liu, and Jin Ma. 2023. T2ranking: large-scatl scale chinese benchmark for passage ranking.
P(alg Σ, P(Ala
(10)
Boyang Xue, Weichao Wang. Hongru Wang, Fei Mi, Rui Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2023. Improving fac- tual consistency for proweedge-rrounder dialogue systems via knowledge enhancement and alignment.
where P(k|g, K.) is the knowledge acquisition pro- cess and P(a|g. k) = II P(ay|a1ı 1, k) is the autoregressive answer generation process of the reader model based on the acquired knowledge. Retrieval-based knowledge acquisition methods use retrieval model to retrieve the most rel- evant knowledge from the knowledge resource K. = {di, d2,...,,d
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators.
,dk] composed K documents:
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's 's song in the ocean: A survey hallucination in large language models.
sim(gdd
P(k dl..
(11)
where sim(-) function is used measure the sim- ilarity, e.g., cosine similarity, between the query and the knowledge document.
Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Poiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models.
Generation -based knowledge acquisition meth- ods prompt a large language model to directly gen- erate the required knowledge:
P(klg. II
prompt) (12)
where prompt denotes the zero-shot few-shot prompt and the LLM is regarded as the knowledge resource IC and Pk stands for the distribution in- duced by the LLM.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ring Wen. 2023. Large language models for information retrieval: A survey.

---

Dataset
Prompts
Best
Topic: [topic] 'nn Query: [query] 'nn Related wikiiedia knowledge:
Topic: [topic] inn Generate background document from Wikipedia to answer the given question. 'n (query)
NQ
Topic: [topic] inn Generate Wikipedia knowledge to answer the given question. in Question: [query] 1an Wikipedia knowledge:
Topic: [topic] (n Generate Wikipedia to answer the given question. (a Question: [quety] 1/a Wikipedia:
Topic: [topic] la Query: [utterance] 'n Related Wikipedia knowledge:
Wikipedia reply to the utterance. 'an [utterance] 'n Topic: [topie] lan Generate Wikipedia knowledge to answer the given question.) y Utterance: [utterance] \an Wikspedia knowledge:
Topic: [lopic] 'n Generate a hackground document
Wow
Topic: [topic] (u Generate a Wikipedia answer given question ja Question: [ulterance]] Wikipedia:
Table 11: List of human prompts we tried for zero-shot knowledge generation, evaluated on the validation set of NQ, WoW. I represents placeholder, and 'utterance" denotes the last utterance of the dialogue partner. We use v to denote the prompt achieving the best performance.
Dataset
Prompts
Best
Topic: [topic] \mn Query: (query] Reluted Wikipedia knowlledge: [knowledge]
Topic: [topic] |n Query: Iquery) 'nn Knowledge: (knowledge)
NQ
Toppc: [topic] Query: (query) Document: (knowledge)
Topic: [topic) 'an Gieneraee a background document from Wikipedia to answer the given question. \/2 (query) J 'nn (knowledge)
Topic: (topic) 'n Generate Wikipedia to answer the given question. (n Question: Iquery) 'n Wikipedia: [knowledge]
Topic: [topic] (m Query: [utterance] 'n Related Wikipedia knowledge: [knowledge)
Topic: [topic] \n Query: [utterance] in Knowledge: [knowledge]
Wow
Topic: [topic] ia Query: [utterance] yn Document: (knowledge)
Topic: [topic] a Generate background decument from Wikipedia to reply the utteranee. 'a [utterance) 'a (knowlodge)
Topic: [topic] in Generate a Wikipediaa to unswer the given question. i Question: (utterance) (nn Wikipediin (knowledgge)
Table 12: List of example templates we tried for few-shot knowledge generation.
Datsset
Prompts
Best
Topic: [topic] 'ys Presage: [knuwlldge) 'nn Query: [query] 'nn Janswer)
Topic: [topic) in Read the passage answer the question belon:) Passage: [knowledge] 'n Question: [query] Anewer:: [answer]
NQ
Topic: [iqiic] ( Using the knowkelge finm the passgg in amewer the quelive heloel Pavaage [knowlllge] la Question: [query) 'n Answer (anower)
Topic: [lopic] 'a Passage [knowklgge] la Spesker 1: [stierance) 'un Speoker 2: (response]
Topic: [ropic] Knowledge: (knowalgee) (n Speaker I: [uttennee]] (a Speaker [response]
WoW
Topic: (lopic) Grounding document: (knowlolge) in Speaker I: [uttemmoe] (n Speake (response) Paesag: (engeledlee] ledge] Query: (untennce in [knowkdgee] Speaker (stierance) Speaker [response]
(requnne)
Topic: [topic] bo Using knowledge
passage, complete dialogue
Table 13: List of example templates we tried for few-shot answer generation.
B Analysis of Reference Knowledge
knowledge.
We investigated the importance of reference knowl- edge in evaluating the factuality of generated knowledge. Specifically, we conducted FLAN- T5 experiments on the WoW dataset using zero- shot approach. Two sets of experiments were per- formed: one included reference knowledge in the retrieved evidence pool, while the other did not. Figure 4 illustrates our findings, indicating that the group with reference knowledge exhibits a clear advantage when the number of retrieved evidence is limited. However, as the number of retrieved evidence increases, the performance of both groups converges. These results suggest that reference knowledge is dispensable, particularly when sig- nificant amount of evidence is available. When the number of retrieved evidence surpasses ten, the impact of reference knowledge becomes negligible. We hope this will provide valuable insights for fu- ture designs of factuality assessment for generated
C Details of Baselines
DPR (Karpukhin et al., 2020) is a supervised dense retrieval model trained on several QA datasets (in- cluding NQ) to retrieve the most passages given a query.
most relevant Wikipedia
FLAN-T5 (Wei et al., 2022) is enhanced ver- sion of T5 (Raffel et al., 2020) that is instruction- finetuned in 1.8k NLP datasets to acquire the gen- eralization ability to unseen tasks.
LLaMA (Touvron et al., 2023) is an foundation language model trained on publicly available datasets and shows competitive perfor- mance with the best models, including GPT-3 (175B) and PaLMM
ChatGPT is sibling model to InstructGPT (Ouyang et al., 2022) that is trained to follow in- structions in a prompt and provide a detailed re- sponse. We adopt sott-davinci-00. version for

---

evaluation.
H Detailed Correlations between Intrinsic and Extrinsic Metries
D Details of Datasets
We listed the detailed correlations between intrinsic and extrinsic metrics for LLaMA, FLAN-TS, and ChatGPT on the NQ dataset in Table 15.
Natural Questions (NQ) (Kwiatkowski et al., 2019) is an open-domain QA dataset, where the questions are mined from real Google search queries. The corresponding ground truth knowl- edge and the answers to the questions are para- graphs and short spans in the Wikipedia pages.
Table of Model Size Impact
We list the specific numerical values of perfor- mance scaling with the model size in Table 16, including LLaMA-6SB/33B/T and FLAN-T5- I1B/3B/780M.
Wizard of Wikipedia (WoW) (Dinan et al., 2018) is a knowledge grouudde dialogue dataset de- signed for informationn-seeking scenarios, where one speaker introduces knowledge related to topic to the other speaker by grounding his/her responses in specific sentence from a Wikipedia page.
Details of Human Evaluation
J11 Human Annotation
We conducted a human evaluation with 400 sam- ples from the NQ and WoW test set. Among these, 320 samples were from the zero-shot setting in the NQ dataset, involving all four models, while 80 samples were from the few-shot setting in the WoW dataset, involving model (ChatGPT), Three pert annotators, who were familiar with the tasks, were employed rate the acquired knowledge and generated answers based on four intrinsic perspec- tives and two extrinsic perspectives. Each perspec- tive was scored on a scale of 0, 1, or 2, representing unacceptable. acceptable, and excellent, respec- tively. The average kappa value of the annotation is 0.612 on 20% cross-annotatio data. The detailed instructions for the human annotation be found in Table 17.
E Implementation Details
All the metries designed are model-based met- rics, utilizing solely off-the-shelf models. We present the models used in Table 14.
F Prompts for Knowledge Generation
F.1 Zero-shot Prompts
our experiments, we observed that zero-shot prompting was highly unstable. Therefore, we con- duct experiments using multiple human prompts and select the most effective ones for the WoW and NQ datasets. The human prompts evaluate listed in Table 11.
Note for factuality assessment, the reliable evi- dence for the generated knowledge is acquired by the following process: For each sentence k, we use it as the query to search Google, and regard the topl Wikipedia webpage as a reliable knowl- edge source to verify the factuality of this sentence. Another point worth noting is that for the evalua- tion of validity in WoW, reused the factuality evaluation process since the responses in WoW are open-ended.
F.2 Few-shot Prompts
In the few-shot setting, our prompt is constructed using k randomly chosen examples from the train- ing set:
prompt (example; An... exampley, An sxampletest
The example templates utilized for knowledge gen- eration are provided in Table 12. Please note that exampletest differs from example, as it does not contain (knowledge) in the placeholder.
J.2 Human Evaluation Results on WoW
Based on the provided annotations, we assessed the correlation between ChatGPT's automatic metries and human judgement on the WoW dataset. The results are presented in Table 18.
G Prompts for Answer Generation
J.3 Baseline Metries
We adopt few-shot prompting on the LLaMA model answer generation and the example tem- plates used for answer generation are provided in Table 13.
We compared it with three reference-reliant metrics in knowledge evaluation. Their definitions and calculation methods are as follows:

---

Metric
Model
Link
NLJ-RoBERTe-large ColBERTv2
Factuality
Relevance
BFRT-ranking-Be
GPT-neo-2.7B
Coherence
Momentum
Informativeness
GPT-neo-2.7B
Helpfulness
LLaMA-65B
NLI-RoRFRTaa-arge ColBERTv2
Validity
ht http::///ggtt....ooo///sooooottr
Table 14: List of all models that we use in designing our framework.
Instrinsic
ence knowledge. We adopt audopt-HNI HNE when comput- ing the correlation with human judgement.
Model
Extrinsic
Fact.
Rel.
Coh-sent. Coh-para. Info
Entailment Ratio (ER) (Lee et al., 2023) also in-
FLAN-TS helpful.
0.15' -0.21*
0.20*
-0.21*
0.02
validity
0.23* -0.16"
0.14*
-0.10*
0.07
troduces an NLI-based approach to assess factual knowledge by measuring its entailing relationship with ground-rrutbreference knowledge. The en- tailment ratio is computed as follows: Entaile= ENTAlLgea| |ALLgeal, where ALLgen is set of all generated knowledge, and ENTAIl-gn is the set of generated knowledge that can be entailed by the NLI model. Specifically, we use the entailment probability of each example as its ER score.
helpful.
0.03
0.05
0.06
-0.09t
-0.01
LLAMA
validity
0.091 0.07
0.05
-0.66
-.03
CHATGPT helpful.
0.16
0.03
0.08
0.02
-0.04
validity 0.22 0.13*
0.02
0.091
0.03
Table 15: The Somers' correlation between intrinsic and extrinsic metrics in zero-shot setting on NQ. Corre- lation scores with y-value < 0.05 are marked with f.
F1 of knowledge (F1) (Liu et al., 2022) employs a unigram F1 score to evaluate the quality of gener- ated knowledge. This metric measures the overlap between the generated knowledge and the reference knowledge by evaluating word-level matches. By assessing the degree of agreement, the Fl metric provides an estimation of the knowledge quality, specifically from relevance perspective.
Model
Size
Fact.
Rel.
Coh.
Info
Help.
Val.
65B
0.942
0.732 0.824
0.757
0.219
0.420
LLaMA
33B
0.656
0.633 0.734
0.608
0.203
0.402
7B
0.773
0.626 0.805
0.662
0.1
0.375
11B
0.584
0,685
0.778
0.673
00.14
FLAN-T5
3B
0,657
0.663 0.816
0.708
-0.155
0.324
780M
0.506 0.729 0.793 0.729
162
0.252
Table 16: Performance on NQ with varying sizes of FLAN-T5 and LLaMA as knowledge generators. The max(O,) .) operation in Eq.6 has been excluded to empha- size the sequential relationship among different sizes of FLAN-TS. Bold and Underlined results represent the best and second-best performances for each model, respectively.
-LL-week-supervised (Kryscinski et al., 2020b) train a classification model on constructed data to perform consistency checking on (document, sentence) pairs. We chose the factCC version as our baseline.
NLJ-decoppos-claim (Glover al., 2022b) found that in general, sentence-level decomposi- tion is preferable for the hypothesis side of the NLI input. So we also decompose the generated knowledge into sentences and then aggregate the sentence-level scores to produce a document-level score.
Hallucinated NE Ratio (HE) (Lee et al., 2023) proposed a NE-based metric that is designed with an intuition that a model is hallucinating (making factual errors) if it generates an NE that does not appear in the reference knowledge source. The NE-based metric can be calculated as: HNE |HLLLUNE| ALLNE] where ALLNE is the set of all the NÈs detected in the LM generation, and HALLLUE a subset of NE that does not appear in the reference Wikipedia knowledge. Note that evaluating NEER requires the existence of refer-
NLI-multitask fine-tunes the DeBERTa-v3-large model FEVER and two NLI datasets.
Exact Match (EM) (Rajpurkar et al., 2016) use Ex-
Match measure the percentage of predictions that match its ground truth answers exactly.

---

Dimension
Value
Description
All seutences in are factually accurate and the information in them can be verified with reliable evidence. & contains least one seutence with non-verified information, while others are factually accurate. factual error that is incomsistent with reliahle knowledge.
Factuality
It contains least sentence with
&: is highly relevant the topic and query/utterance.
Relevance
& is relevant to the topic but less relevant to the querylatterance. & irrelevant
topic querylutterance.
& is very coherent and flucut (do consider truncation end due the maximum gencration length).
Coherence
k: has some minor incoherence lack of fluency, phrase or sentence repetition, but it does not aflect understanding. & has significant cohereace and flueacy issuca that are hard to understand.
contains informative content that you know before. & contains limited trivial information against your knowledge.
Informative
k fails to provide any meaningful information.
k directly provide ides or comaiis the correct answer.
Helpfulness
& indiroctly help generating conrect
& does not contain any usetul information for the correct answer.
The answer generated based on & is correc.
Validity
correctness generated answer camnot determined.
The answer generated based & is completely incorrect.
Table 17: Annotation guideline of LLM generated knowledge.
Model
Fact.
Rel.
Coh.
Info.
Help.
Val.
CharGPT
0.71
0.57
0.52
0.40
0.79
0.54
Table 18:
Somer's D correlation of metrics with the human annotation on WoW ). p-value for all results are 0.05. We report the maximum for coherence.


Reviews:
[{"review_id": "CrA6n4kYFp", "reviewer": "Reviewer_rAA7", "summary": "", "questions": "**Question A**: Whether the Helpfulness score is bias?\n\nIn Line 315, the intuitive baseline of Helpfulness is generating a response without any external knowledge instead of randomly sampled knowledge. The unrelated knowledge can negatively impact response generation. If so, the helpfulness metrics may be inflated. I am wondering whether these two types of baselines perform equally.\n\n**Question B**: Could you describe the meaning of evidence, the reason to collect evidence, and the collection method In Line 347?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and positive feedback! Your recognition of our work's importance, its role in bridging the knowledge evaluation gap, and its potential to facilitate future studies is deeply encouraging. We're also pleased that you acknowledged the thoroughness of our evaluations and the instructive observations they provide. Following this, we will sequentially address your comments and supplement our work with new experimental results.\n\n\n#### **Response for ​​\"evaluation framework is hard to generalize to languages besides English\".**\n\nIndeed, our framework relies on several common NLP components, possibly presenting generalization challenges for non-English languages, a dilemma faced by most NLP methodologies. However, model variants have been developed in other languages, such as Chinese [1][2][3], indicating potential for broader applications. Our work primarily aims to propose a general framework that can be applied to any language, so with the development of corresponding models in non-English languages, our method can be directly applied. We appreciate your insightful comments and will include further discussions regarding this issue in the Limitations section.\n\n\n#### **Response for \"Before applying the proposed metrics to three LLMs, this paper does not validate the effectiveness of the metrics\".**\n\nActually, our paper indeed includes the 'validation of metric effectiveness'. We apologize for any confusion caused. For readability considerations, we initially placed the validation in **Chapter 6** and **Appendix.J.** The intention was to first present the exploratory findings and analyses, which we thought would hold more immediate value for readers. However, we acknowledge your insightful point about the importance of showcasing the metrics' effectiveness before applying them. We appreciate your suggestion and will make adjustments to enhance the rigour of our presentation in the revised version.  \n\n\n#### **Response for \"it is essential to validate whether the metrics align with expectations when computed on the ground-truth knowledge\".**\n\nTo better demonstrate the effectiveness of our metrics, we further supplemented our evaluation by computing our metrics on the ground-truth labels, after filtering out empty and noisy knowledge annotations in the data. The results are presented in Table 1.\n\n|Fact-con|non-verif|Fact-incon|Rel|sent-coh|para-coh|inform|helpful|valid|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|99.60%|0.40%|0.00%|0.957|0.0366|0.7803|0.9241|0.4326|81.32%|\n|\n\nTable 1: Evaluation metrics computed on ground-truth knowledge annotations\n\n\n\nThe results were as expected — higher than all model results, except for coherence. The coherence discrepancy is due to two factors. First, the Perplexity (PPL) evaluation tends to favour text generated by language models [4]. Nonetheless, this does not prevent us from using it to gauge the fluency of knowledge generated by large language models. Secondly, the text generated by large language models indeed shows better paragraph coherence than human essays as they focus on transitions and are well-structured [5].\nWe will include these results and corresponding explanations in the revised paper. We hope this response addresses your concerns.\n\n\n\n#### **Response for \"whether these two types of baselines perform equally\".**\n\nGenerating a response without any external knowledge could be another viable baseline for Helpfulness. And we did consider the baseline of generating a response without any external knowledge but due to space constraints, we did not include this in the paper. We've supplemented additional experiments in NQ and detailed the motivation for choosing random knowledge as the baseline. \n\n**Supplementary experiments.**\nNotably, our experimental results in Table 2 show these two different implementations perform comparably. And, both implementations do not affect the relative ranking of model performance. We consider this consistency might be due to the minimal overlap between the given query and randomly sampled knowledge, leading the model to disregard the knowledge.\n\n|Metrics|DPR|Flan-T5|LLaMA|ChatGPT|Flan-T5|LLaMA|ChatGPT|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| |supervised||zero-shot|||few-shot||\n|random\u000bknow|0.1236|0.0000|0.2191|0.1461|0.0000|0.2528|0.1966|\n|without\u000bknow|0.1086|0.0000|0.2057|0.1314|0.0000|0.2400|0.1827|\n|\n\nTable 2. Comparison of Helpfulness Scores on the NQ Dataset with Different Baselines\n\n**Motivation for why we use random knowledge.**\nThe reason we chose to use random knowledge as a baseline is that our setup aligns more with the setting of open-book QA.  In such scenarios, some form of pre-acquired knowledge, regardless of its quality, is always provided as input. \n\nWe truly appreciate your valuable comments about the Helpfulness score with a deep insight into our method. We will add the corresponding experiments and discussion in the final version, and we believe this additional analysis would strengthen our paper. \n\n\n\n\n#### **Response for \"describe the meaning of evidence, the reason to collect evidence, and the collection method\".**\n\n● **Evidence.** In our scenario, \"evidence\" refers to Wikipedia Knowledge that is used to verify the correctness of generated knowledge. As an illustration, consider the generated knowledge 'The first Nobel Prize in Physics was awarded to Wilhelm Conrad Röntgen in 1901'. To confirm the correctness of this statement, which we aren't immediately sure of, we need to procure external knowledge. This might involve searching the Wikipedia page about 'the first Nobel Prize in Physics' to locate information that either validates or refutes the model's statement. In this context, corresponding knowledge on the Wikipedia page serves as our evidence.\n\n● **Reason to collect evidence.** There are three primary reasons to collect evidence:\n\n(1)  We need to use reliable external knowledge as evidence to test the correctness of the generated knowledge, just like the example we have discussed above. \n\n(2)  The ground-truth knowledge in the dataset is not sufficient to fulfill this function (please refer to Table 10 for the NLI baseline result). Ideally, the real 'ground truth' should be annotated in response to model-generated knowledge. However, the 'ground-truth knowledge' in the dataset typically corresponds to annotations made against pre-existing human answers, which may not align with the content of model-generated knowledge. For instance, it might only contain the name of the Nobel Prize laureate (information from the human answer), without encompassing the year of the award (additional information from the model answer). This discrepancy underscores the necessity for a more comprehensive evidence collection to ensure the thorough and accurate evaluation of the generated knowledge. \n\n(3)  In real-world scenarios, it's not viable to have ground-truth knowledge readily available, making the collection of evidence a necessary step. \n\n● **Collection Method.** To ensure comprehensive coverage of the knowledge generated by the model, the following approach is employed. Firstly, the generated knowledge is decomposed into individual sentences. Each sentence is then utilized as a query, with Wikipedia serving as the knowledge base, and ColBERTv2 acting as the retrieval model. The objective is to retrieve several of the most relevant evidence related to each sentence in order to verify its correctness. Once all the sentences have been validated, the results are aggregated using formula 1 to determine the overall correctness of the entire knowledge.\n\nWe thank the reviewer for such valuable comments, we will detail them in the revised version.\n\n\n#### **Response for typo.**\nThank you for pointing this out. We will correct them in our revised version.\n\n&nbsp;\n***\n\nWe hope that our answers can address your concerns satisfactorily and improve the clarity of our contribution. We would be grateful if you could re-evaluate our paper. We look forward to receiving your further feedback.\n\n***\n&nbsp;\n\n**References**\n\n[1] Hai Hu et al. 2020, OCNLI: Original Chinese Natural Language Inference\n\n[2] Xiaohui Xie et al. 2023, A large-scale Chinese Benchmark for Passage Ranking\n\n[3] Guimin Huang et al. 2017, A discourse coherence model for analyzing Chinese students' essay\n\n[4] Yafu Li et al. 2023, Deepfake Text Detection in the Wild\n\n[5] Biyang Guo et al. 2023, How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The reliability of knowledge generated by LLMs is important. This paper proposes a comprehensive evaluation framework to bridge the gap in knowledge evaluation and releases the code for further studies.\n\n2. The paper conducts thorough evaluations and provides several interesting and instructive observations about generated knowledge of LLMs.", "reasons_to_reject": "1. The evaluation framework is hard to generalize to languages besides English.\n\nThis framework relies on several assistant models, such as an NLI model, a ranking model, and a discourse coherence model. The availability of these assistant models poses a problem in other languages and hinders generalization.\n\n2. Before applying the proposed metrics to three LLMs, this paper does not validate the effectiveness of the metrics. \n\nExcept for Section 6, it is essential to validate whether the metrics align with expectations when computed on the ground-truth knowledge, especially when the ground-truth knowledge is available in the used datasets.", "questions_to_authors": "**Question A**: Whether the Helpfulness score is bias?\n\nIn Line 315, the intuitive baseline of Helpfulness is generating a response without any external knowledge instead of randomly sampled knowledge. The unrelated knowledge can negatively impact response generation. If so, the helpfulness metrics may be inflated. I am wondering whether these two types of baselines perform equally.\n\n**Question B**: Could you describe the meaning of evidence, the reason to collect evidence, and the collection method In Line 347?", "paper_topic_and_main_contributions": "This paper tackles the problem of the evaluation of knowledge generated by LLMs. Prior evaluation methods only access the intrinsic quality of knowledge and overlook their extrinsic reliability on downstream tasks. Therefore, this paper proposes an evaluation framework named CONNER that considers six fine-grained aspects of intrinsic and extrinsic evaluations. The evaluation metrics consist of four intrinsic aspects (Factuality, Relevance, Coherence, and Informativeness) and two extrinsic aspects (helpfulness and Validity). \n\nThe main contribution is the evaluation framework that first includes extrinsic aspects and does not require human-labeled ground truth knowledge, and the insightful findings about knowledge generated by LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- According to Line 329, should the denominator of Equation (7) be the average rather than the sum?"}, "extra_text": {}}, {"review_id": "X8IzRG7giM", "reviewer": "Reviewer_2CNk", "summary": "", "questions": "A. For the ranking BERT model, which model from the relevant paper is used. The paper proposed two variants for ranking, which one was used and how was it adapted for measuring relevance?\n\nB. Line 269 typo: coh_para should be coh_sent\n\nC. Line 298 typo - S_ent ?\n\nD. For the informativeness measure entropy could be increased without the text being informative? Any text as long as it’s new will be considered informative. Is that what the expected behaviour of the metric ?\n\nE. Line 322 - clarify where the Loss is being used exactly.\n\nF. Table 2 - why is sentence coherence so low for dpr (basically all models) ? What does it mean if coherence values are so low ?\n\nG. For qualitative examples (Table 4), please include paragraphs for the same query for better comparison (dpr query is different).\n\nH. Para from line 407 - how was this correlation done? What does correlation between metrics actually reveal - more clarification on what this experiment actually measures would be good.\n\nI. Line 550 type 43.45", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer 2CNk for the detailed comments and careful reading. Your recognition of our work as the first evaluation focused on understanding the use of model-generated knowledge is inspiring. We are also heartened that you find our findings interesting and valuable to the community. Following this, we will sequentially address your comments, provide clarifications on some issues, and enhance our work with new experimental results.\n\n### **Response for Motivation**\nTo address the concern about our motivation, we first elaborate on the limitations of existing metrics for each perspective and then discuss the motivation and design process of our framework. \n\n**1. Limitations of existing metrics.** We have supplemented the motivation for the Factuality, Relevance, and Extrinsic metrics mentioned in your review due to their unique design compared to existing work.\n\n● **Factuality.** Existing factuality metrics [1,2,4] normally determine the factual correctness of generated content by assessing their consistency with golden/human-annotated references. This method proves effective for summarization tasks but exhibits significant shortcomings in the burgeoning field of knowledge generation.\n\nOne major limitation is the potential discrepancy between 'ground-truth knowledge' in the dataset and model-generated content. For instance, a model may generate knowledge beyond the reference knowledge scope. The correctness of these extra portions cannot be verified against reference knowledge, potentially leading to the low correlation between existing factuality scores and human annotations, as shown in **Table 1**.\n\nFurthermore, the absence of annotated references in real-world situations renders these metrics ineffective. This highlights the urgent need for more adaptive factuality evaluation methods in the evolving domain of knowledge generation.\n\n\n● **Relevance.** Previous relevance metrics usually rely on word overlap or similarity with human-written references [6,7]. However, in the context of knowledge generation, these metrics encounter two significant drawbacks:\n\nFirst, They struggle to align well with the knowledge acquisition scenario, where large language models (LLMs) act as generative search engines. The main goal here is to assess the relevance of the generated knowledge to the given query, similar to a search engine's role. In this context, measuring similarity with annotated knowledge is indirect and may not accurately represent relevance, potentially leading to the low correlation between these metrics and human-annotated relevance scores, as depicted in **Table 10** in the paper.\n\nSecond, similar to factuality metrics, reference reliance becomes a significant bottleneck in real-world applications where such references are not readily available.\n\n● **Extrinsic.** Extrinsic metrics such as EM (Exact Match) and F1 score can indeed be used to assess the quality of answers. However, their over-reliance on literal matches often results in low recall. For instance, \"PRC\" and \"China\" refer to the same entity, but these metrics would yield a score of 0 due to the difference in literal representation. This limitation is likely the reason for their underperform in **Table 2**.\n\nAdditionally, their assessment of \"quality\" is rather vague. For example, we may wish to gauge the factual accuracy of downstream tasks using the generated knowledge or assess how much the upstream knowledge aids downstream tasks. These prompt the need for more granular downstream metrics. \n\n**2. Motivation and design process - “Why did we choose these six metrics”.**\nGiven these limitations, there is a clear need for a framework that can provide a fine-grained evaluation of knowledge generation from essential perspectives in reference-free scenarios. This was the primary motivation for designing our proposed framework.\nThe process of identifying these essential perspectives began with a detailed manual investigation of the LLaMA's responses to 160 samples from the NQ and WoW datasets (80 each). We sought to understand the errors in knowledge generation and answer generation made by the model.\n\nTo achieve this, we employed thematic analysis [A1], a widely recognized method for identifying patterns within data. Our process started with the extraction of preliminary patterns from subtle errors in the knowledge and answers, which were then aggregated into broader patterns. We further validated these patterns using an additional 40 questions, ensuring that they adequately represented the spectrum of errors. \n\nAs a result, we discerned four primary error categories in knowledge generation and two in answer generation. These categories formed the foundation for our four intrinsic metrics for knowledge evaluation and two extrinsic metrics for answer evaluation.\n\nWe hope these explanations help improve the understanding of our work to warrant new metrics.\n\n&nbsp;\n\n### **Response for Factuality**\n\nThank you for your insightful comments. We have organized our response to more clearly address the three key areas you've mentioned: the connection with previous work, comparisons with advanced models, and why we choose off-the-shelf models.\n\n**\"Connection to Previous Evaluation Measures\".**  Our work significantly diverges from previous studies[1,2], especially in defining and evaluating 'factuality'. The latter focuses on 'closed' tasks like abstractive summarization, dealing with input-output contradictions. As noted in **lines 157-166**, they evaluate 'factuality' by comparing input and output, requiring a more advanced, task-specific NLI model.\nConversely, our open-ended tasks like open-domain QA and dialogue do not permit direct correctness judgement based on the input question. 'Factuality' is assessed by how well the model's output aligns with world knowledge, necessitating the **output decomposition**, **detailed evidence collection** and **results aggregation** processes as integral parts of our factuality metrics.\n\n**Compared with More Advanced Baselines.** As shown in Table 1, we have also supplemented three more advanced methods as the baselines of our Factuality metrics and computed their correlation with human annotation of actuality.\n\n(1)  NLI-multitask, which fine-tunes the DeBERTa-v3-large model on FEVER and two NLI datasets. \n\n(2)  weak-supervised[1], which was trained on constructed data to perform consistency checking on (document, sentence) pairs. We chose the factCC version as our baseline.\n\n(3)  NLI-decompose-claim[2], which found that in general, sentence-level decomposition is preferable for the hypothesis side of the NLI input. So we decompose the generated knowledge into sentences and then aggregate the sentence-level scores to produce a document-level score.\n\n|Metric|DPR|FLAN-T5|LLaMA|ChatGPT|\n|:---:|:---:|:---:|:---:|:---:|\n|Factuality|0.65|0.66|0.66|0.63|\n|NLI-off-the-shelf|0.22|0.6|0.22|0.17|\n|NLI-multitask|0.18|0.51|0.26|0.32|\n|weak-supervised[1]|0.12|0.43|0.19|0.16|\n|NLI-decompose-claim[2]|0.23|0.47|0.27|0.38|\n|\n\nTable 1. Correlation of Factuality Scores with Human Evaluation Across Different Baseline Methods\n\nOur approach delivers competitive Factuality scores across all models. This is because our method decomposes the generated knowledge into individual sentences, and collects corresponding evidence for each sentence. This allows for a more comprehensive coverage of every component of the generated knowledge, as opposed to simply using ground-truth knowledge. This nuanced approach enhances the model's capacity to accurately generate factual information, resulting in an improved Factuality score.\n\n**Why do we \"use off-the-shelf model\".** While fine-tuning off-the-shelf NLI models for specific tasks can help overcome limitations, our goal is to provide a more universally applicable solution. Our preference for general models over task-specific ones ensures that our evaluation framework remains useful across a broad range of knowledge-intensive tasks, even in real-world scenarios.\n\nWe appreciate your keen insights and will ensure these points are clarified in the revised manuscript.\n\n&nbsp;\n\n### **Clarification for Relevance**\n\nWe divide our response into two parts to respond to your questions about the motivation for using a BERT ranking model and to make some clarification about baselines.\n\n**1. Motivation for Using BERT Ranking Model.**\nOur choice to use a BERT ranking model is driven by three main factors:\n\n **(1) Knowledge Generation as a Generative Search Process.** In the process of generating knowledge, large language models (LLMs) effectively function as generative search engines. The most crucial aspect of such a process is to evaluate the relevance of the acquired knowledge document to the given query. The BERT ranking model's training data comes from the Bing search engine, with results manually annotated for relevance. This makes the BERT ranking model particularly suited to precisely evaluate the relevance of a document (or in our case, knowledge generated by a language model) to a given query.\n\n**(2)  Absence of Reference in Real-World Scenarios.** In real-world applications, ground-truth knowledge is often not available. The BERT model, trained to rank matching pairs of queries and knowledge documents, proves effective in estimating the relevance of the knowledge to the given query, even in the absence of golden knowledge.\n\n**(3) Potential Misalignment with Pre-Annotated Reference.** Even when datasets have pre-annotated references, they may not align perfectly with the knowledge generated by the model. In our dataset, the 'ground-truth knowledge' often corresponds to annotations made against pre-existing human answers, which may not perfectly align with the knowledge generated by the model. Therefore, we find it more meaningful to measure the retrieval relevance between the query and the model-generated knowledge.\n\n\n**2.  Clarification on Baselines**\nWe understand your concerns about the baselines of relevance evaluation and wish to offer some clarifications:\n\n● **F1 Score:** The F1 score was indeed calculated in our evaluation. As outlined in Table 10 of our paper, we used the F1 score to measure the overlap between the ground-truth knowledge and the knowledge generated by the model.\n\n● **Knowledge-F1**: Knowledge-F1 is not designed for evaluating knowledge generation. It measures the overlap between the model's response and the knowledge used during response generation. This makes Knowledge-F1 more suitable for assessing response generation in knowledge-grounded dialogue tasks, not knowledge-generation tasks.\n\n● **Accuracy**: Accuracy is not suitable for text generation tasks, which is the focus of our work. \n\nWe appreciate your insightful feedback. It has helped us clarify our research and will undoubtedly improve our work. Thank you for your thorough review.\n\n&nbsp;\n\n### **Response for Extrinsic Metrics**\nTo address your suggestions, we supplemented our extrinsic evaluation with a comparison of EM and F1 scores and calculated their correlation with human evaluations. Given that EM and F1 scores are primarily used to assess the correctness of downstream answers, we utilized them as baselines for our validity metric.\n\nThe results of our supplementary evaluations are in Table 2:\n\n|Metric|DPR|FLAN-T5|LLaMA|ChatGPT|\n|:---:|:---:|:---:|:---:|:---:|\n|Validity|0.83|0.73|0.85|0.82|\n|EM|0.59|0.51|0.54|0.61|\n|F1|0.74|0.67|0.76|0.77|\n|\n\nTable 2. Correlation with Human Annotation of Answer Correctness Across Different Methods\n\nFrom the results, it is evident that our validity metric demonstrates a significantly higher correlation with human evaluation of answer correctness compared to the traditional metrics.\nWe will add this evaluation in our revised version. We hope that these additions address your concerns and illustrate the effectiveness of our extrinsic evaluation.\n\n&nbsp;\n\n### **Response for \"focuses on findings before establishing the efficacy of metrics\"**\nWe understand your concerns about the presentation order in our paper.  In our initial structure, we chose to present the validation of the metric efficacy in Chapter 6 and Appendix J, intending to first share the exploratory findings and analyses, which we believed might be of more immediate interest and value to readers. \n\nHowever, we acknowledge your insightful point about the importance of showcasing the metrics' effectiveness earlier in the paper. We appreciate your suggestion and will make adjustments to enhance the rigour of our presentation.\n\n&nbsp;\n\n### **Clarification for \"clubbing data from multiple models and datasets for computing correlations\"**\nWe would like to clarify that we did not \"club data from multiple models and datasets for computing correlations\". Rather, we have calculated correlations separately for each dataset and model. As outlined in **Table 10 and Appendix J.1** in our manuscript, each model's performance has been independently evaluated on each dataset, without merging the results across different data sources. By adopting this approach, we have strived to ensure that our analysis is as fair and unbiased as possible, and that it reflects the true performance of each model on each dataset. \n\nWe hope this addresses your concern about the correlation study.\n\n&nbsp;\n\n### **Clarification for \"each correlation for each metric is computed against the same scores.\"**\nWe would like to clarify that we did not use a uniform score for correlation computations w.r.t different metrics.  Instead, as mentioned in **lines 556-557 and 957-959** of our manuscript, human annotators assigned scores of 0, 1, 2 for each of the intrinsic and extrinsic metrics we used, representing 'Unacceptable, Acceptable, and Excellent', respectively. \n\nFurthermore, we provided detailed 'Human annotation guidelines for each perspective/metric' in **Table 17** of our paper. This approach ensures that each metric has a direct and meaningful comparison with human judgments.\n\nWe hope this clarification addresses your concern about the comparability between human annotations and individual metrics. \n\n&nbsp;\n\n### **Response for \"Use of Somers' D Correlation\"**\nWe appreciate your attention to the choice of correlation measurement used in our study. We would like to clarify our rationale for selecting Somers' D and provide additional analysis using the Pearson correlation to address your concerns.\n\n**Motivation for Using Somers' D.**\nOur choice of Somers' D correlation was primarily motivated by its superior performance in handling ties of a biased random variable [A2]. In this method, correlations are measured against a biased random variable $X \\in[0,1]$, represented by the potential perturbation or error presence indicator within data. For each score Y considered, it is calculated using the formula $D(Y\\mid X)=\\tau(X, Y) / \\tau(X, X)$. This capability of Somers' D is particularly beneficial for our study and aligns with our evaluation characteristics.\n\n**Supplementary Analysis Using Pearson Correlation.**\nTo address your concern further, we conducted a supplementary analysis using the Pearson correlation. The results are in Table 3, which mirrors those obtained with Somers' D, strengthening the robustness of our conclusions. We present a comparison of the results obtained using both correlation methods:\n\n|Metric|Correlation|DPR|FLAN-T5|LLaMA|ChatGPT|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Factuality|Somers' D|0.65|0.66|0.66|0.57|\n| |Pearson|0.74|0.68|0.67|0.57|\n|Relevance|Somers' D|0.69|0.37|0.55|0.54|\n| |Pearson|0.75|0.4|0.64|0.64|\n|Coherence|Somers' D|0.53|0.58|0.44|0.49|\n| |Pearson|0.48|0.51|0.36|0.42|\n|Informative|Somers' D|0.3|0.17|0.35|0.32|\n| |Pearson|0.44|0.21|0.32|0.34|\n|helpfulness|Somers' D|0.75|0.45|0.81|0.69|\n| |Pearson|0.64|0.41|0.64|0.72|\n|validity|Somers' D|0.83|0.73|0.85|0.82|\n| |Pearson|0.89|0.87|0.94|0.89|\n|\n\nTable 3. Comparison of Correlations Using Pearson and Somers' Methods Across Different Metrics and Models\n\nAs seen from the results, there is a consistent pattern of correlation across different metrics and correlation methods, with no significant deviations. Regardless of the correlation method used, the relative rankings of the models' performance remain consistent, which further illustrates that our choice of Somers' D does not adversely impact the accuracy of our evaluation.\n\nWe hope that this response addresses your comments and we appreciate your suggestions for improving the rigor of our work.\n\n&nbsp;\n\n### **Response for additional baselines**\nAs advised, we have already addressed the concerns about the baselines in our previous responses.\n\n**Factuality Baselines.**  As recommended, we have incorporated references [1,2] as factuality baselines and presented the results in Table 1.\n\n**Relevance Baselines.** Regarding Knowledge-F1[6] for relevance, we explained in our previous response that they primarily assess overlap between the response and its' grounding knowledge, which does not apply to the relevance evaluation for the generated knowledge.  As for the EM (Exact Match) and F1 Score used in [7], we've demonstrated these metrics respectively in Table 2 and Table 10 (in our paper).\n\nThank you for your feedback.\n\n&nbsp;\n\n### **Response for BERT-ranking model**\nAs specified in Table 14 of our paper, we utilized the BERT Large model, which was trained on the TREC-CAR training set.\n\nThis model was chosen based on its suitability for our task, as just discussed in the 'Motivation of using BERT-ranking-model' part. We aimed to capitalize on the model's existing capabilities rather than adapt it to our specific needs. This approach aligns with the \"Why do we use off-the-shelf models\" part, as mentioned in our prior response.\n\nTherefore, we decided to use the off-the-shelf BERT-ranking model as the component of our metric without any custom adaptations for measuring relevance. This strategy allowed us to maintain the broad applicability of our metrics.\n\n&nbsp;\n\n### **Clarification for \"the expected behaviour of informativeness\"**\nWe would like to clarify that our measure of informativeness is not strictly about the novelty of the text, but rather its ability to reduce the uncertainty introduced by a query.  We clarify this from two perspectives:\n\n**Conceptual Understanding of Informativeness.**\nIn our study, we employ the conditional entropy of the acquired knowledge k given a query q. Conditional entropy can be understood as a measure of the uncertainty (or randomness) remaining in the 'knowledge' (k) once the 'query' (q) is known. In other words, it quantifies the 'uncertainty reduction' in the knowledge given a specific query.\n\n**Illustrative Example of Expected Behaviour.**\nTo more intuitively understand the expected behaviour of 'informativeness', we consider a toy counterexample to illustrate a scenario where output may be novel but not informative:\n\n**Query:** What is the capital of the United States?\n\n**Acquired Knowledge:** Washington.\n\nIn this case, the query is very straightforward, so the model's uncertainty is very low. The average probability of generating the \"Washington\" is high. Our informativeness score is 1 minus this average probability, thus resulting in a low informativeness score.\nConversely, if the acquired knowledge was \"China,\" the model's probability of generating this knowledge would certainly be low, leading to a high informativeness score.\n\nSimilarly, for a very hard query where the model doesn't know the answer, any knowledge provided might have a high informativeness score because they all have low output probabilities from the model (since the model is uncertain about the answer).\n\nTherefore, the informativeness measure isn't just about novelty, but more about the ability of the text to reduce the uncertainty introduced by a query. We hope this clarifies the intended behaviour of our metric.\n\n&nbsp;\n\n### **Response for typo**\nApologies for the confusion.  The clearer representation of the equation should be as follows:\n\n$\nS_\\texttt{help}(q, a, k, k_{1}^-, \\cdots, k_{u}^-) \n=\\max \\left(0, 1 - \\frac{\\mathcal{L}(q,k,a)}{\\frac{1}{u} \\sum\\nolimits_{i=1}^{u} \\mathcal{L}(q,k_{i}^-,a)}\\right) \n=\\max \\left(0, 1 - \\frac{\\log P(a|q, k)}{\\frac{1}{u} \\sum\\nolimits_{i=1}^{u} \\log P(a|q,k_{i}^-)}\\right) \n$\n\nThanks for pointing this out. We will also address other typos in our revised version.\n\n&nbsp;\n\n\n### **Response for low sentence-coherence of DPR**\n\nApologies if the coherence scores in Table 2 were not sufficiently explained. We explain it from two aspects: understanding the generally low coherence scores and explaining DPR's specific low score.\n\n**Understanding Generally Low Coherence Scores.** Our sentence coherence measure is based on the inverse of Perplexity (PPL), a common evaluation metric in natural language processing. Typically, PPL values for language models tend to range between 40-50, which results in lower sentence coherence scores when inverted. Despite the low absolute values, this does not impede our evaluation. The sentence coherence measure can effectively differentiate between models, providing a comparative assessment of their performance.\n\nHowever, it doesn't prevent us from using the sentence-level coherence measure to compare different large language models, as we only need to compare the relative values among different LLMs, not the absolute values.\n\n**Explaining DPR's Low Sentence-level Coherence Score.** The low coherence score for the Dense Passage Retrieval (DPR) and other models is mainly attributed to the nature of the knowledge passages they retrieve.\n\nCurrently, the method [A3] for constructing the knowledge base involves truncating Wikipedia passages to a fixed length of 100 tokens. This often results in 'incomplete' passages, where the beginnings and ends of sentences are often cut off. This truncation can significantly impact the coherence of the text, leading to lower coherence scores.\n\nWe hope this explanation provides a comprehensive understanding of the coherence scores in our study.\n\n&nbsp;\n\n### **Response for qualitative examples.**\nWe will adjust the DPR query to match the others and update Table 4 accordingly. And we appreciate your suggestion for a more clear qualitative analysis.\n\n&nbsp;\n\n### **Response for pairwise correlation.**\nIn Tables 5 and 6, we conducted pairwise correlations between two extrinsic and four intrinsic metrics across different datasets. To derive a more generalizable rule, we averaged the results from Language Model-based methods (LLMs) and compared them with the Dense Passage Retrieval (DPR), a retrieval-based method. This approach is designed to highlight the differences between the generative and retrieval methods.\n\nThe main goal of our analysis was to ascertain which aspects of knowledge generation (intrinsic evaluation metrics) have the most substantial impact on the performance of downstream answer generation (extrinsic evaluation metrics). By correlating internal and external metrics, we can obtain a deeper understanding of the key factors that influence the performance of the external metrics. This method provides a quantifiable approach to pinpoint the most influential intrinsic properties. We hope this explanation offers a more comprehensive understanding of our correlation analysis and its objectives.\n\n&nbsp;\n\n### **Clarification for '43.45'.**\nTo clarify, our present values are relative improvements, not absolute ones. Specifically, we intended to convey that the improvement on the 'helpfulness' metric reached (0.209 - 0.146) / 0.146 = 43.15%, not 43.45% in the 'validity' metric. Our apologies for the confusion. Thanks for bringing this to our attention.\n\n&nbsp;\n\n### **Response to Missing References.**\nThank you for pointing that out. We will incorporate it in our revisions.\n\n&nbsp;\n\n***\nWe hope that our answers can address your concerns satisfactorily and improve the clarity of our contribution. We would be grateful if you could re-evaluate our paper. We look forward to receiving your further feedback.\n\n***\n&nbsp;\n\n**References**\n\n[A1] Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.\n\n[A2] Alan Agresti. Analysis of ordinal categorical data, volume 656. John Wiley & Sons, 2010.\n\n[A3] Vladimir Karpukhin et. al 2020, Dense passage retrieval for open-domain question answering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. First evaluation targeted towards understanding the use of model generated text as context for knowledge intensive tasks. The paper highlights the importance of studying the context and its impact in addition to just measuring downstream performance.\n2. Interesting findings - With a lot of concerns about factuality of generated text, the findings that even with lower factuality, the impact on downstream task impact is not significant will be interesting to the community. They show that relevance and coherence of the outputs are more important that small factual mistakes.", "reasons_to_reject": "1. Motivation and explanations for each metric proposed is not clear - The evaluation of some aspects do not follow standard evaluation measures and some new automatic metrics are used without providing sufficient motivation. There is minimal grounding in prior research in such evaluation metrics and does not discuss limitations in them to warrant new metrics. Since the goal of the paper is to evaluate models, the metrics used to measure quality of generation should be robust and accurate to ensure that findings are reliable. Therefore motivation and explanations on why the existing metrics were not applicable is important. I am not fully convinced that the current evaluation measures chosen adequately measure the concerned aspect, which cast doubt on the validity of the findings.\n\n    i) Factuality - Prior work has explored evaluating factuality of text given evidence [1,4, 5] via models trained to predict factuality. [2,3] has shown limitation of using off-the-shelf NLI models for detecting factual errors. How does the proposed metric connect to previous evaluation measures and is it experimentally better at detecting factual errors?\n\n    ii) Relevance - Following [6, 7], F1 and Knowledge-F1 or Accuracy wrt to retrieved/gold evidence is commonly used to measure relevance. It’s not clear how a BERT ranking model would measure relevance well. Explaining the motivation of using a BERT ranking model and citing and comparing with the prior evaluation measures is needed.  \n\n    iii) Extrinsic Evaluation - Prior work Open-Domain QA use Exact-Match and F1 scores  for evaluating downstream task performance [8,9]. The helpfulness metric measured using ‘cross entropy losses of answer generation’ is interesting but approximate and not exact metrics. Validity metric evaluates entailment which again is a semantic match but might not compute exact evaluation of the downstream performance. Incorporating the prior metrics in addition to the proposed ones or comparing against them to show the effectiveness of the new metric is important.\n \n2. Weak evaluation of proposed metrics - Since some of the metrics proposed are not commonly used or are new, the first evaluation should center around the efficacy of the evaluation measures. Currently, the paper focuses on findings before establishing the efficacy of metrics in measuring different aspects. The correlation study done in Sec 6 is limited and very briefly explained.\n\n    i) Effect on model and dataset bias - [10] discusses the concern of clubbing data from multiple models and datasets for computing correlations. Biases like ‘model detects one dataset/model well’ leak into the correlation and compromise final results. The paper suggests computing and reporting ‘partial correlation’ to correct this problem.\n\n    ii) Human and metric are measuring different aspects and hence not comparable - The metrics proposed and used in the paper are ‘Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity’ and the human annotations obtained are a 0,1,2 label for ‘Unacceptable, Acceptable, and Excellent’. I don’t think these are comparable, especially considering that each correlation for each metric is computed against the same scores. Ideally, most work obtain human annotations for each aspect being measured and compute correlations for each pair.\n\n    iii) Using Sommers D for correlation - What is the motivation for using Sommers D? The statistic measures the association between an ordinal dependent variable and an ordinal independent variable. How is it applicable in this setting. Most prior work use Pearson correlation or Spearman rank correlation [10, 11, 12], is there a reason a different statistic was chosen ?\n\n3. Weak baselines in metric evaluation - As mentioned in motivation for metrics, prior research in different evaluation measures need to be citied and compared against. E.g, [1,2] should be used as factuality baselines, [6,7] should be used for relevance in addition to F1.\n\n[1] Kryściński et. al 2019, Evaluating the Factual Consistency of Abstractive Text Summarization\n\n[2] Glover et. al 2022, Revisiting text decomposition methods for NLI-based factuality scoring of summaries\n\n[3] Utama et. al 2022, Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization\n\n[4] Honovich et. al 2022 TRUE: Re-evaluating Factual Consistency Evaluation\n\n[5] Ji et, al. 2022, Survey of Hallucination in Natural Language Generation\n\n[6] Schuster et, al 2021, Retrieval augmentation reduces hallucination in conversation\n\n[7] Komeili et al 2021, Internet-Augmented Dialogue Generation\n\n[8] Chen et al 2017, Reading Wikipedia to Answer Open-Domain Questions\n\n[9] Karpukhin et al 2020, Dense Passage Retrieval for Open-Domain Question Answering\n\n[10] Pagnoni et al 2021, Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark \nfor Factuality Metrics\n\n[11] Devaraj et al 2022, Evaluating Factuality in Text Simplification\n\n[12] Dror et al 2018, The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing", "questions_to_authors": "A. For the ranking BERT model, which model from the relevant paper is used. The paper proposed two variants for ranking, which one was used and how was it adapted for measuring relevance?\n\nB. Line 269 typo: coh_para should be coh_sent\n\nC. Line 298 typo - S_ent ?\n\nD. For the informativeness measure entropy could be increased without the text being informative? Any text as long as it’s new will be considered informative. Is that what the expected behaviour of the metric ?\n\nE. Line 322 - clarify where the Loss is being used exactly.\n\nF. Table 2 - why is sentence coherence so low for dpr (basically all models) ? What does it mean if coherence values are so low ?\n\nG. For qualitative examples (Table 4), please include paragraphs for the same query for better comparison (dpr query is different).\n\nH. Para from line 407 - how was this correlation done? What does correlation between metrics actually reveal - more clarification on what this experiment actually measures would be good.\n\nI. Line 550 type 43.45", "paper_topic_and_main_contributions": "The paper aims to evaluate LMs as knowledge generators. Recent works have proposed to leverage LMs to generate text to be used as context for knowledge-intensive tasks. But there is a concern about the quality and factual inaccuracies in the generated text and the impact of such inaccuracies in the downstream task. This paper addresses such concerns by proposing to compresively evaluate such generated knowledge across six factors — Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. They propose an automatic metric for each aspect and evaluate generated knowledge from 3 LLMs (Flan-T5, Llama and ChatGPT) and text from retrieval system (DPR) across using these metrics. Their results indicate that LLM generate text are less factual but more relevant than retrieved text. The lower factuality doesn’t significantly effect the downstream tasks since the LLM generated text have higher helpfulness and validity. The additionally experiment and show effect of model size, long tail generation and long form generation on factuality of the generated text. Finally they show how to leverage these metrics to improve the few-shot prompts and generated text to get better performance on the downstream tasks. \nThe main contributions of this work:\n\n1. A framework for evaluating knowledge generated from LMs for knowledge intensive task.\n\n2. Automatic metrics for each aspect of generation to be evaluated.\n\n3. Evaluation of model generated text in context of knowledge intensive tasks like open-domain QA and knowledge grounded dialogue.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Glover et. al 2022, Revisiting text decomposition methods for NLI-based factuality scoring of summaries\n\n[2] Utama et. al 2022, Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization\n\n[3] Honovich et. al 2022 TRUE: Re-evaluating Factual Consistency Evaluation\n\n[4] Ji et, al. 2022, Survey of Hallucination in Natural Language Generation\n\n[5] Schuster et, al 2021, Retrieval augmentation reduces hallucination in conversation\n\n[6] Komeili et al 2021, Internet-Augmented Dialogue Generation\n\n[7] Chen et al 2017, Reading Wikipedia to Answer Open-Domain Questions\n\n[8] Pagnoni et al 2021, Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics\n\n[9] Devaraj et al 2022, Evaluating Factuality in Text Simplification\n\n[10] Liu et, al 2020, Evaluating Text Coherence at Sentence and Paragraph Levels", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Since the goal of the paper is an evaluation framework, it is ideally ok to center results around findings and analysis of models. But if some new evaluation measures are introduced, first presenting the efficacy of the measures and then focusing on findings increases the confidence in the findings.\n2. There are a few typos which seem to stem for revisions, carefully correcting them would be good."}}, {"review_id": "ZZPZ6bWCHE", "reviewer": "Reviewer_QUWs", "summary": "", "questions": "1. In the calculation of the Informativeness score, it is unclear how Eq.(5) is related to Eq.(6). In my understanding, the $k$ in $H(k|q)$ on the LHS of Eq.(5) is a random variable rather than a $k$ with a definite value. $H(k|q)$ here represents the entropy of the knowledge distribution under the given query $q$, with a larger entropy implying a more uniform knowledge distribution under the given query. But the final Informativeness score in Eq.(6) is solely obtained by subtracting the geometric mean of the generation probabilities of all tokens in $k$ from 1 under the given $q$ and $k$, which seems unrelated to the entropy defined in Eq.(5). Could you explain the connection here?\n2. In the calculation of the Helpfulness score, is there a coefficient $u$ missing in the numerator of Eq.(7)? Also, there lacks a necessary explanation of the relationship between $L(q, k, a)$ and $P(a|q, k)$.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are profoundly grateful for your positive and insightful feedback on our work! Your recognition of our paper being the pioneer to propose an automatic evaluation framework for LLM-generated knowledge, and its role in filling a gap in existing literature is highly encouraging. Your endorsement of our experimental investigations' guidance for downstream tasks, and the reliability through a sufficient number of manual experiments are particularly encouraging. We will now address your points individually.\n\n\n#### **Response for Informativeness**\nWe appreciate your insightful feedback. We realize that there appears to be a disconnect between Eq.(5) and Eq.(6). As you noted, Eq.(5)'s $H(k|q)$ represents the entropy of the knowledge distribution for a specific query, with a lower value implying a more nonuniform knowledge distribution under the given query.\n\nHowever, computing $H(k|q)$ requires a distribution over an infinite set of variable-length knowledge sequences, which is impractical as we only possess information for a specific 'k'. To address this, we employ an autoregressive language model to factorize the joint distribution over 'k' into a product of conditionals over a finite set of tokens as follows:\n\n$P(k | q)= \\\\prod\\\\nolimits\\_{t=1}^{M} P( k_t | k\\_{1:t-1}, q)$\n\nThis conditional independence assumption permits us to define a model on a finite token space of knowledge sequence. Therefore, instead of the sequence level, we estimate uncertainty at the token level, which considers uncertainty in the prediction of a single 'k'.\n\nWe then employ an approximation method, as suggested by [1][2] and apply '1 - the geometric mean of the generation probabilities of all tokens in k under the given q'. This serves as an effective approximation of uncertainty reduction, which we denote as $S\\_{info}$. Empirically, it aligns with $H(k|q)$ in that a score of 0 indicates the absence of useful information, which corresponds to a case where $P(k|q)=1$, suggesting the most uneven knowledge distribution.\n\nIn summary, Eq.(5) represents an ideal estimation of knowledge distribution, while Eq.(6) provides a practical approximation under specific knowledge. \n\nWe hope this explanation clarifies the relationship between these two equations. Your astute observation has helped us to provide this clarification.\n\n&nbsp;\n\n#### **Response for Factuality Equation**\nApologies for the confusion.  The clearer representation of the equation should be as follows:\n\n$\nS_\\texttt{help}(q, a, k, k_{1}^-, \\cdots, k_{u}^-) \n=\\max \\left(0, 1 - \\frac{\\mathcal{L}(q,k,a)}{\\frac{1}{u} \\sum\\nolimits_{i=1}^{u} \\mathcal{L}(q,k_{i}^-,a)}\\right) \n=\\max \\left(0, 1 - \\frac{\\log P(a|q, k)}{\\frac{1}{u} \\sum\\nolimits_{i=1}^{u} \\log P(a|q,k_{i}^-)}\\right) \n$\n\nHere, $\\mathcal{L}(q,k,a)$ signifies the cross-entropy loss of the answer generation process, and $P(a|q,k)$ denotes the probability of the correct answer. We calculate the cross-entropy loss based on this probability.\n\nWe will ensure that this correction, as well as the explanations, are incorporated in our revised version.\n\n&nbsp;\n\n#### **Response for Typo**\nThank you very much for pointing out the typo. We will correct this in our revised manuscript. Your attention to detail is greatly appreciated.\n\n&nbsp;\n\n**References**\n\n[1] Kenton Murray et. al 2018, Correcting length bias in neural machine translation\n\n[2] Andrey Malinin et. al 2021, Uncertainty estimation in autoregressive structured prediction", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is the first to propose an automatic evaluation framework for LLM-generated knowledge. Many previous works have utilized the knowledge generated by LLMs, but lack an evaluation of whether this knowledge is reliable or effective. This work fills this gap nicely.\n2. The experimental part of the paper investigates the quality of the knowledge generated by the model and its impact on downstream tasks through Somers’ correlation score, which provides guidance for how to better use model-generated knowledge in downstream tasks.\n3. The authors conduct a sufficient number of manual experiments and show a high correlation with the scores automatically obtained by CONNER, suggesting the reliability of this method for automatic evaluation.", "reasons_to_reject": "1. Some of the metrics used by this method are not explained clearly enough. (Refer to question 1 and 2)", "questions_to_authors": "1. In the calculation of the Informativeness score, it is unclear how Eq.(5) is related to Eq.(6). In my understanding, the $k$ in $H(k|q)$ on the LHS of Eq.(5) is a random variable rather than a $k$ with a definite value. $H(k|q)$ here represents the entropy of the knowledge distribution under the given query $q$, with a larger entropy implying a more uniform knowledge distribution under the given query. But the final Informativeness score in Eq.(6) is solely obtained by subtracting the geometric mean of the generation probabilities of all tokens in $k$ from 1 under the given $q$ and $k$, which seems unrelated to the entropy defined in Eq.(5). Could you explain the connection here?\n2. In the calculation of the Helpfulness score, is there a coefficient $u$ missing in the numerator of Eq.(7)? Also, there lacks a necessary explanation of the relationship between $L(q, k, a)$ and $P(a|q, k)$.", "paper_topic_and_main_contributions": "This paper develops a comprehensive knowledge evaluation framework (CONNER) to evaluate the LLM's knowledge generation capability. CONNER involves both intrinsic and extrinsic evaluation, where the intrinsic assessment focuses on the quality of generated knowledge while the extrinsic aspect considers the applicability of the generated knowledge to specific downstream tasks (QA, knowledge-grounded dialogue). The evaluation results contribute to the understanding of how model generated knowledge affects downstream tasks, and yields practical experience for how to leverage LLM-generated knowledge for downstream tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 296: The index under $\\Sigma$ should be $t$ rather than $i$"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of evaluating the knowledge extracted from LLMs. The proposed evaluation metrics consist of four intrinsic aspects (Factuality, Relevance, Coherence, and Informativeness) and two extrinsic aspects (Helpfulness and Validity). The authors propose an automatic metric for each aspect and evaluate generated knowledge from 3 LLMs (Flan-T5, Llama and ChatGPT) and text from retrieval system (DPR) across using these metrics. Their results indicate that LLM generate text are less factual but more relevant than retrieved text, and that the higher relevance is ultimately important for downstream tasks.
The reviewers appreciate the importance of verifying the information generated by LLMs and the interesting analysis provided by the paper. 

The reviewers initially had some concerns specifically with respect to the validity / efficacy of the metrics, especially the informativity metric. However, these concerns seem to have been largely resolved in the discussion with the authors such that it seems likely that the authors can address the concerns in the revision of the paper.
Decision: Accept-Main