Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknown

Authors: Lifu Tu, Rui Meng, Shafiq Joty, Yingbo Zhou, Semih Yavuz
Primary area: generative models
Keywords: long-form generation, Factuality

Abstract:
Large language models (LLMs) have demonstrated strong capabilities in text understanding and generation. However, they often lack factuality, producing a mixture of true and false information, especially in long-form generation. In this work, we investigates the factuality of long-form text generation across various large language models (LLMs), including GPT-4, Gemini-1.5-Pro, Claude-3-Opus, Llama-3-70B, and Mistral. Our analysis reveals that factuality scores tend to decline in later sentences of the generated text, accompanied by a rise in the number of unsupported claims.
Furthermore, we explore the effectiveness of different evaluation settings to assess whether LLMs can accurately judge the correctness of their own outputs: Self-Known (the percentage of supported atomic claims, decomposed from LLM outputs, that the corresponding LLMs judge as correct) and Self-Unknown (the percentage of unsupported atomic claims that the corresponding LLMs judge as incorrect). The results indicate that even advanced models like GPT-4 and Gemini-1.5-Pro fail to achieve perfect Self-Known scores, while their Self-Unknown scores remain notably above zero, reflecting ongoing uncertainty in their self-assessments.
Moreover, we find a correlation between higher Self-Known scores and improved factuality, while higher Self-Unknown scores are associated with lower factuality. Interestingly, even without significant changes in the models' self-judgment (Self-Known and Self-Unknown), the number of unsupported claims can increases, likely as an artifact of long-form generation. These findings show the limitations of current LLMs in long-form generation, and provide valuable insights for improving factuality in long-form text generation.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729293932040

thread_type:
official_review

<thread_object>
creator: reviewer_rxnh

<content>
summary:
This paper investigate the factuality in long-form text generation. Using the existing datasets, it defines the Self-Known and Self-Unknown scores judged by the corresponding LLMs themselves. Based on several sets of statistics and empirical experiments, the authors draw several findings which might be useful for the community.

strengths:
1. Long-form factuality is a promising research direction.

weaknesses:
1. The statistical observations are based on an existing dataset of Biography from Min et al., 2023, which impair the contribution of this work.

2. The key finding of this work, *factuality scores tend to decline in later sentences of the generated text, accompanied by a rise in the number of unsupported claims*, is obvious and not exciting. This is a general challenge for LLMs to generate long-form texts in the later parts in terms of factuality, coherency, creativity, etc.

3. This work does not provide any insightful reasons, either. As stated in Section 2.1, *We hypotheses the possible reasons are the error propagation and these generated claims are with low confidence by LLMs*, this is not convincing.

4. The results presented in the main text overlap somewhat. For example, I don't quite get why reporting both the percentage and number of supported and unsupported atomic claims in Figure 1.

5. It is abrupt to discuss retrieval-augmented generation in Section 5.3. I don't quite get the necessity of introducing RAG.

6. The presentation is not good, which should be proofread one more time. For example, the duplicate descriptions of *Mixtral* in line 18 and *in this work* in line 42. Line 59: *Our findings indicate that sentences generated **later** in the sequence generally demonstrate higher factuality.* I think it should be *sentences generated **early** in the sequence generally demonstrate higher factuality*?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562903767

thread_type:
official_review

<thread_object>
creator: reviewer_8sw2

<content>
summary:
The paper explores the challenges large language models (LLMs) face in maintaining factual accuracy in extended text outputs, analyzing models like GPT-4, Gemini-1.5-Pro, and Llama-3-70B. The author first estimate the factuality score based on ‘’Self-Known and Self-Unknown’, and the evaluation shows that the score is well aligned with human evaluation. The study shows that factuality often declines over longer text, with later sentences containing more unsupported claims. Using new metrics, Self-Known, Self-Unknown and factuality score, the authors assess how well LLMs recognize their own factuality; they find that even advanced models have limited ability to self-assess accurately, achieving only modest success in identifying correct or incorrect claims. The findings emphasize the need for improved self-assessment in LLMs to enhance factual consistency, particularly in long-form generation.

strengths:
a.	The problem explored in the paper is interesting, and worth further exploration.
b.	The authors estimate a factuality score based on Self-Known and Self-Unknown, which makes sense

weaknesses:
a.	The presentation of the paper is confusing, the logic line of the paper is very unclear. For example, the paper proposes to use three ways to evaluate the self-known and self-unknown, however, no details is provided for all the methods. And in Section 4, an estimation of the factuality score is proposed, and in section 5, they mentioned that they measure the factuality score using FActScore, then which score do they used in the following experiments?
b.	As an analytical paper, it lacks insightful findings or conclusions. It has been found in some previous works that the model generates more unsupported atomics in the later sentences, and this is a known problem with long-form generation.
c.	 Some errors in the paper writing,  e.g. in line 60, they said ‘Our findings indicate that sentences generated later in the sequence generally demonstrate higher factuality.’ and in line 85, they said ‘...LLMs typically exhibit lower factuality scores in the later segments of long-form text.’ , which are contradictory.

questions:
a.	Which factuality score did you use for evaluation, the FActScore or the estimation in section 4?
b.	Can you elaborate more on the assumption in line 250?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674409800

thread_type:
official_review

<thread_object>
creator: reviewer_ksoe

<content>
summary:
This paper explores the issue of factuality in long-form text generation by large language models (LLMs), specifically addressing their tendencies to produce a mix of true and false information. The study examines six advanced LLMs, including GPT-4, Claude-3-Opus, and Mixtral, to investigate how factuality scores evolve throughout the text. The authors propose two novel evaluation metrics, Self-Known and Self-Unknown, to measure whether LLMs can accurately self-assess the correctness of their generated content. Results reveal that while LLMs generally perform well in earlier parts of the text, factuality tends to decline in later sections. Additionally, higher Self-Known scores correlate with better factuality, while higher Self-Unknown scores indicate the opposite. The authors also test various evaluation settings, emphasizing the importance of incorporating external knowledge to improve models’ factual accuracy.

strengths:
The introduction of Self-Known and Self-Unknown scores provides a fresh perspective on evaluating LLMs' self-assessment capabilities, offering valuable insights into their strengths and limitations. The paper systematically analyzes multiple LLMs, covering different evaluation settings and utilizing both human and automated tools for factuality assessment. This extensive examination adds rigor to the study. By highlighting the consistent decline in factuality in long-form outputs, the research addresses a critical challenge for deploying LLMs in real-world applications, making the findings highly relevant for model improvement. The observed relationships between factuality and self-assessment metrics could guide future work on enhancing LLMs’ self-awareness and knowledge calibration, which is vital for developing reliable generative AI models.

weaknesses:
1. The difference between Figure 1 (a) and Figure 1 (b) is not illustrated. It will be beneficial to add illustrations in caption.
2. The proposed method can perform evaluation to long-term generation by the model itself, but it is time-consuming to do evaluation for every atomic statement. The detection itself has a very strong trade-off between performance and inference costs.
3. There is not a way to adjust or convert the fact errors into correct facts. The proposed method only covers fact error detection.
4. Missing results for LLAMA family makes the results less comprehensive.
5. The scope of the proposed method is a bit limited as it can only be applied to factuality tasks which can be separated into atomic statements. However, some factuality tasks may not explicitly contains atomic statements.

questions:
As in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715359169

thread_type:
official_review

<thread_object>
creator: reviewer_3uij

<content>
summary:
This paper studied factuality in long text generation. It studies the position of the atomic fact and finds that the latter part of the text generated is more likely to be not factual. It invented two metrics: self-Known (the percentage of correct atomic facts by predicted as correct atomic facts) and Self-Unknown (the percentage of incorrect atomic facts are correctly predicted as incorrect). The authors claim the two metrics can help measure the true factuality score.

strengths:
- The authors did extensive analysis to support our intuition that there tends to more factual errors in the latter part of the LLM's response. The authors show this applies to different models and also applies to RAG scenario.

weaknesses:
- The Finding where model tends to have more factuality errors in later part of the output is kind of accepted conclusion. So the findings is no surprising
- The conclusion about the relationship between factuality score and Self-Known and Self-Unknown is not established. Firstly, I think there is something wrong when transiting from Eq (1) to Eq (2) and it is hard to understand the definition of "y_correct" and "y_wrong". Secondly, the assumption that there is no "over confidence" is not correct.  Lastly, it will be helpful if authors can verified the conclusion in Figure 3 in more than 1 datasets.

questions:
1. what are definitions of  the definition of "y_correct" and "y_wrong? can you show why Eq (1) can transit to Eq (2)?
2. I think "self-Known" is also called, True Negative Rate (if we consider the labe of being not-factual as positive) and "Self-Unknownw" will be "True Postiive Rate". Is this correct?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the factuality of long-form text generation by large language models (LLMs), revealing that factuality declines in later sections of generated text. Introducing two metrics, Self-Known and Self-Unknown, the study assesses LLMs' ability to self-evaluate the correctness of their outputs, finding that even advanced models like GPT-4 exhibit limitations in self-assessment. The findings highlight the need for improved self-evaluation mechanisms to enhance factual consistency in long-form generation.

Strength:
Studying the factuality problem in LLM is an important topic, especially how it evolves over the long-form generation situations.

Weakness:
The main weakness of this paper is that it provides relatively less insights about the problem. The factuality decays over at the later positions of long-form generation is not surprising. And defining the two metrics measuring their correlations are not a significant enough contribution. In addition, the work is only carried out on one dataset and should be evaluated on more datasets. As reviewers pointed out that testing the claims on one dataset cannot provide significant findings.

Overall, due to limited contribution and insights provided by this work, this paper cannot be accepted in its current form.