SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation

Authors: Song Duong, Florian Le Bronnec, Alexandre Allauzen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: faithfulness, hallucination, conditional text generation, natural language processing, large language models

Abstract:
Large Language Models (LLMs), when used for conditional text generation, often produce hallucinations, i.e., information that is unfaithful or not grounded in the input context. This issue arises in typical conditional text generation tasks, such as text summarization and data-to-text generation, where the goal is to produce fluent text based on contextual input. When fine-tuned on specific domains, LLMs struggle to provide faithful answers to a given context, often adding information or generating errors. One underlying cause of this issue is that LLMs rely on statistical patterns learned from their training data. This reliance can interfere with the model's ability to stay faithful to a provided context, leading to the generation of ungrounded information. We build upon this observation and introduce a novel self-supervised method for generating a training set of unfaithful samples. We then refine the model using a training process that encourages the generation of grounded outputs over unfaithful ones, drawing on preference-based training. Our approach leads to significantly more grounded text generation, outperforming existing self-supervised techniques in faithfulness, as evaluated through automatic metrics, LLM-based assessments, and human evaluations.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730419552930

thread_type:
official_review

<thread_object>
creator: reviewer_ajyu

<content>
summary:
The paper presents SCOPE, a self-supervised framework that enhances the faithfulness of LLMs in conditional text generation tasks like summarization and data-to-text generation. The framework employs a two-stage approach: initial fine-tuning followed by preference tuning using contrastive learning. The key innovation lies in its method of automatically generating synthetic unfaithful samples by blending grounded and context-free generation, which are then used to train the model to prefer context-aligned outputs over hallucinated ones. Through comprehensive evaluation across multiple datasets, SCOPE demonstrates significant improvements in output faithfulness, achieving up to 14% gains according to automatic metrics, with additional improvements confirmed through both GPT-4 and human evaluations.

strengths:
- An innovative self-supervised approach to improving LLM faithfulness through synthetic unfaithful sample generation
- The method shows strong performance across multiple benchmarks, with notable improvement in data-to-text tasks. 
- The paper explains the methodology and experimental setup well, using helpful figures and clear explanations of core ideas, such as the noisy sample generation and preference tuning.

weaknesses:
- While SCOPE performs well on general summarization and data-to-text tasks, evaluating it on domain-specific datasets (e.g., biomedical or financial) could better demonstrate its robustness in high-stakes settings where hallucinations are especially problematic.
- SCOPE is tested on 7B models only; applying it to models of different sizes would clarify its scalability and help determine whether it generalizes effectively to larger different models. Perhaps the improvement using SCOPE on larger models will be marginal. 
- The evaluation on summarization task used ROUGE and AlignScore (an entailment metric). While SCOPE scores slightly lower ROUGE scores than other baselines, the authors did not use other faithfulness metrics in summarization research to further evaluate.

questions:
1. What is the rationale behind splitting the dataset into 50-50 for the two phases? Would other split also work?
2. Do you anticipate that SCOPE would perform similarly on domain-specific datasets, like those in healthcare or finance?
3. Can SCOPE’s approach to unfaithful sample generation and preference tuning generalize to larger or different model architectures?
4. The $\alpha$ for CAD is set to very small values for tasks other than XSum. Why is it the case?
5. The main evaluation metrics used are BLEU/NLI/ROUGE/AL. Did you consider any additional faithfulness metrics or error types that might capture different facets of faithfulness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583980422

thread_type:
official_review

<thread_object>
creator: reviewer_esuq

<content>
summary:
The paper investigates the common issue of hallucinations in large language models (LLMs) during conditional text generation. To enhance faithfulness, the authors introduce SCOPE, a novel self-supervised fine-tuning approach that generates unfaithful samples and then trains the model to prefer context-grounded outputs over these fabricated examples. By using this approach, SCOPE achieves substantial improvements in generating grounded and faithful responses, as validated by automatic metrics and evaluations from both LLMs and human judges. The study's findings demonstrate SCOPE’s effectiveness across diverse tasks, including data-to-text and summarization, where it consistently outperforms existing self-supervised techniques.

strengths:
1. The research problem addressed in this paper is significant, as faithfulness remains a major challenge for contextual generation in LLMs.
2. The authors propose an innovative method for generating negative examples for DPO training, utilizing two models—the fine-tuned model and the pre-trained model—selecting the next token based on a weighted combination of their distributions. This approach is independent of external tools like NER.
3. The experimental results demonstrate that their method outperforms baseline models across multiple tasks.

weaknesses:
1. Some of the baselines appear to be training-free decoding methods, while others require training, similar to the method proposed in the paper. The authors should clarify which methods are training-free and which are not, as they require different resources to implement.
2. Additional ablation studies are needed, such as applying the same training framework with other negative sampling methods like NER replacement or negation. Otherwise, the effectiveness of the negative sampling itself remains uncertain.
3. Experiments should include the faithfulness score of the generated negative examples on the training set to directly demonstrate the impact of negative sampling.

questions:
1. In the noisy generation step, is fluency in the output ensured? Since each token is selected based on the weighted sum of the original and fine-tuned LM distributions, could it happen that the token with the highest probability from the combined distribution is not among the top tokens in either individual distribution? For instance, if the two models’ distributions differ significantly, could an unsuitable token receive a relatively high score in each distribution and then emerge as the highest-scoring token after the weighted sum?
2. Is the generated negative examples all unfaithful? it would be useful to provide statistics or evaluation results indicating the proportion of genuine negative samples. This could potentially be estimated using the evaluation metrics applied in the experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730940407632

thread_type:
official_review

<thread_object>
creator: reviewer_nujj

<content>
summary:
This paper introduces a new learning framework for conditional text generation for the purpose of increasing the faithfulness of generated text with respect to the context input. The learning framework consists of two main steps: finetuning and contrastive learning. For contrasting learning, dispreferred samples are created in an unsupervised manner by utilizing a decoding method for generating certain tokens from the untrained language model, thereby introducing tokens which are not contextually consistent with the task. In several experiments, SCOPE is shown to be more effective than other baselines, including various decoding methods. SCOPE shows to be more faithful than traditional finetuning and SCOPE output is more preferred by humans and by GPT-4 judges.

strengths:
Relatively novel methodology that uses the process of collaborative decoding for the purpose of generating dispreferred text. The method of collaborative decoding has been used in the past to combine text from a general LLM and an expert LLM to improve performance. In this setting, however, collaborative decoding is utilized to generate inconsistencies in the text randomly based on a threshold alpha.

Experiments in the tasks of summarization and data-to-text in several datasets which highlight the strength of SCOPE compared to traditional finetuning and other baseline decoding methods. Additionally, the large variety of metrics, human evaluation, and GPT-4 evaluation indicates that SCOPE is effective at increasing the faithfulness of the model for these two specific tasks.

weaknesses:
Lack of human analysis of the unsupervised generated data: it is unclear whether the the dispreferred samples are worse due to the changes in the language distribution, or if the dispreferred samples contain information or text that is inconsistent with the context. If the unwanted generation mainly comes from the changes in the language distribution, then SCOPE is not learning to be more faithful to the context. This is a key distinction which may limit the interpretability of SCOPE for being a more faithful method of conditional text generation. One way to mitigate this concern would be to conduct a human analysis on a sample of generated data and determine if the generated data contains text inconsistent with the context or if the inserted text from the base LM is merely different in language distribution.

Lack of comparisons to instruction-tuned models. The methodology of SCOPE, at its essence, is very similar to the standard method of reinforcement learning with human feedback, where the model is first finetuned and then preference optimized with binary data consisting of a preferred sample and an unwanted sample. Due to the similarities, it would be valuable to compare SCOPE to RLHF methods and compare the degree of faithfulness of the model. There is evidence that instruction-tuned models are more "helpful" to users' inputs in the prompt, which correlates highly with faithfulness. Using the instruction-tuned model as a baseline, or running SCOPE on an instruction-tuned model is important to validate SCOPE as a novel, effective method.

questions:
What are the impacts of training set size on the SCOPE methodology? It seems that for a small dataset, utilizing half of the dataset for finetuning and utilizing the other half for preference optimization may not be as effective as traditional finetuning.

Is there a way to rate the quality of the dispreferred samples in 3.2? For example, are there generated texts with more inconsistencies that are worse than other samples? Is there an impact on the potential difference in quality of the generated texts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730957234620

thread_type:
official_review

<thread_object>
creator: reviewer_fcdz

<content>
summary:
Scope is a self-supervised framework for improving the faithfulness of conditional text generation in LLMs. The approach uses a novel two-stage process: first fine-tuning on half the dataset, then using a mixture of that model and the base model to generate synthetic unfaithful samples for preference learning. The authors evaluate their method on data-to-text and summarization tasks.

strengths:
- They study an important problem in the subfield -- factuality and faithfulness issue in conditional text generation. The research is motivated.
- Interesting two-stage training pipeline, and the approach to generate synthetic unfaithful samples.
- Provides analysis of preference training dynamics as well as the ablation for the noise parameter $\alpha$

weaknesses:
- The preference learning stage uses β=0.1 for all models and datasets (A.2) without justification or ablation, despite this being a critical hyperparameter in preference learning.
- No evidence the method works on larger models. For their main claimed contribution ("improving faithfulness" in finetuning), testing on 7B models seems insufficient. Is it possible to include a few larger models, e.g. 14B, if not 70B scales.

questions:
1. Have you investigated whether the method's effectiveness varies with model size? The limitation to 7B models leaves open questions about scalability.
2. The choice of α in [0.4, 0.6] seems empirically driven. Could it vary for different tasks / models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731387724834

thread_type:
official_review

<thread_object>
creator: reviewer_hu8d

<content>
summary:
LLMs often struggle to generate fully grounded information to a provided context. There is a rich literature on the hallucination issues in generative models (https://arxiv.org/abs/2005.00661). This has become an even bigger priority as LLMs are being adapted widely. 

This paper addresses the hallucination issues in LLMs and proposes a self-supervised method for synthetically generating a training set of negative responses. Using these negative examples, models are then trained to encourage the generation of grounded outputs over unfaithful ones using preference-based training. 

The authors empirically demonstrate that the proposed method helps models generate more faithful texts, when evaluated automatically and by humans, on data-to-text generation and summarization.

strengths:
The paper is trying to address generative hallucination, a common problem in generative models. The paper is fairly well written and easy to follow. 

The paper brings new insights on the behavior of preference-tuning with the synthetically generated negative examples.

The proposed approach appears to improve response faithfulness, at a minor cost of fluency.

weaknesses:
The method for synthetically generating negative examples is not novel, as the authors claim. Several papers (see some pointers below) in the past have used similar methods for generating negative or diverse sets of samples to better calibrate their models. 

The paper should present a detailed analysis of the synthetic data being generated. For example, it would be nice to see faithfulness score distributions in negative examples. Also a qualitative analysis showing types of hallucinations being generated would have been very interesting. 

Faithfulness is a pointwise metric, but the human evals mostly focus on SxS ratings. It would have been nice to see pointwise evaluations of faithfulness. 

It would have been nice to see comparisons to other ways of generating negative examples, for example, prompting models to generate negative examples.

Other interesting work in this space that might be interesting to discuss here: 

Brio or Slic: https://arxiv.org/abs/2203.16804, https://arxiv.org/abs/2210.00045
Calibrating models for faithfuness: https://arxiv.org/pdf/2310.08764
https://renzhaochun.github.io/assets/pdf/26596-Article%20Text-30659-1-2-20230626.pdf
https://arxiv.org/pdf/1910.08684 from the point of sub-sequence sampling
Comparison with CoT or planning based methods for controlling hallucinations.

- 
After the rebuttals, I have adjusted the score. Thanks for conducting additional experiments and evaluations. They will be valuable to the readers.

questions:
We are moving towards general purpose models, instead of a task-specific model. How does the proposed method generalize to other tasks, while achieving higher factuality/faithfulness for data-to-text generation and summarization? Will it be much harder to tune the value of alpha when tried with multiple tasks together?

SFT on a specific task often  suffers from catastrophic forgetting. It would be interesting to see how post training baselines do on these tasks. The post training often aims to improve faithfulness along other criteria. 

Could you add a detailed analysis of synthetically generated data wrt faithfulness and fluency? Also how does the proposed method compare with other synthetic data generation methods? 

Reference responses often have hallucinations. For example, XSum gold summaries often have hallucinated content. How does this effect the data generation? How do you ensure that an example is a positive example with respect to faithfulness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732451722168

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their thoughtful feedback and are grateful for the overall positive reception. In response to their suggestions, we have made several additions to the paper, including significant new experimental results:

- (Reviewer aJYu) assessments with FactCC and QuestEval, two faithfulness-focused metrics for summarization,
- (Reviewers hu8D) an evaluation of SCOPE on Alpaca, a generalist instruction-following dataset,
- (Reviewer aJYu) experiments on PubMed, a dataset for medical evidence summarization,
- (Reviewers fCdZ, aJYu) evaluations of SCOPE on a larger model (Llama2-13b),
- (Reviewers hu8D, NUJj, eSUQ) a quantitative and qualitative (human) analysis of synthetic noisy samples
- (Reviewer fCdZ) ablations on the $\beta$ hyperparameter used in preference tuning.

We are excited about these new results and hope they provide further confirmation of the effectiveness of our approach.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SCOPE, a self-supervised framework designed to improve the faithfulness of large language models (LLMs) in conditional text generation tasks, such as summarization and data-to-text generation. By leveraging a two-stage process of fine-tuning and contrastive learning, the method synthesizes negative examples (unfaithful and contextually ungrounded) and trains models to prioritize grounded outputs. Extensive evaluations, including human and GPT-4 assessments, demonstrate that SCOPE significantly outperforms existing techniques in generating faithful text, achieving notable gains across multiple benchmarks.

Strengths:

* The paper addresses significant challenges in generative models: factuality and faithfulness.
* The proposed two-stage training pipeline is both novel and well-motivated, and it is independent of external tools and resources, unlike many prior works.
* Extensive experiments across summarization and data-to-text tasks show improvements over baseline methods. The results are well-supported by a variety of metrics, human evaluations, and GPT-4 assessments, emphasizing the method's effectiveness and applicability to multiple benchmarks. The work demonstrates an ability to improve response faithfulness at a minor cost to fluency.
* The paper provides comprehensive ablation studies and analyses.

Weaknesses:
* Limited novelty: Reviewers expressed concerns about novelty, as generating artificially bad examples is not entirely new. However, the authors highlighted significant technical differences, and the simplicity of the method (requiring only a few lines of code) remains a strong point.
* Insufficient analysis: The paper lacks a detailed qualitative and quantitative analysis of the generated negative examples. However, the authors have begun to address this by including plots and analyses in the appendix.
* Scalability concerns: The experiments focus on 7B-scale models, omitting larger models where faithfulness improvements might be less pronounced. However, this focus is reasonable, as the lack of faithfulness is more apparent in smaller models. Additionally, the authors included extra experiments on 13B models, which yielded similar improvements to those observed with 7B models.
* There were other concerns, such as the fixed hyperparameter β, which were addressed during the discussion period.

After the discussion period, it appears that most major concerns have been addressed, except for the fact that there are other methods that also rely on generating artificially bad examples (though these approaches differ technically). Given the importance of the problems addressed (factuality and faithfulness in LLMs), the simplicity of the method, the positive results, and the comprehensiveness of the analysis, I recommend accepting the paper.