Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Year: 2023

Abstract:
Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual-entailment rewards to optimize for factual consistency and explore the ensuing tradeoffs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience and conciseness of the generated summaries.

heading: 1 Introduction
text: Recent advancements in abstractive summarization systems (Zhang et al., 2019; Liu et al., 2022b) are often impeded by their tendency to output information that is either contradicting or unsupported by their input article, often termed as “hallucinations” or factual inconsistency (Falke et al., 2019; Maynez et al., 2020; Pagnoni et al., 2021). While these systems produce highly relevant and coherent text, this lack of factual consistency often limits their wide-spread adoption in real-world applications. An example is depicted in Figure 1, where the highlighted statement in the summary, while plausible, has no support in the input article.
∗Equal contribution
Since widely-used metrics such as ROUGE (Lin, 2004) were shown to be inefficient for detecting hallucinations, many recent research efforts introduced novel automatic metrics for measuring factual consistency (Kryscinski et al., 2020; Goyal and Durrett, 2020; Scialom et al., 2021, inter alia). We propose to leverage these automatic metrics within a reinforcement learning (RL) framework at training time. Specifically, we apply textual entailment assessment (a.k.a. natural language inference, or NLI; Dagan et al., 2005; Bowman et al., 2015) between the source article and the generated summary as a reward.
Our reward is based on the well studied textual entailment task (Pavlick and Kwiatkowski, 2019; McCoy et al., 2019; MacCartney and Manning, 2007, inter alia), for which there are many publicly available datasets (Nie et al., 2020; Liu et al.,
6252
2022a). While these NLI datasets are not specific to summarization, it was shown that classifiers trained on these datasets perform well in detecting factual inconsistencies in summarization and other generative tasks (Honovich et al., 2022). Because faithful summaries must be textually entailed from the corresponding input documents, using such a reward explicitly should guide a summarization model towards generating more factually consistent summaries. Yet, a high-quality summary should also be coherent and contain relevant information (Fabbri et al., 2021), aspects which may not be captured by entailment alone. Moreover, a reward that is based only on entailment raises the risk of degenerate solutions, leading to either highly extractive (Ladhak et al., 2022) or less informative summaries (“reward hacking”; Amodei et al., 2016; Skalse et al., 2022; Pan et al., 2022).
To address these issues, we propose Reinforcement Learning with Entailment Feedback (RLEF): Start with a model trained to produce summaries with the conventional cross-entropy objective, and further fine-tune it using RL with an entailmentbased reward. Throughout the RL procedure, we constrain the candidate models to stay close to the initial model. This way, while the model is being corrected for higher consistency, it also retains other summarization capabilities that were learnt with the maximum-likelihood (MLE) objective. In this work we explore the consistent vs. informative trade-off in our RL-based summaries w.r.t. various aspects including model scale, regularization and decoding strategies. We find those aspects to be highly important and interdependent for the final model performance, highlighting the importance of carefully tuning them.
Our work stands in contrast to two prior RLbased approaches. The first approach induces a reward function from human feedback that encompasses various task-specific requirements into a single value (Böhm et al., 2019; Stiennon et al., 2020). Collecting such feedback is expensive and requires dedicated data collection for each target task. In contrast, we use readily-available models and datasets for the reward, which address a specific aspect of generation that is generic across many different tasks. Other works modeled the reward using different similarity functions between the reference and the generated summaries (Pasunuru and Bansal, 2018; Gunasekara et al., 2021), thus requiring reliable reference data. Instead, our
reward function evaluates the generated output only w.r.t. the input, enabling to train using RL on data without reference summaries. We evaluated our approach on the widely used XSum (Narayan et al., 2018a) dataset, using both automated metrics and human raters. The results show considerable improvements over strong baselines for factual consistency, salience, and conciseness of the generated summaries.

heading: 2 Method
text: We would like to increase factual consistency using an entailment-based reward, while retaining the high salience and coherence that current summarization models already obtain. To achieve this, we propose to initialize an RL policy with a summarization model trained on supervised data (the anchor model). From there, in each RL-based training step we update the parameters according to two signals: an entailment reward and a regularization term grounded on the anchor model. During RL training, the entailment reward directs the model towards increased faithfulness, while the regularization term keeps the model from drifting to degenerate solutions and “forgetting” how to summarize. The process is illustrated in Figure 2.

heading: 2.1 RLEF: RL from Entailment Feedback
text: Problem Formulation. We denote the input document and output summary as x, y respectively. Let V denote the input and output vocabulary, and y:n = (y1, ..., yn) denote the generated summary up to the n-th token. We define the token-wise generative summarization process as a deterministic Contextual Markov Decision Process (CMDP, Hallak et al. 2015) with observable context, where the context is the input text x, the state at the n-th token generation is the sequence generated thus far y:n−1, and the action space is defined over the vocabulary V. A policy π(· | y:n−1, x), is a probability distribution over all tokens in V, conditioned on the context and state. We note that following this formulation, the policy is identical to a tokenlevel auto-regressive language model (Bengio et al., 2003). The RL objective is to find the optimal policy, which maximizes the cumulative reward signal.
Rewards. We use an NLI classification model as a factual consistency reward signal. Since the model is trained to evaluate complete utterances and expects as input a grammatical premise (document) / hypothesis (summary) pair, we use sequence-level rewards and define the token-level NLI reward to be zero on every token except for the end-of-sequence (EOS) token. For the EOS token we set the reward to be the log-probability for an “entailment” decision according to the NLI classifier, using x as the premise and y:n as the fully generated hypothesis:
rNLI(yn; y:n−1, x) =
{ NLI(y:n;x) yn = [EOS];
0 otherwise,
where [EOS] is an end-of-sequence symbol, and NLI(y:n;x) = log Pr(entailment | y:n, x).
To retain the summarization capabilities of the anchor model, we use Kullback-Leibler (KL) regularization to keep the RL-based policy close to the supervised anchor policy (Jaques et al., 2017):
rKL(yn; y:n−1, x) = log πSL(yn | y:n−1, x) πRLθ (yn | y:n−1, x) .
This term is added to the NLI reward, producing the final token-level reward:
r(yn; y:n−1, x) = (1− α)rNLI(yn; y:n−1, x) + αrKL(yn; y:n−1, x) . (1)
The hyperparameter α enables controlling the tradeoff between enforcing faithfulness through the reward and remaining close to the anchor policy.
Training Algorithm. We train the policy to optimize for the rewards defined in Equation (1) using an on-policy actor-critic policy gradient (PG) approach. Since we keep proximity to the anchor model via the KL penalty reward, the algorithm can be considered a regularized PG algorithm, similarly to works by Geist et al. (2019); Shani et al. (2020); Abdolmaleki et al. (2018); Tomar et al. (2022); Vaswani et al. (2021); see Appendix C for a detailed formulation. Specifically, two models are learned: a policy (the generation model) and the expected value of the policy (the value network). We use the supervised model to initialize the parameters of both models, with the exception that the last layer of the value network outputs single scalars instead of a distribution over the vocabulary.
The RL training process consists of the following stages: (1) Generating summaries with the current policy and (2) Scoring the summaries using the reward signal. Then, (3) Policy and value networks are trained, jointly: the policy is trained via the PG loss while using the value for generalized advantage estimation (GAE, Schulman et al. (2016)); the value is trained via standard bootstrapping, using the GAE predictions. Notably, this process does not require reference summaries for learning the policy. More details regarding the algorithm and losses can be found in Appendix A.

heading: 2.2 Decoding at Inference Time
text: As a direct consequence of RL training, the model explicitly learns to generate tokens with the goal of maximizing the long-term sequence reward. This is in contrast to MLE-based training, where the model learns to generate each token myopically, requiring heuristic decoding strategies such as beam-search to plan ahead. As a result, we can use the more efficient temperature sampling instead of beamsearch when decoding from an RL-trained policy.1

heading: 3 Experimental Design

heading: 3.1 Data
text: We focus on XSum (Narayan et al., 2018a), an abstractive summarization dataset that poses challenges around factual consistency. XSum is compiled from 200K web-scraped BBC news articles, where the lead (introductory) sentence in every article is taken as the summary, and the rest of the
1We found that temperature sampling is sufficient for RL, while beam-search is required to improve the supervised policy.
sentences are taken as the source document. Due to this formulation, XSum summaries may contain additional information that was not repeated in the rest of the sentences. Indeed, prior work found that only 20% of the reference summaries in XSum are entailed from their source document (Maynez et al., 2020), and that summarization systems trained on XSum are likely to generate factually inconsistent summaries. For this reason we find XSum suitable for our experiments, as we would like to see if the RL-based reward could alleviate the factual inconsistencies that supervised models learn to generate based on this data.
We also experiment on two additional datasets to compare to prior work. The TL;DR dataset (Völske et al., 2017), using the same cleaned version provided by Stiennon et al. (2020), which contains 120K Reddit posts and their short summaries, and the CNN/DM (Nallapati et al., 2016) dataset. The latter contains 200K news articles and their bulletpoint highlights, which are mostly copied excerpts from article sentences. In this work we focus on abstractive summarization, and therefore evaluate our methods on CNN/DM with models trained, both supervised and reinforced, over TL;DR.

heading: 3.2 Entailment Model
text: In this work we focus on combining an existing entailment model as a reward in an RL framework. We employ the NLI classifier from Honovich et al. (2022) across our study as a reward as well as for evaluation and data labelling for baseline methods. It was trained over the ANLI dataset (Nie et al., 2020) with the T5-XXL architecture. The classifier produces the characters ‘1’ or ‘0’ as its output for binary entailment and non-entailment decisions, respectively. We pose the source document as the premise and the predicted summary as the hypothesis, and use the log-probability of the decoded character ‘1’ conditioned on the input as our reward.2 We leave improvements to the underlying factual consistency models for future efforts. See Section 6 for more discussion about different factual consistency models.

heading: 3.3 Baseline Methods
text: SL. Our supervised learning baseline is obtained by fine-tuning a T5 model on document-summary pairs. We use the T5X framework (Roberts et al.,
2We use an empirically validated classification threshold of 0.5 for entailment decisions.
2022) for fine-tuning with batch size of 32 and keep the other hyperparameters to their default values (see Appendix A for details). Fine-tuning is stopped once the model converges in terms of ROUGE on the validation set. This supervised baseline will also be used as the initialization checkpoint of our RL methods. Decoding a summary using this model is implemented using beam search.
Filtered. Similar to the SL approach, with the distinction that we filter out training data where the summaries are not entailed by the input document according to our NLI model. This filtering leaves 60% of the original XSum training set. We train the model similarly to the SL model, and evaluate on the full validation and test splits, without filtering.
CTRL. Inspired by Filippova (2020); Rashkin et al. (2021b), we train the model on the full training set to explicitly differentiate between generating faithful and unfaithful summaries: each training document is prepended with a phrase indicating if the target summary is entailed or not according to our NLI model. At inference, since we aim to produce consistent summaries, each document is always prepended with the phrase denoting an entailing summary, and continue decoding the summary using beam search. Other parameters are similar to the SL method.
FactPegasus. Wan and Bansal (2022) employ a tailored pre-training setup similar to PEGASUS (Zhang et al., 2019) that also takes factual consistency into account, and combine it with data pre-processing, and contrastive learning to generate more faithful summaries.
CLIFF. Cao and Wang (2021) propose a contrastive learning objective that distinguishes between reference and heuristically created noisy summaries.
RLHF. Stiennon et al. (2020) uses an RL approach with a reward model that learns from human comparisons of summaries. They iteratively add new feedback from humans for summaries generated by the current policy, and re-train the reward model. We use their publicly released samples of the TL;DR validation set and the CNN/DM test set.

heading: 3.4 Proposed Models
text: We train two flavors of RL-based models. The first, RLEFL, gives a lower weight to the regularization
reward by setting α = 0.1 and the sampling temperature to 1. The second model, RLEFH, gives a higher weight to the regularization reward with α = 0.2 and a sampling temperature of 0.3. We altered both the α values and the sampling temperatures since we saw that both parameters affect the trade-off between factual consistency, as measured by the NLI model, and lexical similarity, as measured by ROUGE (see Figure 3). For additional implementation details see Appendix A.

heading: 3.5 Automatic Evaluation Metrics
text: We report the common lexical n-gram overlap evaluation metrics and a set of factual consistency metrics, as the former were shown to be ill-suited for detecting unfaithful outputs (Falke et al., 2019; Pagnoni et al., 2021).
For factual consistency, we report NLI, which is the percent of entailed summaries according to our NLI classifier, and the Q2 score (Honovich et al., 2021). Q2 is similar to QAGS (Wang et al., 2020) and QuestEval (Scialom et al., 2021) but was shown to work better on XSum data (Honovich et al., 2022) with higher correlation with human judgements.
When optimizing for faithfulness, an RL policy may resort to less abstractive summaries that are copied verbatim from the source (Ladhak et al., 2022), or less informative ones with a reduced level of detail. To explicitly measure these attributes in a summary, we report extractiveness metrics: COVERAGE and DENSITY (Grusky et al., 2018), where the first measures the percent of summary tokens that also appear in the document, while the second measures a quantity similar to the average length of extractive spans in the summary. Finally, we report the average summary length3 (LENGTH).

heading: 3.6 Manual Evaluation Protocol
text: We asked human evaluators to rate a sample of the XSum test-set from several selected methods. Each summary was evaluated by 3 different raters. Inspired by Fabbri et al. (2021), we pose 4 questions outlining comprehensibility, attribution, salience and conciseness (see example in Figure 5 in the appendix). To get conclusive results, similarly to Rashkin et al. (2021a) we request binary yes/no answers and ask to answer “No” for any slight devi-
3We use SequenceMatcher::get_matching_blocks from the python standard library to compute the set of extractive spans. Texts are tokenized with NLTK (Loper and Bird, 2002).
ation from the desired property. For unfaithful summaries, the evaluator also provides the offending phrase. Our evaluator pool consists of 11 workers that successfully completed a short training round of 10 examples (for details, see Appendix B).

heading: 4 Results
text: Automatic Evaluation. Table 1 presents the automatic evaluation results on the XSum test set, comparing the supervised baselines to the two RLbased models (RLEFL, RLEFH).
The table shows that the RL-based models achieve the highest entailment scores as measured by the NLI and Q2 metrics. Notably, the RL approach is the most effective approach to utilize the NLI signal, scoring favorable compared to supervised baselines Filtered and CTRL, which leverage the same signal.
Analyzing ROUGE reveals the trade-off between the entailment and other summarization traits. Without strong regularization, RLEFL scores highest on entailment but lower on ROUGE, indicating that in order to reach higher factual consistency, the model pushed farther away from the supervised starting point. The more strongly reg-
Size Method COMPREHENSION ATTRIBUTION SALIENCE CONCISENESS
XXL SL 99.0 ± 1.1 27.3 ± 5.0 61.6 ± 5.5 35.0 ± 5.4 Filtered 96.3 ± 2.1 31.3 ± 5.2 61.3 ± 5.5 34.3 ± 5.3 RLEFL 98.7 ± 1.3 56.6 ± 5.6 78.0 ± 4.7 61.0 ± 5.5 RLEFH 98.0 ± 1.5 39.0 ± 5.5 70.6 ± 5.1 45.3 ± 5.6 Base RLEFH 96.0 ± 2.2 38.3 ± 5.5 64.3 ± 5.4 44.3 ± 5.6 CLIFF 99.3 ± 0.9 28.3 ± 5.1 58.3 ± 5.6 33.3 ± 5.3 XSum reference 99.3 ± 0.9 23.6 ± 4.8 62.6 ± 5.4 30.3 ± 5.2
Table 3: Human evaluation results over 100 test set samples, each summary rated by 3 workers, results are micro-averaged. Each value corresponds to a percentage of positive answers per category with 95% confidence intervals around the sample proportion. Highest values for each model size are in bold.
ularized RLEFH achieves a ROUGE score on par with the CTRL and SL baselines, suggesting that our KL-regularization prevented the policy from drifting.
Looking at extractiveness, the Density metric suggests that RL policies do not resort to copying text, and the increased Coverage implies that they tend to use more terms from the document, suggesting fewer hallucinations. Lower ROUGE scores may hint at lower quality summaries for the less regularized entailment model, yet the other metrics actually point at higher conciseness. We next present our human evaluation to shed light on these differences, and analyze whether the improvement in entailment is also captured by human readers, and whether the lexical divergence from the reference summary affects has implications on salience or conciseness.
Human Evaluation. The results of our human evaluation are detailed in Table 3. Our raters fully agreed on 60% of the examples regarding attribution. From attribution (factual consistency) perspective, the results strengthen the evidence that the RL approach is superior to other methods by a large gap. Interestingly the XSum reference summaries scored lowest with 23.6%, showing that they are ill-suited to serve as faithful references for ROUGE and similar reference-based metrics. Notably, the human attribution evaluation was much stricter than the NLI metric, with much lower scores for all models, and we analyze this discrepancy in Section 5.
Surprisingly, the RLEF models outperforms all other models also on Salience and Conciseness. Specifically, the less regularized RLEFL learned to generate not only the most factually consistent summaries but also to improve on Salience and Conciseness, indicating that they are correlated w.r.t human quality perception.
Comparison with RLHF. We applied our RL approach on the TL;DR dataset. We used the same input format and data split as in Stiennon et al. (2020) for both the supervised and RL training processes. For the supervised model (SL) we used hyper-parameters identical to our previous experiments (see Appendix A) except for a batch size of 128 and learning rate of 2e-4.
We compared our results using automated metrics with the RLHF approach (Stiennon et al., 2020). This approach is also based on the T5 model and uses a similar RL setup, yet it employs a reward model trained on task-specific human preferences and applying a KL-based anchor. The results, detailed in Table 2, show that RLEF achieves higher entailment scores in both NLI and Q2 metrics, while our supervised model is on par with RLHF. We also note that RLHF produces noticeably different and longer summaries compared to our supervised baseline, while RLEF maintains similar length and ROUGE to the supervised baseline.
We also compared the two approaches in a transfer learning setting, where we predicted a summary on a different dataset (CNN/DM) using models trained on TL;DR. The results show similar trends, with higher entailment score for RLEF. These results hint at the benefit of utilizing a general NLI reward function, which managed to outperform the domain-specific RLHF reward both on the source domain and on a transfer setting.

heading: 5 Analysis
text: Regularization and Sampling Temperature. Figure 3 describes an ablation experiment where we vary the regularization α and the decoding temperature and measure the effect on different automatic metrics. Higher sampling temperature correlates with higher entailment and lower ROUGE scores. We conjecture that this is since higher temperature generates more diverse summaries, which amplifies exploration away from the original gold references. A similar phenomenon is observed when considering token length, as lower temperature policies produce summaries closer in length to the data-mean than their higher temperature counterparts.
As for the regularization coefficient α, we observe the expected trade-off: lower regularization (smaller α) leads to higher entailment (NLI), lower similarity to the supervised summary (ROUGE),
and higher Coverage. These may be explained by removal of external hallucinations that often use vocabulary terms that are unrelated to the document.
Surprisingly, in each KL setting, the lower temperature policy favors more document-aligned terms (perhaps for their higher initial probability), yet this is not reflected in the NLI metric, that stays lower than its higher-temperature counterpart. We also observe that the summaries get shorter with less regularization, as the policy learns to mention fewer details as a way to alleviate generating inconsistencies.
Model Size. We tested our approach with different model sizes to study the effect of scale in the RLEFH setup. We compared T5-Base (220M parameters), T5-Large (770M) and T5-XXL (11B), using the same hyper-parameters for all three models. Figure 4 shows the entailment rate on the XSum validation set during RL-finetuning. For all model sizes, our approach improved the entailment ratio over the supervised model by a large margin.
However, while the Large and XXL models changes the average summary length only slightly, the Base model completely degenerates, “hacking” the NLI reward by generating summaries that are half as short as the reference. This suggests that higher-capacity models are essential to prevent reward hacking, perhaps due to two possible reasons. First, the larger policies have higher generalization capabilities overall and can better accommodate different rewards, such as entailment and summarization regularization in our case. Second, since the anchor model uses the same architecture, the higher capacity anchor model is more robust to changes in the summary and produces lower scores for less informative or more extractive summaries.

heading: 5.1 Manual Analysis.
text: To gain more insight into the inner workings of RLEF, we propose two manual inspections about the types of changes being induced by the policy, and analysis of attribution errors found by our human evaluation procedure.
Changes to the summary during RL training. We study the changes that the RLEFH policy induces on a summary during RL training, focusing on the changes that cause a flip in entailment decision. We sample 200 documents from the validation set for which we obtain the predicted summary at different checkpoints throughout the RL training process in 4K steps intervals. We apply the NLI classifier for each document and summary list, and select 60 examples for which the NLI decision has flipped between any pair of consecutive checkpoints, and study what changes have been made to the summary that caused the flip. Notably, most flips occur only once during training, and from the non-entailed to the entailed decision. Examples are shown in Table 4 together with our categorization of the changes, with some summaries morphing in more than one way. We notice that for summaries produced by RLEFH most changes are local, meaning that the main predicate clause and the
Summary before the NLI flip Summary after the NLI flip NLI Description
1 Two astronauts who spent a year living on the International Space Station have landed in Florida. Two astronauts who spent a year living on the International Space Station have returned to Earth. 3 Abstractive Rephrasing 2 Afghan forces have repelled an advance by Taliban fighters on the northern city of Kunduz, officials say. Afghan forces have been battling Taliban insurgents in the northern city of Kunduz. 3 Abstractive Rephrasing 3 A senior Nigerian military official has said militant Islamist group Boko Haram is no longer a threat, after a mosque attack that left at least 82 people dead. A senior Nigerian official has denied that Islamist militant group Boko Haram was behind a mosque attack in the north in which more than 100 people were killed. 3 Claim Change 4 Two people have been arrested on suspicion of manslaughter after a threeyear-old boy died at a water park. Two people have been arrested after a four-year-old boy died at a water park. 3 Argument Omission 5 Bolton Wanderers manager Lee Trotter has apologised after he and striker Aaron Lennon swore at fans on live television. Bolton Wanderers manager Lee Trotter has apologised after he and teammate Gary Caldwell swore at fans on live television. 7 Argument Change
Table 4: Examples of summaries for the same document on consecutive checkpoints during RL training, before and after the NLI classification of the summary has flipped. The summaries maintain a stable main structure which enables manual inspection. The main changes are highlighted in gray, the NLI column depicts the entailment decision for the latter summary, and the description specifies the type of the semantic change.
core participants remain the same throughout most checkpoints. We classified 13 out of 60 examples as abstractively rephrased, where a specific detail is replaced with a broader description, e.g. returned to earth instead of landed in Florida (ex. 1). However, we also found that 27 examples contained argument omissions, where verbal arguments or noun modifiers with typically non-core semantic roles (Palmer et al., 2005) are removed (e.g. Locative or Temporal descriptions). See for example the “Cause for arrest” omission in ex. 5. Such omissions keep the information regarding the main participants intact, while lowering the risk of errors around non-core details. Other changes included claim changes (16 cases) where a predicate has been replaced (see ex 3), argument replacements (8 cases), and other non-specific alterations.
Attribution error analysis. We analyzed attribution errors from the human evaluation of our best policy, RLEFL, aggregated by majority vote. We inspect the offending phrase supplied by the evaluator for 39 out of 100 examples that are found to be non-attributable. 28 are considered as a local hallucination, mostly confirming to addition of personal names, numbers, places, and roles that did not appear in the article. For example, an article mentioned Kevin O’Malley without alluding to his job title, while the summary referred to him as the Irish Ambassador. While Kevin O’Malley was indeed an Irish ambassador, the model should not add such details if they are not explicitly mentioned in the article. Since most of these examples were found as entailing by our reward, this may point at issues with the NLI model that are due to knowledge conflicts between its parametric and contextual knowledge (Neeman et al., 2022). The rest of the examples include 5 contradictions and 5 major hallucinations.

heading: 6 Related Work
text: RL for text generation. RL has been applied to many text generation tasks like neural machine translation (Wu et al., 2018; Leblond et al., 2021), extractive summarization (Narayan et al., 2018b; Wu and Hu, 2018; Gao et al., 2019; Arumae and Liu, 2019), abstractive summarization (Chen and Bansal, 2018) and others (Bahdanau et al., 2017; Welleck et al., 2019; Bai et al., 2022a; Ouyang et al., 2022; Bai et al., 2022b).
Specifically for summarization, prior RL approaches used different reference-based metrics as a reward function. In Pasunuru and Bansal (2018), two reward signals are measured between the generated and reference summaries: lexical overlap (ROUGE) to gauge salience and an entailment score to measure factual consistency. Gunasekara et al. (2021) employed a similar approach with question-answering, they produced QA pairs conditioned on the generated summary to detect inconsistencies with the reference, and another set of QAs conditioned on the reference to measure salience. Additionally, Nan et al. (2021) proposed QUALS, a more computationally efficient QA approach, that was used in a contrastive learning setting. While their approach could be used without comparing outputs to reference summaries, they observed that adding such comparisons with the reference is essential for the stability of their method. We note that for some datasets, reference summaries are likely to contain factual errors (Maynez et al., 2020), decreasing the effectiveness of reference-based rewards.
Other RL methods, instead of explicitly defining the quality of a summary suggest to model it directly from human feedback (Böhm et al., 2019; Ziegler et al., 2019; Wu et al., 2020; Stiennon et al., 2020). This technique can prevent errors due to ref-
erences that are misaligned with human judgment. While it is a promising approach, it also requires acquiring task-specific annotation, which can be labor-intensive.
Another hybrid approach interleaves a crossentropy objective with policy gradients (Pang et al., 2021) in multi-document summarization (MDS). They use an in-domain NLI model, for which they annotate their MDS dataset with entailment decisions. To stabilize their policy they employ an additional GAN-like training regime and add a discriminator loss between generated and reference summaries to their reward.
Trade-offs in consistency models. The choice of which factual consistency approach to use has interesting consequences for the RL setup. Our work employs a binary NLI decision that does not point towards the specific inconsistent parts in the output summary. Consequently, the reward is assigned to the final token of the summary, leaving proper credit assignment to the RL algorithm. Other methods, specifically those based on question-answering (Durmus et al., 2020; Wang et al., 2020; Honovich et al., 2021) can frame misaligned answers in the generated summary and assign the reward explicitly to the offending tokens. However, these QA-QG based methods may be much slower to compute. Our reward requires a single forward pass using a transformer model over the document-summary pair, in comparison, QA-QG approaches require generating answer candidates, questions, answers from both sources and computing answer alignment. Some of this complexity is remedied by generating jointly questions-and-answers (Nan et al., 2021), but it still requires a lengthy decoding of QA pairs. A different NLI-based approach decomposes the document and summary into smaller blocks of sentences (Laban et al., 2022) and aggregates the final decision over a matrix of block-level NLI scores. Such approach could aid the RL algorithm with credit assignment when generating long summaries. In practice, the abstractive summarization datasets in this study use short single sentence summaries.

heading: 7 Conclusions and Future Work
text: We propose to leverage NLI models as a readymade, reference-free reward signal for RL training of factually consistent abstractive summarization models. Our experiments and thorough analysis with automatic and human evaluation show promis-
ing results for this approach, with our RL approach outperforming all baselines on factual consistency, while maintaining and even improving on other desired summarization attributes as well.
In future work, we would like to extend this approach to other grounded generation tasks, like knowledge-driven dialog. In addition, we find it interesting to explore additional reference-free reward models for other summarization attributes (or for other tasks). Then, an important research direction would be to understand how to properly adapt our method to work with multiple such rewards.
Limitations
While our approach shows promising results in both automatic and human evaluation, it relies on two significant pillars: a strong entailment model and a strong initial summarization model. The NLI model implicitly encodes the biases and other data regularities that were part of the NLI training set into the generated summaries of our policy. This is well demonstrated by the gap between human attribution judgements and the automatic NLI metric. Our RL policies cannot improve on factual consistency errors if they are undetectable by the NLI reward. Hopefully, as NLI capabilities get better, so will the efficacy of RLEF and the abilities to automatically flag hallucinations and contradictions.
Secondly, a strong summarization model is essential for our method in two ways: as an initialized starting point for RL exploration and as an anchor point to a policy. While our RL training does not require any reference data and opens the possibility to use more un-summarized documents, it would probably not succeed as well without initializing from a high-quality supervised model.
Another limitation is that our experiments suggest that model size is important when using RLEF (Figure 4): both our summarization and NLI models are 11B parameters models. We believe it is important to further understand how to make our approach more robust to smaller models, to increase its computational efficiency and availability.
Ethics Statement
Our work aims at solving the ethical issue of addressing misinformation in automated text generation tasks. Yet, adopting automatic summarization by real users can amplify misinformation in cases where the model still makes an error or when the input text itself is not trustworthy. As we stated
in the limitations, our trained models heavily rely on other predictive models and therefore carry the biases of their training data, and may implicitly encode these into our generative process. Therefore, we believe that to reach real-world use, not just our method should be scrutinized but also the NLI and summarization datasets that were used to train these models. Thus, such methods should be used with caution and combined with other techniques to ensure humans are capable of judging the validity of the information generated by the model.

heading: Acknowledgements
text: We would like to thank our anonymous reviewers for their thorough comments and insightful suggestions.

heading: A Experimental Details
text: RL Algorithm Details. We use an actor-critic on-policy PG algorithm with a learned value function Vψ and a parameterized policy πθ to maximize the RL objective. The policy gradient w.r.t. to the regularized reward r(yt; y:t−1, x) defined in Equation (1) is
∇θJ(θ)
= Ex,y∼πθ [ T∑
t=0
∇θ log πθ(yt|y:t−1, x)Gαt ] ,
where for brevity we denote Gαt =∑T t′=t r(yt; y:t−1, x), the accumulated regularized return. For more details on the derivation of this expression, and framing the regularized objective as an RL problem, we refer the reader to Appendix C.
We use the value Vψ as a baseline, a statedependent function that can be subtracted in the policy gradient without changing it. This leads to the following equivalent policy gradient
∇θJ(θ) = Ex,y∼πθ [ T∑
t=0
∇θ log πθ(yt|y:t−1, x)×
( Gαt − Vψ(y:t−1, x)
)]
= Ex,y∼πθ [ T∑
t=0
∇θ log πθ(yt|y:t−1, x)AGAEψ (y:t) ]
where Aψ is termed the advantage function. Applying this PG can be regarded a variant of the REINFORCE (Williams, 1992) algorithm with a baseline. In practice, we replace the advantage in the expression above by generalized advantage estimation (GAE, Schulman et al., 2016), which allows to better control the bias-variance trade-off via the λ parameter:
AGAEψ (yt; y:t−1, x) = +∞∑
t′=t
(γλ)t ′−t×
( r(yt′ ; y:t′−1, x) + γVψ(y:t′ , x)− Vψ(y:t′−1, x) ) .
Finally, the above policy gradient definition leads to the following per-example loss for learning the policy πθ,
Lπ(θ)(y:t, x) = AGAEψ (y:t, x) log πθ(yt|y:t−1, x),
where the gradients are only propagated here w.r.t. the policy parameters.
The value Vψ itself is learned via regression towards the return estimate induced by GAE, which is equivalent to minimizing the GAE advantage:
LV (ψ)(y:t, x) = ( AGAEψ (y:t, x) )2 .
We now describe more intricate implementation details and hyper parameter choices.
RL Implementation Details. Given that we operate in the finite horizon setting, we naturally set the discount factor γ to 1. Similarly to the PPO algorithm (Schulman et al., 2017b), we normalize the advantages in a given batch of data so that they approximately follow a standard normal distribution. We also normalize the value loss by dividing it by the variance of the batch returns. An important difference between our implementation and the standard (regularized) PG implementation is that instead of treating KL penalties along a given sequence as immediate rewards, we accumulate those and treat the resulting quantity as a sequence-level penalty. We found this to lead to more stability in the RL procedure.
Unlike the conventional RL setting where both the policy and value are randomly initialized, in our case the policy is already fine-tuned to solve the required task. Thus, to make the value function accurate w.r.t. the already initialized policy, we observed that we needed a small number of iterations before the value estimation is sufficiently accurate to avoid detrimental policy gradients. To do so, we run RL fine-tuning for 20K steps, with a warmup of 5K steps for the value network. We also noticed that it was beneficial to use distinct values for the policy and value learning rates, so we decouple them in practice.
Optimization. We use Adafactor (Shazeer and Stern, 2018) with a learning rate warmup phase: the learning rate is linearly annealed from zero to the specified asymptotic value.
Hyperparameter Search. We noticed that the optimal value of the policy and value learning rates are highly correlated. Hence, we propose a decoupled hyperparameter search: we start by finding a suitable value learning rate by keeping the policy fixed. We then follow a standard grid search to find suitable values for the remaining hyperparameters including the policy learning rate, temperature and the regularization coefficient α. Specifically, in our hyperparameter sweep we used temperatures [0.1, 0.3, 1.0] and α values between 0.1 and 0.8
with a grid size of 0.1. Thus overall, our main sweep for the XXL model consisted of 24 runs of 20K iterations.
We list all the hyperparameters used (unless different values are mentioned in the text) in Table 5. For the learning rate warmup and policy update delay, note that the number of steps reported correspond to gradient steps of the RL fine-tuning procedure.
SL Implementation details. For the SL models, we decode summaries with beam search with a beam width of 4 and a brevity penalty of 0.6. For training we use the same optimizer with base learning rate of 0.001, batch size of 32, and a dropout rate of 0.1.
Resources. We used TPU-v4 chips to train all the models mentioned. Each of our T5-XXL based RLEF experiment ran for approximately 17 hours on 64 TPU chips. Furthermore, our main hyper parameters sweep included 24 such experiments, accounting for 1088 TPU-days.

heading: B Evaluator Demographics, UI and Instructions
text: We employed full-time hourly workers to rate the summary quality. Our raters consist of native English speakers, nationals from the U.S. and U.K. that hold graduate (70%) and high-school (30%) diplomas. We supplied them with 2 pages of instructions and additional examples, and conducted an initial pilot study and training batch before proceeding to rate the summaries. The UI that we used
is displayed in Figure 5. In what follows we attach the guidelines presented to the raters in the human evaluation described in Section 3.6. The guidelines are loosely based on Rashkin et al. (2021a).
B.1 Guidelines
In this task you will be presented with a news article and multiple summaries of the article, and you are asked to evaluate the summary quality. You will rate each summary with 4 yes/no questions. These questions ask if the summary is: Comprehensible and understandable. Attributable (supported) by the article - no contradicting or unattested information. Captures the main idea(s) behind the article. Concise - does not contain additional details beyond the key information in the article. Read carefully the text and the summary. The summaries may appear very fluent and well-formed, but contain slight inaccuracies that are not easy to discern at first glance.
Q1: Comprehensibility. An incomprehensible summary is not understandable due to significantly malformed phrases and sentences that are difficult to comprehend or make sense of. If there is any part of the summary that is unclear or hard to understand or malformed (e.g., partially cut-off or contains strange characters), select "No, not fully comprehensible". Summary When you leave it late, you leave it late is adding interest to your pension money as a result of the financial crisis. o, not fully comprehensible

heading: Q2: Attributable (Supported) by the article.
text: A fully supported summary contains information that can be found in the source article. No information in the summary is unattested when compared against the source news article. In other words, if you can say that “According to the news article. . . ” with the summary following this phrase, you should answer, “Yes, it is attributable.” If some key details in the summary are not supported by the article (e.g. missing from the article), inaccurately represent the information in the article, or contradicted by the article, then please mark “No, not fully attributable.”
Q3: Main Idea. A main idea captures a fact or theme that is central to the article’s discussion. It should involve the people, locations, or events that the article focuses on. If a main idea was removed from the original article, it would change the meaning, focus, or argument of the article. Note that
this question is NOT asking whether the summary includes ONLY main ideas.
In Q3, to the best of your ability try to distinguish between the following cases, some may be more rare than others:
• The summary is fully supported (yes to Q2) and captures the main idea (yes to Q3).
• The summary is fully supported (yes to Q2), but ignores the central point of the document (No in Q3).
• The summary contradicts the document in minor details or hallucinates some information (No to Q2), but the idea behind the document is mostly captured even if some details are incorrect (Yes to Q3).
• The summary contradicts the document in key details (No to Q2) to the level where the main idea is unrecoverable or largely missed (No to Q3).
Q4: Conciseness. A summary is concise if it includes only the necessary details and the important information in the article. If it includes any details which are not central to the article, it should be marked as "No, it is not concise". A summary may be concise even if some details are contradicting (i.e. you marked “No, it is not fully attributable” in
Q2) as long as those were part of the main idea of the article.
In Q4 we are trying to find if the summary contains substantial information that does not belong to the main idea. If some minor details in the summary are contradicting, yet they are part of the main idea, then this summary is still concise, the system made an error of attribution, but not of overgeneration.

heading: C Fine-Tuning Language Models with Reinforcement Learning
text: C.1 Language Generation as a Contextual Markov Decision Problem
In this appendix, we explain the connection between arbitrary language generation tasks and the Markov Decision Process (MDP) framework (Howard, 1960) which is widely used in RL. We recall that an MDP M is a tuple M = (S,A, γ, r, P ), where S is a state space, A is an action space, γ ∈ [0, 1] is a discount factor, r : S × A → [−rmax, rmax] is a bounded reward function and P : S×A → ∆S is a transition kernel. ∆χ denotes the standard simplex over χ. We represent sequential decision-making strategies as policies π : S → ∆A. At any point in time t, a policy π interacts in an MDP by observing the current state st, selecting an action at ∼ π(·|st), and accordingly receiving a reward rt = r(st, at), before ob-
serving a new state st+1 ∼ P (·|st, at). We define the return as the discounted sum of rewards in one episode of interaction: Gt = ∑T t′=t γ
t′rt′ , where T is called the horizon and is potentially infinite. We now introduce Contextual MDPs (CMDPs) (Hallak et al., 2015). They model the fact that a fixed context is available and determines the nature of rewards and dynamics. Formally, a CMDP is a tuple Mc = (C, fM ), where C is a context space and fM : c ∈ C → M is a function that maps a context to the corresponding MDP.
Any language generation task can be seen as the following interactive process: a language model observes the current state st = y:t−1 and context c = x, that is both the input text x and the text generated so far y:t−1, and selects a token at = yt. Thus, we can view any language generation task as a CMDP Mc = (C, fM ) with fM (c) = (S,A, γ, r(· ; c), P (· ; c)), with the policy π being the language model itself. The state space S is the set of all potential generations (either complete or incomplete). We suppose that the maximum length of generated text T , which is equivalent to the horizon, and that of the input text Tc are finite, which is a common assumption in NLP. Accordingly, if we note V the vocabulary (the set of all admissible tokens), we have S = ∪Ti=0Vi. Similarly, we have the context space C = ∪Tcj=0Vj . The action space A is the set of tokens that the policy can output at any point in time, that is the vocabulary, hence A = V. The discount factor γ is arbitrary and can be set to 1 given that the horizon is supposedly finite. The reward function r is also arbitrary, but in the case of interest exposed in the main text we set it to:
r(st, at; c) = { NLI(y:t;x) if yt = [EOS] or t = T, 0 otherwise.
Finally, and most importantly, the transition kernel is deterministic:
P (st+1|st, at; c) =    1 if [EOS] ∈ st and st+1 = st, 1 if [EOS] /∈ st and st+1 = y:t, 0 otherwise.
Indeed, any state that contains an [EOS] token can be considered an absorbing state.
C.2 Language Generation From a Pre-Trained Model as a Regularized Markov Decision Problem
While the previous formalism applies to all language generation tasks, we now describe a formal-
ism that specifically applies to the language generation task that is explored in the main text: language generation when a pre-trained model is available. It models the fact that we want generated text to be likely according to the pre-trained model, which we call anchor model in what is next. We note the corresponding policy πanchor. We consider the following reward function:
r(st, at) = (1− α)r(st, at) + αrKL(st, at),
with the regularization term:
rKL(st, at) = log π anchor(at|st)− log π(at|st),
where r is the reward function defined previously and α is a scalar controlling the regularization strength. We recall that the Kullback-Leibler (KL) divergence between the current policy and the anchor policy has the expression:
KL(π||πanchor)(st) = − Eat∼π [ log πanchor(at|st)− log π(at|st) ] .
Hence, the regularization term is an unbiased estimator for the KL divergence between current and anchor policies. Intuitively, it encourages the learned policy to keep a distribution that is close to the distribution over tokens induced by the anchor policy (the fine-tuned model). Since the learned policy evolves along training, the reward function we described is non-stationary, that is the reward for a given state-action pair (s, a) changes with the policy π. Hence, the modified MDP is best viewed as a regularized MDP (Geist et al., 2019). We define the KL regularizer as Ω(π) = KL(π||πanchor), which is a strongly convex function. We can show that this formalism is equivalent to the MDP with the non-stationary reward function described above.
C.3 Defining the Reinforcement Learning Objective
In this section, we show that the regularized reward defined in Equation (1) can be used together with any PG based algorithm. To do that, we show that for any MDP (see Appendix C.1), the policy gradient can be easily re-derived for our regularization scheme when using parameterized policies. This repeats the derivations in Schulman et al. (2017a); Geist et al. (2019). We denote trajectories τ = {s0} ∪ {at, st+1}T−1t=0 . By a slight abuse of notations we denote the probability of a given trajectory
under the policy π as π(τ), that we can decompose as π(τ) = P (s0) ∏T−1 i=0 π(ai|si)P (si+1|si, ai). We also denote Gt as the return of a trajectory starting from time-step t. Now, denote a parameterized policy πθ, and define the standard RL objective,
J(θ) = Eτ∼πθ [ T∑
t=0
r(st, at) ] ,
= Eτ∼πθ [G0].
The goal of RL is to find a parameterization θ∗ that maximizes the following objective:
θ∗ ∈ arg max θ J(θ).
The policy gradient theorem states that
∇θJ(θ) = Eτ∼πθ [ T∑
t=0
∇θ log πθ(at|st)Gt ] .
We now place ourselves in the specific regularized MDP defined in Equation (1) and Appendix C.2, with the reward regularization scheme, rα(s, a) = (1 − α)r(s, a) + α log πanchor(a|s)πθ(a|s) . Define the RL objective of interest, which adds a regularization term to the reward function,
J(θ) = Eτ∼πθ [ T∑
t=0
(1− α)r(st, at)
+ α log πanchor(at|st) πθ(at|st)
] .
For r(st, at), we repeat standard steps to rederive the corresponding policy gradient. However, we need to have a separate treatment for the KL regularization reward log π
anchor(at|st) πθ(at|st) , as it explicitly
depends on θ. We have:
∇θEτ∼πθ [ T∑
t=0
log πanchor(at|st) πθ(at|st)
]
= −∇θEτ∼πθ [ T∑
t=0
log πθ(at|st) πanchor(at|st) ]
= −∇θ ∑
τ
πθ(τ)
T∑
t=0
log πθ(at|st)
πanchor(at|st) ,
= − ∑
τ
∇θ ( πθ(τ) T∑
t=0
log πθ(at|st) πanchor(at|st) ) ,
= − ∑
τ
∇θπθ(τ) T∑
t=0
log πθ(at|st)
πanchor(at|st) ︸ ︷︷ ︸
A
− ∑
τ
πθ(τ)∇θ T∑
t=0
log πθ(at|st)
πanchor(at|st) ︸ ︷︷ ︸
B
.
We keep A as is and show that B is equal to 0:
B = ∑
τ
πθ(τ)∇θ T∑
t=0
log πθ(at|st)
πanchor(at|st)
= ∑
τ
πθ(τ) T∑
t=0
∇θ log πθ(at|st)
πanchor(at|st) ,
= ∑
τ
πθ(τ)∇θ T∑
t=0
log πθ(at|st),
= ∑
τ
πθ(τ)∇θ log πθ(τ),
= ∑
τ
∇θπθ(τ),
= ∇θ ∑
τ
πθ(τ),
= 0.
By putting all the pieces together we get the expression of the policy gradient for the modified RL objective:
∇θJ(θ)
= Eτ∼πθ [ T∑
t=0
∇θ log πθ(at|st) T∑
t′=t
rα(st, at) ] ,
Denoting Gαt , the return of the trajectory when using rα, this can be rewritten as,
∇θJ(θ) = Eτ∼πθ [ T∑
t=0
∇θ log πθ(at|st)Gαt ] .
Note that we recovered the standard policy gradient for the regularized reward rα (and corresponding return Gα). This means that by treating rα as the reward we can use any policy gradient method, to solve the new objective. Because this holds for any MDP, it holds for the specific MDP defined in Appendix C.1 for the summarization task. To see how this is concretely used in our approach to construct the PG losses, we refer the reader to Appendix A.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We discuss the limitations of our work in section 8 Limitations.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We discuss potential risks in our ethics statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2-3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We cite the dataset creators in section 3.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 3. We use well-known and publicly released datasets, with CC-BY-4 or MIT open licenses
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The intended use of the summarization datasets we employed is to advance research in summarization, as we did.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use standard, well-known and publicly available datasets. Some of the content has been filtered by the creators of the datasets to remove problematic content. We publish only aggregated numerical rating data collected by our evaluators.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3, we briefly mention the content type of each dataset.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, briefly, as we use the standard train validation and dev splits or splits used in earlier works (and we explicitly mention and cite those).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 3 and appendix A.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3 and Appendix A

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sections 2-3, Appendices

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 4-5. We report results of a single run for automatic metrics on mainstream dataset splits. Human evaluation results have confidence intervals.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Appendix A
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3-4, Appendix B

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3, Appendix B
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We hired full-time annotators and explained the essence of their work is for research purposes in NLP.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: A similar protocol was determined in previous studies.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3, Appendix B

heading: Future Work
text: In future work, we would like to extend this approach to other grounded generation tasks, like knowledge-driven dialog. In addition, we find it interesting to explore additional reference-free reward models for other summarization attributes (or for other tasks). Then, an important research direction would be to understand how to properly adapt our method to work with multiple such rewards. Limitations
While our approach shows promising results in both automatic and human evaluation, it relies on two significant pillars: a strong entailment model and a strong initial summarization model. The NLI model implicitly encodes the biases and other data regularities that were part of the NLI training set into the generated summaries of our policy. This is well demonstrated by the gap between human attribution judgements and the automatic NLI metric. Our RL policies cannot improve on factual consistency errors if they are undetectable by the NLI reward. Hopefully, as NLI capabilities get better, so will the efficacy of RLEF and the abilities to automatically flag hallucinations and contradictions. Secondly, a strong summarization model is essential for our method in two ways: as an initialized starting point for RL exploration and as an anchor point to a policy. While our RL training does not require any reference data and opens the possibility to use more un-summarized documents, it would probably not succeed as well without initializing from a high-quality supervised model. Another limitation is that our experiments suggest that model size is important when using RLEF (Figure 4): both our summarization and NLI models are 11B parameters models. We believe it is important to further understand how to make our approach more robust to smaller models, to increase its computational efficiency and availability