SlowGuess/ACL_2020_OCR huggingface dataset:
Daniel Kang
ddkang@stanford.edu
hashim@stanford.edu
Neural language models are usually trained to match the distributional properties of large-scale corpora by minimizing the log loss. While straightforward to optimize, this approach forces the model to reproduce all variations in the dataset, including noisy and invalid references (e.g., misannotations and hallucinated facts). Even a small fraction of noisy data can degrade the performance of log loss. As an alternative, prior work has shown that minimizing the distinguishability of generated samples is a principled and robust loss that can handle invalid references. However, distinguishability has not been used in practice due to challenges in optimization and estimation. We propose loss truncation: a simple and scalable procedure which adaptively removes high log loss examples as a way to optimize for distinguishability. Empirically, we demonstrate that loss truncation outperforms existing baselines on distinguishability on a summarization task. Furthermore, we show that samples generated by the loss truncation model have factual accuracy ratings that exceed those of baselines and match human references.
Learning to generate text is a core part of many NLP tasks, including summarization (Nallapati et al., 2016), image captioning (Lin et al., 2014), and story generation (Roemmele, 2016). A common challenge to all these tasks is that references from the training distribution are not unique and contain substantial variations in phrasing and content (Wiseman et al., 2017; Dhingra et al., 2019). Learning to generate under a set of diverse and noisy references is challenging as some variations ought to be learned (e.g., paraphrasing) while others should not (e.g., hallucinated facts, ignoring prompts).
Existing training procedures for models seek to
match the underlying distribution, leading to models that replicate and sometimes even amplify unwanted behaviors such as hallucination during generation. For example, neural language models often produce fluent text that is unfaithful to the source (Tian et al., 2019; Wiseman et al., 2017; Lee et al., 2018). Existing work (Fan et al., 2018; Holtzman et al., 2019) has primarily addressed these issues by constructing decoders that implicitly remove unwanted variation when generating (see §6 for a detailed discussion of task-specific losses).
In this work, we argue that this phenomenon is not model specific, but is due to the widely-used log loss: we demonstrate that log loss is not robust to noisy and invalid references (§2). In particular, log loss requires that models assign probabilities to all potential test reference sequences. As a result, log loss is sensitive to outliers: invalid or noisy references with small probability mass can cause large changes in model behavior. We show that the brittleness of log loss, together with the noise in existing generation datasets, lead to low-quality and unfaithful generated text.
Instead of optimizing log loss, which has little correlation with model output quality (Theis et al., 2016; Hashimoto et al., 2019; Gamon et al., 2005), recent work on diverse generation models has proposed optimizing for the distinguishability of samples from the model and the reference. Distinguishability provides a natural and appealing guarantee: samples that are indistinguishable from human generated text will be as high quality as human generated text. Furthermore, we show that optimizing for distinguishability is robust in the face of noisy and even invalid data. Despite its appeal, distinguishability has not been widely used due to statistical and computational challenges. For example, existing methods that directly optimize for distinguishability have yet to match even naive log loss based baselines (Caccia et al., 2018).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718-731 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We propose a modification to the log loss, loss truncation, that has the benefits of distinguishability while being efficient to train. Loss truncation is as efficient to train as log loss, nearly as robust as distinguishability, and provides distinguishability guarantees via an upper bound. It achieves these properties by modifying the standard log loss to adaptively remove examples with high log loss. We additionally extend loss truncation with a sequence-level rejection sampling scheme that generates higher quality sequences by restricting the outputs to be high probability sequences.
We show that loss truncation with direct and rejection sampling outperforms standard log loss based generation methods (beam search, full sampling, top-  $k$ , and top-  $p$  sampling) on distinguishability, as measured by the HUSE score (Hashimoto et al., 2019). We additionally study the factual accuracy of a summarization system trained on loss truncation and show that our proposed approach produces summaries which improve upon all baselines (including beam searched models) and match references on factual accuracy.
2 Motivation and Problem Statement
Task and Background. We consider a natural language generation task with a conditional language model, where we are given a context  $x$  drawn from  $p(x)$  and our probabilistic model  $\hat{p}(y \mid x)$  produces an output  $y$  by approximating a (usually human) reference distribution  $p_{\mathrm{ref}}(y|x)$ .
In order to achieve this, many existing models are trained to minimize the Kullback-Leibler (KL) divergence,
$$
\operatorname {K L} \left(p _ {\text {r e f}} \mid \mid \hat {p}\right) = \underbrace {- E _ {p _ {\text {r e f}}} [ \log \hat {p} ]} _ {\text {l o g l o s s}} + \underbrace {E _ {p _ {\text {r e f}}} [ \log p _ {\text {r e f}} ]} _ {\text {n e g e n t r o p y}}. \tag {1}
$$
We refer to the first term of this divergence as the log loss of a model. The second term is commonly ignored as it is a constant with respect to the model. Minimizing the log loss has several practical benefits: 1) it is written as an expected loss (and is thus straightforward to optimize via stochastic gradient descent), 2) it factorizes across tokens in autoregressive modeling, and 3) it provides a guarantee on a model's goodness of fit (Eq (1)).
Unfortunately, log loss also suffers from several drawbacks. It is known to have little correlation with a model's sample quality and it can be brittle to invalid references in the training data.
Figure 1: Fitting a mixture of Gaussians with a single Gaussian using distinguishability (TV) and log loss (KL). As shown, log loss is extremely sensitive to outliers, resulting in poor estimation.
Log loss is not robust to noise. The KL divergence has intuitively correct behavior when each input  $x$  has a single correct reference  $y$ : it will maximize the probability of the single correct reference. However, log loss can be problematic when there are multiple correct references, of which some are invalid or difficult to model.
In particular, log loss is sensitive to invalid or noisy data because it requires that the model assign high probabilities to all potential references. Log loss is unbounded above: a model assigning zero probability to even a single reference makes the model incur an infinite overall loss.
We show a well-known example of this behavior with synthetic data. We consider fitting a single Gaussian to a mixture of two Gaussian in Figure 1. The reference distribution (blue) has a valid set of references at zero as well as variation that the model does not expect (e.g., invalid or noisy references) on the right. Minimizing the log loss results in a suboptimal model that is forced to span both groups. Furthermore, post-hoc processing the model does not help, as even the most likely output under the log loss trained model ( $\sim 3$ ) has low probability under the reference distribution.
In natural language generation, training sets can contain invalid or poor quality references. As such, these types of problems manifest themselves in tasks such as summarization (hallucinating facts), story generation (ignoring prompts and constraints), and captioning (ignoring parts of the image).
Much of the existing literature on faithful generation has focused on designing better models for valid references (via copying or attention constraints), but the example in Figure 1 shows that this alone may not be sufficient. The Gaussian 'model' in this case perfectly fits the mixture component
Context: For the first time in five years, Microsoft corp. is finally unveiling a new system for operating personal computers.
Title: Microsoft Makes Long-Awaited Software Upgrade Available to Businesses Thursday.
Figure 2: Example of an article title from the Gigaword dataset that requires hallucinating new facts such as 'Thursday' (colored red).
at zero but is still brittle because it cannot simultaneously fit the other group of (invalid) samples. Resolving this will require either a model which is designed explicitly to capture invalid references or a loss function that can ignore them.
Case Study: Hallucination in Summarization We show that low-probability reference sequences (e.g., Figure 1) are pervasive by examining the Gigaword summarization dataset (Rush et al., 2017). We manually classified 300 titles into two categories: 1) requires hallucinating new facts and 2) directly entailed from the context. We show an example of a reference that requires hallucination in Figure 2. In this example, a model that assigns high probability to the new fact (Thursday) must also frequently hallucinate dates on other examples.
We show the fraction of examples in each category in Table 1. As shown,  $35\%$  of titles require hallucinating new facts. Others have found this phenomenon to be pervasive in other datasets (Krysciński et al., 2019), including the CNN/DM dataset (See et al., 2017).
Studying the log loss of these examples<sup>1</sup>, we note that the average log loss of titles that require new facts is over  $1.7 \times$  the average loss of the titles that are directly entailed (Table 1) and the high-loss examples are clearly dominated by examples which require hallucination (Figure 3). In fact, we find that over  $80\%$  of examples with greater than 40 log loss requires some form of hallucination.
These statistics are similar to the toy example we presented earlier in Figure 1. A small but nontrivial fraction of invalid and unexpected data force the model to incur high losses. Much like in the earlier example, we can see that a model which aims to have low log loss on this dataset must spend a substantial amount of effort learning to hallucinate.
Distinguishability. Given that large-scale data
New facts	Directly entailed
Percent	35%	65%
Avg. log loss	34.3	20.5
Table 1: Fraction of the data and log loss of titles that require hallucinating new facts (left column) and titles that are entailed from the context (right column). As shown,  $35\%$  of titles require hallucinating new facts and the average log loss of titles requiring new facts is over  $1.7\times$  the loss of the directly entailed sequences.
Figure 3: Normalized histogram of log losses for titles that require hallucinating new facts compared to those that can be directly entailed. As shown, titles requiring new facts incur significantly higher loss and more than  $80\%$  of examples with greater than 40 log loss require hallucinating new facts.
will inevitably contain annotation errors and noise, we might ask whether there are effective alternatives to the KL divergence for training models. The distinguishability of samples from a model compared to the reference is one such objective. Distinguishability has recently gained attention as a way to learn and evaluate models based on both sample quality and diversity (Hashimoto et al., 2019; Zhou et al., 2019; Zellers et al., 2019; Gehrmann et al., 2019). We show that this objective also serves as a naturally robust alternative to the KL divergence for learning language models. Unfortunately, directly optimizing for distinguishability (e.g., via generative adversarial networks) is challenging (Caccia et al., 2018) and we show this works poorly in practice ( $\S 5$ ).
Distinguishability is defined as the error rate of an optimal classifier which seeks to distinguish samples from both the model and reference, and we will formally define this via the mixture
$$
y | x, z \sim \left\{ \begin{array}{l l} p _ {\text {r e f}} (y | x) & \text {i f} z = 1 \\ \hat {p} (y | x) & \text {i f} z = 0 \end{array} \right.
$$
where  $z\sim \mathrm{Bernoulli}\left(\frac{1}{2}\right)$ . We can now define  $L^{*}$  to be twice the optimal error in identifying samples
1The log loss was computed from a standard language model, see §5 for details.
from the model
$$
L ^ {*} := 2 \inf  _ {f \in \mathcal {X} \times \mathcal {Y} \rightarrow [ 0, 1 ]} \mathbb {P} [ f (x, y) \neq z ] \quad (2)
$$
Our measure of distinguishability, the total variation (TV) distance, is a linear function of this error
$$
| \hat {p} - p _ {\mathrm {r e f}} | _ {T V} = 1 - L ^ {*}
$$
where  $\hat{p}$  and  $p_{\mathrm{ref}}$  refer to the joint distributions  $\hat{p}(y|x)p(x)$  and  $p_{\mathrm{ref}}(y|x)p(x)$  for brevity. Note that distinguishability is inherently robust to the addition of any small fraction of noisy data (Donoho et al., 1988). Unlike the log loss, the model's loss on an example for TV is upper bounded by 1 (Eq 2). We show an example of TV's robustness in Figure 1, where a small amount of noise does not substantially affect the learned distribution.
Log loss as a surrogate for distinguishability. Distinguishability is both robust and provides sample quality guarantees, but is challenging to optimize (Caccia et al., 2018). One approach to optimize for distinguishability is to find an appropriate surrogate loss which serves as an upper bound. This is analogous to the use of logistic or hinge losses as a way to optimize for classification accuracy. For log loss, Pinsker's inequality (Csiszar and Korner, 2011) relates the KL divergence and distinguishability as
$$
| \hat {p} - p _ {\mathrm {r e f}} | _ {T V} ^ {2} \leq \frac {1}{2} \cdot \mathrm {K L} (p _ {\mathrm {r e f}} | | \hat {p}). \tag {3}
$$
This explains the empirical success of log loss in low-uncertainty situations, where KL is sufficiently small and this bound becomes tight.
Our approach will be to modify the log loss slightly by truncating the distribution. This truncated loss will be as easy to optimize as log loss, while being more robust and providing a tighter variant of Pinsker's inequality.
3 Loss Truncation
Intuition. We would like the model to ignore data that would force it to unnecessarily hallucinate at test time. Concretely, recall the toy example (Figure 1); there is a set of invalid references that force the model to be degenerate. If we could remove these these invalid references by truncating the distribution, the resulting model would be high quality. We can show that this intuition is theoretically justified, and that truncating (i.e., removing) an appropriate  $c$ -fraction of the data provides tighter bounds on the distinguishability of the model.
Improved log losses for distinguishability. We will demonstrate that log loss with an appropriate  $c$ -fraction of the data removed provides guarantees on distinguishability. We will define the set of truncated distributions as the set of distributions with any  $c$ -fraction of data removed
$$
\mathcal {P} _ {c, p} := \left\{q _ {0}: p = (1 - c) q _ {0} + c q _ {1} \text {f o r s o m e} q _ {1} \right\}.
$$
A simple lemma shows that all elements in  $\mathcal{P}_{c,p}$  are  $c$ -close to  $p$  in TV (Appendix B).
Now we state our main result,
Proposition 1. For any  $c \in [0,1]$  and  $p_{\mathrm{t}} \in \mathcal{P}_{c,p_{\mathrm{ref}}}$ ,
$$
| \hat {p} - p _ {\mathrm {r e f}} | _ {T V} ^ {2} \leq \frac {1}{2} K L (p _ {\mathrm {t}} | | \hat {p}) + 2 c + c ^ {2}
$$
See Appendix B for the proof. Namely, distinguishability is bounded by the log loss with respect to the truncated distribution and a small constant. Furthermore, this upper bound is valid for any  $c$ , although different  $c$  will change the tightness of the bound and produce different models.
This truncated bound can be substantially tighter than Pinsker's inequality. Consider for example a model that can perfectly capture  $(1 - c)$  fraction of the data, but  $c$ -fraction of the reference outputs cannot be generated by the model and receive probability zero. In this case, the distinguishability (TV) is  $c$ , the KL divergence is infinite, while our truncated bound is  $\sqrt{c^2 + 2c}$ . This suggests that appropriately truncating high-loss examples makes log loss robust and allows us to use log loss as a surrogate for distinguishability, even in the presence of invalid and noisy references.
Loss truncation. Given that the log loss on any  $c$ -fraction of the data is a surrogate loss for distinguishability (Eq (6)), a key parameter to optimize is the truncated distribution  $p_{\mathrm{t}}$ . An oracle solution would exhaustively search over  $p_{\mathrm{t}}$  and which data to drop. However, exhaustively searching through  $\mathcal{P}_{c,p_{\mathrm{ref}}}$  is a combinatorial optimization problem and infeasible. Our approach will be to optimize  $p_{\mathrm{t}}$  with a heuristic. The truncated objective takes the form of a log loss and negative entropy term,
$$
- \mathbb {E} _ {p _ {\mathrm {t}}} [ \log \hat {p} (y \mid x) ] + \mathbb {E} _ {p _ {\mathrm {t}}} [ \log p _ {\mathrm {t}} (y \mid x) ]
$$
and we will select  $p_{\mathrm{t}}$  by dropping the examples with the highest log loss, treating the negative entropy term as being upper bounded by zero.
This heuristic is straightforward to compute, provides an upper bound on distinguishability, and
Figure 4: Pinsker's inequality, our bound, and the total variation squared of parameter estimates for different parameter estimates  $(c = 0.2)$ . As shown, loss truncation can significantly improve bounds over Pinsker's inequality and, in this case, has a nearly identical minimizer to directly minimizing total variation.
matches our earlier observation that high-loss examples are correlated with invalid examples we would like the model to ignore (see Table 1).
As an example of how our heuristic can improve estimation and tightness in bounds, consider the earlier toy example in Figure 1. In this example, we find the optimal mean for a single Gaussian with fixed variance which fits mixture of two Gaussians. Figure 4 shows the objective function value implied by the TV loss, log loss (Pinsker's bound), and our  $c$ -truncated bound as a function of the Gaussian mean. We find that log loss provides an upper bound on distinguishability (via Pinsker's inequality) but is loose and results in a low quality estimate. In contrast,  $c$ -truncation results in a nearly identical minimizer as directly minimizing TV.
4 Implementing Truncation
Our algorithm has three components at training time. First, it trains a model on all the data using standard hyperparameters, which we refer to as "hotstarting" the model. Second, it tracks a running estimate of the  $1 - c$  quantile of the losses during training. Third, it performs gradient updates on examples that are below the current  $1 - c$  quantile estimate. We present the pseudocode in Algorithm 1 and describe each step in detail below.2
Hotstarting. First, our algorithm hotstarts the model (hotstart  $(M)$  in Alg. 1) by training with the standard log loss. Hotstarting address two challenges in optimizing the truncated loss. First, losses are uninformative at the start of training so trun
cating examples based on these losses will result in dropping valid examples. We have empirically found that truncating after hotstarting primarily drops invalid references, which avoids this problem. Second, hotstarting allows the model to transfer information from the entire dataset to the clean  $1 - c$  fraction of the data. Examples that cause a model to hallucinate may still contain valid information about the fluency of a sentence, which hotstarting can capture. This is effectively pretraining our model on the entire data before learning to generate on the clean subset. We have found this procedure to be effective in practice.
Quantile estimation. Second, our algorithm keeps track of the  $1 - c$  quantile over the distribution of losses. For each new minibatch  $B$ , we update an online estimate of the  $1 - c$  quantile (estimateQuantile  $(M, B)$  in Alg. 1). To estimate this quantile, our algorithm constructs a histogram over the last 10,000 examples seen during training and estimates the empirical  $1 - c$  quantile every 10,000 examples.<sup>3</sup>
Loss dropping. Third, our algorithm will perform minibatch stochastic gradient descent while excluding examples that have losses above the current top  $1 - c$  quantile estimate  $q$  (truncatedUpdate  $(M,B,q)$  in Alg. 1). Dropping can be accomplished in automatic differentiation packages (e.g., Tensorflow and PyTorch) by setting the loss on the given example to zero.
4.2 Generating High-Probability Samples
Thus far, our goal has been to robustly learn the underlying distribution. However, in some cases, a user may wish to only generate high confidence sequences, which will ideally correspond to high quality sequences.
To generate such samples, we propose sequence-level rejection sampling.
Recall that our truncation heuristic selects for the  $1 - c$  quantile of the distribution. For a user-defined level  $\alpha$ , our rejection sampling scheme will aim to generate samples from the  $1 - c \cdot \alpha$  quantile.
To perform rejection sampling, given a model and a user-defined rejection level  $\alpha$ , we first sample  $N$  sequences (e.g., titles in a summarization task). Then, we sample a random sequence from the  $\alpha \cdot N$  smallest samples as measured by log loss. Ideally,
For datasets with fewer than 10,000 examples, we can perform this procedure over the entire dataset.
2Our code is available at https://github.com/ddkang/loss_dropper.
Algorithm 1: The proposed loss truncation procedure with three components (see main text for details for each component).
this procedure will return a sample in the  $1 - c\cdot \alpha$  quantile of  $p_{\mathrm{ref}}$
We show that rejection sampling can outperform baselines in generating factual summaries (§5). We further show examples of selected and rejected samples in Appendix A.
Dataset and Task. We primarily evaluate loss truncation on abstractive summarization in the form of generating news headlines from an article. We selected this task to highlight that loss truncation can improve sample quality and factual accuracy, while also achieving the secondary goal of diversity for abstractive systems (See et al., 2017; Krysciński et al., 2019).
We evaluated on the Gigaword summarization task (Rush et al., 2017) as in Gehrmann et al. (2018). While there are other summarization datasets, we chose Gigaword for the following reasons. First, it is large enough that sample quality defects are not caused by a lack of data. Second, the dataset is structured so that neither model nor computation is the bottleneck in performance: the standard sequence-to-sequence models are competitive on the Gigaword dataset. Third, while Gigaword dataset is known to have noise, this matches the behavior of existing annotation errors (Beigman and Klebanov, 2009; Klebanov and Beigman, 2010) and uncertainty (Krysciński et al., 2019).
To show that loss truncation is applicable beyond summarization, we also performed a preliminary evaluation of our approach on the E2E NLG task. In E2E, the goal is to generate restaurant reviews from meaning representations (Dušek et al., 2019).
Model and Baselines. We used a standard LSTM architecture with global attention for summariza
tion that has been used for the Gigaword summarization task in the past (Gehrmann et al., 2018). The learning rate and hyperparameters are given in Appendix C. For the E2E task, we use a standard model with the exact settings as in Puzikov and Gurevych (2018).