SlowGuess/ACL_2020_OCR huggingface dataset:
Context: Southwest China's Sichuan province has decided to build an inter-city high-tech industrial belt to serve development of Western China.
Title: Sichuan to Build High-Tech Industrial Belt
(b) Example of a title that can be directly generated from the context.
Figure 6: Examples of titles that require hallucinating new facts and titles that are directly entailed from context.
A Examples of Titles and Generations
Examples of ground truth titles. We present examples of titles in Figure 6 that require factual hallucination and can be directly entailed from context.
Examples of generated titles. We present examples of titles that from rejection sampling that are selected and that were rejected in sampling in Figure 7. As shown, rejected titles tend to be of lower quality.
B Proof of Lemma and Proposition
Lemma. We prove the lemma that all elements in  $\mathcal{P}_{c,p}$  are close to  $p$  in total variation.
$$
\sup_{q_{0}\in \mathcal{P}_{c,p}}|q_{0} - p|_{TV}\leq c
$$
Proof. By definition of  $\mathcal{P}_{c,p}$ , for any  $q_0$  there exists a  $q_1$  such that  $p = cq_1 + (1 - c)q_0$  so,
$$
\left| q _ {0} - p \right| _ {\mathrm {T V}} = \left| c q _ {0} - c q _ {1} \right| _ {\mathrm {T V}} \leq c
$$
Figure 7: Examples of sampled titles that were selected and rejected in rejection sampling at  $\alpha = 0.1$ .
Proposition. We prove that the truncated log loss bounds total variation.
Context: At least two people have tested positive for the bird flu virus in Eastern Turkey, health minister Recep Akdag told a news conference Wednesday.
Ground truth: Two test positive for bird flu virus in Turkey
Selected sample: Two reported positive for bird flu in Eastern Turkey
Rejected sample: Two officials fail to get good for bird flu in Eastern Turkey
Context: British investment fund Fidelity has increased its stake in Puma, the German maker of sportswear and equipment, to just over five percent, Puma said on Thursday.
Ground truth: Private equity firm Fidelity raises stake in Puma to over five pct
Selected sample: Fidelity increases stake in Puma
Rejected sample: Boost higher first-half stake in Puma says Puma
$$
\left| \hat {p} - p _ {\text {r e f}} \right| _ {\mathrm {T V}} ^ {2} \tag {4}
$$
$$
\leq \left(\left| \hat {p} - p _ {\mathrm {t}} \right| _ {\mathrm {T V}} + \left| p _ {\mathrm {t}} - p _ {\text {r e f}} \right| _ {\mathrm {T V}}\right) ^ {2} \tag {5}
$$
$$
\leq \frac {1}{2} \mathrm {K L} \left(p _ {\mathrm {t}} | | \hat {p}\right) + 2 c + c ^ {2} \tag {6}
$$
which follows from the triangle inequality, Pinsker's inequality, and using Lemma 1 to bound the remaining terms by  $c$ .
Summarization model hyperparameters. We used a standard OpenNMT-py model with global attention for all sequence-to-sequence experiments (Klein et al., 2017). It has a single LSTM layer in the encoder and two in the decoder.
For the baseline model, we train for 200,000 steps with SGD and an initial learning rate of 1. For the loss truncated model, we hotstart with 100,000 minibatch updates and subsequently with 100,000 minibatch updates with the truncated loss with an initial learning rate of 0.1.
(a) Prompt for measuring HUSE.
(b) Prompt for measuring factuality.
Figure 8: Prompts for measuring HUSE and factuality.
$k$  and  $p$  selection. A key parameter in top-  $k$  and top-  $p$  sampling are  $k$  and  $p$  respectively. These parameters trade off between diversity and quality. To select these values, we chose values of  $k$  and  $p$  that had similar entropies to our model trained with loss truncation.
Specifically,  $k = 100$  and  $p = 0.9$  matched loss truncation at  $c = 0.6$  for summarization (entropies of 18.08, 20.01, and 17.93 respectively).  $k = 2$  and  $p = 0.4$  matched rejection sampling for summarization at  $c = 0.6$ ,  $\alpha = 0.1$  (entropies of 3.71, 4.02, and 3.84 respectively).
D Crowd Worker Setup and Prompts
Crowdsourcing setup. For all human evaluations, we used Amazon Mechanical Turk (all prompts shown below). We sampled 312 context/title pairs to measure HUSE. For each generated title, we asked 9 crowd workers to measure the typicality of the generated title, as in Hashimoto et al. (2019). Each crowd worker responded to 24 generated titles.
For measuring factuality, we sampled 312 examples and for each example, we asked two crowd workers how much information in the generated title was present in the article.
Prompts. We show crowd worker prompts for measuring HUSE and factuality in Figure 8. The HUSE prompt was directly taken from Hashimoto
Condition	ROUGE-L
Truncation, c = 0.9	24.3
Truncation, c = 0.8	24.9
Truncation, c = 0.7	24.0
Truncation, c = 0.6	23.2
top-k = 100	22.8
top-p = 0.9	22.8
Table 5: ROUGE-L scores for loss truncation at various  $c$  and entropy-matched top-  $k$  and top-  $p$  decoding for summarization. As shown,loss truncation outperforms on ROUGE-L for a range of  $c$  .
Condition	BLEU
Truncation, c = 0.9	0.72
Truncation, c = 0.8	0.71
Truncation, c = 0.7	0.70
Truncation, c = 0.6	0.69
Truncation, c = 0.5	0.69
Baseline	0.64
0.72	0.64
Table 6: BLEU scores for loss truncation at various  $c$  and the baseline model on the E2E task. As shown, loss truncation outperforms the baseline on BLEU score at a range of hyperparameters.
et al. (2019) with an extra control.
E Further experiments
E.1 Sensitivity to  $c$
We investigate the sensitivity of loss truncation to the hyperparameter  $c$ . To do so, we vary  $c$  and measure ROUGE-L and BLEU scores, for summarization and E2E respectively.
We show results for summarization in Table 5 and E2E in Table 6 along with baselines. As shown, truncation outperforms on automated metrics on a variety of hyperparameter settings on automated metrics. We leave a full investigation of sensitivity to  $c$  as future work.
E.2 Combining Loss Truncation and Decoders
As loss truncation is a training method, it can be combined with alternative methods of decoding at inference time. As such, we perform a preliminary investigation of using top-  $k$  and top-  $p$  decoding with loss truncation.
We show ROUGE-L of loss truncation combined with various decoders and baselines for summarization in Table 7. As shown, top-  $k$  and top-  $p$  de
Condition	ROUGE-L
Log-loss, beam	41.4
Log-loss, full sampling	27.9
Truncation, top-k = 100	33.4
Truncation, top-k = 2	38.9
Truncation, top-p = 0.9	35.1
Truncation, top-p = 0.1	40.9
Table 7: Loss truncation combined with top-  $k$  and top-  $p$  decoding.
coding work with loss truncation and can improve sample quality.