SlowGuess/NAACL_2022_OCR huggingface dataset:
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
A More Details on Experimental Setup
Following PEGASUS, we pre-train on the C4 dataset, a large collection of documents from Common Crawl. We evaluate our pre-trained model on three downstream abstractive summarization datasets: XSum, WikiHow, and Gigaword. XSum is a collection of articles from the British Broadcasting Corporation, Gigaword is a large collection of news articles headlines, and WikiHow consists of how-to articles.
We show the dataset statistics for pre-training and fine-tuning in Table 5, where we present the number of examples in the training, validation, and test splits. We also show the number of examples corrected using the replace and remove method. All datasets are from datasets (Lhoest et al., 2021).
We use the ROUGE package provided by transformers (Wolf et al., 2020). We follow the instructions provided by the authors of the factuality metrics to set up and run their code. We report all scores of our models from single runs.
We use transformers library for the training script and the checkpoints of the pre-trained models. We use the default setting, including the AdamW optimizer and the linear rate scheduler. We also use mixed precision for both pre-training and fine-tuning the models. We conduct our experiments on the RTX A6000 GPU with 48GB memory and the A100 GPU with 40GB memory. BART-base model has 139M parameters, and PEGASUS* and FACTPEGASUS have 175M parameters.
A.4 Pre-training Setup
Model Architecture. We use the same architecture as BART-base. Specifically, the model has  $L = 6$ ,  $H = 768$ ,  $F = 3072$ ,  $A = 12$ , where  $L$  is the number of layers,  $H$  is the hidden size,  $F$  is the dimension for feed-forward layer, and  $A$  is the number of self-attention heads. We use the SentencePiece (Kudo, 2018) unigram model tokenizer from PEGASUS with a vocabulary size of 96,103.
Sentence Selection Criteria. Before pretraining the full model, we first determine the best sentence selection criteria that produces more factual summaries with comparable quality. We experiment with sentence selection criteria that use ROUGE-1, ROUGE-2, and ROUGE-L, as well as combining each with FactCC. To understand the effect of the pre-training objective on factuality directly, we evaluate the performance on the XSum dataset without applying any of our proposed fine-tuning modules. Following Zhang et al. (2020), we report the models' relative performance to the base model, which only uses ROUGE-1 as the selection criteria. We use the normalized ROUGE F1 scores  $\frac{1}{3} \left( \frac{R1}{R1_{base}} + \frac{R2}{R2_{base}} + \frac{RL}{RL_{base}} \right)$  where  $R1_{base}$ ,  $R2_{base}$ , and  $RL_{base}$  are the ROUGE F1 scores of the base model. We similarly report the factuality metrics by normalizing each score by that of the base model. We take the complement of token error and sentence error as token accuracy and sentence accuracy, respectively, to present all metrics where higher is better.
Similar to previous works (Lewis et al., 2020; Zhang et al., 2020; Raffel et al., 2020) that save computational resources when selecting strategies for pre-training, we pre-train these model on the realnewslike subset of the C4 dataset with less steps.
Pre-training Details. We use a learning rate of 1e-4, a weight decay of 0.01, and set the maximum number of input tokens to be 512 and a maximum number of output tokens to be 256. We use a batch size of 256. We pre-train the full model for 750,000 steps with a warm-up of 20,000 steps, and only pretrain the smaller models for the sentence selection criteria experiment for 250,000 steps. Pre-training the smaller models takes 30 hours, and pre-training the full model takes 90 hours.
Calculating FactCC Score. In practice, running FactCC on each sentence-document pair of the pretraining data is expensive. Thus, we opt to only calculate the FactCC score for the top 5 sentences
Dataset	Train	Validation	Test	Corrector
Replace	Remove
C4	364,868,892	364,608	-	-	-
realnewslike	13,799,838	13,863	-	-	-
XSum	204,045	11,332	11,334	54,036	152,716
WikiHow	157,252	5,559	5,577	8,077	71,936
Gigaword	3,803,957	178,651	1,951	115,896	1,296,168
Table 5: Dataset Statistics. We show the number of examples in each split, as well as the number of training examples changed using the replace and remove strategy of the corrector.
according to the ROUGE score between the sentence and the rest of the document.
A.5 Fine-tuning Setup
For all datasets, we use a label smoothing of 0.1. For decoding, we use a beam size of 6 for all datasets. Task-specific hyper-parameters are shown in Table 6. Fine-tuning on XSum and WikiHow takes 8 hours, and fine-tuning on Gigaword takes 11 hours. Decoding on XSum and Gigaword takes half an hour, while decoding WikiHow takes an hour. We use 5 negative examples for the contractor and set  $\lambda$  to 5 when calculating the combined loss. We set the temperature  $\tau$  to 0.05.
For fine-tuning DAE and CLIFF, we follow the authors' instructions and fine-tune BART-base with their respective code and hyper-parameters. For WikiHow and Gigaword, we use the same hyperparameters as above.
B Implementation Details for Corrector and Contrastor
B.1 Corrector
We use spaCy's NER model to find entities in the document and summary. Entities in the summary sentence are considered nonfactual if no matching document entities with the same string are found. We have previously experimented with the additional requirement of matching entity type similar to Kryscinski et al. (2020), but we find that this constraint unintentionally causes some correct entities to be considered hallucinating, leading to unnecessarily less informative summaries when removed.
Given hallucinated entities, we can perform either replace or remove operations. For replace, we find document entities whose words are all contained in the selected entity.
For the remove method, we need to make sure to also remove any related words. We use spaCy's
dependency parser to systematically remove those. The algorithm is as follows: We first add all the tokens in the selected hallucinated entity to the list of tokens to remove. Then, we recursively find all parents that contain the dependency relation of  $\text{pobj}$  and  $\text{prep}$  without any other children and add those to the tokens to remove. Finally, we add all children that do not have the label compound, relcl, and fixed. The final set of words will then be removed in the summary sentence.
We qualitatively observe that this approach can cover most of the edge cases that would otherwise result in ungrammatical sentences. Nevertheless, this method is not perfect. We include some sample output with the remove method in Figure 7. The algorithm is good at removing entities and related words, such as prepositions, as illustrated in example 1, 3, and 5. However, we observe that it will create ungrammatical sentences when the hallucinated entity is the subject (example 2), or the object of a transitive verb (example 6).
We leave exploration with the best systematic correction algorithm or models for future work.
B.2 Contrastor
Similar to Kryscinski et al. (2020), we generate hallucinated summaries by performing entity perturbation on the original summaries. We find entity candidates using the NER labels and sort them into three categories: We include MONEY, QUANTITY, and CARDINAL as number, DATE and TIME as date, and all other labels as named entities. We randomly select a factual entity in the summary and replace it with an entity belonging to the same category.
For extrinsic hallucinations, we sample candidates of the same category from the training corpus but exclude those present in the document. For the intrinsic case, we select to consider the entities from the document. The number of negative examples for all tasks is 5.
<sup>6</sup>We use the en_core_web_trf model.
Dataset	Learning rate	Num Steps	Warmup	Batch size	Max Input tokens	Max Target tokens
XSum	3e-5	15k	500	256	512	64
WikiHow	3e-5	15k	500	256	512	256
Gigaword	3e-5	50k	2000	256	128	32
Table 6: Hyperparameters for fine-tuning on the three tasks.
Pos.	XSum	WikiHow	Gigaword
R1	R2	RL	R1	R2	RL	R1	R2	RL
1	32.84	11.32	25.35	21.02	4.85	4.85	26.19	9.09	22.92
2	24.10	5.90	18.02	20.65	4.80	14.80	22.89	7.22	20.03
3	21.23	4.30	15.69	20.81	4.89	14.93	22.89	7.22	20.03
4	19.52	3.47	14.41	20.61	4.79	14.77	22.89	7.22	20.03
5	18.77	3.03	13.86	20.72	4.85	14.82	22.89	7.22	20.03
6	18.22	2.80	13.51	20.69	4.82	14.87	22.89	7.22	20.03
C Connector Result
This mask-token fine-tuning technique can be seen as a form of prompting, where we elicit our desired faithful abstractive summarization behavior from the pre-trained model directly. Specifically, we consider this as null-prompting (Logan et al., 2021), where using the mask token as the prompt can achieve competitive results with manually engineered prompts. Conveniently, since the mask token during pre-training already serves as a placeholder of where the summary sentence should be generated, it naturally serves as a valid prompt. Figure 1b shows an example of adding the mask token before the first sentence and thus creating a similar setup for pre-training.
We first need to determine the best position of mask token, as discussed in Section 3.2.1, where we insert the mask token before the  $i$ th sentence of the document, where  $i = 1,2,\ldots,6$ , and select the best position that achieves the highest ROUGE score on the dev collection. We report ROUGE score of all positions in Table 7 for the three datasets. Interestingly, we observe that the best mask token position for all datasets is before the first sentence. This agrees with the dataset generation of XSum: the summary is taken from the first sentence of the original article. For Gigaword, there is not a change after the first sentence, since the document only consists of a single sentence.
Table 7: ROUGE score on validation set when the mask token is placed at different position. Pos. indicates placing the mask token before the  $i$  th sentence. Pos. 1 indicates the beginning of the document.
Model	R1	R2	RL
BART-base	19.75	2.61	12.81
PEGASUS*	18.03	2.65	13.02
FACTPEGASUS	32.97	11.42	25.41
Table 8: ROUGE score in zero-shot setting on XSum. We apply the connector to our model. FACTPEGASUS outperforms BART base and PEGASUS* on all metrics.
Num Examples	RL	tok err↓	sent err↓	FactCC
0	25.44	7.69	38.88	35.14
1	22.36	7.69	38.85	35.15
10	23.44	7.69	38.85	37.78
100	25.44	5.15	36.48	37.95
1,000	27.03	5.67	32.97	34.38
Table 9: Full Result of zero-shot and few-shot experiments.
D.1 Sentence Selection Criteria Result
We report the full result for the sentence selection criteria in Table 10. Surprisingly, each sentence selection criteria that uses FactCC excels in one specific factuality metric: R1+FactCC is best at FactCC, R2+FactCC is best at sentence error, and RL+FactCC is best for token error.
D.2 Zero-shot and Few-shot
We present additional results of the zero-shot and few-shot experiments here.
Zero-shot We first report the reference-based result of the two baseline models and FACTPEGA-
Model	RL	tok err↓	sent err↓	FactCC
R1	29.04	12.31	60.65	23.93
R1+FC	28.99	12.13	59.93	24.81
R2	29.08	12.12	59.59	23.67
R2+FC	28.65	12.13	59.48	24.37
RL	29.23	12.17	60.08	23.06
RL+FC	28.62	12.10	59.63	24.58
Table 10: Full result of pre-trained models with different sentence selection criteria shown in Figure 3. We denote the criteria with FactCC with (+FC).
SUS in Table 8. Due to the mismatch of pretraining and fine-tuning, we observe that both baseline models perform much worse than their result when fully trained. However, with the help of the connector, we observe 11.5 ROUGE-1 points increase for our model compared to the baseline models, and almost four times and double the score for ROUGE-2 and ROUGE-L, respectively.
Few-shot We show FACTPEGASUS's full result of the few-shot experiment in Table 9.
D.3 Fine-tuning ablation on Gigaword and WikiHow
We report ablation of each fine-tuning components on Gigaword and Wikihow. The result can be found in Table 11. We observe similar trend as Table 3, where each component improves the performance. For WikiHow and Gigaword, the extrinsic method for contrastive learning perform the best. We think that this is due to the fact that the two tasks do not contain rich entities in the document, and thus require introduction of additional entities from the training corpus.
D.4 Fine-tuning ablation using PEGASUS*
We similarly perform the same ablation using the PEGASUS* model, which we present in Table 12. We observe similar trend as Table 3. We note that using our pre-trained model factGSG achieves better factuality than PEGASUS* in each setting.
E Human Evaluation Detail
To ensure high-quality annotations, we select the workers from the United States and have more than 10,000 number of HITS approved as well as an approval rate greater than  $98\%$ . In addition, we also create a qualification test where we rate the factuality of the selected generated summaries. Such examples include cases where some summaries hallucinate the first name of a person, which the
workers should mark them as not factual. Only workers with the correct annotation can perform the actual task.
To avoid giving too much text to the workers, we select the most important sentences and replace the less relevant sentences with an ellipsis. For each of the summaries, we select the ten most relevant sentences from the document by cosine similarity of the sentence embedding using SentenceTransformer $^{7}$  (Reimers and Gurevych, 2019). We combine and show all the selected relevant sentences from each summary. Since the summaries are similar, we see a large overlap of the relevant sentences.
We give the following prompt, which we modify from Dreyer et al. (2021):
- consistency/factuality: Please avoid using general knowledge, and only consider it in the context of the provided document. Select not consistent if facts in the summary are not supported by the document, such as cases like these:
1. The summary contradicts the information in the document. The summary might say "A fire broke out in Seattle", but a document says it broke out in Portland. Or the summary might say "the Republicans won the election", but the document indicates the Democrats won instead
2. The summary adds (hallucinates) a fact that is not mentioned anywhere in the document. For example, the summary might say that "A fire broke out at 2 am", but the document doesn't mention the time when the fire broke out.
- Informativeness: Please select informative if the summary expresses the main points of the document. Summary should contain relevant and important information and few unimportant details. If you select the summary to be not consistent with the document, please only consider the consistent information when evaluating this category.
The order of the summary is randomly shuffled. Each task consists of three unique workers, where we take the mean as the scores for this document. The final score is the mean factuality score across all documents. The average time for each task is
<sup>7</sup>We use the all-mpnet-base-v2 model.
Model	WikiHow	Gigaword
RL	tok err↓	sent err↓	FactCC	RL	tok err↓	sent err↓	FactCC
factGSG	30.16	9.55	46.84	99.12	34.39	2.72	22.30	56.89
+ corrector replace	30.14	9.71	47.60	98.92	34.45	2.68	21.27	55.20
+ corrector remove	29.91	9.40	47.39	99.19	34.33	2.53	20.25	59.71
+ corrector combined	30.00	9.30	46.86	99.14	34.07	2.49	20.45	58.85
+ contrastor intrinsic	30.21	9.53	46.94	99.15	34.50	2.72	21.48	56.18
+ contrastor extrinsic	30.15	9.52	46.76	99.19	34.03	2.59	20.91	56.63
+ contrastor + corrector	29.91	8.23	44.59	99.21	34.38	2.46	20.04	58.74
FACTPEGASUS	29.33	7.86	42.40	99.41	34.23	2.30	19.32	60.02
Table 11: Fine-tuning ablation on Wikihow and Gigaword. We combine the modules by using the corrector combined and contrastor extrinsic. Results of the final model is copied from Table 1.
Model	RL	tok err↓	sent err↓	FactCC
PEGASUS*	33.17	12.33	60.01	24.14
+ corrector replace	32.83	10.57	55.07	24.44
+ corrector remove	30.53	6.49	40.12	34.30
+ corrector combined	31.51	6.33	39.51	32.35
+ contrastor intrinsic	32.30	11.57	58.21	24.57
+ contrastor extrinsic	33.16	12.31	60.08	24.14
+ contrastor + corrector	31.46	6.22	39.46	32.39
PEGASUS* full	31.49	6.24	39.37	32.43
Table 12: Fine-tuning ablation on XSum using PEGASUS*. We combine the modules by using the corrector combined and contrastor intrinsic. We name the model with all three components as PEGASUS* full.
around 3 minutes and we pay 0.6 USD per task, hence an hourly rate of  $\geq \$ 12$  per hour.
We use bootstrap test (Efron and Tibshirani, 1993) to determine statistical significance between two summaries with  $p < 0.05$ .
E.1 Sample Outputs
We include some sample outputs in Figure 8 and Figure 9.
Example 1 (XSum)
Summary: Tap water in 80,000 homes in Lancashire has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water for three weeks.
Remove: Tap water in homes has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water.
Example 2 (XSum)
Summary: Great Britain's Becky James won her second Olympic silver of Rio 2016 by finishing second in the women's sprint.
Remove: Won her second Olympic silver of Rio 2016 by finishing second in the women's sprint.
Example 3 (WikiHow)
Summary: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic. Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye in Sharpies is usually weak against it.
Remove: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic. Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye is usually weak against it.
Example 4 (WikiHow)
Summary: Call 911 if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your daily routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys.
Remove: Call if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys.
Example 5 (Gigaword)
Summary: xinhua summary of asia-pacific stocks news on tuesday februar ##
Remove: summary of stocks news on tuesday ##
Example 6 (Gigaword)
Summary: cuba urges eu to drop its common position
Remove: cuba urges to drop its common position
Figure 7: Summaries changed using the corrector. We mark hallucinated entities in the summaries with red.
XSum Example
Article: The Scots started their Six Nations campaign with a thrilling first win over Ireland in four years. They beat France for the first time in 10 years last season at home, but have lost on their past nine trips to Paris. "It is a long time ago since we won there but we understand there is a reason for that," said Laidlaw. "France are a good team and it is a hard place to win. We are under no illusions this week - we need to be on top form to get something out of the game. "We'll need accuracy and we need to be physical and aggressive because the French are big men. But we need to be subtle and clever in our attack. "If we do that, and move France about and play for 80 minutes, I would back our fitness against them. But we obviously need to be in the game right to the death and not let France get any momentum." Media playback is not supported on this device France have not finished in the top half of the Six Nations since 2011, but impressed in narrow defeats by New Zealand and Australia in the autumn, and came close to a shock victory against England at Twickenham in their Six Nations opener. "They have improved massively under [coach] Guy Noves and have not been far away from any team they have played," Laidlaw noted. "They have some really dangerous strike runners throughout the team. "As much as we need to understand why we beat them last year, we need to develop that, be clever and smart and be able to score points." We need to keep doing what we are doing well, keep our skills high and if we do that again this weekend, we are going to put ourselves in a strong position. "The start of the game is massively important, whether it is our kick-off, or or us receiving. It is very important psychologically for them as it is for us. "If we can start the game playing in the right areas of the field for the first 20 minutes and not muck about too much in our own half, hopefully we will put them under pressure and upset the crowd a little bit." Media playback is not supported on this device
Reference: Captain Greg Laidlaw says Scotland must be "subtle and clever" as well as match France physically if they are to secure a first win in Paris since 1999.
Bart-base: Scotland head coach Greig Laidlaw says his side are "under no illusions" when they face France at Twickenham on Saturday.
PEGASUS*: Scotland coach Steve Laidlaw has warned France that they need to be "smart and smart" in their attack if they are to win their first Six Nations game of the season against the French in Paris on Saturday.
CLIFF: Scotland head coach Mark Laidlaw says his side need to be " subtle and clever" in their attack against France in Paris on Saturday.
DAE: Scotland must be " subtle and clever" in their attack when they face France at Twickenham on Saturday, says head coach Gordon Laidlaw.
FACTPEGASUS: Scotland captain Laidlaw says his side need to be "subtle and clever" in their attack if they are to beat France.
Gigaword Example
Article: the investigating magistrate and prosecutor handling the case involving the french association , zoe ’ s ark , arrived tuesday in the eastern town of abeche where they met the #### children the association was attempting to fly to france , according to reports .
Reference: chadian court officials go to UNK to investigation into children's case
Bart-base: zoe's ark investigating magistrate arrives in abeche
PEGASUS*: prosecutor in zoe's ark case arrives in abecha
CLIFF: prosecutor meets zoe's ark children
DAE: zoe's ark investigating magistrate prosecutor arrive in abecha
FACTPEGASUS: prosecutor arrives in abeche to investigate french association
Figure 8: Example summaries from XSum and Gigaword. Nonfactual components are marked with red.
WikiHow Example