SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 6 shows the balanced accuracy values reported by the model at different points during its training, on both the synthetic and human-annotated test sets. The graph clearly shows that performance on the human annotated dataset (CNN/DM) has high variance, compared to the held-out dataset accuracies which has a steadily increasing performance. This behavior was observed for
Ent-C	Gen-C
CNN/DM
Sent-Factuality	96.4	91.2
DAE	95.4	97.3
DAE-Weak	94.8	97.8
XSUM
Sent-Factuality	96.1	97.9
DAE	94.3	97.1
DAE-Weak	95.3	95.9
Table 7: Performance of factuality models when trained on synthetic factuality training datasets on their held-out test sets.
ENTITY-CENTRIC
Figure 6: Performance of different train checkpoints on a held-out dataset and on the human annotated dev set for models trained on the synthetic data in the CNN/DM domain.
GENERATION-CENTRIC
both ENT-C and GEN-C domains; however, ENT-C exhibited more variance. This indicates that the synthetic datasets are targeting a different error distribution, and optimizing for the synthetic distribution does not necessarily improve results on the actual generation errors.
C Transferability of human annotations across generation models within the same domain
In Section 5.2, we demonstrate that for highly abstractive domains like XSUM, we require human annotated data to train factuality models. However, even within the same summarization domain (say XSUM), it is prohibitively expensive to collect human annotations for each summarization model that we may wish to evaluate. Here, we investigate whether the factuality annotations collected for one summarization model be used to identify factuality errors in summaries generated by other models. These experiments are done within the same domain (XSUM)
We create new training and test sets from the XSUM-HUMAN dataset. We create 2 types of train
ing datasets for each of the 5 models annotated in that dataset: (1) All-models train set: This contains (A,S) pairs from all models, including the models being evaluated (2000 pairs from other models, 200 pairs from same model) and (2) Other-models train set: This contains (A, S) pairs from the rest of the models (2000 pairs). Evaluation is done on the remaining 300 (A,S) pairs for each summarization model. We train the best performing factuality model, i.e. the DAE model for all these settings.
BERTS2S	PTGEN	TCONVS2S	TRANS2S
All	79.6	75.8	76.7	84.5
Others	82.3	77.0	74.1	85.3
Results are outlined in table 8. These show that the performance is similar for both All-models and Other-models settings for all models considered. This indicates that for the given set of summarization models considered (all trained on the same summarization training dataset), human annotations from one generation model can be used to evaluate factuality for other models.
We use the Huggingface Library (Wolf et al., 2019) for all our experiments. All our factuality models are trained by fine-tuning the pre-trained ELEcTRA (electra-base-discriminator, 110M parameters) model. We perform 5 hyper parameter trials to select the best set of hyper parameters, varying the learning rate. The final hyper-parameters are:
Table 8: Performance of models trained on All-models dataset vs Other-models dataset.
Computing Infrastructure	32GB NVIDIA V100 GPU
Max Seq Length	512
Optimizer	Adam
Optimizer Params	β = (0.9, 0.999), ε = 10-8
Learning Rate Decay	Linear
Learning rate	2e-5
Weight Decay	0
Warmup Steps	0
Max Gradient Norm	1
Batch size	8
Epochs	3
For models with high variance (sent-factuality model from section 5.2), we report average of 3 runs by initializing with a random seed.
The hyperparameters for training the BART summarization models are given in Table 10. Parameter settings used during decoding to generate summaries on the test set are also included
Table 9: Hyperparameters used for fine-tuning the factuality models.
For training
Computing Infrastructure	32GB NVIDIA V100 GPU
Max Input Seq Length	512
Max Output Seq Length	128
Optimizer	Adam
Optimizer Params	β = (0.9, 0.999), ε = 10-8
Learning Rate Decay	Linear
Learning rate	2e-5
Weight Decay	0
Warmup Steps	0
Max Gradient Norm	1
Batch size	8
Epochs	10
For decoding
Num beams	6
Length Penalty	2
No repetition size	3-grams
Min-Length	10
Max Length	60
Table 10: Hyperparameters used for fine-tuning and decoding using the BART-based summarization models.
E Human Study
Figure 7 provides an screenshot of the Amazon Mechanical Turk tasks used to obtain human judgements for factuality of generated summaries, as outlined in Section 6.2. Workers were presented with a source article and 3 corresponding summaries. Each of these summaries were marked as Factual or Non-Factual. Additionally, they were asked to highlight the span within the summary that was erroneous.
Given below is a news article on the left hand side. On the right side are 3 claims related to the article. Your task is to determine whether EACH claim is Correct (supported by the news article) or Incorrect (unsupported by the news article). Please read the news article carefully. A claim may be incorrect because it mis-states information in the article (e.g. say person X did something instead of person Y) or introduces new information not in the article.
- If the claim is Correct, copy-paste the phrase 'The claim in correct' in the corresponding text box.
- If the claim is Incorrect, copy-paste the part of the claim that is unsupported by the article in the corresponding text box. Most of the claims will be at least mildly incorrect. If a claim is completely incorrect, you can copy-paste the whole claim.
supporters and colleagues gathered outside the alvorada palace to bid her farewell, some handing her flowers. ms rousseff was dismissed last week after the senate found her guilty of manipulating the budget. she denies wrongdoing and has dismissed her impeachment as a "coup d'etat". brazilian television showed ms rousseff walking out of the presidential residence surrounded by former ministers and congressmen from her workers' party. how will history look back on impeachment? profile: dilma rousseff supporter cecilia monteir, 56, said she was "very, very sad, feeling like the country will be left a bit orphaned". more supporters awaited ms rousseff as she arrived at an airport to board a plane to the southern city of porto alegre, her adopted hometown. on her arrival she was greeted by more well-wishers. hours after the impeachment vote, ms rousseff's vice-president michel temer, was sworn in, ending 13 years in power for the left-wing workers' party. he will serve out ms rousseff's term until 1 january 2019. ms rousseff has filed an appeal at the supreme court against the senate's decision but correspondents say it has very little chance of succeeding.
brazilian president dilma rousseff has left the presidential palace in Rio de janeiro after her impeachment by the lower house of parliament.
$\bigcirc$  Correct
Copy paste the incorrect part from the above claim
Claim 2
brazilian president dilma rousseff has left the presidential palace in the capital, brussels, a day after she was impeached by the senate.
brazilian president dilma rousseff has left her home after being impeached by the country's senate, hours after being sworn in for a second term.
Figure 7: Screenshot of the Mechanical Turk experiment. Given an input articles, the annotators were tasked with evaluating the factuality of 3 model generated summaries on a binary scale.