SlowGuess/NAACL_2022_OCR huggingface dataset:
In addition to the BOOT-INPUTS CI method proposed by Deutsch et al. (2021b), the authors also proposed BOOT-SYSTEMS and BOOT-BOTH. Each of the three methods makes assumptions about whether the set of  $N$  systems and  $M$  input documents are fixed or variable during the bootstrapping calculation. For instance, BOOT-INPUTS assumes the  $N$  systems are always the same and the  $M$  input documents are random, then subsequently resamples  $M$  input documents on each bootstrapping iteration to calculate the confidence interval. BOOT-SYSTEMS does the opposite by resampling which  $N$  systems are used while holding the original  $M$  input documents fixed. BOOT-BOTH assumes both the systems and inputs are variable.
Figs. 7 and 8 contain the  $95\%$  CIs for ROUGE, BERTScore, and QAEval on the SummEval and REALSumm datasets using the BOOT-SYSTEMS and BOOT-BOTH methods calculated using all  $M_{\mathrm{test}}$  test instances and only the  $M_{\mathrm{jud}}$  annotated instances (BOOT-INPUTS included in the main body
Figure 7: The  $95\%$  CIs calculated using the BOOT-SYSTEMS bootstrapping method with  $M_{\mathrm{jud}}$  summaries in orange and  $M_{\mathrm{test}}$  in blue.
of the paper, Fig. 4). The widths of the BOOT-BOTH CIs decreased by  $14\%$  and  $12\%$ , whereas the BOOT-SYSTEMS CIs only decreased by  $1\%$  and  $6\%$ .
The BOOT-SYSTEMS widths likely decreased less because its estimation of  $r_{\mathrm{SYS}}$  is not dependent on the variance of the system score estimates. Since the set of  $M$  input documents is fixed, the system scores do not change at all during bootstrapping, so increasing the number of summaries used to estimate those scores should not have a major effect on the estimation of  $r_{\mathrm{SYS}}$ .
B Additional  $r_{\mathrm{SYS}} \Delta(\ell, u)$  Results
Fig. 9 contains the  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlations for when  $\ell = 0$  for ROUGE-1, ROUGE-2, and ROUGE-L, equivalent to those shown in Fig. 6 in the main body of the paper (ROUGE-1 is shown in both). The ROUGE-2 and ROUGE-L results are largely consistent with those of ROUGE-1. The metrics' correlations to human annotations are low (or even negative) when the differences between system scores are small. As more pairs are added that differ by larger margins, the correlations increase.
Figs. 10 and 11 contain the  $r_{\mathrm{SYS}}\Delta (\ell ,u)$  correlations for ROUGE, BERTScore, and QAEval for various combinations of  $\ell$  and  $u$  on both the SummEval and REALSumm datasets. The first rows of each heatmap are plotted in Figs. 6 and 9.
Figure 8: The  $95\%$  CIs calculated using the BOOT-BOTH bootstrapping method with  $M_{\mathrm{jud}}$  summaries in orange and  $M_{\mathrm{test}}$  in blue.
We see that as the allowed score gap between system pairs is allowed to increase (i.e., adding "easier" pairs to rank), the correlation increases by a large margin over the correlation on pairs close in score. All of the metrics have nearly perfect correlation when the system pairs are separated by large margins.
C Summarization Paper Survey
To estimate the difference in ROUGE-1 score that is commonly reported in papers, we performed a survey of recently published summarization papers. We selected papers from 2020 and 2021 that were published in a *ACL conference (including Findings), had "summary" or "summarization" in the title, proposed a new system, and compared systems on the CNN/DailyMail dataset with ROUGE. We selected the differences between the best two models that were compared with ROUGE on the test set. We did not include ablation experiments for which the differences are likely smaller than the differences between the top two performing systems. The results are shown in Table 1. The average reported difference was found to be 0.49 ROUGE-1.
Figure 9: The  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlations on the SummEval (top) and REALSumm (bottom) datasets for  $\ell = 0$  and various values of  $u$  for ROUGE-1, ROUGE-2, and ROUGE-L. The  $u$  values were chosen to select the  $10\%$ ,  $20\%$ ,  $\ldots$ ,  $100\%$  of the pairs of systems closest in score. Each  $u$  is displayed on the top of each plot.
Paper	Model 1	Score 1	Model 2	Score 2	Δ
Xu et al. (2020b)	SAGCopy Indegree-2	42.56	MASS+Copy	41.71	0.85
Xu et al. (2020a)	DISCOBERT w. GR & GC	43.77	BERTSum	43.25	0.52
Huang et al. (2020)	BART	44.16	ASGARD-DOC + RROUGE + RCloze	43.93	0.23
Li et al. (2020)	EDUSum	41.40	Fast-Abs	40.88	0.52
Zhong et al. (2020)	MATCHSUM (RoBERTa-Base)	44.41	BERTSum	43.85	0.56
Wang et al. (2020)	BERTSum+TA	43.06	BERTSum	42.13	0.93
Jia et al. (2020)	HAHSum-Large	44.68	MatchSum-Base	44.41	0.27
Zou et al. (2020)	STEP (GIGA-CM)	44.07	UniLM	43.47	0.60
Yang et al. (2020)	TED 10L8H	38.73	Pretrained 10L8H	38.38	0.35
Desai et al. (2020)	MatchSum + CUPS	44.69	MatchSum	44.41	0.28
Xiao et al. (2020)	IMP + BERT (MLM)	37.53	IMP + XLNet (PLM)	37.04	0.49
Jin and Wan (2020)	Ours	41.70	CopyTransformer	41.39	0.31
Jia et al. (2021)	DifferSum-Large	44.70	MatchSum-Base	44.41	0.29
Nan et al. (2021)	Q-C-O	44.70	MLE	44.24	0.46
Xing et al. (2021)	Shuffling	41.00	Our Method	40.88	0.12
Liu and Liu (2021)	SimCLS	46.67	GSum	45.94	0.73
Padmakumar and He (2021)	PacSum	40.26	Lead-k	39.69	0.57
Balach et al. (2021)	Explicit-Structure Attention	39.63	Pointer-Generator + Coverage	39.07	0.56
Huang and Kurohashi (2021)	DiscoBERT	43.77	Proposed	43.61	0.16
Liu et al. (2021a)	UniLMv2 + SKD + Noisy T + Noisy S	43.77	UniLMv2	43.45	0.43
Chen and Yang (2021)	S-BART w. Discourse & Action	46.07	Multi-View Seq2Seq	45.56	0.51
Liu et al. (2021b)	GSum-Fine-Tuned	46.18	Base	45.93	0.25
Dou et al. (2021)	BART + MatchSum	45.94	BART	44.66	1.28
Average	0.49
Table 1: A survey of recent summarization papers published in *ACL conferences and the differences in ROUGE-1 score they reported on the CNN/DailyMail dataset. Narayan et al. (2020) is not included because it is not clear which results were considered comparable to their model.
Figure 10:  $r_{\mathrm{SYS}} \Delta(\ell, u)$  correlations for various combinations of  $\ell$  and  $u$  (see §4.2) for ROUGE (top), BERTScore (middle), and QAEval (bottom) on SummEval (left) and REALSumm (right). The values of  $\ell$  and  $u$  were chosen so that each value in the heatmaps evaluates on  $10\%$  more system pairs than the value to its left. For instance, the first row evaluates on  $10\%, 20\%, \ldots, 100\%$  of the system pairs. The second row evaluates on  $10\%, 20\%, \ldots, 90\%$  of the system pairs, never including the  $10\%$  of pairs which are closest in score. The first row of each of the heatmaps is plotted in Fig. 6. The correlations on realistic score differences between systems are in the upper left portion of the heatmaps and contain the lowest correlations overall. Evaluating on all pairs is the top-rightmost entry, and the "easiest" pairs (those separated by a large score margin) are in the bottom right.
Figure 11: See Fig. 10 for a description of the heatmaps, shown here for ROUGE-2 (top) and ROUGE-L (bottom).