SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ablation Study In Equation 1, we propose to aggregate chunk-sentence scores by taking the maximum score over context chunks and then average over claim sentences (mean-max). Another reasonable choice is to take the minimum score over claim sentences instead of the average (min-max). We evaluate the two aggregation methods on factual consistency evaluation tasks and report the results in Table 20. Overall, the mean-max method leads to better correlation with human annotation. We speculate this is because taking the average helps remove some of the noise introduced by the alignment estimator, while the min-max setup would be easily affected by underestimation.
D.3 Question Answering with Unanswerable Question
Simplified Natural Questions For this experiment, we construct a new SQuAD-style QA dataset with unanswerable questions, Simplified Natural Questions (Simplified NQ), base on Natural Questions [75]. For each sample (an search query and a Wikipedia article) in Natural Questions, [75] ask five annotators to find 1) an HTML bounding box (the long answer; e.g., paragraphs, tables, list items, or whole list) containing enough information to infer the answer to the query (long answer), and 2) a short span within the long answer that answers the question (the short answer). For both short and long answers, the annotators can alternatively indicate that an answer could not be found.
For the purpose of constructing Simplified NQ, we consider a sample to be answerable if at least 2 annotators identified both long and short answers. In this case, we use the most popular (among the annotators) short answer as the ground truth answer, and the most popular long answer containing the selected short answer as the context. Conversely, if less than 2 annotators identified any long answer, and less than 2 annotators identified any short answer, we consider the sample to be unanswerable and use a random paragraph from the article as the context. We discard all remaining samples to
Table 10: The performance of multi-task RoBERTa-base and ALIGN-base on the in-domain datasets. The last row is the averaged performance of each model on these datasets.
Task	Dataset	Multitask-base	ALIGN-base
NLI	MNLI-mm	87.61	87.54
MNLI-m	87.68	87.82
ANLI-1	65.10	65.30
ANLI-2	48.20	48.70
ANLI-3	46.17	45.50
SNLI	91.33	90.78
Fact Verification	NLI-FEVER	76.50	76.78
VitaminC	89.82	89.79
STS	SICK	89.01	90.71
STSB	87.86	89.03
Paraphrase	PAWS	93.94	92.33
PAWS-QQP	92.32	91.88
QQP	90.66	90.07
QA	RACE-m	78.34	76.95
RACE-h	71.56	68.84
Multi-RC	83.83	82.20
BoolQ	81.32	81.07
SciQ	92.10	92.40
Coreference	GAP	81.65	81.35
Average	80.79	80.48
Table 11: The performance of multi-task RoBERTa-base and ALIGN-base on the zero-shot datasets. Multi-task RoBERTa uses task-specific prediction heads for each of its training sets. To test it on zero-shot datasets, we evaluate a set of "reasonable" heads obtained during training (e.g., we use heads trained on NLI datasets for NLI evaluation tasks) and report the average (avg. of heads), best (best head), and worst (worst head) performance of these heads. "Best head" assumes oracle access to evaluation results, and is thus effectively an unrealistic upper bound. The last row is the averaged performance of each configuration on these datasets.
Task	Dataset	Multitask-base (avg. of heads)	Multitask-base (best head)	Multitask-base (worst head)	ALIGN-base
NLI	AXB	74.94	76.18	72.37	75.09
AXG	63.58	65.73	60.39	59.83
CB	80.00	83.93	76.79	76.79
RTE	81.41	81.95	81.23	83.39
WNLI	56.34	60.56	47.89	52.11
SE14T1	57.86	58.45	57.17	90.72
Paraphrase	MRPC	69.60	71.19	68.23	65.97
QA	DREAM	67.04	73.98	57.03	71.34
Quartz	59.09	63.52	54.46	59.69
Average	67.76	70.61	63.95	70.55
Table 12: The performance of ALIGN-base and the instruction finetuned T5-base (IF-T5-base) on the in-domain tasks. The last column shows the average performance on the in-domain tasks.
Model (Size)	NLI	Fact Verification	STS	Paraphrase	QA	Coreference	Average
ALIGN-base (125M)	70.9	83.3	89.9	91.4	78.2	81.4	82.5
IF-T5-base (222M)	53.7	68.7	62.0	83.2	30.3	42.9	56.8
Table 13: The performance of ALIGN-base and the instruction finetuned T5-base (IF-T5-base) on the zero-shot tasks. The last row shows the average performance on the in-domain tasks.
Dataset	Alignment-base	IF-T5-base
AXB	75.1	65.3
AXG	59.8	50.8
CB	76.8	58.9
RTE	83.4	59.9
WNLI	52.1	47.9
SE14T1	90.7	48.5
MRPC	66.0	66.1
DREAM	71.3	35.8
Quartz	59.7	50.6
AVG	70.5	53.8
Table 14: The performance of RoBERTa-NLI and FLAN-Alpaca on the zero-shot datasets (of alignment model). The gray number shows the specific dataset is appeared in the training set of FLAN-T5. We report the average performance of each model in the last row.
RoBERTa-NLI	FLAN-Alpaca
base	large	base	large	xlarge
Model Parameters	125M	355M	220M	770M	3B
NLI	AXB	75.2	79.2	53.6	72.3	77.2
AXG	59.6	73.6	49.4	72.5	88.8
CB	85.7	87.5	78.6	78.6	87.5
RTE	81.2	88.1	72.9	79.8	87.0
WNLI	52.1	50.7	40.8	62.0	71.8
SE14T1	91.2	93.1	65.0	72.4	77.3
Paraphrase	MRPC	38.7	42.3	66.7	75.4	83.1
QA	DREAM	63.9	73.0	63.5	76.8	89.5
Quartz	54.6	65.6	68.1	87.4	90.2
Average	66.9	72.6	62.1	75.2	83.6
Table 15: The ROC AUC of each metric on the SummaC benchmark. CGS and XSF are abbreviations of CogenSumm and XSumFaith, respectively. The strongest performance on each dataset is shown in bold. The last column shows the average performance on each dataset in the SummaC benchmark.
SummaC Benchmark
CGS	XSF	PolyTope	FactCC	SummEval	FRANK	AVG
BERTScore	63.1	49.0	85.3	70.9	79.6	84.9	72.1
BLEURT	60.8	64.7	76.7	59.7	71.1	82.5	69.2
BARTScore	74.3	62.6	91.7	82.3	85.9	88.5	80.9
CTC	76.5	65.9	89.5	82.6	85.6	87.3	81.2
UniEval	84.7	65.5	93.4	89.9	86.3	88.0	84.6
QAFactEval	83.4	66.1	86.4	89.2	88.1	89.4	83.8
ALIGN-base (Ours)	80.6	76.1	87.5	93.1	88.6	89.5	85.9
ALIGN-large (Ours)	88.4	74.6	92.5	94.9	92.3	91.3	89.0
Table 16: The ROC AUC of each metric on the TRUE benchmark. The datasets with asterisks(*) appear in the training set of the alignment model. We compute both the overall average on all datasets (Average) and average without PAWS, FEVER, VitaminC datasets (Average-ZS). The latter shows the zero-shot performance of ALIGN. Bold indicates the best performance on a dataset.
BERTScore	BLEURT	BARTScore	CTC	UniEval	QAFactEval	ALIGN -base (Ours)	ALIGN -large (Ours)
TRUE Benchmark	FRANK	84.0	81.6	87.8	87.1	88.1	88.5	90.4	91.4
SummEval	72.3	68.0	78.9	79.8	81.2	80.9	79.5	83.8
MNBM	52.5	65.5	63.5	65.0	66.8	67.3	76.4	74.4
QAGS-C	70.6	71.2	83.9	77.3	86.5	83.9	80.3	89.0
QAGS-X	44.3	56.2	60.2	67.7	76.7	76.1	79.6	83.2
BEGIN	86.4	86.6	86.7	72.0	73.6	81.0	81.3	81.1
Q2	70.2	72.9	65.1	66.8	70.4	75.8	77.2	79.2
DialFact	68.6	73.0	60.8	63.7	80.4	81.8	83.3	85.1
PAWS*	78.6	68.4	77.1	63.1	80.1	86.1	97.6	98.4
FEVER*	54.2	59.5	66.1	72.5	92.1	86.0	94.7	94.9
VitaminC*	58.2	61.8	64.2	65.0	79.1	73.6	97.8	98.3
Average	67.2	69.5	72.2	70.9	79.5	80.1	85.3	87.2
Average-ZS	68.6	71.9	73.4	72.4	78.0	79.4	81.0	83.4
Table 17: The Pearson correlation coefficients of various metrics on other datasets mentioned in Section 4.2.1. Q-XSum and Q-CNNDM are abbreviations of QAGS-XSum and QAGS-CNNDM, respectively. F-XSum and F-CNNDM are abbreviations of FRANK-XSum and FRANK-CNNDM, respectively. The last column shows the average performance on each dataset. The best performance is shown in bold.
Other Datasets - Pearson
XSumFaith	SummEval	Q-Xsum	Q-CNNDM	F-Xsum	F-CNNDM	SamSum	AVG
BERTScore	13.0	33.1	-10.6	51.7	13.0	51.7	10.9	23.3
BLEURT	38.7	23.8	13.2	45.2	15.6	37.5	8.1	26.0
BARTScore	29.3	35.5	16.3	71.5	23.7	51.9	15.0	34.7
CTC	27.2	54.7	30.6	64.5	20.0	54.5	16.9	38.3
UniEval	23.9	57.8	45.5	66.7	27.2	58.3	23.2	43.2
QAFactEval	30.3	61.6	44.2	68.4	32.1	64.6	38.9	48.6
ALIGN-base (Ours)	33.2	57.8	51.1	60.9	31.2	61.8	21.1	45.3
ALIGN-large (Ours)	28.8	66.7	53.9	76.1	38.9	68.9	47.7	54.4
Table 18: The Pearson correlation coefficients of various metrics on other datasets mentioned in Section 4.2.1. The format in this table follows Table 17.
Other Datasets - Spearman
XSumFaith	SummEval	Q-Xsum	Q-CNNDM	F-Xsum	F-CNNDM	SamSum	AVG
BERTScore	13.4	31.5	-8.9	46.2	12.7	45.1	13.1	21.9
BLEURT	37.0	23.6	12.4	43.4	13.9	37.6	6.7	24.9
BARTScore	29.8	39.1	17.0	68.1	20.0	53.3	16.3	34.8
CTC	29.8	41.7	30.6	57.3	20.4	49.4	17.7	35.3
UniEval	25.3	44.3	50.0	67.6	26.7	54.0	22.8	41.5
QAFactEval	31.9	42.8	44.1	63.1	25.5	53.7	35.9	42.4
ALIGN-base (Ours)	38.8	42.0	52.7	56.1	25.5	56.4	22.3	42.0
ALIGN-large (Ours)	32.1	47.9	57.4	71.6	30.0	61.8	46.7	49.7
Table 19: The Kendall's tau correlation coefficients of various metrics on other datasets mentioned in Section 4.2.1. The format in this table follows Table 17.
Other Datasets - Kendall's tau
XSumFaith	SummEval	Q-Xsum	Q-CNNDM	F-Xsum	F-CNNDM	SamSum	AVG
BERTScore	9.2	24.9	-7.3	36.3	10.4	34.7	10.7	17.0
BLEURT	25.3	18.6	10.1	33.9	11.4	28.8	5.5	19.1
BARTScore	20.2	31.0	13.9	55.6	16.3	41.4	13.3	27.4
CTC	20.2	33.2	25.1	45.7	16.6	38.2	14.4	27.6
UniEval	17.0	35.3	40.9	54.4	21.8	42.4	18.7	32.9
QAFactEval	23.2	34.0	36.2	50.5	22.4	42.2	30.1	34.1
ALIGN-base (Ours)	26.6	33.4	43.1	45.5	20.8	44.4	18.2	33.1
ALIGN-large (Ours)	21.9	38.4	47.0	59.6	24.5	49.5	38.2	39.9
Table 20: The comparison between the mean aggregation method and the min aggregation method. The bold number represents a better performance when comparing the aggregation method.
ALIGN-base mean-max	ALIGN-base min-max	ALIGN-large mean-max	ALIGN-large min-max
SummaC	85.9	86.1	89.0	89.1
TRUE	85.3	84.2	87.2	86.2
Other-Pearson	45.3	43.7	54.4	52.1
Other-Spearman	42.0	42.3	49.7	49.2
Other-Kendall	33.1	33.1	39.9	38.8
avoid ambiguity (e.g., some samples might only have long answers but not short answers). This results in a total of 3336 answerable samples and 3222 unanswerable samples in the validation set.
Prompts and QA Inference For FLAN T5, we follow [48] and use the following prompt: Context: {context}\nQuestion: {question}\nAnswer:
For GPT-3.5, we use a prompt with additional instructions:
Find the answer to the question from the given context. When the question cannot be answered with the given context, say "unanswerable". Just say the answer without repeating the question.\nContext: {context}\nQuestion:{question}\nAnswer:
At inference time, we truncate the context if necessary such that the entire input is at most around 2000 tokens long (2000 for FLAN T5, 2040 for GPT-3.5 to account for the longer prompt). We use greedy decoding for FLAN T5, a the default chat completion settings for GPT-3.5. When FLAN T5 outputs "unanswerable", we interpret it as predicting the sample to be not answerable. Similarly, if GPT-3.5's output contains any of "unanswerable", "no answer", "context does not provide an answer", we consider the prediction to be unanswerable.
Additional Results In addition to FLAN T5 and GPT-3.5, we also experiment with Electra [88], one of the top performing single models on the SQuAD v2 leaderboard, for reference. Specifically, we reproduce Clark et al. [88]'s design that use a QA prediction head to jointly predict the answer span and unanswerable probability. As shown in Table 21, while Electra is a strong performer on SQuAD v2 and Simplified NQ, adding the alignment verifier to GPT-3.5 and FLAN T5 greatly reduces the performance gap. Additionally, on ACE-whQA, our design (both FLAN T5 and GPT-3.5 with alignment verifiers) outperforms Electra.
We present the additional ablation result on factual consistency evaluation tasks in Table 22. This part follows Section 4.4, where we use the same checkpoints that are trained on incrementally added tasks. Result shows the training tasks are generally compatible and effective, though we notice adding fact verification and paraphrase detection tasks lead to a slightly performance drop. We speculate it is due to the paraphrase detection task, where a text pair is expected to have exactly the same information.
Table 21: Additional experiment results on QA with unanswerable questions including Electra. The best model for each task/metric is shown in **bold**.
SQuAD v2	ACE-whQA	Simplified NQ
EM	F1	AUC	EM	F1	AUC	EM	F1	AUC
Electra	86.47	89.37	0.97	52.32	55.59	0.87	70.81	74.13	0.88
GPT-3.5	52.53	63.96	0.76	67.98	71.98	0.77	58.37	68.61	0.81
Flan T5	75.72	79.01	0.83	26.29	29.24	0.51	38.24	44.98	0.58
GPT-3.5 + Verifier (Ours)	67.19	77.63	0.93	79.02	80.91	0.84	56.16	57.40	0.86
FLAN T5 + Verifier (Ours)	83.72	86.55	0.95	75.75	77.60	0.90	64.93	67.99	0.83
The ALIGN-base model, which uses all the possible training data, gets the best performance on every factual consistency evaluation task.
Table 22: Ablation results on factual consistency evaluation tasks. Each row corresponds with a model trained with data adapted from incrementally more types of tasks. For example, the model on the second row is trained with NLI, Fact Verification and Paraphrase tasks. The model on the last row is the same as Alignment-base. We report the average performance for each evaluation tasks. The last column shows the overall average for the factual consistency evaluation tasks. The best performance for each task is shown in bold.
Training Tasks	Factual Consistency Evaluation Tasks
SummaC	TRUE	Other-Pearson	Other-Spearman	Other-Kendall	Average
+NLI	78.1	77.5	32.6	33.6	26.3	49.6
+FV, Para	74.9	80.3	27.6	27.2	21.1	46.2
+Coref, Sum, IR, STS	84.2	83.7	39.4	36.8	28.8	54.6
+QA (Alignment-base)	85.9	85.3	45.3	42.0	33.1	58.3
D.5 Evaluation Data Contamination Analysis
As we train and evaluate our models by adapting existing datasets, a subset of our evaluation data could be contaminated with training data. To understand the impact of data contamination, we perform a post hoc overlap analysis. Following [2], for each evaluation example, we check if any of its n-grams exist in the training set, where  $n$  is the 5th percentile example length for the evaluation dataset in words (in practice we clamp  $n$  to between 8 and 13 to avoid spurious collisions and limit run time). We consider an evaluation example "dirty" if there is at least one overlapping  $n$ -gram or "clean" otherwise. Due to resource constraints (e.g., available RAM), we sample up to 1000 examples from each NLU (Section 4.1) and factual consistency evaluation (Section 4.2) dataset for analysis.
The results are shown in Table 23. Overall, the SummaC benchmark and the other datasets we use in the factual consistency evaluation experiment have the least number of dirty examples, with other datasets having varying level of overlap with training data. One notable case is ANLI, where almost all examples are marked as dirty. We manually examine a small selection of ANLI examples, and find that most of the overlapping n-grams come from the premise portion of the examples (they overlap with examples from DocNLI and HotpotQA in the training set). As noted by [2], this n-gram overlap metric tends to show a large number of false positives for datasets constructed from common data sources. For instance, one possible explanation for a high dirty count is that training and evaluation examples share texts sourced from the Internet (e.g., Wikipedia) as supporting information (e.g., HotpotQA contexts and ANLI premises), which does not necessarily leak evaluation set answers. Thus, a high number of dirty examples alone does not meaningfully indicate contamination.
A more reliable indicator of data contamination is the difference in metric scores when evaluating using the clean subset compared to the full datasets. If removing the dirty samples (i.e., using the clean subset) leads to significantly worse metric scores, then the model might have overfit to the overlapping training data. As show in Table 23, NLI-FEVER and RACE-h have the biggest drop in metric scores. However, overall, removing dirty examples does not meaningfully reduce metric scores (the average difference is -0.1). Thus, the data contamination does not have a significant impact on our evaluation results and conclusions.
Table 23: Data contamination analysis results for evaluation datasets. For each evaluation dataset, we report the percentage of examples containing n-grams that also exist in training set ("dirty %"); the performance of ALIGN-large on the entire dataset ("full eval set"; see Section 4 for details), the clean subset, and the dirty subset; and lastly, the difference between the clean subset and the entire dataset ("Clean vs. Full"). We exclude subsets with less than 100 examples to reduce variance.
Experiment	Dataset	Dirty %	Accuracy / Pearson Correlation / AUC
Full Eval Set	Clean Subset	Dirty Subset	Clean vs. Full
Language Understanding In-Domain	MNLI-mm	2.8	90.3	90.0	—	-0.3
MNLI-m	5.4	90.3	88.6	—	-1.8
ANLI-1	99.1	75.8	—	75.9	—
ANLI-2	99.4	52.4	—	52.3	—
ANLI-3	99.9	52.3	—	51.7	—
SNLI	0.5	91.8	91.4	—	-0.4
NLI-FEVER	68.9	77.8	73.6	79.1	-4.1
VitaminC	18.5	91.8	90.4	94.6	-1.4
SICK	52.4	91.5	93.2	90.0	1.7
STSB	12.2	89.8	88.8	91.3	-1.0
PAWS	16.5	92.6	91.7	92.7	-0.9
PAWS-QQP	78.4	93.8	97.3	92.8	3.5
QQP	34.1	91.3	93.2	91.8	1.9
RACE-m	34.0	86.8	88.0	82.6	1.3
RACE-h	28.8	81.6	78.7	83.7	-3.0
Multi-RC	44.0	87.8	87.0	85.7	-0.9
BoolQ	49.5	87.7	85.9	90.9	-1.8
Quail	0.4	78.6	78.5	—	-0.1
SciQ	20.1	93.7	92.4	99.0	-1.3
GAP	1.6	88.6	87.5	—	-1.1
Language Understanding Zero-Shot	AXB	1.2	79.6	79.8	—	0.1
AXG	0.0	72.5	72.5	—	0.0
CB	12.5	89.3	87.8	—	-1.5
RTE	6.5	89.9	89.2	—	-0.7
WNLI	0.0	54.9	54.9	—	0.0
SE14T1	50.6	86.9	90.5	84.8	3.5
MRPC	32.0	67.9	68.7	66.9	0.8
DREAM	0.0	81.1	81.4	—	0.3
Quartz	24.5	79.2	77.7	83.9	-1.5
Factual Consistency Evaluation SummaC	CogenSumm	2.0	88.4	88.6	—	0.2
XSumFaith	0.7	74.6	75.2	—	0.5
PolyTope	1.3	92.5	91.9	—	-0.6
FactCC	9.3	94.9	95.3	—	0.5
SummEval	4.0	92.3	92.6	—	0.3
FRANK	2.0	91.3	90.2	—	-1.0
Factual Consistency Evaluation TRUE	FRANK	2.8	91.4	91.4	—	0.0
SummEval	4.6	83.8	84.4	—	0.6
MNBM	0.5	74.4	76.5	—	2.1
QAGS-C	0.4	89.0	89.0	—	0.0
QAGS-X	0.4	83.2	83.1	—	-0.1
BEGIN	27.9	81.1	81.0	82.3	-0.1
Q2	41.6	79.2	81.3	77.8	2.1
DialFact	28.9	85.1	85.2	89.0	0.1
PAWS*	19.4	98.4	98.8	97.1	0.4
FEVER*	54.4	94.9	96.2	94.9	1.3
VitaminC*	17.5	98.3	98.3	99.6	-0.1
Factual Consistency Evaluation Other	XSumFaith	0.8	28.8	30.1	—	1.3
SummEval	3.3	66.7	64.9	—	-1.8
Q-Xsum	0.4	53.9	53.6	—	-0.2
Q-CNNDM	0.4	76.1	76.1	—	0.0
F-Xsum	2.6	38.9	45.7	—	6.8
F-CNNDM	2.6	68.9	67.9	—	-1.0
SamSum	0.0	47.7	47.7	—	0.0
Average	-0.1