SlowGuess/NAACL_2022_OCR huggingface dataset:
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dušek, Chris Chinenye Emezue, Varun Ggangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics.
Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021. Dialfact: A benchmark for fact-checking in dialogue. arXiv preprint arXiv:2110.08222.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Martin Heidegger. 2001. On the essence of truth. The Nature of Truth: Classic and Contemporary Perspectives, 1:295-316.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^2$ : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019. Largescale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning
(CoNLL), pages 44-54, Hong Kong, China. Association for Computational Linguistics.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2021. Summac: Re-visiting nlibased models for inconsistency detection in summarization. arXiv preprint arXiv:2111.09525.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Katherine Lee, Orhan First, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. NeurIPS 2018 Workshop on Interpretability and Robustness for Audio, Speech, and Language.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. 2017. The RepEval 2017 shared task: Multi-genre natural language inference
with sentence representations. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 1-10, Copenhagen, Denmark. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI).
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, especially dolphins: Addressing contradictions in dialogue modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1699-1713, Online. Association for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829, Online. Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1):71-106.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2021. Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. Association for Computational Linguistics.
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 751-762, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Libo Qin, Tianbao Xie, Shijue Huang, Qiguang Chen, Xiao Xu, and Wanxiang Che. 2021. Don't be contradicted with anything! CI-ToD: Towards benchmarking consistency for task-oriented dialogue system. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2357-2367, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021a. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 704-718, Online. Association for Computational Linguistics.
Ehud Reiter and Craig Thomson. 2020. Shared task on evaluating accuracy. In Proceedings of the 13th International Conference on Natural Language Generation, pages 227-231, Dublin, Ireland. Association for Computational Linguistics.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Asso
ciation for Computational Linguistics: Human Language Technologies, pages 624-643, Online. Association for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thomas Scialom and Felix Hill. 2021. Beametrics: A benchmark for language generation evaluation evaluation. arXiv preprint arXiv:2110.09147.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921-927, Online. Association for Computational Linguistics.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. **Scrolls: Standardized comparison over long language sequences.**
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERIFICATION (FEVER) shared task. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language
inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 100-110, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15-33, Online. Association for Computational Linguistics.
Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document-level natural language inference. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 4913-4922, Online. Association for Computational Linguistics.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive summarization. arXiv preprint arXiv:2009.13312.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393-1404, Online. Association for Computational Linguistics.
A Additional Data Statistics
Tables 6 and 7 presents statistics regarding the length of the grounding text and the generated text for TRUE's datasets, respectively.
Dataset	Min len.	Max len.	Median len.	Avg len.
FRANK	102	1005	550	548
SummEval	100	540	367	359
MNBM	8	10315	287	383
QAGS-CNNDM	73	360	325	318
QAGS-XSUM	218	520	339	351
BEGIN	7	64	23	23
\( Q^2 \)	6	71	21	23
DialFact	4	174	22	26
PAWS	5	37	21.0	21
FEVER	8	286	44	59
VitaminC	1	265	26	28
Table 6: Grounding length statistics for TRUE.
Dataset	Min len.	Max len.	Median len.	Avg len.
FRANK	2	126	40	41
SummEval	5	133	61	63
MNBM	2	52	19	19
QAGS-CNNDM	23	85	47	49
QAGS-XSUM	9	31	18	18
BEGIN	5	40	13	14
\( Q^2 \)	7	44	15	16
DialFact	4	69	16	17
PAWS	5	37	21	21
FEVER	2	36	8	8
VitaminC	1	103	12	13
Table 7: Generated text length statistics for TRUE.
We train all models using the t5x library.[21]
QG-QA For our reimplementation of  $Q^2$  (Honovich et al., 2021) we use T5-11B as the pretrained model for QG, QA and NLI, while Honovich et al. (2021) used T5-Base, ALBERT (Lan et al., 2019), and RoBERTa (Liu et al., 2019) for the QG, QA and NLI models, respectively. We use a maximum length of 2048 tokens for the input. We set the F1 token overlap threshold to 0.54 by tuning it on a held-out dataset. We use beam search with a beam size of 4 to generate multiple questions, and use the first question that passes the validation threshold.
NLI We fine-tune a T5-11B model on ANLI (Nie et al., 2020) for  $25K$  steps with a learning rate of  $10^{-4}$  and a batch size of 32. During inference we use a maximum input length of 2048 tokens.
C ROC Curves
Figure 2 presents the ROC curves for the different datasets studied in TRUE, using the best-performing metrics.
$^{21}$ https://github.com/google-research/t5x
Dataset	Pos ROUGE_L	Neg ROUGE_L	ROUGE_L diff	Pos F1	Neg F1	F1 diff
FRANK	0.105	0.060	0.045	0.165	0.103	0.062
SummEval	0.181	0.141	0.041	0.282	0.244	0.038
MNBM	0.044	0.047	0.003	0.079	0.084	0.006
QAGS-CNNDM	0.215	0.170	0.045	0.281	0.249	0.031
QAGS-XSUM	0.051	0.050	0.002	0.082	0.080	0.002
BEGIN	0.465	0.159	0.306	0.553	0.207	0.346
\( Q^2 \)	0.228	0.169	0.059	0.368	0.264	0.104
DialFact	0.302	0.200	0.102	0.394	0.249	0.144
PAWS	0.832	0.734	0.098	0.938	0.934	0.003
FEVER	0.174	0.179	0.005	0.276	0.258	0.018
VitaminC	0.314	0.270	0.044	0.362	0.290	0.072
Table 8: Average overlap between the generated text and the grounding, measured using ROUGE-L and simple F1 token-overlap, taking the grounding to be the reference text. The “Pos” columns contain the statistics for the grounded text, while the “Neg” columns contain the statistics for the ungrounded text.
Figure 2: ROC curves for the best performing methods.
ANLI+Q2	ANLI+SCZS	Q2+SCZS	CTC	SCConv	ROUGE-L	BLEU4
FRANK	89.6	91.1	90.4	87.5	88.9	80.1	78.0
SummEval	80.7	83.0	82.0	76.0	79.8	68.8	60.2
MNBM	75.6	77.1	74.6	72.3	67.2	47.5	49.3
QAGS-C	86.0	84.7	86.4	73.4	79.6	67.1	63.9
QAGS-X	81.8	85.1	79.3	73.1	76.1	52.9	48.6
BEGIN	85.7	82.1	85.7	77.9	81.6	86.4	84.6
Q2	83.0	76.9	83.9	85.3	77.5	66.8	64.3
DialFact	89.4	84.5	90.2	83.5	81.2	71.2	72.5
PAWS	90.5	89.7	91.4	86.0	88.2	82.2	77.3
FEVER	94.0	94.6	93.9	84.8	86.7	49.9	51.1
VitaminC	90.3	96.4	96.5	84.9	97.5	59.9	59.6
Avg, w/o VitC, FEVER	84.7	83.8	84.9	79.4	80.0	69.2	66.5
Table 9: ROC AUC results for metrics that were not reported in Table 3.
Ensemble	Q2	ANLI	SCZS	BLEURT	QuestEval	FactCC	BARTscore	BERTscore
FRANK	90.8	87.8	89.2	88.6	83.2	86.4	73.9	88.3	86.0
BEGIN	85.9	78.0	82.8	84.2	82.2	81.4	65.0	83.7	86.0
DialFact	88.6	85.0	75.9	82.1	72.2	76.3	55.1	65.5	64.3
PAWS	92.4	90.1	87.3	89.7	67.1	70.1	65.1	77.3	76.4
VitaminC	96.7	83.4	89.6	98.4	63.0	67.8	56.8	64.1	63.5
Avg. w/o VitC	89.4	85.2	83.8	86.2	76.2	78.5	64.8	78.7	78.2
Table 10: ROC AUC results for the different metrics on the TRUE test set. We exclude VitaminC from the average calculation as  $\mathrm{SC}_{\mathrm{ZS}}$  was trained on VitaminC. The highest score in each row (excluding the Ensemble) is in bold and the aforementioned SC results are in strikethrough.
Ensemble	Q2metric	ANLI	SCZS	F1	BLEURT	QuestEval	FactCC	BARTscore	BERTscore
FRANK	83.2	82.9	83.5	80.9	67.5	76.0	74.2	73.9	80.2	77.0
SummEval	79.4	77.3	72.9	75.0	64.2	66.4	67.0	65.3	68.9	73.7
MNBM	69.6	66.5	66.7	64.5	40.4	53.7	66.1	62.9	61.0	60.1
QAGS-C	81.3	78.3	75.3	72.8	62.6	68.9	62.6	70.6	74.5	66.8
QAGS-X	77.4	63.6	80.3	72.8	51.0	56.5	56.9	65.3	54.4	52.3
BEGIN	77.9	73.0	76.0	79.7	80.0	78.1	75.5	66.6	77.4	80.5
Q2dataset	76.8	76.1	66.6	71.3	61.4	67.0	65.8	60.5	59.7	66.1
DialFact	83.4	80.5	70.9	75.8	66.4	67.0	69.9	53.4	60.8	61.2
PAWS	83.4	83.5	80.5	81.5	50.2	64.2	64.3	60.2	72.2	71.1
FEVER	90.4	82.7	90.2	87.6	52.3	57.7	69.4	57.2	60.5	58.7
VitaminC	90.8	76.0	82.3	93.5	58.0	58.0	62.2	55.0	59.1	58.6
Avg- w/o VitC,FEVER	79.2	75.7	74.7	74.9	60.4	66.4	66.9	64.3	67.6	67.6
Table 11: Accuracy results for the different metrics on the TRUE development set. Note that thresholds were tuned on the development set itself. We exclude VitaminC and FEVER from the average calculation as  $\mathrm{SC}_{\mathrm{ZS}}$  was trained on VitaminC that includes examples from FEVER. The highest score in each row (excluding the Ensemble) is in bold and the aforementioned SC results are in strikethrough.
Ensemble	Q2	ANLI	SCZS	BLEURT	QuestEval	FactCC	BARTscore	BERTscore
FRANK	83.0	81.5	82.0	79.0	76.6	73.0	72.1	80.7	75.6
BEGIN	76.8	74.1	76.8	78.9	74.3	73.4	62.09	74.8	78.1
DialFact	80.9	78.1	68.4	74.2	67.1	69.0	52.5	58.6	60.2
PAWS	84.8	84.1	82.1	82.3	62.9	64.8	60.7	70.9	69.8
VitaminC	92.1	77.5	83.9	94.2	59.0	63.3	55.5	59.8	58.0
Avg. w/o VitC	81.4	79.4	77.3	78.6	70.2	70.0	62.1	71.3	70.9