SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4 shows the evidence extraction results. With MCS, we achieve an increase of  $29.71\%$  on cell recall and  $13.22\%$  on the overall evidence recall over FaBULOUS. The result indicates that considering only one table is not enough and we should pay attention to the relevance scores of the input tables especially when the cell selector is somewhat weak.
We evaluate the effect of each part of DCUF with a collection of ablation experiments. The Experiment settings are as follows. (1) w/o Table Format Encoding We only use the unified text format evidence encoding for verdict prediction. (2) w/o Table Format Encoding We only use the unified table format evidence encoding for verdict predic
Figure 4: Few shot results on the development set.
tion. (3) w/o Dual-Channel Predictor We use the verdict predictor in the baseline to predict the veracity label.
Table 4: Categorized evidence recall on the development set. The recall is calculated with at most 3 tables, 5 sentences and 25 table cells.
Models	Feverous	Accuracy
DCUF	35.77	72.91
w/o Table Format Encoding	35.16	72.05
w/o Text Format Encoding	32.41	66.80
w/o Dual Channel Predictor	27.91	58.40
Table 5: Ablation results on the development set.
The results are shown in Table 5. The FEVEROUS score and accuracy drop consistently when we remove the unified table format encoding or the unified text format encoding. Especially, the FEVEROUS score drops by  $3.34\%$  when only using the unified tabular encoding for prediction. With only unified text format encoding, the FEVEROUS score drops by  $0.61\%$ , which may contribute to pre-trained table models, such as TAPAS, being still weaker compared to pre-trained language models. However, with our carefully-designed context-aware unified format conversion, verdict prediction upon one format encoding outperforms all previous results. To relieve the effect of different evidence extraction methods, we train the verdict prediction model in the baseline with our evidence extraction results. We see a great drop, with accuracy dropping of  $14.51\%$  and FEVEROUS score by  $7.86\%$ , which proves that our combined unified format verdict prediction model can keep the context needed when converting the evidence format and help the extracted evidence perform early interaction in a unified format.
Few-shot Results Figure 4 shows the FEVER-OUS score on the development set when we train unified DCUF, DCUF without text format encoding and DCUF without table format encoding verdict predictors on  $1\%$ ,  $5\%$  and  $10\%$  instances of the training set respectively. We find that, training on only  $10\%$  instances, all of the three settings outperform previous SOTA results, with the dual channel predictor achieving the best FEVEROUS score of  $33.9\%$ . Meanwhile, with only  $1\%$  of the training set, namely, 713 instances, the unified text format predictor outperforms previous SOTA result by  $0.9\%$ . These improvements may contribute to our carefully-designed format conversion methods. The text format evidence converted from cells is similar to sentences in natural language when keeping much context information in the conversion procedure. And the concatenated claim-evidence string is the same as previous fact verification tasks on which the RoBERTa checkpoint is fine-tuned. Meanwhile, a given claim, and a single global evidence table are consistent to the input requirements in the pre-training step of TAPAS. With few new parameters introduced and an input form strictly complying with the requirements of pre-trained models and previous single format fact-checking tasks, DCUF could make the most of the pre-training stage and, thus, learn well in the few-shot setting.
We also find that when training with only  $1\%$  instances, the unified text format predictor outperform other two settings. As the number of training instances increase, dual-channel predictor learns how to combine information from the two channel, thus achieving better results.
Error Analysis We find that when converting cells to text with rule-based methods, there are inevitably noise or not fluent sentences introduced.
One problem is caused by the latent header cells. As shown in Table 6(a), the cell "Team" is selected but not explicitly marked as a header cell, so a meaningless sentence "Senior career*: Years for Aramais Yepiskoposyan is Team." is derived from this cell. Meanwhile, although the header cells indicating the attribute type are usually nouns, there are some exceptions. Infobox Table 6(b) is an example. The selected cell "Sir Roger Manwood" is converted to a fluent and meaningful evidence sentence "Information: Founder for Sir Roger Manwood's School is Sir Roger Manwood". However, "Information: Established for Sir Roger Manwood's School is 1563; 457 years ago." is not a sentence
(a) Case A
Wikipedia page: Aramais Yepiskoposyan
Senior career*
Years	Team	Apps
1986-1991	FC Ararat Yerevan	10
1997	FC Kuban Krasnodar	8
(b) Case B
Wikipedia page: Sir Roger Manwood's School
Information
Founder	Sir Roger Manwood
Established	1563; 457 years ago
(c) Case C Wikipedia page: List of The Simpsons cast members
Episodes	Actor	Character(s)
179	Marcia Wallace	Edna Krabappel
52	Phil Hartman	Troy McClure
Table 6: Example tables in the FEVEROUS dataset.
that conforms to English grammar. And sometimes, the latent row header which indicates the object name is confusing. For example, the cell "Marcia Wallace" in Table 6(c) is converted to a sentence "Actor for 179 is Marcia Wallace." Without the header "Episodes", what "179" refers to is confusing. Assessing and polishing the converted sentence may help to solve the problems presented above.
Fact Verification over Unstructured Evidence Thorne et al. (2018) proposed FEVER, a large-scale dataset of claims based on Wikipedia articles. Language models have better performance compared to other methods e.g. ESIM-based models(Hanselowski et al., 2018; Nie et al., 2019). BERT-based models make the prediction based on collected evidence in a direct aggregating rule(Soleimani et al., 2020) or a graph-based approach(Zhou et al., 2019; Zhong et al., 2020).
Fact Verification over Structured Evidence Benchmarks for fact verification on structured evidence are built on tables collected from Wikipedia(Chen et al., 2020) or scientific articles(Wang et al., 2021). Many previous works search latent programs as an intermediary to reason over the given table. They directly encode programs (Chen et al., 2020) or construct heterogeneous graphs (Shi et al., 2020; Yang et al., 2020) with the claim, the table and the programs. Another
way is to linearize the input table and perform table pre-training (Chen et al., 2020) and add additional table-aware embeddings (Herzig et al., 2020; Eisenschlos et al., 2020) to enhance the table encoding. However, in these datasets, the evidence is only one given table, and models are not requested to find out the evidence cells explicitly.
Fact Verification over Structured and Unstructured Evidence FEVEROUS (Aly et al., 2021) is the first dataset of fact verification on structured and unstructured evidence. Many previous works follow the baseline settings and convert all evidence to text format to perform evidence interaction. They transform each cell to header-value pairs (Aly et al., 2021; Malon, 2021) or in a cell location indication type (Kotonya et al., 2021a). They pay less attention to making the converted text more consistent with natural language expressions or identifying what the context cells represent. Bouziane et al. (2021) propose to convert all evidence to tables. They simply convert each sentence to a 2-cell table with the Wikipedia title and itself instead of packing closely-tied evidence and building a global evidence table. There are also works focusing on the first two steps two improve the final results. Saeed et al. (2021) propose to add a document re-ranker to strengthen the document retrieval. Multi-hop Dense Retriever (Bouziane et al., 2021) and T5 generator (Malon, 2021) are introduced to better extract multi-hop evidence.
In this paper, we propose DCUF, a dual channel unified format model for fact verification over structured and unstructured data. With context-aware evidence format conversion, DCUF gets a unified text format representation of all evidence and a global evidence table of them at the same time. The dual channel design helps us make the most of existing pre-trained language/table models to encourage all evidence pieces to interact with each other in their best forms as early as possible. Experiments show that, with dual-channel unified format encoding, our proposed DCUF achieves state-of-the-art performance and also comparable results in few-shot settings.
This work is supported in part by NSFC (62161160339). We would like to thank the anonymous reviewers and action editors for their helpful
comments and suggestions. For any correspondence, please contact Yansong Feng.
Rami Aly, Zhijiang Guo, M. Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. Feverous: Fact extraction and verification over unstructured and structured information. *ArXiv*, abs/2106.05707.
Mostafa Bouziane, Hugo Perrin, Amine Sadeq, Thanh Nguyen, Aurélien Cluzeau, and Julien Mardas. 2021. Fabulous: Fact-checking based on understanding of language over unstructured and structured information. In Proceedings of the Fourth Workshop on Fact Extraction and VERIFICATION (FEVER), pages 31-39.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with intermediate pre-training. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 281-296, Online. Association for Computational Linguistics.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 103-108, Brussels, Belgium. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Kelvin Jiang, Ronak Pradeep, and Jimmy Lin. 2021. Exploring listwise evidence reasoning with T5 for fact verification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021,
(Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 402-410. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Neema Kotonya, Thomas Spooner, Daniele Magazzeni, and Francesca Toni. 2021a. Graph reasoning with context-aware linearization for interpretable fact extraction and verification. ArXiv, abs/2109.12349.
Neema Kotonya, Thomas Spooner, Daniele Magazzeni, and Francesca Toni. 2021b. Graph reasoning with context-aware linearization for interpretable fact extraction and verification. arXiv preprint arXiv:2109.12349.
Canasai Kruengkrai, Junichi Yamagishi, and Xin Wang. 2021. A multi-level attention model for evidence-based fact checking. In ACL/IJCNLP (Findings), volume ACL/IJCNLP 2021 of Findings of ACL, pages 2447-2460. Association for Computational Linguistics.
Christopher Malon. 2021. Team papelo at feverous: Multi-hop evidence pursuit. Proceedings of the Fourth Workshop on Fact Extraction and VERIFICATION (FEVER).
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6859-6866. AAAI Press.
Mohammed Saeed, Giulio Alfarano, Khai Nguyen, Duc-Hong Pham, Raphael Troncy, and Paolo Papotti. 2021. Neural re-rankers for evidence retrieval in the feverous task. Proceedings of the Fourth Workshop on Fact Extraction and VERIFICATION (FEVER).
Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. 2020. Learn to combine linguistic and symbolic information for table-based fact verification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5335-5346, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Amir Soleimani, Christof Monz, and Marcel Worring. 2020. Bert for evidence retrieval and claim verification. Advances in Information Retrieval, 12036:359.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Kateryna Tymoshenko and Alessandro Moschitti. 2021. Strong and light baseline models for fact-checking joint inference. In ACL/IJCNLP (Findings), volume ACL/IJCNLP 2021 of Findings of ACL, pages 4824-4830. Association for Computational Linguistics.
Nancy XR Wang, Diwakar Mahajan, Marina Danilevsk Rosenthal, et al. 2021. Semeval-2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (sem-tab-facts). arXiv preprint arXiv:2105.13995.
Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2020. Program enhanced fact verification with verbalization and graph attention network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7810-7825, Online. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180, Online. Association for Computational Linguistics.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
A Statistics of the FEVEROUS dataset
The FEVEROUS is an open-domain English dataset. It contains 87,026 claims, and the claim length is 225.3 on average. Each claim averagely needs 1.4 sentences and 3.3 cells (0.8 tables) to be verified. 34,963 instances need only text format evidence, 28,760 only table format and 24,667 need a combination of the two formats. There are 49,115 instances labeled SUPPORTS, 33,669 labeled Refutes and the rest 4,242 instances are labeled NEI. Detailed label and evidence distributions are shown in Table 7.
Train	Dev	Test
Supported	41,835(59%)	3,908(50%)	3,372 (43%)
Refuted	27,215(38%)	3,481(44%)	2,973 (38%)
NEI	2,241 (3%)	501 (6%)	1,500 (19%)
Total	71,291	7,890	7,845
Sentences	31,607(41%)	3,745(43%)	3589 (42%)
Cells	25,020 (32%)	2,738(32%)	2816 (33%)
Sentence+Cells	20,865 (27%)	2,468 (25%)	2062 (24%)
Table 7:FEVEROUS Distribution.