SlowGuess/NAACL_2022_OCR huggingface dataset:
T5 (template): We used text from representation template-based sentence as input to the T5 model.
5.2 Modelling Approaches
Based on the previously described table representation methods, we evaluated state-of-the-art NLP models on PubMedTab. We use models previously applied in table fact-checking (BERT, ALBERT, RoBERTa) (Chen et al., 2020b; Gupta et al., 2020; Aly et al., 2021), as well as domain-specific models (BioBERT, BlueBERT, ClinicalBERT), pretrained on large-scale health datasets. We describe the models below:
BERT: We used the uncased BERT-base (Devlin et al., 2019) model from huggingface library<sup>10</sup>. ALBERT: A transformer-based model that extends BERT with a parameter-reduction technique, resulting in lower memory consumption and higher training speed (Lan et al., 2020).
10https://huggingface.co
Our Dataset	TabFact	InfoTabs	FEVEROUS
Total number of tables	1,942	16,573	2,540	28,760
% of tables with caption	21%	100%	100%	22%
% of tables with header	56.9%	100%	100%	97%
% of tables with <5 rows	23.1%	0.1%	7.5%	18%
% of tables with =>5 rows & <= 10 rows	53.8%	40.7%	56%	44%
% of tables with >10 rows	23.1%	59.2%	36.5%	38%
Ratio of cells with only string content	30.6%	40.1%	45.8%	34%
Ratio of cells with numerical content	59%	53.6%	35.5%	40%
Avg number of characters per cell	13.4	8.6	22.6	17.3
Table 5: Comparison of table fact checking datasets.
Train	Valid	Test	Sum
Support	810	106	103	1019
Refute	370	46	46	462
NEI	373	43	45	461
Sum	1553	195	194	1942
RoBERTa: We used the RoBERTa-Large model released by Nie et al. (2020). The model was pretrained on SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), ANLI (Nie et al., 2020), and FEVER (Thorne et al., 2018).
BioBERT: A domain-specific BERT model, pretrained on PubMed abstracts and PMC full-text articles (Lee et al., 2020). The model was finetuned on two NLI datasets, SNLI and MultiNLI.
BlueBERT: The model was pre-trained on PubMed abstracts and MIMIC-III clinical notes, a database of electronic health records from ICU patients at a Boston hospital (Peng et al., 2019).
ClinicalBERT: A BERT model which was pretrained on MIMIC-III data (Huang et al., 2019).
TAPAS: An extension to BERT which uses additional, table-specific embeddings (column embeddings, row embeddings, rank embeddings) that capture the table structure (Herzig et al., 2020). We experiment with TAPAS on our dataset as it achieved good performance on the TabFact dataset.
We partitioned the dataset into training  $(80\%)$ , test  $(10\%)$ , and validation  $(10\%)$  sets. Table 6 shows the class distribution across the dataset split. We performed hyper-parameter search on the validation set and evaluated the following parameters for each model before selecting the best-performing combination:  $\{4,8,16\}$  for batch size,  $\{1e-3,1e-5,1e-7\}$  for learning rate,  $\{2,3,4,5\}$  for training epochs, and  $\{0.01,0.001,0.0001\}$  for
Table 6: Class distribution across dataset split.
Represent.	All	Sup.	Ref.	NEI
BERT	concatenation	0.60	0.72	0.28	0.81
template sent.	0.57	0.78	0.04	0.89
template concat.	0.57	0.75	0.11	0.85
T5 concat.	0.55	0.75	0.07	0.83
T5 template	0.53	0.71	0.03	0.84
ALBERT	concatenation	0.55	0.72	0.15	0.79
template sent.	0.58	0.69	0.27	0.79
template concat.	0.55	0.71	0.17	0.78
T5 concat.	0.54	0.74	0.07	0.83
T5 template	0.55	0.75	0.11	0.79
RoBERTa	concatenation	0.69	0.79	0.44	0.84
template sent.	0.70	0.77	0.48	0.84
template concat.	0.66	0.75	0.39	0.84
T5 concat.	0.73	0.78	0.52	0.89
T5 template	0.68	0.74	0.45	0.84
BioBERT	concatenation	0.57	0.68	0.29	0.76
template sent.	0.60	0.71	0.33	0.76
template concat.	0.58	0.68	0.3	0.75
T5 concat.	0.58	0.68	0.33	0.73
T5 template	0.58	0.71	0.30	0.74
BlueBERT	concatenation	0.50	0.72	0.04	0.77
template sent.	0.56	0.71	0.23	0.74
template concat.	0.54	0.69	0.20	0.75
T5 concat.	0.52	0.70	0.13	0.72
T5 template	0.54	0.68	0.22	0.72
ClinicalBERT	concatenation	0.51	0.75	0	0.78
template sent.	0.58	0.72	0.20	0.83
template concat.	0.58	0.74	0.19	0.80
T5 concat.	0.55	0.76	0.10	0.80
T5 template	0.55	0.73	0.13	0.78
TAPAS	0.48	0.67	0.28	0.48
Table 7:  $F_{1}$  (macro) score for different state-of-the-art models and table representations on PubMedTab.
weight decay.
We evaluated and compared the table representation and modelling approaches, and report the overall (macro)  $F_{1}$  score and the  $F_{1}$  scores for each class in Table 7.
Table Representations. The resulting  $F_{1}$  scores across all models and veracity classes remained overall the same when different methods for table representation were applied. The template-based
Dataset	All	Sup.	Ref.	NEI
Concat.	PubHealthTab	0.69	0.79	0.44	0.84
InfoTabs	0.78	0.78	0.76	0.81
TabFact	0.49	0.34	0.65	-
FEVEROUS	0.68	0.89	0.87	0.29
T. sent.	PubHealthTab	0.70	0.77	0.48	0.84
InfoTabs	0.77	0.77	0.73	0.81
TabFact	0.44	0.23	0.65	-
FEVEROUS	0.66	0.88	0.85	0.27
T. concat.	PubHealthTab	0.66	0.75	0.39	0.84
InfoTabs	0.78	0.78	0.75	0.81
TabFact	0.50	0.36	0.65	-
FEVEROUS	0.67	0.88	0.86	0.26
T5 concat.	PubHealthTab	0.73	0.78	0.52	0.89
InfoTabs	0.73	0.72	0.69	0.77
TabFact	0.47	0.29	0.65	-
FEVEROUS	0.64	0.86	0.83	0.22
T5 temp.	PubHealthTab	0.68	0.74	0.45	0.84
InfoTabs	0.72	0.72	0.68	0.77
TabFact	0.46	0.25	0.67	-
FEVEROUS	0.64	0.86	0.83	0.24
Table 8:  $F_{1}$  score for RoBERTa with different representation methods on various table fact-checking datasets.
sentence approach outperforms other representation techniques in terms of the overall  $F_{1}$  score for four out of six models (i.e. ALBERT, BioBERT, BlueBERT, and ClinicalBERT). However, for all four models, the difference to the second highest scoring representation was relatively small, between 0.02 and 0.03. Thus, choosing between concatenation and template did not seem to influence the overall claim classification.
Models. RoBERTa outperformed the other models across all representations, followed by BioBERT. The highest macro  $F_{1}$  score (0.73) was obtained using RoBERTa with T5 concatenation. The BioBERT model outperformed BERT, ALBERT and all other domain-specific models for all representations except concatenation where BERT yielded a slightly higher overall  $F_{1}$  score. Surprisingly, TAPAS achieved the lowest score. We believe that this is attributed to the small dataset; while TAPAS is one of the best-performing models on TabFact (Eisenschlos et al., 2020), our training set is much smaller, which can pose a challenge to the BERT-based model.
Performance on refuted claims. Across all applied models and table representations, we obtained a noticeable low  $F_{1}$  score for PubHealthTab refuted claims compared to the two other veracity classes, support and NEI. The  $F_{1}$  scores ranged from 0 (ClinicalBERT with concatenation) to 0.52 (RoBERTa and T5 concatenation).
To determine if this scenario was specific to our dataset, we compared the  $F_{1}$  scores we obtained
on our dataset using RoBERTa with other table fact-checking datasets. The results are shown in Table 8. While the  $F_{1}$  score for PubMedTab refuted claims was between 0.39 and 0.52 using RoBERTa, this value was between 0.65 and 0.87 for refuted claims from TabFact, InfoTabs and FEVERIOUS. Whilst the low performance of RoBERTa on FEVEROUS NEI claims can be attributed to the imbalanced class distribution (Aly et al., 2021), this is not the case for PubMedTab as the three veracity classes {support, refute, NEI} are present in a ratio of 2:1:1 in our training set. We believe that the comparably low performance of RoBERTa on PubMedTab refute claims is due to the fact that state-of-the-art representation and modelling approaches were previously evaluated on Wikipedia evidence tables. These approaches seem to struggle with noisy web tables: lacking table captions and headers, a higher ratio of numerical content, and a lower ratio of string-only content (see Section 4.4) could pose a challenge for generating table representations and for pre-trained models previously evaluated on tables from single data sources.
The results we obtained using RoBERTa on TabFact are lower compared to the other datasets. Whilst Chen et al. (2020b) do not report the results per class, the overall  $F_{1}$  score we obtained is comparable to their baseline.
We introduced PubHealthTab, a table-based dataset for evidence-based fact checking centred on real-world public health claims. Our dataset comprises 1,942 claim-table pairs, with tabular evidence data extracted from websites similar to those used by fact checkers. We described the dataset creation process and the steps taken to minimise biases and correlations. We evaluated state-of-the-art representation and modelling approaches and showed that the RoBERTa model achieves the highest performance on PubHealthTab across all representation methods compared to other models. In contrast to previous table-based fact-checking datasets that contain tables from single data sources, state-of-the-art models struggle to correctly classify refute claims from PubHealthTab against evidence consisting of mostly numerical data or with noisy text headers, making PubHealthTab a challenging dataset for table-based fact-checking research.
The PubMedTab dataset can be used for developing and evaluating fact checking systems intended for a real-world context. The labels supports, refutes and not enough information describe a claim's veracity given the evidence table. We do not make any statement on PubMedTab claims' truthfulness in a real-world context.
We obtained ethical clearance prior to crowdsourcing from the relevant authority in the academic institution. We informed the participants about the data being collected and its purpose. Participants had the opportunity to withdraw at any time and to provide feedback at the end of each task. All workers were from English speaking countries. The payment was above the minimum wage and decided based on the time workers spent on the pilot tasks. For the first and third tasks we paid 0.75USD (2.5 minutes per task on average) and for the second 1.35USD (average 5 minutes per task).
The authors acknowledge support from the Distributed AI (DAI) research group at King's College London for creating the dataset.
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVERIOUS: fact extraction and verification over unstructured and structured information. CoRR, abs/2106.05707.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4685-4697, Hong Kong, China. Association for Computational Linguistics.
Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2015. Soylent: a word processor with a crowd inside. Communications of the ACM, 58(8):85-94.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference.
In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942, Online. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. TabFact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR. OpenReview.net.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026-1036, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Thomas Diggelmann, Jordan L. Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. CoRR, abs/2012.00614.
Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 2623-2634. ACM.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with intermediate pre-training. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 281-296, Online. Association for Computational Linguistics.
Stefan Evert. 2005. The statistics of word cooccurrences: word pairs and collocations.
Alvan R Feinstein and Domenic V Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of clinical epidemiology, 43(6):543-549.
Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in crowdsourcing microtasks. In Design for Teaching and Learning in a Networked World - 10th European Conference on Technology Enhanced Learning, EC-TEL, volume 9307 of Lecture Notes in Computer Science, pages 100-114. Springer.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics.
Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. ManyModalQA: Modality disambiguation and QA over diverse inputs. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI, pages 7879-7886. AAAI Press.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. CoRR, abs/1904.05342.
Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831, Vancouver, Canada. Association for Computational Linguistics.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3441-3460, Online. Association for Computational Linguistics.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740-7754, Online. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR. OpenReview.net.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159-174.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433-6441, Online. Association for Computational Linguistics.
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeno, Rubén Miguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021. The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR, volume 12657 of Lecture Notes in Computer Science, pages 639-649. Springer.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Jeppe Nørregaard and Leon Derczynski. 2021. DanFEVER: claim verification dataset for Danish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 422-428, Reykjavik, Iceland (Online). Linkoping University Electronic Press, Sweden.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 1470-1480, Beijing, China. Association for Computational Linguistics.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58-65, Florence, Italy. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:140:1-140:67.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624-643, Online. Association for Computational Linguistics.
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1451-1465, Online. Association for Computational Linguistics.
James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. CoRR, abs/2104.00640.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021. SemEval2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TABFACTS). In Proceedings of the 15th International
Workshop on Semantic Evaluation (SemEval-2021), pages 317-326, Online. Association for Computational Linguistics.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426, Vancouver, Canada. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
A.1 Dataset Creation
We evaluated the following conditions for the second crowdsourcing task. Workers could only submit their work if all checks were passed:
- A veracity label is selected for the adjusted claim.
- Minimum 2.5 seconds are spend on each HIT page for adjusting the claim.
- Adjusted claim length is between 5 and 30 tokens.
- The adjusted claim is different from the initial claim.
- The adjusted claim text does not contain ambiguous words, i.e. maybe, probably, mostly, occasionally, frequently, might, many, few, some, several, most of, sometimes.
- The adjusted claim does not contain negation words, i.e. not, never, none, nobody.
After hyperparameter tuning on the validation set, we selected the following parameters for the different modelling approaches displayed in Table 9.
Welcome to the Table-Claim-Checking-Task.
We will show you seven claims related to public health. Each claim comes with a table. For each claim, you will be asked to answer the following questions: Is the table displayed supporting or refuting the claim? Is the table related (= mention the same topic) or unrelated to the claim? Answer the question using only information from the table. Please avoid using any additional sources or own knowledge.
We will show you some examples of how to solve this task on the next page.
Afterwards, we will ask you three questions. You can progress to the task by answering these questions correctly. The task also includes some checks. You can only submit and receive payment by passing them successfully.
The data collected in this study consist of your inputs, the time you spent on the task and the buttons clicked. This experiment received ethical clearance on 17th April, 2021 from with registration number The data controller for this project will you have any questions or need further assistance, please contact
By continuing with the task, you agree to take part in this research project and consent for the data collected to be used for the purpose of this study.
Figure 3: Introduction text for table relevance and verification task.
Welcome to the Table-Claim-Generation-Task.
We will show you five claims related to public health. Each claim comes with an accompanied table. For each claim, you will be asked to do the following: Adjust the claim so that it can be verified or refuted given the table. Write the claim using only information from the table. Avoid using any additional sources or own knowledge.
Figure 4: Introduction text for claim adjustment task.