Venue: NAACL 2021

# Incorporating External Knowledge to Enhance Tabular Reasoning

J. Neeraja

IIT Guwahati

jneeraja@iitg.ac.in &Vivek Gupta\({}^{*}\)

University of Utah

vqupta@cs.utah.edu &Vivek Srikumar

University of Utah

svivek@cs.utah.edu

###### Abstract

Reasoning about tabular information presents unique challenges to modern NLP approaches which largely rely on pre-trained contextualized embeddings of text. In this paper, we study these challenges through the problem of tabular natural language inference. We propose easy and effective modifications to how information is presented to a model for this task. We show via systematic experiments that these strategies substantially improve tabular inference performance.

+
Footnote †: _Correspondings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2799–2809 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Natural Language Inference (NLI) is the task of determining if a hypothesis sentence can be inferred as true, false, or undetermined given a premise sentence Dagan et al. (2013). Contextual sentence embeddings such as BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019), applied to large datasets such as SNLI Bowman et al. (2015) and MultiNLI Williams et al. (2018), have led to near-human performance of NLI systems.

In this paper, we study the harder problem of reasoning about _tabular_ premises, as instantiated in datasets such as TabFact Chen et al. (2019) and InfoTabS Gupta et al. (2020). This problem is similar to standard NLI, but the premises are Wikipedia tables rather than sentences. Models similar to the best ones for the standard NLI datasets struggle with tabular inference. Using the InfoTabS dataset as an example, we present a focused study that investigates (a) the poor performance of existing models, (b) connections to information deficiency in the tabular premises, and, (c) simple yet effective mitigations for these problems.

We use the table and hypotheses in Figure 1 as a running example through this paper, and refer to the left column as its keys.1 Tabular inference is challenging for several reasons: (a) **Poor table representation**: The table does not explicitly state the relationship between the keys and values. (b) **Missing implicit lexical knowledge** due to limited training data: This affects interpreting words like '_fewer_', and '_over_' in H1 and H2 respectively. (c) **Presence of distracting information**: All keys except _No. of listings_ are unrelated to the hypotheses H1 and H2. (d) **Missing domain knowledge about keys**: We need to interpret the key _Volume_ in the financial context for this table.

Footnote 1: Keys in the InfoTabS tables are similar to column headers in the TabFact database-style tables.

In the absence of large labeled corpora, any modeling strategy needs to explicitly address these problems. In this paper, we propose effective approaches for addressing them, and show that they lead to substantial improvements in prediction quality, especially on adversarial test sets. This focused study makes the following contributions:

1. We analyse why the existing state-of-the-art BERT class models struggle on the challenging task of NLI over tabular data.
2. We propose solutions to overcome these challenges via simple modifications to inputs using existing language resources.

Figure 1: A tabular premise example. The hypotheses H1 is entailed by it, H2 is a contradiction and H3 is neutral i.e. neither entailed nor contradictory.



3. Through extensive experiments, we show significant improvements to model performance, especially on challenging adversarial test sets.

The updated dataset, along with associated scripts, are available at [https://github.com/utahnlp/knowledge_infotabs](https://github.com/utahnlp/knowledge_infotabs).

## 2 Challenges and Proposed Solutions

We examine the issues highlighted in SS1 and propose simple solutions to mitigate them below.

Better Paragraph Representation (BPR):One way to represent the premise table is to use a universal template to convert each row of the table into sentence which serves as input to a BERT-style model. Gupta et al. (2020) suggest that in a table titled t, a row with key k and value v should be converted to a sentence using the template: "The k of t are v." Despite the advantage of simplicity, the approach produces ungrammatical sentences. In our example, the template converts the _Founded_ row to the sentence _"The Founded of New York Stock Exchange are May 17, 1792; 226 years ago."_.

We note that keys are associated with values of specific entity types such as Money, Date, Cardinal, and Bool, and the entire table itself has a category. Therefore, we propose type-specific templates, instead of using the universal one.2 In our example, the table category is _Organization_ and the key _Founded_ has the type Date. A better template for this key is "\({}^{*}\!\mathrm{t}\) was k on v", which produces the more grammatical sentence _"New York Stock Exchange was Founded on May 17, 1792; 226 years ago."_. Furthermore, we observe that including the table category information i.e. _"New York Stock Exchange is an Organization."_ helps in better premise context understanding.3 Appendix A provides more such templates.

Footnote 2: The construction of the template sentences based on entity type is a one-time manual step.

Footnote 3: This category information is provided in the InfoTabS and TabFact datasets. For other datasets, it can be inferred easily by clustering over the keys of the training tables.

Implicit Knowledge Addition (KG implicit):Tables represent information _implicitly_; they do not employ connectives to link their cells. As a result, a model trained only on tables struggles to make lexical inferences about the hypothesis, such as the difference between the meanings of _'before'_ and _'after'_, and the function of negations. This is surprising, because the models have the benefit of being pre-trained on large textual corpora.

Recently, Andreas (2020) and Pruksachatkun et al. (2020) showed that we can pre-train models on specific tasks to incorporate such implicit knowledge. Eisenschlos et al. (2020) use pre-training on synthetic data to improve the performance on the TabFact dataset. Inspired by these, we first train our model on the large, diverse and _human-written_ MultiNLI dataset. Then, we fine tune it to the InfoTabS task. Pre-training with MultiNLI data exposes the model to diverse lexical constructions. Furthermore, it increases the training data size by \(433\)K (MultiNLI) example pairs. This makes the representation better tuned to the NLI task, thereby leading to better generalization.

Distracting Rows Removal (DRR)Not all premise table rows are necessary to reason about a given hypothesis. In our example, for the hypotheses H1 and H2, the row corresponding to the key _No. of listings_ is sufficient to decide the label for the hypothesis. The other rows are an irrelevant distraction. Further, as a practical concern, when longer tables are encoded into sentences as described above, the resulting number of tokens is more than the input size restrictions of existing models, leading to useful rows potentially being cropped. Appendix F shows one such example on the InfoTabS. Therefore, it becomes important to prune irrelevant rows.

To identify relevant rows, we employ a simplified version of the alignment algorithm used by Yadav et al. (2019, 2020) for retrieval in reading comprehension.

First, every word in the hypothesis sentence is aligned with the most similar word in the table sentences using cosine similarity. We use fastText Joulin et al. (2016); Mikolov et al. (2018) embeddings for this purpose, which preliminary experiments revealed to be better than other embeddings. Then, we rank rows by their similarity to the hypothesis, by aggregating similarity over content words in the hypothesis. Yadav et al. (2019) used inverse document frequency for weighting words, but we found that simple stop word pruning was sufficient. We took the top \(k\) rows by similarity as the pruned representative of the table for this hypothesis. The hyper-parameter \(k\) is selected by tuning on a development set. Appendix B gives more details about these design choices.

Explicit Knowledge Addition (KG explicit):We found that adding _explicit_ information to enrich 

keys improves a model's ability to disambiguate and understand them. We expand the pruned table premises with contextually relevant key information from existing resources such as WordNet (definitions) or Wikipedia (first sentence, usually a definition).4

Footnote 3: [https://github.com/facebook-of-the-bertbert-language-for-wordnet-based-learning-and-play-in-view-of-words-with-training-from-existing-to-use-the-topologies-of-the-billion-2019-thinking-2018-thinking-2](https://github.com/facebook-of-the-bertbert-language-for-wordnet-based-learning-and-play-in-view-of-words-with-training-from-existing-to-use-the-topologies-of-the-billion-2019-thinking-2018-thinking-2) 

BprAs shown in Table 2, with BPR, we observe that the RoBERTa\({}_{L}\) model improves performance on all dev and test sets except \(\alpha_{3}\). There are two main reasons behind this poor performance on \(\alpha_{3}\).

First, the zero-shot \(\alpha_{3}\) data includes unseen keys. The number of keys common to \(\alpha_{3}\) and the training set is \(94\), whereas for, dev, \(\alpha_{1}\) and \(\alpha_{2}\) it is \(334\), \(312\), and \(273\) respectively (i.e., 3-5 times more). Second, despite being represented by better sentences, due to the input size restriction of RoBERTa\({}_{L}\) some relevant rows are still ignored.

KG implicitWe observe that _implicit_ knowledge addition via MNLI pre-training helps the model reason and generalize better. From Table 2, we can see significant performance improvement in the dev and all three test sets.

DrrThis leads to significant improvement in the \(\alpha_{3}\) set. We attribute this to two primary reasons: First, \(\alpha_{3}\) tables are longer (\(13.1\) keys per table on average, vs. \(8.8\) keys on average in the others), and DRR is important to avoid automatically removing keys from the bottom of a table due to the limitations in RoBERTa\({}_{L}\) model's input size. Without these relevant rows, the model incorrectly predicts the neutral label. Second, \(\alpha_{3}\) is a zero-shot dataset and has significant proportion of unseen keys which could end up being noise for the model. The slight decrease in performance on the dev, \(\alpha_{1}\) and \(\alpha_{2}\) sets can be attributed to model utilising spurious patterns over irrelevant keys for prediction.8 We validated this experimentally by testing the original premise trained model on the DRR test tables. Table 5 in the Appendix C shows that without pruning, the model focuses on irrelevant rows for prediction.

Footnote 8: Performance drop of dev and \(\alpha_{2}\) is also marginal i.e. (dev: 79.57 to 78.77, \(\alpha_{1}\): 78.27 to 78.13, \(\alpha_{2}\): 71.87 to 70.90), as compared to InfoTabS WMD-top3 j.e (dev: 75.5 to 72.55,\(\alpha_{1}\): 74.88 to 70.38, \(\alpha_{2}\): 65.44 to 62.55), here WMD-top3 performance numbers are taken from Gupta et al. (2020).

KG explicitWith _explicit_ contextualized knowledge about the table keys, we observe a marginal improvement in dev, \(\alpha_{1}\) test sets and a significant performance gain on the \(\alpha_{2}\) and \(\alpha_{3}\) test sets. Improvement in the \(\alpha_{3}\) set shows that adding external knowledge helps in the zero-shot setting. With \(\alpha_{2}\), the model can not utilize spurious lexical correlations9 due to its adversarial nature, and is forced to use the relevant keys in the premise tables, thus adding explicit information about the key improves performance more for \(\alpha_{2}\) than \(\alpha_{1}\) or dev. Appendix F shows some qualitative examples.

Footnote 9: The hypothesis-only baseline for \(\alpha_{2}\) is 48.5% vs. \(\alpha_{1}\): 60.5 % and dev: 60.5 % (Gupta et al., 2020)

### Ablation Study

We perform an ablation study as shown in table 3, where instead of doing all modification sequentially one after another (+), we do only one modification at a time to analyze its effects.

Through our ablation study we observe that: (a) **DRR** improves performance on the dev, \(\alpha_{1}\), and \(\alpha_{2}\) sets, but slightly degrades it on the \(\alpha_{3}\) set. The drop in performance on \(\alpha_{3}\) is due to spurious artifact deletion as explained in details in E. (b) **KG explicit** gives performance improvement in all sets. Furthermore, there is significant boost in performance of the adversarial \(\alpha_{2}\) and \(\alpha_{3}\) sets.10 (c) Similarly, **KG implicit** shows significant improvement in all test sets. The large improvements on the adversarial sets \(\alpha_{2}\) and \(\alpha_{3}\) sets, suggest that the model can now reason better. Although, implicit knowledge provides most performance gain, all modifications are needed to obtain the best performance for all sets (especially on the \(\alpha_{3}\) set).11

Footnote 10: The KG explicit step is performed only for relevant keys (after DRR).

Footnote 11: We show in Appendix D, Table 6, that implicit knowledge addition to a non-sentential table representation i.e. Struc Chen et al. (2019); Gupta et al. (2020) leads to performance improvement as well.

## 4 Comparison with Related Work

Recently, there have been many papers which study several NLP tasks on semi-structured tabular data. These include tabular NLI and fact verification tasks such as TabFact Chen et al. (2019), and InfoTabS Gupta et al. (2020), various question answering and semantic parsing tasks Pasupat and Liang (2015); Krishnamurthy et al. (2017); Abbas et al. (2016); Sun et al. (2016); Chen et al. (2020); Lin et al. (2020), _inter alia_), and table-to-text generation and its evaluation (e.g., Parikh et al., 2020; Radev et al., 2020). Several, models for better representation of tabl

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Premise & Dev & \(\alpha_{1}\) & \(\alpha_{2}\) & \(\alpha_{3}\) \\ \hline Para & 75.55 & 74.88 & 65.55 & 64.94 \\ \hline DRR & 76.39 & 75.78 & 67.22 & 64.88 \\ KG explicit & 77.16 & 75.38 & 67.88 & 65.50 \\ KG implicit & **79.06** & **78.44** & **71.66** & **67.55** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation results with individual modifications.

 

et al., 2020), TaBERT Yin et al. (2020), and Tabstruc Zhang et al. (2020) were recently proposed. Yu et al. (2018, 2021) and Eisenschlos et al. (2020) study pre-training for improving tabular inference, similar to our MutliNLI pre-training.

The proposed modifications in this work are simple and intuitive. Yet, existing table reasoning papers have not studied the impact of such input modifications. Furthermore, much of the recent work focuses on building sophisticated neural models, without explicit focus on how these models (designed for raw text) adapt to the tabular data. In this work, we argue that instead of relying on the neural network to _"magically"_ work for tabular structures, we should carefully think about the representation of semi-structured data, and the incorporation of both implicit and explicit knowledge into neural models. Our work highlights that simple pre-processing steps are important, especially for better generalization, as evident from the significant improvement in performance on adversarial test sets with the same RoBERTa models. We recommend that these pre-processing steps should be standardized across table reasoning tasks.

## 5 Conclusion & Future Work

We introduced simple and effective modifications that rely on introducing additional knowledge to improve tabular NLI. These modifications governs what information is provided to a tabular NLI and how the given information is presented to the model. We presented a case study with the recently published InfoTabS dataset and showed that our proposed changes lead to significant improvements. Furthermore, we also carefully studied the effect of these modifications on the multiple test-sets, and why a certain modification seems to help a particular adversarial set.

We believe that our study and proposed solutions will be valuable to researchers working on question answering and generation problems involving both tabular and textual inputs, such as tabular/hybrid question answering and table-to-text generation, especially with difficult or adversarial evaluation. Looking ahead, our work can be extended to include explicit knowledge for hypothesis tokens as well. To increase robustness, we can also integrate structural constraints via data augmentation through NLI training. Moreover, we expect that structural information such as position encoding could also help better represent tables.

## Acknowledgements

We thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. We also thank the support of NSF grants #1801446 (SATC) and #1822877 (Cyberlearning) and a generous gift from Verisk Inc.

## References

* Abbas et al. (2016) Faheem Abbas, M. K. Malik, M. Rashid, and Rizwan Zafar. 2016. Wikiqa -- a question answering system on wikipedia using freebase, dbpedia and infobox. _2016 Sixth International Conference on Innovative Computing Technology (INTECH)_, pages 185-193.
* Andreas (2020) Jacob Andreas. 2020. Good-enough compositional data augmentation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7556-7566, Online. Association for Computational Linguistics.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Chen et al. (2019) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tbfact: A large-scale dataset for table-based fact verification. In _International Conference on Learning Representations_.
* Chen et al. (2020) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. _Findings of EMNLP 2020_.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanotto. 2013. Recognizing textual entailment: Models and applications. _Synthesis Lectures on Human Language Technologies_, 6(4):1-220.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Eisenschlos et al. (2020) Julian Eisenschlos, Syrine Krichene, and Thomas Mueller. 2020. Understanding tables with intermediate pre-training. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 281-296.
* Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as 

semi-structured data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2309-2324, Online. Association for Computational Linguistics.
* Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Muller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4320-4333, Online. Association for Computational Linguistics.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. _arXiv preprint arXiv:1612.03651_.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1516-1526.
* Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-sql semantic parsing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 4870-4888.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. _arXiv preprint arXiv:1907.11692_.
* Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
* Parikh et al. (2020) Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In _Proceedings of EMNLP_.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1470-1480.
* Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Hutt, Xiaoyi Zhang, Richard Yuanhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5231-5247.
* Radev et al. (2020) Dragomir Radev, Rui Zhang, Amrit Rau, Abhin Anand Sivaprasad, Chichuan Hsieh, Nazneen Fatema Rajani, Xiangru Tang, Aadir Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. _arXiv preprint arXiv:2007.02871_.
* Sun et al. (2016) Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In _Proceedings of the 25th International Conference on World Wide Web_, pages 771-782.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Yadav et al. (2019) Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Alignment over heterogeneous embeddings for question answering. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2681-2691.
* Yadav et al. (2020) Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020. Unsupervised alignment-based iterative evidence retrieval for multi-hop question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4514-4525, Online. Association for Computational Linguistics.
* Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8413-8426, Online. Association for Computational Linguistics.
* Yu et al. (2021) Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing. _International Conference of Learning Representation_.
* Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3911-3921.
* Zhang et al. (2020) Hongzhi Zhang, Yingyao Wang, Siriu Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020. Table fact verification with structure-aware transformer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1624-1629, Online. Association for Computational Linguistics 

BPR Templates

Here, we are listing down some of the diverse example templates we have framed.

* For the table category _Bus/Train Lines_ and key _Disabled access_ with Bool value YES, follow template: "\(t\) has \(k\)."

**Original Premise Sentence** _"The Disabled access of Tukwila International Boulevard Station are Yes."_

**BPR Sentence** _"Tukwila International Boulevard Station has Disabled access."_

* For the table category _Movie_ and key _Box office_ with Money type, follow template: "In the \(k\), \(t\) made \(v\)."

**Original Premise Sentence** _"The Box office of Brokeback Mountain are $178.1 million."_

**BPR Sentence** _"In the Box office, Brokeback Mountain made $178.1 million."_

* For the table category _City_ and key _Total_ with Cardinal type, follow template: "The \(k\) area of \(t\) is \(v\)."

**Original Premise Sentence** _"The Total of Cusco are 435,114."_

* For the table category _Painting_ and key _Also known as_, follow template: "The \(k\) area of \(t\) is \(v\)."

**Original Premise Sentence** _"The Also known as of Et in Arcadia ego are Les Bergers d'Arcadie."_

**BPR Sentence** _"Et in Arcadia ego is Also known as Les Bergers d'Arcadie."_

* For the table category _Person_ and key _Died_ with Date type , follow template: "\(t\) k on \(v\)."

**Original Premise Sentence** _"The Died of Jesse Ramsden are November 1800 (1800-11-05) Brighton, Sussex."_

## Appendix B DRR: fastText and Binary weighting

fastText:For word representation, Yadav et al. (2019) have used BERT and Glove embeddings. In our case, we prefer to use fastText word embeddings over Glove because fastText embedding uses sub-word information which helps in capturing different variations of the context words. Furthermore, fastText embeddings is also as better choice than BERT for our task because 1. Firstly, we are embedding single sentential form of diverse rows instead of longer context similar paragraphs, 2. Secondly, all words (especially keys) of the rows across all the tables are used only in one context, whereas BERT is useful when same word is used with different contexts across paragraphs, 3. Thirdly, in all tables, the number sentences to select from is bounded by maximum rows in the table, which is a small number (8.8 in train, dev, \(\alpha_{1}\), \(\alpha_{2}\) and 13.1 in \(\alpha_{3}\)), and 4. Lastly, using fastText is much faster to compute than BERT for obtaining embeddings.

Binary weighting:Since, we are embedding single sentential form of diverse rowsinstead of longer context related paragraphs, we found that using binary weighting \(0\) for stop words and \(1\) for others is more effective than the idf weighting, which is useful only for longer paragraph context with several lexical terms.

## Appendix C Hyperparameters \(k\) vs test-sets accuracy

We also trained a model both train and tested on the DRR table premise for increasing values of the hyper parameter \(k\), as shown in Table 1. We also test the model trained on the entire para on pruned para with increasing value of hyperparameters \(k\in\{2,3,4,5,6\}\) for the test sets \(\alpha_{1}\), \(\alpha_{2}\), and \(\alpha_{3}\). In all cases, except \(\alpha_{3}\), the performance with larger \(k\) is better. The increase in performance, even with \(k>4\), shows that the model is using more then required keys for prediction. Thus, the model is utilising the spurious pattern in irrelevant rows for the prediction.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline Train & Dev & k=2 & k=3 & k=4 & k=5 & k=6 \\ \hline +DRR & +DRR & 77.61 & 77.94 & 78.16 & 78.38 & 79.00 \\ BPR & +DRR & 71.72 & 74.83 & 77.50 & 78.50 & 79.00 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dev accuracy with increasing hyper parameter \(k\) trained with both BPR and +DRR table.

 



## Appendix D TabFact Representation Experiment

Table 6 implicit knowledge addition effect on non-para _Struc_ representation i.e. a key value linearize representation as "key k : value v", rows separated by semicolon ";" Gupta et al. (2020); Chen et al. (2019). Here too the implicit knowledge addition leads to improvement in performance on all the sets.

## Appendix E Artifacts and Model Predictions

In Table 7 we show percentage of example which were corrected after modification and vice versa. Surprisingly, there is a small percentage of examples which are predicted correctly earlier with original premise (Para) but predicted wrongly after all the modifications (Mod), although such examples are much lesser than opposite case. We suspect that earlier model was also relying on spurious pattern (artifacts) for correct prediction on these examples earlier, which are now corrupted after the proposed modifications. Hence, the new model struggle to predict correctly on such examples.

In the next section F, we also shows qualitative examples, where modification helps model predictor correctly. We also provide some examples via distracting row removal modification, where model fails after modification.

## Appendix F Qualitative Examples

In this section, we provide examples where model is able to predict well after the proposed modifications. We also provide some examples, where model struggles to make the correct prediction after distracting row removal (DRR) modification.

### Bpr

Original PremiseThe Birth name of Eva Mendes are Eva de la Caridad Mendez. Eva Mendes was Born on March 5, 1974 (1974-03-05) (age 44) Miami, Florida, U.S.. The Occupation of Eva Mendes are Actress, model, businessswoman. The Years active of Eva Mendes are 1998 - present. The Partner(s) of Eva Mendes are Ryan Gosling (2011 - present). The Children of Eva Mendes are 2.

Better Paragraph PremiseEva Mendes is a person. The birth name of Eva Mendes is Eva de la Caridad Mendez. Eva Mendes was born on March 5, 1974 (1974-03-05) (age 44) Miami, Florida, U.S.. The occupation of Eva Mendes is Actress, model, businesswoman. The years active of Eva Mendes was on 1998 - present. The partner(s) of Eva Mendes is Ryan Gosling (2011 - present). The number of children of Eva Mendes are 2.

HypothesisEva Mendes has two children.

Result and ExplanationIn this example from \(\alpha_{2}\), the model predicts Neutral for this hypothesis with orignal premise. However, forming better sentences by adding the _"number of children are 2"_ (highlighted as green) in case of CARDINAL type for the category PERSON helps the model understand the relation and reasoning behind the children and the number two and arrive at the correct prediction of entailment.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Premise & Dev & \(\alpha_{1}\) & \(\alpha_{2}\) & \(\alpha_{3}\) \\ \hline Struc & 77.61 & 75.06 & 69.02 & 64.61 \\ + KG implicit & **79.55** & **78.66** & **72.33** & **70.44** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Accuracy on InfoTabS data for Struc representation of Tables. Here, + represents the change with respect to the previous row.

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline Para & Mod & Dev & \(\alpha_{1}\) & \(\alpha_{2}\) & \(\alpha_{3}\) \\ \hline ✓ & \(\times\) & 6.77 & 7.83 & 9.27 & 10.01 \\ × & ✓ & **10.94** & **12.55** & **14.33** & **16.05** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Correct vs Incorrect Predictions for Para model Gupta et al. (2020) and the model after the modifications (Mod).

\begin{table}
\begin{tabular}{c|c} \hline \hline Premise & Label \\ \hline Human Label (Gold) & Entailed \\ Original Premise & Neutral \\  +BPR & Entailed \\ \hline \hline \end{tabular}
\end{table}
Table 8: Prediction after BPR. Here, + represents the change with respect to the previous row.





### KG implicit

Original PremiseJanet Leigh is a person. Janet Leigh was born as Jeanette Helen Morrison (1927-07-06) July 6, 1927 Merced, California, U.S. Janet Leigh died on October 3, 2004 (2004-10-03) (aged 77) Los Angeles, California, U.S.. The resting place of Janet Leigh is Westwood Village Memorial Park Cemetery. The alma mater of Janet Leigh is University of the Pacific. The occupation of Janet Leigh are Actress, singer, dancer, author. The years active of Janet Leigh was on 1947-2004. The political party of Janet Leigh is Democratic. The spouse(s) of Janet Leigh are John Carlisle (m. 1942; annulled 1942), Stanley Reames (m. 1945; div. 1949), Tony Curtis (m. 1951; div. 1962), Robert Brandt (m. 1962). The children of Janet Leigh are Kelly Curtis, Jamie Lee Curtis.

Hypothesis AJanet Leigh's career spanned over 55 years long.

Hypothesis BJanet Leigh's career spanned under 55 years long.

Result and ExplanationIn this example from \(\alpha_{2}\), the model without implicit knowledge and the model with implicit knowledge addition predict the correct label on the Hypothesis A. However for Hypothesis B which is an example from \(\alpha_{2}\), and originally generated by replacing the word "**over**" to word "**under**" in the Hypothesis A and flipping gold label from entail to contradiction, the ealier model which is using artifacts over lexical patterns arrive to predict the original wrong label entail instead of contradiction. On adding implicit knowledge while training, the model is now able to reason rather than relying on artifacts and correctly predicts contradiction. Note, that both hypothesis A and hypothesis B require exactly same reasoning for inference i.e. they are equally hard.

### KG explicit

#### Drr

Original PremiseThe pronunciation of Fluorine are (FLOOR-een, -in, -yn) and (FLOR-een, -in, -ym). The allotropes of Fluorine is alpha, beta. The appearance of Fluorine is gas: very pale yellow , liquid: bright yellow , solid: alpha is opaque, beta is transparent. The standard atomic weight are, std(f) of Fluorine is 18.998403163(6). The atomic number (z) of Fluorine is 9. The group of Fluorine is group 17 (halogens). The period of Fluorine is period 2. The block of Fluorine is p-block. The element category of Fluorine is Reactive nonmetal. The electron configuration of Fluorine is [He] 2s 2 2p 5. The electrons per shell of Fluorine is 2, 7. The phase at stp of Fluorine is gas. The melting point of Fluorine is (F-2) 53.48 K (-219.67 \({}^{\circ}\)C, -363.41 \({}^{\circ}\)F). The boiling point of Fluorine is (F 2 ) 85.03 K (-188.11 \({}^{\circ}\)C, -306.60 \({}^{\circ}\)F). The density (at stp) of Fluorine is 1.696 g/L. The when liquid (at b.p.) of Fluorine is 1.505 g/cm 3. The triple point of Fluorine is 53.48 K, 90 kPa. The critical point of Fluorine is 144.41 K, 5.1724 MPa. The heat of vaporization of Fluorine is 6.51 kJ/mol. The molar heat capacity of Fluorine is C p : 31 J/(mol\(\cdot\)K) (at 21.1 \({}^{\circ}\)C), C v : 23 J/(mol\(\cdot\)K) (at 21.1 \({}^{\circ}\)C). The oxidation states of Fluorine is -1 (oxidizes oxygen). The electronegativity of Fluorine is Pauling scale: 3.98. Fluorine was ionization energies on 1st: 1681 kJ/mol, 2nd: 3374 kJ/mol, 3rd: 6147 kJ/mol, (more). The covalent radius of Fluorine is 64 pm. The van der waals radius of Fluorine is 135 pm. The natural occurrence of Fluorine is primordial. The thermal conductivity of Fluorine is 0.02591 W/(m\(\cdot\)K). The magnetic ordering of Fluorine is diamagnetic (-1.2x10 -4 ). The cas number of Fluorine is 7782-41-4. The naming of Fluorine is after the mineral fluorite, itself named after Latin fluo (to flow, in smelting). The discovery of Fluorine is Andre-Marie Ampere (1810). The first isolation of Fluorine is Henri Moissan (June 26, 1886). The named by of Fluorine is Humphry Davy.

Distracting Row Removal (DRR)The first isolation of Fluorine is Henri Moissan (June 26, 1886). The group of Fluorine is group 17 (halogens). The discovery of Fluorine is André-Marie Ampere (1810). Fluorine was ionization energies on 1st: 1681 kJ/mol, 2nd: 3374 kJ/mol, 3rd: 6147 kJ/mol, (more).

HypothesisFlourine was discovered in the 18th century.

\begin{table}
\begin{tabular}{c|c} \hline Premise & Label \\ \hline Human Label (Gold) & Contradiction \\ Original Premise & Entailed \\ + KG implicit & Contradiction \\ \hline \end{tabular}
\end{table}
Table 10: Prediction on Hypothesis B (from \(\alpha_{2}\)). Here, + represents the change with respect to the previous row

\begin{table}
\begin{tabular}{l l} \hline Premise & Label \\ \hline Human Label (Gold) & Entailed \\ Original Premise & Entailed \\ + KG implicit & Entailed \\ \hline \end{tabular}
\end{table}
Table 9: Prediction on Hypothesis A. Here, + represents the change with respect to the previous row

Result and ExplanationIn this example from the \(\alpha_{3}\) set, removing distracting rows (sentence except the one in green and blue) definitely helps as there are irrelevant distracting noise and also make premise paragraph long beyond BERT maximum tokenization limits. Before DRR is applied, the model predicts neutral due to a) distracting rows and b) required information i.e. relevant keys-rows highlighted as green being removed due to maximum tokenization limitation (it's second last sentence). However, after DRR, the prune information retained is only the relevant keys highlighted as green and thus the model is able to predict the correct label.

Negative ExampleIn some examples distracting row removal for DRR remove an relevant rows and hence the model failed to predict correctly on the DRR premise, as shown below:

Original PremiseEt in Arcadia ego is a painting. Et in Arcadia ego is also known as Les Bergers d'Arcadie. The artist of Et in Arcadia ego is Nicolas Poussin. The year of Et in Arcadia ego is 1637 - 1638. The medium of Et in Arcadia ego is oil on canvas. The dimensions of Et in Arcadia ego is 87 cm 120 cm (34.25 in 47.24 in). The location of Et in Arcadia ego is Muse du Louvre.

Distracting Row Removal (DRR)Et in Arcadia ego is a painting. The artist of Et in Arcadia ego is Nicolas Poussin. The medium of Et in Arcadia ego is oil on canvas. The dimensions of Et in Arcadia ego is 87 cm 120 cm (34.25 in 47.24 in).

HypothesisThe art piece Et in Arcadia ego is stored in the United Kingdom

Result and ExplanationIn this example from the Dev set, the DRR technique used removes the required key "_Location_" (highlighted in red) from the para representation. Hence, the model here predicts neutral as the information regarding where the painting is stored i.e. "_Location_" is removed in the DRR, which the model require for making the correct inference. While in original para, this information is still present and the model is able to arrive at the correct label. Another interesting observation is RoBERTa\({}_{L}\) knows Musee du Louvre is a museum in the United Kingdom, showing sign of world-knowledge.

Negative ExampleIn another negative examples distracting row removal for DRR got the relevant rows correct but still the model failed to predict correct label due to spurious correlation, as shown below:

Original PremiseIdicracy is a movie. Idicocracy was directed by Mike Judge. Idicocracy was produced by Mike Judge, Elysa Koplovitz, Michael Nelson. Idiocracy was written by Eth Cohen, Mike Judge. Idiocracy was starring Luke Wilson, Maya Rudolph, Dax Shepard. Idiocracy was music by Theodore Shapiro. The cinematography of Idiocracy was by Tim Suhrstedt. Idiocracy was edited by David Rennie. The production company of Idiocracy is Ternion. Idiocracy was distributed by 20th Century Fox. The release date of Idiocracy is September 1, 2006. The running time of Idiocracy is 84 minutes. The country of Idiocracy is United States. The language of Idiocracy is English. The budget of Idiocracy is $2-4 million. In the box office, Idiocracy made $495,303 (worldwide).

Distracting Row Removal (DRR)Idiocracy was directed by Mike Judge. Idiocracy was produced by Mike Judge, Elysa Koplovitz, Michael Nelson. Idiocracy was written by Eth Cohen, Mike Judge. Idiocracy was edited by David Rennie.

HypothesisIdicoracy was directed and written by the same person.

\begin{table}
\begin{tabular}{c|l} \hline \hline Premise & Label \\ \hline Human Label (Gold) & Contradiction \\ Original Premise & Contradiction \\  +DRR & Neutral \\ \hline \hline \end{tabular}
\end{table}
Table 11: Prediction after DRR. Here, + represents the change with respect to the previous row.

\begin{table}
\begin{tabular}{c c} \hline \hline Premise & Label \\ \hline Human Label (Gold) & Contradiction \\ Original Premise & Neutral \\  +DRR & Contradiction \\ \hline \hline \end{tabular}
\end{table}
Table 12: Prediction after DRR. Here, + represents the change with respect to the previous row.

\begin{table}
\begin{tabular}{c l} \hline \hline Premise & Label \\ \hline Human Label (Gold) & Contradiction \\ Original Premise & Contrradiction \\  +DRR & Neutral \\ \hline \hline \end{tabular}
\end{table}
Table 13: Prediction after DRR. Here, + represents the change with respect to the previous row.

 

Result and ExplanationIn this example from the Dev set, the model before DRR predicts the correct label but however on DRR, it predicts incorrect label of neutral. Despite the fact that both the relevant rows require for inference (highlighted in green) is present after DRR. This shows, that the model is looking at more keys than required in the initial case, which are eliminated in the DRR, which force the model to change it prediction. Thus, model is utilising spurious correlation from irrelevant rows to predict the label.

Original PremiseJulius Caesar was born on 12 or 13 July 100 BC Rome. Julius Caesar died on 15 March 44 BC (aged 55) Rome. The resting place of Julius Caesar is Temple of Caesar, Rome. The spouse(s) of Julius Caesar are Cornelia (84-69 BC; her death), Pompeia (67-61 BC; divorced), Calpurnia (59-44 BC; his death).

Original Premise+KG explicitJulius Caesar died on 15 March 44 BC (aged 55) Rome. **The resting place of Julius Caesar is Temple of Caesar, Rome.** Julius Caesar was born on 12 or 13 July 100 BC Rome. The spouse(s) of Julius Caesar are Cornelia (84-69 BC; her death), Pompeia (67-61 BC; divorced), Calpurnia (59-44 BC; his death). KEY: Died is defined as pass from physical life and lose all bodily attributes and functions necessary to sustain life . KEY: Resting place is defined as a cemetery or graveyard is a place where the remains of dead people are buried or otherwise inferred . KEY: Born is defined as british nuclear physicist (born in germany) honored for his contributions to quantum mechanics (1882-1970) . KEY: Spouse is defined as a spouse is a significant other in a marriage, civil union, or common-law marriage .

HypothesisJulius Caesar was buried in Rome.

\begin{tabular}{c|l} \hline \hline Model & Label \\ \hline Human Label (Gold) & Entailed \\ Original Premise & Neutral \\  + KG explicit & Entailed \\ \hline \hline \end{tabular} Table 14: Prediction after KG explicit addition. Here, + represents the change with respect to the previous row.