SlowGuess/NAACL_2021_OCR huggingface dataset:
J. Neeraja*
IIT Guwahati
jneeraja@iitg.ac.in
Vivek Gupta*
vgupta@cs.utah.edu
Reasoning about tabular information presents unique challenges to modern NLP approaches which largely rely on pre-trained contextualized embeddings of text. In this paper, we study these challenges through the problem of tabular natural language inference. We propose easy and effective modifications to how information is presented to a model for this task. We show via systematic experiments that these strategies substantially improve tabular inference performance.
Natural Language Inference (NLI) is the task of determining if a hypothesis sentence can be inferred as true, false, or undetermined given a premise sentence (Dagan et al., 2013). Contextual sentence embeddings such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), applied to large datasets such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), have led to near-human performance of NLI systems.
In this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as TabFact (Chen et al., 2019) and InfoTabS (Gupta et al., 2020). This problem is similar to standard NLI, but the premises are Wikipedia tables rather than sentences. Models similar to the best ones for the standard NLI datasets struggle with tabular inference. Using the InfoTabS dataset as an example, we present a focused study that investigates (a) the poor performance of existing models, (b) connections to information deficiency in the tabular premises, and, (c) simple yet effective mitigations for these problems.
We use the table and hypotheses in Figure 1 as a running example through this paper, and re
New York Stock Exchange
Type	Stock exchange
Location	New York City, New York, U.S.
Founded	May 17, 1792; 226 years ago
Currency	United States dollar
No. of listings	2,400
Volume	US$20.161 trillion (2011)
H1: NYSE has fewer than 3,000 stocks listed.
H2: Over 2,500 stocks are listed in the NYSE.
H3: S&P 500 stock trading volume is over $10 trillion.
Figure 1: A tabular premise example. The hypotheses H1 is entailed by it, H2 is a contradiction and H3 is neutral i.e. neither entailed nor contradictory.
fer to the left column as its keys. $^{1}$  Tabular inference is challenging for several reasons: (a) Poor table representation: The table does not explicitly state the relationship between the keys and values. (b) Missing implicit lexical knowledge due to limited training data: This affects interpreting words like 'fewer', and 'over' in H1 and H2 respectively. (c) Presence of distracting information: All keys except No. of listings are unrelated to the hypotheses H1 and H2. (d) Missing domain knowledge about keys: We need to interpret the key Volume in the financial context for this table.
In the absence of large labeled corpora, any modeling strategy needs to explicitly address these problems. In this paper, we propose effective approaches for addressing them, and show that they lead to substantial improvements in prediction quality, especially on adversarial test sets. This focused study makes the following contributions:
1. We analyse why the existing state-of-the-art BERT class models struggle on the challenging task of NLI over tabular data.
2. We propose solutions to overcome these challenges via simple modifications to inputs using existing language resources.
<sup>1</sup>Keys in the InfoTabS tables are similar to column headers in the TabFact database-style tables.
*The first two authors contributed equally to the work. The first author was a remote intern at University of Utah during the work.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2799-2809 June 6-11, 2021. ©2021 Association for Computational Linguistics
3. Through extensive experiments, we show significant improvements to model performance, especially on challenging adversarial test sets. The updated dataset, along with associated scripts, are available at https://github.com/utahnlp/knowledge_infotabs.
2 Challenges and Proposed Solutions
We examine the issues highlighted in  $\S 1$  and propose simple solutions to mitigate them below.
Better Paragraph Representation (BPR): One way to represent the premise table is to use a universal template to convert each row of the table into sentence which serves as input to a BERT-style model. Gupta et al. (2020) suggest that in a table titled t, a row with key k and value v should be converted to a sentence using the template: "The k of t are v." Despite the advantage of simplicity, the approach produces ungrammatical sentences. In our example, the template converts the Founded row to the sentence "The Founded of New York Stock Exchange are May 17, 1792; 226 years ago".
We note that keys are associated with values of specific entity types such as MONEY, DATE, CARDINAL, and BOOL, and the entire table itself has a category. Therefore, we propose type-specific templates, instead of using the universal one. In our example, the table category is Organization and the key Founded has the type DATE. A better template for this key is "t was k on v", which produces the more grammatical sentence "New York Stock Exchange was Founded on May 17, 1792; 226 years ago." Furthermore, we observe that including the table category information i.e. "New York Stock Exchange is an Organization" helps in better premise context understanding.3 Appendix A provides more such templates.
Implicit Knowledge Addition (KG implicit): Tables represent information implicitly; they do not employ connectives to link their cells. As a result, a model trained only on tables struggles to make lexical inferences about the hypothesis, such as the difference between the meanings of 'before' and 'after', and the function of negations. This is surprising, because the models have the benefit of being pre-trained on large textual corpora.
Recently, Andreas (2020) and Pruksachatkun et al. (2020) showed that we can pre-train models on specific tasks to incorporate such implicit knowledge. Eisenschlos et al. (2020) use pre-training on synthetic data to improve the performance on the TabFact dataset. Inspired by these, we first train our model on the large, diverse and human-written MultiNLI dataset. Then, we fine tune it to the InfoTabS task. Pre-training with MultiNLI data exposes the model to diverse lexical constructions. Furthermore, it increases the training data size by 433K (MultiNLI) example pairs. This makes the representation better tuned to the NLI task, thereby leading to better generalization.
Distracting Rows Removal (DRR) Not all premise table rows are necessary to reason about a given hypothesis. In our example, for the hypotheses H1 and H2, the row corresponding to the key No. of listings is sufficient to decide the label for the hypothesis. The other rows are an irrelevant distraction. Further, as a practical concern, when longer tables are encoded into sentences as described above, the resulting number of tokens is more than the input size restrictions of existing models, leading to useful rows potentially being cropped. Appendix F shows one such example on the InfoTabS. Therefore, it becomes important to prune irrelevant rows.
To identify relevant rows, we employ a simplified version of the alignment algorithm used by Yadav et al. (2019, 2020) for retrieval in reading comprehension.
First, every word in the hypothesis sentence is aligned with the most similar word in the table sentences using cosine similarity. We use fast-Text (Joulin et al., 2016; Mikolov et al., 2018) embeddings for this purpose, which preliminary experiments revealed to be better than other embeddings. Then, we rank rows by their similarity to the hypothesis, by aggregating similarity over content words in the hypothesis. Yadav et al. (2019) used inverse document frequency for weighting words, but we found that simple stop word pruning was sufficient. We took the top  $k$  rows by similarity as the pruned representative of the table for this hypothesis. The hyper-parameter  $k$  is selected by tuning on a development set. Appendix B gives more details about these design choices.
Explicit Knowledge Addition (KG explicit): We found that adding explicit information to enrich
2The construction of the template sentences based on entity type is a one-time manual step. 3This category information is provided in the InfoTabS and TabFact datasets. For other datasets, it can be inferred easily by clustering over the keys of the training tables.
keys improves a model's ability to disambiguate and understand them. We expand the pruned table premises with contextually relevant key information from existing resources such as WordNet (definitions) or Wikipedia (first sentence, usually a definition).<sup>4</sup>
To find the best expansion of a key, we use the sentential form of a row to obtain the BERT embedding (on-the-fly) for its key. We also obtain the BERT embeddings of the same key from WordNet examples (or Wikipedia sentences).<sup>5</sup> Finally, we concatenate the WordNet definition (or the Wikipedia sentence) corresponding to the highest key embedding similarity to the table. As we want the contextually relevant definition of the key, we use the BERT embeddings rather than noncontextual ones (e.g., fastText). For example, the key volume can have different meanings in various contexts. For our example, the contextually best definition is "In capital markets, volume, is the total number of a security that was traded during a given period of time." rather than the other definition "In thermodynamics, the volume of a system is an extensive parameter for describing its thermodynamic state".
3 Experiment and Analysis
Our experiments are designed to study the research question: Can today's large pre-trained models exploit the information sources described in §2 to better reason about tabular information?
Datasets Our experiments use InfoTabS, a tabular inference dataset from Gupta et al. (2020). The dataset is heterogeneous in the types of tables and keys, and relies on background knowledge and common sense. Unlike the TabFact dataset (Chen et al., 2019), it has all three inference labels, namely entailment, contradiction and neutral. Importantly, for the purpose of our evaluation, it has three test sets. In addition to the usual development set and the test set (called  $\alpha_{1}$ ), the dataset has two adversarial test sets: a contrast set  $\alpha_{2}$  that is lexically similar to  $\alpha_{1}$ , but with minimal changes in the hypotheses
and flip entail-contradict label, and a zero-shot set  $\alpha_{3}$  which has long tables from different domains with little key overlap with the training set.
Models For a fair comparison with earlier baselines, we use RoBERTa-large (RoBERTa $_L$ ) for all our experiments. We represent the premise table by converting each table row into a sentence, and then appending them into a paragraph, i.e. the Para representation of Gupta et al. (2020).
Hyperparameters Settings For the distracting row removal (+DRR) step, we have a hyperparameter  $k$ . We experimented with  $k \in \{2,3,4,5,6\}$ , by predicting on +DRR development premise on model trained on original training set (i.e. BPR), as shown in Table 1. The development accuracy increases significantly as  $k$  increases from 2 to 4 and then from 4 to 6, increases marginally (1.5% improvement). Since our goal is to remove distracting rows, we use the lowest hyperparameter with good performance i.e.  $k = 4$ .
Train	Dev	k = 2	k = 3	k = 4	k = 5	k = 6
BPR	DRR	71.72	74.83	77.50	78.50	79.00
Table 1: Dev accuracy on increasing hyperparameter  $k$  .
Table 2 shows the results of our experiments.
Premise	Dev	α1	α2	α3
Human	79.78	84.04	83.88	79.33
Para	75.55	74.88	65.55	64.94
BPR	76.42	75.29	66.50	64.26
+KG implicit	79.57	78.27	71.87	66.77
+DRR	78.77	78.13	70.90	68.98
+KG explicit	79.44	78.42	71.97	70.03
Table 2: Accuracy with the proposed modifications on the Dev and test sets. Here,  $+$  represents the change with respect to the previous row. Reported numbers are the average over three random seed runs with standard deviation of 0.33 (+KG explicit), 0.46 (+DRR), 0.61 (+KG implicit), 0.86 (BPR), over all sets. All improvements are statistically significant with  $p < {0.05}$  ,except  ${\alpha }_{1}$  for BPR representation w.r.t to Para (Original). Here the Human and Para results are taken from Gupta et al. (2020).
Usually multi-word keys are absent in WordNet, in this case we use Wikipedia. The WordNet definition of each word in the key is used if the multi-word key is absent in Wikipedia.
5We prefer using WordNet examples over definition for BERT embedding because (a) an example captures the context in which key is used, and (b) the definition may not always contain the key tokens.
$^{6}$ Appendix C has more details about hyperparameters.  ${ }^{7}$ Indeed, the original InfoTabs work points out that no more than four rows in a table are needed for any hypothesis.
BPR As shown in Table 2, with BPR, we observe that the  $\mathrm{RoBERTa}_L$  model improves performance on all dev and test sets except  $\alpha_3$ . There are two main reasons behind this poor performance on  $\alpha_3$ .
First, the zero-shot  $\alpha_{3}$  data includes unseen keys. The number of keys common to  $\alpha_{3}$  and the training set is 94, whereas for, dev,  $\alpha_{1}$  and  $\alpha_{2}$  it is 334, 312, and 273 respectively (i.e., 3-5 times more). Second, despite being represented by better sentences, due to the input size restriction of RoBERTa $_{L}$  some relevant rows are still ignored.
KG implicit We observe that implicit knowledge addition via MNLI pre-training helps the model reason and generalize better. From Table 2, we can see significant performance improvement in the dev and all three test sets.
DRR This leads to significant improvement in the  $\alpha_{3}$  set. We attribute this to two primary reasons: First,  $\alpha_{3}$  tables are longer (13.1 keys per table on average, vs. 8.8 keys on average in the others), and DRR is important to avoid automatically removing keys from the bottom of a table due to the limitations in RoBERTa $_L$  model's input size. Without these relevant rows, the model incorrectly predicts the neutral label. Second,  $\alpha_{3}$  is a zero-shot dataset and has significant proportion of unseen keys which could end up being noise for the model. The slight decrease in performance on the dev,  $\alpha_{1}$  and  $\alpha_{2}$  sets can be attributed to model utilising spurious patterns over irrelevant keys for prediction. $^{8}$  We validated this experimentally by testing the original premise trained model on the DRR test tables. Table 5 in the Appendix C shows that without pruning, the model focuses on irrelevant rows for prediction.
KG explicit With explicit contextualized knowledge about the table keys, we observe a marginal improvement in dev,  $\alpha_{1}$  test sets and a significant performance gain on the  $\alpha_{2}$  and  $\alpha_{3}$  test sets. Improvement in the  $\alpha_{3}$  set shows that adding external knowledge helps in the zero-shot setting. With  $\alpha_{2}$ , the model can not utilize spurious lexical correlations $^{9}$  due to its adversarial nature, and is forced to use the relevant keys in the premise tables, thus
adding explicit information about the key improves performance more for  $\alpha_{2}$  than  $\alpha_{1}$  or dev. Appendix F shows some qualitative examples.
We perform an ablation study as shown in table 3, where instead of doing all modification sequentially one after another  $(+)$ , we do only one modification at a time to analyze its effects.
Through our ablation study we observe that: (a) DRR improves performance on the dev,  $\alpha_{1}$ , and  $\alpha_{2}$  sets, but slightly degrades it on the  $\alpha_{3}$  set. The drop in performance on  $\alpha_{3}$  is due to spurious artifact deletion as explained in details in Appendix E. (b) KG explicit gives performance improvement in all sets. Furthermore, there is significant boost in performance of the adversarial  $\alpha_{2}$  and  $\alpha_{3}$  sets.[10] (c) Similarly, KG implicit shows significant improvement in all test sets. The large improvements on the adversarial sets  $\alpha_{2}$  and  $\alpha_{3}$  sets, suggest that the model can now reason better. Although, implicit knowledge provides most performance gain, all modifications are needed to obtain the best performance for all sets (especially on the  $\alpha_{3}$  set).[11]
Premise	Dev	α1	α2	α3
Para	75.55	74.88	65.55	64.94
DRR	76.39	75.78	67.22	64.88
KG explicit	77.16	75.38	67.88	65.50
KG implicit	79.06	78.44	71.66	67.55
Table 3: Ablation results with individual modifications.
4 Comparison with Related Work
Recently, there have been many papers which study several NLP tasks on semi-structured tabular data. These include tabular NLI and fact verification tasks such as TabFact (Chen et al., 2019), and InfoTabS (Gupta et al., 2020), various question answering and semantic parsing tasks (Pasupat and Liang, 2015; Krishnamurthy et al., 2017; Abbas et al., 2016; Sun et al., 2016; Chen et al., 2020; Lin et al., 2020, inter alia), and table-to-text generation and its evaluation (e.g., Parikh et al., 2020; Radev et al., 2020). Several, models for better representation of tables such as TAPAS (Herzig
8 Performance drop of dev and  $\alpha_{2}$  is also marginal i.e. (dev: 79.57 to 78.77,  $\alpha_{1}$ : 78.27 to 78.13,  $\alpha_{2}$ : 71.87 to 70.90), as compared to InfoTabS WMD-top3 i.e (dev: 75.5 to 72.55,  $\alpha_{1}$ : 74.88 to 70.38,  $\alpha_{2}$ : 65.44 to 62.55), here WMD-top3 performance numbers are taken from Gupta et al. (2020).
The hypothesis-only baseline for  $\alpha_{2}$  is  $48.5\%$  vs.  $\alpha_{1}$ :  $60.5\%$  and dev:  $60.5\%$  (Gupta et al., 2020)
10The KG explicit step is performed only for relevant keys (after DRR).
<sup>11</sup>We show in Appendix D, Table 6, that implicit knowledge addition to a non-sentential table representation i.e. Struc (Chen et al., 2019; Gupta et al., 2020) leads to performance improvement as well.
et al., 2020), TaBERT (Yin et al., 2020), and TabStruc (Zhang et al., 2020) were recently proposed. Yu et al. (2018, 2021) and Eisenschlos et al. (2020) study pre-training for improving tabular inference, similar to our MutliNLI pre-training.
The proposed modifications in this work are simple and intuitive. Yet, existing table reasoning papers have not studied the impact of such input modifications. Furthermore, much of the recent work focuses on building sophisticated neural models, without explicit focus on how these models (designed for raw text) adapt to the tabular data. In this work, we argue that instead of relying on the neural network to "magically" work for tabular structures, we should carefully think about the representation of semi-structured data, and the incorporation of both implicit and explicit knowledge into neural models. Our work highlights that simple pre-processing steps are important, especially for better generalization, as evident from the significant improvement in performance on adversarial test sets with the same RoBERTa models. We recommend that these pre-processing steps should be standardized across table reasoning tasks.
We introduced simple and effective modifications that rely on introducing additional knowledge to improve tabular NLI. These modifications govern what information is provided to a tabular NLI and how the given information is presented to the model. We presented a case study with the recently published InfoTabS dataset and showed that our proposed changes lead to significant improvements. Furthermore, we also carefully studied the effect of these modifications on the multiple test-sets, and why a certain modification seems to help a particular adversarial set.
We believe that our study and proposed solutions will be valuable to researchers working on question answering and generation problems involving both tabular and textual inputs, such as tabular/hybrid question answering and table-to-text generation, especially with difficult or adversarial evaluation. Looking ahead, our work can be extended to include explicit knowledge for hypothesis tokens as well. To increase robustness, we can also integrate structural constraints via data augmentation through NLI training. Moreover, we expect that structural information such as position encoding could also help better represent tables.
We thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. We also thank the support of NSF grants #1801446 (SATC) and #1822877 (Cyberlearning) and a generous gift from Verisk Inc.
Faheem Abbas, M. K. Malik, M. Rashid, and Rizwan Zafar. 2016. Wikiqa — a question answering system on wikipedia using freebase, dbpedia and infobox. 2016 Sixth International Conference on Innovative Computing Technology (INTECH), pages 185–193.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556-7566, Online. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Findings of EMNLP 2020.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. *Synthesis Lectures on Human Language Technologies*, 6(4):1-220.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Julian Eisenschlos, Syrine Krichene, and Thomas Mueller. 2020. Understanding tables with intermediate pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 281-296.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as