SlowGuess/NAACL_2021_OCR huggingface dataset:
semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516-1526.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-sql semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4870-4888.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of EMNLP.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470-1480.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247.
Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Nazneen Fatema Rajani, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871.
Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of the 25th International Conference on World Wide Web, pages 771-782.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Alignment over heterogeneous embeddings for question answering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2681-2691.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020. Unsupervised alignment-based iterative evidence retrieval for multi-hop question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4514-4525, Online. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing. International Conference of Learning Representation.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921.
Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020. Table fact verification with structure-aware transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1624-1629, Online. Association for Computational Linguistics.
A BPR Templates
Here, we are listing down some of the diverse example templates we have framed.
- For the table category Bus/Train Lines and key Disabled access with BOOL value YES, follow template: "t has  $k$  ."
Original Premise Sentence "The Disabled access of Tukwila International Boulevard Station are Yes."
BPR Sentence "Tukwila International Boulevard Station has Disabled access."
- For the table category Movie and key Box office with MONEY type, follow template: "In the  $k$ ,  $t$  made  $\nu$ ."
Original Premise Sentence "The Box office of Brokeback Mountain are $178.1 million."
BPR Sentence "In the Box office, Brokeback Mountain made $178.1 million."
- For the table category City and key Total with CARDINAL type, follow template: "The  $k$  area of  $t$  is  $\nu$ ."
Original Premise Sentence "The Total of Cusco are 435,114."
BPR Sentence "The Total area of Cusco is 435,114."
- For the table category Painting and key Also known as, follow template: "The  $k$  area of  $t$  is  $v$ ."
Original Premise Sentence "The Also known as of Et in Arcadia ego are Les Bergers d'Arcadie."
BPR Sentence "Et in Arcadia ego is Also known as Les Bergers d'Arcadie."
- For the table category Person and key Died with DATE type, follow template: "tk on v."
Original Premise Sentence "The Died of Jesse Ramsden are November 1800 (1800-11-05) (aged 65) Brighton, Sussex."
BPR Sentence "Jesse Ramsden Died on 5 November 1800 (1800-11-05) (aged 65) Brighton, Sussex."
B DRR: fastText and Binary weighting
fastText: For word representation, (Yadav et al., 2019) have used BERT and Glove embeddings. In our case, we prefer to use fastText word embeddings over Glove because fastText embedding uses sub-word information which helps in capturing different variations of the context words. Furthermore, fastText embeddings is also as better choice than BERT for our task because 1. Firstly, we are embedding single sentential form of diverse rows instead of longer context similar paragraphs, 2. Secondly, all words (especially keys) of the rows across all the tables are used only in one context, whereas BERT is useful when same word is used with different contexts across paragraphs, 3. Thirdly, in all tables, the number sentences to select from is bounded by maximum rows in the table, which is a small number (8.8 in train, dev,  $\alpha_{1}$ ,  $\alpha_{2}$  and 13.1 in  $\alpha_{3}$ ), and 4. Lastly, using fastText is much faster to compute than BERT for obtaining embeddings.
Binary weighting: Since, we are embedding single sentential form of diverse rows instead of longer context related paragraphs, we found that using binary weighting 0 for stop words and 1 for others is more effective than theidf weighting, which is useful only for longer paragraph context with several lexical terms.
C Hyperparameters  $k$  vs test-sets accuracy
We also trained a model both train and tested on the DRR table premise for increasing values of the hyper parameter  $k$ , as shown in Table 1. We also test the model trained on the entire para on pruned para with increasing value of hyperparameters  $k \in \{2,3,4,5,6\}$  for the test sets  $\alpha_{1}, \alpha_{2}$ , and  $\alpha_{3}$ . In all cases, except  $\alpha_{3}$ , the performance with larger  $k$  is better. The increase in performance, even with  $k > 4$ , shows that the model is using more then required keys for prediction. Thus, the model is utilising the spurious pattern in irrelevant rows for the prediction.
Train	Dev	k=2	k=3	k=4	k=5	k=6
+DRR	+DRR	77.61	77.94	78.16	78.38	79.00
BPR	+DRR	71.72	74.83	77.50	78.50	79.00
Table 4: Dev accuracy with increasing hyper parameter  $k$  trained with both BPR and +DRR table.
k	α1	α2	α3
2	71.44	67.33	64.83
3	75.05	69.33	67.33
4	77.72	69.83	68.22
5	77.77	70.28	69.28
6	77.77	70.77	69.22
D TabFact Representation Experiment
Table 6 implicit knowledge addition effect on nonpara Struc representation i.e. a key value linearize representation as "key k : value v", rows separated by semicolon ";" (Gupta et al., 2020; Chen et al., 2019). Here too the implicit knowledge addition leads to improvement in performance on all the sets.
Table 5: Accuracy of model trained with original table but tested with DRR table with increasing hyper parameter  $k$  on all test sets.
Premise	Dev	α1	α2	α3
Struc	77.61	75.06	69.02	64.61
+ KG implicit	79.55	78.66	72.33	70.44
E Artifacts and Model Predictions
In Table 7 we show percentage of example which were corrected after modification and vice versa. Surprisingly, there is a small percentage of examples which are predicted correctly earlier with original premise (Para) but predicted wrongly after all the modifications (Mod), although such examples are much lesser than opposite case. We suspect that earlier model was also relying on spurious pattern (artifacts) for correct prediction on these examples earlier, which are now corrupted after the proposed modifications. Hence, the new model struggle to predict correctly on such examples.
Table 6: Accuracy on InfoTabS data for Struc representation of Tables. Here, + represents the change with respect to the previous row.
Para	Mod	Dev	α1	α2	α3
✓	×	6.77	7.83	9.27	10.01
×	✓	10.94	12.55	14.33	16.05
In the next section F, we also show qualitative examples, where modification helps model predict
correctly. We also provide some examples via distracting row removal modification, where model fails after modification.
F Qualitative Examples
In this section, we provide examples where model is able to predict well after the proposed modifications. We also provide some examples, where model struggles to make the correct prediction after distracting row removal (DRR) modification.
F.1 BPR
Original Premise The Birth name of Eva Mendes are Eva de la Caridad Méndez. Eva Mendes was Born on March 5, 1974 (1974-03-05) (age 44) Miami, Florida, U.S.. The Occupation of Eva Mendes are Actress, model, businesswoman. The Years active of Eva Mendes are 1998 - present. The Partner(s) of Eva Mendes are Ryan Gosling (2011 - present). The Children of Eva Mendes are 2.
Better Paragraph Premise Eva Mendes is a person. The birth name of Eva Mendes is Eva de la Caridad Méndez. Eva Mendes was born on March 5, 1974 (1974-03-05) (age 44) Miami, Florida, U.S.. The occupation of Eva Mendes is Actress, model, businesswoman. The years active of Eva Mendes was on 1998 - present. The partner(s) of Eva Mendes is Ryan Gosling (2011 - present). The number of children of Eva Mendes are 2.
Hypothesis Eva Mendes has two children.
Table 7: Correct vs Incorrect Predictions for Para model (Gupta et al., 2020) and the model after the modifications (Mod).
Premise	Label
Human Label (Gold)	Entailed
Original Premise	Neutral
+BPR	Entailed
Table 8: Prediction after BPR. Here, + represents the change with respect to the previous row.
Result and Explanation In this example from  $\alpha_{2}$ , the model predicts Neutral for this hypothesis with original premise. However, forming better sentences by adding the "number of children are 2" (highlighted as green) in case of CARDINAL type for the category PERSON helps the model understand the relation and reasoning behind the children and the number two and arrive at the correct prediction of entailment.
F.2 KG implicit
Original Premise Janet Leigh is a person. Janet Leigh was born as Jeanette Helen Morrison (1927-07-06) July 6, 1927 Merced, California, U.S. Janet Leigh died on October 3, 2004 (2004-10-03) (aged 77) Los Angeles, California, U.S.. The resting place of Janet Leigh is Westwood Village Memorial Park Cemetery. The alma mater of Janet Leigh is University of the Pacific. The occupation of Janet Leigh are Actress, singer, dancer, author. The years active of Janet Leigh was on 1947-2004. The political party of Janet Leigh is Democratic. The spouse(s) of Janet Leigh are John Carlisle (m. 1942; annulled 1942), Stanley Reames (m. 1945; div. 1949), Tony Curtis (m. 1951; div. 1962), Robert Brandt (m. 1962). The children of Janet Leigh are Kelly Curtis, Jamie Lee Curtis.
Hypothesis A Janet Leigh's career spanned over 55 years long.
Hypothesis B Janet Leigh's career spanned under 55 years long.
Premise	Label
Human Label (Gold)	Entailed
Original Premise	Entailed
+ KG implicit	Entailed
Table 9: Prediction on Hypothesis A. Here,  $^+$  represents the change with respect to the previous row
Premise	Label
Human Label (Gold)	Contradiction
Original Premise	Entailed
+ KG implicit	Contradiction
Table 10: Prediction on Hypothesis B (from  $\alpha_{2}$ ). Here,  $+$  represents the change with respect to the previous row
Result and Explanation In this example from  $\alpha_{2}$ , the model without implicit knowledge and the model with implicit knowledge addition predict the correct label on the Hypothesis A. However for Hypothesis B which is an example from  $\alpha_{2}$ , and originally generated by replacing the word "over" to word "under" in the Hypothesis A and flipping gold label from entail to contradiction, the earlier model which is using artifacts over lexical patterns arrive to predict the original wrong label entail instead of contradiction. On adding implicit knowledge while training, the model is now able to reason rather than relying on artifacts and correctly predicts contradiction. Note, that both hypothesis A
and hypothesis B require exactly same reasoning for inference i.e. they are equally hard.
F.3 KG explicit
F.4 DRR
Original Premise The pronunciation of Fluorine are (FLOOR-een, -in, -yn) and (FLOR-een, -in, -yn). The allotropes of Fluorine is alpha, beta. The appearance of Fluorine is gas: very pale yellow, liquid: bright yellow, solid: alpha is opaque, beta is transparent. The standard atomic weight are, std(f) of Fluorine is 18.998403163(6). The atomic number (z) of Fluorine is 9. The group of Fluorine is group 17 (halogens). The period of Fluorine is period 2. The block of Fluorine is p-block. The element category of Fluorine is Reactive nonmetal. The electron configuration of Fluorine is [He] 2s 2 2p 5. The electrons per shell of Fluorine is 2, 7. The phase at stp of Fluorine is gas. The melting point of Fluorine is (F-2)  $53.48\mathrm{K}$  (-219.67 °C, -363.41 °F). The boiling point of Fluorine is (F 2 ) 85.03 K (-188.11 °C, -306.60 °F). The density (at stp) of Fluorine is 1.696 g/L. The when liquid (at b.p.) of Fluorine is 1.505 g/cm 3. The triple point of Fluorine is 53.48 K, 90 kPa. The critical point of Fluorine is 144.41 K, 5.1724 MPa. The heat of vaporization of Fluorine is 6.51 kJ/mol. The molar heat capacity of Fluorine is C p : 31 J/(mol·K) (at 21.1 °C), C v : 23 J/(mol·K) (at 21.1 °C). The oxidation states of Fluorine is -1 (oxidizes oxygen). The electronegativity of Fluorine is Pauling scale: 3.98. Fluorine was ionization energies on 1st: 1681 kJ/mol, 2nd: 3374 kJ/mol, 3rd: 6147 kJ/mol, (more). The covalent radius of Fluorine is 64 pm. The van der waals radius of Fluorine is 135 pm. The natural occurrence of Fluorine is primordial. The thermal conductivity of Fluorine is 0.02591 W/(m·K). The magnetic ordering of Fluorine is diamagnetic (-1.2×10  $-4$ ). The cas number of Fluorine is 7782-41-4. The naming of Fluorine is after the mineral fluorite, itself named after Latin fluo (to flow, in smelting). The discovery of Fluorine is André-Marie Ampère (1810). The first isolation of Fluorine is Henri Moissan (June 26, 1886). The named by of Fluorine is Humphry Davy.
Distracting Row Removal (DRR) The first isolation of Fluorine is Henri Moissan (June 26, 1886). The group of Fluorine is group 17 (halogens). The discovery of Fluorine is André-Marie Ampère (1810). Fluorine was ionization energies on 1st:  $1681\mathrm{kJ / mol}$ , 2nd:  $3374\mathrm{kJ / mol}$ , 3rd:  $6147\mathrm{kJ / mol}$ , (more).
Hypothesis Flourine was discovered in the 18th century.
Premise	Label
Human Label (Gold)	Contradiction
Original Premise	Neutral
+DRR	Contradiction
Result and Explanation In this example from the  $\alpha_{3}$  set, removing distracting rows (sentence except the one in green and blue) definitely helps as there are irrelevant distracting noise and also make premise paragraph long beyond BERT maximum tokenization limits. Before DRR is applied, the model predicts neutral due to a) distracting rows and b) required information i.e. relevant keysrows highlighted as green being removed due to maximum tokenization limitation (it's second last sentence). However, after DRR, the prune information retained is only the relevant keys highlighted as green and thus the model is able to predict the correct label.
Negative Example In some examples distracting row removal for DRR remove an relevant rows and hence the model failed to predict correctly on the DRR premise, as shown below:
Original Premise Et in Arcadia ego is a painting. Et in Arcadia ego is also known as Les Bergers d'Arcadie. The artist of Et in Arcadia ego is Nicolas Poussin. The year of Et in Arcadia ego is 1637 - 1638. The medium of Et in Arcadia ego is oil on canvas. The dimensions of Et in Arcadia ego is 87 cm 120 cm (34.25 in 47.24 in). The location of Et in Arcadia ego is Musee du Louvre.
Distracting Row Removal (DRR) Et in Arcadia ego is a painting. The artist of Et in Arcadia ego is Nicolas Poussin. The medium of Et in Arcadia ego is oil on canvas. The dimensions of Et in Arcadia ego is  $87~\mathrm{cm}$ $120~\mathrm{cm}$  (34.25 in 47.24 in).
Hypothesis The art piece Et in Arcadia ego is stored in the United Kingdom
Table 11: Prediction after DRR. Here,  $+$  represents the change with respect to the previous row.
Premise	Label
Human Label (Gold)	Contradiction
Original Premise	Contradiction
+DRR	Neutral
Result and Explanation In this example from the Dev set, the DRR technique used removes the required key "Location" (highlighted in red) from the para representation. Hence, the model here predicts neutral as the information regarding where the painting is stored i.e. "Location" is removed in the DRR, which the model require for making the correct inference. While in original para, this information is still present and the model is able to arrive at the correct label. Another interesting observation is RoBERTa $_L$  knows Musee du Louvre is a museum in the United Kingdom, showing sign of world-knowledge.
Negative Example In another negative examples distracting row removal for DRR got the relevant rows correct but still the model failed to predict correct label due to spurious correlation, as shown below:
Original Premise Idiocracy is a movie. Idiocracy was directed by Mike Judge. Idiocracy was produced by Mike Judge, Elysa Koplovitz, Michael Nelson. Idiocracy was written by Etan Cohen, Mike Judge. Idiocracy was starring Luke Wilson, Maya Rudolph, Dax Shepard. Idiocracy was music by Theodore Shapiro. The cinematography of Idiocracy was by Tim Suhrstedt. Idiocracy was edited by David Rennie. The production company of Idiocracy is Ternion. Idiocracy was distributed by 20th Century Fox. The release date of Idiocracy is September 1, 2006. The running time of Idiocracy is 84 minutes. The country of Idiocracy is United States. The language of Idiocracy is English. The budget of Idiocracy is $2-4 million. In the box office, Idiocracy made $495,303 (worldwide).
Distracting Row Removal (DRR) Idiocracy was directed by Mike Judge. Idiocracy was produced by Mike Judge, Elysa Koplovitz, Michael Nelson. Idiocracy was written by Etan Cohen, Mike Judge. Idiocracy was edited by David Rennie.
Hypothesis Idiocracy was directed and written by the same person.
Table 12: Prediction after DRR. Here,  $+$  represents the change with respect to the previous row.
Premise	Label
Human Label (Gold)	Entailed
Original Premise	Entailed
+DRR	Neutral
Table 13: Prediction after DRR. Here,  $+$  represents the change with respect to the previous row.
Result and Explanation In this example from the Dev set, the model before DRR predicts the correct label but however on DRR, it predicts incorrect label of neutral. Despite the fact that both the relevant rows require for inference (highlighted in green) is present after DRR. This shows, that the model is looking at more keys than required in the initial case, which are eliminated in the DRR, which force the model to change its prediction. Thus, model is utilising spurious correlation from irrelevant rows to predict the label.
Original Premise Julius Caesar was born on 12 or 13 July 100 BC Rome. Julius Caesar died on 15 March 44 BC (aged 55) Rome. The resting place of Julius Caesar is Temple of Caesar, Rome. The spouse(s) of Julius Caesar are Cornelia (84-69 BC; her death), Pompeia (67-61 BC; divorced), Calpurnia (59-44 BC; his death).
Original Premise + KG explicit Julius Caesar died on 15 March 44 BC (aged 55) Rome. The resting place of Julius Caesar is Temple of Caesar, Rome. Julius Caesar was born on 12 or 13 July 100 BC Rome. The spouse(s) of Julius Caesar are Cornelia (84-69 BC; her death), Pompeia (67-61 BC; divorced), Calpurnia (59-44 BC; his death). KEY: Died is defined as pass from physical life and lose all bodily attributes and functions necessary to sustain life. KEY: Resting place is defined as a cemetery or graveyard is a place where the remains of dead people are buried or otherwise interred. KEY: Born is defined as british nuclear physicist (born in germany) honored for his contributions to quantum mechanics (1882-1970). KEY: Spouse is defined as a spouse is a significant other in a marriage, civil union, or common-law marriage.
Hypothesis Julius Caesar was buried in Rome.
Model	Label
Human Label (Gold)	Entailed
Original Premise	Neutral
+ KG explicit	Entailed
Table 14: Prediction after KG explicit addition. Here, + represents the change with respect to the previous row.
Result and Explanation In this example from  $\alpha_{2}$ , the model without explicit knowledge predicts neutral for the hypothesis as it is not able to infer that resting place is where people are buried, so it predicts neutral as it implicitly lack buried key understanding. On explicit KG addition (highlighted as blue+ green), we add the definition of resting place to be the place where remains of the dead
are buried (highlighted as green). Now the model uses this extra information (highlighted as green) plus the original key related to death (highlighted in bold) to correctly infer that the statement Caesar is buried in Rome is entailed.