SlowGuess/NAACL_2022_OCR huggingface dataset:
mBERT's performance gets boosted in EnTranslate-Test as mBERT is pre-trained on a significant amount of English data which makes it extremely competitive in modeling English NLU tasks. MuRIL performs similarly even though it is trained on lesser amount of English data. This could be due to Indic artifacts like sentence structure and inadequately transliterated named entities being present in the back-translated sentences which MuRIL has been trained to handle better than mBERT.
5.2 Across Languages
We observe that the models perform best on Hindi (hi) and Bengali. This is expected as they are high resource languages in the Indic context. Additionally, as explained in §5.1, we note that pre-training or fine-tuning on Bengali aids the performance on Assamese due to their high degree of relatedness. Table 3 shows the measure of agreement across the languages. We note that almost all languages agree on  $55\%$  of the predictions on the dev set and the  $\alpha_{1}$  test set. This reduces to  $38\%$  on the  $\alpha_{3}$  test set. This indicates that for a majority of examples from the non-adversarial test sets, MuRIL performs uniformly across languages. However, its performance across languages starts varying more on the adversarial test sets ( $\alpha_{2}$  and  $\alpha_{3}$ ).
5.3 Train-Eval Strategies
Translate-Train's results show that the multilingual models converge and perform adequately when fine-tuned on EI-INFO TABS. Moreover, when fine-tuned along with English data - as described
Figure 2: Here, we compare human benchmarks (H), predictions of the best performing model trained on INFOTABS from (Neeraja et al., 2021) (I) and predictions of our best performing model (E), MuRIL (Multilingual-Train), on the examples annotated with reasoning category in the dev split for Oriya (left) and Hindi (right).
in Bilingual-Train - mBERT and IndicBERT perform marginally better while MuRIL doesn't report a change in performance. MuRIL, when fine-tuned on all languages as described in Multilingual-Train, performs best on EI-INFOTABS and forms the benchmark for this task. mBERT and IndicBERT, however, perform worse on Multilingual-Train when compared to Bilingual-Train. This indicates that these models fail to generalise their reasoning ability across all languages and aren't as multilingually robust as MuRIL. The results on Translate-Test are the lowest across all train/eval strategies which indicates a poor Zero-shot Cross-Linguial Transfer from INFOTABS to EI-INFOTABS. However, the performance of MuRIL on Translate-Test is comparable with its performance on Translate-Train unlike mBERT and IndicBERT. This indicates that MuRIL can generalize well across English and Indic languages which are linguistically distinct.
Translate-Train-X and Bilingual-Train-X evaluate the average Cross-Linguial Transfer performance of the models trained in TranslateTrain and Bilingual-Train. We observe higher performance in Bilingual-Train-X over Translate-Train-X which indicates that addition of English training data aids the Cross Lingual Transfer from one Indic language to another. Moreover, the average performance of MuRIL on Bilingual-Train-X is comparable to that on Translate-Train which suggests that MuRIL robustly generalises across Indic languages. Both, Bilingual-Train-X and Translate-Train-X perform better than Translate-Test due to high
Figure 3: Consistency Matrix which measures the deviation of our best performing model, MuRIL (Multilingual-Train)'s predictions on the  $\alpha_{1}$  test set for Hindi as compared to that of RoBERTaLARGE on the  $\alpha_{1}$  test set of INFOTABS.
language relatedness among Indic languages when compared with English. The results on EnTranslate-Test are extremely promising for both MuRIL and mBERT. Their performance is very close to that of the best performing model, MuRIL, on Translate-Train. This indicates that back-translation doesn't lead to a significant loss in information required for the bTNLI task.
5.4 Tabular Reasoning Categories
We conduct a fine-grained analysis on how our best model, MuRIL (Multilingual-Train), performs on various reasoning categories. We present the results in Figure 2 for Hindi and Odiya. We observe that MuRIL performs similarly on EIINFOTABS as RoBERTaLARGE does on INFOTABS for entity type, named entity, negation, numerical, quantification and simple lookup reasoning types. Additionally, MuRIL performs better for the co-reference resolution reasoning type. This is broadly
Figure 4: Confusion Matrix for the predictions of our best performing model, MuRIL (Multilingual-Train), on the Hindi  $\alpha_{1}$  set from EI-INFO TABS.
observed across all the Indic languages. Both RoBERTa $_{\text{LARGE}}$  and MuRIL perform poorly for knowledge and common sense, multi-row, co-reference, and temporal reasoning types.
5.5 Across Adversarial Test Splits
The results for the other evaluation sets  $\alpha_{2}$  and  $\alpha_{3}$  are provided in Appendix §E. Across all the experiments, we note that the fine-tuned models perform best on  $\alpha_{1}$ , followed by  $\alpha_{2}$  and  $\alpha_{3}$  respectively. Moreover, we note that on most baselines, the average performance of a fine-tuned model drops by roughly  $10\%$  when tested on  $\alpha_{2}$  or  $\alpha_{3}$ . This is similar to the observations reported on INFOTABS (Neeraja et al., 2021) and presented in Table 4. Low performance of the multilingual models on the  $\alpha_{2}$  test set of EI-INFOTABS indicates that (a.) multilingual models learn shallow lexical features to make inferences on EI-INFOTABS just like the monolingual models do on INFOTABS, (b.) and IndicTrans carefully captures the lexical adversity in the  $\alpha_{2}$  test set of INFOTABS. This commends the ability of IndicTrans to handle lexical nuances. Low performance on  $\alpha_{3}$  test set of EI-INFOTABS suggests that the multilingual models learn categorical features and perform adversely when evaluated on unseen category.
5.6 EI-INFOTABS v/s INFOTABS
Table 4 reports the human benchmarks and the baselines with the BPR linearization algorithm on each validation set in INFOTABS. We observe that the baselines on EI-INFOTABS are within an absolute margin of  $10\%$  when compared to those on INFOTABS. This suggests that EI-INFOTABS is more challenging than INFOTABS which was
expected due to the presence of (a.) bilinguality within the premise-hypothesis pair, and (b.) the low resource nature of Indic languages.
Figure 3 reports the consistency of predictions of MuRIL on the  $\alpha_{1}$  test set of Hindi EIINFOTABS when compared against that of RoBERTaLARGE on the  $\alpha_{1}$  test set of INFOTABS. We observe that MuRIL behaves noticeably different than RoBERTaLARGE. MuRIL disagrees with RoBERTaLARGE on  $47\%$  of examples with the Contradiction and Entailment labels. However, for Neutral labels, it only disagrees on around  $36\%$  of the examples. Moreover, from our discussion in §5.4, we observe that MuRIL outperforms RoBERTaLARGE on certain reasoning categories.
Model (Rep)	Dev	α1	α2	α3
BERTB (BPR)	63.00	63.54	52.57	48.17
RoBERTaB (TabFact)	68.06	66.7	56.87	55.26
RoBERTAL (BPR)	76.42	75.29	66.50	64.26
RoBERTAL (TabFact)	77.61	75.06	69.02	64.61
Human	79.78	84.04	83.88	79.33
Table 4: The human benchmarks and several baselines on evaluation set of INFO TABS as reported in Gupta et al. (2020) (TabFact) and Neeraja et al. (2021) (BPR). Here subscript  $\mathrm{X}_{\mathrm{L}}$  and  $\mathrm{X}_{\mathrm{B}}$  represent  $\mathrm{X}$  model L: Large and B: Base versions respectively.
However, the models fine-tuned on EI-INFOTABS broadly mimic the performance of RoBERTaLARGE on INFOTABS. Figure 4 presents the confusion matrix of MuRIL's predictions on the  $\alpha_{1}$  test set of Hindi. We observe a similar distribution across all Indic languages. As noted in Gupta et al. (2020), MuRIL also tends to predict Neutral hypotheses with the highest confidence as they mostly contain out of table or subjective information terms. Moreover, both models confuse Entailment with Contradiction inference label and vice-versa. We observe that the model predictions on EI-INFOTABS is similar to RoBERTaLARGE predictions on INFOTABS.
EI-INFOTABS is the first Tabular NLI dataset in the Indic context which enables preliminary studies in this field. Moreover, it introduces bilinguality for fact verification scenarios which is of huge significance in low resource contexts. It motivates the development of cross-lingual reasoning models, and helps in evaluation of robustness of multilingual models. For instance, our experiments on EI-INFOTABS clearly indicate
that MuRIL is a significantly more robust multilingual model when compared to mBERT as it is able to generalize its reasoning ability across all Indic languages.
Although, we explain how machine translation doesn't affect the semantics of the hypotheses, it does come with a few challenges. We identified a few instances wherein the IndicTrans model translates named entities, instead of transliterating them. This is observed only, but not always, when a named entity has an English dictionary word in it. For instance, "Death Proof", name of a movie, gets translated and not transliterated in two out of nine hypotheses containing the phrase. This is mostly observed in the Movies category. However, this doesn't affect our reasoning models and they perform on par on this category when compared with RoBERTaLarge's performance on INFOCTS. This is so because such translations when shallow parsed indicate that the translated entity still acts as the Noun Phrase in the sentence. This helps the translation, though technically imperfect, retain the intended semantic structure.
Tabular Reasoning. Tabular NLI has been of keen interest recently. Datasets like TabFact (Chen et al., 2020b), INFOTABS (Gupta et al., 2020) were the first resources on TNLI and they enabled a fine-grained examination of the task. Beyond NLI, there has been a thorough examination of various other NLP tasks on semi-structured data. For instance, question answering (Abbas et al., 2016; Chen et al., 2020c; Zayats et al., 2021; Oguz et al., 2020; Chen et al., 2021, and others), semantic parsing and retrieval (Krishnamurthy et al., 2017; Sun et al., 2016; Pasupat and Liang, 2015; Lin et al., 2020, and others), tabular probing (Gupta et al., 2021), generative tasks including table-to-text (Parikh et al., 2020; Nan et al., 2021; Yoran et al., 2021; Chen et al., 2020a,d, and others). Other works have explored creating task-independent representations for Wikipedia infoboxes (Herzig et al., 2020; Yin et al., 2020; Zhang et al., 2020; Iida et al., 2021; Pramanick and Bhattacharya, 2021; Glass et al., 2021, and others), and boosting tabular reasoning by pre-training and external knowledge incorporation (Neeraja et al., 2021; Varun et al., 2022, and others).
Multilingual Models. Multilingual, and specifically Cross-Linguual transfer (Deshpande
et al., 2021; Patil et al., 2022, and other), has been widely discussed in the context of low resource languages. Several datasets (Conneau et al., 2018; Yang et al., 2019; Ponti et al., 2020; Artetxe et al., 2020; Nivre et al., 2016; Lewis et al., 2021, and others), benchmarks and leaderboards (Hu et al., 2020; Liang et al., 2020; Ruder et al., 2021; Khanuja et al., 2021b, and others), and evaluation frameworks (Tarunesh et al., 2021; K et al., 2021; Srinivasan et al., 2021) have emerged which focus entirely on evaluation of multilingual NLU. Further, multilingual language models have been developed for (a.) Natural Language Understanding (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Chi et al., 2021; Chung et al., 2021, and others), (b.) and Natural Language Generation (Xue et al., 2021; Fan et al., 2021, and others).
Indic Resources. Indic NLP, recently, has seen a recent surge in the number of datasets (Ramesh et al., 2022; Roark et al., 2020; Haddow and Kirefu, 2020a; Abadji et al., 2022; Kolluru et al., 2021, and others), multilingual models (Dabre et al., 2021; Kakwani et al., 2020; Khanuja et al., 2021a, and others), toolkits (Arora, 2020; Bhat et al., 2015; Jain et al., 2020, and others), translation systems (Ramesh et al., 2022), and dedicated benchmarks for evaluation (Kakwani et al., 2020; Krishna et al., 2021). This has enabled the Indian NLP research community to construct competent models for a variety of challenging NLP tasks.
We motivate and introduce the bilingual tabular NLI for fact verification tasks, and release EI-InfOTABS- a first of its kind tabular NLI dataset for making inferences in 11 Indic languages over English tabular data. Our robust quality estimation experiments show that the machine translated datasets closely preserve the semantics of the source and are fluent. We show that pre-trained multilingual models find this task challenging, however, still perform close to the benchmarks on InfOTABs with Translate-test and Translate-train providing good performance. The analysis also shows the similarity of inference capabilities across languages. The dataset offers immense potential as it opens up avenues in (a) multilingual tabular NLI, (b) bilingual claim verification, (c) and evaluation of multilingual models.
9 Ethical Considerations
In terms of demographic and socioeconomic characteristics, we attempted to establish a balanced, bias-free dataset. The EI-INFOTABS dataset is derived from the INFOTABS dataset, which is devoid of bias. The only possible source of prejudice can be the translation pipeline. Our qualitative analysis indicates that translation quality is reasonably good and there aren't any observable biases like gender in the translation. The dataset is intended and useful for studying language model representations in a cross-lingual and structured data setting. The paper points out that low-resource languages can benefit from reasoning over structured data in other languages. This is a relatively new research topic and further work will help understand limitations as well as uncover new directions. Hence, we recommend the use of this dataset at this point exclusively for scholarly, non-commercial purposes.
We thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. The authors also thank Manila Devaraj, Arka Singha, Anagh Chattopadhyay, Maitrey Mehta, Souvik Banerjee, Rupambara Padhi, Jayant Duneja, Nithila Prakash, Tanvi Kamble, Anirudh Palutla for helping with annotations for quality estimation. Additionally, we appreciate the inputs provided by Vivek Srikumar and Ellen Roloff. Vivek Gupta acknowledges support from Bloomberg's Data Science Ph.D. Fellowship.
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoit Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv eprints, page arXiv:2201.06642.
Faheem Abbas, M. K. Malik, M. Rashid, and Rizwan Zafar. 2016. Wikiqa — a question answering system on wikipedia using freebase, dbpedia and infobox. 2016 Sixth International Conference on Innovative Computing Technology (INTECH), pages 185–193.
Eytan Adar, Michael Skinner, and Daniel S Weld. 2009. Information arbitrage across multi-lingual wikipedia. In Proceedings of the Second ACM International
Conference on Web Search and Data Mining, pages 94-103.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497-511, San Diego, California. Association for Computational Linguistics.
Gaurav Arora. 2020. iNLTK: Natural language toolkit for indic languages. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 66-71, Online. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Patti Bao, Brent Hecht, Samuel Carton, Mahmood Quaderi, Michael Horn, and Darren Gerge. 2012. Omnipedia: bridging the wikipedia language gap. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1075-1084.
Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tammewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015. Iiit-h system submission for fire2014 shared task on transliterated search. In Proceedings of the Forum for Information Retrieval Evaluation, FIRE '14, pages 48-53, New York, NY, USA. ACM.
Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In International Conference on Learning Representations.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942, Online. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020c. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Findings of EMNLP 2020.
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020d. Logic2Text: High-fidelity natural language generation from logical forms. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2096–2111, Online. Association for Computational Linguistics.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. 2021. Xlm-e: Cross-lingual language model pre-training via electra. CoRR, abs/2106.16138.
Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059-7069.
Alexis CONNEAU and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, and Pratyush Kumar. 2021. Indicbart: A pre-trained model for natural language generation of indic languages.
Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2021. When is bert multilingual? isolating crucial ingredients for cross-lingual transfer. arXiv preprint arXiv:2110.14782.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short
Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1-48.
Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1212-1224, Online. Association for Computational Linguistics.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33-41, Sofia, Bulgaria. Association for Computational Linguistics.
Vivek Gupta, Riyadh A Bhat, Atreya Ghosal, Manish Srivastava, Maneesh Singh, and Vivek Srikumar. 2021. Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning. arXiv preprint arXiv:2108.00578.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. arXiv preprint arXiv:1902.01382.
Barry Haddow and Faheem Kirefu. 2020a. Pmindia - a collection of parallel corpora of languages of india.
Barry Haddow and Faheem Kirefu. 2020b. Pmindia - A collection of parallel corpora of languages of india. CoRR, abs/2001.09907.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.