SlowGuess/NAACL_2022_OCR huggingface dataset:
Chaitanya Agarwal<sup>1*</sup>, Vivek Gupta<sup>2*</sup>, Anoop Kunchukuttan<sup>3,4</sup>, Manish Shrivastava<sup>1</sup>
$^{1}$ LTRC, IIIT Hyderabad;  $^{2}$ University of Utah;  $^{3}$ AI4Bharat;  $^{4}$ Microsoft
chaitanya.agarwal@research.iit.ac.in; vgupta@cs.utah.edu;
ankunchu@microsoft.com; m.shrivastava@iit.ac.in
Existing research on Tabular Natural Language Inference (TNLI) exclusively examines the task in a monolingual setting where the tabular premise and hypothesis are in the same language. However, due to the uneven distribution of text resources on the web across languages, it is common to have the tabular premise in a high resource language and the hypothesis in a low resource language. As a result, we present the challenging task of bilingual Tabular Natural Language Inference (bTNLI), in which the tabular premise and a hypothesis over it are in two separate languages. We construct EI-INFOTABS: an English-Indic bTNLI dataset by translating the textual hypotheses of the English TNLI dataset INFOTABS into eleven major Indian languages. We thoroughly investigate how pretrained multilingual models learn and perform on EI-INFOTABS. Our study shows that the performance on bTNLI can be close to its monolingual counterpart, with translate-train, translate-test and unified-train being strongly competitive baselines.
Tabular Natural Language Inference (TNLI) is the task of classifying whether a textual hypothesis is an entailment, contradiction or a neutral extension of the given tabular premise. The task requires a broad range of reasoning abilities, including but not limited to the ability to make lexical, spatiotemporal, and semantic deductions. Recently published datasets, TabFact (Chen et al., 2020b) and INFO TABS (Gupta et al., 2020), have enabled the examination of the TNLI task. Moreover, sophisticated models based on deep contextual embeddings like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), etc. trained under
Joe Strummer
Birth Name	John Graham Mellor
Born	1952-08-21 Ankara, Turkey
Died	2002-12-22 Broomfield, Somerset, England
Genres	Punk Rock, Post Punk
Occupation(s)	Musician, Songwriter, Radio Host, Actor
Instruments	Vocals, Guitar, Piano
Years Active	1970-2002
Labels	CBS, Sony, Hellcat, Mercury
Associated Acts	The 101ers, The Clash
H1: John Graham Mellor plays less instruments than the number of labels he has worked for.
H2: Joe Strummer changed his surname after he became a guitar player.
H3: Joe Strummer was active in the sports industry for over three decades.
$\mathrm{H1}_{hi -TRL}$  :jon graham melar un lebaloan ki sankhya kti tulana mean kam vadya bajate haian jinake lie unhoanne kam kiyahai
$\mathrm{H2}_{hi -TRL}$  : jo stramar ne ek gitar vadak banane ke badapana upanam badal liya
H3<sub>hi-trl</sub>: jo stramar tin dashakoan se khel udyog mean sakriya the
Figure 1: Tabular premise followed by human written hypotheses (H1, H2, H3). H1 is entailed entirely from the premise, H2 is neither entailed nor contradictory, and H3 is contradictory.  $\mathrm{H}1_{hi -TRL}$ ,  $\mathrm{H}2_{hi -TRL}$ , and  $\mathrm{H}3_{hi -TRL}$  are the transliterations of Hindi translations of the former, released as a part of our EI-INFOTABS dataset.
supervision on heuristic adaptations of these datasets perform adequately.
Typically, and to the authors' best knowledge, fact verification tasks, specifically TNLI, have been examined only in a monolingual setting wherein, the tabular premise and the textual hypothesis are in the same language. However, many semistructured/tabular data sources exist only in English but require verification of hypotheses over those data sources in other languages, as discussed in  $\S 2$ . Therefore, we examine a modified tabular NLI task by introducing bilinguality within the premise
*Equal Contribution † Corresponding Author
Human Language Technologies, pages 4018 - 4037
and hypothesis pair. To understand this modified task, consider the example in Figure 1. The table presented in the figure has been extracted from the English Wikipedia article on Joe Strummer<sup>1</sup>, a well known musician. Following which, are transliterated hypotheses in Hindi (hi) (and their English (en) translation) which are related to the information presented in the given table. We show transliterated hypotheses only for the ease of comprehension. We use native scripts for each language in the EI-INFOTABS dataset. For bilingual tabular NLI, a reasoning model should be able to predict the inference label entail for  $\mathrm{H}1_{hi}$ , neutral for  $\mathrm{H}2_{hi}$  and contradict for  $\mathrm{H}3_{hi}$  given the English table as the primary context. In summary, our contributions are as follows:
- We introduce the task of bilingual tabular NLI (bTNLI) wherein the tabular premise is in a high resource language, while the textual hypothesis is in a low resource language. This is a practical, real world setting for fact verification on semistructured tabular data which is further illustrated in §2.
- We create EI-INFOTABS, a dataset consisting of machine translated hypotheses in 11 Indian languages, while retaining the English tabular premises from the INFOTABS dataset. Through extensive studies shown in §3, we confirm that EI-INFOTABS is of good quality, and preserves properties important to study the bTNLI task.
- We explore several multilingual models for the bTNLI task, and establish strong baselines and share findings about their performance across multilingual models, languages, train-eval techniques, tabular reasoning categories, adversarial test splits, and both datasets (INFOTABS and EI-INFOTABS).
Overall, EI-INFOTABS dataset and our proposed train-eval strategies enable thorough examination of the challenging task of bTNLI. Furthermore, the former aslo serve as a quality benchmark for evaluating the robustness of multilingual models. The dataset and the associated scripts, are available at https://enindicinfotabs.github.io.
Why Tabular NLI? Tabular data is termed as semi-structured as it is neither truly unstructured
data like raw text, nor is it entirely structured like a database. Although semi-structured data is based on a structured scaffold, the content can be freeform text with variable length and type. Moreover, unlike a database, there is no homogeneity across various data points in a shared context. Such structural ambiguity imposes a significant cognitive load while reasoning about it. However, such data is ubiquitous in the real world (e.g. web pages, fact sheets, information tables) and we frequently make inferences from it.
Chen et al. (2020b) argue that reasoning about semi-structured data is broadly two-fold in nature. It consists of (a.) Linguistic Reasoning: a semantic deconstruction of the semi-structured data (b.), and Symbolic Reasoning: a symbolic execution on the tabular structure. For instance,  $\mathrm{H}_{2}$  in Figure 1 requires linguistic reasoning over the phrase "became a guitar player" from the "Occupation", and the "Instruments" rows of the concerned table.  $\mathrm{H}_{1}$  requires symbolic reasoning in the form of conditional and arithmetic operations on the "Labels" and "Instruments" rows. Whereas,  $\mathrm{H}_{3}$  requires a combination of the two types of reasoning. Such interwoven reasoning criteria makes it challenging to model Tabular NLI task.
Why Indic Languages? Indian society is largely multilingual and consists of 122 major and 1599 other languages and dialects spanning 6 language families with over 1.3 billion native speakers². Out of these, 30 languages have more than 1 million native speakers each and over 1 billion speakers cumulatively³. Moreover, India has the second largest online presence with over 749 million internet users and is expected to grow to over 1.5 billion users by  $2040^4$ . So, development of competent reasoning models for the Indic context is essential.
However, due to unfair linguistic bias on the web (Miquel-Ribe and Laniado, 2020; Joshi et al., 2020), there is a disproportionate distribution of text resources for Indian languages. Indian languages have a limited number of internet resources. Thus, they are often known as low web resource languages (LRL) (Khemchandani et al., 2021). For instance, Wikipedia entries in Hindi are just  $2\%$  of those in English, and Wikipedia entries in Assamese and Oriya are 7 times lesser than those in Hindi. This implies that a significant fraction of
2 Wikipedia Indian Languages 3 2011 Indian Census
4 www.statista.com
$^{1}$  Joe Strummer Wikipedia
articles and sometime even complete categories are discussed only in the English language Wikipedia (Bao et al., 2012).
Although, efforts have been made to bridge this gap (Adar et al., 2009; Kumaran et al., 2010), there still exist several limitations: (a.) table extraction for an article across languages is a challenge due to absence of Wikipedia page links, their infobox tables or important keys of tables, (b.) even if tabular data exists, infobox tables in Indian languages are not updated as regularly as their English equivalents (Minhas et al., 2022) which leaves us with outdated and untrustworthy tabular data for inference, (c.) and lastly, table translation while maintaining the intent, context, and the same quality of the source English language is difficult. Often, accurate translation requires the distinction of a language specific domain expert. Due to above reasons, tabular data is mostly absent from Indic Wikipedia articles.
Thus, fact verification in a bilingual setting wherein, the premise is in English and the claim/hypothesis is in an Indic language, is of great significance. Moreover, recent advances in multilingual language models (Khanuja et al., 2021a; Kunchukuttan, 2020), datasets (Roark et al., 2020; Ramesh et al., 2022), and translation systems (Ramesh et al., 2022) for Indian languages have enabled quality examination of several Indic NLU tasks which serves as additional motivation to evaluate the task of bTNLI for Indic languages before other low resource languages.
3 EI-INFOTABS Dataset
EI-INFOABS is an English-Indic bTNLI extension of INFOABS (Gupta et al., 2020), an English TNLI dataset. INFOABS consists of 23,738 pairs of tabular premises and textual hypotheses. The hypotheses are human written short assertions with an accompanying NLI label, and the tabular premises are based on 2,540 Wikipedia infoboxes from 12 diverse categories. Moreover, it consists of additional adversarial test sets apart from  $\alpha_{1}$  which is the standard test set and is lexically and topically similar to the train set -  $\alpha_{2}$  is the lexically adversarial test set which maintains topical similarity and  $\alpha_{3}$  is the topically adversarial test set. The dev and test sets  $(\alpha_{1},\alpha_{2},$ $\alpha_{3})$  cumulatively consist of 7200 table-hypothesis pairs equally splits on all four sets.
EI-INFO TABS extends it by providing machine
translated hypotheses in 11 major Indic languages namely Assamese (as), Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (kn), Malayalam (ml), Marathi (mr), Odia (or), Punjabi (pa), Tamil (ta), and Telugu (te) for each tabular premise. In this section, we discuss the EI-INFOTABS construction and verification.
3.1 EI-INFOTABS Construction
To construct EI-INFOTABS, we machine translated the English hypotheses provided in INFOTABS to 11 major Indian languages as described earlier. We use IndicTrans (Ramesh et al., 2022), an open-sourced state-of-the-art Indic NMT model. IndicTrans is trained on the Samanantar dataset (Ramesh et al., 2022), which is the largest publicly available parallel corpus for Indic languages. Moreover, it outperforms (a) commercial NMT systems like GoogleTranslate and Bing Microsoft Translator, and (b) open-source multilingual models like OPUSMT (Tiedemann and Thottingal, 2020), mBART50 (Liu et al., 2020) and mT5 (Xue et al., 2021).
3.2 EI-INFO TABS Verification.
Given the absence of Indic reference data, it becomes challenging to measure the quality of the translations, and subsequently, of EI-INFOTABS. In this section, we describe our robust quality estimation approach to validate EI-INFOTABS.
Automatic Evaluation. We use BERTScore (Zhang* et al., 2020), an automatic scoring metric for sentence similarity, between the source and back-translated English sentences. We use IndicTrans to generate Indic to English back-translated data.
BERTScore is known to correlate better with human judgment at the sentence level (Zhang* et al., 2020) compared to conventionally used MT evaluation metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). BERTScore calculates word level semantic similarity whereas the conventional MT metrics focus on word overlap. The results are presented in Table 1. We notice high semantic similarity scores for all the languages. However, when we analyse the examples with low scores, we note that the scores are almost always low due to the error added during the back-translation phase. The back-translation introduces
5 https://translate.google.co.in/
$^{6}$  https://www.bing.com/Translator
errors due to incorrect transliteration of Named Entities. Consider the following example:
- Femme aux Bras Croisés is open for public viewing.
- Back-translated: The Ox Brass Crossox is open to the public
- Hindi Translation(Transliterated): fem auksa bras kroisaiksa janata ke lie khulā hai
The Hindi translation of the original sentence is perfect, however, the named entity "Femme aux Bras Croisés" when back-translated becomes "Ox Brass Crossox" and yields a low BERTScore of 0.86. This is broadly identified as qualitative feedback for most of the sentences with low scores across all the languages. Around  $20\%$  of the examples yield a BERTScore of 1.0 and are deemed perfect translations when reviewed by native speakers.
Human Evaluation. Broadly, we follow the guidelines recommended in (Agirre et al., 2016) to conduct human evaluation. We (a.) diversely sample source-translation pairs in each language, (b.) prepare a common Direct Assessment (Graham et al., 2013) scoring strategy, and (c.) get the sampled data evaluated on the basis of that strategy.
Diverse Sampling. We sample 50 diverse hypotheses from the dev split of EI-INFOTABS for each Indic language. Using the k-DPP algorithm (Kulesza and Taskar, 2011) over the mBERT sentence representations, we're able to achieve syntactically and semantically diverse samples spanning the different table categories.
Direct Assessment. We adopt the human evaluation strategy for low resource machine translation laid out in (Guzmán et al., 2019). We ask native Indic language speakers proficient in English to score a source-translation pair from 0-100. The score highlights the perceived translation quality of the source-translation pair. For each language, we get the samples annotated by two different annotators. In Table 1, we report the average scores for each language along with the Pearson correlation coefficient  $(r)$  as a measure for inter-rater reliability. For more details on human evaluation strategy refer to Appendix §C.
Discussion. We report our evaluation results in Table 1. Automatic evaluation and our corresponding analysis on it shows that EI
Language	DA	BSIT	BSGT	r
Bengali ('bn')	0.87	0.95	0.99	0.64
Marathi ('mr')	0.81	0.94	0.98	0.68
Gujarati ('gu')	0.89	0.95	0.98	0.38
Oriya ('or')	0.94	0.94	0.98	0.35
Hindi ('hi')	0.89	0.96	0.99	0.40
Punjabi ('pa')	0.86	0.95	0.98	0.34
Kannada ('kn')	0.87	0.95	0.98	0.70
Tamil ('ta')	0.85	0.94	0.98	0.59
Malayalam ('ml')	0.85	0.94	0.98	0.50
Telugu ('te')	0.84	0.94	0.98	0.39
Assamese ('as')	0.83	0.94	-	0.65
Table 1: Here, we compare the Average Direct Assessment (DA) scores provided by native speakers with Average BERTScore F1 scores for IndicTrans En-Indic-En back-translated data  $(\mathbf{BS}^{IT})$ , and Average BERTScore F1 scores for Google-Translate En-Indic-En back-translated data  $(\mathbf{BS}^{GT})$ . Additionally, we also present the Pearson correlation coefficient as a measure of inter-rater reliability. Higher score implies better quality for each of the metric.
INFOTABS consists of fluent, semantically accurate translations across all Indic languages. Moreover, we note competitive Direct Assessment scores for each language, and a positive  $r$  value which indicates that the native speakers agree on the good quality of EI-INFOTABS.
4 Experimental Pipeline
We design our experimental pipeline along the lines of the research question: How well do existing pretrained multilingual language models perform on the bTNLI task? In this section, we propose various modeling strategies and examine how they might address the challenges and nuances of the proposed inference bTNLI task.
4.1 Table Representations
It is necessary to linearize semi-structured tabular data into a textual premise in order to reduce the task of Tabular Inferencing to a standard NLI task for which existing state-of-the-art language models can be adapted directly. We use and compare the previously proposed linearization methods (a.) Better Paragraph Representations (BPR) (Neeraja et al., 2021), (b.) and Premise as Structure - TabFact (Chen et al., 2020b; Gupta et al., 2020) (cf. Appendix §A). Henceforth, by premise, we refer to the linearized representation of the tabular premise i.e. the infobox table.
Strategy	Model	bn	hi	gu	pa	mr	te	ta	ml	kn	as	or	ModAvg
Translate-Train	mBERT	62	64	61	62	61	60	61	59	61	60	35	59
IndicBERT	54	54	53	48	51	51	52	47	54	34	53	50
MuRIL	67*	67*	67*	66	65	65	66	66	66	65	64	66
LnAvg	61	62	60	59	59	57	60	56	60	53	51	57
Translate-Test	mBERT	53	55	50	54	51	50	53	51	52	45	34	50
IndicBERT	37	35	35	34	36	36	34	38	39	39	38	36
MuRIL	63	65*	62	62	62	60	61	61	62	60	59	62
LnAvg	51	52	49	50	50	49	49	50	51	48	44	49
Bilingual-Train	mBERT	63	66	62	64	62	63	63	62	64	62	36	61
IndicBERT	53	53	52	53	52	50	52	54	53	54	53	53
MuRIL	68*	67	66	67	65	67	66	66	66	65	65	66
LnAvg	61	62	60	61	60	60	60	61	61	60	51	60
Multilingual-Train	mBERT	63	64	62	63	62	62	61	62	63	62	36	60
IndicBERT	53	54	53	52	52	50	51	50	50	53	51	52
MuRIL	67	68*	67	67	66	66	67	67	67	66	66	67
LnAvg	61	62	61	61	60	59	60	60	60	60	51	60
EnTranslate-Test	mBERT	65	67*	63	66	63	62	64	62	64	62	62	64
IndicBERT	56	57	56	57	55	56	57	57	57	57	56	56
MuRIL	65	67*	65	65	63	62	64	63	64	61	62	64
LnAvg	63	64	61	63	60	60	62	61	62	60	60	62
Translate-Train-X	mBERT	55	55	53	54	53	53	54	53	53	50	36	51
IndicBERT	41	41	39	36	39	40	40	40	40	34	40	39
MuRIL	64	65*	64	63	64	64	63	63	64	63	62	64
LnAvg	53	53	52	51	52	52	52	52	52	49	46	51
Bilingual-Train-X	mBERT	56	56	55	56	56	56	55	55	55	55	41	54
IndicBERT	42	42	41	40	41	41	41	41	42	42	41	41
MuRIL	65*	64	65*	65*	64	65*	65*	65*	65*	65*	65*	65
LnAvg	54	54	54	53	54	54	54	54	54	54	49	53
Table 2: Performance in terms of accuracy when evaluated on the  $\alpha_{1}$  test set. Higher value implies better performance. Here, LnAvg represents the average accuracy for a language across all models, while ModAvg represents the average accuracy of a model across all languages. A value in Purple represents the best accuracy for that model across all languages. An underlined value in Blue represents the best accuracy for that language across all models. A value in Green with an asterisk(*) represents the cases where language-wise and model-wise values coincide. As we fine-tune on a specific Indic language in the train-eval strategies Translate-Train-X and Bilingual-Train-X, we report the training average of accuracy on the remaining 10 Indic languages for them. We do not include the results of XLM-RoBERTa as the model fails to converge on these experiments on multiple runs with a distinct set of hyper-parameters as explained in Appendix §D. The results for the  $\alpha_{2}$  and  $\alpha_{3}$  sets are provided in Appendix §E.
Owing to the multilingual setting of this task, we utilise pre-trained multilingual models to encode the linearized English tabular premise along with the Indic hypothesis into contextual representations for classification. We consider two kinds of pretrained multilingual models (a.) Indic Specific which includes IndicBERT and MuRIL due to their indic specific pre-training, and (b.) Generic which includes mBERT and XLM-Roberta due to their pre training in more than hundred languages. For more details refer to Appendix §B.
4.3 Training and Evaluation Strategies
In order to examine the inter-woven relationships among the 11 languages, and the corresponding impact on multilingual models' performance, we design a set of train-eval strategies for this task.
Translate-Train: We fine-tune and evaluate the models on EN-IN $_i$  premise-hypothesis pairs where IN $_i$  is one of the 11 Indic languages. This baseline evaluates the performance of the multilingual models on EI-INFOTABS when fine-tuned on Indic hypotheses. We also evaluated these models across all languages i.e. cross lingual zero-shot setting Translate-Train-X.
Translate-Test: We fine-tune the multilingual models on EN-EN premise-hypothesis pairs from the INFOTABS dataset and evaluate on EN-IN $_i$  premise-hypothesis pairs. This baseline evaluates the Zero-shot Cross-Lingual Transfer ability of the reasoning models from INFOTABS to EIINFOTABS.
Bilingual-Train: We fine-tune the multilingual models on both EN-EN and  $\mathrm{EN - IN}_i$  premise-hypothesis pairs, and evaluate on  $\mathrm{EN - IN}_i$  premise
hypothesis pairs. This baseline evaluates whether addition of English hypotheses while fine-tuning aids the performance of the multilingual models prepared in Translate-Train. We also evaluated these models across all languages i.e. cross lingual zero-shot setting Bilingual-Train-X.
Multilingual-Train: We fine-tune the multilingual models on all available training data across all Indic languages and the English language. We evaluate the models on EN-IN $_i$  premise-hypothesis pairs on each 11 Indic languages. This baseline assesses if fine-tuning on several languages to produce a unified multilingual model improves performance.
EnTranslate-Test: We fine-tune the multilingual models on EN-EN premise-hypothesis pairs from INFOTABS and evaluate on EN-ENIN $_i$  premise-hypothesis pairs where ENIN $_i$  represents IN $_i$  to EN back-translated hypotheses. This approach evaluate the translate then test baseline on the EIINFOTABS dataset.
In this section, we discuss and analyse the results obtained on conducting the experiments as per the various strategies laid out in  $\S 4$ . We present the results in Table 2 for each experiment on the  $\alpha_{1}$  test set using the BPR linearization algorithm. The values represent classification accuracy. We analyze the findings thoroughly across multilingual models, languages, train-eval techniques, tabular reasoning categories, adversarial test splits, and both datasets (INFOTABS and EI-INFOTABS).
5.1 Across Multilingual Models
We observe that MuRIL performs best across all languages and experiments except EnTranslate-Test, beating IndicBERT and mBERT. MuRIL's superior performance can be justified on the grounds of (a) the large size of the hidden layers, (b) Indic specific pre-training data, and (c) Indic specific pre-training objectives (Khanuja et al., 2021a). MuRIL's architecture consists of 237M parameters, compared to mBERT's 167M and IndicBERT's 33M, which makes it extremely competitive on any Indic NLU task. IndicBERT's relatively small size explains why it performs the worst, even though it is pre-trained on Indic specific data. mBERT comes in a close second to MuRIL, failing to perform adequately only on Odia (or). mBERT isn't pre-trained on Assamese (as) or Odia which justifies its extremely low performance on
Odia. However, we note competitive results on the Assamese language. This could be attributed to the fact that Assamese is closely related to Bengali  $(bn)$  linguistically. They both share the Bengali-Assamese script and are mutually intelligible (Khemchandani et al., 2021).
#	dev	α1	α2	α3
0	15.56%	16.33%	27.61%	25.67%
1-3	11.17%	10.83%	11.39%	14.22%
4-6	7.16%	7.5%	6.72%	9.61%
7-9	9.55%	10.67%	10.22%	12.67%
10-11	56.56%	54.67%	44.06%	37.83%
Table 3: Percentage of examples predicted correctly by our best performing model for the given number of Indic languages. For instance,  $7.16\%$  of examples in the dev set are predicted correctly for at least 4 and at max 6 Indic languages.