Venue: NAACL 2022

# Bilingual Tabular Inference: A Case Study on Indic Languages

 Chaitanya Agarwal\({}^{1}\), Vivek Gupta\({}^{2*}\), Anoop Kunchukuttan\({}^{3,4}\), Manish Shrivastava\({}^{1}\)

\({}^{1}\)LTRC, IIIT Hyderabad; \({}^{2}\)University of Utah; \({}^{3}\)AI4Bharat; \({}^{4}\)Microsoft chaitanya.agarwal@research.iiit.ac.in; vgupta@cs.utah.edu; ankunchu@microsoft.com; m.shrivastava@iitit.ac.in

Equal Contribution Corresponding Author

###### Abstract

Existing research on Tabular Natural Language Inference (TNLI) exclusively examines the task in a monolingual setting where the tabular premise and hypothesis are in the same language. However, due to the uneven distribution of text resources on the web across languages, it is common to have the tabular premise in a high resource language and the hypothesis in a low resource language. As a result, we present the challenging task of bilingual Tabular Natural Language Inference (bTNLI), in which the tabular premise and a hypothesis over it are in two separate languages. We construct EI-InfoTabS: an English-Indic bTNLI dataset by translating the textual hypotheses of the English TNLI dataset InfoTabS into eleven major Indian languages. We thoroughly investigate how pretained multilingual models learn and perform on EI-InfoTabS. Our study shows that the performance on bTNLI can be close to its monolingual counterpart, with translate-train, translate-test and unified-train being strongly competitive baselines.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4018 - 4037

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Tabular Natural Language Inference (TNLI) is the task of classifying whether a textual hypothesis is an entailment, contradiction or a neutral extension of the given tabular premise. The task requires a broad range of reasoning abilities, including but not limited to the ability to make lexical, spatio-temporal, and semantic deductions. Recently published datasets, TabFact Chen et al. (2020) and InfoTabS Gupta et al. (2020), have enabled the examination of the TNLI task. Moreover, sophisticated models based on deep contextual embeddings like BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), etc. trained under supervision on heuristic adaptations of these datasets perform adequately.

Typically, and to the authors' best knowledge, fact verification tasks, specifically TNLI, have been examined only in a monolingual setting wherein, the tabular premise and the textual hypothesis are in the same language. However, many semi-structured/tabular data sources exist only in English but require verification of hypotheses over those data sources in other languages, as discussed in SS2. Therefore, we examine a modified tabular NLI task by introducing bilinguality within the premise

Figure 1: Tabular premise followed by human written hypotheses (H1, H2, H3). H1 is entailed entirely from the premise, H2 is neither entailed nor contradictory, and H3 is contradictory. H1\({}_{hi-trl}\), H2\({}_{hi-trl}\), and H3\({}_{hi-trl}\) are the transliterations of Hindi translations of the former, released as a part of our EI-InfoTabS dataset.



and hypothesis pair. To understand this modified task, consider the example in Figure 1. The table presented in the figure has been extracted from the English Wikipedia article on _Joe Strummer_1, a well known _musician_. Following which, are transliterated hypotheses in Hindi (hi) (and their English (en) translation) which are related to the information presented in the given table. We show transliterated hypotheses only for the ease of comprehension. We use native scripts for each language in the EI-InfoTabS dataset. For bilingual tabular NLI, a reasoning model should be able to predict the inference label entail for H1\({}_{hi}\), neutral for H2\({}_{hi}\) and contradict for H3\({}_{hi}\) given the English table as the primary context. In summary, our contributions are as follows:

Footnote 1: Joe Strummer Wikipedia

\(\bullet\) We introduce the task of bilingual tabular NLI (bTNLI) wherein the tabular premise is in a high resource language, while the textual hypothesis is in a low resource language. This is a practical, real world setting for fact verification on semi-structured tabular data which is further illustrated in SS2.

\(\bullet\) We create EI-InfoTabS, a dataset consisting of machine translated hypotheses in 11 Indian languages, while retaining the English tabular premises from the InfoTabS dataset. Through extensive studies shown in SS3, we confirm that EI-InfoTabS is of good quality, and preserves properties important to study the bTNLI task.

\(\bullet\) We explore several multilingual models for the bTNLI task, and establish strong baselines and share findings about their performance across multilingual models, languages, train-eval techniques, tabular reasoning categories, adversarial test splits, and both datasets (InfoTabS and EI-InfoTabS).

Overall, EI-InfoTabS dataset and our proposed train-eval strategies enable thorough examination of the challenging task of bTNLI. Furthermore, the former aslo serve as a quality benchmark for evaluating the robustness of multilingual models. The dataset and the associated scripts, are available at [https://enindicinfotabs.github.io](https://enindicinfotabs.github.io).

## 2 Motivation

Why Tabular NLI?Tabular data is termed as semi-structured as it is neither truly unstructured data like raw text, nor is it entirely structured like a database. Although semi-structured data is based on a structured scaffold, the content can be freeform text with variable length and type. Moreover, unlike a database, there is no homogeneity across various data points in a shared context. Such structural ambiguity imposes a significant cognitive load while reasoning about it. However, such data is ubiquitous in the real world (e.g. web pages, fact sheets, information tables) and we frequently make inferences from it.

Chen et al. (2020) argue that reasoning about semi-structured data is broadly two-fold in nature. It consists of (a.) Linguistic Reasoning: a semantic deconstruction of the semi-structured data (b.) , and Symbolic Reasoning: a symbolic execution on the tabular structure.For instance, H\({}_{2}\) in Figure 1 requires linguistic reasoning over the phrase "_became a guitar player_" from the "_Occupation_", and the "_Instruments_" rows of the concerned table. H\({}_{1}\) requires symbolic reasoning in the form of conditional and arithmetic operations on the "_Labels_" and "_Instruments_" rows. Whereas, H\({}_{3}\) requires a combination of the two types of reasoning. Such interwoven reasoning criteria makes it challenging to model Tabular NLI task.

Why Indic Languages?Indian society is largely multilingual and consists of 122 major and 1599 other languages and dialects spanning 6 language families with over 1.3 billion native speakers2. Out of these, 30 languages have more than 1 million naive speakers each and over 1 billion speakers cumulatively3. Moreover, India has the second largest online presence with over 749 million internet users and is expected to grow to over 1.5 billion users by 20404. So, development of competent reasoning models for the Indic context is essential.

Footnote 2: Wikipedia Indian Languages 3  2011 Indian Census 4 www.statista.com

However, due to unfair linguistic bias on the web (Miquel-Ribe and Laniado, 2020; Joshi et al., 2020), there is a disproportionate distribution of text resources for Indian languages. Indian languages have a limited number of internet resources. Thus, they are often known as low web resource languages (LRL) (Khemchandani et al., 2021). For instance, Wikipedia entries in Hindi are just \(2\%\) of those in English, and Wikipedia entries in Assamese and Oriya are 7 times lesser than those in Hindi. This implies that a 

articles and sometime even complete categories are discussed only in the English language Wikipedia Bao et al. (2012).

Although, efforts have been made to bridge this gap Adar et al. (2009); Kumaran et al. (2010), there still exist several limitations: (a.) **table extraction** for an article across languages is a challenge due to absence of Wikipedia page links, their infobox tables or important keys of tables, (b.) even if tabular data exists, infobox tables in Indian languages are not updated as regularly as their English equivalents Minhas et al. (2022) which leaves us with **outdated and untrustworthy tabular data** for inference, (c.) and lastly, **table translation** while maintaining the intent, context, and the same quality of the source English language is difficult. Often, accurate translation requires the distinction of a language specific domain expert. Due to above reasons, tabular data is mostly absent from Indic Wikipedia articles.

Thus, fact verification in a bilingual setting wherein, the premise is in English and the claim/hypothesis is in an Indic language, is of great significance. Moreover, recent advances in multilingual language models Khanuja et al. (2021); Kunchukuttan (2020), datasets Roark et al. (2020); Ramesh et al. (2022), and translation systems Ramesh et al. (2022) for Indian languages have enabled quality examination of several Indic NLU tasks which serves as additional motivation to evaluate the task of bTNLI for Indic languages before other low resource languages.

## 3 EI-InfoTabS Dataset

EI-InfoTabS is an English-Indic bTNLI extension of InfoTabS Gupta et al. (2020), an English TNLI dataset. InfoTabS consists of \(23,738\) pairs of tabular premises and textual hypotheses. The hypotheses are human written short assertions with an accompanying NLI label, and the tabular premises are based on 2,540 Wikipedia infoboxes from 12 diverse categories. Moreover, it consists of additional adversarial test sets apart from \(\alpha_{1}\) which is the standard test set and is lexically and topically similar to the train set - \(\alpha_{2}\) is the lexically adversarial test set which maintains topical similarity and \(\alpha_{3}\) is the topically adversarial test set. The dev and test sets (\(\alpha_{1}\), \(\alpha_{2}\), \(\alpha_{3}\)) cumulatively consist of 7200 table-hypothesis pairs equally splits on all four sets.

EI-InfoTabs extends it by providing machine translated hypotheses in 11 major Indic languages namely Assamese (_as_), Bengali (_bn_), Gujarati (_gu_), Hindi (_hi_), Kannada (_kn_), Malayalam (_ml_), Marathi (_mr_), Odia (_or_), Punjabi (_pa_), Tamil (_ta_), and Telugu (_te_) for each tabular premise. In this section, we discuss the EI-InfoTabS construction and verification.

### EI-InfoTabS Construction

To construct EI-InfoTabS, we machine translated the English hypotheses provided in InfoTabS to 11 major Indian languages as described earlier. We use IndicTrans Ramesh et al. (2022), an open-sourced state-of-the-art Indic NMT model. IndicTrans is trained on the Samannantar dataset Ramesh et al. (2022), which is the largest publicly available parallel corpus for Indic languages. Moreover, it outperforms (a) commercial NMT systems like Google-Translate 5 and Bing Microsoft Translator 6, and (b) open-source multilingual models like OPUS-MT Tiedemann and Thottingal (2020), mBART50 Liu et al. (2020) and mT5 Xue et al. (2021).

Footnote 5: [https://translate.google.co.in/](https://translate.google.co.in/)

Footnote 6: [https://www.bing.com/translator](https://www.bing.com/translator)

### EI-InfoTabS Verification.

Given the absence of Indic reference data, it becomes challenging to measure the quality of the translations, and subsequently, of EI-InfoTabS. In this section, we describe our robust quality estimation approach to validate EI-InfoTabS.

#### Automatic Evaluation.

We use BERTScore Zhang* et al. (2020), an automatic scoring metric for sentence similarity, between the source and back-translated English sentences. We use IndicTrans to generate Indic to English backwardated data.

BERTScore is known to correlate better with human judgment at the sentence level Zhang* et al. (2020) compared to conventionally used MT evaluation metrics like BLEU Papineni et al. (2002) and ROUGE Lin (2004). BERTScore calculates word level semantic similarity whereas the conventional MT metrics focus on word overlap. The results are presented in Table 1. We notice high semantic similarity scores for all the languages. However, when we analyse the examples with low scores, we note that the scores are almost always low due to the error added during the back-translation phase. The back-translation introduces 

errors due to incorrect transliteration of _Named Entities_. Consider the following example:

\(\bullet\) Femme aux Bras Croises is open for public viewing.

\(\bullet\) Back-translated: The Ox Brass Crossox is open to the public

\(\bullet\) Hindi Translation(Transliterated): fem auksa bras kroisaiksa janata ke lie khulai hai

The Hindi translation of the original sentence is perfect, however, the named entity "_Femme aux Bras Croises_" when back-translated becomes "_Ox Brass Crossox_" and yields a low BERTScore of 0.86. This is broadly identified as qualitative feedback for most of the sentences with low scores across all the languages. Around 20% of the examples yield a BERTScore of 1.0 and are deemed perfect translations when reviewed by native speakers.

Human Evaluation.Broadly, we follow the guidelines recommended in Agirre et al. (2016) to conduct human evaluation. We (a.) diversely sample source-translation pairs in each language, (b.) prepare a common Direct Assessment Graham et al. (2013) scoring strategy, and (c.) get the sampled data evaluated on the basis of that strategy.

Diverse Sampling.We sample 50 diverse hypotheses from the dev split of EI-InfoTabS for each Indic language. Using the k-DPP algorithm Kulesza and Taskar (2011) over the mBERT sentence representations, we're able to achieve syntactically and semantically diverse samples spanning the different table categories.

Direct Assessment.We adopt the human evaluation strategy for low resource machine translation laid out in Guzman et al. (2019). We ask native Indic language speakers proficient in English to score a source-translation pair from 0-100. The score highlights the perceived translation quality of the source-translation pair. For each language, we get the samples annotated by two different annotators. In Table 1, we report the average scores for each language along with the Pearson correlation coefficient (\(r\)) as a measure for inter-rater reliability. For more details on human evaluation strategy refer to Appendix SSC.

Discussion.We report our evaluation results in Table 1. Automatic evaluation and our corresponding analysis on it shows that EI-InfoTabS consists of fluent, semantically accurate translations across all Indic languages. Moreover, we note competitive Direct Assessment scores for each language, and a positive \(r\) value which indicates that the native speakers agree on the good quality of EI-InfoTabS.

## 4 Experimental Pipeline

We design our experimental pipeline along the lines of the research question: _How well do existing pretrained mutlilingual language models perform on the bTNLI task?_ In this section, we propose various modeling strategies and examine how they might address the challenges and nuances of the proposed inference bTNLI task.

### Table Representations

It is necessary to linearize semi-structured tabular data into a textual premise in order to reduce the task of Tabular Inferencing to a standard NLI task for which existing state-of-the-art language models can be adapted directly. We use and compare the previously proposed linearization methods (a.) Better Paragraph Representations (BPR) Neeraja et al. (2021), (b.) and Premise as Structure - TabFact Chen et al. (2020); Gupta et al. (2020) (cf. Appendix SSA). Henceforth, by premise, we refer to the linearized representation of the tabular premise i.e. the infobox table.

\begin{table}
\begin{tabular}{p{71.1pt} p{28.5pt} p{28.5pt} p{28.5pt}|p{28.5pt}} \hline Language & DA & BS\({}^{GT}\) & BS\({}^{GT}\) & \(r\) \\ \hline Bengali (’bn’) & 0.87 & 0.95 & 0.99 & 0.64 \\ Marathi (‘mr’) & 0.81 & 0.94 & 0.98 & 0.68 \\ Gujarati (‘gu’) & 0.89 & 0.95 & 0.98 & 0.38 \\ Oriya ( ‘or’) & 0.94 & 0.94 & 0.98 & 0.35 \\ Hindi (‘hi’) & 0.89 & 0.96 & 0.99 & 0.40 \\ Punjabi (‘pa’) & 0.86 & 0.95 & 0.98 & 0.34 \\ Kannada (‘kn’) & 0.87 & 0.95 & 0.98 & 0.70 \\ Tamil (‘ta’) & 0.85 & 0.94 & 0.98 & 0.59 \\ Malayalam (‘ml’) & 0.85 & 0.94 & 0.98 & 0.50 \\ Telugu (‘te’) & 0.84 & 0.94 & 0.98 & 0.39 \\ Assamese (‘as’) & 0.83 & 0.94 & - & 0.65 \\ \hline \end{tabular}
\end{table}
Table 1: Here, we compare the Average Direct Assessment (DA) scores provided by native speakers with Average BERTScore F1 scores for IndicTrans En-Indic-En back-translated data (BS\({}^{IT}\)), and Average BERTScore F1 scores for Google-Translate En-Indic-En back-translated data (BS\({}^{GT}\)). Additionaly, we also present the Pearson correlation coeffecient as a measure of inter-rater reliability. Higher score implies better quality for each of the metric.

 



### Multilingual Models

Owing to the multilingual setting of this task, we utilise pre-trained multilingual models to encode the linearized English tabular premise along with the Indic hypothesis into contextual representations for classification. We consider two kinds of pretrained multilingual models (a.) **Indic Specific** which includes IndicBERT and MuRIL due to their indic specific pre-training, and (b.) **Generic** which includes mBERT and XLM-Roberta due to their pre training in more than hundred languages. For more details refer to Appendix SSB.

### Training and Evaluation Strategies

In order to examine the inter-woven relationships among the 11 languages, and the corresponding impact on multilingual models' performance, we design a set of train-eval strategies for this task.

#### Translate-Train:

We fine-tune and evaluate the models on En-IN premise-hypothesis pairs where IN is one of the 11 Indic languages. This baseline evaluates the performance of the multilingual models on EI-InfoTabS when fine-tuned on Indic hypotheses. We also evaluated these models across all languages i.e. cross lingual zero-shot setting **Translate-Train-X**.

#### Translate-Test:

We fine-tune the multilingual models on En-En premise-hypothesis pairs from the InfoTabS dataset and evaluate on En-IN premise-hypothesis pairs. This baseline evaluates the Zero-shot Cross-Lingual Transfer ability of the reasoning models from InfoTabS to EI-InfoTabS.

#### Bilingual-Train:

We fine-tune the multilingual models on both En-En and En-IN premises-hypothesis pairs, and evaluate on En-IN premise

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c} \hline \hline
**Strategy** & **Model** & bn & hi & gu & pa & rrr & te & ta & ml & kn & as & or & **ModAvg** \\ \hline \multirow{3}{*}{**Translate-Train**} & mBERT & 62 & 64 & 61 & 62 & 61 & 60 & 61 & 59 & 61 & 60 & 35 & 59 \\  & IndicBERT & 54 & 54 & 53 & 48 & 51 & 51 & 52 & 47 & 54 & 34 & 53 & 50 \\  & MuRL & \(\mathbf{67^{*}}\) & \(\mathbf{67^{*}}\) & \(\mathbf{67^{*}}\) & \(\mathbf{66^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{65}\) & \(\mathbf{64}\) & \(\mathbf{66}\) \\  & LnAvg & 61 & 62 & 60 & 59 & 59 & 57 & 60 & 56 & 60 & 53 & 51 & 57 \\ \hline \multirow{3}{*}{**Translate-Test**} & mBERT & 53 & 55 & 50 & 54 & 51 & 50 & 53 & 51 & 52 & 45 & 34 & 50 \\  & IndicBERT & 37 & 35 & 35 & 34 & 36 & 36 & 34 & 38 & 39 & 39 & 38 & 36 \\  & MutRL & \(\mathbf{63}\) & \(\mathbf{65^{*}}\) & \(\mathbf{62}\) & \(\mathbf{62}\) & \(\mathbf{62}\) & \(\mathbf{60}\) & \(\mathbf{61}\) & \(\mathbf{61}\) & \(\mathbf{62}\) & \(\mathbf{60}\) & \(\mathbf{59}\) & \(\mathbf{62}\) \\  & LnAvg & 51 & 52 & 49 & 50 & 50 & 49 & 49 & 50 & 51 & 48 & 44 & 49 \\ \hline \multirow{3}{*}{**Bilingual-Train**} & mBERT & 63 & 66 & 62 & 64 & 62 & 63 & 63 & 62 & 64 & 62 & 36 & 61 \\  & IndicBERT & 53 & 53 & 52 & 53 & 52 & 50 & 52 & 54 & 53 & 54 & 53 & 53 \\  & MuRL & \(\mathbf{68^{*}}\) & \(\mathbf{67}\) & \(\mathbf{66}\) & \(\mathbf{67}\) & \(\mathbf{65}\) & \(\mathbf{67}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{65}\) & \(\mathbf{65}\) & \(\mathbf{66}\) \\ \cline{2-11}  & LnAvg & 61 & 62 & 60 & 61 & 60 & 60 & 60 & 61 & 61 & 60 & 51 & 60 \\ \hline \multirow{3}{*}{**Multilingual-Train**} & mBERT & 63 & 64 & 62 & 63 & 62 & 62 & 61 & 62 & 63 & 62 & 36 & 60 \\  & IndicBERT & 53 & 54 & 53 & 52 & 52 & 50 & 51 & 50 & 50 & 53 & 51 & 52 \\  & MuRL & \(\mathbf{67^{*}}\) & \(\mathbf{68^{*}}\) & \(\mathbf{67}\) & \(\mathbf{67}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{67}\) & \(\mathbf{67}\) & \(\mathbf{67}\) & \(\mathbf{66}\) & \(\mathbf{66}\) & \(\mathbf{67}\) \\ \cline{2-11}  & LnAvg & 61 & 62 & 61 & 61 & 60 & 59 & 60 & 60 & 60 & 51 & 60 \\ \hline \multirow{3}{*}{**EnTranslate-Test**} & mBERT & 65 & \(\mathbf{67^{*}}\) & 63 & 66 & 63 & 62 & 64 & 62 & 64 & 62 & 62 & 64 \\  & IndicBERT & 56 & 57 & 56 & 57 & 55 & 56 & 57 & 57 & 57 & 56 & 56 \\  & MuRL & \(\mathbf{65}\) & \(\mathbf{67^{*}}\) & \(\mathbf{65}\) & \(\mathbf{65}\) & \(\mathbf{63}\) & \(\mathbf{62}\) & \(\mathbf{64}\) & \(\mathbf{63}\) & \(\mathbf{64}\) & \(\mathbf{61}\) & \(\mathbf{62}\) & \(\mathbf{64}\) \\ \cline{2-11}  & LnAvg & 63 & 64 & 61 & 63 & 60 & 60 & 62 & 61 & 62 & 60 & 60 & 62 \\ \hline \multirow{3}{*}{**Translate-Train-X**} & mBERT & 55 & 55 & 53 & 54 & 53 & 53 & 54 & 53 & 53 & 50 & 36 & 51 \\  & IndicBERT & 41 & 41 & 39 & 36 & 39 & 40 & 40 & 40 & 40 & 34 & 40 & 39 \\  & MuRL & \(\mathbf{64}\) & \(\mathbf{65^{*}}\) & \(\mathbf{64}\) & \(\mathbf{63}\) & \(\mathbf{64}\) & \(\mathbf{63}\) & \(\mathbf{63}\) & \(\mathbf{64}\) & \(\mathbf{63}\) & \(\mathbf{64}\) & \(\mathbf{63}\) & \(\mathbf{62}\) & \(\mathbf{64}\) \\ \cline{2-11}  & LnAvg & 53 & 53 & 52 & 51 & 52 & 52 & 52 & 52 & 52 & 49 & 46 & 51 \\ \hline \multirow{3}{*}{**Bilingual-Train-X**} & mBERT & 56 & 56 & 55 & 56 & 56 & 56 & 56 & 55 & 55 & 55 & 41 & 54 \\ \cline{2-11}  & IndicBERT & 42 & 42 & 41 & 40 & 41 & 41 & 41 & 41 & 42 & 42 & 41 & 41 \\ \cline{1-1}  & MuRL & \(\mathbf{65^{*}}\) & \(\mathbf{64}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{64}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) & \(\mathbf{65^{*}}\) \\ \cline{1-1}  & LnAvg & 54 & 54 & 54 & 54 & 53 & 54 & 54 & 54 & 54 & 54 & 54 & 49 & 53 \\ \hline \end{tabular}
\end{table}
Table 2: Performance in terms of accuracy when evaluated on the \(\alpha_{1}\) test set. Higher value implies better performance. Here, LnAvg represents the average accuracy for a language across all models, while ModAvg represents the average accuracy of a model across all languages. A value in Purple represents the best accuracy for that model across all languages. An underlined value in Blue represents the best accuracy for that language across all models. A value in Green with an asterisk(\({}^{*}\)) represents the cases where language-wise and model-wise values coincide. As we fine-tune on a specific

hypothesis pairs. This baseline evaluates whether addition of English hypotheses while fine-tuning aids the performance of the multilingual models prepared in Translate-Train. We also evaluated these models across all languages i.e. cross lingual zero-shot setting **Bilingual-Train-X**.

**Multilingual-Train:** We fine-tune the multilingual models on all available training data across all Indic languages and the English language. We evaluate the models on En-In\({}_{i}\) premise-hypothesis pairs on each 11 Indic languages. This baseline assesses if fine-tuning on several languages to produce a unified multilingual model improves performance.

**EnTranslate-Test:** We fine-tune the multilingual models on En-En premise-hypothesis pairs from InfoTabS and evaluate on En-En\({}_{i}\) premise-hypothesis pairs where EnIN\({}_{i}\) represents In\({}_{i}\) to En back-translated hypotheses. This approach evaluate the translate then test baseline on the EIInfoTabS dataset.

## 5 Results and Analysis

In this section, we discuss and analyse the results obtained on conducting the experiments as per the various strategies laid out in SS4. We present the results in Table 2 for each experiment on the \(\alpha_{1}\) test set using the BPR linearization algorithm. The values represent classification accuracy. We analyze the findings thoroughly across multilingual models, languages, train-eval techniques, tabular reasoning categories, adversarial test splits, and both datasets (InfoTabS and EI-InfoTabS).

### Across Multilingual Models

**We observe that MuRIL performs best across all languages and experiments except EnTranslate-Test, beating IndicBERT and mBERT.** MuRIL's superior performance can be justified on the grounds of (a) the large size of the hidden layers, (b) Indic specific pre-training data, and (c) Indic specific pre-training objectives Khanuja et al. (2021). MuRIL's architecture consists of 237M parameters, compared to mBERT's 167M and IndicBERT's 33M, which makes it extremely competitive on any Indic NLU task. IndicBERT's relatively small size explains why it performs the worst, even though it is pre-trained on Indic specific data. mBERT comes in a close second to MuRIL, failing to perform adequately only on Odia (_or_). mBERT isn't pre-trained on Assamese (_as_) or Odia which justifies its extremely low performance on Odia. **However, we note competitive results on the Asamese language. This could be attributed to the fact that Assamese is closely related to Bengali (_bn_) linguistically. They both share the Bengali-Assumese script and are mutually intelligible Khemchandani et al. (2021).**

mBERT's performance gets boosted in EnTranslate-Test as mBERT is pre-trained on a significant amount of English data which makes it extremely competitive in modeling English NLU tasks. MuRIL performs similarly even though it is trained on lesser amount of English data. This could be due to Indic artifacts like sentence structure and inadequately transliterated named entities being present in the back-translated sentences which MuRIL has been trained to handle better than mBERT.

### Across Languages

We observe that the models perform best on Hindi (_hi_) and Bengali. This is expected as they are high resource languages in the Indic context. **Additionally, as explained in SS5.1, we note that pre-training or fine-tuning on Bengali aids the performance on Assamese due to their high degree of relatedness.** Table 3 shows the measure of agreement across the languages. We note that almost all languages agree on 55% of the predictions on the dev set and the \(\alpha_{1}\) test set. This reduces to 38% on the \(\alpha_{3}\) test set. **This indicates that for a majority of examples from the non-adversarial test sets, MuRIL performs uniformly across languages. However, its performance across languages starts varying more on the adversarial test sets (\(\alpha_{2}\) and \(\alpha_{3}\)).**

### Train-Eval Strategies

Translate-Train's results show that the multilingual models converge and perform adequately when fine-tuned on EI-InfoTabS. Moreover, when fine-tuned along with English data - as described

\begin{table}
\begin{tabular}{l r r r r} \hline \# & dev & \(\alpha_{1}\) & \(\alpha_{2}\) & \(\alpha_{3}\) \\ \hline
0 & 15.56\% & 16.33\% & 27.61\% & 25.67\% \\
1-3 & 11.17\% & 10.83\% & 11.39\% & 14.22\% \\
4-6 & 7.16\% & 7.5\% & 6.72\% & 9.61\% \\
7-9 & 9.55\% & 10.67\% & 10.22\% & 12.67\% \\
10-11 & **56.56\%** & **54.67\%** & **44.06\%** & **37.83\%** \\ \hline \end{tabular}
\end{table}
Table 3: Percentage of examples predicted correctly by our best performing model for the given number of Indic languages. For instance, 7.16% of examples in the dev set are predicted correctly for at least 4 and at max 6 Indic languages.

 

in Bilingual-Train - mBERT and IndicBERT perform marginally better while MuRIL doesn't report a change in performance. **MuRIL, when fine-tuned on all languages as described in Multilingual-Train, performs best on** EIHofTabS **and forms the benchmark for this task**. mBERT and IndicBERT, however, perform worse on Multilingual-Train when compared to Bilingual-Train. This indicates that these models fail to generalise their reasoning ability across all languages and aren't as multilingually robust as MuRIL. The results on Translate-Test are the lowest across all train/eval strategies which indicates a poor Zero-shot Cross-Lingual Transfer from InfoTaB$ to EI-InfoTabS. However, the performance of MuRIL on Translate-Test is comparable with its performance on Translate-Train unlike mBERT and IndicBERT. This indicates that MuRIL can generalize well across English and Indic languages which are linguistically distinct.

Translate-Train-X and Bilingual-Train-X evaluate the average Cross-Lingual Transfer performance of the models trained in Translate-Train and Bilingual-Train. **We observe higher performance in Bilingual-Train-X over Translate-Train-X which indicates that addition of English training data aids the Cross Lingual Transfer from one Indic language to another. Moreover, the average performance of MuRIL on Bilingual-Train-X is comparable to that on Translate-Train which suggests that MuRIL robustly generalises across Indic languages. Both, Bilingual-Train-X and Translate-Train-X perform better than Translate-Test due to high language relatedness among Indic languages when compared with English. The results on EnTranslate-Test are extremely promising for both MuRIL and mBERT. Their performance is very close to that of the best performing model, MuRIL, on Translate-Train. This indicates that back-translation doesn't lead to a significant loss in information required for the bTNLI task.

### Tabular Reasoning Categories

We conduct a fine-grained analysis on how our best model, MuRIL (Multilingual-Train), performs on various reasoning categories. We present the results in Figure 2 for Hindi and Odiya. We observe that MuRIL performs similarly on EIHofTabS as RoBERTa\({}_{\text{lLarge}}\) does on InfoTabS for entity type, named entity, negation, numerical, quantification and simple lookup reasoning types. Additionally, MuRIL performs better for the co-reference resolution reason

Figure 3: Consistency Matrix which measures the deviation of our best performing model, MuRIL (Multilingual-Train)’s predictions on the \(\alpha_{1}\) test set for Hindi as compared to that of RoBERTa\({}_{\text{lLarge}}\) on the \(\alpha_{1}\) test set of InfoTabS.

Figure 2: Here, we compare human benchmarks (H), predictions of the best performing model trained on InfoTabS from Neeraja et al. (2021) (I) and predictions of our best performing model (E), MuRIL (Multilingual-Train), on the examples annotated with reasoning category in the dev split for Oriva (left) and Hindi (right).

 

observed across all the Indic languages. Both RoBERTaLARGE and MuRIL perform poorly for knowledge and common sense, multi-row, co-reference, and temporal reasoning types.

### Across Adverserial Test Splits

The results for the other evaluation sets \(\alpha_{2}\) and \(\alpha_{3}\) are provided in Appendix SE. Across all the experiments, we note that the fine-tuned models perform best on \(\alpha_{1}\), followed by \(\alpha_{2}\) and \(\alpha_{3}\) respectively. **Moreover, we note that on most baselines, the average performance of a fine-tuned model drops by roughly 10% when tested on \(\alpha_{2}\) or \(\alpha_{3}\).** This is similar to the observations reported on InfoTabS (Neeraja et al., 2021) and presented in Table 4. Low performance of the multilingual models on the \(\alpha_{2}\) test set of EI-InfoTabS indicates that (a.) multilingual models learn shallow lexical features to make inferences on EI-InfoTaSb just like the monolingual models do on InfoTabS, (b.) and IndicTrans carefully captures the lexical adversity in the \(\alpha_{2}\) test set of InfoTabS.This commends the ability of IndicTrans to handle lexical nuances. Low performance on \(\alpha_{3}\) test set of EI-InfoTabS suggests that the multilingual models learn categorical features and perform adversely when evaluated on unseen category.

### Ei-InfoTabS v/s InfoTabS

Table 4 reports the human benchmarks and the baselines with the BPR linearization algorithm on each validation set in InfoTabS. We observe that the baselines on EI-InfoTabS are within an absolute margin of 10\(\%\) when compared to those on InfoTabS. **This suggests that EI-InfoTabS is more challenging than InfoTabS which was expected due to the presence of (a.) bilinguality within the premise-hypothesis pair, and (b.) the low resource nature of Indic languages.**

Figure 3 reports the consistency of predictions of MuRIL on the \(\alpha_{1}\) test set of HindiEI-InfoTabS when compared against that of RoBERTaLARGE on the \(\alpha_{1}\) test set of InfoTabS. We observe that MuRIL behaves noticeably different than RoBERTaLARGE. MuRIL disagrees with RoBERTaLARGE on 47\(\%\) of examples with the Contradiction and Entailment labels. However, for Neutral labels, it only disagrees on around 36\(\%\) of the examples. Moreover, from our discussion in SS5.4, we observe that MuRIL outperforms RoBERTaLARGE on certain reasoning categories.

However, the models fine-tuned on EI-InfoTabS broadly mimic the performance of RoBERTaLARGE on InfoTabS. Figure 4 presents the confusion matrix of MuRIL's predictions on the \(\alpha_{1}\) test set of Hindi. We observe a similar distribution across all Indic languages. As noted in Gupta et al. (2020), MuRIL also tends to predict Neutral hypotheses with the highest confidence as they mostly contain out of table or subjective information terms. Moreover, both models confuse Entailment with Contradiction inference label and vice-versa. We observe that the model predictions on EI-InfoTabS is similar to RoBERTaLARGE predictions on InfoTabS.

## 6 Further Discussion

Ei-InfoTabS is the first Tabular NLI dataset in the Indic context which enables preliminary studies in this field. Moreover, it introduces bilinguality for fact verification scenarios which is of huge significance in low resource contexts. It motivates the development of cross-lingual reasoning models, and helps in evaluation of robustness of multilingual models. For instance, our experiments on EI-InfoTabS clearly indicate

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model (Rep)** & Dev & \(\alpha_{1}\) & \(\alpha_{2}\) & \(\alpha_{3}\) \\ \hline BERTa (BPR) & 63.00 & 63.54 & 52.57 & 48.17 \\ RoBERTa (TabFact) & 68.06 & 66.7 & 56.87 & 55.26 \\ RoBERTa (BPR) & 76.42 & 75.29 & 66.50 & 64.26 \\ RoBERTa (TabFact) & 77.61 & 75.06 & 69.02 & 64.61 \\ Human & **79.78** & **84.04** & **83.88** & **79.33** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The human benchmarks and several baselines on evaluation set of InfoTabS as reported in Gupta et al. (2020) (TabFact) and Neerajá et al. (2021) (BPR). Here subscript X\({}_{\text{L}}\) and X\({}_{\text{B}}\) represent X model L: Large and B: Base versions respectively.

Figure 4: Confusion Matrix for the predictions of our best performing model, MuRIL (Multilingual-Train), on the Hindi \(\alpha_{1}\) set from EI-InfoTabS.

 

that MuRIL is a significantly more robust multilingual model when compared to mBERT as it is able to generalize its reasoning ability across all Indic languages.

Although, we explain how machine translation doesn't affect the semantics of the hypotheses, it does come with a few challenges. We identified a few instances wherein the IndicTrans model translates named entities, instead of transliterating them. This is observed only, but not always, when a named entity has an English dictionary word in it. For instance, "_Death Proof_", name of a movie, gets translated and not transliterated in two out of nine hypotheses containing the phrase. This is mostly observed in the _Movies_ category. However, this doesn't affect our reasoning models and they perform on par on this category when compared with RoBERTaLARGE's performance on InfoTabS. This is so because such translations when shallow parsed indicate that the translated entity still acts as the Noun Phrase in the sentence. This helps the translation, though technically imperfect, retain the intended semantic structure.

## 7 Related Work

Tabular Reasoning.tabular NLI has been of keen interest recently. Datasets like TabFact Chen et al. (2020), InfoTaBS Gupta et al. (2020) were the first resources on TNLI and they enabled a fine-grained examination of the task. Beyond NLI, there has been a thorough examination of various other NLP tasks on semi-structured data. For instance, question answering Abbas et al. (2016); Chen et al. (2020); Zayats et al. (2021); Oguz et al. (2020); Chen et al. (2021), and others), semantic parsing and retrieval Krishnamurthy et al. (2017); Sun et al. (2016); Pasupat and Liang (2015); Lin et al. (2020), and others), tabular probing Gupta et al. (2021), generative tasks including table-to-text Parikh et al. (2020); Nan et al. (2021); Yoran et al. (2021); Chen et al. (2020),d, and others). Other works have explored creating task-independent representations for Wikipedia infoboxes Herzig et al. (2020); Yin et al. (2020); Zhang et al. (2020); Iida et al. (2021); Pramanick and Bhattacharya (2021); Glass et al. (2021), and others), and boosting tabular reasoning by pre-training and external knowledge incorporation Neeraja et al. (2021); Varun et al. (2022), and others).

Multilingual Models.multilingual, and specifically Cross-Lingual transfer Deshpande et al. (2021); Patil et al. (2022), and other), has been widely discussed in the context of low resource languages. Several datasets Conneau et al. (2018); Yang et al. (2019); Ponti et al. (2020); Artetxe et al. (2020); Nivre et al. (2016); Lewis et al. (2021), and others), benchmarks and leaderboards Hu et al. (2020); Liang et al. (2020); Ruder et al. (2021); Khanuja et al. (2021), and others), and evaluation frameworks Tarunesh et al. (2021); K. et al. (2021); Srinivasan et al. (2021) have emerged which focus entirely on evaluation of multilingual NLU. Further, multilingual language models have been developed for (a.) Natural Language Understanding Devlin et al. (2019); Conneau and Lample (2019); Conneau et al. (2020); Chi et al. (2021); Chung et al. (2021), and others), (b.) and Natural Language Generation Xue et al. (2021); Fan et al. (2021), and others).

Indic Resources.indic NLP, recently, has seen a recent surge in the number of datasets Ramesh et al. (2022); Roark et al. (2020); Haddow and Kirefu (2020); Abadji et al. (2022); Kolluru et al. (2021), and others), multilingual models Dabre et al. (2021); Kakwani et al. (2020); Khanuja et al. (2021), and others), toolkits Arora (2020); Bhat et al. (2015); Jain et al. (2020), and others), translation systems Ramesh et al. (2022), and dedicated benchmarks for evaluation Kakwani et al. (2020); Krishna et al. (2021). This has enabled the Indian NLP research community to construct competent models for a variety of challenging NLP tasks.

## 8 Conclusion

We motivate and introduce the bilingual tabular NLI for fact verification tasks, and release EIRInfoTabS- a first of its kind tabular NLI dataset for making inferences in 11 Indic languages over English tabular data. Our robust quality estimation experiments show that the machine translated datasets closely preserve the semantics of the source and are fluent. We show that pre-trained multilingual models find this task challenging, however, still perform close to the benchmarks on InfoTabS with Translate-test and Translate-train providing good performance. The analysis also shows the similarity of inference capabilities across languages. The dataset offers immense potential as it opens up avenues in (a) multilingual tabular NLI, (b) bilingual claim verification, (c) and evaluation of multilingual models 

Ethical Considerations

In terms of demographic and socioeconomic characteristics, we attempted to establish a balanced, bias-free dataset. The EI-InfoTabS dataset is derived from the InfoTabS dataset, which is devoid of bias. The only possible source of prejudice can be the translation pipeline. Our qualitative analysis indicates that translation quality is reasonably good and there aren't any observable biases like gender in the translation. The dataset is intended and useful for studying language model representations in a cross-lingual and structured data setting. The paper points out that low-resource languages can benefit from reasoning over structured data in other languages. This is a relatively new research topic and further work will help understand limitations as well as uncover new directions. Hence, we recommend the use of this dataset at this point exclusively for scholarly, non-commercial purposes.

## Acknowledgement

We thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. The authors also thank Manila Devaraj, Arka Singha, Anagh Chattopadhyay, Maitrey Mehta, Souvik Banerjee, Rupambara Padhi, Jayant Duneja, Nithila Prakash, Tanvi Kamble, Anirudh Palutla for helping with annotations for quality estimation. Additionally, we appreciate the inputs provided by Vivek Srikumar and Ellen Riloff. Vivek Gupta acknowledges support from Bloomberg's Data Science Ph.D. Fellowship.

## References

* Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoit Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. _arXiv e-prints_, page arXiv:2201.06642.
* Abbas et al. (2016) Faheem Abbas, M. K. Malik, M. Rashid, and Rizwan Zafar. 2016. Wikiqa -- a question answering system on wikipedia using freebase, dbpedia and infobox. _2016 Sixth International Conference on Innovative Computing Technology (INTECH)_, pages 185-193.
* Adar et al. (2009) Eytan Adar, Michael Skinner, and Daniel S Weld. 2009. Information arbitrage across multi-lingual wikipedia. In _Proceedings of the Second ACM International Conference on Web Search and Data Mining_, pages 94-103.
* Agirre et al. (2016) Eneko Agirre, Carmene Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 497-511, San Diego, California. Association for Computational Linguistics.
* Arora (2020) Gaurav Arora. 2020. iNLTK: Natural language toolkit for indicle languages. In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_, pages 66-71, Online. Association for Computational Linguistics.
* Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Bao et al. (2012) Patti Bao, Brent Hecht, Samuel Carton, Mahmood Quaderi, Michael Horn, and Darren Gergle. 2012. Omnipedia: bridging the wikipedia language gap. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_, pages 1075-1084.
* Bhat et al. (2015) Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tammewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015. litt-h system submission for fire2014 shared task on transliterated search. In _Proceedings of the Forum for Information Retrieval Evaluation_, FIRE '14, pages 48-53, New York, NY, USA. ACM.
* Chen et al. (2021) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In _International Conference on Learning Representations_.
* Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7929-7942, Online. Association for Computational Linguistics.
* Chen et al. (2020b) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-based fact verification. In _International Conference on Learning Representations_.
* Chen et al. (2020c) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020c. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. _Findings of EMNLP 2020_ 

Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020d. Logic2Text: High-fidelity natural language generation from logical forms. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2096-2111, Online. Association for Computational Linguistics.
* Chi et al. (2021) Zewen Chi, Shaohian Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. 2021. Xlm-e: Cross-lingual language model pre-training via electra. _CoRR_, abs/2106.16138.
* Chung et al. (2021) Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In _International Conference on Learning Representations_.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. _Advances in Neural Information Processing Systems_, 32:7059-7069.
* CONNEAU and Lample (2019) Alexis CONNEAU and Guillaume Lample. 2019. Cross-lingual language model pretraining. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XnL! Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Dabre et al. (2021) Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, and Pratyush Kumar. 2021. Indicbart: A pre-trained model for natural language generation of indic languages.
* Deshpande et al. (2021) Am meet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2021. When is bert multilingual? isolating crucial ingredients for cross-lingual transfer. _arXiv preprint arXiv:2110.14782_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Manddeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. _Journal of Machine Learning Research_, 22(107):1-48.
* Glass et al. (2021) Michael Glass, Mustafa Canim, Alfio Gliozzo, Sanem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1212-1224, Online. Association for Computational Linguistics.
* Graham et al. (2013) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 33-41, Sofia, Bulgaria. Association for Computational Linguistics.
* Gupta et al. (2021) Vivek Gupta, Riyaz A Bhat, Atreya Ghosal, Manish Srivastava, Maneesh Singh, and Vivek Srikumar. 2021. Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning. _arXiv preprint arXiv:2108.00578_.
* Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2309-2324, Online. Association for Computational Linguistics.
* Guzman et al. (2019) Francisco Guzman, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. _arXiv preprint arXiv:1902.01382_.
* a collection of parallel corpora of languages of india.
* A collection of parallel corpora of languages of india. _CoRR_, abs/2001.09907.
* Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Muller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4320-4333, Online. Association for Computational Linguistics.

 

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In _International Conference on Machine Learning_, pages 4411-4421. PMLR.
* Iida et al. (2021) Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: Pretrained representations of tabular data. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3446-3456, Online. Association for Computational Linguistics.
* Jain et al. (2020) Kushal Jain, Adwait Deshpande, Kumar Shridhar, Felix Laumann, and Ayushman Dash. 2020. Indic transformers: An analysis of transformer language models for indian languages.
* Jiang and Marhee de Marneffe (2019) Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Evaluating BERT for natural language inference: A case study on the CommitmentBank. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6086-6091, Hong Kong, China. Association for Computational Linguistics.
* Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293, Online. Association for Computational Linguistics.
* K et al. (2021) Karthikeyan K, Aalok Sathe, Somak Aditya, and Monojit Choudhury. 2021. Analyzing the effects of reasoning types on cross-lingual transfer performance.
* Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokuul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratvush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4948-4961, Online. Association for Computational Linguistics.
* Khanuja et al. (2021a) Simran Khanuja, Diskha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021a. Murli: Multilingual representations for indian languages.
* Khanuja et al. (2021b) Simran Khanuja, Melvin Johnson, and Partha Talukdar. 2021b. MergeDistill: Merging language models using pre-trained distillation. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2874-2887, Online. Association for Computational Linguistics.
* Khemchandani et al. (2021) Yash Khemchandani, Sarvesh Mehtunai, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1312-1323, Online. Association for Computational Linguistics.
* Kolluru et al. (2021) Keshav Kolluru, Martin Rezk, Pat Verga, William W Cohen, and Partha Talukdar. 2021. Multilingual fact linking. _arXiv preprint arXiv:2109.14364_.
* Krishna et al. (2021) Kalpesh Krishna, Deepak Nathani, Xavier Garcia, Bidisha Samanta, and Partha Talukdar. 2021. Fewshot controllable style transfer for low-resource settings: A study in indian languages. _arXiv preprint arXiv:2110.07385_.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1516-1526, Copenhagen, Denmark. Association for Computational Linguistics.
* Kulesza and Taskar (2011) Alex Kulesza and Ben Taskar. 2011. k-dpps: Fixed-size determinantal point processes. In _ICML_.
* Kumaran et al. (2010) A Kumaran, Naren Datha, B Ashok, K Saravanan, Anil Ande, Ashwani Sharma, Sridhar Vedantham, Vidya Natampally, Vikram Dendi, and Sandor Maurice. 2010. Wikibabel: A system for multilingual wikipedia content. In _American Machine Translation Association (AMTA) Workshop_. Citeseer.
* Kunchukuttan (2020) AnoopKunchukuttan. 2020. The IndicNLP Library. [https://github.com/anoopkunchukuttan/indic](https://github.com/anoopkunchukuttan/indic)\({}_{mlp}\)library/blob/master/docs/indicinlp.pdf.
* Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and R Soricut. 2019. A lite bert for self-supervised learning of language representations. _arXiv preprint arXiv:1909.11942_.
* Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1000-1008, Online. Association for Computational Linguistics.
* Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Davin Jiang, Guihong Cao, et al. 2020. Xylue: 

A new benchmark dataset for cross-lingual pretraining, understanding and generation. _arXiv preprint arXiv:2004.01401_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4870-4888, Online. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. _Transactions of the Association for Computational Linguistics_, 8:726-742.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Minhas et al. (2022) Bhavnick Minhas, Anant Shankhdhar, Vivek Gupta, Divyanshu Aggrawal, and Shuo Zhang. 2022. Xinftotabs: Evaluating multilingual tabular natural language inference. In _Proceedings of the Fifth Workshop on Fact Extraction and VERification (FEVER)_. Association for Computational Linguistics.
* Miquel-Ribe and Lanialdo (2020) Marc Moiquel-Ribe and David Lanialdo. 2020. The wikipedia diversity observatory: A project to identify and bridge content gaps in wikipedia. In _Proceedings of the 16th International Symposium on Open Collaboration_, pages 1-4.
* Nan et al. (2021) Lingvong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiaun Hsieh, Xiangru Tang, Aadir Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethius Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Opendomain structured data record to text generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 432-447, Online. Association for Computational Linguistics.
* Neeraja et al. (2021) J. Neeraja, Vivek Gupta, and Vivek Srikumar. 2021. Incorporating external knowledge to enhance tabular reasoning. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2799-2809, Online. Association for Computational Linguistics.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsafraty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Oguz et al. (2020) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Pestherlievy, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020. Unified open-domain question answering with structured and unstructured knowledge. _arXiv preprint arXiv:2012.14610_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1173-1186, Online. Association for Computational Linguistics.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1470-1480, Beijing, China. Association for Computational Linguistics.
* Patil et al. (2022) Vaidehi Patil, Partha Talukdar, and Sumita Sarawagi. 2022. Overlap-based vocabulary generation improves cross-lingual transfer among related languages. _arXiv preprint arXiv:2203.01976_.
* Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2362-2376, Online. Association for Computational Linguistics.
* Pramanick and Bhattacharya (2021)Aniket Pramanick and Indrajit Bhattacharya. 2021. Joint learning of representations for web-tables, entities and types using graph convolutional network. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 

1197-1206, Online. Association for Computational Linguistics.
* Ramesh et al. (2022) Gowtham Ramesh, Sumanath Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022. Samannatur: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages. Transactions of the Association for Computational Linguistics, 10:145-162.
* Roark et al. (2020) Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johnry, Isin Demirsahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: the Dakshina dataset. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 2413-2423, Marseille, France. European Language Resources Association.
* Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10215-10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Srinivasan et al. (2021) Anirudh Srinivasan, Sunayana Siitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. 2021. Predicting the performance of multilingual nlp models. _arXiv preprint arXiv:2110.08875_.
* Sun et al. (2016) Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In _Proceedings of the 25th International Conference on World Wide Web_, pages 771-782.
* Tarunesh et al. (2021) Ishan Tarunesh, Somak Aditya, and Monojit Choudhury. 2021. Lonli: An extensible framework for testing diverse logical reasoning capabilities for nli. _arXiv preprint arXiv:2112.02333_.
* Taylor (1953) Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. _Journalism quarterly_, 30(4):415-433.
* building open translation services for the world. In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_, pages 479-480, Lisboa, Portugal. European Association for Machine Translation.
* Varun et al. (2022) Yerram Varun, Aayush Sharma, and Vivek Gupta. 2022. Trans-kblstm: An external knowledge enhanced transformer bilstm model for tabular reasoning. In _Proceedings of Deep Learning Inside Out (DeeLO): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_. Association for Computational Linguistics.
* Xue et al. (2021) Liting Xue, Noah Constant, Adam Roberts, Mihir Kale, Ramil Al-Rfou, Aditya Siddharn, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
* Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8413-8426, Online. Association for Computational Linguistics.
* Moran et al. (2021) Ori Moran, Alon Talmor, and Jonathan Berant. 2021. Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills. _arXiv preprint arXiv:2107.07261_.
* Zayats et al. (2021) Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021. Representations for question answering from documents with tables and text. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2895-2906, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020. Table fact verification with structure-aware transformer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1624-1629, Online. Association for Computational Linguistics.
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.

## Appendix A Details: Table Representation

1. **Premise as Paragraph**: Chen et al. (2020b), Gupta et al. (2020) employ universal templates to construct close to natural language sentences for isolated cells in a 

row, and then, concatenate them to obtain a single paragraph representation. Gupta et al. (2020) suggest constructing sentences of the form _"The k of t is v"_ for a cell having key k, value V in a table with title t. E.g. in figure 1 for the row _Born_ the premise sentence would be _"The born of Joe Strummer is 21 August 1952 (1952-08-21) Ankara, Turkey"_ However, Neeraja et al. (2021) identify that such templates can often lead to ungrammatical sentences and propose the _Better Paragraph Representation_ (BPR) approach. BPR utilises type specific templates based on the entity type of a key, and the overall category of the table itself resulting in grammatical sentences. Neeraja et al. (2021) note a significant increase in performance while employing BPR over the universal template. We adopt BPR as one of our representation approaches. E.g. for same _Born_ key in figure 1 the premise sentence with BPR representation would be _"Joe Strummer was born on August 21, 1952 (1952-08-21) at Ankara, Turkey"_
2. **Premise as Structure**: Unlike the natural language like _Premise as Paragraph_ representations, here, we try to represent the row as structural text as proposed by Chen et al. (2020b). Every isolated cell in a row is represented as _"k : v"_ where k is the key, and v is the value of the cell. A row's structural representation is a semi-colon ";" separated sequence of the structural representations of all the isolated cells in that row. E.g. for the same _Born_ key in figure 1 the premise sentence will be represented as _"Born : August 21, 1952 (1952-08-21)_, Ankara, Turkey"_ Oscar Corpus 8 Tensorflow Datasets

## Appendix B Details: Multilingual Models

Indic Specific:This class of multilingual models are pre-trained entirely on Indic language data along with English. We use _MuRIL Base_Khanuja et al. (2021a), and _IndicBERT_ (kak) pre-trained multilingual models. _MuRIL_ is a _BERT_Devlin et al. (2019) based model trained with _Masked Language Modeling_Taylor (1953) and _Translation Language Modeling_CONNEAU and Lample (2019) objectives. It is trained on (a.) Common Crawl OSCAR corpus7 and Wikipedia8 monolingual data for 16 Indic languages along with the English language, (b.) PMIndia Haddow and Kirefu (2020b) along with other in-house parallel corpora, (c.) and the Dakshina Dataset Roark et al. (2020) along with other parallel in-house transliterated corpora. _IndicBERT_ is an _ALBERT_Lan et al. (2019) based model trained on _IndicCorp_ (kak).

Footnote 7: footnotemark: Oscar Corpus 8 TensorFlow Datasets

Generic:This class of multilingual models are pre-trained on a wide array of languages from around the world. We use _mBERT-cased_Devlin et al. (2019) and _XLM-RoBERta_ (con) pre-trained multilingual models.

## Appendix C Human Evaluation Strategy

We requested our colleagues who are native speakers and are proficient in English to help us with this task while disclosing the intentions. We provide them with instructions adopted from the Direct Assessment Graham et al. (2013) strategy for low resource machine translation in Guzman et al. (2019). We sample 50 pairs of source, translation pairs and ask the annotators to provide a continuous score between 0 to 100. 0-10 range represents a translation that is completely incorrect and inaccurate. 70-90 range represents a translation that closely preserves the meaning of the source sentence while the 90-100 range represents a perfect translation.

\begin{table}
\begin{tabular}{l|r r r r} \hline \multicolumn{4}{l}{**Hyper Parameter**} & \multicolumn{1}{l}{**XLM-RoBERta****IndicBERT****MueRIL–base-cased****mBERT-cased****n**} \\ \hline
**Initial Learning Rate** & [1e-4,1e-9] & 5e-5 & 5e-5 & 5e-5 \\
**Batch Size** & 128 & 128 & 128 & 128 \\
**Weight Decay** & 0.01 & 0.01 & 0.01 & 0.01 \\
**Max Seq Length** & 128 & 128 & 128 & 128 \\
**Model Size** & 278M & 33.7M & 237M & 177M \\
**Warmup Steps** & 500 & 500 & 500 & 500 \\ \hline \end{tabular}
\end{table}
Table 5: Hyper Parameters used for Fine-Tuning the corresponding multilingual models.

 

Model Hyper-Parameters

Table 5 reports the hyper-parameters used for fine-tuning the multilingual models on EI-InfoTabS. We use the Huggingface Transformers9 library to script these experiments. We were unable to successfully converge XLM-RoBERTa in multiple runs spanning a distinctive set of hyper-parameters. Figure 5 shows the loss plots for XLM-RoBERTa and mBERT when fine-tuned on EI-InfoTabS. It is distinctively visible that XLM-RoBERTa is unable to converge on EI-InfoTabS on a significant amount of steps unlike mBERT.

Footnote 9: Transformer Hugging Face

Fine-Tuning Settings.We follow the conventionally used pipeline for fine-tuning _BERT_ for Sequence Classification Jiang and de Marneffe (2019). We concatenate the premise and the hypothesis strings using a [SEP] token in between them, prepend this sequence with a [CLS] token, tokenize this sequence using the pre-trained tokenizer for the respective model, and provide the obtained sequence as input to the pre-trained model. We attach a three-way classification head with cross-entropy loss on top of the pooled output obtained from the previous step. With an initial learning rate of 5e-05 with AdamW optimizer Loshchilov and Hutter (2018), we fine-tune each model on 4 1080Ti GPUs with a batch size of 32 per GPU over 10 epochs.

## Appendix E Performance on the \(\alpha_{2}\) and \(\alpha_{3}\) Adversarial Sets

Tables 6 and 7 report the results for the adverserial test sets \(\alpha_{2}\) and \(\alpha_{3}\) respectively using the BPR linearization method.

 



\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline \hline
**Strategy** & **Model** & bn & hi & gu & pa & mr & te & ta & ml & kn & as or & **ModAvg** \\ \hline  & mBERT & 51 & 52 & 48 & 49 & 48 & 48 & 49 & 48 & 49 & 47 & 36 & 48 \\
**Translate-Train** & IndicBERT & 46 & 44 & 44 & 44 & 46 & 46 & 45 & 45 & 46 & 34 & 45 & 44 \\  & MuRL & \(\mathbf{56^{*}}\) & \(\mathbf{56^{*}}\) & \(\mathbf{52}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{52}\) & \(\mathbf{55}\) & \(\mathbf{53}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{53}\) & \(\mathbf{54}\) \\  & LnAvg & 51 & 51 & 48 & 49 & 49 & 49 & 50 & 48 & 50 & 45 & 45 & 49 \\ \hline  & mBERT & 47 & 47 & 44 & 45 & 44 & 44 & 45 & 43 & 46 & 41 & 34 & 44 \\
**Translate-Test** & IndicBERT & 41 & 38 & 37 & 38 & 39 & 38 & 38 & 38 & 38 & 42 & 39 & 39 \\  & MuRL & \(\mathbf{52}\) & \(\mathbf{51}\) & \(\mathbf{51}\) & \(\mathbf{50}\) & \(\mathbf{50}\) & \(\mathbf{51}\) & \(\mathbf{51}\) & \(\mathbf{53^{*}}\) & \(\mathbf{53^{*}}\) & \(\mathbf{53^{*}}\) & \(\mathbf{49}\) & 50 & 51 \\  & LnAvg & 47 & 45 & 44 & 44 & 44 & 45 & 44 & 45 & 44 & 41 & 44 \\ \hline  & mBERT & 52 & 52 & 50 & 51 & 50 & 50 & 51 & 49 & 51 & 50 & 37 & 49 \\
**Bilingual-Train** & IndicBERT & 45 & 45 & 45 & 48 & 47 & 45 & 46 & 45 & 46 & 44 & 45 & 45 \\  & MuRIL & \(\mathbf{56^{*}}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{56^{*}}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{53}\) & \(\mathbf{54}\) & \(\mathbf{56^{*}}\) & \(\mathbf{54}\) & \(\mathbf{53}\) & \(\mathbf{54}\) \\  & LnAvg & 51 & 51 & 50 & 51 & 50 & 50 & 50 & 49 & 51 & 49 & 45 & 50 \\ \hline  & mBERT & 50 & 51 & 51 & 50 & 50 & 51 & 49 & 48 & 50 & 48 & 35 & 48 \\
**Multilingual-Train** & IndicBERT & 46 & 46 & 46 & 47 & 45 & 45 & 44 & 44 & 45 & 45 & 44 & 45 \\  & MuRIL & \(\mathbf{55}\) & \(\mathbf{55}\) & \(\mathbf{56^{*}}\) & \(\mathbf{55}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{55}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{55}\) \\  & LnAvg & 50 & 51 & 51 & 51 & 50 & 50 & 49 & 49 & 50 & 49 & 45 & 50 \\ \hline  & mBERT & 55 & 55 & 55 & 53 & 53 & 54 & 54 & 54 & 54 & 54 & 53 & 54 \\
**EnTranslate-Test** & IndicBERT & 48 & 47 & 47 & 48 & 47 & 47 & 46 & 46 & 46 & 47 & 47 & 47 \\  & MuRIL & \(\mathbf{56^{*}}\) & \(\mathbf{55}\) & \(\mathbf{55}\) & \(\mathbf{56^{*}}\) & \(\mathbf{54}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{55}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{55}\) \\  & LnAvg & 53 & 52 & 52 & 52 & 51 & 52 & 51 & 52 & 51 & 52 & 51 & 52 & 52 \\ \hline  & mBERT & 45 & 45 & 44 & 45 & 42 & 42 & 45 & 41 & 45 & 40 & 35 & 43 \\
**Translate-Train-X** & IndicBERT & 40 & 38 & 37 & 36 & 37 & 39 & 38 & 40 & 38 & 34 & 38 & 38 \\  & MuRIL & \(\mathbf{54^{*}}\) & \(\mathbf{53}\) & \(\mathbf{52}\) & \(\mathbf{54^{*}}\) & \(\mathbf{53}\) & \(\mathbf{52}\) & \(\mathbf{53}\) & \(\mathbf{51}\) & \(\mathbf{52}\) & \(\mathbf{54^{*}}\) & \(\mathbf{52}\) & \(\mathbf{53}\) \\  & LnAvg & 46 & 45 & 44 & 45 & 44 & 44 & 45 & 44 & 45 & 42 & 41 & 44 \\ \hline  & mBERT & 47 & 47 & 46 & 46 & 46 & 46 & 46 & 44 & 46 & 47 & 43 & 46 \\
**Bilingual-Train-X** & IndicBERT & 39 & 40 & 39 & 39 & 39 & 39 & 40 & 39 & 40 & 39 & 40 & 39 & 39 \\  & MuRIL & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{55^{*}}\) & \(\mathbf{54}\) & \(\mathbf{53}\) & \(\mathbf{53}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) & \(\mathbf{54}\) \\  & LnAvg & 46 & 47 & 46 & 47 & 47 & 47 & 46 & 46 & 46 & 46 & 47 & 45 & 46 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance in terms of accuracy when evaluated on the \(\alpha_{2}\) test set. Higher value implies better performance. Here, LnAvg represents the average accuracy for a language across all models, while ModAvg represents the average accuracy of a model across all languages. A value in Purple represents the best accuracy for that model across all languages. An underlined value in Blue represents the best accuracy for that language across all models. A value in Green with an asterisk(\({}^{*}\)) represents the cases where language-wise and model-wise values coincide. As we fine-tune on a specific Indic language in the fine-tuning strategies Translate-Train-X and Bilingual-Train-X, we report the training average of the concerned language. We do not include the results of XLM-RoBERTa as the model fails to converge on these experiments on multiple runs with a distinct set of hyper-parameters as explained in Appendix §D.





\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline
**Strategy** & **Model** & bn & hi & gu & pa & mr & te & ta & ml & kn & as & or & **ModAvg** \\ \hline  & mBERT & 47 & 48 & 46 & 45 & 46 & 46 & 47 & 46 & 43 & 35 & 45 \\
**Translate-Train** & IndicBERT & 43 & 44 & 40 & 42 & 41 & 42 & 43 & 39 & 43 & 33 & 41 & 41 \\  & MuRIL & **52** & **54*** & **52** & **53** & **52** & **51** & **52** & **51** & **54** & **51** & **50** & 52 \\  & LnAvg & 47 & 49 & 46 & 47 & 46 & 47 & 47 & 45 & 48 & 43 & 42 & 46 \\ \hline  & mBERT & 44 & 46 & 43 & 45 & 43 & 45 & 46 & 43 & 44 & 39 & 33 & 43 \\
**Translate-Test** & IndicBERT & 36 & 36 & 35 & 34 & 35 & 35 & 35 & 35 & 35 & 36 & 34 & 35 \\  & MuRIL & **53*** & **52** & **51** & **51** & **51** & **51** & **50** & **51** & **50** & **49** & 51 \\  & LnAvg & 45 & 44 & 43 & 43 & 43 & 43 & 43 & 44 & 43 & 42 & 39 & 43 \\ \hline  & mBERT & 49 & 49 & 47 & 47 & 48 & 46 & 49 & 47 & 49 & 46 & 34 & 46 \\
**Bilingual-Train** & IndicBERT & 42 & 41 & 42 & 42 & 40 & 42 & 41 & 44 & 42 & 42 & 41 & 42 \\  & MuRIL & 52 & **53*** & **52** & **51** & **51** & **51** & **52** & **51** & **53** & **51** & **52** \\  & LnAvg & 48 & 48 & 47 & 47 & 46 & 45 & 47 & 47 & 47 & 42 & 47 \\ \hline  & mBERT & 47 & 47 & 46 & 47 & 46 & 45 & 46 & 45 & 47 & 46 & 36 & 45 \\
**Multilingual-Train** & IndicBERT & 42 & 41 & 42 & 40 & 40 & 42 & 41 & 41 & 40 & 42 & 40 & 41 \\  & MuRIL & **54*** & **54*** & **52** & **53** & **52** & **53** & **53** & **54*** & **54*** & **53** & **53** \\  & LnAvg & 47 & 47 & 47 & 47 & 47 & 46 & 47 & 47 & 46 & 47 & 47 & 43 & 46 \\ \hline  & mBERT & 51 & 52 & 50 & 51 & 50 & 50 & 52 & 49 & 50 & 50 & 49 & 50 \\
**EnTranslate-Test** & IndicBERT & 46 & 48 & 46 & 46 & 46 & 46 & 46 & 45 & 46 & 45 & 44 & 46 \\  & MuRIL & **53*** & **52** & **51** & **51** & **51** & **50** & **51** & **50** & **50** & **51** & **50** & **48** & **51** \\  & LnAvg & 50 & 51 & 49 & 49 & 49 & 49 & 49 & 48 & 49 & 48 & 47 & 49 \\ \hline  & mBERT & 44 & 44 & 43 & 44 & 43 & 44 & 44 & 43 & 44 & 41 & 34 & 42 \\
**Translate-Train-X** & IndicBERT & 37 & 36 & 36 & 35 & 36 & 37 & 37 & 37 & 36 & 33 & 36 & 36 \\  & MuRIL & **51** & **52*** & **52*** & **52*** & **52*** & **51** & **51** & **52*** & **51** & **51** & **51** \\  & LnAvg & 44 & 44 & 43 & 44 & 43 & 44 & 44 & 45 & 44 & 42 & 40 & 43 \\ \hline  & mBERT & 45 & 44 & 44 & 44 & 44 & 44 & 45 & 44 & 45 & 44 & 37 & 44 \\
**Bilingual-TrainX** & IndicBERT & 37 & 36 & 36 & 36 & 36 & 37 & 38 & 37 & 37 & 37 & 36 & 37 \\  & MuRIL & **51** & **51** & **51** & **51** & **51** & **51** & **51** & **52*** & **52*** & **51** \\  & LnAvg & 44 & 44 & 43 & 44 & 44 & 44 & 45 & 44 & 44 & 42 & 44 & 42 \\ \hline  & mBERT & 45 & 44 & 44 & 44 & 44 & 44 & 45 & 44 & 45 & 44 & 37 & 44 \\
**Bilingual-TrainX** & IndicBERT & 37 & 36 & 36 & 36 & 36 & 37 & 38 & 37 & 37 & 37 & 36 & 37 \\  & MuRIL & **51** & **51** & **51** & **51** & **51** & **51** & **51** & **52*** & **52*** & **51** \\  & LnAvg & 44 & 44 & 44 & 44 & 44 & 44 & 44 & 44 & 44 & 44 & 42 & 44 & 44 \\ \hline \end{tabular}
\end{table}
Table 7: Performance in terms of accuracy when evaluated on the \(\alpha_{3}\) test set. Higher value implies better performance. Here, LnAvg represents the average accuracy for a language across all models, while ModAvg represents the average accuracy of a model across all languages. A value in Purple represents the best accuracy for that model across all languages. An underlined value in Blue represents the best accuracy for that language across all models. A value in Green with an asterisk(*) represents the cases where language-wise and model-wise values coincide. As we fine-tune on a specific Indic language in the fine-tuning strategies Translate-Train-X and Bilingual-Train-X, we report the training average of the concerned language. We do not include the results of XLM-RoBERTa as the model fails to converge on these experiments on multiple runs with a distinct set of hyper-parameters as explained in Appendix SSD.