Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention
Authors: Negar Foroutan
Mohammadreza Banaei
Karl Aberer
Antoine Bosselut
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-lingual
Multilingual
Reasoning

Abstract: In this work, we study whether multilingual language models (MultiLMs) can transfer logical reasoning abilities to other languages when they are fine-tuned for reasoning in a different language. We evaluate the cross-lingual reasoning abilities of MultiLMs in two schemes: (1) where the language of the context and the question remain the same in the new languages that are tested (i.e., the reasoning is still monolingual, but the model must transfer the learned reasoning ability across languages), and (2) where the language of the context and the question is different (which we term code-switched reasoning). On two logical reasoning datasets, RuleTaker and LeapOfThought, we demonstrate that although MultiLMs can transfer reasoning ability across languages in a monolingual setting, they struggle to transfer reasoning abilities in a code-switched setting. Following this observation, we propose a novel attention mechanism that uses a dedicated set of parameters to encourage cross-lingual attention in code-switched sequences, which improves the reasoning performance by up to 14% and 4% on the RuleTaker and LeapOfThought datasets, respectively.

Breaking the Language Barrier:
Improving Cross-Lingual Reasoning with Structured Self-Attention
Negar Foroutan", Mohammadreza Banaci', Karl Aberer, Antoine Bosselut
EPFL
{negar foroutan, www.mmadreaaaa.... nohmaara banaei, antoine. antoiiee
Abstract
Context:
Monolingual:
Q1: The cat sees the rebbit.
The rebbit is rough. the cat likes the rabbit then the cat the rabbit. The tiger is round. The likes the rabbit.
Q2: The does visit the tiger.
this work, we study whether multilingual language models (Multil.s) can transfer logi- cal reasoning abilities to other languages when different
Code-Switched:
Le chat voit le lapin.
Q2: Le chat ne visite pas le tigre.
they are fine-tuned for reasoning language. We evaluate the cross- longull rea- soning abilities of MultiLMs in two schemes: (I) where the language of the context question remain the same in the new languages that are tested (i.e,, the reasoning is still mono- lingual, but the model must transfer the learned reasoning ability across languages), and (2) where the language of the context and the ques- tion is different (which reasoning). On two logical reasoning datasets, RuleTaker and LeapOrThought, we demon- strate that although MultiLMS can transfer soning ability across languages in a monolin- gual setting, they struggle to transfer reasoning abilities in code-switched setting. Following this observation, we propose a novel attention mechanism that uses dedicated of param- eters to encourage cross-lingus lingual attention in code-switched sequences, which improves reasoning performance by up 14% and 4% on the RuleTaker and LeapOfThought datasets, respectively.'
example of monolingual and code- switched reasoning. In code-switched reasoning, the context and question are in different languages.
Figure
the
language models (MultiLM) such mBERT (De- vlin et al., 2019), mT5 (Xue al., 2021), and XLM-R (Conneau al., 2020), which have been shown to generalize setting (Pires et al., 2019a; Conneau and Lample, 2019). The cross-lingual transfer is often enabled by fine-tuning the MultiLM on a high-resource lan- guage (typically English) and then evaluating it on other target languages.
code-switched
zero-shot cross-lingual
However, most of the recent efforts reasoning-related tasks have been centered around English, our knowledge of the multilingual rea- soning capabilities of language models remains limited. In this work, we investigate the logical reasoning capabilities of MultiLMs, especially in monolingual and structured code-switched? set- tings (Figure 1). In the monolingual setting, the context and the question are in the same language. In the structured code-switched setting, we refer to a setting where the context and question are in two different languages. The code-switched set- ting can found many realistic scenarios, such as when non-English speakers may ask questions about information that is unavailable in their native language (Asai et 2021).
1 Introduction
Recent studies show that language models (LMs) are capable of logically reasoning over natural lan- guage statements (Clark et al., 2020b), reasoning with their implicit knowledge (Talmor et al., 2020), and performing multi-step reasoning via chain-of- thought prompting when the model size large enough (Wei et al., 2022b; Kojima et al., 2022: Wei et al., 2022a). A separate line of work has focused on pre-training language models on multilingual corpora to enable knowledge transfer across dif- ferent languages. These efforts led to multilingual
For both reasoning settings, conduct exper- iments using the RuleTaker dataset (Clark et al., 2020b), which contains artificial facts and rules, and the LeapOrThought dataset (Talmor al., 2020), which incorporates real-world knowledge
"Equal contribution
'Our code is available at htrps//githu..co/nngga foroutan/muliiinggual wawpgguull--d-sssiiceeeeeee reasoind....snngnggg.......
'Throughout the paper, we will use the terms "structured code-switching" and "code-switching" interchungeably.

---

into the reasoning context. Our results show that although MultiLMs perform well when fine-tuned in different languages (i.e., high in-language per- formance when fine-tuning and testing on the same language), their cross-lingual transfer can vary con- siderably, especially in the code-switched setting. We posit that the lack of code-switched data in MultiLM pre-training data makes fine-tuning code-switched data inconsistent with pre-training. To improve the code-switched reasoning capabil- ities of MultiLMs, we propose two methods. First, we propose a dedicated cross-lingual query matrix (section 4.1) to better model cross-lingual atten- tion when the MultiLMs receive code-switched se- quences as input. This query matrix is pre-trainn pre-trained on unsupervised code-switched data, either shared across all language pairs or specific single one. Then, we propose a structured attention dropout (see section 4. 4),, in which we randomly mask atten- tion between tokens from different languages (i.e., context-question attentions) during training. This masking makes the fine-tuning phase more con- sistent with the pre-training by regularizing cross- lingual attention.
where Le + Lg. Thus, to evaluate the multilingual reasoning ability of MultiLMs, we first define four different evaluation setups based on the language of context or statement: (1) both the context and statement are always in one language (monolingual reasoning): (2) the context is always in one lan- guage, and the statement can be in any language; (3) the context can be in any language, but the state- ment is always in language; and (4) both the context and statement can be in any language.
To have a reasonable baseline to compare with the code-switched setups, we first focus on the monolingual evaluation (1), in which we evaluate the reasoning ability of MultiLMs for nine typolog- ically different languages. Then, by fine-tuning the models on code-switched data, we evaluate their performance for setups (2) and (3) where either the language of the context or the language of the question is different from the training data. This evaluation aims to study the possibility of teach- ing models to reason across languages in a code- switched setting, and to investigate the extent they can transfer their reasoning other code-switched data formats. Finally, hypothesize that in order to succeed in setup (4), the model would have be strong setups (2) and (3). Since our exper- imental results show that the MultiLMs struggle in these two setups, we focus on improving their performance for setups (2) and (3).
By mixing the two methods, we also experiment with an interfered variant of the cross-lingual query, which considerably improves cross- cooss-lnggual gener- alization, especially in code-switched settings. We evaluate our methods for the code-switched setting and show they improve the cross- lingull transfer of MultiLMs by 14% and 4% for the RuleTaker and LeapOfThought datasets, respectively.
3 Multilingual Reasoning
In this section, we describe our evaluation of the logical reasoning capabilities of MultiLMs for monolingual and code-switched settings.
2
Motivation
Most prior work on reasoning with language mod- els remains limited to monolingual (English) sys- tems (Han et al., 2021; Sanyal et al., 2022: Shi et al., 2023; Tang et al., 2023). In this work, we investigate the reasoning abilities of MultiLMs, for- mulating an analysis in formal reasoning that eval- uates MultiLMs on their ability resolve logical statements. Given set of facts and rules con- text (in natural language sentences), the task is to predict whether a given statement is true.
3.1 Analysis Setup
We run experiments two datasets focusing on multi-hop logical reasoning over natural lan- guage knowledge:
RuleTaker. This is a set of five datasets, each constrained by the maximum depth of inference re- quired to prove the facts used in its questions (Clark et al., 2020b). This dataset is generated with the closed-world assumption (CWA), assuming a state- ment is false if it is not provable. Each example consists of facts and rules (i.e., context) and state- ment (more details in Appendix A.1).
In our multilingual reasoning setting, we assume given set of languages = {L,, L2, ..., LN}; ), and define L. and Ly as the context and statement lan- guages, respectively. Typically, MultiLMs are eval- uated in a monolingual setup where T_e = Tag How- ever, if MultiLMs are truly multilingual, we posit that they should also be able to reason in a scenario
LeapOfThought (LoT). This dataset comprises 30K true or false hypernymy inferences, ver- balized using manually written templates (Talmor

---

al., 2020). The hypernymy relations and proper- ties are derived from WORDNET (Fellbaum, 1998) and CONCEPTNET (Speer et al., 2017). This dataset contains two main test sets; EXPLICIT REA- SONING which performs inference over explicit nat- unal language statements, and IMPLICIT REASON- ING where the model must reason by combining the context with missing information that should be implicitly encoded by the model. We create modified version of LoT, and use the IMPLICIT REASONING test set in our evaluation. The dataset modification pipeline and the reason behind using only the IMPLICIT evaluation setting is further dis- cussed in Appendix A.2.
hops), the model's performance drops for both in- language and cross-lingual evaluation settings, and the performance gap between different source lan- guages increases. Moreover, when increasing the depth, zero-shot cross-lingual performance suffers more compared to in-language performance, show- ing that as the complexity of the task increases, the harder it becomes to generalize to other languages.
For the LoT dataset, the model must learn to reason by combining its implicit knowledge of hy- pernyms with the given explicit knowledge. How- ever, the model's performance differs for different languages, suggesting that the model's ability access and use the implicit knowledge is not the same for all languages. We also observe that a lan- guage with high in-language performance does not necessarily have a high zero-shot cross-lingual per- formance. We hypothesize that for some languages, the model starts learning in-language noises that are not generalizable to other languages.
Models. We conduct all our experiments using the cased version of multilingual BERT (mBERT; Devlin al. 2019) and the base version of XLM- R (Conneau al., 2020). We train a binary clas- sifier on top of the model's classification token (e.g., [CLS] in mBERT) to predict whether a given statement/question is true or false. The model's input is /CLS) context [SEP] statement [SEP] and the (CLS) output token is used for the classification. For evaluation, we measure the model's 's accuracy. We use full fine-tuning for these experiments. The random baseline is 50% (binary classification).
We generally observe the same patterns for the XLM-R model (see Appendix B) when finc-tuned on the monolingual Rule'aaker and LoT datasets.
3.3 Reasoning Over Code-Switch Data
When we fine-tune the model using a code- switched data format, the context is in one language and the statement we use English as an anchor language for the con- text (i.e., en-X) or for the statement (i.e., X-en). In the fine-tuning phase, we learn the task using the en-X data format, and evaluate it on both en-X and X-en data formats. The models' average in- language and zero-shot cross-lingual performance are shown in Table
Languages. Both RuleTaker and LoT datasets are only available in English. We translated these two datasets into eight languages using the Google Translate API. We have chosen typologi- cally diverse languages covering different language families: Germanic, Romance, Anyan, and Semitic, and including both high- and medium- resource languages from the NLP perspective. These languages include French (fr), German (de), Chinese (zh), Russian (ru), Spanish (es), Farsi (fa), Italian (it), and Arabic (ar).
another. In our experiments,
For Depth-O of the RuleTaker dataset, mBERT is able to learn the reasoning task almost perfectly for most languages. As the depth of the task in- creases, the performance of the code-switched rea- soning declines. This decline more pronounced at higher depths compared to the monolingual sce- nario. While the model is capable of learning rea- soning within this framework, its zero-shot gener- alization to other code-switched data, such as en-X (where the context language remains English but the statement language changes), is poor. Reason- ing over two languages poses a greater challenge than reasoning within monolingual data due to the need for information alignment across languages. Consequently, the transferability of such tasks to other language pairs becomes more challenging.
3.2 Reasoning Over Monolingual Data
The average in-language and cross-lingual zero- shot performance of mBERT for each source lan- guage are depicted in Table For the cross-lingual zero-shot performance, we first fine-tune models on a single source language, test it on other languages, and then take an average of these results.
On the RuleTaker dataset, the model is able learn the task for the Depth-0 subset nearly perfectly for almost all the languages, exhibit- ing relatively high cross-lingual transfer perfor- mance (~-8%)). However, for models trained on higher depths (i.e,, requiring more reasoning

---

RuleTaker
LeapOrThought
Depth-D
Depth-1
Depts-2
Depth-3
in-lang.
cross-ling.
in-lang.
cross-ling.
in-lang.
cross-ling.
in-lang.
cross-ling.
in-lang.
cross-ling.
ca 100.00
87.96
93.37
73,60
88.00
67.91
88.46
67.13
81.15
62.11
99.40
87.06
90.50
86.64
67.55
65.12
Ea 99.41
90.04
84.64
63.53
64.37
de
89.57
90.77
76.67
85.41
71.57
83,10
70.74
77.11
67.03
ar 09,48
80,30
90.35
72.32
84.81
67.79
82.62
62.21
69.62
67,71
us 99.99
89.68
91.84
76.20
88.16
72:29
85.79
68.75
75.25
64,22
zh 100.00
87.48
92:43
72.46
89.04
68.13
85.94
66.25
84.12
62.32
ru 99.97
89.61
90.54
78.05
86.43
70.88
84.01
67.08
70.60
68.02
93.14
70.43
64.68
Average
99.78
87.67
91.44
74.47
86.93
69.08
84.77
67.07
75.56
65.06
Table I: Monolingual Setting: In-language and cross-lingual zero-shot performance (accuracy) of the mBERT model for the RuleTaker and LeapOrThought datasets. Cross-lingual performance is the average performance of the model being fine-tuned on single source language and then zero-shot transferred other languages.
Taker
LeapOrThought
Depth-0
Depch-1
Depth-2
Depch-3
in-lang
cn-X
X-en
in-lang:
cn-X
X-en
in-lang.
cn-X
X-en
in-lang:
cn-X
X-en
in-lang.
cn-X
X-en
en-fr
53.47
55.28
51.85
51.83
83.26
50.28
71.48
en-fa
97:46
54,04
52.05
87.72
62.17
51.56
70.95
53.64
51.29
62:26
50.95
50.52
74.99
73.82
65.93
en-de
99.53
$4.26
$2.69
88.85
52.67
$1.87
83.97
55.32
52.73
79.08
53:48
51.61
77.60
71.16
65.09
un-ur
85.93
53.73
$2.36
67.05
57.83
$1.92
68.54
55,33
$1.74
61.29
52.78
$0.76
77.09
75.35
64.57
50.91
58.20
55,53
51.72
75.62
72.55
en-ah 100.00
52.68
51.81
92.34
54.70
51.31
81.41
54.92
51.24
68.74
52.83
50.00
84.85
68.92
61.09
en-ru
98.03
58.40
52.06
94.02
39.70
50.64
80.28
57.69
$1.96
73.89
56.26
50.87
76.57
74.64
65.11
cn-it
99.91
56.26
54.68
92.25
88
50.94
85.59
54.58
$1.20
79.50
53.29
$1.11
75.38
70.53
66.90
Average
97.53
55.18
53.18
88.22
36.20
51.38
80.63
55.56
51.89
73.26
53.66
50.86
78.17
72.94
66.39
Table 2: Code-Switched Setting: In-language and cross-lingual zero-shot performance (accuracy) of the mBERT model for the RuleTaker and LeapOfThought datasets. In-language performance corresponds to evaluating the model same language as the training data.
On the LoT dataset, the model performs quite well on the code-switched data, outperforming the monolingual scenario for nearly all languages. The relatively high code-switched performance shows that the language of the context plays an important role in accessing the implicit knowledge encoded in the model's parameters, as the model must rely on this knowledge to solve the task. Providing the context in English facilitates access to implicit knowledge compared to other languages. This is also inline with the empirical observation that gen- eralization to X-en is considerably worse than en- X. We generally observe the same pattern for the XLM-R (see Appendix B) when fine-tuned on the monolingual RuleTTaker and LoT datasets.
stay mostly monolingual (Devlin et al., 2019; Con- neau et al.. 2020). When these models are fine- tuned code-switched downstream task, unlike the pre-training phase, tokens from different lan- guages can attend each other, which, as demon- strated in Tables and 8, results in poor gener- alization to other code-switched language pairs. We also observe that self-attention patterns consid- erably change when we compare code-switched langagee and cross-lingual samples' attention patterns (see Figure 4).
4.1 Approach
In order to make the fine-tuning phase more consis- tent with the pre-training, we propose two sets of methods to better handle the cross-lingual interac- tions of tokens.
Following the empirical observations showing MultiLMs struggle to transfer reasoning abilities in code-switched setting, we propose a novel attention mechanism mitigate the lack of code- switched transfer in these models.
Cross-lingual Query To better model the cross- lingual attention for code-switched tasks, we pre- train a cross-lingual query matrix Qeroxs (while keeping all other parameters frozen) on code- switched unsupervised data (more experimental
4 Cross-lingual-aware Self-Attention
Although MultiLMs have been pre- pre-trained on mul- tilingual corpora, individual inputs to the model
'The two samples are semantically the same, only having different statement languages.

---

Context
Question
Structured Attention Dropout As mentioned earlier, poor generalization of MultiLMs in code- switched settings can attributed to inconsistency between the pre-training and fine-tuning phases, where the former mostly deals with monolingual attention while the latter needs to handle cross- lingual attention well. We propose that the con- sistency can be improved by limiting the cross- lingual attention in the fine-tuning phase (i.e., reg- ularizing computational interactions between lan- guages). As demonstrated Figure this can be achieved by randomly masking attention scores (i.e., attention dropout), with probability Pmasks when tokens from different languages attend to each other. Moreover, to ensure reliable bridge between context and question, we never mask the attention scores of the first token (e.g., [CLS] in mBERT) help the model better flow informa- tion between two sections. Table 11 demonstrates the importance of structured attention dropout for better generalization in code-switched settings.
Conteext
+
Question
Figure 2: Illustration of the drop attention scheme. Due to the input's code-switched structure, we want to limit the attention between context and question tokens. can be seen that tokens from the same language can fully attend to each other, but there a dropout (white cells) when cross-lingual attention is being applied. In order to ensure reliable bridge between context and question, first token (e.g.. [CLS] mBERT) attends to all tokens, and also all tokens attend to the first token.
Interfering Cross-lingual Query Given the promising performance of the attention dropout for code-switched tasks, we experiment with a variation of cross-lingual query, where queries Q and Qcross also partially handle cross-lingual and monolingual attentions, respectively (see Fig- 3b). We empirically observe that having atten- tion masks that could randomly interfere with each other generally results in better performance (see Table 12) compared to the attention masks pro- posed in Figure 3a. In this scheme, M₁ and M2 are generated randomly and online," but once sampled, the same masks will be used for all the attention heads in all layers (more details in Ap- pendix D). Due better empirical performance, this variation of the cross-lingual query will be used for all the following experiments.
details in section 4.2). More specifically, we use two sets of attention masks, M1 and M2, where Mi enforces the query matrix Q to focus only monolingual attentions, and M2 constrains the cross-lingual query Qcross to cross-lingual atten- tions (see Figure 3.a). Formally, the self-attention probabilities for a given attention head, up (row-wise) normalization term, are computed as below:
Mi O exp( Vd )+ + M2 O exp( K K
QKT
Vd
where Q and K are the query and key matrices, d is the model's hidden dimension, and @ represents the Hadamard product. It is worth noting that this scheme still allows attention between all tokens; however, monolingual and cross-lingual attentions are handled by different query matrices.
4.2 Experimental Setup
All models are trained with the AdamW opti- mizer (Loshchilov and Hutter, 2017) using the Hug- gingFace Transformers (Wolf et al., 2020) imple- mentation for Transformer-based (Vaswani et al., 2017) models. The hyperparameters used for per- forming different experiments can be found in Ap- pendix C. All the reported scores are averaged over three different seeds.
The proposed Qeross can either be pre-trained for a single language pair (e.g., en-fr pair where context is in English and question/statement is in French), or it can be shared across many language pairs We show in Section 4.3 that having language- pair specific Qeross enables modularity, meaning model that trained on given source language pair can perform considerably better on another language pair by just swapping the source Qcrose matrices with the target ones.
A given sample can have different attention masks in different epochs.

---

Fine-tuning Setup. As Bitfit fine-tuning outper- forms full fine-tuning for all our experiments, we only report the Bitfit results here (Zaken et al., 2021). In Bitfit tuning, only biases are tuned in the MultiLM encoder, together with classifier and pooler parameters.
model using a language-speeific query matrix (Pair Qeross). we use the query matrix of the target lan- guage during the inference (only the weights). For example, while doing the zero-shot evaluation on en-zh, we use the en-zh cross-lingual query matrix instead of the one from the fine-tuned model.
Language Pairs. To show the effectiveness of the proposed method, we fine-tune the models on four typologically diverse languages (language of the statement), namely fr, de, zh, and ru. Our anal- ysis shows that combining monolingual and code- switched data the fine-tuning step improves the reasoning performance. Morcover, a multilingual reasoner should be able to reason over both mono- lingual and code-switched data. So, for this set of evaluations, we use combination of English and en-X (half of each) as the training dataset, which we denote mix(en, en-X).
4.3 Experimental Results
Table 3 shows the average zero-shot transfer per- formance (accuracy) for the RuleTaker dataset. For both mBERT and XLM-R, introducing a shared cross-lingual query matrix (Shared Qeross) im- proves the reasoning accuracy. These results under- score the significance of maintaining consistency between the pre-training and fine-tuning phases in code-switched downstream tasks to facilitate effec- tive transfer learning.
Using specific query matrix for each lan- guage pair (Pair Qeross) further boosts the cross- lingual transfer performance across most tested settings (up to 18%). In this scenario, there is a dedicated set of parameters to learn the attention patterns for a language pair rather than having them share the same number of parameters among many different language pairs. In other words, dedicated parameters help the model learn attention patterns for specific language pairs.
Pre-training Cross-lingual Query. We train shared (Shared Qeross) or language-pair specific (Pair Qcross) cross-lingual query matrix. For Shared Qerosss a shared cross-lingual query is trained on a parallel code-switched variant of XNLI (Conneau et al., 2018a), where an English premise is followed by the same premise but in another language. For Pair Qcroes, train cross- lingual query for each en-X language pair again using the XNLI dataset. In both cases, only the cross-lingual query matrix is trained and the rest the parameters are frozen. The training happens for 500K iterations.
Interestingly, in many cases, our approach also improves the transfer performance for monolingual data (mono). We hypothesize that, by having a separate cross-lingual query matrix, the model does not need to learn the cross-lingual attention pattern using the same parameters, reducing the chance of overfitting to the code-switched format.
Baselines. We compare the performance of the proposed method against two baselines: (1) The pre-trained model (original) (2) model pre- trained on code-switched data (CS-baseline). For the CS- baseline, we pre-train the model the par- allel swit-switched variant of XNLI (similar the data we use learn the shared cross-lingual query matrix) for 500K iterations to adapt the model to the code-switched setting.
We also conducted comparison with a code- switched baseline in which the MultiLM is pre- trained on code-switched version of XNLI. The code-switched baseline (CS-baseline) showed im- proved transfer results for en-X format and, in some cases, performed competitively with the Pair Qcross approach. However, it negatively af- fected performance in monolingual and X-en sce- narios, particularly for the mBERT model. In essence, the model exhibited overfitting the lan- guage pairs in en-X format it was trained on, mak- ing it unable to generalize effectively to monolin- gual and other code-switched formats. On the other hand, both Shared Qaross and Pair Qcrass demon- strated the ability to generalize their reasoning to
Cross-lingual Evaluation. For all the experi- ments, we evaluate the zero-shot performance of the model on (1) monolingual setting (where both context and question are in one language), (2) an en-X code-switched setting (where the context is in English and the question in other languages), and (3) a X-en code-switched setting (where the question is in English and the context is in other lan- guages). For the case when we Bitfit fine-tune tune the
'There is no Pair Qeross for en-fa and en-it (as they are not part of the XNLI dataset), and all the transfer results for these two languages are fully zcro-shot.

---

mBERT
Train Data
Method
Depth 0
Depch-1
Depth-2
Depch-3 3
mono
en-X
X-en
mono
en-X
X-en
mono
en-X
X-en
mona
en-X
X-en
Original
65.38
60.81
70.76
60.48
67.43
62.
55.55
62.48
$7.94
51.04
CS haseline
78.93
74.72
54,96
67.59
68.25
53.90
63.16
67.50
52.60
62.57
66.89
50.95
mix(en, en-fr)
Shared Qrees
92.52
70.07
65.16
77.72
67.26
63.93
74.81
64.23
58.97
70.46
63 86
55.75 75
Pair Q.eess
93.65
77.79
68.27
77.44
68.55
63.76
73.78
68.23
61.27
71.39
67.70
60.12
Original
88.71
66.75
59.10
68.98
58.64
56.69
73.39
62.88
55.66
63.45
57.36
50.84
CS. S-baseline
84.77
74.73
68.08
$3.99
63.58
67.47
62.23
50.73
mix(en, en-do)
Stared Qrross
91.39
70.10
76.74
61.64
64.38
71.98
57.26
Pair Qeenss
94.11
76.32
69.85
77.38
68.31
63.22
73.79
68.42
62.16
70.56
67.23
61.86
Original
91.69
69.25
60.23
76.49
60.40
$7.09
68.99
S7.62
$2,93
65.60
$7.70
50.05
CS- CS-haseline
83.65
75.92
54.68
71.06
69.96
55.49
64.80
66.84
52.47
60.06
58.93
48.71
mixion, en-ni)
Shared Qorees
76.22
70.35
79.80
68.77
65.44
74.06
71.89
57.19
Pair Q.eess
92.23
7W.31
64.50
67.97
70.98
60.10
Original
91.20
65.58
59.80
76.43
63.02
57.20
68.23
55.40
52.47
65.03
56.55
50.85
CS-haseline
83.16
70.22
57.46
67.34
66.87
54.29
66.29
65.73
53.01
60.72
63.64
52.21
mixten, en zh)
Shared Qoreae
93.70
68.49
64.59
75.11
65.11
62.00
73.42
66
59.03
69.98
62.01
57.62
Pair Qeross
93.21
72.09
78.93
64,22
66.68
71.34
60.05
XLM-R
Original
95.39
69.43
64.09
79.85
65.35
59.55
62.94
74.71
CS baseline
94.89
71.03
61.41
81.11
67.08
57.16
75.78
65.32
52.33
72.28
63.77
51.16
mix(en, on-fr)
Shared Q.eoss
95.92
74.78
70.87
79.82
68.46
63.84
79.99
70.64
62.
77.26
68.55
60.81
Pair Qeenss
95.94
78.36
75,80
83.64
70.17
63.94
81.39
71.59
64.37
76.05
69.04
60.12
Original
94.95
65.72
64.94
82.58
64.99
62.03
78.74
63.88
57.02
75.06
64.87
58.02
CS-haseline
91.92
72.53
76.70
66.64
54,29
65.58
74.98
51.87
mix(en, en-de)
Shared Qorees
96.23
71.29
70.95
81.95
67.25
64.27
82,14
70.48
63.28
75.26
67.16
57.01
Pair Qeenss
96.19
73.61
70.89
84.33
68,40
65.23
80.11
71.72
64.55
76.73
69:29
59,78
Original
94.46
72.86
63,94
80.80
66.55
59.53
78.23
65.90
55.78
74.35
63.05
53.32
CS-baseline
95.02
74.63
60.42
80.96
71.53
54.85
78.30
67.56
52.84
68.59
64.93
49.43
mix(en, en-ru)
Shared Qorees
80.20
83.73
72.16
68.02
71.31
56.23
Pair Qcraus
95.14
81.77
78.49
86.64
74.04
64.15
80.53
71.42
60.72
77.03
68.96
58.29
Original
95.61
71.13
65.80
82.29
65.44
60.53
76,93
62.36
53.87
75.93
61.67
53.35
CS baseline
94.67
73.76
57.92
81.86
67.79
55.41
78.40
65.58
52,74
74.39
62.67
49.57
mixien, en zh)
Shared Qerecs
96.56
77.10
74.84
84.52
71.96
61.35
81.39
71.31
63.25
75.61
67.42
55,40
Pair Qeeaas
75.00
72.39
87.55
71.83
62.88
71.08
76.03
61.42
Table 3: Average cross-lingual transfer of mBERT and XLM-R models on RuleTaker datasets to monolingual samples (mono) and code-switched language pairs (en-X X and X-en). The original is the pre-trained model and the CS-baseline is the model that pre-trained on code-switched data. Shared Qeroas and Pair Quross: refer to cases where the cross-lingual query matrix Qcrose is either shared across many language pairs or is specific to each language pair, respectively.
the X-en format. We also perform qualitative analysis of self-attention patterns for our proposed method in Figure 5, showing that the attention pat- terns remain more similar between in-language and cross-lingual code-switched samples (unlike Fig- ure 4). We hypothesize that the attention pattern sta- bility makes the MultiLM more language-neutred
dataset. Table 4 illustrates the average zero-shot transfer performance for this dataset. For this dataset, our proposed method also enhances the reasoning ability of the models for all examined language pairs. However, the degree of improve- ment observed is smaller compared to the Rule- Taker dataset. In the case of the implicit reasoning task within the LoT dataset, the model must rely both contextual cues and implicit knowledge to successfully solve the task. Conversely, for the Rule Taker dataset, the model is required to fully reason over the context. Consequently, for implicit reasoning, the model only partially uses contextual information, resulting in a lesser impact on per- formance when improving cross-lingual context- question attentions.
Regarding the cross- cross-linguse lingual transfer across lan- guages, we observe that the reasoning ability of the model is not transferred across languages equally (Appendix F). The more similar the lan- guages, the higher the transfer performance is. For example, the model trained on en-fr has its highest transfer performance in Latin languages (e.g., es, it, en-es, and en-it). For almost all cases, and regard- less of the training data language, en-fa and en-ar are the hardest languages transfer to.
4.4 Generalization to other Reasoning Tasks
To study the effect of the cross-lingual query ma-
trix on an implicit reasoning task, we expand our ex- perimentation to include the LeapOrThought (LoT)
So far, our experiments have focused on the logical reasoning ability of MultiLMs, either in monolin-

---

MRERT
XUM-R
5 Related Work
Source Disa
Method
X-m
Origatal
65.71
71.59
67.59
69.81
73.39
71.70
Reasoning in NLP. Language models (LMs) have demonstrated their ability to perform logical reasoning over natural language statements (Clark
CS baseline
62.18
66.92
62.19
70,00
70.92
09.08
mix(en, en-fr)
Quea
6961
73.27
71.45
74.51
72.22
Prir Ques
67.5
76.79
71,13
71.12
74.20
73,09
Origesi
6805
70.53
71.77
71.48
mixiun, en-de)
CS basuline
63.07
67.78
64.25
09.58
72.57
70.19
et al., 2020b; Chen al., 2023). They can also
Shured Qeease
67.48
75.50
71.52
10,00
73.22
72.07
Quees
69,09
76.17
72.62
70.03
73.55
72.75
leverage their implicit knowledge for reasoning pur- poses (Talmor et al., 2020) and exhibit multi-step reasoning capabilities by utilizing chain-of- thtughhht prompting, even with minimal demonstrations or instructions, when the model size is sufficiently large (Wei et al., 2022b; Kojima et al., 2022; Wei et al., 2022a; Tang al., 2023). In parallel to English-centric efforts on reasoning tasks, there have been attempts to create multilingual reasoning datasets to evaluate the cross-lingual abilities of pre- trained MultiLMs (Conneau et al., 2018b; Artetxe et al., 2020; Clark et al., 2020a; Hu et al., 2020; Shi et al., 2022). Recent pre-trained large MultiLMs like BLOOM (Scao et al., 2022), BLOOMZ (Muen- nighoff et al., 2022), and XGLM (Lin et al., 2021), exhibited promising few-shot performance a variety of cross-lingual reasoning datasets using in- context learning (Brown et al., 2020). Prior works studied the reasoning ability of MultiL.S in the context of open-retrieval answer generation (Asai et al., 2021), and mathematical problem-solving in a multilingual setting via chain-of-thoughh reason- ing (Shi et al., 2022). This work conducts the first investigation of the logical reasoning capability of MultiLMs and proposes cross-lingual laae-ll-aw-aar tention mechanism to improve their performance.
Origina
67.46
73.87
67.65
70.28
71.60
70.62
CS basaanee
62.37
68.03
62.48
70.11
73.85
70.15
mixion, en ra)
Stured Qreese
67.84
74.59
69.65 85
70,10
73.20
71.91
68.7
71.99
72.43
Origine
67.99
73.62
70.52
70:05
72.27
72.80
CS-hreeline
64.55
64.61
72.42
70.23
mixien,
Shared Qcnass
69.19
74.88
71.45
70.20
73.00
72.15
69.08
76.38
72.96
70.21
73.28
72:51
Table 4: Average cross-lingual transfer of mBERT and XLM-R on LoT dataset to monolingual samples (mono) and code-switched language pairs (en-X and X-en). The original is the pre-trained model and CS-baseline refers to the model pre-trained on code-switched data. Shared Qeross and Pair Qerose. refer to cases where cross- lingual query is either shared across many language pairs or is specific to each language pair, respectively.
Source Data
Model
mono
en-X
X-en
en
Original
69.42
63.16
68.79
Original
68.35
67.43
65.18
CS CS-buseline
68.26
70.58
65 89
mix(en, en-fr)
Shared Qeroaa
68.30
69.22
70.16
Pair Qeroaa
69.31
71.53
72.40
Table 5: Performance (accuracy) of mBERT model for the XNLI dataset in both monolingual and code- switched evaluation settings.
gual or code-switched settings. However, to demon- strate the proposed method's generalization to other reasoning tasks, we extend our experiments to the XNLI dataset. To create structured code-switched inputs for this task, we change the language of the premise and the hypothesis for a given input. More specifically, in a code-switched setting (e.g., en- fr), the premise is in English, and the hypothesis is in French. We fine-tune the mBERT model on a combination of EN and code-switched EN and FR data (mix(en, en-fr)), then zero-shot transfer it to other languages for monolingual evaluations (excluding en and fr) and other language pairs for code-switched evaluation (excluding en-fr and fr- en pairs). Table 5 presents the performance of the mBERT model with the linssllingual query com- pared to the baselines in both monolingual and code-switched settings. We observe ~4% improve- ment on en-X, ~% on X-en, and competitive per- formance on monolingual evaluation setups, indi- cating the effectiveness of our proposed method on downstream tasks other than logical reasoning.
aware at-
Cross-lingual Transfer. MultiLMs such mBERT (Devlin et al., 2019), XLM-R (Con- neau al., 2020), mT5 (Xue et al., 2021), and XGLM (Lin al., 2021) have achieved state of the art results in cross-lingual understanding tasks by jointly pre-training Transformer models (Vaswani et al., 2017) on many languages. These mod- els have shown effective cross- cross-lingual transfer for many tasks, including named entity recogni- tion (Pires et al., 2019b; Wu and Dredze, 2019; Foroutan et al., 2022), cross- cross-lingus lingual natural lan- guage inference (Conneau et al., 2018a: Hu et al., 2020), question answering (Lewis et al., 2019), and commonsense reasoning (Tikhonov and Ryabinin, 2021). This study focuses on the cross-lingus transfer performance of MultiLMs in the context of logical reasoning.
Code-switched NLP. Code-switching is a lin- guistic phenomenon of alternating between two

---

more languages within a single conversation text. In recent years, code-switching-relace research has been growing in the NLP commu- nity. The growth is motivated by the increasing need for NLP systems to handle code-switched data and call to pay more attention to multilingual- ism and low-resource languages (Dogruöz et al., 2021; Winata et al., 2022; Jose et al., 2020; Sitaram al., 2019). Previous research has been done for diverse range of tasks such Language Identifica- tion, Part of Speech Tagging, Sentiment Analysis, and Automatic Speech Recognition (Winata et al., 2021; Khanuja et al., 2020; Ostapenko et al., 2022; Tarunesh et al.. 2021). To the best of our knowl- edge, this work is the first to study logical reasoning in the context of code-switched NLP. Furthermore, majority of prior studies have focused word- level code-switching, where the language of certain words in text randomly changes. However, our investigation delves into the realm of "structured swithssstch wherein language transitions oc- cur at a section level.
of real-world knowledge in the reasoning context (compared artificial facts and rules in RulcTaker), in line with Tang et al. (2023) observation that lan- guage models perform better when provided with commonsense-consittnen context, and struggle with artificial ones.
7 Limitations
In this work, we evaluate our proposed method encoder-only language models, and the impact of this method on autoregressive models and encode- decoder-only models has not been explored, leav- ing room for further investigation and evaluation. Moreover, our experiments are limited to relatively small language models (less than one billion param- eters), and the results and our findings do not nec- essarily extend to large language models. Further- more, we should highlight that the scope of ex- periments constrained by the availability of mul- tilingual data and computational resources. Con- sequently, our evaluation limited to two specific datasets and covers only nine languages. While we strive for diversity in our selection, it is impor- tant to recognize that broader and more extensive datasets encompassing a wider range of languages could offer additional perspectives and potentially reveal new insights.
6 Discussion
this study, we explored the effectiveness of MultiLMs in a code-switched setting and found that while these models exhibit strong reasoning capabilities in monolingual settings, they strug- gle when it comes to code-switching. To ad- dress this, we first proposed the structured at- tention dropout, which encourages the model to rely less on cross-lingual attention when dealing with code-switched data. This simple method con- siderably improved cross-lingual transfer to other code-switched languages, demonstrating the impor- tance of structured attention for this setting. We then proposed a novel structured attention mech- anism, incorporating the cross-lingual query, that helps the model to better handle cross-lingual at- tention in the code-switched setting. The pro- posed cross-lingual query matrix, pre-trained on unsupervised code-switched data, significantly im- proved the cross-lingual transfer to other code- switched language pairs in all studies settings, demonstrating the importance of code-switched alignment for MultiLMs. We also observed bet- ter cross-lingual code-switched performance for the LeapOrThought dataset (real-world knowledge contexts) compared to RuleTaker (utilizing artifi- cial facts and rules). We attribute LeapOrThought's better code-switched performance to the usage
Acknowledgments
We thank Debjit Paul, Syrielle Montariol, Ange- lika Romanou, and Deniz Bayazit for their helpful discussions and feedback on our paper. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), In- nosuisse (PFFS-21-29), the EPFL. Science Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AI.
References
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Ha- jishirzi. 2021. One question answering model for many languages with cross-lingual dense passage trieval. Advances in Neural Information Processing Systems, 34:7547-7560.

---

Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international confer- ence on machine learning, pages 41-48.
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
A. Seza Dogruöz, Sunayana Sitaram, Barbara E. Bul- lock, and Almeida Jacqueline Toribio. 2021. A sur- vey of code-switching: Linguistic and social per- spectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1654-1666, Online. Association for Computational Linguisties.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celiky- ilmaz, and Antoine Bosselut. 2023. RECKONING: reasoning through dynamic knowledge encoding. CoRR, abs/2305.06349.
Christiane Fellbaum. 1998. WordNet: An electronic lexical database. MIT press.
Negar Foroutan, Mohammadreza Banaei, Rémi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discov- ering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Con- ference on Empiricul Methods in Natural Language Processing, pages 7560-7575, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a. TyDi QA: bench- mark for iitormation-seeiing question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454- 470.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020b. Transformers soft reasoners over language. arXiv preprint wxXx:22002.05867
Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, and Nanyun Peng. 2021. Ester: A machine reading comprehension dataset for reason- ing about event semantic relations. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 7543-7559.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 4411 1421. PMLR.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Navya Jose, Bharathi Raja Chakravarthi. Shardul Suryawanshi, Elizabeth Sherly, and John P Mc- Crae. 2020. A survey of current datasets for code- switching research. In 2020 6th international con- ference on advanced computing and communication systems (ICACCS), pages 136-141. IEEE
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018a. Xnli: Evaluating cross- lingual sentence representations. arXiv preprint arXiv:1809.05053
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NI.P. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 3575-3585, Online. Association for Computational Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2483 Brussels, Belgium. Association for Computational Linguisties.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint DrXiv:2205.11911
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sehastian Riedel, and Holger Schwenk. 2019. Mlqa: Eval- uating cross-lingual extractive question answering. arXiv preprint 1910:7711.

---

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112 2tt:::2122..060 10668.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Gold- berg, and Jonathan Berant. 2020. Leap-of-thought: Teaching pre-trained models to systematically rea- son over implicit knowledge. Advances in Neural Information Processing Systems, 33:20227-20237.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
arXiv:1711.05001.
Niklas Muennighoft, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng. Song- -hng Zhu, Yitao Liang, and Muhan Zhang- 2023. Large language models are in-context seman- tic reasoners rather than symbolic reasoners. arXiv preprint arX5v:2305. 14825.
arXiv:2211.01786
Alissa Ostapenko, Shuly Wintner, Melinda Fricke, and Yulia Tsvetkov. 2022. Speaker information can guide models to better inductive biases: case study on predicting code-switching. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pages 3853-3867, Dublin, Ireland. Association for Compu- tational Linguistics.
Ishan Tarunesh, Syamantak Kumar, and Preethi Jyothi. 2021. From machine translation code-switching: Generating high-quality code-switched text. arXiv preprint 1itiv::2177.04485
Alexey Tikhonov and Max Ryabinin. 2021. It's All in the Heads: Using Attention Heads as Baseline for Cross- Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IICNLP 2021, pages 3534-3546, Online. Association for Computational Linguistics.
Telmo Pires. Eva Schlinger, and Dan Garrette. 2019a. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 4996-5000 Flo- rence, Italy. Association for Computational Linguis- tics.
Lifu Tu, Caiming Xiong, and Yingbo Zhou. 2022. Prompt-tuning can be much better than fine-tuning on cross- cross-lingual understanding with multilingual lan- guage models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5478- 5788-585,.40 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019b. arXiv
How multilingual is multilingual bert? preprint arXiv:1906.01502
Soumya Sanyal, Zeyi Liao, and Xiang Ren. 2022. Ro- bustir: A diagnostic benchmark for evaluating logical robustness of deductive reasoners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9614-9631.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana 1lié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2..05000
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint 2rXiv:2006.0666
Freda Shi. Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanacl Schärli, and Denny Zhou. 2023, Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00 00093.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fdd Chi, Quoc Lee, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 12111:2201.11903.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay. Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057
Genta Indra Winata, Alham Fikri Aji, Zheng-Xoo Xin Yong. and Thamar Solorio. 2022. The decades progress on code-switching research in nlp: systematic survey on trends and challenges. arXiv preprint arXiv:2212.09660.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Kr- ishna Rallabandi, and Alan W Black. 2019. A survey of code-switched speech and language processing. arXiv preprini urXiv:I904.000 097440
Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2021. Are multilingual models effective in code- switching? arXiv preprint arXiv:2103. 13309.

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of- oftte-o--hhearr the-art natural language processing. In Proceedings of the 2020 Conference on Empiricul Methods in Natural Language Processing: System Demonstrations, pages 48845, Online. Association for Computational Linguistics.
A Dataset Details
This section further elaborates on the datasets that are used in our experiments. Both datasets in this study are translated using Google Translate API to investigate our proposed method's cross-lingual transfer. Starting from the English dataset, the samples are translated into eight other languages. namely, French (Fr), Farsi (Fa), German (De), Ara- bic (Ar), Spanish (Es), Chinese (Zh), Russian (Ru), and Italian (It). Below we discuss in more detail each studied dataset.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, be- cas: The surprising cross-lingual effectiveness of bert. arXiv preprint 1904::99990990
A.1 RuleTTaker Dataset
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021, mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 202/ Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
RuleTaker dataset (Clark et al., 2020b) is a set of five datasets, requiring various depths of inference to answer the questions. Each dataset consists of examples in the form of triple: (context, state- ment, answer). The context is composed of a series of facts and rules, while the statement represents the question that needs to be proven and the answer is either "T" (true) if the statement logically follows from the context, or "I" (false) if it does not (false under a closed-world assumption, CWA). All the facts, rules, and question statements are expressed in synthetic English. Essentially, each example represents self-contained logical theory in lin- guistic form, with question asking, "Is true?" The dataset generation procedure ensures that ev- ery question can be answered by formal reasoner. given the closed-world assumption (CWA).
Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bitfit: Simple parameter-effice efficcient fine-tuning for transformer-based masked language- models. arXiv preprint 2r6iv:2106.1011 10199.
Each dataset limited by the maximum level of inference needed to validate the facts employed in its corresponding questions. These datasets are categorized based their depth restrictions (up to depths D 0, D < 1, D < 2, D S 3 and D S respectively). A depth of D = 0 implies that the accurate facts can be readily "proven" by directly looking them up within the given context, without requiring any inference. The fifth dataset, encompasses questions that span up to a depth of 5. This dataset serves test to assess the ability to generalize depths not encountered during train- ing on the other four datasets. In our experiments, we use datasets with depths 4. Each dataset contains 100k examples randomly split 70/10/20 into train/dev/test partitions.
A.2 LeapOrThought (LoT) Dataset
The primary focus the LoT dataset (Talmor al., 2020) revolves around specific form inference that integrates implicit taxonomic knowledge (such

---

Experiment
EXPLICIT REASONING
IXPLICIT REASONING
EXPLICIT dev set), even if the statement is in medium-resouree language like Farsi or Ara- bic (context being English). We hypothesized that the model is mostly relying on the context for reasoning; therefore, the statement being in a medium/low-resoune language does not necessar- ily impact the model's performance. We validate this hypothesis by training context-only model (without having access to respective statements), and surprisingly this model performs ~94% on the EXPLICIT dev set (see Table 6). In order to ensure that the model can not get non-random accuracy by relying only on the context, we randomly negate 50% of statements (also negating the respective la- bels), so that a context-only model would perform randomly. The resulting dataset is the modified LoT that is used in all experiments in the paper. In order to further investigate artifacts present in the modified LoT dataset, we inject noise into the statement (without changing the context) as following:
Context only
94,56
68.98
Subject swap
83.08
51.46
Object swap
91.85
53.6
Subject Object swap
87.5
52.36
Table 6: This table investigates the artifacts present in the LcapOfThought dataset. We evaluate the model's 's performance when either different parts of given sam- ple removed (e.g., context-only model) or different noises are injected into the statement (e.g., swapping the subject in the statement with a randomly selected entity). The experiments that involve swapping entities in the statement are performed on the modified version of LoT, as discussed in section A.2.1.
hypernymy and meronymy) with explicit rules derived from natural language. The hypernymy relations and properties are derived from WORD- NET (Fellbaum, 1998) and CONCEPTNET (Speer et al., 2017). Each example consists of two com- ponents: (1) a hypothesis, which is textual state- ment that can be either true or false, and (2) ex- plicit knowledge, represented as list of textual statements. These statements can be classified as either facts, which describe properties of specific entities, or rules, which describe properties of a par- ticular class. The explicit knowledge carefully constructed to ensure that the truth value of the hypothesis cannot be determined solely based on the provided information. It necessitates the inclu- sion of additional knowledge encoded within the language model. This dataset contains two main sets; EXPLICIT REASONING which performs inference over explicit natural language statements, and IMPLICIT REASONING where the model must reason by combining the context with missing in- formation that should be implicitly encoded by the model. The dataset consists of 30,906 training ex- amples, 1,289 development examples, and 1,289 test examples.
Swapping statement's subject with a randomly selected entity from the whole dataset
Swapping statement's object with a randomly selected entity from the whole dataset
Swapping statement's subject and object with a randomly selected entity from the whole dataset
As demonstrated in Table 6, given the EXPLICIT evaluation results, the model can still get high rea- soning performance even when the entities in the context and statement are not consistent. However, as reasoning performance on IMPLICIT evaluation set drops to (almost) random when noise are in- jected into the statement entities, we believe that LoT artifacts have less effect on this evaluation set- ting. Therefore, to evaluate the MultiLM's 's reason- ing performance, we use the IMPLICIT evaluation throughout the paper.
A.2.1 Discussion on LoT Dataset Artifacts
LoT dataset was designed to test how well NLP models can (possibly) reason using real-world knowledge. However, as we show in this section, the dataset has some artifacts that causes the NLP models to take shortcuts instead of actually per- forming the reasoning. In the following analysis, we only focus on the original English LoT dataset (and not on the translated samples).
B Multilingual Reasoning: XLM-R results
Sections 3.2 and 3.3 discussed the in-language and cross-lingual performance of the mBERT model on monolingual and code-switched data. This section evaluates the XLM-R model on the same evaluation settings as mBERT.
In our preliminary experiments on this dataset, we observed that MultiLMs perform surprisingly high in cross lingual code-switched settings (on
Table 7 demonstrates the average in in-language and cross-lingual zero-shot performance XLM-R

---

Rakerr
LeapOrThought
Depth-O
Depth-1
Depti-2
Depth-3
in-lang.
cross-ling.
in-lang.
cross-ling.
in-lang.
cross-ling.
in-lang.
cress-ling
in-lang.
cross-ling.
en 100.00
77.75
87.24
76.26
76.70
70.36
99.40
87.10
81.53
83,92
77.78
81.33
73.78
74.64
80.49
85.46
72.97
72.11
69.07
de 99.33
80.
84,92
79.08
81.15
73.60
78.32
69.09
ar 99.13
76.29
84,43
71.13
69.70
71.71
es 99.96
90.53
82.00
84.34
75.61
71.83
80.60
71.25
99.96
77.63
82.56
69.10
81.95
67.69
82.62
66.35
ru 99.81
87.20
77.37
82.34
70.16
81.35
73.40
73.47
71.36
99.75
92.18
78.38
84.72
78.41
74.34
73.89
72.29
Average
99.70
91.95
87.00
79.06
84.44
74.50
81.83
71.10
75.78
69.80
Table 7: Monolingual Setting: In-language and cross-lingual zero-shot performance (accuracy) of the XLM-R model for the Rule Taker and LeapOrThought datasets.
Rule Taker
LeapOrThought
Depth.0 0
Depth
Depth 2
Depth 3
ln-lang.
en-X
X-en
lang.
en-X
X-en
in-lang.
en-X
X-en
lang.
en-X
X-en
lang........................ll
X-en
en-Ir
98.47
$4.18
52.38
88.18
61.10
57.61
85.54
54.54
50.45
83.14
52.50
50.66
84,02
78.41
75.40
eu-fa
98.61
63.70
55.51
88.99
60.63
$6.39
85:90
60.59
56.32
74.38
$4.92
52.78
85.96
82.57
72.06
en-de
99.62
59.59
$2.98
92.68
58.41
$3.06
85.17
57.25
55.14
86.09
54.75
50.45
87.20
80.55
74.45
en-ur
60.36
$7.55
57.40
62.21
55.45
56.21
51.20
77.06
70.16
100.00
60.01
52.32
92.59
63.06
56.99
87,97
59.17
53.61
77.48
55.70
51.33
86.89
81.30
77.60
en-ch 99.98
62.36
54.45
87.00
63.01
58.23
85.55
58.78
$6.99
82.96
$7.59
53.71
88.75
82.13
79.24
en-ru
99.93
64,07
$5.85
80.11
64.59
$1.57
86.85
57.40
$1.32
85.35
56.95
48.85
81.85
78.55
74.13
cn-it
64.16
89.11
61.03
54.45
53.39
56,80
50.73
74.96
70:41
Average
99.44
61.05
54.66
86.85
61.15
54.96
85.82
58.78
$4.08
80.58
55.68
51.21
84.49
79.44
74.18
Table 8: Code-Switched Setting: In-language and cross-lingual performance (accuracy) of the XLM-R model for the RulcTaker and LeapOfThought datasets.
RuleTaker
better cross-lingual generalization than full fine- tuning. In our experiments, also found out that using PEFT method like Bitfit considerably im- proves our cross-ling looss-l transfer across different languages.
Triining I angezge
Training Merhod
Deptl-0 O
Degth-1
ct-X
X-ex
cxxX
X-ex
minien, mnfr)
Full FT
87.57
63:05
59.40
68.95
59.72
5630
70.26
58.16
PT
62.55
58.56
63.84
$7.02
misies, sh)
Nide
91.20
65.66
49.00
16.43
63002
$7.20
FuE
02.00
19.13
58.30
36.67
59.10
68.95
58.64
56.09
Full FT
90.22
63.17
36.78
73.33
39.94
55.65
Table demonstrates the generalization improve- ment brought by Bitfit over full fine-tune base- line for the Rule'Taker dataset, especially in code- switched settings. We observed similar pattern for other RuleTaker depths and the LoT dataset. It is worth noting that using a PEFT method especially helps with transfer to code-switched tasks, which is our main focus in this paper.
mixton en-es)
Bifa
91.00
00.25
60.23
76.47
60:40
57.00
Table 9: Performance of fully fine-tuned versus Bitfit- tuned mBERT models the RuleTaker dataset. Bitfit- tuned models perform better or competitively to the fully fine-tuned setting, especially on the code-switched evaluation setups.
for each source language in a monolingual setting. Code-switched evaluation results are depicted in Table 8.
C.2 Curriculum Learning
For depths 2 and 3 of RuleTakkrr dataset, which involves more reasoning hops, we observed that curriculum learning (Bengio et al., 2009) makes the XLM-R training more robust. The curriculum learning is performed by first training the MultiLM for 3 epochs on subset of dataset that has depth 0 (i.e., no hop is needed for reasoning), and then the training is continued on the full dataset. This tech- nique not only makes the XLM-R training more robust, but also improves the final reasoning per-
Experimental Setup Details
C.1 Full Fine-tuning Versus Bitfit
As discussed in section 4.2, our proposed model and the baselines' performances in Tables and are achieved by Bitfit tuning (Zaken et al., 2021). It has been previously observed by Tu et al. (2022) that parameter-efficien fine-tuning (PEFT) has

---

formance.
attention pattern of our proposed model which in- corporates cross-lingual query.
C.3 Hyperparameters
We hypothesize that in order to have a reason- able cross-lingual performance, the cross-lingual samples' attention pattern should not change sig- nificantly compared to the in-language samples. Figure 4 visualizes the attention pattern between tokens in the last (baseline) mBERT layer across all attention heads. The mBERT model is fine-tuned on the mix(en, en-fr) depth-O of RuleTaker dataset, so the en-fr sample is considered in in-language and the en-ar sample is considered a zero-shot transfer. is worth noting the two samples are semantically the same and only the questions are in different languages. Comparing the two samples' attention patterns, we can see that the attention pattern con- siderably changes (especially the strong attention signals getting much weaker when en-ar sample is given as input), which to some extent explains the poor generalization of the baseline models to other code-switched tasks.
The hyperparameters for all the experiments is pro- vided in Table 10 for both mBERT and XLM-R models. We use the AdamW optimizer with warmup ratio of 0.1 for all experiments.
Cross-lingual Query
This section further discusses the methods pro- posed in section 4.1.
D.1 Structured Attention Dropout
As previously discussed section 4,1, limiting the cross-lingual attention in the fine-tuning makes this phase more consistent with the pre- pre-training, where the MultiLM mostly deals with monolingual atten- tions. Table 11 demonstrates that applying dropout cross- lingull attenions (see Figure 2) consider- ably improves lingualingual generalization in code- switched settings. Table 11 results are achieved by a 40% dropout on cross-lingual attentions (i.e., Pmaa = 0.4)
In contrast, as demonstrated by Figure 5, the attention pattern of our proposed method, which in- corporates cross gross-lingual query, is much more stable between in-language (i.e., en-fr sample), and the zero-shot transfer (i.e., en-ar sample). We believe that the observed stability in the attention patterns makes our models more language-neune pared to the baseline, which also demonstrated by the significant cross-lingual improvements over the baselines in Tables 3 and 4.
D.2 Interfering Cross-lingual Query
Inspired by the promising performance of the struc- tured attention dropout, we propose setting where the query matrix Q also partially handles the cross- lingual attentions, and cross-lingual query Qeross partially handles monolingual attentions. The only difference between the interfering cross-lingual query and the non-interfering scheme is their re- spective attention masks, M1 and M2. as illustrated in Figure 3. We also empirically demonstrate in Ta- ble 12 that the interfering scheme consistently per- forms better generalization than the non-interfering one, especially in the code-switched settings. For all the fine-tuning experiments with the interfering cross-lingual query, we use a 70% attention dropout (i.e., Pmosk = 0.7), meaning that 70% of cross- lingual attentions for query Q, and 70% of mono- lingual attentions for query Qerass are masked.
com-
F Detailed Cross-lingual Query Results
Tables and demonstrated the average cross- lingual transfer to either monolingual or code- switched settings. This section demonstrates the detailed cross-lingual performance of models with cross-lingual query and the Original and CS- baseline. Tables 13 and 14 present the detailed cross-lingual transfer of mBERT trained on the TukeTaker and LeapOfThought datasets, respec- tively. Tables 15 and 16 present similar detailed cross-lingual performance of XLM-R model on the RuleTaker and LeapofThough datasets, respec- tively.
E Attention Visualization
As discussed earlier, MultiLMs perform well on in- language, but when they are transferred to other lan- guages (especially code-switched languages) their performance hinders considerably (see Table 2). This section first analyzes the attention pattern of baseline models, both on in-language and cross- lingual evaluation settings. Then, we analyze the

---

mBERT
Train Method
Dataset
Epoch Iteration
Batch Size
Learning Rate
Evaluation Metric
Attention Dropout Probability
RuleTaker
32
le-5
Accuracy
N/A
LeapOrThought
32
le:5
Accuracy
N/A
RuleTaker
Accuracy
0.7
BitFit
LeupOfThought
4e-4
Accuracy
0.7
Pre-training
XNI.I dalasel
Perplexily
XLM-R
Train Method
Dataset
Epoch Iteration
Batch Size
Leurning Rate
Evaluation Metric
Attention Dropout Probability
RuleTaker
5e-6
Accuracy
N/A
Full FT
LeupOfThought
56-6
Accuracy
N/A
RuleTaker
Accuracy
0.7
BitFit
LepOOThoough
Accuracy
Pre-training Q
XNLI dataset
500,000
Perplexity
Table 10: Hyperparameters the pre-training and fine-tuning experiments for mBERT and XLM-RoBERT models.
Learning rate decays linearly from the initial value zero.
Transfer Setting
Training Method
Drop attention
Monolingual
en-X
X-en
Full Fine
Yes
90.00
65.57
62.74
89.48
62.68
59.53
Biditt
89.14
65.38
60.81
Table 11: Average cross-lingual transfer of mBERT model when tuned on mixture of English and English- French (mix(en, en-fr)) RuleTakkrr dataset (depth-0). The (zero-shot) cross- lioss-lingual transfer to code-switched tasks gets considerably better with structured attention dropout (see section 4.1), either in full fine-tune Bit- (Zaken et al., 2021) tuning.
Setting
Training Method
Interfering
Monolingual
en-X
Yes
93.65
77.79
68.27
Bidit
No
91.96
73,08
66.28
Table 12: Average cross-lingual transfer of mBERT model when fine-tuned on a mixture of English and English-French (mix(en, en-fr)) RuleTaker dataset (depth-O). Both models incorporate language-patt specific cross-lingual query (i.e., Pair Qcross) are trained with Bitfit tuning. The only difference between the two runs is whether an interfered version of the cross- longull query is used not. We can observe that the interfered variant consistently outperforms the other variant, in monolingual and sode-switched settings.

---

Dupth-8
09.50
10.33
17:31
19:21
12.9
12:33
71.5
12.00
2031
50.36
59.35
-0.77
SLB
10.07
10.00
23.01
6322
55.52
53.45
$2.47
04.55
89:34
19.6
71.5
8197
09:79
61.58
93.08
9935
76-25
25.56
61:51
16:6
45.43
5150
58.55
57.66
51.55
53.15
Stared Quoes
189
36.21
7211
14.65
59.59
5399
8551
90.71
95.18
98.75
83.33
15.01
62.59
74.96
50.35
56.6
59.33
10.33
33.71
69335
75.99
5504
55.54
92.90
Pat: Maen
99.00
27.04
0138
1457
64.52
94.99
12.07
19:23
13.44
16:17
72400
B101
52.19
05115
02.84
72.44
25.78
71.34
75.96
01-07
51.55
54.35
Orgiul
64
01
96.11
7513
91.15
55.11
03.11
79.71
56.11
36.1
60.79
7161
56.11
01.5
$1.33
55.66
68.53
71.00
53.35
58.55
01.35
53.9
$1.02
$1.35
C5-taudie
09.80
87.90
67:21
16.12
11:53
12.12
$4.55
06.56
18.11
79.66
72.66
19.35
5US5
62.59
68.21
64:23
5984
5741
$2.9
Shard
91.51
81.81
91.34
75-6
00.11
T1.99
3136
56.6
51.03
Urgtual
60:71
99.58
05.35
77:32
71.11
19:19
16-22
72-80
60.24
75
16.33
79:4
7509
Pat Moeen
19.30
14.07
$1.83
12.00
89:35
13.04
19:31
12.32
94355
$2.32
765
79.2
76.96
48.52
17:11
73.02
62.79
72
76.96
77.91
71.34
9A81
Orgael
59.44
15.35
K2:4
00:00
1584
55.34
85.55
90
54.15
55.30
66.75
10.66
91.54
56.99
56.01
54.50
6966
00.00
34.55
6354
6836
55.79
81.25
32.99
rablee 10:00
Stave Quee
CS-baudee
64.04
99.50
50.30
61.99
07:56
58.31
11:28
60.95
13:34
94
45-55
97.50
74-07
91.37
14.8
91.03
92.03
05.88
77:52
74.57
6606
68/14
12A1
65.16
19.1
96.50
65:30
66:72
96-46
96.65
-6.77
$1-99
26NG
51.34
71.58
05
50.00
66.00
57.79
52.62
67.57
63:40
47.85
57.96
55.52
97.84
73.03
$7.99
5243
$3.95
hi: Creas
169
96.30
69.75
14.41
55.64
96.76
96.96
09.65
66.36
74.90
71.38
13.73
19:46
03.56
72.05
+1.37
56,35
16:31
13.08
59.39
72.69
75.91
9018
85.75
$3.57
Rek-Taker Dupth1
Orginl
13.59
5479
Quees
75/9
71.31
Quees
5004
74.99
15.66
82.55
14
72:01
4015
69:82
5:99
91.2
Orgiol
13.0
31.36
01.15
(76)
09177
66.95
16.5
12:17
03.53
61.07
13.11
57
16.96
5317
15.21
61:33
61:10
53.11
54.93
$7.90
5546
13.88
51.57
CSteddee
$2.55
65.65
50.90
64.11
05:35
85.55
6624
68:39
64.59
14.14
(6/
11.19
12.56
66.11
9136
51.75
06776
16.01
59.00
$5.59
54.4
55.35
15/4
55.39
52.9
Shanet Quee
10:30
74.35
7621
18:56
12:57
76.77
71.00
13.53
13.98
73377
06.57
66:12
62.47
62.19
51.17
$4.50
2608
68:22
13.70
2461
6317
05.30
55.32
15.77
5472
Tul Qoees
67.0
00:30
71:31
61.75
61:11
8357
74:00
15.59
K2.04
70.99
67:51
T833
(4:73
66-1
11:00
5:81
06:57
10:17
71.03
5541
6567
65.6
58:11
99666
5537
Orginl
$5.0
73.91
07.75
15:00
11:32
7669
16:22
13.22
1859
6834
60.49
66.76
56:85
68.16
91.17
35.21
54.0
6368
66.08
5321
5837
3936
5682
33.99
5972
$1:3
70:11
68.9
18.14
06.8
8111
71.96
43.11
5600
99.9
Stard
76-42
1218
07:7
Orgiul
CStediie
$3.00
00:72
62.39
71.77
61.06
06.10
1917
57
5491
55.8
35.56
51.9
Staned Qua.n
10:22
74.0
9652
17:44
63:88
7566
79,22
9434
01.20
79:71
63.1
15.31
56-47
35,33
69A1
10.24
26.99
5335
5632
54.44
55:42
Pail Oween
9115
6631
81.34
71.44
01.1
79.3
71.14
01.43
71.56
61/00
(9:51
$0.53
06-33
70.00
55-0
(051
(3:33
(6.11
51.00
31:18
00/
9191
73.66
58335
Hal Taker Pepth-2
Orgiul
6150
31:09
51.%5
71.01
15.09
7147
60.55
10.10
11.06
74.10
66.76
61:55
16.75
96.34
00:94
5::00
15:24
61:17
13.66
10.75
57.33
17.77
51.31
51.9
9131
CSteddee
$3.59
12.35
57.35
56.71
03:59
6225
0003
61.01
02.0
91.92
68.35
14.54
66.15
00.00
70.38
$1.08
01.32
54.16
53.99
5831
54.43
5-40
50.56
$1.33
50.4
Stane
13.5
77.34
1800
07:17
73,56
13.0
700
51330
$7.54
13557
02.9
63.55
5132
Tall
$181
76.57
76.66
11.31
62:35
Urgiaal
76.40
53:51
ShasiQ
$4.9
7646
10.38
9115
62.16
01.70
76-41
66:31
15.66
61:35
6006
$2.4
Pat Veen
15:28
36.40
1110
19:09
7538
T1J4
13.6
73.33
07:34
TEA6
69:44
66:31
67:48
73.11
23.65
63.3
5654
Orgiel
8034
71.39
00.31
12.19
8359
9544
063
66.55
89:24
31.1
36:22
36.11
26:35
0623
35.0
34.55
5600
26:42
5450
3031
3535
5:31
32.36
31.9
3154
12000% CHI
CSibuddze
13:14
65:08
54.90
36.00
56.77
61.75
61.17
75.00
19:19
55339
3591
1143
6::99
7015
63.49
77.68
7151
7177
67.67
65:57
62.33
6611
(8:43
10-31
06.00
S051
T13F
06.30
67:91
52.77
06.96
54.66
61.06
53.00
49.00
5120
53.76
54.22
63:03
01.55
13.11
51.6
66:00
55.66
51.99
5111
$7.53
he Quen
1434
17.35
65.77
75.04
19.34
76455
72.22
12.09
16.16
71.22
64.06
68:54
1146
71.66
90.88
5605
62.99
66.13
66.71
53463
65.33
63366
900
62.44
53.1
Orgtul
1338
7170
61.4
6031
01.0
97.4
35,70
64135
$3.00
$4.00
51.31
55.16
5255
5002
CSteidac
Shand Quee
Qoees
Rok:TskerBpph.
Doyth33
Drgital
13.34
64.33
68:33
15:33
53.39
66.27
63255
91.7
10:27
62.14
23:00
5421
36.22
SU12
52.77
37:35
23.44
48.33
12:77
40/2
4632
81:11
57.21
14.44
41477
53.22
(8/66
1931
74.57
60:90
(8:99
06/4
2013
91.01
(011
53.56
13.25
18.56
53.17
$1.5
19.29
93.38
Staee Queen
$1.20
71.77
42/9
7434
64:11
12.30
65.47
68:72
11.34
75.36
09:71
68.00
59.71
6017
57.90
55.65
55.74
10.50
58.66
49.72
57.59
60.90
500
59:55
45.47
Tel Quen
61:30
35.34
67:89
75
6553
7187
67.5
18.91
98.74
78.90
09:34
71.45
16
6 50
70.71
57.99
61.33
74:14
00:33
53.47
67.55
69.66
57.79
55
5533
Orginll
15.43
07.33
57.36
64.11
59:18
64.1
39.01
6439
91.33
30.06
55.55
51.54
53.51
9015
51.38
32.44
54.14
56.00
48.16
51.11
31.9
22,90
45.38
48.22
CS-audiie
12.00
5936
53.54
59.36
71.96
MUSI
67.44
77.44
53.73
54.66
51.35
4691
4672
Shand Quees
$1.58
7101
68.11
T8.16
96.11
66.36
96/4
11.38
63.41
71.11
54.63
40.55
Orghul
csesl
Staned Qu...
CStendiee
83:39
61.00
72:44
45.4
6002
59.99
19:55
3441
6612
60.99
30
1329
61.52
55.33
12:01
35115
62:0
61.12
63.69
66.0
10J0
5438
51.55
55.19
35381
61.10
45:1
2038
2333
57333
49:1
56.32
28.11
54.35
43.33
5434
Pai Qoeen
63.61
30:30
98.91
2931
17.01
71.66
6661
(8:00
03
(8:31
65:66
83.51
$1.51
5540
6062
(0:00
00.30
53.96
63:15
07.75
03:39
5613
Drgbbal
14:00
55.54
57.36
66.30
63.33
05.14
6206
64.3
06.38
5752
57.40
25.0
73-57
53AT
50.00
52.00
51.8
5424
$3.00
48.09
51.34
51.09
4616
82
4631
CS huudii
61:0
59.99
53.35
0556
5353
0375
9:39
17.11
6458
63.11
05.81
63
58.70
63.36
65.5
19.8
05.55
1661
53.95
1555
59885
9.51
90.0
Stared Qre.n
8:30
75.12
5683
11A1
6457
1218
003
6125
1241
5413
62.75
62.55
15.99
5695
$1.00
35.90
$2.84
63.0
55.00
5852
5939
90.5
55.4
65.5
51.2
Pul:
$3.0
33.41
13.12
7347
18.10
07.25
08.12
08.94
6459
9933
57.41
0630
07.36
5634
Table 13: Cross-lingual transfer of mBERT model on the Taker datasets either monolingual samples or code-switched language pairs (en-X and X-en). The original is the pre-trained model, and the CS-baseline is the model that continues pre-training on code-switched data. Shared Qcross and Pair Q refer to cases where the cross-lingual query is either shared across many language pairs or is specific to each language pair, respectively. Scores are averaged across three different seeds.

---

The Di
Mch-ll
CHO
d-ca
drm
Orgen
63.85
6131
14.6
60:33
7549
1331
12.05
7181
91315
75.40
68.15
07.25
12.55
80.35
1251
65,35
71,255
03.30
C5 laeka
71.00
6340
9033
64.36
$1.33
(5.
63.0
5:355
58.55
(6:11
74.5
$300
56.48
03:23
13.0
66.00
5.36
5031
56.35
06.25
$248
Band Qu1
75.80
6035
72:30
09.8
1356
1546
$4.55
06.05
1533
134
7571
71.1
TL-E
NL11
7081
00:42
11,00
71.39
13.55
3634
6350
70.9
75.17
Ner Meea
170
1548
06:11
120
11.0
14.0
27.8
12.00
28.35
10.
7734
13.3
16.44
11
1168
01:0
34
113
7091
6171
7733
16.1
08:51
10.00
25.15
61%
7157
09.5
14
3057
06.90
35.17
6557
STAL
18.34
6518
1534
7640
1159
5687
43:99
6540
08.43
13.97
6537
62.35
13.66
E3m
Shand
6388
3133
71.45
3638
9138
80.25
66:33
13.9
75.44
77A8
7137
72.34
12.1
72.81
65.00
71.14
66.43
15.33
8631
6121
1440
12.16
12:0
Par
61.38
19.33
66.1
04:11
3639
43.29
7114
16.11
22,45
Tel
9633
0835
X633
9121
11.4
60.2
61.9
7534
1471
14.9
5181
11.8
1479
9611
66.35
12.9
71.37
00:1
146
7100
68.5
1336
1611
403
Band Rooe
Choweker
1417
73.25
TL0
5TAL
0135
08:1
5156
7114
94.33
68.30
7448
1131
07.37
13.88
7448
$4.00
00:7
1511
110
98.00
7244
1716
7406
00.65
72.52
04.66
1564
7230
01.
06:3
14.77
7548
100
5537
13:2
1148
$7.55
09.57
5581
7111
03.00
94.35
60.35
7659
15.72
7623
63.85
7842
70.9
7152
72.52
7416
7465
4827
15.3
7547
70.7
7313
5611
Table 14: Cross-lingual transfer of mBERT model on the LeapOrThought dataset to either monolingual samples or code-switched language pairs (en-X and X-en). The original is the pre-trained model, and the CS- CS-baseline is the model that continues pre-training on code-switched data. Shared Qcrosa and Pair Qeross refer to cases where the cross-lingual query either shared across many language pairs or is specific to each language pair, respectively. Scores are averaged across three different seeds.
Dee
SHE
Cigadd
12:10
1031
1136
ILD
6713
18000
TA25
363
3430
65.
61:11
7675
766
2533
10:00
15.9
19.3
JAR
7110
W00. (9-0
13
100.0
16.33
11M
12th
(92)
12h
12.0
0631
341
+1.0
13.4
9641
11.9
7356
133
7100
0.18
11.1
7310
14.0
1334
11:01
1931
1131
10.0
1933
11.1
7436
1558
7:36
Rellbor Dep--1
Stane
8:00
9008
12.0
88.08
911
142
9631
PAN
1655
569
0431
5.15
21-8
10.0
7531
700
400 (M/7)
13 a
2010
7130
16:4
413
108
520
443
1133
6.10
6.4)
61-8
30.66
1630
11.0
2530
300
1180
9631
31.8
00.5
1781
0.38
0635
9531
9031
15.
5:31
9:1
13
14:
04.00
1331
64.
18.11
34.8
2481
390
3913
12.4
7552
6.4
5480
533
7500
0344
1000
1633
763
114
46.0
608
1200
1439
(23)
9433
50.5
5601
5412
9013
5.1
5611
1155
9150
12.91
1130
1841
2642
11.1
7330
1058
7354
5166
3743
540
9431
5473
25.0
00:00
9.11
11:
11.9
5014
13.00
15-1
1913
01
31.06
1213
(43)
002
0121
14.4
250
300
312
105
15.00
0333
9.50
755
055
1101
5111
5.41
513
1110
06.11
Kcm
20:0
1936
443
305
405
3618
8000
7400
1561
5.31
5464
500
113
11.0
14:1
15.6
1134
4555
63:11
515
64.3
012
5000
55
055
535
514
200
1942
706
15.0
11.3
136
762
61
7344
11.0
7531
03.0
01.0
539
6350
3600
565
063
569
25
Ospad
8555
1255
3.58
73.6
133
6235
7599
622
3074
314
5455
5011
8611
209
0116
31.1
11.9
1611
150
1.11
1114
100
8218
1281
0.6
753
4/3
132
121
TUE
1856
14.4
41.3
SAY
258
0231
3144
21.2
11:18
11:45
1015
4041
7541
7015
1100
05.5
15:11
0.
8.75
154
1536
1.31
5114
J.M
6310
16.18
36.0
1200
11.
12.2
3421
24-3
1856
3650
66.
30.2
1631
8.
6.5
M......
15.0
5.14
4.55
700
11.0
7140
41.1
16.
0.55
10.8
7534
3.3
(740
1831
110
4835
6113
06.
01
11.
6518
13.0
0.15
602
5331
12.34
11:34
737
81.3
096
1781
71,9
442
7534
1032
TME
11.8
5122
5235
03.0
5541
561
(1.0
562
366
56.1
Ospad
13400
54.80
6879
71.44
73545
14-2
2002
540
550
56.96
11:08
75%
0631
73
1800
100
9m
191
3116
1.8
064
M.3
11.
0634
1626
613
1931
164
4.3
5.0
1236
3651
065
2012
w00 04.04
0234
0113
11.1
06.0
1420
1881
11.0
11
1026
11
0641
1941
11.4
1966
20.0
6.01
1830
1111
11.0
3600
20.0
1020
3133
011
0114
Table 15: Cross-lingual transfer of the XLM-R model on the Raker datasets either monolingual samples or code-switched language pairs (en-X and X-en). The original is the pre-trained model, and the CS-baseline is the model that continues pre-training on code-switched data. Shared Qernes and Pair Qeross refer to cases where the cross-lingual query is either shared across many language pairs or is specific each language pair, respectively. Scores are averaged across three different seeds.

---

Context
Question
Context
Question
Context
Context
1
M
Question
Question
Context
Context
M
8 2
M2
Question
Question
8
(a) Non-interfering Atlention Masks
(b) Interfering Attention Masks
Figure 3: Illustration of the attention masks in Section 4.1. In the proposed scheme, two sets of independent query matrices (Q and Qerosa) collaborate to compute the attention scores. Matrix M1 enforces the Q matrix to mostly focus monolingual attentions, and matrix M2 constrains the Qcroas mostly handle tross-lingual attentions. The difference between masks in the two figures are the structured attention dropout probability being either one (left) or less than one (right). is worth noting that the first token (e.g.. g. [CLS] in mBERT) is used bridge in both Mj and M2, meaning its respective attentions are not masked.
68
87
71
moh
mb
CE:ET
2:m
12:m
27m
Organl
15.30
11.34
0622
1233
0352
7255
7W775
06.23
$3.33
73377
75,40
76.14
13.51
71.77
76.11
0107
7146
12.0
12.34
10.15
73.53
71.3
70.5
34.09
3632
#8008.00 48-2)
Stavd Quaes Qoen
CSibudiie
11.11
15:34
09.34
03.35
05:00
6563
71.15
1210
6349
0332
7155
7147
7506
7666
68.11
06.73
06.23
66.89
73.95
74.00
71.36
7006
72.16
7106
7357
7181
7006
26.4
7675
73.55
70.7
06-66
6983
66.75
13.31
71.11
08.02
73.00
6833
7551
7335
34.7
07.31
7071
9606
71.76
1533
31.55
30.55
56.11
he: Qoen
19.35
72.30
6518
1434
83.0
1453
7612
06.00
14.00
74.6
73.0
16.12
7531
7611
7032
14.17
1426
12.2
15.87
72.5
35.41
91.4
Drghul
18.34
7186
7231
72:46
6611
15.8
66.00
12.54
344
3115
Queas
Oceas
Orginal
1233
7128
11.30
CSbaudine
71.31
7436
65.99
13.09
Stave Qeei
16:35
6911
7111
77/6
71.1
75:54
71/6
75.25
21.55
66.24
68:13
$3.94
hii Rree
19.29
67.11
6967
7536
59112
3421
7806
60.2
64
14.47
7401
14.99
79
04.32
N654
75.99
7003
10.39
1426
15.54
7533
86.5
7214
12.9
$9.0
Orgial
15.58
63.30
66.7%
73
03:45
7191
7999
16.1
55.0
N03
71.94
73.75
74:6
69:00
76.1
2061
66.53
68.74
7166
73.33
3141
7067
74311
36
$3.9
CSteddec
19.35
82.81
66.66
12.38
6334
902
73.99
68.5
74.85
68.74
65.00
73.51
1152
3333
67.75
13.85
52554
Shane
17.60
13311
au4
71.84
7137
75.46
68.19
7424
8631
13.33
71.58
9691
Table 16: Cross-line lingsllingual transfer of XLM-R model on the LeapOrThought dataset to either monolingual samples or code-switched language pairs (en-X and X-en). The original is the pre-trained model, and the CS-baseline is the model that continues pre-training on code-switched data. Shared Qcroes and Pair Q Qcrose refer to cases where the cross-lingual query either shared across many language pairs is specific each language pair, respectively. Scores are averaged across three different seeds.

---

(a) fr sample (in-lange) language)
Heads
Layers
(b) en-ar sample (zero-shot transfer)
Figure 4: Attention visualization of the baseline mBERT model for in-language (en-fr) and zero-shot transfer (en-ar), both from depth-0 of the RuleTaker dataset. The underlying mBERT model is fine-tuned on the mix(en, en-fr) of the RuleTaker depth-O dataset. We hypothesize that the poor cross-lingual transfer of baseline models other code-switched languages partially originates from instability of attention patterns across different languages depicted in above figures.

---

Heads
Layers
(a) en-fr sample (in-language)
Heads
(b) en-ar sample (zero-shot transfer)
Figure 5: Attention visualization of the mBERT model with cross-lingual query for in-language (en-fr) and zero-shot transfer (en-ar), both from depth-0 of the RuleTaker dataset. The underlying mBERT model is fine-tuned on the mix(en, en-fr) of the RuleTaker depth-0 dataset. We can see that attention patterns our proposed model is more stable between in language and cross- cross-lingual samples, compared baseline model in Figure 4.


Reviews:
[{"review_id": "q1219dphk9", "reviewer": "Reviewer_Gdvn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the importance and rigor of our analysis and the effectiveness of the proposed method. \n\n---\n\nRegarding the clarification questions asked by the reviewer:\n\n\n\n\n>**\\[Q1\\]**: The reviewer asks for clarification on Table 1, which reports results for when we fine-tune models on a single source language, test it on other languages, and then take an average of these results. The reviewer’s interpretation is correct. We report the average cross-lingual transfer performance of each source language. We will clarify this in the camera-ready.\n\n---\n>**\\[Q2\\]**: The reviewer asks for clarification on **“mono”** performance in Table 3, where we report the zero-shot cross-lingual transfer performance of the proposed method and the baselines for two monolingual and code-switched settings. Column **“mono”** shows the zero-shot transfer performance of the model in a **“monolingual”** setting (when context and question are in one language). More precisely, in Table 3, we fine-tune the model on English + fr/de/ru/zh (e.g., mix(en, en-fr)) and zero-shot transfer it to unseen languages (e.g., it, es, fa, ar). The results of this column shows that our approach improves the transfer performance for monolingual reasoning in many languages.\n\n---\n>**\\[Q3\\]**: The reviewer asks why we include monolingual data (English) while learning cross-lingual query matrices (line 370). For the **pre-training** phase, we only use code-switched data to learn cross-lingual query matrices while the rest of the parameters are frozen. However, for the **fine-tuning** phase (when we learn the reasoning task), we include English training data to train a multilingual reasoner that can reason over both monolingual and code-switched data. During the inference on monolingual data (e.g., English), the model’s original query matrix is used to compute attention scores, and for the code-switched data, both the original and the cross-lingual query matrices are involved.\n\n---\n>**\\[Concern Statement 2\\]**: \nThe reviewer is concerned about the translation of the datasets and asks why we did not use manually verified datasets like XNLI and XCOPA:\nWe use logical reasoning tasks to study the cross-lingual reasoning ability of multilingual models because they allow us to control the exact degree of code-switched text in the contexts and questions. As there is no multilingual logical reasoning dataset, we automatically translated existing English datasets (i.e., RuleTaker and LeapOfThought). \nHowever, to show the proposed method’s generality and address the reviewer’s concern, we performed an experiment on the XNLI dataset. We fine-tuned the mBERT model on a combination of EN and code-switched EN and FR data (mix(en, en-fr)), then zero-shot transfer it to other languages for monolingual evaluations (excluding en and fr) and other language pairs for code-switched evaluation (excluding en-fr and fr-en pairs). Finally, we get the average of these zero-shot transfer results (accuracy). We observe ~4%  improvement on en-X,  ~7% on X-en, and competitive performance on monolingual evaluation setups, indicating the effectiveness of our proposed method on downstream tasks other than logical reasoning.\nThe languages included in the evaluation are German, Russian, Spanish, Chinese, and Arabic, and the detailed results are:\n>>| Fine-tune Data Language |      Model     |    mono    |  en_X |  X-en |\n>>|:-----------------------:|:--------------:|:----------:|:-----:|:-----:|\n>>|            en           |    Original    | 69.42  [1] | 63.16 | 68.79 |\n>>|      mix(en, en-fr)     |    Original    |    68.35   | 67.43 | 65.18 |\n>>|      mix(en, en-fr)     |   CS-Baseline  |    68.26   | 70.58 | 65.89 |\n>>|      mix(en, en-fr)     | Shared Q_cross |    68.30   | 69.22 | 70.16 |\n>>|      mix(en, en-fr)     |  Pair Q_cross  |    69.31   | 71.53 | 72.40 |\n\n>We will include the complete evaluation results for the XNLI dataset in the final version of the paper. \\\nFor the XCOPA dataset, unfortunately, the training dataset is only available in English and would need translation for the fine-tuning of code-switched data.\\\n**[1]** Hu, Junjie, et al. \"Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.\" International Conference on Machine Learning. PMLR, 2020\n\n---\n>**\\[Concern Statement 3\\]**: \nThe reviewer is concerned about the code-switched setting being artificial. Although in this paper, we address the structured code-switched problem in a controlled setup, the code-switched setting can be found in many realistic scenarios, such as when non-English speakers may ask questions about information that is unavailable in their native language. In this case, the model needs to combine the context information in one language with the question in another. That’s why it’s important for the model to have the ability to perform in such a setting.\n\nWe will add more explanations in the body and captions of the tables to clarify the reported results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an interesting and rigorous analysis on how multilingual models are suffering in the code switching transfer scenario and offers convincing explanations. This analysis is an interesting scientific study on the limits of these multilingual models with regard to cross lingual transfer. \n\n- The proposed methods prove to be very effective as it significantly improved the performance of models on the transfer setting.", "reasons_to_reject": "- The main issue of the paper is that the experimental details are very unclear and the reader has to guess in many cases. Below are some of such examples:\n(1). For the cross-lingual results in Table 1, did the authors fine-tune the model on the source language and test on every other language and then take an average of the results of these target language performance? It would be good to explicitly state this.\n(2). What is the mono performance in Table 3? Is it first training from English + de/fr/ru/zh and fine-tune on fr/de/ru/zh, and then zero-shot transfer to the other three unseen languages from fr/de/ru/zh?\n(3). In line 370, the author(s) mentioned that only cross lingual query matrix was trained, but why is there monolingual English training data (in this case there is no cross lingual query matrix and all the parameters will be fixed)?\n\n- All the analysis is based on Google translations of the datasets, which introduces additional noise due to potential errors in machine translation. Why not use existing manually written/verified datasets such as XNLI, XCOPA?\n\n- Maybe a limitation here rather than a weakness. The code-switching setup in this study is a little bit artificial with the analysis and the model being specific to the datasets and tasks, while code-switching in natural language tends to switch specific phrases and has specific rules and tendencies. That being said, I think the analysis from the paper is a valid scientific study on the hypothetical language switching behaviours of multilingual models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents evaluation analysis of multilingual models on monolingual and code-switching language transfer using two logical reasoning datasets. The key finding from the evaluation is that while multilingual models show high zero-shot monolingual transfer (when both context and question of the task are in the same language), they suffer in the code-switching scenario (where context and question are from different languages). The paper then proposes an attention mechanism to train dedicated cross-lingual query matrices. This proposed mechanism shows improvement in the code-switching transfer experiments. The contributions are: (1) evaluation analysis on code-switching transfer for multilingual models (2) proposal of a new attention mechanism that improves multilingual models' performance in code-switching transfer.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dt7R4wAKRN", "reviewer": "Reviewer_SPqx", "summary": "", "questions": "I have a few suggestions that could improve the clarity of the paper:\n- The reason for using BitFit for fine-tuning models is not clear in the main paper. I would suggest bringing a part of the C1 results to the main paper. This is interesting that BitFit enhances cross-lingual performance.\n-  The results of A.2.1 is quite interesting to me. It would be better to have a short mention of this in the main paper. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comment on the novelty of the analysis and the efficiency of the proposed method.\n\n---\n>**\\[Q1, Q2\\]**: The reviewer has suggestions regarding the explanation of Bitfit fine-tuning and the artifacts of the LeapOfThought dataset:\nIn the main body, we briefly refer to our observation on Bitfit fine-tuning and the artifacts of LeapOfThougth, which are present in the appendix (i.e., C1 and A.2.1). However, as suggested by the reviewer, we are willing to move the related information from the appendix to the main body for the final version.\\\nWe will add more details on these aspects with the extra page we get for the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has a fresh look at analyzing the cross-lingual reasoning ability of multilingual models using code-switched structured instances. Besides, the proposed method is a computationally efficient approach (compared to the pre-training) that can adopt existing multilingual models to the code-switched data. Structurally, the paper is well-written and easy to follow. ", "reasons_to_reject": "Some missing details in the results (or the motivation) make understanding difficult. I've added more specifics in the \"Questions for the authors\" to clarify the findings.", "questions_to_authors": "I have a few suggestions that could improve the clarity of the paper:\n- The reason for using BitFit for fine-tuning models is not clear in the main paper. I would suggest bringing a part of the C1 results to the main paper. This is interesting that BitFit enhances cross-lingual performance.\n-  The results of A.2.1 is quite interesting to me. It would be better to have a short mention of this in the main paper. ", "paper_topic_and_main_contributions": "The paper examines the cross-lingual reasoning ability of mBERT and XLM-r models. Using the Google Translator API, the authors translate the RuleTaker benchmark and LeapOfThught dataset from English into eight languages, including French, German, Chinese, Russian, Spanish, Farsi, Italian, and Arabic. They evaluate the performance of multilingual models in the cross-lingual zero-shot setting and code-switched format where the context and statement languages are not the same.\nThe proposed results show that in the monolingual and cross-lingual settings (where the fine-tuning language differs from the evaluation), the multilingual models have high performance on this task. However, as the depth of reasoning increases, the performance drops. The results show a similar trend in the code-switched setting where the fine-tuning and evaluation data have the same languages (e.g., en-fr). Nevertheless, the performance is pretty low when the statement or context language is not English and differs from the fine-tuning language.\nTo enhance the cross-lingual performance of this setup, the paper suggests a cross-lingual attention approach. In the proposed method, they learn a new cross-lingual query matrix using code-switched unsupervised data. The learned query matrix could be shared between all possible language pairs or be specific for two languages. During the evaluation, they use the cross-lingual query matrix instead of the fine-tuned one. The results show that the proposed method is effective for both monolingual instances and code-switched ones.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 108: an extra 'a'. "}, "extra_text": {}}, {"review_id": "WxCgb05wjk", "reviewer": "Reviewer_ocuD", "summary": "", "questions": "A. Based on the experimental results, what is the eventual answer to the research question of whether MLMs can transfer logical reasoning abilities to other languages? Do the results provide conclusive evidence of successful cross-lingual transfer in reasoning tasks, or are there limitations and challenges that need to be addressed?\n\nB. Throughout the discussion of the results, several hypotheses are proposed to explain certain observations (e.g., line 221). Could any evidence or analysis be provided to support each of these hypotheses? Also, how does the lack of code-switched data in pre-training affect the reasoning abilities of MLMs (line 070)? Are any experiments or analyses conducted to validate this hypothesis and establish a stronger connection with the proposed attention mechanism?\n\nC. Without any reference to the previous works and comparison in the same settings, how to verify, for example, whether the implementation and the reported results are correct or not?\n\nD. How to measure the independent contributions of each module of the proposed self-attention mechanism and assess the significance of the improved performance? For instance, can the improvement be primarily attributed to the regularization effects by the dropout over the attention metrics that mitigate potential overfitting issues?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments about our thorough experiments and analysis and our novel perspective on code-switching.\n\n---\n### Questions:\n\n>**\\[Q_A\\]:** The reviewer asks for an eventual answer for the logical reasoning ability of MultiLMs. Our experiments and analysis conclude that multilingual language models can exhibit strong logical reasoning capabilities in monolingual settings and show high cross-lingual transfer performance. However, they struggle when it comes to the code-switched setting. So, the main challenge and limitation is that these models fail to generalize their reasoning ability in a code-switched scenario. The code-switched setting can be commonly found in realistic scenarios where questions and contexts are in different languages, and it is important for multilingual language models to be able to handle such a scenario.\n\n---\n>**\\[Q_B\\]:** The reviewer asks for clarification on line 221, regarding potential sources of systemic bias in the LeapOfThought dataset. Our hypothesis is based on the (empirical) observation of artifacts in the Explicit development set of the English LoT dataset (see Appendix A2.1). We hypothesize that some artifacts in the Implicit dev set also exist that might enable the model to learn in-language shortcuts that are not generalizable to other languages.\\\nThe reviewer also asks for more explanation about why the lack of code-switched data in pre-training affects the model’s performance in our multilingual setting:\nLine 070 highlights MultiLMs’ struggle to reason over code-switched data (i.e., non-monolingual reasoning). These models have not seen code-switched data during pre-training, so they are not optimized for such an input. Hence, when the data format and domain are not consistent for pre-training and fine-tuning, the model is more likely to overfit the fine-tuning data and not transfer to other data formats.\nTo demonstrate the effect of adding code-switched data for the pre-training phase, we have a code-switched baseline (**CS-baseline**), in which the MultiLM is pre-trained on a code-switched version of XNLI (continue LM pre-training using a code-switched data). This baseline improved transfer results significantly for en-X format, showing the inclusion of code-switched data in the pre-training phase and making it consistent with the fine-tuning phase plays a role in the model’s performance to reason over code-switched data (Table 3). However, this baseline negatively affects the performance in the monolingual scenario and performs worse than our proposed cross-lingual query matrix method.  \n\n---\n>**\\[Q_C\\]:** The reviewer is concerned that our method is not compared to results from previous work. As we are the first ones to design an experimental framework for code-switched logical reasoning, previous work has been done only on English data. We are happy to include the English baselines from these work for the final version of the paper to make it easier for the reader to compare. Here are the baseline results (accuracy) for English using our implementation on RoBERTA-Large (used in both RuleTaker and LeapOfThought papers):\n>>|     Model     | RuleTaker [1] | RuleTaker | RuleTaker | RuleTaker | LeapOfThought [2] |\n>>|:-------------:|:-------------:|:---------:|:---------:|:-----------------:|:-----------------:|\n>>|               |    Depth-0    |  Depth-1  |  Depth-2  |  Depth-3  | Implicit dev set  |\n>>| RoBERTA-Large |    100.00     |   99.67   |   99.22   |   98.58   |       88.39       |\n>>|     mBERT     |    100.00     |   93.37   |   88.00   |   88.46   |       81.15       |\\\n\n>We note these numbers are higher than our english-language results for mBERT, but that (1) mBERT is based on the BERT-small architecture of ~110M parameters while RoBERTa-large has ~345M parameters, and (2) RoBERTa-Large was trained on far more data. We will publicly release the code and the datasets used in this work, along with the final version of the paper.\n\n---\n>**\\[Q_D\\]:** The reviewer asks for the contribution of each component of the proposed method (structured dropout & cross-lingual query). \nWe have done an ablation study to evaluate the effect of each component in isolation in Tables 10 and 11 (in Appendix D.1 and D.2) and mention them in the main body (lines 317 and 328 of Section 4). The results show that both cross-lingual query matrix and structured drop attention play roles in the improvement of the model’s performance.\nWe will move these results from the appendix to the main body with the extra page we get for the camera-ready version.\\\n[1] Clark, Peter, Oyvind Tafjord, and Kyle Richardson. \"Transformers as soft reasoners over language.\" arXiv preprint arXiv:2002.05867 (2020).\\\n[2] Talmor, Alon, et al. \"Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge.\" Advances in Neural Information Processing Systems 33 (2020): 20227-20237.\n\n---\n### Concern Statements:\n>**\\[Concern Statement 1\\]:** \nThe reviewer is concerned that the paper is compact because we address two research questions. We agree that this is a dense paper; however, because no prior work addressed the cross-lingual logical reasoning in MultiLMs, we had to first evaluate their performance on this task to highlight the problem in the code-switched setting and then propose a method to mitigate the problem. We view highlighting the problem as an important contribution of this paper.\n\n---\n>**\\[Concern Statement 2\\]:** \nThe reviewer is concerned about the research question not being addressed clearly throughout the paper. In both the Abstract and Introduction, we set two main research questions for the paper: 1) studying the cross-lingual transfer ability of MultiLMs in monolingual and code-switched settings, and 2) proposing a method to improve the cross-lingual performance of the model in logical reasoning. Section 3 is dedicated to answering the first question and highlighting the model’s struggle in a code-switched setting. We propose a solution to mitigate the problem in Section 4, which our results confirm provides an improvement.\n\n---\n>**\\[Concern Statement 5\\]:**\nThe reviewer has suggestions for the structure of the paper.\\\nWe thank the reviewer for pointing out what could be clearer. If accepted, we will incorporate some of these suggestions in our revision for the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper breaks new ground to study the reasoning capabilities of MLMs in multilingual settings. By introducing code-switching as a fresh perspective, the paper extends the evaluation of MLMs to more realistic and challenging scenarios, providing valuable insights into how these models handle logical statements in different language contexts, particularly under code-switching.\n+ The paper is well-motivated. Firstly, previous works on logical reasoning of language models have mainly focused on monolingual settings, primarily English. Secondly, while there have been studies on the cross-lingual transfer of multilingual language models, they have focused on tasks related to language understanding rather than explicitly investigating the transfer of reasoning abilities. Lastly, the paper highlights that the current multilingual evaluation protocols do not encompass the code-switched setting, which is a common occurrence in real-world language usage.\n+ To overcome the challenges of code-switched reasoning, the authors propose an attention mechanism called cross-lingual aware self-attention. By incorporating techniques such as cross-lingual queries, structured attention dropout, and interfering cross-lingual queries, the proposed mechanism aims to encourage the model to effectively utilize cross-lingual information and align it with the given context.\n+ The paper conducts thorough experimental evaluations on benchmark datasets, specifically RuleTaker and LeapOfThought. By providing empirical evidence of the effectiveness of the proposed method on these datasets, the paper establishes a benchmark for performance comparison and enables future research to build upon these findings in the direction of code-switched reasoning of MLMs.", "reasons_to_reject": "+ The paper attempts to lie its contribution in two research goals: evaluating the cross-lingual logical reasoning abilities of MLMs and proposing a method to enhance their multilingual reasoning under coder-switching. However, due to the compressed content, it can be challenging to fully understand and grasp both aspects. For example, although the lack of code-switched data in pre-training is mentioned as a potential reason, there are no subsequent tests or strong connections with the proposed attention mechanism. Meanwhile, the proposed method lacks support from further analysis, and many details can only be found in the Appendix. This limits the clarity and thoroughness of the contributions.\n+ The research question is not clearly addressed throughout the paper. While the initial goal is stated as studying the transfer of logical reasoning abilities in MLMs across different languages, the paper does not provide a definitive answer and empirical analysis to this question. Instead, the focus shifts towards proposing a method to improve the performance of MLMs in the code-switched setting. This deviation from the original research question could cause difficulties in following the idea.\n+ It appears that the experimental results presented in the paper are not built upon previous work with a formally defined evaluation protocol. The authors introduce both baselines and proposed methods implemented by themselves without any reference to the existing works. This lack of comparison makes it challenging to assess the reliability of the results and the reproducibility of the proposed methods.\n+ As mentioned in the first point, there is a lack of clarity on the specific contribution of each module of the proposed cross-lingual-aware self-attention mechanism. While the experimental results indicate that the attention mechanism improves performance, without a detailed analysis or ablation study to isolate the impact of each module, it becomes challenging to understand the proposed methods and assess the significance of the observed improvement.\n+ The presentation of the work is not clear. The non-standard organization leads to confusion and hinders understanding. For example, the problem definition is embedded within the motivation section. The analysis of reasoning over monolingual data under the multilingual reasoning section seems out of place and lacks clear justification. The conclusions drawn from the analysis are not explicitly stated, creating a gap between this analysis and the later proposed method. Throughout the experiments, numerous settings involve various language pairs and methods; however, they lack clear and formal definitions and introductions in the first place. As a result, it becomes challenging to follow the flow and understand the statements from these results.", "questions_to_authors": "A. Based on the experimental results, what is the eventual answer to the research question of whether MLMs can transfer logical reasoning abilities to other languages? Do the results provide conclusive evidence of successful cross-lingual transfer in reasoning tasks, or are there limitations and challenges that need to be addressed?\n\nB. Throughout the discussion of the results, several hypotheses are proposed to explain certain observations (e.g., line 221). Could any evidence or analysis be provided to support each of these hypotheses? Also, how does the lack of code-switched data in pre-training affect the reasoning abilities of MLMs (line 070)? Are any experiments or analyses conducted to validate this hypothesis and establish a stronger connection with the proposed attention mechanism?\n\nC. Without any reference to the previous works and comparison in the same settings, how to verify, for example, whether the implementation and the reported results are correct or not?\n\nD. How to measure the independent contributions of each module of the proposed self-attention mechanism and assess the significance of the improved performance? For instance, can the improvement be primarily attributed to the regularization effects by the dropout over the attention metrics that mitigate potential overfitting issues?", "paper_topic_and_main_contributions": "While previous research has primarily focused on the cross-lingual understanding of multilingual language models (MLMs), the area of monolingual reasoning, particularly in the code-switched setting, has received less attention. This paper aims to fill this gap by addressing the challenge of enhancing cross-lingual transfer, with a specific focus on code-switched alignment, in language models. The key contributions of this paper start with a thorough analysis of MLMs in two distinct scenarios: reasoning over monolingual data and reasoning over code-switched data. The findings from these analyses serve as the foundation for proposing an attention mechanism called cross-lingual aware self-attention. This attention mechanism encourages the model to effectively incorporate cross-lingual attention when processing code-switched sequences. To achieve this, the authors employ techniques such as cross-lingual queries, structured attention dropout, and interfering cross-lingual queries. The experimental evaluation conducted on the RuleTaker and LeapOfThought benchmarks provides evidence supporting the effectiveness of the proposed method. Overall, this paper offers a fresh perspective on challenging the cross-lingual transfer of MLMs in the code-switched setting and presents a corresponding method to mitigate the problem.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The paper would benefit from more comprehensive coverage of related works, not only for the purpose of comparison, motivation, and references but also to provide a thorough discussion of the differences between the proposed approach and existing methods. The following one is an example, and many more can be found.\n\n[1] It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning (Tikhonov & Ryabinin, Findings 2021)", "Typos_Grammar_Style_and_Presentation_Improvements": "In general, it is recommended to clearly define the research question, streamline the organization, connect the conclusion from the initial analysis to the introduction of the new method, establish a formal evaluation protocol with references to existing works, conduct additional analysis, and ensure clarity and readability when discussing the experimental results. It may also be worth considering how to strike a balance between showcasing the problem and introducing new methods in a cohesive manner, given limited pages."}}]

Author rebuttal:

Meta-review: The paper focuses on improving the cross-lingual reasoning of multilingual language models (here XLM-R). 

It introduces a learnable query layer (as part of the self-attention layers) trained specifically on code-mixed data to help the model to reason on context in one language to answer a question in another language. 
The method leads to better cross-lingual performance compared to baselines. 

Reasons to Accept: 
- New empirical evidence that adding adapter-like parameters as the query in the self-attention and explicitly trained in the cross-lingual setting helps the cross-lingual reasoning abilities of the MLMs.  (R1, R2, R3)

Reasons to Reject: 
- Lack of mention, discussion, and comparison with related methods such as adapters (Pfeiffer et al. 2020)
- Running the experiments exclusively on machine-translated may lead to noisy results (R3). 
- Cross-lingual reasoning defined in the paper is a synthetic task that requires further discussion and comparison with existing cross-lingual tasks (R1 and R3)
Decision: Accept-Findings