InteMATs: Integrating Granularity-Specific Multilingual Adapters for Cross-Lingual Transfer
Authors: Meizhen Liu
Xu Guo
He Jiakai
Jianye Chen
Fengyu Zhou
Siu Cheung Hui
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual Language Model Enhancement
Cross-lingual Transfer
Parameter-Efficient Method

Abstract: Multilingual language models (MLLMs) have achieved remarkable success in various cross-lingual transfer tasks. However, they suffer poor performance in zero-shot low-resource languages, particularly when dealing with longer contexts. Existing research mainly relies on full-model fine-tuning on large parallel datasets to enhance the cross-lingual alignment of MLLMs, which is computationally expensive. In this paper, we propose InteMATs, a novel approach that integrates multilingual adapters trained on texts of different levels of granularity. To achieve this, we curate a multilingual parallel dataset comprising 42 languages to pre-train sentence-level and document-level adapters under the contrastive learning framework. Extensive experiments demonstrate the effectiveness of InteMATs in improving the cross-lingual transfer performance of MLLMs, especially on low-resource languages. Finally, our comprehensive analyses and ablation studies provide a deep understanding of the high-quality representations derived by InteMATs.

InteMATs: Integrating Granularit-SSpecffi Multilingual Adapters for Cross-Lingual Transfer
Meizhen Liu', Xu Guo?, Jiakai He', .Jaaye Chen', Siu Cheung Hui", Fengyu Zhou'
Shanndng University, "Nanyang Technological University [meizhen. 1iu. n, jiakai. he, jyccn)@mail. sdu. edu. cn
[xu0008 asschui asschui jentu. ventu.edu edu. sg, thoufengyu@sdu. edw. cn
Abstract
Chi al.. 2022; Luo et al., 2021; Conneau et al., 2019) propose to enhance MILMM representations by continuing the pre-training on more parallel data using techniques such as translation language mod- eling (e.g.. InfoXLM (Chi et al.. 2021a) and XLM- ALIGN (Chi et al., 2021b)), contrastive learning (e.g., mSimCSE (Wang et al., 2022)), and so on (Yang al., 2020; Feng al., 2022).
Multilingual language models (MLLMs) have achieved remarkable success in various cross- lingual transfer tasks. However, they suffer poor performance in zero-shot low-resource languages, particularly when dealing with longer contexts. Existing research mainly re- lies on full-model fine-tuning on large paral- lel datasets to enhance the cross-lingual align- ment of MLLMs, which is computationally expensive. In this paper, we propose InteM- ATs, a novel approach that integrates multi- lingual adapters trained texts of different levels of granularity. To achieve this, we cu- rate a multilingual parallel dataset compris- ing 42 languages pre-train sentence-level and document-level adapters under the con- trastive learning framework. Extensive experi- ments demonstrate the effectiveness of InteM- ATs in improving the cross-lingual transfer performance of MLLMs, especially on low- resource languages. Finally, our comprehen- sive analyses and ablation studies provide deep understanding of the high-quality repre- sentations derived by InteMATs.
Nevertheless, these approaches often require fine-tuning the entire backbone model, which is computationally expensive and can result catas- trophic forgetting (Kirkpatrick et al., 2017). More- over, they mainly focus on enhancing sentence- level representations. As a result, their performance gains on document-level cross-lingual tasks are not as impressive sentence-level ones (Hu et al., 2020), indicating a need for finer-grained perspec- tive in enhancing cross-lingual alignment. Motivated by recent advancements in parameter- efficient adaptation for large language models (LLMs) (Ding et al., 2022; Guo and Yu, 2022), we propose a novel approach named InteMATs, which stands for Integrating tranularity-speciin Multi- lingual Adapters, to enhance cross-lingual transfer performance of MILMs. Adapter tuning works by training set of adapter modules conditioned on frozen LLM (Houlsby et al., 2019). They can equip MLLMs with new knowledge (Hou et al.. 2022) and facilitate language-specifi adaptation (Pfeiffer et al., 2020, 2022) without modifying pre- trained parameters. Different from them, we offer a new perspective for enhancing cross-lingual align- ment by exploiting set of multilingual adapters pre-trained different levels of text granularity.
1 Introduction
Multilingual language models (MLLMs), such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2019), have achieved remarkable success on wide range of cross-lingual tasks (Zweigenbaum al., 2017: Artetxe and Schwenk, 2019a; Nivre et al., 2020; Zhang et al., 2019; Clark et al., 2020; Artetxe et al., 2020a). They are pre-trained on large multilingual data using pretext tasks such as masked language modeling (MLM). To apply MILMs to downstream tasks, we need to fine-tune a separate copy for each specific task.
To obtain these adapters, we curate mul- tilingual parallel dataset consisting of 42 lan- guages from Wikipedia' and process them into the sentence-level corpus (Dgr) and document-level corpus (Dirr). Recent findings (Park et al., 2023) reveal that contrastive learning (CL) tends to ex-
However, due limited data during pre-training. MLLMs may struggle capture cross-lingual alignment for low-resource languages, resulting in subpar performance. To address this issue, sub- sequent works (Chi et al., 202 lb; Liu et al., 2022;
htttps:/een https:://enwwikipeeeee wwkipedia org/wiki htte/wikiMMaiinPPPP Page

---

tract global information, whereas masked image modeling (MIM) focuses on capturing local fea- tures, which well explained earlier results (Wang et al., 2022). Therefore, we employ CL to train adapters to capture global cross-lingual alignment information to augment the MLLM representations that may initially focus extracting local informa- tion. Specifically, we first train set of multilingual adapters on Dst and DDT respectively. Then, learn to fuse them by incorporating layer-wise fu- sion modules (e.g., AdapterFusion (Pfeiffer al., 2021)) and train them on downstream data to deter- mine the fusion weights.
research specifically addressing the enhancement of document-level cross-lingual representations.
Adapters for MLLMs Recently Adapter-based approaches (Houlsby et al., 2019; Pfeiffer et al., 2020; Artetxe et al., 2020b: Li and Liang, 2021; Pfeiffer et al., 2022) have gained popularity as a parameter-cfficient alternative to traditional fine- tuning for large language models (LLMs). These adapters are inserted between the transformer lay- ers and are learned from data conditioned on a frozen LLM. They learn language-specifie transfor- mations to facilitate quick adaptation to new lan- guages (Artetxe et al., 2020b) or new tasks (Pfeif- fer et al., 2020). Recent research show that it is possible to inject new knowledge into MLLMs to enhance their cross-lingual representations (Hou et al., 2022). Different from previous research, this paper fills the gap of granularity perspective in cross- lingsll alignment via adapter tuning.
Extensive experiments five kinds of cross- lingual transfer tasks, including sentence-level and document-level ones, demonstrate that InteM- AT's significantly improves the cross-lingual trans- fer performance of MLLMs (i.e., mBERT and XLMR). In particular, InteMATs excels the state- of-thc-art baseline by 4% on BUCC (Zweigen- baum et al., 2017), 7% on Tatoeba (Artetxe and Schwenk, 2019a), and 3.5% on TydiQA (Clark al., 2020). Notably, InteMAT's brings substan- tial 30% improvement over its backbone MLLMs in low-resource languages that are never seen dur- ing pre-training, demonstrating the high quality of the representations Icarned by InteMATs. We finally conduct a comprehensive analysis on InteM- ATs to unravel the distribution of contributions on each component and layer-wise wise impact within In- teMATs. We will make our data and model publicly available for future research.
Contrastive Learning for Language Models Contrastive learning (CL) (Gao et al., 2021; : He et al., 2022; Chen et al., 2020a,b) has shown much promise in NLP for its capability of capturing dis- criminative information in an unsupervised manner. Many research works have adopted this pretext task in their MLLM pre-training, such as mSim- CSE (Wang al., 2022), LASER (Artetxe and Schwenk, 2019b), and InfoXLM (Chi et al., 2021a). However, these approaches require fine-tuning the entire backbone, resulting in non-trivial computa- tional overhead. This paper circumvents this chal- lenge by applying CL for pre-training multilingual adapters while leaving MLLMs frozen.
2 Related Works
Cross-lingual Representation Learning Exist- ing researches mainly adopt the full-model fine- tuning approach with monolingual or parallel cor- pus to obtain cross-lingual representations. They employ pretext tasks such as masked language mod- eling (MLM) (Devlin al., 2019; Chi al., 2021b), casual language modeling (CLM) (Conneau and Lample, 2019), and translation language modeling (TLM) (Chi et al., a,), to train MLLMs. How- ever, from the results reported on the XTREME benchmark (Hu et al., 2020), we observe de- crease in cross- lins-llnguall transfer performance for MLLMs as the input text length increases (Hu et al., 2020). Previous researches mainly focus on im- proving cross-lingual alignment for sentence rep- resentations (Wang et al., 2022; Feng et al., 2022: Artetxe and Schwenk, 2019b). There is a lack of
3 Preliminaries
We start by introducing some basic knowledge about adapter tuning and contrastive learning.
Adapter Tuning: Adapter tuning (Houlsby al., 2019) is a parameter-efficient transfer learning technique for adapting large pre-trained models to downstream tasks based on adapters (Rebuffi et al., 2017). These adapters are small, new task- specific modules inserted between the layers of a pre-trained model. Instead of fine-tuning the entire model, adapter tuning only trains the parameters of the adapter modules while keeping the pre-trained model fixed. This approach allows us to specialize a pre-trained model in different tasks while retain- ing the knowledge acquired during pre- training...

---

Following Houlsby al. (2019), we use &w denote pre-trained model with parameters w: (2)). For adapter tuning, a new function, ( is composed, where parameters v denote all the adapter modules and w is copied from pre- trained weights. The architecture of an adapter module is shown in Figure (a). It consists of two feed-forward layers and non-linear activation function. For the hidden states h4 at layer I, an adapter module works as follows:
unraveled the complementary properties of CL and masked image modeling objectives, we use CL train adapters to augment the MLLMs that have been initially trained with the MLM objective. The goal is to enhance the cross-lingual alignment of MLLMs with pluggable adapters while retaining their pre-learned knowledge. In the second stage, show how to integrate these adapters for cross- lingual transfer tasks. In this paper, we employ AdapterFusion (Pfeiffer et al., 2021) for this pur- pose ($4.3). InteMATs, however, is not limited the choice of MLLMs and fusion algorithms. We show the working mechanisms InteMATs and a concrete example of MATs-ST in Figure 1.
(hx) w/ (Rel.U(WPhi)) +.. (I)
where 7? represents the down-projection matrix, W/ represents the up-projection matrix, and ReLU is the activation function. ry represents the resid- ual information from the original input, which by- passes the adapter's transformations.
4.1 Sentence-level Multilingual Adapters
Notations. We designate English as the source language and all the other languages as target languages. To train sentence-level multilingual adapters (MATs-ST), we curate entity-aligned parallel dataset consisting of N sentences in J lan- guages: DSt = {Di} where D { We employ a pre-trained MLLM as the text en- coder and use the hidden states from the penulti- mate layer of the MLLM as text representations. For example, with mBERT, I; is encoded into a sequence of m + token representations: h; =< e((CLS)), e(ti), .... e((EP) >. Here, CLL) and [SEP] are classification and separator tokens specially used for learning positional and structural information.
InfoNCE: InfoNCE (InfoMax with Noise Con- trastive Estimation) (Oord et al., 2018) is a loss function commonly used in self-supervised learn- ing, particularly in contrastive learning methods. The goal is maximize the mutual information be- tween related samples while minimizing it between unrelated samples, facilitating the discovery of dis- criminative features an unsupervised manner.
Given a sample æ, and a set of N random sam- ples, X = { J. which contains one pos- itive sample 2+ and N 1 negative samples x-,, we minimize the following negative log- ::eliike;:hood:
aim(n x//TT
InfoNCE -E log
(2)
Ex.xx (.))
The pre-training objective. For every English sample xt" we randomly select another English sample x.f and N non-English samples x to cre- ate contrastive data pairs. We denote the average hidden states of the sequence of tokens as h,,, and take it the sentence representation for given sentence sample. Similarly, we use her as the rep- resentation for the aligned entity.
where 7 is temperature hyperparameter. Follow- ing SimCSE (Gao et al., 2021), we employ cosine similarity as a measure to compare the representa- tions of positive pairs and negative pairs: h h2 |aa] In this paper, we use pre-trained models such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2019) encode the input texts and only train all the adapters using the InfoNCE objective.
We train MATs-ST on top of a fixed MLLM by minimizing the following contrastive loss on the sentence-level corpus DST:
4 The InteMATs Approach
We now introduce InteMATs, which integrates mul- tilingual adapters to enhance the representations of fixed MILM. InteMATs involves two stages. In the first stage, pre-train two multilingual adapters to be specialized in processing texts of different levels of granularity: sentence-level multi- lingual adapters (MATs-ST) 1)4. and document- level multilingual adapters (MATs-DT) ($4.2). In- spired by the findings (Park et al., 2023) which
(MAT:-ST(V.) = NCE( hy; (3)
DST
+ InfoNCE(h hyi vs), (4) DST
where the superseript en represents English and j [1, J] represents one of the J languages. v, denotes the parameters of MATs-ST. We encourage cross-lingual alignment at both sentence-level and entity-level representations.

---

Add Norm
Adagfer-
Adapter-Pusiie
Updating MAT
cauliyy sentence
Updating
Fuddonn
SoftAlax
MAIs-ST
MILMs with MATS ST
MLLMs win MABS ST
Puramette Share
Add & Neras
us: [ClnncPT] is an untifical
it: [ChsGCPP] e in charbot besato inseligguuz artifiiille
Fi
Up
intillience chatbet developed by Opennl and released November
apprendiento
sviluppato
Add & Norm
FF Dowa
FF
OpenAI convenasione omn mento urreno zhr.
UMC
MATx-DT
MAT :ST
h1
Seutence English aligued Corpus
(u) Trunsformer layerwith InieMATs
(b) Training for MATsST
Figure 1: (a): The structure of InteMATss in each transformer layer. includes two multilingual adapters (i.c., MATs-ST and MATs-DT) and an adapter- fusion module. The fusion module learns the Key, Value, Query matrices to fuse two pre-trained adapters. MATs-ST and MATs-DT DT are fixed when updating the fusion module. MHA means multi-heade attention mechanism. Norm meams addition and layer normalization operation. FEN means the feed-forward neural network. (b): An example of training MATs-ST on the sentence-level parallel corpus. The MLLM backbone is fixed during training.
4.2 Document-level Multilingual Adapter
4.3 Integrating Multilingual Adapters
As Hu et al. (2020) demonstrates, the cross-lingual transfer performance of MILMs tends degen- erate longer texts, potentially due to their lim- ited capability of understanding longer contexts, Motivated by this observation, we specially train document-level multilingual adapters (MATs-DT) for MI.LMs. Similar to the pre-training of MATs- ST, we curate document-level parallel dataset comprising J languages: Divr {D*) For each document in DJ, we also average the hidden states from the penultimate layer an MLLM as the document-level representation: hat- The pre-training objective. The pre-training setup MATs-DT is similar that of MATs-ST except using longer input texts. We encourage adapters to capture cross-lingual alignment on the document- level corpus, DDT, by minimizing the following contrastive loss:
We now introduce the second stage: knowledge composition. Inspired by AdapterFusion (Pfeiffer et al., 2021), which trains multiple spec-fie adapters on top of a shared pre-trained model. We propose to train InteMATs in similar way to fuse the multilingual adapters on cross-lingual tasks. By incorporating both MATs-ST and MATs-DT at each transformer layer, seek a more comprehen- sive view for capturing transferrable features from varying lengths of context.
As illustrated in Figure 1, we append the AdapterFusion module right after the MATs-ST and MATs-DT modules, followed by residual connection to the original transformer output, hr, at layer 1. The outputs of MATs-ST (hat,) and MATs-DT (haz.) are used as inputs for both the Value and Key transformations, and the new hidden states after fusion are as follows:
Q: h2W QI RIxd
(6)
(MATE-DT(Va) = InfoNCE(h 1 va), (5)
Dor
K₁ W,, ks has We), Ki E R2xd
(7)
Vi [wa, W.), Vi E 1xddd
(8)
where Vy denotes the parameters MATs-DT. Note that we do not explicitly include the loss for entity-level aligmment as Eq.4. Our experiments show that incorporating such constraint does not improve cross-lingual transfer performance and, in fact, leads to a decrease in performance. This suggests conflict between entity-level alignment, which focuses on capturing local information, and document-leved aligmment, which emphasizes cap- turing global information.
O softmax(Q;K] / E E
(9)
where d represents the size of hidden states. [. indicates the concatenation of vectors, and
Wg. Wi, W, are d' d matrices for computing cross attentions (Vaswani et al., 2017).
We train InteMATs on each lingull task to determine the task -seccifc importance weights for MATs-ST and MATs-DT. In this process, the

---

parameters of SATs-ST and MATs-DT are fixed while the parameters of the fusion module, vs are optimized by minimizing, e.g., cross cross-entropy loss, on each task- sss--specifie dataset Dr:
dataset (Köksal and Özgür, 2020), we compare with MTMB (Köksal and Özgür, 2020) which intro- duces distant supervision to enhance MLLMs, and MLKG (Hou et al., 2022) which incorporates mul- tilingual knowledge graphs into MLLMs. 4) On XQuAD dataset (Artetxe et al., 2020a), we compare with two adapter-based methods: MLKG (Hou et al., 2022) and MAD-X (Pfeiffer et al., 2020) which uses one task-specific adapter. Note that we only report the results based on large version of XLMM for MLKG and KMLM.
(ImeMATx(BH)
E (m.y)e
llog = u) (10)
5 Experiments
In this section, we present the evaluation details and results of InteMATs across sentence-level and document-level cross-lingual tasks.
5.1 Experimental Setup
Evaluation Setup. We divide the languages in each downstream task into two types: Sup.: stands for supervised language, which is used for fine- tuning, and ZS: stands for zero-shoi language, which is only used for testing.
Pre-training Corpus. We collect large, entity- aligned, multilingual dataset from Wikipedia?. Each data sample is summary text with max- imum length of 381 to describe an entity, which spans multiple languages. The dataset covers 42 languages in total, including those extensively stud- ied in the popular XTREME benchmark (Hu et al., 2020). We treat English as the source language and ensure each English sample has at least four parallel samples from other languages. We utilize the first sentence of each summary text and curate the sentence-level parallel dataset DST for training MATs-ST and use the entire text DDT for train- ing MATs-DV More details about the dataset can be found in Appendix A.I.
5.3 Cross-lingual Semantic Textual Similarity
We first evaluate the models on the Cross- ponss-lingual Semantic Textual Similarity (STS) task (Cer et al., 2017) to assess the quality of their representations in capturing universal semantics. Following mSim- CSE (Wang et al., 2022), we average the embed- dings from the first and last layers.
Table I presents the semantic textual similarity among the Arabic (ar), Spanish (cs), English (en). and Turkish (tr) languages. Comparing InteMATTs (XLMR) with the SOTA model, mSimCSE, we ob- serve that on monolingual groups (ar->an, >ar, es -es), mSimCSE demonstrates the highest textual similar- ity, while on cross-lingual groups (ar-ten, es-ten, tr-ten), InteMATs produces the highest textual similarity. Note that mSimCSE employs the large version of XLMR as the backbone during pre- training. This implies that InteMATs can enhance the representations of XLMR in capturing cross- lingual semantics. Moreover, mSimCSE requires fine-tuning the entire MLLM in the pre-training stage while InteMATs only trains set of adapters (See Appendix 5. 5.1.... However, when conditioned on mBERT, it does not bring improvement, show- ing the limit on the choice of MLLMs.
MLLM Backbones.
We experiment on three representative MILMs, the base version of mBERT (Devlin et al., 2019), both the base and large versions of XLMR (Conneau al., 2019), show the effects on different model types and model scales. Hyperparameters setup under each MLLM can be found in Appendix A.I.
5.2 Baselines
We compare InteMAT's against the following 12 baselines. 1) MILMs without adapters: we di- rectly fine-tune the backbone MLLMs, mBERT, and XLMR (base and large versions), on down- stream cross-lingual tasks. We also compare with another four SOTA MLLMs including VECO (Luo al., 2021), XLM-ALIGN (Chi al., 2021b), KMLM (Liu et al., 2022), and XLM-E (Chi et al., 2022). 2) On sentence retrieval tasks, we compare with two SOTA methods: mSimCSE (Wang et al., 2022) and InfoXLM (Chi et al., 2021a), which fine- tune the entire MLLM on their pre-training corpus before applying to downstream tasks. 3) On RELX
Table I: Spearman rank correlation (p) on the cross -ingull semantic textual similarity task (higher, better).
Models
(5-ben
tr:tcn
Avg.
MSimCSE
72.3
48.4
83.7
57.6
53.4
63.1
MBEKT
23.6
17.3
38.5
InteMATs, (mBERT)
53.4
26.0
66.0
21.4
15.9
36.5
XLM
44.0
15.
65.1
6.5
6.5
26.
InteMATs (XLMB
47.8
33.8
74.5
42.2
43.3
48.3
XLMBa
26.2
10.7
10.5
33.8
InteMATs (XI MR
60.9
59.2
Phctps://g //github con/ www///rtinnnnnnaaoooce httpar/tiiinnnnll////idpieeeeeee

---

5.4 Cross-lingual Sentence Retrieval
(Hou et al., 2022), we report the performance on the source language, Sup(en), the average performance on all the other n zero-shot languages, ZS(n), and the average performance on all languages in the dataset.
We use BUCC (Zweigenbaum et al., 2017) and Tatoeba (Artetxe and Schwenk, 2019a) datasets to evaluate the cross-lingual sentence retrieval per- formance of InteMATs. Specifically, given a sam- ple from the source language, e.g.. English, the model should correctly retrieve all the similar sam- ples from other xx languages, and vice versa. On Tatoeba, we follow XLM-E (Chi et al., 2022) and mSimCSE (Wang et al., 2022) and report the results on both the 14 common languages (Tatoeha-14) and all the 36 languages (Tatoeba-36).
Table 3 presents the performance comparison results. Generally, InteMAT's still takes the high- est spot both datasets, especially zero-shot languages (ZS). Interestingly, the average perfor- mance of InteMAT's matches that of VECO on PAWS-X. However, VECO requires fine-tuning the entire XLMR model and is supported with a huge pre-training corpus covering 50 languages (Luo et al., 2021) while InteMAT's only trains a few pluggable adapters. Compared with its backbone models, mBERT and XLMR, InteMAT's consis- tently enhances their cross- -inguul zero-shot perfor- mance by 2% 3.2%. However, the performance gains on the source language (en) are not consistent, indicating that InteMAT's may focus on enhancing the cross- longull alignment on target languages.
Table 2 presents the overall performance compar- ison. InteMATs, when conditioned on Koges outperforms all the baselines by a large margin, establishing a new State-of-the-as on the unsuper- vised settings of BUCC and Tatoeba. On all se- lected MLLMs, InteMAT's outperforms its counter- parts regardless of the model scale, indicating that there is significant potential for enhancing cross- lingual alignment in the representations provided by pre-trained MLLMs. InteMATs outperforms the second-best state-of-the-art model, mSimCSE, implying that pre-training adapters through CL is better than the traditional full-model fine-tuning approach in aligning cross-lingual representations. Moreover, InteMATs outperforms other competi- tive MLLMs, namely XLM-E and InfoXLM, which employ the same backbone but with distinct pre- training tasks.
Table 3:
The cross-lingut sentence classification perfor- mance. We report F1 score for RELX and Accuracy (Acc.) for PAWS-X. X. Results of VECO, XLM-E, XLM-ALIGN, KMLM, MTMB, and MLKG are taken from their original pupers.
PAWS-X
Moddl
Sap.ten)
28 (4)
Avg.
Sup.tcn)
ZS (6)
Avg.
VECO XIM-ALIGI XLM-6
96.2
87.4
88.7
95.1
85.4
86.8
87.1
KML
88.0
MTMB
63.6
59.6
60.1
MLKG
64.1
60.4
61.1
$1.9
InteMAT's (MBERT)
59.4
60.2
83.2
84.9
Table 2: The cross-lingual sentence retrieval performance. We report the average Fl score of four languages for BUCC and the average accuracy @ score for Tatoeba.
XLMReay
63.1
5911
59.9
91.7
85.0
36.1
InteMAT's (XLM
61.1
61.7
61.7
95.7
87.5
88.7
5.6 Cross-lingual Syntactic Analysis
BUCC-4
Tatoeha-14
Tatoeha-36
Models
We conduct evaluations on a cross-lingual Part-of- Speech (POS) dataset (Nivre et al., 2020) from the
miimCSE
87,5
82.0
78.0
INFOXLM
80.6
77.8
68.6
67.3
XLM-F
74.4
72.3
65.0
62.3
XTREME benchmark (Hu et al., 2020) to assess
MBERTT
36,7
48.8
37.2
34.9
34.3
InteMATss (MBERT)
68.5
59.3
37.0
34.9
model's 's capability of capturing syntactic structures
IntoMATs (XLMR
XLMRsur
80.8
76.2
82.5
61.7
81.1
71.6
29.7
73.8
61.7
7L8
and grammatical properties across 33 languages.
XLMReww
76.7
65.8
42.5
53.5
39.7
All models are trained on English data.
IncMATss (XLMR..))
91.5
92.3
90.9
As shown in Table 4, InteMATss achieves com- parable performance to XLLG-ALINN and outper- forms VECO and XLM-E on average. This sug- gests that instead of fine-tuning the entire MLLMs, transferring the syntactic knowledge acquired from English data to other languages can be casily achieved by only tuning a few adapters. It holds true regardless of the choice of MLLMs. More- over, InteMATs achieves more performance gains for zeros-hoo languages, showing the advantage of adapter tuning for MLLMs.
5.5 Cross-lingual Sentence Classification
We conduct performance evaluation on RELX (Köksal and Özgür, 2020), the cross-lingual rela- tion extraction dataset, and PAWS-X X (Zhang et al., 2019), the paraphrase detection dataset. RELX con- tains five languages and PAWS-X contains seven languages. We follow the XTREM benchmark (Hu et al., 2020) and train adapters only on English data and test on non- -nn-Ennlishhh data. Following MLKG

---

Table 4: The cross-lingual structure prediction performance. Accuracy (Acc.) is used for POS evaluation.
mance of Sen. They are subsequently tested on other languages, yielding an average performance of Sigt. The performance gap. A = Sen - Segt= serves as an evaluation metric to assess the degree of cross-lingual transfer degradation.
Models
POS
Sup.ien)
(32)
Avg.
VECO
95.9
75.1
XLM-ALIGNN
95.9
76.0
XLM-E
74.2
On all but the POS task, InteMATs based on XaMRe demonstrates the lowest cross-lingual performance degeneration, highlighting its advan- tage in enhancing cross-lingual knowledge transfer.
KMLM
72.8
MBERT
95.5
70.8
71.3
72.5
XLMR
85.9
72.2
72.9
IricMATs (XIMR
96.0
72.4
73.1
XLMRow
96.1
73.1
738
InteMATs M
75.6
Table 6: The performance gap (A) between source lan- guage and target languages on cross- cross-lingual transfer tasks. A lower score indicates better cross-lingual transferability.
5.7 Cross-lingual Question-Answering
Question-answerr (QA) requires model to un- derstand given long context so as to correctly answer the questions by extracting the text span for the true answer from the context. We conduct evaluations on two popular multilingual QA bench- marks: XQuAD (Artetxe et al., 2020a) and TydiQA (Clark et al., 2020).
Model
PAWS-X
RELX
POS
XQUAD (FTEMM
TydiQA (FIEM)
(Acc.)
EBE
14.0
24.7
20.9/2500
17.622.1
InteMAT's (mBERT)
11.7
4.3
23.5
22.4/55.9
17.4/20.8
XLMR
9.7
4,0
25
10.9166.4
7.2/16.4
VECO
8.8
21.4
11.31662
4.2/10.2
XLM-ALKiN
12.3/172
8.2/2.8
IacMAAAs (XLMR
21.2
10.5/158
3.0/.6
Table 5 presents the performance comparison between InteMATs and SOTA models. In gen- eral, InteMATs excels all the full-model fine-tuning baselines (VECO, XLM-E, XLM-ALIGN, and KMLM), and adapter-based baselines (MLKG and MAD-X). Conditioned on the same large version of XLMR, InteMAT's surpasses the second-best baseline, KMLM, by 0.3% on XQuAD and 3. 3% on TydiQA, at low cost of training. On both supervised and zero-shot benchmarks, InteMATs consistently outperforms fine-tuning the backbone models, mBERT and XLMR, indicating the ad- vantage of incorporating multilingual adapters into pre-trained MILMs.
5.9 Scaling to Low-resource Languages
We further study how InteMATs performs on lan- guages out of our pre-training corpus. We use eight low-resource languages from the Tatoeba dataset: Cantonese (yue), Vietnamese (vie), Tagalog (tgl), Irish (gle). Georgian (kat), Khmer (khm), Telugu (tel), Serbian (srp).
Table presents the results. We find that di- rectly applying MLLMs on low-resource languages yields poor performance, with an average accuracy of approximately 25% ~ 35%. In contrast, when conditioned on XLMRiarges InteMAT's significantly improves the performance to 62%. These findings confirm our hypothesis that pre-trained MLLMs ex- hibit poor cross munss-lingual alignment low-resource languages due their scarcity of training data dur- ing pre-training. InteMATs effectively enhances MILMSs by capturing better cross-lingual align- ment information, enabling generalization to un- seen low-resource languages. Appendix A.2 pro- vides more details.
Table The cross- cross-lingual We report their Fl score Exact Matching (FI/EM) scores.
performance.
XQBAD
Tyst04
Ave:
Sep.om)
VECO
57,67655
76.38003
77.3/51.
71.3858.2
67. 144.0
67.649.1
MEM-ALBUN
49.7754.
74.1/901
00-496:3
62.194455
SMLM
71.055.7
67.0154
MILKI
NU477A
75 T1M
70-09333
MAIXX
543/726
23.195577
nBERT
57
Inck (IBERT
859033
63.548.
65.5/044
7566631
24233
60:24466
ALMillew
11.31123
35 1NNTST
12 T1233
1133755
Table 7: The performance on the low-resource languages from Tatoeba dataset for cross-lingual retrieval accuracy (Acc.).
InchMAAT (XLMM
91.3719
787888
69.31533
60
46 N24
48.003.7
XUXE-
363957
7512903
71 395
14 3053
TneMATs MR.
87,4764
77.46388
71.0530
Model
suc
vi
ka
km
Ang.
MBERT
3.1
39.4
15.4
103
2022
0.3
17.1
43.7
21.1
5.8 Closing the Cross-Lingual Transfer Gap
InteMATs (nDERT)
26.8
61.0
16.0
11.8
196
0.4
14.1
44.6
24.3
XL MR
24.7
67.7
29.7
16.7
57.3
19.8
308
58.6
35.7
JaseAAT (NLM
38.5
$1.7
42.9
25.5
53.4
35.5
51.7
75.0
50.7
We summarize the above cross-lingual perfor- mance results and conclude that InteMAT's can effectively mitigate the issue of cross-lingual per- formance degeneration in pre-trained MILMs, as shown in Table 6. All models are trained using the source English language, achieving a perfor-
XLMK
213
413
18.3
322
1359
15.7
25.0
47.4
21.
ImeMiTTx MMR...))
52.8
91.6
31.1
08.8
83.6
62.3
5.10 Ablation Study and Analysis
We conduct ablation studies on six tasks to un- ravel the individual impact of each component in

---

InteMATs. Specifically, we compare the backbone MLLMs, MATs-ST, MATs-DT, and InteMATs. The ablation results in Table show that MATs-ST generally outperforms MATs-DT on sentence- tasks, while MATs-DT performs better than MATs- ST on document-level tasks. Both MATs-ST and MAT'sDDT outperform the backbone MLLMs, par- ticularly with substantial gains on BUCC, Tatoeba, and TydiQA By effectively incorporating these two modules, InteMATs achieves the best perfor- mance across the benchmarks.
alignments, transferring this alignment knowledge to understanding tasks and yielding performance improvements.
5.10.2 Analysis of Pre-training Tasks
We evaluate the pre-training tasks of CL's advan- tage in capturing global cross-lingual alignment information. We compare with mBERT-FT, which is fully fine-tuned on the entire raw text Drrs and InteMATs-MLMM, which uses the pre- pre-training task of MLM instead of CL to train each adapters.
Table 10 shows the results. We can observe that MLLMs with adapters can achieve performance en- hancements all tasks but Tatoeba. This suggests employing external adapters for training is more effective for incremental cross-lingual knowledge's learning, than full model fine-tuning. Meanwhile,
Table 8: The ublation results for both MLLMs on various cross-lingual language tasks,
BUCC
Tatoeba (Ace)
RELX
PAWS-X
XQuAD (FNEND
TydiQA (FREM)
Models
(FI)
(FI)
(Aec.)
mBERT bucksbone
MIBERT
56.7
34.6
58.3
81.9
64.549.4
59,74339
MATsST
68.0
34.3
59.9
83.4
65.2449.8
58.142.0
MATs-TT
67.5
35.4
58.7
84.5
65/4/50.7
59,7743.6
InteMAT's excel over other MLLMs on four tasks,
ImcMATs
68.5
36,0
60.2
84.9
65.5/50.4
60,2/44.6
showing average gains of 1.6% over InteMATs- MLM, 2% over Initialized, and 3% over mBERT-
XIMRaag bockbone
XL
76.7
46.6
59.9
M6.4
76.660.88
65.1745.
MAT--TT
91.3
84.3
60.9
88.4
761.
70.1/51.
FT. These results underscore that applying the pre-training task of CL to train adapter can en- hance global cross-lingual alignment information, enabling knowledge transferring to understanding tasks and boosting performance.
MATS-DT
87.6
74.1
59.9
88.0
77. 562.1
70.91533
ImMMATS
91.5
85.4
61.7
88.7
77.666..0
71.15377
5.10.1 Analysis of Pre-training Corpus
We further study whether our proposed different granularities of pre-training corpus boosts MLLMs* performance on cross-lingual understanding tasks. We compare with Initialized (adding a randomly initialized adapter without pre-training) and Mixed (adding an adapter trained on a non-distinguishing granularity of the pre-training corpus), as shown in Table 9.
Table 10: The performance on various goss-lingual tasks under different pre pre-training lasks.
POS (Acc.)
Tatoeba (Acc.)
BUCC (FI)
TydiQA (FIEEM
Model
mBEKT-FT
70.3
37.2
64.1
54 8/40.5
Initialized
71.3
34.1
67.1
57.9/40.6
MM.MA
71.4
35.7
68.5
56.0/40.6
InteMATs
72.5
36.0
68.5
60.2/44.6
Table 9: The performance on various cross-lingual tasks under the MLLMs pre-trained on different pre-training corpus.
5.10.3 Analysis of Fusion Activation
PAWS-X (Ace.)
Tatocha (Acc.)
RELX (FI)
XQuAD (FTEM)
TydiQA (FTEM
Model
InteMATs learns to fuse knowledge from MATs-ST and MATs-DT for different tasks. We retrieve the activation values of its fusion module and visualize the weight distributions for four tasks in Figure 2 (POS, RELX, XQuAD, and TydiQA), with increas- ing input text lengths.
mBERTT backbone
Initaaizzdd
83.6
34.1
57.
65.4/50.4
88.8/4377
Mixed
84.2
37.6
56.4
64.4/49.3
53.5/38.6
InteMATs
65,5/50.0
60.2/44.6
XLMRarg backboue
Initialized Mixed
65.8
60.7
77.4/6..0
69.8/53.1
86.0
73.9
57.4
77.2/61.8
67.8/49.6
InteMATs
88.7
85.4
61.7
77.6/62.0
71.1/5.77
We observe a consistent pattern in InteMATs. where favors MATs-ST for sentence-level tasks and MATs-DT for document-level tasks. Specifi- cally, on POS and RELX tasks, it relies more on the representations from MATs-ST, while on XQuAD and TydiQA, it relies more MATs-DT. More- over, as the network goes deeper, the degree of reliance on MATs-DT increases. This finding con- firms with our intuition that granularity-specin adapters are specialized in handling texts of vary- ing lengths. As result, InteMATs can effectively
We find that InteMATs consistently outperform Initialized across all tasks, with an average im- provement of 2% on mBERT and 4.2% on XLMR backbones. Meanwhile, InteMATs surpass Mixed on all tasks except Tatoeba, achieving average gains of 2.1% on mBERT and 1.4% on XLMR back- bones. These findings emphasize that InteMATss by pre-training individually on more diverse range of text granularities can precisely capture and in- tegrate different text granularities of cross-lingual

---

KOVT
cross-lingual understanding tasks across various text granularities, requiring fewer computational parameters and smaller training corpus.
6
0 9011
2
6
2011
Table 1I: The implementation details of the existing MILMSs and InTeMATs.
Trt MiNgght
Layer
14-
&Pre-trained Purameters
Model
#Lauguages
Data Size
11 9-11 Layer
3015
MMTE XLM-E
375M
103
Layer
840M
100
2,500GB
VECO
662M
1,360GR
Figure 2: InteMAT's activation different layers. We follow
XLM-ALING
279M
2,187GB
the setting from Pfeiffer et al. (2021) calculate the SoftMax activation for the two adapters, MATs-ST and MATs-DT.
MLKG (XLM
20M
36GB
InteMATs (mBERT)
3M
42
2.TGB
InteMATs (
13M
2,1GB
leverage these adapters to enhance cross-lingual alignment regardless of the specific task at hand.
Conclusion
5.10.4 Layer-wise Representation Analysis
In this paper, we introduce InteMATs, novel ap- proach that integrates granularit--specifi multi- lingual adapters, including sentence-leve multilin- gual adapters (MATs-ST) and document-level mul- tilingual adapters (MATs-DT), enhance MLLMS in cross-lingual transfer tasks. On top of fixed MLLM (c.g., mBERT and XLMR), we train these adapters on our curated parallel corpus using the contrastive learning objective. Our experiments demonstrate that InteMAT's significantly enhanced the cross-lingual transfer performance of MLLMs across sentence-level and document-level tasks. Our comprehensive analyses show that InteMAT's can automatically leverage corresponding adapters when dealing with different kinds of tasks.
We examine InteMAT's layer by layer to unravel on which layers it enhances cross-lingual transfer performance. Figure 3 compares InteMAT's and XLMR in both base and large versions. We re- port the sentence retrieval accuracy on the Tatoeba dataset (Artetxe and Schwenk, 2019a) using repre- sentations from each transformer layer.
X0
10
101214161980
Limitations
Figure 3:
Average accuracy 36 language pairs from Tatoeba dataset the xx->en directions.
We identify few limitations of our current work.
We find that InteMATs achieves similar perfor- mance to XLMR in the carly layers. However, in later layers (layer 2 onwards for the base version and layer 9 onwards for the large version), InteM- ATs significantly outperforms XLMR. This com- parison reveals that the cross-lingual transfer capa- bility of InteMATs is gradually developed in the later layers, which provides improved cross-lingual alignment. This finding complies with previous research that later layers of Transformers tend to extract high-level features (Clark et 2019).
First, InteMATss demonstrates limited im- provements on structure prediction tasks, iee., POS dataset. This is not surprising as syntac- tic structures are not universal across different languages. However, is possible to share knowledge between languages from the same family, e.g., Romance languages (es, pt, it, fr, ro). We encourage future researchers to pay more attention to the syntactic cross-lingual alignment for MLLMs.
Second, the publicly available benchmarks for cross-lingual transfer evaluation dom- inated by sentence-level tasks. As a result, performance comparisons on existing bench- marks could be inadequate to demonstrate a model's capability of handling longer contexts and transfer that ability to different languages.
5.11 Model Configuration
We compare the pre-training budget for enhanc- ing the cross-lingual alignments against existing MILMS in Table 11. The results reveal that In- teMATs offers better parameter efficiency during pre-training and performance improvements on

---

A more comprehensive cross-lingual transfer benchmark is needed.
Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey Hinton. 2020a. A simple framework for contrastive learning of visual representations. In In- rernational conference on machine learning, pages 1597-1607. PMLR.
Reproducibility Statement
We elaborate the experiment settings and hyper- parameters in the Appendix A. We will publish our collected parallel datasets soon, as well as our code.
Xinlei Chen, Haoqi Fan, Ross Girshick. and Kaiming He. 2020b. Improved baselines with momentum con- trastive learning. arXiv preprint arXiv:2003. 2449977..
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang. Xia Song. Xian-Ling Mao, He-Yan Huang. and Ming Zhou. 2021a. Infoxlm: An information-thcoreet framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- Language Technologies, pages 3576-33 -3888.
Ethics Statement
All procedures performed in this work were in ac- cordance with ethical standards. We don't have any ethical concerns in this work.
A Acknowledgements
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian- Ling Mao, Heyan Huang, and Furu Wei. 2021b. Improving pretrained cross-lingual language mod- els via self-labeled word alignment. arXiv preprint arXiv:2106.06388
We extend our gratitude to the anonymous review- ers for their valuable feedback and insights. This work was completed at NTU Singapore and was supported by the China Scholarship Council. Fund- ing was also provided by the National Key R and D Program of China (Grant No.2017YFB13N0 1302400)) and Shandong Province Agricultural Major Ap- plication Technology Innovation Project (Grant No.5D201991014).
Zewen Chi, Shaohan Huang. Li Dong, Shuming Ma. Bo Zheng, Saksham Singhal, Payal Bajaj. Xia Song, Xian-Ling Mao, He- Yan Huang, et al. 2022. XIm-e: Cross-lingual language model pre-training via electra. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6170-6188 77882
References
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, 2020. Tydi qa: A benchmark for information-seell question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingua transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4677.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACI. Workshop BlackboxNI P: Analyzing and Interpreting Neural Networks for NLP. pages 276-286, Florence, Italy. Association for Com- putational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross- cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguisties.
Mikel Artetxe and Holger Schwenk. 2019a. Mas- sively multilingual sentence embeddings for zero- shot cross- transfer and beyond. Transactions of the Association for Computational Linguistics. 7:597-610.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In An- nual Meeting of the Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019b. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610 610.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Daniel Cer, Mona Diab, Encko E Agirre, Iñigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarity multilingual and cross- lingual focused evaluation. In The 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, NAACL-HLI 2019. Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 1 11th International Joinl Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 4582- 4597. Association for Computational Linguistics.
Ning Ding. Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan. Weize Chen, al. 2022. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models. arXiv preprint arXiv:2203. 06904.
Linlin Liu, Xin Li, Ruidan He, Lidong Bing, Shafiq Joty, and Luo Si. 2022. Enhancing multilingual lan- guage model with massive multilingual knowledge triples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 6878-6890.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari- agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \Volume I: Long Papers), pages 878-891.
vazhagan, and Wei Wang. 2022. Language- Language-agnostic
Fuli Luo, Wei Wang. Jiahao Liu, Yijia Liu, Bin Bi, Song- fang Huang, Fei Huang, and Luo Si. 2021. Veco: Variable and flexible cross- cross-lingual pre-training for language understanding and generation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 3980-30 33944
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021.
Simese: Simple contrastive learning of sentence em- beddings. In 2021 Conference on Empirical Meth- ods in Natural Language Prucessing, EMNLP 2021, pages 6894-6918 1910. Association for Computational Linguistics (ACL).
Xu Guo and Han Yu. 2022. On the domain adaptation and generalization of pretrained language models: A survey, arXiv preprint (rXxv:2211.031555
Joakim Nivre, Marie- Catherinee de Marneffe, Filip Ginter, Jan Hajic. Christopher D Manning, Sampo Pyysalo, Sebastian Schuster. Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043,
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009,
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.08:: 1807.03748.
Yifan Hou, Wenxiang Jiao, Meizhen Liu, Carl
Allen, Zhaopeng Tu, and Mrinmaya Sachan. 2022. Adapters for enhanced modeling of multilingual knowledge and text. In EMNLP 2022.
Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. 2023. What do self- supervised vision transformers learn? In The Eleventh International Conference on Learning Rep- resentations.
Neil Houlsby, Andrei Giurgiu, Stanistaw Kamil Jas- trzebski. Bruna Halila Morrone, Quentin de Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Syl- vain Gelly. 2019. Parameter efficient transfer learn- ing for ulp.
Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre- training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3479- 3495. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 441 1 1-4421. PMLR.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins. Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwe Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 (14(13):351-3356
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destruetive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503.
Abdullatif Köksal and Arzucan Özgür. 2020. The relx dataset and matching the multilingual blanks for cross-lingual relation classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 340-350.
Jonas Pfeiffer, Ivan Vulić, Tryna Gurevych, and Sebas- tian Ruder. 2020. Mad-x: An adapter-based frame- work for multi-task cross-lingual transfer. arXiv preprint 2r8

---

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances in neural informa- tion processing systems, 30.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Mllia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Yau-Shian Wang, Ashley Wu, and Graham Neubig. 2022. English contrastive learning learn uni- versal cross-lingual sentence embeddings, arXiv prints, pages arXiv-2211.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hemandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, al. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 87-94.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Téchnologies, Volume (Long and Short Papers). pages 1298- 1308.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the second buce shared task: Spot- ting parallel sentences in comparable corpora. In Proceedings of 10th Workshop on Building and Using Comparable Corpora, pages 60-66 67.

---

A Appendix
our adapters effectively facilitate mBERT in captur- ing global representation patterns across languages, thereby enabling substantial cross-lingual enhance- ments.
A.I Training Details
Training Data Statistics. We present the details of the collected parallel training corpus in Table 12, which covers 42 diverse of languages. Especially, for DDr, we filter out summaries whose tokenized input sequence lengths are greater than 384. For both DSr and DDT, we split the corpus into train- ing and validation data in a 9 1 ratio.
I-bomain
Un-Oomaln
Sentenne level Sample Set
Dooument leve Sample Set
Simtiamm tarttrrt
Cosine
Training Settings. Experiments are conducted on 1 Tesla V100 GPUs. We set the maximum input sequence length 384, with a batch size of 8 and accumulation step of 2. The learning rate is set 5e and we use the Adam optimizer with a warm-up step of le4 during training and the random seed is set as 123. Since we adopt the random sampling method and contrastive learning, i.e. we randomly sample aligned text in different languages as positive samples, we only take several 12 hours for training MATs-ST, and 2.5 days at more for training MATs-DT. Both training epochs for MATs-ST and MATs-DT are set as 5 epochs.
neter
MAD
MAD-U7
Intel
noter
M28-57
M3B-0T
Figure 4: The cosine similarities of aligned emheddings of parallel sample sets.
A.3 Detailed Experimental Results
We provide detailed results for each language the cross-lingual language tasks. Specifically, we present the results for cross-lingual sentence-lewe retrieval benchmarks in Tables 13 (based on BUCC dataset), while the results for the cross- cross-lingual re- lation extraction and classification benchmark are displayed in Tables 14 (on RELX dataset) and 15 (on PAWS-X dataset). The results for cross-lingual structure prediction are shown in Table 16. Mean- while, we present the results for cross-lingual ques- tion answering tasks in Table 17 (on XQuAD dataset) and 18 (on TydiQA dataset).
A.2 Scaling to Unseen I Languages
Since we adopt contrastive learning loss for multi- lingual adapters training, aiming to enhance cross- lingual alignments by learning the universal fea- tures across languages. To assess the generalization to unseen languages of InteMATs, we collected two groups of parallel sample sets: sentence-leve corpus and document-level corpus. Both differ- ent level of corpus involve two language set: In- Domain set, including languages in pre-training corpus, and Un-Domain set, including languages out-of pre-training corpus
We employ the cosine similarity to assess the cross-lingual alignments of individual adapters (MATs-ST and MATs-DT) and InteMAT's on both sentence-level and document-level corpus, and mBERT base version is selected backbone mod- els. The results are shown in Figure 4. Compared to the vanilla mBERT, the mBERT with any multilin- gual adapters all consistently and significantly en- hance the similarities of aligned embeddings across different text granularities both the In-Domain sei and Un- Domain set, while InteMAT's demon- strate the highest performance. This indicates that
'The la-Domain set primarily utilizes the validation set of the collected corpus A.I. We collect additional parallel data to form an Un- no-aon set, which includes languages such as dv, gl. ha, ic, ka, ps, ro, sd, st, tg. it, wuu, yuc.

---

Table 12: Statistics about the parallel training corpus in different languages. Meanwhile, means Indo-Puropean, which has the most members. The language selection aims to cover a wide range language tasks.
Language
ISO Code
I Language Family
# of Instances
English en
IE: Germanic
216920
Spanish
es
[E; Romance
98813
Arabic
ur
Afro-Asiatic
20366
Bengali bn
IE: Indo-Aryan
4258
Greek el
IE: Hellenic
4541
Danish da
IF: Germanic
36151
German de
IE: Germanic
100601
French fr
IE: Romance
108312
Hebrew he
Afro-Asiatic
7453
Hungarian hu
Uralic
29473
Indonesian id
Austronesian
15677
Italian it
IR: Romance
79465
Japanese
ja
Japonic
3789
Korean
ko
Koreanic
4784
Dutch
nl
IE: Germanic
89033
Malay
Austronesian
9480
Portuguese
pt
IE: Romance
50019
Polish
pl
IE: Slavic
46420
Russian
IE: Slavic
39592
Swedish
Telugu sv
IE: Germanic
42197
Dravidian
1065
Tamil th
Dravidian
992
Thai
Kra-Dai
1494
Turkish
Turkic
12005
Ukrainian
uk
IE: Slavic
19650
Vietnamese
vi
Austroasiatic
15880
Mandarin
zh
Sino-Tibetan
Basque
eu
Basque
$7766
Swahili
sw
Niger-Congo
22308
Javanese
jv
Austronesian
16855
Estonian
ct
46641
Tagalog
tl
Austronesian
12577
Yoruba
yo
Niger-Congo
7407
Afrikaans af
IE: Germanic
34982
Urdu
ur
IE: Indo-Aryan
40075
Burmese
my
Aino-Tibetan
5867
Georgian
ka
Kartvelian
25455
Malayalam
ml
Dravidian
38048
Marathi
mr
IF: Indo-Aryan
31969
Bulgarian
bg
IE: Slavic
39987
Kazakh
kk
Turkic
21018
Persian fa
IE: Iranian
74166
Table 13: The detailed results of Fl scores at the embeddings of layer-7 for mBERT and layer-13 13 for XLMR on BUCC dataset.
Model
de
fr
zh
A4g.(4)
mBERT
64.3
62.5
53.6
57.5
MATs-ST
72.1
74.1
63.2
MATs-DT
72.4
71.8
62.0
67.5
InteMAT's
72.7
70.5
65.9
65.0
68.5
XLMR wuge
83.8
73.2
82.9
67.0
76.7
SA
94.1
89.4
93.4
88.3
91.3
MATs-DT
90.9
84.4
89.6
85.7
87.6

---

Table 14: The detailed results of Fl score on RELX dataset for each languages.
Models
Sup.(en)
de
fr
tr
ZS,(4)
Avg((5)
mBERT
61.8
57.5
57.9
58.3
55.8
57.4
58.3
MATs-ST
58.5
63.1
60.2
59.3
58.2
60.2
59.9
MATs-DT
59.6
58.8
61.9
57.5
55.8
58.5
58.7
InteMATs
63.7
59.9
61.8
58.1
57.6
59.4
60.2
XLMR
63.1
58.0
59.5
59.1
59.1
MATs-ST
61.2
61.0
60.3
62.1
59.9
60.8
60.9
MATs-DT
61.3
62.3
59.8
54.7
59.6
InteMATs
61.1
61.8
62.1
61.5
61.2
61.7
61.7
Table 15: The detailed results of Acc. on PAWS-X dataset for each languages.
Models
Sup.(en)
de
ja
ko
zh
ZS.(6)
Avg.(7)
mBERT
94.0
85.7
87.4
73.0
69.6
77.0
81.9
MATs-ST
94.5
87.1
87.5
74.9
73.2
79.5
82,9
83.4
MATs-DT
93.7
86.5
89.2
88.5
77.9
75.3
80.1
81.6
84.5
InteMAT's
94.9
87.8
89.3
88.1
78.5
75.3
80.3
84.9
XLMR
94.7
89.7
90.1
90.4
78.7
79.0
82.3
85.0
86.4
MATs-ST
95.8
91.8
92.1
92.4
80.1
82.3
84.3
87.2
88.4
MATs-DT
95.1
90.7
91.7
91.9
80,9
81.6
84.1
86.8
88.0
InteMATss
95.7
92.7
92.3
92.4
80.7
82.7
84.4
87.5
88.7
Table 16: The detailed results of Acc.
POS dataset for each languages.
Model
Sup.(en)
ar
lg
de
el
es
et
eu
fu
fr
he
hu
ja
mBERTT
95.5
86.6
56.2
65.0
85.2
81.1
86.9
79.1
60.7
66.7
78.9
84.3
56.2
67.2
78.3
71.0
88.4
49.2
MAT=DT
95.4
87.0
53.7
87.4
87.4
84.0
87.8
80..1
64.4
67.2
79.4
83.7
53.7
65.4
79.5
71.3
88.7
72.1
InteMATs
95.3
86.0
53.9
88.0
86.8
83.1
88.3
80.2
63.8
66.3
79.4
86.8
53.7
68.2
79.2
71.5
39.5
46.6
67.5
SA
96.0
66.9
88.6
89.0 .0
88.2
86.6
86.3
73.4
69.1
85.9
86.2
65.8
73.8
83.1
73.0
88.0
25.9
MAT&DT
96.1
89.1
67.7
88.7
89.1
87.6
88.9
86.5
74.6
70.8
85.9
87.4
67.2
74.3
82.6
72.4
89.3
37.8
96.1
88.
85.8
Model
lik
vi
zh
7.8.32)
Avg.(33)
MBERT
70.3
49.5
69.4
56.2
83.3
59.0
75.9
41.7
$1.4
63.3
$7.0
53.2
55.7
61.6
10.3
71.3
MATs
72.5
49.4
71.7
$8.4
84.8
86.4
60.4
77.5
39.4
83.7
69.8
55.1
56.0
55.3
64.2
71.2
71.9
MAT-DT
46.2
49.0
88.6
86.4
86.4
60.9
76.1
40.5
83.6
67.7
56.8
56.2
71.4
InteMATs
73.2
50.2
73.8
88.6
87.2
87.5
60.3
76.2
40.9
84.4
68.9
56.8
56.3
57.7
65.4
71.8
72.5
XLMR
78.1
53.9
50.8
$9.3
87.8
89.3
65.2
50.0
47.2
92.2
76.3
70.3
50.8
24.6
25.7
73.1
13.3
38.3
MAT=DT
79.4
52.8
80.7
89.6
87.2
90.0
64.8
85.7
52.8
92.4
75.8
58.8
29.1
49
74.9
75.5
InteMATs
79.3
56.2
82.7
89.5
87.7
90.1
65.5
86.8
52.1
93.7
77.8
70.5
58.6
29.7
45.4
74.9
75.6
Table 17: The detailed results of FI/EM XQuAD dataset for each languages.
Models
Sugute)
ar
de
hi
th
tr
vi
vh
75.(10)
Avg/1I)
mBERT
83.5/72.2
61 61.511..1
70.6:51.0
62.6/11.9
75.5/56.9
59.2/16.0
71.3/533
42.7/33.5
55,690.
69.5-19.6
58.0/18.3
62.697.2
61.599.1
MATs
85.47338
62.7445.4
72.9156.6
63.144..5
75.3/56.6
57.7442.9
72.955
44.1/33.
54,033.88
70.155.3
58.4/48.6
63.2/47.4
65.2/49.8
MAT--TT
85.073.9
6369
75.6/58.2
55.300.7
65.4/5077
ImtMMTss
63.0447.2
73.1/56.
55.23997
70.5:5011
X
86.575.7
68.6719.0
804863.
19336177
$2.0639
76.059.7
30.061
74,9688
7AR555.6
19.125000
59.350.0
75,N993
76.50008
MAT-ST
88.3/77.1
75.959.2
$1.83355
806661.7
83.086.33
76.760.1
80.163.6
73.860.3
76.66111
M0AV0077
53.2445.2
76.299.
77.5661
MATs DT
87.275.88
75.6/58.2
802263.0
804463.4
$2.3169.0
76.9/59.8
$0.5/61.9
74./624
76.880L1
81.5/62.7
56.5/88.4
76.58007
77.5462.
IneeMAT
87.476.4
75.3/58.2
X1.785.3
8046619
83.16649
76.8/9.7
50.16645
74.76612
77.1808
30A8611
56.64884
76,6686
77.6/62.1
Table 18: The detailed results of FI/EM score TydiQA dataset for each languages.
Models
Sup.(en)
id
2S.18)
Avg.9)
mBERT
75.3/63.6
62.2/42.8
49.3/32.7
59.7/45.3
64.8/45.8
58.8/50.0
60.0/38.8
57.5/37.9
49.6/38.4
57.7/41.5
59.7/43.9
MATs-ST
73.7/61.8
61.0/99.8
48.0/319
59.0/41.8
63.8/46.4
51,9/4.00
61.5/42.0
59.5/36.
44.7/36.6
163969
58.1/4 1/4.0
MATs-DT
74.0/63.2
63.4/112
55.4/40.7
59.3/42.1
63.3/47.1
54.6444.6
60.0/40.9
$7.1/34.1
49.9/38.6
$7.8/41.2
59.7/43.6
InteMATs
75.6/63.1
64.7/43.2
53.5/41.8
50.3/42.7
63.6748.7
53.9/44.1
62.3/42.5
58.9/36.6
49.8/38.4
58.2/42.3
60.2/44.6
XLMRan
51.665..8
67.6/30.4
64.044788
70.5/53.2
77.4661.9
31.9/10,9
67.0/42.1
66.1/48.1
70.1/43.6
64.3/43.5
65.1/45.0
MATs-TT
73. 166.1
73.3/52.9
64.2/4511
72.5/54.7
5/6.3
59.7/45.7
66.7/39.2
74.6/55.6
70. 1/51/11..
70.1/51.
MATs-DT
73.9/61.1
74.6/55.4 4
66.4/52.1
73.9/57.5
78.9163.5
59.7/46.4
67. 1/38.8
69 2/50/
74.8/53.7
70.9/53.3
70.9/53.3
InteMATs
73.8/61.3
75.7/55.5
74.3/88.4
78.8/64.6
68.5/50.6
70.8/52.7
71.1/53.7


Reviews:
[{"review_id": "DPD6z0LLUe", "reviewer": "Reviewer_R9P3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive suggestions and positive comments on the soundness and significance of our approach.\n\nQ1. **[A fairer comparison: applying contrastive learning (CL) to XLMR]**:\n\n- **[Full-model finetuning of MLLMs with CL]**: The baseline models, mSimCSE and InfoXLM, were obtained by applying CL to train the entire XLMR model on multilingual data pairs. They have already proved the effectiveness of using CL as a pretraining objective to enhance the cross-lingual representations of MLLMs. Nevertheless, they need to train the entire MLLM backbone and therefore consume much more language data. E.g., InfoXLM uses 93 languages while ours uses only 43 languages. The cross-lingual performance of our parameter-efficient solution, InteMATs, is 4-7% higher than mSimSCE and 7-20% higher than InfoXLM on BUCC and Tatoeba.\n\n- **[Parameter-efficient tuning of MLLMs with CL]**: The baseline, MLKG, has proved the effectiveness of using CL as a pretraining objective to incorporate knowledge into MLLMs with adapters trained on knowledge graph triples. Yet, the pretraining corpus of MLKG is 36G in size. Our approach employs only 1.95G pretraining data but achieved around 1 percent higher than MLKG on zero-shot languages of the RELX and XQUAD benchmarks. Please refer to Table 10 for a detailed comparison of parameters and data consumption.  \n\n- **[Our value]**: We put forth the hypothesis that text granularities may play a crucial role in determining the effectiveness of cross-lingual alignment during pre-training for different tasks. Figure 2 shows that sentence-level tasks put higher weights on sentence-level adapters while document-level tasks put higher weights on document-level adapters, which supports the importance of differentiating text granularities in pretraining for different tasks. To avoid manual differentiation, we propose to automatically learn the weights to adapt to different tasks regardless of text granularities. \n\nQ2. **[Missing references]**:\n\n- Thanks for referring us to these references. We will include them in the section of Related Works. In short, Ansell et al. and Wang et al. proposed different approaches to train **language-specific** adapters while we contribute a new kind of generalized **language-agnostic** adapters for MLLMs. Language-specific adapters require training a different set of adapters for every different language while ours do not need. We show that ours can even improve a bit (averagely 15-38% for XLMR) on unseen languages (Section 5.9). All three references focus on limited evaluations such as syntactic analysis and machine translation (Baziotis et al.) while we conducted a more comprehensive evaluation on five kinds of cross-lingual tasks: syntactic analysis, textual similarity, sentence retrieval, sentence classification, and question-answering. \n\nQ3. **[Introduction of baselines]**:\n\n- Thanks for your suggestion. Due to the page limit, we only managed to fit a summary of the baselines in the current version. We will revise Section 5.2 to provide more details and comparisons in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Sound Approach: InteMATs integrates multilingual adapters trained at different granularity levels (sentence-level and document-level). This approach has the potential to address the limitations of multilingual language models (MLLMs) on zero-shot low-resource languages and longer contexts, offering a unique and promising solution to a significant problem in the field.\n\n2. Extensive experiments: The paper presents extensive experiments evaluating the effectiveness of InteMATs across various cross-lingual understanding tasks. The reported performance enhancements compared to existing MLLMs demonstrate the method's efficacy. Moreover, the reduction in the number of parameters updated during the integration process highlights the efficiency gains achieved by InteMATs, making it a valuable contribution to the field.", "reasons_to_reject": "1.Limited Comparative Evaluation: The use of paired data during InteMATs pretraining results in relatively modest improvements on each dataset. The comparison with unsupervised methods like mBERT, XLMR, and mSimCSE is provided as baseline; however, a fairer comparison could be achieved by applying contrastive learning to XLMR first. The absence of a comprehensive benchmarking analysis and failure to address potential trade-offs in model accuracy or other aspects of performance undermines the significance of the reported results.\n\n2. Missing Citations: Despite numerous related papers on multilingual adapters, the authors overlook citing them in their work. Some of the missing citations are listed below for reference. (Please refer to Missing References)\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed InterMATs, a method that integrates multilingual adapters trained at different granularity levels, to address the low performance of multilingual language models (MLLMs) on zero-shot low-resource languages, particularly when dealing with longer contexts. \n\nThe approach involves pre-training sentence-level and document-level adapters using the contrastive learning framework on a multilingual parallel dataset. Sentence-level multilingual adapters (MATs-ST) process texts at the sentence level, while document-level multilingual adapters (MATs-DT) capture global information from document-level texts. The adapters are then integrated using AdapterFusion, enhancing the representations of a fixed MLLM.\n\nExtensive experiments are conducted to evaluate the effectiveness of InteMATs, and the results show performance enhancements across various cross-lingual understanding tasks while updating fewer parameters than other MLLMs.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Ansell et al., MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer, findings of EMNLP 2021\n2. Wang et al., Efficient Test Time Adapter Ensembling for Low-resource Language Varieties, findings of EMNLP 2021\n3. Baziotis et al., Multilingual Machine Translation with Hyper-Adapters, EMNLP 2022\n\n\nOn the cross-lingual transferability of monolingual representations is cited twice. ", "Typos_Grammar_Style_and_Presentation_Improvements": "To enhance the clarity and understanding for readers, I recommend including a brief introduction to each baseline method used in the experiments. This introduction should outline the key features and training methodologies of each baseline and highlight any significant differences between the baselines and InteMATs. Additionally, it is crucial to cite the original papers for each baseline to allow readers to access further details and gain a deeper understanding of the comparison. By addressing these aspects, the paper can offer a more comprehensive and well-informed evaluation of InteMATs against the selected baselines."}}, {"review_id": "NZUo8DulFQ", "reviewer": "Reviewer_JJrR", "summary": "", "questions": "A) Sec. 4.1:\nA characteristic of the proposed method is using a corpus that include \"entity alignment information.\"  I could not imagine the format of the corpus.  Please show examples to readers.  I assume that the entity alignment information means Wikipedia titles among multiple languages or link information in sentences. The example in Fig. 1 (b) seems to be no link information.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and address each of them below:\n\nQ1. **[Which of the corpus, the adapters, fusion module, and contrastive learning worked to improve accuracy?]**:\n\nWe want to clarify that comparing every alternative choice for every component is not the primary goal of this paper. The value of this paper is providing an approach for **automatically integrating** granularity-specific adapters for **any given** task without knowing the text in advance. Examining every step from the beginning produces **too many results** to fit the allowed page limit, and we have put these extensive results in a prior work that has just been accepted. We will put the reference in the final version. \n\n- **[The role of pretraining corpus]**: It is widely accepted that pretraining generally provides a better-initialized model than a randomly initialized one. We additionally report some results here: Equipping mBERT with randomly initialized adapters produces 2% lower performance on Tatoeba, 3% lower on RELX, and 5% lower on TydiQA than with adapters pre-trained on our corpus. Moreover, randomly initialized adapters even undermine the performance of mBERT on many benchmarks. We will release the corpus for future studies.\n\n  | Model | PAWS-X | Tatoeba | RELX | TydiQA |\n  | ---- | ---- |---- |---- |---- |\n  | mBERT | 81.9 | 34.6 | 58.3 | 59.7/43.0 |\n  | +Initilized | 83.6 |34.1 | 57.9 | 55.8/43.7 |\n  | InteMATs | 84.9 | 36.0 | 60.2 | 60.2/44.6 |\n\n- **[The adapters and fusion module]**: We believe we have proved this in Table 8. More analysis can be found in Figures 2, 3, and 4.\n\n- **[The role of contrastive learning (CL)]**: The advantage of using CL as a pretraining objective has been widely studied in previous research. SimSCE is a CL-trained model that continues the pretraining of an MLM-trained language model on additional corpora. mSimSCE and InforXLM are CL-trained multilingual models. MLKG employs CL to incorporate knowledge graphs into MLLMs. etc. The exact result of comparing the choice of CL and MLM for MATs-ST or MATs-DT can be found in our prior work where we demonstrate that CL continues to excel in this problem.\n\nQ2. **[In the low-resource problem, is the corpus or contrastive learning effectively worked to improve the accuracies in Tables 3-7 ?]**:\n\nWe extensively examined the effect of the pretraining corpus and the CL pretraining objective in our prior work. We will make references to relevant conclusions accordingly. However, this paper aims to present a novel InteMATs approach for automatically integrating different granularity-specific adapters. We have compared InteMATs against classical and most recent baselines on five different downstream tasks. The advantages of InteMATs against domain gap (Table 6) and against  **low-resource languages** (Table 7) have already been discussed. The effectiveness of **integration** has also been demonstrated in Table 8, Figures 2, 3, and 4. \n\n- **Does the pretraining corpus benefit low-resource languages?** Yes. We additionally report the results of pre-trained and randomly initialized adapters on Tatoeba as follows. We found that the randomly initialized adapters not only perform worse than pre-trained ones (-38% on XLMR) but can also undermine the performance of MLLMs (-2% on mBERT), indicating that the parallel data can teach adapters with cross-lingual alignment knowledge, particularly useful for XLMR. More results on other low-resource languages can be found in the Appendix from Table 12-16. \n\n  | Model | Avg. | Model | Avg. |\n  | ---- | ---- | ---- | ---- |\n  | mBERT | 24.1 | XLMR | 24.5 |\n  |+random | 21.9 | +random | 24.3 |\n  | InteMATs | 24.2 | InteMATs | 62.3 |\n\nQ3. **[How about training the base pretrained models using the authors' corpora, and then fine-tune the models to the tasks?]**: \n\nAgain, we want to clarify that the corpus is not the only contribution that we make in this paper. We have this result in our prior work where we have found that finetuning the entire mBERT model on the whole pretraining corpus produces 6% lower performance than MATs-ST on BUCC, 2% lower than MATs-ST on RELX, and 1% lower than MATs-DT on XQUAD. Since both MATs-ST and MATs-DT underperform InteMATs in this paper, finetuning the entire model on our corpus must work worse than InteMATs. \n\nQ4. **[About the example of entity-aligned information]**:\n\nWe have given an example of sentence-level corpus in Figure 1 (b). The **ChatGPT** item refers to an aligned entity. More details have been described in Section 5.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Accuracies of various tasks were improved.\n* In the document-level tasks, the accuracy was improved because two adapters effectively worked.", "reasons_to_reject": "* Although the accuracies of various tasks were improved, they were system comparison.  It is unclear what components of the proposed method (the corpus, the adapters and fusion module, and contrastive learning) worked to improve accuracy.  More analyses are necessary.\n\n* Similar to the previous comments, The long context problem was alleviated by the proposed model in Table 8 and Figure 2.  However, in the low-resource problem, it is unclear which component was effectively worked to improve the accuracies in Tables 3-7, the corpus or contrastive learning.\nAdditional experiments are necessary to clarify effectiveness of the components.  For example, the base pretrained models (mBERT and XLM-R) are additionally learned using the authors' corpora, and then fine-tune the models to the tasks.", "questions_to_authors": "A) Sec. 4.1:\nA characteristic of the proposed method is using a corpus that include \"entity alignment information.\"  I could not imagine the format of the corpus.  Please show examples to readers.  I assume that the entity alignment information means Wikipedia titles among multiple languages or link information in sentences. The example in Fig. 1 (b) seems to be no link information.", "paper_topic_and_main_contributions": "This paper presents a multilingual encoder and aims to improve accuracies of NLP tasks in low-resource languages and long contexts.\n\nThe characteristics of the proposed method is as follows.\n1) An adapter for sentences, an adapter for documents, and a fusion module of both adapters were integrated into large pretrained models.\n2) For the corpora, the authors used sentences including entity alignment information and documents obtained from Wikipedia articles in 43 languages.\n3) Only the adapters and fusion module were tuned by contrastive learning that contrast languages.\n\nIn their experiments, accuracies of various tasks (sentence-level and document-level) were improved.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The size of letters in the tables and figures are too small.\nThis is not reader-friendly."}, "extra_text": {}}, {"review_id": "3vqiFOOfnu", "reviewer": "Reviewer_27bY", "summary": "", "questions": "My questions are those raised in reasons to reject:  I would like to better understand impact of each of the experimental choice made by authors \n\n1) training on entity aligned dataset should definitely improve cross-lingual transfer, would it be the main factor of the gains you observe on low-resource languages?\n \n2) I really liked the arguments and idea for contrastive loss training, but I can't see any experimental proof supporting these arguments \n \n\n3) You seem to use average between the first and the last layers as a sample representations (not sure if it is only true for STS task or other tasks as well?). However, the plot on FIgure 3 seems to suggest that the \"best\" repreresentations for cross-lingual transfer seem to be somewhere in the middle.  Quite similar observations were made also at https://arxiv.org/abs/2212.09535. \n How the results would change if you used representations from the layers 6-9 (xlmr-base)/12-16 (xlmr-large) instead?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the value and contributions of our paper to the field. We will revise the experiments' explanations accordingly. \n\n- For the contribution of each of the components in the overall final result, we address specific concerns below:\n\nQ1. **[Does contrastive learning (CL) really capture global features?]**: While this question is beyond the scope of this paper, we’d like to provide our thoughts as follows.\n\nBased on existing research, a quick answer is Yes. The recent findings (Park et al., 2023) from the CV community provided visualized comparisons between the features captured by CL and the features captured by MIM. In NLP, however, it’s not straightforward to define and visualize global and local features from texts. In view of previous results, e.g., SimSCE shows that using CL to continue the pretraining of an MLM-trained language model can produce sentence representations that align better in a geometric space, as evaluated on similarity-based tasks such as sentence retrieval; mSimCSE and InfoXLM extend the use of CL to multilingual settings and show that it is similarly useful to enhance cross-lingual alignment of XLMR; MLKG demonstrated that CL can also be used to teach adapters to learn knowledge graph triples that inherently have certain geometric properties. Therefore, this paper extends from previous findings and uses CL as a pretraining objective to ensure a better cross-lingual alignment capability for our InteMATs. Figure 4 in the Appendix compares the similarity between aligned embeddings, demonstrating that such capability has been effectively acquired by InteMATs.\n\nTheoretically, the loss function of CL uses distance-based measures and it enforces the model to make relevant data closer and irrelevant data far away from each other. This requires the sentence or document representations to emphasize **a set of features** that can make such distinctions in general. Local features, however, only focus on a very small part of the input. It is impossible for a certain kind of local features to clearly distinguish one language from another.\n\nQ2. **[Are all the benefits we see resulting from fusing these adapters?]**:\n\nTable 8 presents the examinations on each of the granularity-specific adapters: MATs-ST and MATs-DT. In general, both sentence-level and document-level adapters can improve the performance of an MLM-trained multilingual model, i.e., mBERT and XLMR. MATs-ST excels at sentence-level tasks (e.g., BUCC and RELX) while MATs-DT excels at document-level tasks (e.g., XQUAD and TydiQA). Fusing these adapters confers the most benefits and the reasons behind this is exhaustively studied in Section 5.10.1 and 5.10.2. \n\nWe show that InteMATs does effectively leverage the corresponding granularity-specific adapters for any given task (Figure 2) without knowing the text granularity in advance. The benefits that InteMATs has brought to a given MLLM increase from the first layer to the last layer (Figure 3), indicating that such capability is mainly acquired on the last layers. \n\nQ3. **[How about training classical adapters (using MLM) on the whole entity-aligned corpus (without distinguishing the granularity)?]**: \n\nThe whole corpus refers to the document-level corpus. We have a prior work that focuses on exploring a more diverse range of text granularities which has just been accepted. Our prior work extensively compared all kinds of pretraining objectives and we found that CL works the best. We bring the results here: training classical adapters on document-level corpus **using CL produces 3-6% higher performance than using MLM** on TydiQA, BUCC, and WK3l. We will revise our explanations and make reference to this paper in our final version. \n\nWe want to emphasize that this paper aims to contribute an approach for **automatically integrating** different kinds of granularity-specific multilingual adapters for **any given task**, and examine whether such integration chooses the appropriate adapters that comply with our intuition.\n\nQ4. **[switch MLM to CL for training adapters]**:\n\nRefer to point 3. The result of MATs-DT in Table 8 is obtained by training adapters with CL on the whole corpus. \n\nQ5. **[Then, consider sentence-level and document-level adapters]**:\n\nPlease refer to Table 8 for the results of incorporating either sentence-level (MATs-ST), document-level adapters (MATs-DT), or integrating both (InteMATs).\n\n\n- We further address the additional questions below:\n\nQ1. **[Is the pretraining corpus the main factor of the gains on low-resource languages?]**:\n\nNo. Both pretraining and integration are necessary. Our prior work shows that pretrained adapters outperform randomly initialized adapters by up to 38% on Tatoeba when conditioned on XLMR. (Please refer to the Q2 of Reviewer 2.) This paper shows that different tasks favor different kinds of granularity-specific multilingual adapters and an effective integration can achieve consistent improvements. Tables 13, 14, 15, and 16 in the appendix present the results on very individual language.\n\nQ2. **[Provide experimental proof supporting the idea for contrastive loss training]**:\n\nIn the appendix, Figure 4 visually presents how CL enhances the cross-lingual alignment for MLLMs.\tAs we mentioned earlier, we have an extensive experimental comparison between CL and MLM in our prior work and we will make reference accordingly in the final version. Again, the proof of CL is not the primary goal of this paper.\n\nQ3. **[Why use the average between the first and the last layers as a sample representation for the STS task?]**:\n\nWe follow mSimSCE to use the average of the first and last layer representations **only** for the STS task for **fair comparison**. In the mSimCSE paper, authors claim that such practice produces the best performance on STS, and the reasons behind were not provided. \n\n**[How about using representations from the layers 6-9 (xlmr-base)/12-16 (xlmr-large) instead?]**: \nWe appreciate this constructive suggestion and we’d like to explore it in future work. This paper is not limited to the evaluations on the Semantic Textual Similarity task only, we have other 4 kinds of cross-lingual tasks: sentence retrieval, sentence classification, syntactic analysis, and QA. For the rest tasks, we just use the last layer representations.\n\nHowever, these deep questions, while worth discussing, do not necessarily override the validity or undermine the soundness of our approach. We thank again the reviewer for driving our attention to so many details and we will make clarifications accordingly in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- all the three proposed contributions (datasets, adapters of different granularities, and contrastive learning) are interesting and useful contributions, the choices are also well motivated and make total sense to try them \n- a lot of baselines were considered and many tasks were addressed, an overview of different methods in the same table is interesting in itself (better presenting these different baseline would help the readability however for the readers who are not necessarily aware of the differences between them)\n - Authors provide an interesting analysis showing the relevance of document-level adapters for the tasks (XSquad, TydiQA)", "reasons_to_reject": "- this work compares proposed method to many other models previously proposed in the litterature. This comparison is quite exhaustive in my point of view and appreciate authors consideres so many baseline \n- However, the major drawback for me is that the contribution of each of the component proposed in this work in the overall final result is not clear: does contrastive learning really favour global features in case of NLP tasks? Or all the benefits we see are from document+sentence level adapters fusion? What if we just train classical adapters on the whole entity-aligned corpus (without dsitinguishing the granulrity): what would be the quality of cross-lingual transfer? If we train such adapters with contrastive loss would it improve cross-lingual transfer? If we distinguish between sentence-level and document-level adapters would it help further? In the current experimental settings authors compare with many different previously proposed methods, but I would assume those baselines were trained on different datasets, and with different training objectives, therefore it is hard to draw meaningful conclusions from these results.", "questions_to_authors": "My questions are those raised in reasons to reject:  I would like to better understand impact of each of the experimental choice made by authors \n\n1) training on entity aligned dataset should definitely improve cross-lingual transfer, would it be the main factor of the gains you observe on low-resource languages?\n \n2) I really liked the arguments and idea for contrastive loss training, but I can't see any experimental proof supporting these arguments \n \n\n3) You seem to use average between the first and the last layers as a sample representations (not sure if it is only true for STS task or other tasks as well?). However, the plot on FIgure 3 seems to suggest that the \"best\" repreresentations for cross-lingual transfer seem to be somewhere in the middle.  Quite similar observations were made also at https://arxiv.org/abs/2212.09535. \n How the results would change if you used representations from the layers 6-9 (xlmr-base)/12-16 (xlmr-large) instead?", "paper_topic_and_main_contributions": "The main goal of this work is to improve cross-lingual transfer in the pretrained multilingual models. To achieve this goal, this work proposes to integrate multilingual adapters of trained on the tasks of different granularities: sentence level + document level adapters. These adapters are then combined via AdapterFusion mechanism. Moreover, inspired by previous work (form CV community) authors suggest that contrastive learning favours global features learning (as opposed to Masked LM training that rather favours local features). Therefore both sentence-level and document-level adapters are trained with Contrastive learning. For training such adapters authors create entity-aligned  dataset  gathered from Wikipedia covering 43 languages in total. The dataset created at sentence-level (first sentence describing each entity)  and document-level (full document). \n\nThus 3 main contributions of this paper can be summarized as\n1) creating of multilingual entity-aligned dataset for sentence-level and document-level granularities\n2) Fusion of sentence-level adapters and document-level adapters in a single module\n3) Contrastive learning of adapters (as opposed to MLM training usually used)\n\nOverall impression: I find that  the ideas this work is advancing are interesting, the experimental results and some of the ablation studies are quite interesting as well.  But I am left with the feeling that the experimental results do not connect well to the originally stated claims.  It is maybe just a matter of the 1-2 more experiments, or better experiments explanation, but in the current status it is quite hard to make this connection.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 4, ZS(32): highest results (XLM_ALIGN) is not in bold"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces, a novel approach to improving cross-lingual transfer in pretrained multilingual models. The method leverages multilingual adapters trained on different granularities (sentence-level and document-level) and combines them using AdapterFusion. These adapters are pretrained using contrastive learning on a newly created multilingual entity-aligned dataset. The paper demonstrates the effectiveness of their approach through extensive experiments, achieving state-of-the-art results on various cross-lingual tasks.

Reviewers raised several concerns, including the need for clearer delineation of contributions, a desire to decouple the impacts of document-level adapters and contrastive learning, and a request for more detailed explanations of experimental choices, particularly regarding low-resource languages.
The authors responded to the concerns by providing clearer explanations of their contributions, emphasizing the non-trivial nature of fusion, and explaining the challenges of decoupling factors tied to the pretraining objective. They addressed the request for more clarity on the impact of experimental choices and offered additional data regarding low-resource languages.

Overall, while the reviewers had concerns about the clarity of contributions and the impact of individual factors, the authors' responses provided reasonable justifications and clarifications.
Decision: Accept-Findings