Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only
Authors: Jinliang Lu
Yu Lu
Jiajun Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: multilinguality
mutlilingual pre-training

Abstract: Recent studies have revealed the remarkable cross-lingual capability of multilingual pre-trained language models (mPLMs), even when pre-trained without parallel corpora (mono-mPLMs). Intuitively, semantic alignments may be the reason behind such capability but remain under-explored. In this work, we investigate the alignment properties from the token perspective in mono-mPLMs and find that the alignments correspond to the geometric similarity of embedding space across different languages. Nevertheless, mono-mPLMs tend to damage this geometric similarity at the higher layers due to the lack of cross-lingual interactions, thus limiting their cross-lingual transfer capabilities. To address this issue, we introduce token-level and semantic-level code-switched masked language modeling, employing the self-induced token alignments to explicitly improve cross-lingual interactions over layers of mono-mPLMs without relying on parallel sentences. We evaluate our method on various natural language understanding tasks and unsupervised machine translation tasks. The results demonstrate that our methods outperform the strong baselines and achieve comparable performance with mPLMs trained with parallel corpora.

"Corresponding author Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only
Jinliang Lu Yu Lu and Jiajun Zhang 1:233
Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Wuhhan AI Research, Wuhan, China {jinliang. lu, yu.lu, jjzhang ia cn
Abstract
Abuulde
coraict
Weak
Hobee Slase
Recent studies have revealed the remarkable cross-lingual capability multilingual pre- trained language models (mPLMs), even when pre-trained without parallel corpora (mono- mPLMs). Intuitively, semantic alignments may be the reason behind such capability but remain under-explored. In this work, we investigate the alignment properties from the token perspec- tive in mono-mPLMs and find that the align- ments correspond to the geometric similarity of embedding space across different languages. Nevertheless, mono-mPLMs tend to damage this geometric similarity at the higher layers due to the lack of cross-lingual interactions, thus limiting their cross-lingual transfer capa- bilities. To address this issue, we introduce token-level and semantic-level code-switched masked language modeling, employing the self- induced token alignments explicitly improve cross- coossllingual interactions over layers of mono- mPLMs without relying on parallel sentences. We evaluate method various natural lan- guage understanding tasks and unsupervised machine translation tasks. The results demon- strate that our methods outperform the strong baselines and achieve comparable performance with MLLMs trained with parallel corpora.'
LM
Sciter
bota
Zoom In
betppaa
Algyneet
(a) The Locatten Token Allgements
The of Taken Alignments
Figure 1: Illustration of properties of token alignments.
from the semantic alignments but remains under- explored. Another line of research involves improv- ing the multilingual pre-training by incorporating the cross-lingual parallel corpora (para-mPLMs) into pre-training (Conneau and Lample. 2019; Cao et al., 2020; Chi et al., 2021a,b,c; Luo et al., 2021; Wei et al., 2021; Ouyang et al., 2021). However, parallel sentences are not always available, es- pecially for low-resource languages (Tran et al.. 2020). And collecting such data often entails sub- stantial costs. Therefore, exploring approaches to improve multilingual pre-training without using parallel corpora is important and worthy of study.
To achieve this, we first conduct analyses in- vestigate the token alignment properties of XLM- R, a strong mono-mPLM that only uses multiple monolingual corpora for pre-training. Our empiri- cal study demonstrates that the cross-lingual token alignments occur at the embedding layer (align- ment location) with surprisingly high alignment curacy (alignment degree) but they become weaker at the higher layers (Figure (a)). We also find that the alignments are geometrically aligned instead of absolutely aligned (alignment format), as shown Figure (b). The phenomenon shows that token embeddings of different languages are separately distributed but geometrically similar.
1 Introduction
Recent studies show that multilingual pre-trained language models (mPLMs) significantly improve the performance of cross-lingual natural language processing tasks. Conventional mPLMs (Devlin et al., 2019; Conneau and Lample, 2019; Con- neau et al., 2020; Xue et al., 2021) typically adopt multiple monolingual corpora (mono-mPLMs) to perform masked language modeling during pre- training, obtaining impressive and stable multi- lingual capabilities, which may intuitively result
We further compare the differences in geomet- ric similarities of representations from the bottom layer to the top layer using mono-mPLMs and para- mPLMs. And we find that the representations be-
'Our code is available in httpp://githhu com/ Jinli InliangLuu66/Seff Improvinn SeaeevinngMuuttlln Multillnguua-PT

---

come geometrically dissimilar at higher layers of mono-mPLMs while para-mPLMs would alleviate the problem by using parallel sentences, obtaining better tross-lingual transfer capability. It shows the necessity explicit cross-lingual interactions. Based on the above observation, we propose self-improving methods to encourage cross-lingual interactions using self-induced token alignments. Intuitively, the masked tokens can be predicted with semantic-equivalent but slightly language-min mixed contexts. Therefore, we first utilize self-induced alignments to perform token-level code-switched masked language modeling (TCS-MLM), which re- quests the model to predict original masked tokens with the semantic-euuivalent but code-switched sur- rounding text. Considering that vanilla replace- ments usually lack diversity, further propose novel semantic-level code-switched masked lan- guage modeling (SCS-MLM) which replaces the context tokens with weighted combination of mul- tiple semantically similar ones in other languages. SCS-MLM involves on-the-fly semantic replace- ments during training, further enhancing the diver- sity of code-switched examples and cross-lingual interactions.
the geometric similarities between represen- lations higher layers while para-mPLMs remain unaffected, showing the necessity of cross-lingual interactions during pre-training.
We propose ooken-levelsemantic-tewe code- switched masked language modeling to en- courage cross-lingual interactions during pre- training, improving the cross-lingual transfer capability without relying parallel corpora.
2 A Closer Look at Multilinguallity
In this section, we take the commonly used XIMM R, the strong mono-mPLM, as an example to show our observation". Specifically, we first investigate the properties of cross-lingual token alignments in mono-mPLMs, showing their relation to geomet- ric similarity. Then, we explore the variation of geometric similarity of representations across dif- ferent layers and demonstrate that the geometric similarity higher layers would be disturbed due to the lack of cross-lingual interactions, hindering the cross- -oons-l transfer capability.
2.1 Language-Specifie Vocabulary
We evaluate our methods on various cross- lingual transfer tasks. Specifically, conduct experiments on natural language understanding tasks, including XNLI (Conneau et al., 2018) and PAWS-X (Hu et al., 2020) for sentence-paii clas- sification, Wikiann (Pan et al., 2017) and UDPOS (Nivre et al., 2018) for structural prediction, MIQQA (Lewis et al., 2020) for question answering, and Tatoeba (Artetxe and Schwenk, 2019) for sentence retrieval. We also perform experiments unsu- pervised machine translation evaluate the perfor- mance of the generation task. Experimental results demonstrate that our methods significantly improve the performance compared with the strong base- lines, even surpassing some mPLMs pre-trained us- ing parallel corpora. Further analysis demonstrates that our methods improve the geometric similarity of representations for different languages, and thus promoting the cross-lingual transfer capability. Our contributions are summarized as follows:
Generally, mPLMs adopt a huge vocabulary shared across 100+ languages. Different languages always have shared and independent tokens. Previous stud- ies (Conneau and Lample, 2019; Pires et al., 2019; Wu and Dredze, 2019) regard the shared token as the source of cross- cross-lingual capability. However, the latent relevance between language-speciff. tokens is not fully exploited.
Suppose that each language in the language set has corresponding corpus Cj. We first record the tokens whose frequeneies are larger than 100 in Ci, obtaining the vocabulary Vi for the specific lan- guage. Then, we remove shared tokens when pro- cessing two languages and ls to avoid the impact of overlapping. Finally, we obtain the language- specific vocabularies independent each other:
Di. {tlt E Vi, and t & V4, freq(t) > 100}
D1 = {1]c & Vi, and i E Vi,, freq(!) > 100}
Our empirical study shows the existence of cross-lingual token alignments in mono- mPLMs. We further measure their accuracy, identify the location, and verify the format.
2.2 Token-Alignments in Mono-mPLMss
After obtaining the language-specific vocabularies from other languages to English, we calculate the
We also conduct analyses on mT5 (Encoder-Decoder) and X-GLM (Deender). The phenomenon also exists regardless of the architecture, which is included in Appendix C.
Comparing mono-mPLMe with para-mPLMMs, we find that mono-mPLMs tend to disturb

---

bg
id
ja
ko
Models
params.
#ig
Avg.
1505
3322
2187
1223
2164
1159
2381
2501
1278
4077
Format of tion is whether the alignments are absolute align- ment or geometrical alignment (Figure 1 (b)). Abso- lute alignment requests the token embeddings are language-agnostie (Artetxe et al., 2017; Lample et al., 2018) 18) while geometrical alignment focuses on the correspondence between tokens different languages (Vulié et al., 2020). The latter just need to have a similar geometric spatial structure while keeping language characteristics (Roy et al., 2020). Hit@ 1 Accuracy
250M
100
44.12
58.61
48.33
52.58
49.03
48.92
47.00
62.38
63.77
59.43
53.42
XLAM-RLAnGn
560M
46.71
58.79
52.25
50.13
47.08
63.97
62.52
59.97
53.90
H Accuracy
XLM-RRASE
250M
100
68.04
72.91
60.86
65.09
59.10
60.74
56.82
75.37
72.77
75.55
66.72
CLM-RLAnGE
560M
69.77
73.81
59.95
66.48
57.95
61.78
56.82
77.09
73.94
76.21
67.38
RSIM scores
XLM-RgAsE
250M
100
0.46
0.64
0.63
0.50
0.64
0.40
0.72
0.52
0.42
0.68
0.56
XLM-R, XLM-RLARGE
560M
100
0.49
0.64
0.56
0.61
0.61
0.53
0.66
0.44
0.45
0.66
0.57
Table I: Alignment accuracy/RSIM translation pairs derived from different sizes of XI.MRRBERTT models
across different languages English. The number below the language is the size of exported cross- mooss-lnngual dictionary.
similarity among token embeddings of XLM-R and directly export high-quality cross cross-lingual token alignments as dictionaries.
The second ques-
Specifically, adopt the cross- doos-dommin similar- ity local scaling (CSLS) (Lample et al., 2018) to compute the token similarity from language X to language Y. For token embeddings æ and y in two languages, the CSLS score computed as:
CSLS(#; v) = 2 cos(x, u) <x( ( (1)
Thus, we visualize the embedding of tokens in the exported dictionaries. The results across five diverse languages shown in Figure 2 (a). We can find that the token embeddings separately distributed in space according the language simi- larity instead of aggregating together, showing that the alignments are geometrical alignment. We also use RSSM* to measure the geometric similarity in Table 1. By contrast, token representations at the top layer aggregate together (Figure 2 (b)).
where rx (x) is the average score from to the K- nearest target neighbourhoods N (xx). And (x()
is vice versa.
1 Σ (EV()) cos(x, Dt) K
TK (x(x)=
(2)
Accuracy of TokknnAAlggmmenss To measure the quality of dictionaries, we collect golden dictio- naries from wiki-dict and MUSE (Lample et al., 2018). The accuracy scores are shown in Table 1. We find that the exported cross-lingual dictionar- ies have good quality, demonstrating that mono- mPLMs learn the alignments between different languages using monolingual corpora only. Par- ticularly, distant language pairs, such as En-Ja (63.97%) and En-Ko (62.52%). have higher ac- curacy scores, while they usually have little over- lapping tokens as anchor points for alignment. The phenomenon directly proves that the cross-lingual ability does not only depend the overlapping tokens in different languages. Another potential factor could be the occurrence words with simi- lar meanings at comparable frequencies across lan- guages (K et al., 2019). Using language modeling as the objective may unearth such regularities and stimulate the cross-lingen lingull transfer capability.
tolan un urbboddkg secallrad
b) D//dddensmaa skalzatin
KIN
Figure 2: The visualization of embedding and states.
Location of Token-Alignments Since top-layer
hidden states aggregate, whether the accuracy of to- ken alignments improves from the bottom to the top layers becomes a question. To answer the question, compute the token alignment accuracy, average
*RSIM means relational similarity, which measures the Pearson correlation coefficient between the cosine similarity matrices of intra-language tokens. It measures the degree of isoorphiss (geometric similarity) of embedding spaces. And the calculation method is included in the appendix B.
Phttps: http://gtttttttete/ httpl//ccooo/////////eeee com//onny//ikidict

---

cosine similarity, and RSIM scores using hidden states of different layers. Figure (a) shows that with the layers becoming higher, token alignment accuracy decreases but the average cosine similar- ity between translation pairs increases, demonstrat- ing that cross-lingual token alignments mainly exist in the embedding layer while higher layers focus on the aggregation of language-specific token rep- resentations. Moreover, Figure 3 (b) shows that the geometric similarities between Language-specitfe token representations at top layers become weaker.
PLMs prioritize complex semantic combinations rather than token features (Jawahar et al., 2019).
Figure 4 (b) compares the average RSIM scores of different mPLMs. VECO and INFOXLM have higher RSIM scores than XLM-R cross layers, showing that parallel corpora would improve the geometric similarity between languages. Further- more, RSIM scores of VECO/INFOXLM across layers are more balanced than XLM-R. It demon- strates that explicit cross-lingual interactions (par- allel corpora) are useful in maintaining geometric similarity in mPLMs, which could be one of the factors contributing better cross-lingual transfer capability than mono-mPLMs.
Taltai Alg1 Acc.
scrass Layar
13H oerom
Our Method
Figure 3: The token alignment accuracy/cosine similar- ity and RSIM across different layers of XLM-R1 XLM-RLACGE. LAACE-
$2.2 demonstrates that mono-mPLMs learn cross- lingual token alignments and export them high-quality dictionaries. $2.3 shows explicit cross- lingual interactions may enhance cross- Pross-lingual transfer capability. These observations motivate us to explore iepfoiing methods to increase cross-lingual interactions without relying on paral- lel corpora. Next, we introduce our proposed token- level/semantic-level code-switch masked language modeling for multilingual pre-training.
2.3 Cross-Lingual Interactions Matters for Geometric Similarity Maintenance
$2.2 shows that language-specific token representa- tions of XLM-R at higher layers aggregate together (Figure 2 (b)) but the alignment and geometric sim- ilarity disturbed (Figure 3). Since para-mPLMMs usually obtain better performance on cross-lingual transfer tasks, we compare the difference between para-mPLMs and XLM-R (mono-mPLM) in the above aspects. Specifically, we choose VECO (Luo et al., 2021) and INFOXLM (Chi et al., 2021 representatives of para-mPLMs, which are pre- trained with monolingual and parallel corpora, ob- taining improvements compared with XLM-R.
3.1 Token-Level Code-Switch MLM (TCS)
Previous code-switch methods either rely on the existing bilingual dictionaries (Lin al., 2020; Chaudhary et al., 2020) or the alignment tools to build the alignment pairs using parallel sen- tences (Ren et al., 2019; Yang et al., 2020). Our work proves that the self-induced dictionaries from mono-mPLMs are accurate enough, which would help mPnss self- teff-iipooiing and do not quire the of prior bilingual knowledge.
(a) Tokkan Align Acc.Aogg Cosine mFMM
RSM min
Therefore, we replace 10~15% tokens using the dictionary to construct multilingual code-switched contexts but keep the masked positions unchanged, forcing the model to predict the masked tokens with different but semantic-eqiivalen contexts. For ample, the original English token sequence (after cutting) is converted into code-switch one using the self-induced dictionary (En-De):
Figure 4: Comparison between XLM-R (mono-mPLM) and VECO/INFOXIM (para-mPL.M), Each score av- eraged by language pairs: En-Ar, En-De, and En-Hi.
Figure 4 (a) shows the phenomenon of token alignment accuracy and cosine similarity across layers. We find that different mPLMs exhibit simi- lar behavior, wherein both mono-mPLM and para- mPLMs tend aggregate token representations while ignoring aligmments at the higher layers. The reason behind this may lie in that higher layers of
A __caa sit _on _the [mask]. + LMLM
4
A Katze _sit _on _the [mask]. + LTCS-MLM Both the original and sode-switched sentences are fed into mono-mPP PPLMs to perform masked lan-

---

guage modeling. The training loss is:
4 Pre-training Settings
C = -MLM + LTCS-MLM
(3)
Pre-training Data We collect monolingual cor- pora from Common Crawl Corpus, which contains about 890GB data for 50 languages. Diffcrent from previous studies, we do not use any bilingual cor- pus. Following (Conneau and Lample, 2019), we sample multilingual data according to a multino- mial distribution with probabilities. Considering the pre-training corpora in N languages with n; training instances for the i-th language, the proba- bility for i-th language can calculated as:
3.2 Semantic-Level Code-Switch MLM (SCS)
Considering that token replacements often lack di- versity, we propose novel semantic-level code- switch method, which replaces 10~15% tokens with the average weighting of its neighbors in an- other language, shown in Figure 5.
Considering that mPLMs provide contextual out- put distributions across the vocabulary and avoid polysemy problems (Tversky and Gati, 1982), we first utilize the mono-mPLM to obtain the output probability distribution across the vocabulary for tokens. Then, we choose top-k tokens according to probabilities and average- weight their embeddings the contextual token representation e::
n Ps
(9)
where is set as 0.7.
Model Configuration Due the restriction of resources, we conduct experiments on the Trans- former encoder models to verify the effectiveness of our method. For fair comparisons with previous studies on natural language understanding tasks, we pre-train a 12-layer Transformer encoder as the BASE model (768 hidden dimensions and 12 atten- tion heads) and 24-layer Transformer encoder the LARGE model (1024 hidden dimensions and 16 attention heads) using fairseq toolkit. The activa- tion function used is GeLU. Following (Chi et al., 202 lb; Luo et al., 2021; Ouyang et al., 2021), we initialize the parameters with XLM-R.
(4)
where p; is the normalized probability of i-th token in the top-k tokens.
After obtaining the contextual representations, we adopt the embedding table search for corre- sponding translations on-the-fi instead of directly using the discrete dictionarics, which would im- prove the diversity of training examples and keep the semantics. Similarly, we also keep top-k trans- lation candidates and average- weighting their em- bedding as the replacement y:
We also pre-train the 6-layer Transformer coder (1024 hidden dimensions and 8 attention heads), which is adopted to evaluate the perfor- mance on unsupervised machine translation.
Ej
(5)
where qj is the normalized CSLS CSLSon-thesyy 6 across the top-k tokens in the corresponding language-specific vocabulary Di
Optimization Settings We use the Adam opti- mizer to train our model, whose learning rate is scheduled with linear decay with 4000 warm-up steps. The peaking learning rates separately set (6) as 2e-4 and le-4 for BASE and LARGE model. Pre- training is conducted using Nvidia A100-80GB GPUs with 2048 batch size. The BASE model takes about month and the LARGE model takes about 2 months for pre-training. Appendix A shows more details about the pre-training settings.
exp(á;)
4j K exp(()) i on (2. (2.
,j), Uj V
(7)
Same as $3.1, we request the mono-mPLMs to perform masked language modeling based on the original examples and semantically code-switched ones. The final training loss is:
5 Experiments on Downstream Tasks
L = LMLM + LSCS-MLM
(8)
5.1 Natural Language Understanding
' is set as in our experiments. "During training, we compute CSI SSe.SSueeeeff scores for to- kens in the same batch to avoid expensive computational costs.
5.1.1 Experimental Settings
We consider four kinds of cross-lingual NLU tasks:

---

MODE)
6. CATOO....
DRSE buntonn
Cdl.
.05
MLL
CAT
EMBEDOING
Wiy
seorch
0.1
cals
0.1
0.1
yrank]
[Twok]
Figure 5:
The illustration of SCS-MLM. First, we use the mPLM to predict the distribution of the context token, obtaining the contextual token representations and searching corresponding translations. Then, we use the weighted representations replace the corresponding token and request the model predict the same masked tokens.
Sentence- Pair Classification We choose XNLI (Conneau et al., 2018) for cross-lingua language inference and PAWS-X (Hu et al., 2020) for cross- lingual paraphrase identification.
the foreign language test sets. We separately de-
scribe the results as follows:
Model
POS
NER
Pre-truining without parallel
corpus
Structural Prediction We choose UDPOS
mBERT (Devlin al., 2019)
70.3
62.2
XLM (Conneau and Lample. 2019)
70.1
61.2
(Nivre et al., 2018) for pos-tagging and Wikiann (Pan et al., 2017) for name entity recognition.
XLM RuASE (Conneau et al., 2020)
73.4
61.9
+TCS MCS-MLM
73.2
62.7
SCSS-LLM
62.6
Question Answering We choose MLQA (Lewis al., 2020) for cross-lingual question answering.
XLM-R (Conneau et al., 2020)
73.8
65.4
TCS-MLM
75.1
66.0
SCS-MLM
76.0
66.6
Cross-lingual Retrieval We choose Tatoeba (Artetxe and Schwenk, 2019) for parallel sentence identification.
Pre-training with parallel corpus
VECO (Luo al., 2021)
75.1
65.7
VECOLABGE 2.0 (Zhang et al., 2023)
75.4
67.2
HICTLLaacE (Wei et al.. 2021)
74.8
66.2
5.1.2 Experimental Results
Table Evaluation results on UDPOS and Wikiann cross-lingual structural prediction tasks. We report the average F1 scores, which are based on five runs with different random seeds.
Model
XNLI
PAWSX
Pre-training without parallel corpus
mBERT (Devlin al., 2019)
65.4
81.9
XLM (Conneau and Lample, 2019)
69.1
80.9
XLM-Ra.ss (our-impl)
74.9
84.4
TCS-MLM
75.2
84.9
Sentence-Pair Classification The cross-lingual
SCS-MLM
75.8
85.1
natural language inference (XNLI) aims deter- mine the relationship between the two input sen- tences, entailment, neural or contradiction. And PAWS-X aims to judge whether the two sentences are paraphrases not.
XLM-RL. (Chen et al,, 2022]
79.2
86.4
-TCS-MLM
80.8
88.2
+SCS-MLM
81.1
88.9
Pre-training with parallel corpus
XLM (Conneau and Lample, 2019)
75.1
Unicoder
75.4
As shown in Table 2, our SCS-MLMBASE sur- passes the baseline models including mBERT, XLM. XLM-RBASE and Unicoder. Morcover, our SCS MLS-MMMLAAR obtains equivalent performance with some pre-trained models using parallel sen- tences, including VECOLARGE and HICTI.LARGE- In contrast, although TCS-MLM also obtains im- provements, it is not as good as SCS-MLM. We suppose that the limited dictionaries would lead to insufficient cross- interactions.
VECOLARRR (Luo al., 2021)
79.9
88.7
VECOLARGE 2.0 (Zhang et al., 2023)
80.4
88.5
HICTLLABeE (Wei al., 2021)
81.0
87.5
ERNIE-M ML (Ouyang et al., 2021)
82.0
89.5
Table 2: Evaluation results on XNLI and PAWS-X cross- lingual text classification tasks. We report the average accuracy across different language test sets, which are based five runs with different random seeds.
We conduct experiments on the above cross- lingual NLU tasks evaluate the cross-lingual transfer capability of our method: fine-tune the model with the English training set and evaluate
Structural Prediction Structural prediction task contains UDPOS and Wikiann. Given a sentence, UDPOS aims to label the pos-tagging for tokens

---

Models
ar
en
65
vi
Avg
Pre-training without parallel corpus
mBERT
45.7
44.9/44.3
77.7
46.6
23.8/29.7
57.1 38.6
37.5/37.3
47.7/41.6
XLM
$4.8/36.3
62.2/47.6
74.9/62.4
68.0/49.8
48.8/27.3
614/418
61.1/996
616/43.5
XLM-RRASE
34.9736.6
60.97 46.7
77.17 64.6
69.47.9.6
39.4742.9
64.5744.7
61.8739.3
63.7746.3
TCS-MLM
37.6/38.5
63.2/48.7
80.1 66.9
68.1/50.2
62.7/
68.0/47.3
63.5/40.0
66.2/48.1
+ SCS-MLM
58.0/39.2
63.5/49.0
81.1/ 68.1
69.0/51.0
63.6/46.2
68.7
65.1/42.5
67.0/49.1
XLM-R
63.1743.5
63.37.3.6
80.67 67.8
56.17
69.2751.6
50.
68.0745.4
70.7752.7
+TCS--MLM
66.4 /46.8
84.3 75.1
74.5 56.4
71.7
53.8/
70.2 46.9
73.0/
-LC
66.6 46.99
70.3/55.3
84.4/71.6
/
71.5/53.9
73.2/53.
70.7 47.7
73.2/55.0
Pre Pre-training
paraltel corpus
VECOLARGE
65.0 /44.6
54.6
83.5 70.6
76.1
53.6
74.0/ 52.9
62.1 37.0
/3.2/
VECOLABUE 2.0
56.3
70.3/54)
84.1 71.4
66.5 /46.5
73.5
/3.2/
67.9 /3.7
72.7 54.3
HICTLLARGE
72.8
ERNIE-MLARGE
67.4/47.2
70.8 / 55.9
84.4 71.5
56.6
72.6 54.7
75.0/ 53.7
71.1/ 47.5
73.7 55.3
Table Evaluation results on MLQA cross-lingual question answering. We report the exact match (EM) scores. The results of TCS-MLM and SCS-MLM are averaged over five runs.
and Wikiann aims to identify name entities. We reported the average score for each dataset. Table 3 shows the results of our models. Com- pared with previous studies, our proposed SCS- MLMLARGE obtains the best results on UDPOS, achieving 76.0 F1 score. For Wikiann, TCS- MLMLARGE and SCS-MLMLARGE also obtain sig- nificant improvements compared with the strong baseline XLM-R. We suppose that the induced dic- tionaries contain the relations entities and pos- tagging across different languages, which promotes improvements in the structural prediction tasks.
525-MAM erpeccetC
Figure 6: Tatoeba results for languages, which are sorted according the performance of XLMM-R
find that our proposed SCS-MLM BASE obtains bet- ter retrieval accuracy scores (average acc. 60.73) than TCS-MI.M (+0.49 acc.) and XI.M-R (+7.46 46 acc.) across language directions, demon- strating the effectiveness of our method.
Cross-lingual Question Answering MLQA aims answer questions based on the given para- graph, which contains 7 languages.
The FI/EM scores are shown in Table We can find that our proposed TCS-MLM and SCS-MLM are significantly better than the strong baseline XLM-R and even surpass some models pre-trained with parallel sentences, such as VECO, VECO 2.0 and HICTL. Although our methods cannot sur- pass ERNIE-MIARGEE they narrow the gaps between mPLMs training with or without parallel sentences, demonstrating the effectiveness of our methods.
5.2 Natural Language Generation UNMT
As our proposed pre-training methods do not rely on parallel sentences, we choose the harder task unsupervised neural machine translation (UNMT) to evaluate the performance on the generation task.
5.2.1 Experimental Results
Table shows the translation performance on WMT14 En-Fr, WMT16 En-De, W En-Ro test sets. We can find that our proposed SCS-MLM can improve the translation quality compared with the strong baselines, XLM and MASS. For exam- ple, SCS- MCSM respectively outperforms XLM and MASS by 1.1 and 1.2 BLEU scores in W En-+Ro. SCS-MLM also surpasses previous stud- ies on average, verifying its effectiveness, More- over, the results also show that our method suit- able for the seq2seq model MASS (Figure
Cross-lingual Retrieval To evaluate the cross- lingual sentence retrieval capability of our models, we choose a subset of the Tatoeba dataset (36 lan- guage pairs), which aims to identify the parallel sentence among 1000 candidates. Following previ- ous studies, we used the averaged representation in the middle layer of different models (XLM-R RBSEE, TCS-MLMBBSE and (CS-MLMBASE) evalu- ate the retrieval task.
The results are shown in Figure We can

---

En
En
En Ro
Pre-trained Models
Avg.
En
Fr->En
En->De
Dn
En En-RRo
Ro->En
XLM
33.4
33.3
26.4
34.3
33.3
31.8
32.1
our re-impl.
37.3
34.7
27.1
33.9
34.5
32.7
33.4
Ren el al. (2019)
35.4
34.9
27.7
35.6
34.9
34.1
33.8
Ai and Fang (2022)
34.1
34.0
27.2
34.9
34.2
32.6
32.8
Ren al. (2021)
34.3
35.0
28.8
35.2
34.5
32.9
33.5
XLM + SCS-MLM (ours)
37.5
35.2
27.7
34.6
35.6
33.8
34.1
MASS
37.5
34.9
28.3
35.2
35.2
33.1
34.0
our re-impl.
37.1
34.7
27.4
34.9
34.9
33.2
33.7
MASS SCS-MASS (ours)
37.9
35.4
27.9
35.8
36.1
33.7
34.5
Table Unsupervised translation performance on WMT14 En-Fr, WMT16 En-De, WMT16 En-Ro. The results of previous studies are picked from corresponding papers.
appendix A.3), demonstrating that our method is independent of the model architectures.
feature space among languages using multiple monolingual corpora. XLM-R (Conneau al., 2020) shows the effects of models when trained on large-scale corpus, establishing strong base- lines for subsequent studies.
5.3 SCS-MLM Improves Geometric Similarity at Higher Layers
$2.3 shows that para-mPLMs would maintain the geometric similarity of language-specift token rep- resentations across layers. As our method incor- porate explicit cross-lingual interactions into pre- training, a similar phenomenon should occur. Therefore, plot the RSIM scores across 24- layer LARGE models for measurement. As shown in Figure (a), compared with the baseline model, our proposed SCS-MLM increases the geometric similarity of different languages across layers. Fig- ure (b) shows the RSIM improvements focus on the higher layers, thereby achieving balanced geometric similarities, akin to the observations of para-mPLMs in Figure 4 (b). It could illustrate the reason why method is effective various cross- Moos-llnggual transfer tasks.
Based on the observation that parallel corpora would help cross-lingual alignment in (Conneau and Lample, 2019), many studies pay attention the usage of the parallel corpus. Unicoder (Huang et al., 2019) employs multi-task learning frame- work to learn cross-lingual semantic representa- tions. ALM (Yang et al.. 2020) and PARADISE (Reid and Artetxe, 2022) uses parallel sentences to construct code-switch sentences. INFOXLM (Chi et al., 2021b) and HICTL (Wei al., 2021) respec- tively employ sentence-level and token-level con- trastive learning for cross-lingual semantic align- ments. VECO (Luo et al., 2021) proposes vari- able framework to enable the model to process understanding and generation tasks. ERNIF-M (Ouyang et al., 2021) generates pseudo-training ex- amples, further improving the overall performance on downstream tasks. (Ai and Fang, 2023) explores the MASK strategy (prototype-word) improve cross-lingual pre-training.
ESMM acreas
Besides the above discriminative models, gener- ative models, such as MASS (Song et al., 2019), mBART (Liu et al., 2020), mT5 (Xue et al., 2021), XGLM (Lin et al., 2022), BLOOM (Scao et al., 2022), also show impressive performance on gen- eration tasks. MASS, mT5 and mBART pre-train the denoising seq2seq model, handling both NLU and NLG tasks. XGLM and BLOOM pre-train extremely large auto-regressive models using ex- tremely large multiple monolingual corpora, show- ing surprising in-context learning (Brown et al.,
Figure (a) RSIM scores across different layers of SCS-MLM (solid lines) and XLM-R (dashed lines). (b) The curve of ARSIM across layers.
6 Related Work
Multilingual pre-trained language models begin with mBERT (Devlin et al., 2019) and XLM (Con- neau and Lample, 2019), which learn the shared

---

2020) capability on cross-lingual tasks. Different from previous studies, our work first investigates the properties of token alignments behind multilinguality and then proposes self- improving methods for multilingual pre-training with only monolingual corpora, alleviating the need for parallel sentences.
national Conference on Compitational Linguistics, Gyeongju, Republic of Korea. In- ternational Committee on Computational I Linguisties.
pages
Xi
Ai and Bin Fang. 2023. On-the-fly cross-lingual masking for multilingual pre-training. In Proceed- ings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 855-876, Toronto, Canada. Association for Computational Linguisties.
7 Conclusion
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 45 451-462, Vancouver, Canada. Association for Computational Linguistics.
In this work, we first investigate the properties of cross-lingual token alignments in mono-mPLMs, and then make comparison between mono- mPLMs and para-mPLMs, demonstrating that geo- metric similarities of higher-layer representations would be damaged without explicit cross-lingual in- teractions, hindering the multilinguality. Therefore, we propose token-level and semantic-level code- switch masked language modeling to improve the cross-lingual interactions without relying on par- allel corpora. Empirical results on language un- derstanding and generation tasks demonstrate the effectiveness of our methods. Future work would adapt our methods to much larger language models.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross lingual transfer and beyond. Transactions of the Association for Computational Linguistics. 7:597-610.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020, Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Acknowledgements
This work is supported by National Key R&D Pro- gram of China 2022ZD0160602 and the Natural Science Foundation of China 62122088.
Limitations
In this paper, we investigate the token-level cross- lingual aligmments but do not focus on the sentence- level alignments, which occur in the middle layer of mPLMs. We leave it as our future work. Then, we propose token-level and semantic-level code- switch masked language modeling for multilingual pre-training. As we analyzed in Appendix C, vari- ous types of mono-mPLMs are capable of forming alignments, but they are unable to preserve geo- metric similarities across layers. This phenomenon is observed irrespective of the language modeling methods employed. However, due to resource limi- tations, we only conduct experiments on XLM-R and MASS models. We will verify the effectiveness of our methods on much larger language models in the future, such as XGLM and BLOOM.
Steven Cao. Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. In International Conference on Learning Representa- tions.
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. Dict-mlm: Improved mul- tilingual pre-training using bilingual dictionaries. arXiv preprint 2tt::20000.22666
Wei-Te Chen, Yandi Xia, and Keiji Shinzato. 2022. Ex- treme multi-label classification with label masking for product attribute value extraction. In Proceedings of the Fifth Workshop on e- e-Commerce and NLP (EC- NLP 5), pages 134-140, Dublin, Ireland. Association for Computational Linguistics.
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang. Saksham Singhal, Xian-Ling Mao, Heyan Huang.
Xia Song, and Furu Wei. 2021a. mT6: Multilingual pretrained text-to-text transformer with translation pairs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 167 1671-, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
References
Xi Ai and Bin Fang. 2022. Vocabulary-infonn informedd lan- guage encoding. In Proceedings of the 29th Inter-

---

Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang. Xia Song. Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021b. InfoXI.M: An cooomation-theorticc framework for pross-linguul language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 3376-3588, On- line. Association for Computational Linguisties.
Unicoder: universal language encoder by pre- training with multiple cross- looss-lingual tasks. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNIP) pages 2485-2494, Hong Kong, China, Association for Computational Linguistics.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy, Association for Computational I Linguistics.
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian- Ling Mao, Heyan Huang, and Furu Wei. 2021c. Im- proving pretrained cross-lingual language models via self-labeled word alignment. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Vol- ume Long Papers), pages 3418-3430, Online. As- sociation for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2019, Cross-lingual ability of multilingual bert: An empirical study. CoRR, abs/19 1912.7740..
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Interna- tional Conference on Learning Representations.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MIQA: Evalu- ating cross- pooss-lnngual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7315- 7330, Online. Association for Computational Lin- guistics.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang. Luke Zettle- moyer, Zormitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Nat- ural Language Processing, pages 2775,, Brus- sels, Belgium. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng. Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2649- 2663, Online. Association for Computational Lin- guistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Fuli Luo, Wei Wang. Jiahao Liu, Yijia Liu, Bin Bi, Song- fang Huang, Fei Huang, and Luo Si. 2021. VECO: Variable and flexible cingsll pre-training for
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong. Linjun Shou, Daxin Jiang, and Ming Zhou. 2019.

---

language understanding and generation. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3980-3994, Online. Association for Computational Linguistics.
Uma Roy, Noah Constant, Rami AI-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Language-agnostic answer retrieval from multilin- gual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), P), pages 5919-5930, Online. Association for Computational Linguisties,
Joakim Nivre, Mitchell Abrams, Željko Agié, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependen- cies 2.2.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana 1lié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-aceess multilingual language model. arXiv preprint arXiv:2211 2200::2211100000.
Xuan Ouyang, Shuohuan Wang, Chao Pang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 27-38, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tin Liu. 2019. Mass: Masked sequence to sequence pre- training for language generation. In International Conference on Machine Learning, pages 5926-5936. PMLR.
Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020. Cross-lingual retrieval for iterative self-supervised training. Advances in Neural Information Processing Systems, 33:2207-22199
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng 2017. Tingsllingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1946,1958, Vancouver, Canada. As- sociation for Computational Linguistics.
Amos Tversky and Itamar Gati. 1982. Similarity, sep- arability, and the triangle inequality. Psychological review, 89(2):123.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Flo- rence, Italy, Association for Computational Linguis- tics.
Ivan Vulié, Sebastian Ruder, and Anders Søgaard. 2020. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192, Online, Association for Computa- tional I Linguistics.
Machel Reid and Mikel Artetxe. 2022. PARADISE::
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing. Heng Yu, and Weihua Luo. 2021. On learning univer- sal representations across languages. In International Conference on Learning Representations.
Exploiting parallel data for multilingual sequence- to-sequence pretraining. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 800-810, Seattle, United States. Association for Computational Lin- guistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas:
The surprising cross- pooss-lingual effectiveness of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNU pages 833- Hong Kong. China, Association for Computational Linguis- tics.
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Explicit cross-lingual pre-training for unsupervised machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 770-779, Hong Kong, China. Association for Computational Lin- guistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021, mT5: A massively multilingual pre- pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483 483-498, On- line. Association for Computational Linguistics.
Shuo Ren, Long Zhou, Shujie Liu, Furu Wei, Ming Zhou, and Shuai Ma. 2021. SemFace: Pre-training encoder and decoder with semantic interface for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- unn Long Pupers). pages 4518-4527, Online. As- sociation for Computational Linguistics.
Jian Yang. Shuming Ma. Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9386-9393.

---

Zhen-Ru Zhang, Chuanqi Tan, Songfang Huang, and Fei Huang. 2023. Veco 2.0: Lonsull language model pre-training with multi-gramularity contrastive learning. arXiv preprint arX.v22004.08205

---

A Experimental Settings
A.3
Experimental Settings of UNMT
We follow the common practices to conduct exper- iments on UNMT benchmarks. For evaluation, we separately adopt newsdev/test 2014 En-Fr, news- devitsst 2016 En-De, newsdev/test 2016 En-Ro development and test sets.
A.1 Pre-Training Data
We use the open-source CC-100 corpora' for the pre-training of BASE and LARGE models. Due to the resource and memory restrictions, we just se- lect 50 languages that cover the downstream tasks and conduct random sampling following(Luv et al., 2021). Table 6 shows the statistics of the monolin- gual data in each language.
For a fair comparison with previous studies, we pre-train the cross-lingual language models with the same model architecture of XLM and MASS. The pre-training data for UNMT is shown in Table 8. We compare SCS-MLM with other UNMT pre- training methods (Ren et al.. 2019, 2021; Song et al., 2019; Ai and Fang, 2022), which have the equivalent number parameters.
Code
Size (GB)
Code
Size (GB)
af
2.07
kk 8.14
ar 10.29
ko 15.36
bg 22.75
lt 10.08
During inference, we use the beam size 1 and length penalty 1.0. To be consistent with previous works, multi wulti-bleu.per to measure the translation quality.
bn 10.5
lv 6.38
19.87
ml 4.23
de 45.05
mr 3,45
el 27.96
ms 5.73
The illustration our method on MASS shown in Figure 8, which similar to Figure 5 but needs to predict multiple adjacent tokens using the sequence-fo-sequeene model.
en 191.85
my 0,52
es 40.35
ne 4.55
et 9.17
nl 13.8
cu 2.95
pl 15.46
fa 26.63
15.2
B RSIM Relational Similarity
fi
19.84
9.56
In NLP, the geometric similarity between two em- bedding spaces can be mcasured by Relational Sim- ilarity (RSIM). Given translation pairs, we first calculate pairwise cosine similarities among intra- language tokens and obtain two vectors a add b:
fr 44.16
46.46
fy 0.27
si 4.5
gd 0.2
Sw
2.56
gu 2.42
ta 7.26
7.01
te 5.66
hi 20.25
tr 14.05
= cos 21), cos((ro- .2), cos(ar, 1) b cos(vo: v1), cos(yo- 12), cos(Us,
hu 8.68
th 20.65
"os Ve-
id 44.12
tl 4.84
it 9.03
vi 42.15
Then, compute the Pearson's 's correlation be- tween add b, which is known as Relational Simi- larity (Vulié al., 2020).
ja 18.86
7.93
jv 0.15
yo 0.0006
5.45
40.14
Table Statistic of pre-training data in our experiments.
C Cross-Lingual Alignments on Different mPLMs
We also evaluate the accuracy of the token align- ments across different language models. The re- sults are shown Table 9, Some examples are included in Figure 11.
A.2 Hyperparameters for Pre-training
Table 7 shows the hyperparameters for pre-training different size models. SMALL models are initial- ized by XLM or MASS. And they are used to con- duct experiments on unsupervised machine transla- tion. BASE model and LARGE model are initialized by XLM-R model and used for various natural lan- guage understanding tasks.
We find that different kinds of language models form cross-lingual alignments. It demonstrates that pre-trained language models automatically learn cross-lingual mapping based on language model- ing regardless of the specific modeling methods (i.e. masked language modeling, text span prediction, or casual language modeling). For different kinds
/https:/ 100/

---

Hyperparameters
SMALL
BASE
LARGE
Toolkit
XLM/ MASS
fairseq
transformers
fairseq
transformers
Layers
6
12
24
Hidden size
1024
768
1024
FFN inner hidden size
4096
3072
4096
FFN dropout
0.1
0.1
0.1
Attention heads
8
12
16
Attention dropout
0.1
0.1
0.1
Embedding size
1024
1024
1024
Stopping criterion
validation patience=
training step=150k
training step=150k
Batch size Learning rate
512
2048
2048
2.00E-05
1.00E-04
1.00F-04
Learning rate schedule
Linear
Linear
Linear
Adam 31
0.9
0.98
0.98
Adam 8g
0.999
0.999
0,999
Weight decay
0
0.01
0.01
Warmup steps
4000
4000
4000
Table 7: Hyperparameters used for pre-training.
(ol Obtoin semanticeequvvveent represtttiooss (LVML MODL)
MASS longuage morilling with comrupted context (TRMI MODE)
Disr Disribusioo
CEL CAT 0.2.
0.6
.05
CAT
022
0.3
Wity
0,1
scarch
0:1
0.1
66
Decoder
MASS Decoder
MASS Encoder
MASS Encoder
Irank
Irasec
Imasii]
Imasit]
[nask]]
Figure 8: prediction.
The illustration of our method on MASS language modeling, which masks multiple adjacent tokens for
Data
Lan.
Sent.
Source
WMTT1 EnwRl
En 50.0M
255
En-De
(Song al., 2019)
De
50.0M
BLEU Scorees
En
179.9M
News Crawl 07-17
En-Fr/Ro
Fr
65.40
65.4M
News Crawl 07-17
2.8M
News Crawl 07-17 + WMT16
35.2
Table 8: Data statistics for unsupervised machine trans- lation training.
00.5
Num. of
Figure 9; Effect k on WMT16 En++Ro.
of pre-trained models, the alignment accuracy in- creases with the size of the parameters. Moreover, all the models show a similar pattern that distant language pairs have higher alignment accuracy.
in Figure 4. Therefore, we argue that explicit cross- lingual interactions still matter regardless of differ- ent architectures.
Furthermore, we also draw the RSIM scores across layers of differrntt mono-mPLMs in Figure 10. We find that different models share similar phenomenon that RSIM scores are higher at the lower layers but lower at higher layers. None of them could keep the geometric similarity balanced like para-mPLMss, VECO, or InfoXLM, as shown
D Ablation Study Effect of
In the proposed method, SCS-MLM, plays an important role. Considering that the pre-training of BASE and LARGE models are time-consuming. we pre-train SMALL models with different & and

---

Models
paramis
#ig
bg
id
ja
ko
ru
Avg.
XLM Ro..-RoTTTT
nOm oum.offpaitss
3322
2187
1223
2164
1159
2381
2501
1278
4077
XLM-R-BASE"
250M
100
44.12
58.61
48.33
32.58
49.03
48.92
47.00
62.38
63.77
59.43
52.73
XIM-R-LARGE
560M
46.71
58.79
48.61
52.25
48.96 94
50.13
47.08
63.97
62.52
59.97
53.09
Multiliguul-TS
num.of,pairs
817
2195
2289
878
1850
477
1563
2176
925
2631
MITS-SMALL
170M
T0o
42.96
59.27
33.56
48.63
55.08
41.09
49.14
55.72
61.41
62.35
32.63
ITT--BASE
390M
100
53.24
64.33
57.14
58.66
62.27
52.62
56.69
65.61
71.03
69.07
60.57
uTT5-LARGE
970M
100
54.96
65.51
58.41
59.34
63.57
51.36
57.90
67.63
73,41
69.49
61.75
3.2B
55.56
73.84
63.59
X-GLM
nom.of.pairs
1271
3332
3049
2212
3165
1451
2862
1312
1638
3636
XGLM-564M
560M
64.59
72.09
65.27
68.67
73.46
72.02
65.62
74.62
73.69
72.28
69.20
XGLM-1.7B
1.7B
30
64.44
71.94
64,55
69.44
73.24
71.95
65.34
76.14
73.69
73.65
69.32
XGLM-229B
2.9B
64.99
72.36
64.81
69.62
73.74
72.50
65.62
76.14
74.05
73.98
69.67
XGLM-7.5B
7.5B
64.83
72.63
65.40
70.71
74.66
73.26
66.46
75.69
73.93
73.79
70.05
Table 9: Prediction accuracy of translation pairs derived from different mPLMs across difterent languages English. Because HHI-RR/mS//XCGA supports different numbers of languages and has different vocabularies, comparisons between different types mPLMs may not be meaningful.
XLM R-BASE
MTTS-BAS
HTS NARGE
NTT ATAS-LLAGGE
En-Ar
3:23
Er-M
0.6
En-D2
tn-De
0.2
0.1
0.2
3.05
3.00
23
30
10.0
21.3
2.5
3.0
7-3
10.0
12.5
XGLM-364M
XGLM
XGLM 2.98
XGLM 7.59
0.7
b1:A1
0.7
E1-ar
0.6
En:D4
Et-O4
0.6
Ex
0.5
Bn
0.5
0.4
0.4
3.4
0.4
0.3
5.3
0.2
0.2
5
10
15
20
25
10
25
20
25
13
20
30
40
50
1.
30
Figure 10: RSIM scores across different layers of different mono-mPLMs.
evaluate the performance on UNMT task (WMT16 En+Roo) As shown in Figure 9, we can find that SCS-MLM obtains the best performance when is set as 8. Therefore, we set 8=8 for all the experi- ments in our paper.
These results are also provided Table 10-11.
Model
de
em
es
fr
ja
zh
avg.
Cross-lingual Transfer
XLM-Ra.s
86.4
93.9 88.8 88.8 76,8 76.4
79.7
84.4
TCS-MLM
87.9
94.4
89.4
89.2
76.3
76.8
80.5
84.9
SCS-MLM
87.9
94.5
89.0
89.0
77.5
77.0
81.0
85.1
E Experimental Results Details
XL R LAMHE
89.7
94.7
90.4
78.7
82.3
86.4
TCS-MLM
90.8
95.8
91.6
91.4
81.8
81.7
84.7
88.2
Due space limitations, we just report the average cross-lingual transfer metric scores in the main paper. The details for each language test set are listed in Table 10-14.
SCS-MLM
91.7 95.7 91.6
92.0
82.8
85.3
88.9
Translate-traine
trainn--tt
XLM-RLARGE
92.2
95.7
92.7
92.5
84.7
85.9
87.1
90.1
SCS-MLM
91.8 95.3 93.0
93,3
86.2
87.6
88.6
90.8
Translate-train-all is another evaluation method for multilingual pre-trained language models. It means fine-tuning multilingual model on the concatenation of all data (English training corpus and translated training corpus in other languages). Although our method mainly focuses on improv- ing cross-lingual transfer capability, can also bring improvements Translate-train-al settings.
Table 10: Cross Cross-lingual transfer results on PAWS-X
rons-lingual paraphrase identification for languages.

---

Langage Pairs
Exparted Tranzatiin Pairs
ug
erporeun strllep) (__ Hamsano, oompletelyy, ies childen), umooee atlizal, Sorrenemoo, gradaa)
Do En
Product Product) (Artii Antidel, (_rejed mepect, kilte ofhure) Koniekt Comeai
E-En
mettouu peolien, Lemoe completly). (gggen, rlaioestipl. Rp0ooins Abrioudyl. i_ arweltouuo outortte)
derelopeneet devdopment) ( pessen 5osemtl, fambia family) pouqquci whyl, sysim 9yttem)
W: En
[_ v.ाt Srutegt) निाह sintto)) 3A0 repurat)) Reatffic, klometrr (_ ftatt mocand)
Ja En
(2 sportal [2 Hhegg) (RE Investmentt (3 businesal,
Ko-Ea
L ectrenel)) L -pogrennll (_ soonotyl.
alwaysi, L
MnEn
Londloogs wnullly winlly) (.idm, pmblem) (_ mod bare) Groym, voyage)
Agggom Jamily)
Pn
milhies _illiian), (_ Jlpin. someonsi, deserdhimmedo development, [_ prahiammntee,
bainaly..
temica
techoque)
Ro
9061, Voppemnnn _aakkeingg. Leeepoun personal)
S--n
Lodan mport) Rd8, noviell, L
-ompteell (_ (opooon, knows), (_ mitrmnel)
To
(106611)) _ Importan, (__ www.uuuuaaa...mm uriltte) [_iiieeeeet probkem)) L_ LiDanuu xinoo, [_ m
T=
oompltttyy, (o_phtt L88 LIPE), Rdide, _iloass) 1_ 2000 various)
(Brem, caltura)) (144 Lagpeeede (eris, twchniquel (miise, Lechologgeee [
Vi-Bi
Lmbn devellog) (aieeaap...omm 1o0mmm_me. mnly). L-gining..... bell, thbch chllleggei _ recommen))
Zh-Bn
(r.. wols) (81. Smit (X
completely 4 improvl. kotel
Figure 11: Some examples that are randomly selected from the exported dictionaries.
Model
ar
bg
de
el
C5
hi
sw
th
ur
Vi
zh
avg
Cross- Longs-Lingua Transfer
XLM-Raast
73.1
78.4
77.1
76.3
85.1
79.4
78.4
70.9
76.5
64.5
73.7
73.1
67.2
75.4
74.7
74.9
TCS-MLM
72.7
78.2
77.3
77.0
84.1
80.3
78.8
71.4
76.4
66.9
73.0
74.2
67.6
75.4
74.4
75.2
SCS MLM
73.7
79.2
78.3
77.1
84.3
79.7
78.4
72.2
77.0
67.5
73.9
75.0
68.1
76.4
75.5
75.8
77.2
83.0
82.5
80.8
88.7
83.7
82.2
75.6
79.1
71.2
77.4
78.0
71.7
79.3
78.2
79.2
+TCS
77.4
78.3
+SCS-MLM
79,9
83.5
82.9
88.8
84.8
77.7
81.3
74.2
79.1
80.6
74.4
80.7
80.2
81.1
Translate-train-ett
XLM-RLaCE +SCS-MLM
82.4
85.3
84,8
85.0
88.9
86.2
84.7
80.2
82.4
77.3
80.9
82.7
77.2
82.8
83.0
82.9
82.9
85.2
84.9
86.0
80.8
83.0
77.8
80.9
77.8
83.0
82.6
83.2
Table 11: Cross-lingual transfer results on XNLI cross-lingual natural language inference for 15 languages.
af
bg
de
fa
fr
he
hi
bu
id
XI.M-Rpas
69.7
86.7
67.9
73.2
72.8
MLM-
88.3
66.8
88.8
88.6
85.8
95.9
88.3
85.8
68.3
69.6
85.3
86.6
67.1
71.9
82,9
72.5
88.9
SCS-MLM
89.0
65.7
88.8
89.6
87.5
95.7
87.4
85.
71.6
69.1
85.0
86.3
66.3
67.5
82.4
72.4
88.6
XLM-RLace
89.8
67.5
88.1
88.5
86.3
96.1
88.3
86.5
72.5
70.6
85.8
87.2
68.3
76.4
82.6
72.4
89.4
TCS-MMM
89.4
67.2
87.0
88.8
96.1
88.2
86.3
76.9
71.2
86.4
86.3
69.0
72.7
83.1
73.0
90.1
MLM
71.0
87.0
68.2
74.3
72.7
ja
kk
ko
mn
mi
tn
te
th
tr
ur
vi
yo
>h
uvg.
XLM-Rassa
27.9
76.3
53.3
82.6
89.3
88.0
89.3
62.7
84.8
45.2
89.9
74.5
64.1
57.9
24.3
33.4
73.6
TCS-MLM
23.1
76.9
52.6
81.7
89.3
88.1
89.3
61.8
85.6
43.0
91.9
74.2
62.1
$7.5
25.9
29.9
73.2
SCS.MI.M
77.1
49.9
91.2
73.9
58.9
XI.MRRAACE
15.9
78.1
53.9
80.8
89.5
87.6
89.5
65.2
86.6
47.2
92.2
76.3
70.3
56.8
24.6
25.7
73.8
TCS MLM
33.5
79.0
52.9
85.7
89.3
87.2
89.6
62.8
85.0
46.7
93.5
76.3
67.8
60.0
33.9
35.7
75.1
-CCS-MLM
35.2
78.4
53.2
83.9
89.6
87.6
89.9
62.8
86.3
54.4
93.9
75.4
68.8
58.0
39.5
54.9
76.0
Table 12: Cross-lingual transfer results on UDPOS (POS) cross-lingual pos-tagging for 33 languages.

---

Model
65
XLM-ReASE
76.2
55.5
78.6
71.8
75.0
75.8
81.8
69.2
71.7
58.7
54.5
76.2
76.2
53.0
69.3
77.7
48.5
78.6
17.7
58.7
TCS-MLM
78.1
78.8
73.2
75.5
73.8
81.9
73.4
72.2
58.1
54.4
76.3
77.5
52.8
69.4
77.5
50.7
78.0
18.8
59.5
SCS-MLM
77.8
49.9
79.9
70.0
75.1
76.0
81.5
76.7
72.5
61.0
50.1
76.5
77.6
55.2
69.3
77.6
48.9
78.1
21.3
63.7
XLM-RLace
78.9
53.0
81.4
78.8
78.8
79.5
84.7
79.6
79.1
60.9
61.9
79.2
80.5
56.8
73.0
79.8
33.0
81.3
23.2
62.5
-TCS-MLM
79.4
47.8
K2.0
75.3
79.9
$1.0
83.4
72.1
81.6
72.3
55.7
81.4
81.3
55.0
72.7
81.8
65.3
82.4
19.3
64.3
SCS-MLM
78.8
55.1
83.0
76.8
79.9
79.3
83.7
77.0
80.8
68.0
63.7
81.1
79.7
55.3
72.4
81.2
56.4
81.8
17.6
64.9
kn
kk
ml
115
my
m1
th
11
tr
XLM ROASE
67.5
43.4
52.0
61.8
62.3
59.9
51.2
80.1
78.7
64.9
69.3
55,1
49.6
0.0
71.6
75.2
66.0
67.7
49.5
25.8
-CS-MLM
68.3
43.3
54.7
63.1
63.1
58.0
54.5
81.0
77.7
64..8
68.2
58.5
53.3
0.0
72.4
77.7
68.5
68.9
50.0
26.4
SCS MLM
69.4
40.8
65.0
63.1
53.4
56.1
80.5
78.9
64.6
69.4
59.4
55.1
0.0
70.8
79.4
67.7
68.6
43.1
28.5
XLM-RLAKEE
73.2
-TU-MLLM
Table 13: Cross-lingual transfer results on Wikiann (NER) cross-lingual name entity recognition for 40 languages.
ar
bg
de
85
et
28
6
he
hi
id
ja
XI.M-Rgus
52,0
35.9
67.1
29.3
88.9
53.7
70.4
43.7
24.3
64.8
64.0
73.2
50.8
50.4
58.5
68.8
65.2
53.8
MLM
62.6
42.6
73.7
36.4
92.3
60.1
78.6
57.6
40.4
71.8
72.6
79.9
56.2
65.9
64.7
76.7
69.3
50.4
-CSCMMMM
42.7
73.3
92.0
59.4
70.3
73.5
57.1
62.8
77.3
70.0
jv
ka
kk
ml
pt
20
SW
18
11
ur
vi
XLM-Rgus
15.1
36.2
32.7
49.3
54.6
37.4
77.2
77.6
70.2
15.9
24.1
28.6
32.3
29.0
55.2
27.7
67.7
59.7
TCS MLM
17.1
42.6
44.9
54.6
56.3
45.3
83,0
82.9
76.1
23.6
26.7
30.8
42.9
37.5
61.1
41.1
73.6
60.3
+SCS-MLM
45.2
57.6
27.0
37.8
74.6
Table 14: Cross-lingual transfer results on Tatoeba cross-lingual sentence retrieval for 34 languages.


Reviews:
[{"review_id": "kOdmnlXk0X", "reviewer": "Reviewer_Jsy5", "summary": "", "questions": "1. Would other MPLMs likewise benefit from the approach you propose? It would make the paper more sound if experiments could be conducted in multiple mPLMs.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging comments!\nWe hope that our study could inspire further exploration into the reasons and mechanisms behind multilinguality and methods to enhance multilingual pre-training models.\n\n**About the Effectiveness of Our Method on Other mPLMs**\n\nOur study demonstrates that our method would benefit XLM-R (Transformer Encoder) and MASS (Transformer Encoder-Decoder) models. Future work would adapt our method to X-GLM to investigate the effectiveness on the Transformer Decoder based mPLMs.\n\n\n\nWe would express our gratitude again for your acknowledgment of our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and well-presented.\n\n2. This paper attempts to improve MPLM using monolingual corpora only and significantly improve the performance compared with the strong baselines.\n\n3. Further analysis demonstrates that the methods improve the geometric similarity of representations for different languages, and thus promoting the cross-lingual transfer capability.\n\n4. The designed empirical study also shows the strong motivation of this paper.", "reasons_to_reject": "Good paper, I have no reason to reject.", "questions_to_authors": "1. Would other MPLMs likewise benefit from the approach you propose? It would make the paper more sound if experiments could be conducted in multiple mPLMs.", "paper_topic_and_main_contributions": "1. This paper investigates the alignment properties from the token perspective in mono-mPLMs and find that the alignments correspond to the geometric similarity of embedding space across different languages.\n\n2. Then this paper introduces token-level and semantic-level code-switched masked language modeling, employing the self-induced token alignments to explicitly improve cross-lingual interactions over layers of mono-mPLMs without relying on parallel sentences.\n\n3. They evaluate the method on various natural language understanding tasks and unsupervised machine translation tasks. The results demonstrate that the methods outperform the strong baselines and achieve comparable performance with mPLMs trained with parallel corpora.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LJrZhwN4Ro", "reviewer": "Reviewer_Swjy", "summary": "", "questions": "Please refer to Reasons To Reject.\n\n1. Some details are missing. How many tokens do you consider for top-k (eq. 4)? \n\n2. In line 289 (\"Therefore, we replace 10∼15% tokens using the dictionary to construct multilingual code-switched\"), how do you decide 10%, 15%, or others?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for your insightful comments, which will help us improve our paper.\n\n**About the Experimental Details**\n\n* Our model is initialized by pre-trained XLM-R and performs second-stage pre-training, akin to prior works such as VECO[1], HICTL[2], InfoXLM[3] and ERNIE-M[4]. Therefore, the replacements are meaningful during training.\n* We use Top-8 tokens for the weighted aggregation. As we analyzed in Appendix C (Figure 9 is placed in the appendix due to the space limits), we find that setting *k* as 8 would obtain the best performance on the downstream task (UNMT). We will clarify this aspect in the main pages of the revised paper.\n* Regarding the replacement ratio.\n\n  * We take a specific example to elaborate. Suppose we have the dictionary *D* with *n* token translation pairs for language En-X and English has *m* tokens. Then, we define the cover rate as *q=n/m* and the replacement rate is *p=0.15/q*. For token *t* in the current batch, if *t* exists in the dictionary *D*, we use a random function to sample a number 0<=*s*<=1. When *s < p*, we replace token *t* with its translations. However, due to the dynamic changes of tokens in different batches, the actual replacement rates may be slightly varied. Thus, we state that \"we replace 10%-15% tokens ...\" in our paper. \n  * Furthermore, some of our preliminary experiments have shown that: when the replacement ratio is too high, the model struggles to make accurate predictions due to the heightened levels of language ambiguity. On the contrary, when the replacement ratio is low, the potential improvements can be restricted. A range of 10-15% is a favorable selection.\n* Finally, we will open-source our code, experimental details, pre-trained language models and the self-induced dictionaries.\n\n**About the Attention Weights**\n\n* We use the following example to illustrate the effectiveness of cross-lingual token substitutions. We use the translation pairs (plan - योजना), (go - mennä), (pandas - 熊猫) to replace the following English sentence and predict the *&lt;mask&gt;* (*_zoo*) token:\n\n  *We plan to go to the zoo to see pandas*         &rArr;       *We योजना to mennä to the &lt;mask&gt; to see 熊猫*\n\n  We extract the attention weights from *&lt;mask&gt;* to other tokens in the last layer of XLM-R and our model. We find that:\n\n  * &lt;s&gt; and &lt;/s&gt; have the highest attention weights than other tokens.\n  * The golden token \"▁zoo\" can be predicted with higher probability by SCS-MLM (0.2862) than XLM-R (0.1046).\n  * The contrast in attention weights for specific tokens (*see*, 熊, 猫) at the *<mask>* position underscores our model's ability to focus more intently on semantically equivalent yet linguistically different tokens compared to vanilla XLM-R. The positive values of $\\Delta$ = Attn(SCS-MLM) - Attn(XLM-R) demonstrate the effectiveness of our method.\n\n  We agree with you that for basic stop words, the assistance of cross-lingual context tokens might be unnecessary.\n\n\n\n**Table 1. Attention weights from *&lt;mask&gt;* to other tokens of XLM-R and SCS-MLM. We also label the gold output \"_zoo\" and the prediction probability.**\n\n|                | &lt;s&gt; | ▁We   | ▁योजना | ▁to   | ▁mennä | ▁to    | ▁the   | &lt;mask&gt;    | ▁to    | ▁see      | ▁         | *熊*        | *猫*        | &lt;/s&gt; |\n| -------------- | :-------: | ----- | ------ | ----- | ------ | ------ | ------ | --------------- | ------ | --------- | --------- | --------- | --------- | ---------- |\n| Attn(XLM-R)          |   0.330   | 0.014 | 0.000  | 0.009 | 0.052  | 0.062  | 0.079  | 0.004           | 0.029  | 0.035     | 0.021     | 0.015     | 0.018     | 0.333      |\n| - *prediction* |           |       |        |       |        |        |        | ▁zoo - p=0.1046 |        |           |           |           |           |            |\n| Attn(SCS-MLM)        |   0.285   | 0.014 | 0.000  | 0.009 | 0.035  | 0.059  | 0.070  | 0.003           | 0.026  | 0.082 | 0.036 | **0.029** | **0.043** | 0.309      |\n| - *prediction* |           |       |        |       |        |        |        | ▁zoo - p=0.2862 |        |           |           |           |           |            |\n| $\\Delta$ = Attn(SCS-MLM) - Attn(XLM-R)      |  -0.045   | 0.000 | 0.000  | 0.000 | -0.017 | -0.003 | -0.009 | -0.001          | -0.003 | 0.047 | 0.015 | **0.014** | **0.025** | -0.024     |\n\n**About the Missing Reference**\n\n* Following your suggestion, we read this paper [5]. It is an impressive work that explores the *MASK* strategy to improve cross-lingual pre-training, which uses average weighted embedding as the \"prototype-word\" to replace *<mask>*, enhancing the cross-lingual knowledge for prediction. Different from [5], our study requests the model to perform correct prediction with semantically equivalent but code-switched contexts, improving the context information usage during prediction. Regrettably, the timeline of EMNLP 2023 submission predates the release of ACL 2023 publications, hence the absence of discussion on this work in our paper. We would like to include this work and discuss the difference between our method and theirs in our revised version.\n\n\n\nThanks again for your thorough review! We sincerely hope that our response addresses your questions and concerns.\n\n\n\n**References**\n\n[1] Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Song fang Huang, Fei Huang, and Luo Si. 2021. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proc. of ACL 2021.\n\n[2] Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning univer827 sal representations across languages. In Proc. of ICLR 2021.\n\n[3] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proc. of NAACL 2021.\n\n[4] Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. In Proc. of EMNLP 2021.\n\n[5] Xi Ai and Bin Fang. 2023. On-the-fly Cross-lingual Masking for Multilingual Pre-training. In Proc. of ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The presentation is good.\n2. The authors present sufficient and clear empirical studies to justify the motivation.\n3. Experiments including 5 tasks support the presented results.", "reasons_to_reject": "1. My major concern is about the details of experimental setups. Concretely, the authors use mono-mPLM to infer substations. However, mono-mPLM is randomly initialized, which means that mono-mPLM only makes random substitutions in early training. Do the authors use pre-trained mono-mPLM? Is the presented method a post-training method? Do random substitutions in early training not impact the final performance? How many tokens do you use for weighted aggregation? \n\n2.  The authors might justify the effectiveness by showing attention weights on the cross-lingual word substitutions. The model might ignore the cross-lingual word substitutions if the context is sufficient for prediction. For instance, the model just predicts some simple stop words. \n\n\nI am very happy to discuss these issues.", "questions_to_authors": "Please refer to Reasons To Reject.\n\n1. Some details are missing. How many tokens do you consider for top-k (eq. 4)? \n\n2. In line 289 (\"Therefore, we replace 10∼15% tokens using the dictionary to construct multilingual code-switched\"), how do you decide 10%, 15%, or others?", "paper_topic_and_main_contributions": "This paper compares parallel-corpora-based multilingual training (para-mPLM) with monolingual-corpora-based multilingual training (mono-mPLM) by analyzing the embedding space and the top-layer space. The author found the geometric similarity of representations across different languages contributes to multilingualism but it is not observed in the top-layer space of mono-mPLM. The authors further argue that cross-lingual interactions at the bottom layer can help the model preserve the geometric similarity for the top-layer space.\n\nTo this end, the authors present a data augmentation method based on cross-lingual word substitutions. To find suitable cross-lingual word substitutions, the authors use mono-mPLM to predict the distribution of a word and search for corresponding translations in the embedding space. Then, a weighted representation of all the translations is used to replace the original word.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There is a parallel work from ACL 2023, though not a reason to reject.  The parallel paper discusses a very similar idea (self-induce candidates and weighted aggregation) but focuses on masking strategies instead of data augmentation (this paper).\n\nXi Ai and Bin Fang. 2023. On-the-fly Cross-lingual Masking for Multilingual Pre-training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 855–876, Toronto, Canada. Association for Computational Linguistics."}}, {"review_id": "W4C57QcYiH", "reviewer": "Reviewer_R5qn", "summary": "", "questions": "A.Line 225 and Figure 3(a), average cos between translation pairs. For computing the sentence embeddings, is the method averaging token embeddings of the input seqence? Why are there different results for token-level and sentence-level semantic alignment? \n \nB.Line 310-311 and Figure 5, how does the output distribution computed? For the input, the token ‘cat’ is provided instead of masked. Does it make sense?\n\nC.Line 352, why is $\\alpha$ set as 0.7, which is different from XLM-R?\n\nD.Is it necessary to use self-induced lexicons instead of gold lexicons? For high-resource languages, the gold lexicons are usually easy to obtain. For low-resource languages, the self-induced lexicons may be in low-quality, leading to worse results on downstream tasks.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful insights and comments on our work.\n\n**About the Difference with Previous Studies**\n\nWe would like to clarify the distinctions between our work and the notable contributions of [1] and [2] in multilingual pre-training. The main difference is utilizing labeled bilingual resources to improve multilinguality [1-2] or exploiting the self-improving approach without reliance on external parallel data.\n\n* *Regarding Scenarios and Motivation*:\n  * [1] and [2] focus on enhancing cross-lingual capabilities by utilizing human-labeled parallel resources, such as bilingual sentences with token alignments[1] or gold dictionaries[2]. They are suitable for scenarios when parallel resources are available. \n  * By contrast, our work unveils the presence of cross-lingual token alignments within mPLMs pre-trained exclusively on monolingual corpora. Moreover, we further demonstrate the self-improving capability of mono-mPLMs through the incorporation of self-induced token alignments.\n* *Regarding Methodology:*\n  * [1] uses L2 loss to draw token representations closer with the help of labeled word alignments, and [2] leverages gold dictionaries from MUSE and Flores to conduct monolingual and parallel denoising tasks, requiring the model to recover the original tokens with substituted translations.\n  * In contrast, our approach utilizes self-induced token alignments to construct context, forcing the model to predict *&lt;mask&gt;* tokens with the semantic-equivalent but code-switched surrounding tokens and thus improving the multilinguality.\n  * Moreover, [1] primarily focuses on the token alignments within a specific parallel sentence pair, while our analysis pertains to the comparison of two language-specific vocabularies.\n\n**About the Questions**\n\n* Q.A - About Average Cosine of Translation Pairs.\n  * We apologize for the misleading regarding \"translation pairs\". In this paper, the \"translation pairs\" are token-level instead of sentence-level. For example, \"happy - سعيد\" and \"interesting - दिलचस्प\". The average cosine means the average scores of 500 token translation pairs for a particular language direction (dash lines in Figure 3(a)). We can find that the average cosine scores become larger, demonstrating that distinct language token representations aggregate at higher layers. By contrast, the token alignments become weaker (solid lines in Figure 3(a)).\n  * The sentence-level alignments (using average pooling of token representations as the sentence representation) are regarded to appear in the middle layer of mPLMs [3-4]. We think that lower layers focus on the token-level features while middle layers focus on the syntactic and sentence-level features [5-6], which result in the difference between token-level and sentence-level alignments.\n\n* Q.B - About the Computation of Distribution.\n  * The distribution $d$ is calculated as $d = hW$, where $h$ denotes hidden states and $W$ is the output projection weights, which are usually tied with the embedding weights.\n  * We believe that replacing context tokens instead of masked positions is meaningful. Masked language modeling (MLM) supposes that the masked tokens can be predicted by the understanding of context. When the context tokens are replaced by the semantic-equivalent tokens in other languages, the model should pay attention to the replaced tokens and perform the correct prediction.\n\n    For the sentence \"*We योजना to mennä to the &lt;mask&gt; to see 熊猫* \", we record the attention weights from *&lt;mask&gt;* to other tokens of XLM-R and our SCS-MLM in Table 1. We find that SCS-MLM pays more attention to \"*see*\", \"熊\" and \"猫\", achieving a higher prediction probability of the gold token \"_zoo\" (0.2862) than XLM-R (0.1046).\n  * Table 2 also shows that SCS-MLM validation loss also gradually decreases during training, corroborating the effectiveness of our approach.\n\n\n\n  **Table 1. Attention weights from *&lt;mask&gt;* to other tokens of XLM-R and SCS-MLM. We also label the gold output \"_zoo\" and the prediction probability.**\n\n|                | &lt;s&gt; | ▁We   | ▁योजना | ▁to   | ▁mennä | ▁to    | ▁the   | &lt;mask&gt;    | ▁to    | ▁see      | ▁         | *熊*        | *猫*        | &lt;/s&gt; |\n| -------------- | :-------: | ----- | ------ | ----- | ------ | ------ | ------ | --------------- | ------ | --------- | --------- | --------- | --------- | ---------- |\n| Attn(XLM-R)          |   0.330   | 0.014 | 0.000  | 0.009 | 0.052  | 0.062  | 0.079  | 0.004           | 0.029  | 0.035     | 0.021     | 0.015     | 0.018     | 0.333      |\n| - *Prediction* |           |       |        |       |        |        |        | ▁zoo - p=0.1046 |        |           |           |           |           |            |\n| Attn(SCS-MLM)        |   0.285   | 0.014 | 0.000  | 0.009 | 0.035  | 0.059  | 0.070  | 0.003           | 0.026  | 0.082 | 0.036 | **0.029** | **0.043** | 0.309      |\n| - *Prediction* |           |       |        |       |        |        |        | ▁zoo - p=0.2862 |        |           |           |           |           |            |\n| $\\Delta$ = Attn(SCS-MLM) - Attn(XLM-R)      |  -0.045   | 0.000 | 0.000  | 0.000 | -0.017 | -0.003 | -0.009 | -0.001          | -0.003 | 0.047 | 0.015 | **0.014** | **0.025** | -0.024     |\n\n\n\n**Table 2. SCS-MLM loss gradually decreases with the increase of training steps.**\n\n| Training Step           | 1k   | 5k   | 10k  | 20k  | 50k  | 100k | 150k |\n| ----------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| SCS-MLM Validation Loss | 2.35 | 2.30 | 2.25 | 2.21 | 2.17 | 2.14 | 2.12 |\n\n\n\n\n* Q.C - About the Sampling Rate.\n\n  * The sampling rate relies on the data sizes of different languages. The sizes of our pre-training corpora are different from XLM-R, leading to different $\\alpha$ settings like previous studies (XLM-R[7]: $\\alpha=0.3$, VECO[8]: $\\alpha=0.5$, InfoXLM[4]: $\\alpha=0.7$). For our data, we find that setting $\\alpha=0.7$ would lead to balanced sampling probabilities for different languages.\n\n* Q.D - The Necessity of Self-Induced Dictionary.\n\n\n  * First, our work introduces valuable insights regarding the potential of exporting cross-lingual dictionaries from mono-mPLMs (multilingual language models pre-trained with multiple monolingual corpora). The proposed method further demonstrates that self-induced dictionaries can enhance the performance of mPLMs like conventional gold dictionaries.\n\n  * Second, we classify the availability of cross-lingual dictionaries based on the resources involved, as shown in Table 3.\n\n    * **CLASS-A:** This category usually has high-resource monolingual corpora and parallel corpora, like widely-used English-centric pairs such as En-Fr and En-Zh. Gold dictionaries for these pairs are readily accessible either as open-source resources ([MUSE](https://github.com/facebookresearch/MUSE), [wikidict](https://github.com/open-dict-data/wikidict-en)) or through extraction from parallel sentences. Our work demonstrates that mPLMs also support producing dictionaries for these language directions with good quality.\n    * **CLASS-B:**  Involving high-resource monolingual corpora but limited parallel corpora, this class encapsulates most language pairs worldwide. In this scenario, acquiring gold dictionaries becomes a challenge. For instance, language pairs like Greek-Hindi (El-Hi) and Tamil-Japanese (Ta-Ja) boast abundant monolingual sentences but lack open-source gold dictionaries. In contrast, our work underlines that mono-mPLMs can effectively yield reliable token translations (self-induced dictionaries), emerging as a viable alternative under these circumstances.\n      We randomly choose some exported token translation pairs of El-Hi and Ta-Ja, which are presented in Table 4-5.\n    * **CLASS-C:** This category involves both low-resource monolingual corpora and parallel corpora, making the acquisition of gold dictionaries, as well as the generation of dictionaries from mPLMs challenging.  Finding effective solutions for these languages necessitates other methodologies and resources. Nonetheless, this class pertains to just a minute fraction of languages. To illustrate, among the 50 languages that we processed, only yo, gd, and fy have this problem.\n\n    In practical scenarios, the availability of gold dictionaries dictates the direct use of such resources. However, in their absence, leveraging self-induced dictionaries from mPLMs emerges as a pragmatic strategy to augment the cross-lingual capabilities of models.\n\n    \n\n  **Table 3. The availability of cross-lingual dictionaries for different resource scenarios.**\n\n  |  Type   |  Monolingual  |   Parallel    | Gold Dictionary | Self-induced Dictionary |\n  | :-----: | :-----------: | :-----------: | :-------------: | :---------------------: |\n  | CLASS-A | High-resource | High-resource |     &check;     |         &check;         |\n  | CLASS-B | High-resource | Low-resource  |     &cross;     |         &check;         |\n  | CLASS-C | Low-resource  | Low-resource  |     &cross;     |         &cross;         |\n\n  **Table 4. Some randomly selected Ja-Ta token translations derived from our XLM-R.**\n\n  | Japanese Token | Tamil Token | English Meaning |\n  | -------------- | ----------- | ---------------- |\n  | それを         | ▁அதை        | it               |\n  | 銀行           | ▁வங்கி       | bank             |\n  | 生まれた       | ▁பிறந்த      | born             |\n  | なぜ           | ▁ஏன்         | why              |\n  | 重要な         | ▁முக்கியமான  | important        |\n  | ということを   | ▁என்பதை      | that             |\n  | 理由は         | ▁காரணம்      | the reason       |\n  | 市場           | ▁சந்தை       | market           |\n  | 今後の         | ▁எதிர்கால    | future           |\n  | 統治           | ▁ஆட்சி       | reign / rule     |\n  | 様々な         | ▁பல்வேறு     | various          |\n  | 最終           | ▁இறுதி      | final            |\n\n  **Table 5. Some randomly selected El-Hi token translations derived from XLM-R.**\n\n  | Greek Token | Hindi Token | English Meaning |\n  | ----------- | ----------- | ---------------- |\n  | ▁ξαφνικά    | ▁अचानक      | suddenly         |\n  | ▁παράδειγμα | ▁उदाहरण     | example          |\n  | ▁φύση       | ▁प्रकृति      | nature           |\n  | ▁ποιότητας  | ▁गुणवत्ता     | quality          |\n  | ▁τοπική     | ▁स्थानीय     | local            |\n  | ▁επίσημη    | ▁आधिकारिक   | official         |\n  | ▁ενδιαφέρον | ▁दिलचस्प     | interesting      |\n  | ▁τύπου      | ▁टाइप       | type             |\n  | ▁βίντεο     | ▁वीडियो     | video            |\n  | ▁Προσωπικ   | ▁व्यक्तिगत    | personal         |\n  | ▁τραγούδι   | ▁गाने        | songs            |\n  | ▁δεκαετίες  | ▁दशक        | decade           |\n\n\n\n  **About the Missing References**\n\n  We will include the missing references in the revised paper.\n\n\n\n  Thank you once more for your comments. We think the discussion about these issues would improve our paper.\n\n\n\n\n  **References**\n\n  [1] Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual Alignment of Contextual Word Representations. In Proc. of ICLR 2020.\n\n  [2] Machel Reid, Mikel Artetxe. 2022. PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining. In Proc. of NAACL 2022.\n\n  [3] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi task benchmark for evaluating cross-lingual generalisation. In Proc. of ICML 2020.\n\n  [4] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proc. of NAACL 2021.\n\n  [5] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language? In Proc. of ACL 2019.\n\n  [6] Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proc. of EMNLP 2019.\n\n  [7] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Prof .of ACL 2020.\n\n  [8] Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Song fang Huang, Fei Huang, and Luo Si. 2021. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proc. of ACL-IJCNLP 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes new training objectives to improve mono-mPLMs using self-induced token alignments instead of external parallel resources.\n- The experimental results show that it can outperform mono-mPLMs and achieve comparable results with para-mPLMs", "reasons_to_reject": "The contributions are incremental. Specifically,\n1. Token alignment analysis (Cao et al. Multilingual Alignment of Contextual Word Representations. ICLR 2020.)\n2. Token-level code-switched masked language modeling (Reid and Artetxe. PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining. NAACL 2022)", "questions_to_authors": "A.Line 225 and Figure 3(a), average cos between translation pairs. For computing the sentence embeddings, is the method averaging token embeddings of the input seqence? Why are there different results for token-level and sentence-level semantic alignment? \n \nB.Line 310-311 and Figure 5, how does the output distribution computed? For the input, the token ‘cat’ is provided instead of masked. Does it make sense?\n\nC.Line 352, why is $\\alpha$ set as 0.7, which is different from XLM-R?\n\nD.Is it necessary to use self-induced lexicons instead of gold lexicons? For high-resource languages, the gold lexicons are usually easy to obtain. For low-resource languages, the self-induced lexicons may be in low-quality, leading to worse results on downstream tasks.", "paper_topic_and_main_contributions": "Current multilingual pretrained language models (mPLMs) are typically pretrained with monolingual corpora for multiple languages, i.e., mono-mPLMs, and may be also further improved using parallel resources, i.e., para-mPLMs. Due to the limited available parallel resource, this paper aims to improve mPLMs without parallel resources.\n\nThe paper firstly conduct analyses to investigate the token properties of a mPLM, i.e., XLM-R, and the results show that cross-lingual token alignments (geometrically aligned instead of absolutely aligned) occur at the embedding layer but become weaker at the higher layers.\n\nBased on these observations, this paper proposes an training objective of token-level and semantic-level code-switched masked language modeling, which employs the self-induced token alignments to explicitly improve cross-lingual interactions over layers of mono-mPLMs without relying on parallel sentences.\n\nThe experimental results show that the proposed method outperforms mono-mPLMs and achieve comparable results with para-mPLMs on various natural language understanding tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Reid and Artetxe. PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining. NAACL 2022\n- Ding et al. A Simple and Effective Method to Improve Zero-Shot Cross-Lingual\nTransfer Learning. COLING 2022"}}]

Author rebuttal:

Meta-review: We thank the author for their insightful paper. This work presents a dictionary-based mask-language modeling (MLMs) pretraining that improves cross-lingual transfer. It first motivates the training method based on a careful analysis of the multilingual internal structure of MLMs (e.g. XLM-R). It then introduces TCS and SCS, which expands masked-language modeling based on dictionary-based cross-lingual pairs and on-the-fly cross-lingual mined candidates. 

Reasons to Accept
- Clarity of argumentation (R1, R2)
- Extensive empirical evidence that the introduced method leads to better cross-lingual transfer (R1, R2, R3)

Reasons to Reject: 
- Novelty of the results: Dictionary-based training for multilingual modeling has been explored in the past (e.g. Cao et al 2020; Wang et al 2022). The paper would benefit from a more detailed discussion and comparison with these papers (as pointed out by R3)

Suggestions: 
- Tables 12 and 13 are tough to read without bolding the best scores. 
- One of the arguments in the paper is that monolingual corpora do not include any parallel data or embedded text and still XLM-R-like models are able to learn geometrically-aligned representations. However, e.g., Wikipedia is somehow parallel across languages (many documents are shared across languages). Additionally, many documents include code-switched text. The paper would benefit from a careful discussion of this point and how this could also explain the multilingual abilities of multilingual MLMs. 

Missing references
- Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation, (Wang et al 2022)
- First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT,  (Muller et al 2021)
Decision: Accept-Findings