Iterative Task-adaptive Pretraining for Unsupervised Word Alignment
Authors: Zhi Li
Gao Xing
Ji Zhang
Yin Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: How to establish a closer relationship between pre-training and downstream task is a valuable question. We argue that task-adaptive pretraining should not just performed before task. For word alignment task, we propose an iterative self-supervised task-adaptive pretraining paradigm, tying together word alignment and self-supervised pretraining by code-switching data augmentation. When we get the aligned pairs predicted by the multilingual contextualized word embeddings, we employ these pairs and origin parallel sentences to synthesize code-switched sentences. Then multilingual models will be continuously finetuned on the augmented code-switched dataset. Finally, finetuned models will be used to produce new aligned pairs. This process will be executed iteratively. Our paradigm is suitable for almost all unsupervised word alignment methods based on  multilingual pre-trained LMs and doesn't need gold labeled data, extra parallel data or any other external resources. Experimental results on six language pairs demonstrate that our paradigm can consistently improve baseline method. Compared to resource-rich languages, the improvements on relatively low-resource or different morphological languages are more significant. For example, the AER scores of three different alignment methods based on XLM-R are reduced by about $4 \sim 5$ percentage points on language pair En-Hi.


Under review as a conference paper at ICLR 2023
ITERATIVE TASK- PRETRAINING FOR UNSU- PERVISED WORD ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
How to establish closer relationship between pre- traininiin and downstream task is valuable question. We argue that task-adaptive pretraining should not be just performed before task. For word alignment task, we propose an iterative self- supervised task-adaptive pretraining paradigm, tying together word alignment and self-supervised pretraining by code-switching data augmentation. When we get the aligned pairs predicted by the multilingual contextualized word embeddings, we employ these pairs and origin parallel sentences to synthesize code-switched sentences. Then multilingual models will be continuously finetuned on the aug- mented code-switchie dataset. Finally, finetuned models will be used to produce new aligned pairs. This process will be executed iteratively. Our paradigm is suitable for almost all unsupervised word alignment methods based on multilingual pre-trained LMs and doesn't need gold labeled data, extra parallel data or any other external resources. Experimental results on six language pairs demonstrate that our paradigm can consistently improve baseline method. Compared to resource-rice -rcch languages, the improvements on relatively low-resource or different morphological languages are more significant. For example, the AER scores of three different alignment methods based on XLM-R are reduced by about ~ 5 percentage points on language pair En-Hi.
INTRODUCTION
Although pre-trained language models (PTLMs) (Devlin et al. 2019b Conneau et al. 2020)trained with massive textual and computational resources have achieved high performance natural language processing tasks, there can be distributional mismatch between the pretraining and target domain corpora. tackle domain discrepancies, domain-adaptive pretraining with a large corpus in the domain of the downstream task is usefully employed, such BioBERT (Lee et al. 2020). However, this approach requires large corpora in the target domain and entails a high computational cost. Gururangan et al. (2020) propose task-adaptive pretraining and explore the benefits of continued pretraining on data from the task distribution. There are also others works (Gu et al. 2020); Karouzos al. 2021 Nishida al. 2021) focusing establishing a closer relationship between pre-training and downstream task. For example, Gu et al. (2020) add a task-guided pre-training stage with selective masking between general pre-training and fine-tuning, Karouzos et al. (202I) simultaneously minimize task-specific loss on the source data and language modeling loss on the target data during fine-tuning.
However, these methods generally follow a fixed paradigm: task task-adaptive pretraining then task training. There is an obvious lack of interactive feedback. Can the output of the task can be used to improve pretraining? See Figure And we find that an iterative self- superviedd task-adaptive pretraining paradigm can designed for unsupervised word alignment tasks. the following, we give detailed introduction about how to design new paradigm.
Continued pretraining of a LM on the unlabeled data of a given task (task-adaptive pretraining) (Gurrangaan et al. 2020) has been shown to be beneficial for task performance. And we think that simply pre-training LMs with MLM or TLM on monolingual parallel sentences is obviously not closely integrated with word alignment task. Based on the assumption that closer interaction between task pre-training and the task itself can improve performance, we propose an iterative seff-supervised continued pretraining paradigm, constantly pushing pre-trained LMs toward the word

---

Under review as a conference paper at ICLR 2023
Tadk-tappine
Tad-staptrve
Word Aligmmn
Uide--wachh Augmoratton
MLM
Toaning
Task
Task
Figure Continued pretraining of a LM on the unlabeled data of given task (task-adaptive pretraining) (Gururangan al. (0200) has been show be beneficial for task performance. However, it only occurs before tasks and is obviously not closely integrated with tasks. For word alignment task, we propose a new paradigm, in which task-adaptive pretraining and word alignment be executed iteratively.
alignment task. We augment sentences with self-labeled pairs and code-switching strategy. When we get the aligned pairs predicted by the multilingual contextualized word embeddings, we employ these pairs and origin parallel sentences to synthesize code-switched sentences. Then multilingual models will be continuously finetuned the augmented code-switched dataset. Finally, finetuned models will used to produce new aligned pairs. This process will be executed iteratively. On the one hand, data augmentation with self-labeled pairs and code-switching strategy alleviates data scarcity. On the other hand, training LMs with MLM on code-switched sentences can bring the different languages closer together in the embedding space and if training LMs on both code-switched sentence and corresponding origin sentences, the code-switched tokens (predicted pairs) will also move towards each other in the embedding space.
Our contribution can be listed as follows:
We design an iterative task-adaptive pretraining paradigm for word alignment, in which task-adaptive pretraining will be performed not only before task but also after task. other words, task-adaptive pretraining and word alignment will be executed iteratively.
Our paradigm is suitable for almost all unsupervised word alignment methods based on multilingual pre-traie pre-trained LMs and doesn't need gold labeled data, extra parallel data or any other external resources. We also don't need to introduce carefully designed loss function and the results can be easily reproduced.
In-depth analysis reveals that training model with standard masked language modeling on larreellgggggggrrrgeeeee targee--agggete and code-switched sentences approximately optimizing the similarity of switched tokens. This can serve as a potential explanation why code- switching can be used improve machine translation and cross-lingual tasks.
We perform experiments on six language pairs and demonstrate that our paradigm can consistently improve baseline methods. For example, the AER scores of three different alignment methods based on XLM-R are reduced by about ~ 5 percentage points on language pair En-Hi.
2
RELATED WORK
2.1 TASK ADAPTIVE PRETRAINING
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. Gururangan al. 2020) propose domain-adaptive pretraining and task-adaptive pretraining. They explore the benefits of continued pretraining on data from the task distribution and the domain distribution. Gu et al. (020)) add a task guiddd pre-training stage with selective masking between general pre-training and fine- sanettuning.... Jin et al. (2022) study lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. KKarouzos et al. (202T) simultaneously minimize task-specific loss the source data and language modeling
2

---

Under review as a conference paper at ICLR 2023
quite good idea.
Algamennt
quite
good
codi-stttcen Augmentationn
De: Das ist cine rocht gute Idee.
model
Diss
lnit
Eiite
reelir
guio
Idee
This
eine
pood
Idee
Close
loop
This
Ist
rocht
good
Idce
rech:
Das
quili
gullo
idea
Trainitx
Model
This
Iost
good
idee
An overview of our iterative task- -ddaptvve pretraining for word alignment. The alignment method is based on multilingual pre- pre-trained LMs and self-labeled pairs obtained by alignment method will be used to augment sentences by code-switching strategy. Then multilingual pre-trained LMs will be finetuned on the augmented sentences by self-supervised learning (masked language modeling). And this process will be executed iteratively.
Figure 2:
loss on the target data during fine-tuning, Nishida et al. (2021) propose a novel fine-tuning process: task-adaptive pre-training with word embedding regularization which runs additional pre-training by making the static word embeddings of pretrained Language Models close to the word embeddings obtained in the target domain with fast'Text. Kang al. (2022) modulate the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts. Different from them, we proposed a new paradigm, in which the output of word alignment task can be used to promote task-adaptive pretraining.
2.2 WORD ALIGNMENT
Statistical alignment models directly build on the lexical translation models such the IBM models (Brown et al. 1993) and their implementations Giza++ (Och & Ney 2003), fast-align (Dyer et al. 2013) and eflomal OOtling
Tiedemann 2016) are widely used for alignment. Based on NMT models (Bahdanau et al.) 2015) trained on parallel corpora, researchers have proposed several methods to extract alignments from them (Cohn al., 2016 Zenkel et al. 2019 Garg et al. 2019 Chen et al.. 2021 Zenkel et al. 2020a Chatterjee et al. 2022). Cohn et al. (216)) and Zenkel et al. (2019) create alignments from attention matrices. Garg et al. (2019) extract discrete alignments from the attention probabilities learnt during regular neural machine translation model training and leverage them in multi-task framework to optimize towards translation and alignment objectives. Zenkel et al. (2019 2020a) both extend the network with additional alignment layer. Chen al.) (202)) propose self supervised word alignment model that takes ad advantage of the full context on the target side, Chatterjee
al. (2022) propose simple architectural modification modern NMT systems to obtain accurate online alignments. However, most NMT-based methods require sufficient amount parallel data to train high quality NMT systems. which limits their application in the Low-resource languages and in domain-specific scenarios without extra parallel data. Sabet et al. (2020) propose effective methods to extract alignments from multilingual contextualized embeddings (Devlin et al., 2019b] Conneau et al. (020)) for word alignments without explicit training on parallel data. Dou & Neubig (2021) indicate that finetuning pre-trained LMs on extra parallel corpus can improve alignment quality,
There are also work supervised neural word alignment (Stengel-Eskin et al. 2019 Nagata et al.)
2020). For example, (Nagata et al. 2020) present novel supervised word alignment method based cross-laane laoos-la span prediction and formalize word alignment problem as question answering task. However, supervised data are not always accessible, making their methods inapplicable many scenarios.
3

---

Under review as a conference paper at ICLR 2023
2.3 CODE-SWITCHING
Code-switching is a prevalent phenomenon in multilingual communities where the words, morphemes and phrases from two or more languages are switched in speech or writing. And it has been employed to improve NMT tasks (Yang et al. 2020a Liu et al.,
2021: Yang et al., 2020b) and cross lingual tasks (Qin et al. 2020 Zhang et al. 2021 Lee et al.
2021 Most of related work attribute the improvement of model performance to an intuitive assumption: Using code-switching data to train models will encourages them to align representations from source and target languages by mixing their context information. In this work, we dig deeper and give an approximate formal explanations for the connection between word alignment and masked language modeling.
3 METHOD
Although multilingual contextualized embeddings of pre- LMs can be employed to achieve reasonable performance even in the absence of explicit training on parallel data, there still a clear gap: the model is trained with language modeling loss function and tested word alignments task. Dou & Neubig (2021) leverage pre-trained LMs and fine-tune them on parallel texts with new objectives designed to improve alignment quality. However, they need extra large amounts of parallel sentences, which is at least thousands of times larger than test sentences and limit their applications on low-resource languages and settings without parallel text. So here is a practical and valuable setting: if we only have a few parallel sentences which need to be aligned and pre-trained LM, can we further improve the performance of alignment methods? Figure 2 illustrates an overview of our paradigm. More accurate alignment results in higher-quality code-switched sentences. And finetuning on higher-quality code-switched sentences will encourage pretrained LMs to align representations from source and target languages by mixing their context information. A better pretrained LMs will obviously improve the accuracy of alignment. This iterative process will promote each other. In the following paragraphs, will elaborate on each part.
3.1 WORD ALIGNMENT
The task of word alignment can be defined as: Given a source- lource-language sentence x (z),. xx,) of length n and its target-language translation y (yi,".. Vmm) of length m, the method of word alignment needs find a set of pairs of source and target words:
A = {( Uy) I1 Ex,
Aligned words are assumed correspond to each other, i.e. for each word pair (zg,8z) Fz and y are semantically similar to each other within the context of the sentence. We focus on improving the methods which can leverage multilingual pre-trained LMs (Devlin et al. 2019h] Conneau et al. 2020) for word alignments by extracting alignments from similarity matrices induced from their contextualized embeddings without relying on parallel data. For each pair of parallel sentences x and y. these methods extract the hidden states of the k -th layer of the multilingual model: hy h Given these contextualized word embeddings, there are
:h",.... ... h and h² = (h many methods to obtain alignments. For example, simple and effective method Argmax (Sabet et al. 2020) is to align vi and 2; when h,; is the most similar to h.ū and vere-.ersa. That is, we set Aij
if arg max, St:,) ^ (i arg max, Sth,) and A,j 0 otherwise. And Sty =ii (ht A some normalized measure of similarity, e.g., cosine-similarity, Some other methods frame alignment as assignment problem (Sabet et al., 2020) or regularized optimal transport problem (Dou & AtyS.j.
is
Neubig 2021 Chi et al. 2021) and is defined by A = argmax Ae (o.1)x
In our setting, these methods will self-label parallel sentences. In order to get high quality aligned pairs, we filter the pairs with a particular threshold e:
Aytter {(a, S > c)
(2)
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Augmented Sentences Sampling
input A pair parallel sentences (x, y), alignment of the t-th iteration A,, alignment of the (t + 1)-th iteration A sampling probability Potd for old pairs, sampling probability Prew for new pairs, the sampling rounds rounds. output :ugmeetedd sentences Sang
S Saug 1
Aold + A innA A++1
Anew Ac+1 (Ap Ac+1)
SETAP & {(Aold. Pola), (Anew: Pnew)]
for 4- rounds do
for (A, E SETAP do
for pair E do
if P 2 random.
then
(x. y) =Code-switch (x,y) pair)
S Sang f S Saug U (x,
3.2 COTC-SWITCHED AUGMENTATION
Code-switching is prevalent phenomenon in multilingual communities where the words, morphemes and phrases from two or more languages are switched in speech or writing. The switched ones usually semantically similar.
Suppose we get p different pairs of source and target words for sentence x = (x1, .n) and y = (.,,... 3m is easy to augment sentences by code-switching. The source words and target words in one pair can be considered as synonyms and can be switched. If we augmented sentences by code-switch sentences with one token at time, then have C₁ kinds choices, which means get C different sentence pairs at most. If the number of switched pairs range from 0 p, then the maximum of different sentence pairs without considering special cases:
C₁ + C C +... C-= = 2
Obviously this is an exponential and impressive data augmentation method. In practice, we do not exhaust all code-switched sentences and the sampling method is illustrated in Algorithm We set Ao 0 and A₁ @ so that when 0. Algorithm still works and in this setting, the 0-th iteration exactly the standard task-adaptive pretraining. When t 1, alignment results of two successive iterations will be used that can assign different sampling distributions to the intersection and new aligned pairs. In general, we give higher probability for new aligned pairs. In addition, we will also add origin parallel sentences to augmented dataset which prepared for masked language modeling.
3.3
MLM ON CODE-SWITCHED DATASET
For masked language modeling (MLM), the input sequence consists of multiple individual tokens. A fraction of the input tokens are chosen randomly and replaced with < MASK >> tokens. Assume that these masked indices are collected together in a set mask(x) and we use å to denote a masked token. Then model with parameters 0 lears to predict mask(x) by the surrounding unmasked tokens X (x))
LMLM =
Σ
log
X Xi
$cmask(xx)
Using code-switched and origin parallel sentences to train model will encourages them to align repre- sentations from source and target languages by mixing their context information. More importantly, self-labeled word pairs have the same surrounding tokens. So training model these sentences will obviously align code-switched tokens in implicit manner and the predicted pairs will also move towards each other in the embedding space. For example, in Figure 2 "Das" and "This" is aligned
5

---

Under review as a conference paper at ICLR 2023
pair. When we train model the code-switched sentence "Das is quite good idea. and origin sentence "This quite good idea. ", the word "Das" and English word "This" will move towards each other in the embedding space because they have same surrounding tokens. We try to dig deeper and give an approximate formal explanations for the connection between word alignment and masked language modeling, which can also serve as potential explanation why code-switching be used to improve machine translation and cross-lingudl tasks.
Proposition: Training model with standard masked language modeling on loure-llnguage..argel language and source-target code-switched sentences is approximately optimizing:
lgt
~ Cx, ~ Cy:
where Cx; represents the contextualized embedding of token æ, in source sentence and cy represents the contextualized embedding of token yi in target sentence. And ex. and ey, are word embeddings of token 24 and y/ in vocabulary. cz ~ ex; represents that ez. is similar to ex;- See appendix for details.
We assume that this approximate similarity is sufficient to induce word alignment. Then we have: Corollary: The performance based on the last layer of optimized model (with iterative task-adaptive pretraining) is better than the best results (usually the eighth layer) of baseline model (without iterative lask-adaptive pretraining).
In section 4
we will test whether this corollary holds in experiments.
4 EXPERIMENTS
Our experiments focus three questions: (1) To what extent can our paradigm improve word alignment across methods, models, languages and layers. (2) The effect of the number of augmented sentences and sampling probability. (3) The detailed ablation study.
4.1 DATASET
Our test data are diverse set of 6 language pairs: Persian,Czeck German, French, Hindi and Romanian, always paired with English. All of them are public dataset: En-Fa(Tavakoli & Faili), En-Cs (Marecek), En-De En-Fr Och & Ney 2000 En-Hi and En-Ro statistics of datasets.
See Table 1 for detailed
Langs
En-Hi
En-Fa
En-Cs
En-De
En-Fr
En-Ro
Size
90
400
2500
508
447
203
[
1409
11606
44292
9612
4038
5033
PVS
0
0
23132
921
13400
O
Table I: Statistics of Datasets. Test sentences of the six gold word alignment datasets used in our experiments: English-Hindi (En-Hi), English-Persian (En-Fa), English-Czecl (En-Cs), English- German (En-De), English- Frenchh (En-Fr), English-Romann Romanian (En-Ro). "Size" refers to the number of sentences. S is sure alignments and P is possible alignments (S C P).
4.2 EVALUATION MEASURES
We use Alignment Error Rate (Och & Ney
2003 as the standard evaluation:
AER = AA JANS S + (ANP)
(5)
A + [S|
where A is a set of predicted alignment edges, S(sure) is (sure) unambiguous alignments and
P(possible) is ambiguous alignments (S с P). We report the percentage.
hhtp::/wwwwi...nnff informatii, wwtpmmatiik.tt-aahhhhhm www.wwwchhh.cceemmm wwwghhenndoggoldAlignnneee hhtpp//wwc.cecs..miii www.h.......c............... /

---

set the Under review as a conference paper at ICLR 2023
En-Hi
En-Pa
En-Cs
En-De
En-Fr
En-Ro
Model
Method
Fj
AER
Fi
AER
Fi
AER
Fi
AER
Fi
AER
F1
AER
IBM2
fast-align
38
62
58
42
78
23
71
30
85
16
68
32
IBM4
Giza++
48
52
43
82
18
78
92
69
eftomal
52
48
63
84
24
91
72
28
mBERT
Argmax
55
45
67
33
86
14
81
19
94
65
35
mBERT
Iter 0
56.2
43.8
68.5
31.5
86.1
13.7
80.7
19.2
94.6
5.3
66.4
33.5
mBERT
Iter
57.5
42.5
30.4
86.2
13.5
81.2
18.7
94.7
68.9
31.0
mBERT
ler 2
57.8
42.1
86.6
13.2
81.4
95.1
69.9
30.1
XLM-R
Argmax
61
39
71
29
87
13
81
19
93
71
29
XLM-R
Iter 0
62.9
37.1
72.9
27.1
88.2
11.6
18.6
94.4
72.4
27.6
XLM-R
65.5
34.5
73.9
26.1
88.9
10.9
17.3
95.0
73.9
26.1
XLM-R
Iter 2
66.2
33.8
74.6
25.4
89.3
10.6
82.5
17.4
95.0
4.8
74.2
25.8
XLM-R
Match
59.9
40.1
68.5
31.5
80.8
19.8
76.4
23.6
87.7
13.6
69.9
30.1
XLM-R
0
61.5
38.5
70.0
30.0
81.9
18.7
77.0
23.0
70.9
29.1
XLM-R
lerr
62.9
37.1
70.8
82.5
18.1
77.4
11.8
71.6
28.3
XLM-R
Iter 2
64.3
35.7
70.7
29.3
82.6
18.0
77.7
22.3
89.6
11.6
71.9
28.1
XLM-R
Iermaa
39
20
92
72
XLM-R
Iter 0
63.8
36.2
73.6
26.4
86.5
13.7
80.2
19.7
92.2
73.0
26.9
XLM-R
Iter
66.1
33.9
75.1
24.9
87.5
12.6
81.8
18.4
93.1
7.3
74.5
25.5
XLM-R
Iter 2
66.6
33.4
75.1
24.9
87.8
12,4
81.5
18.2
93.1
7.4
74.9
25.1
Table 2: Evaluation results six datasets. Argmax, Itermax and Match are three different alignment methods (Sabet al., 2020) based on multilingual contextualized embeddings. Results are average over different runs. Best results are in bold. (For AER, lower is better.)
4.3 BASE METHODS AND MODELS
Our experiments focus on three alignment method based on multilingual pretrained models: Argmax, Itermax and Match, proposed by SimAlign (Sabet et al., 2020) and follow the default setting of repository | without any modification. We use the contextualized embeddings from 8-th layer. We employ multilingual pretrained models: the multilingual BERT model (mBERT), which is pretrained on the 104 largest Wikipedia languages and XLM-RoBERT base (Conneau et al. 2020). which is pretrained on 100 languages on cleaned CommonCrawi data (Wenzek al.) 2020). The pretrained LMs often use subword segmentation techniques (Kudo & Richardson 2018 /Sennrich et al. 2016) and the above alignment extraction methods can only produce alignments on the subword level, we follow previous work (Sabet et al. 2020 Zenkel et al. 2020b] Dou & Neubig 2021) and consider two words to be aligned if any their subwords is aligned. For masked language modeling, we following Devlin et al. (2019a) and use special [MASK] token 80% of the time, random token 10% the time and the original token 10% of the time to perform masking. The batch size is set to 32. Max epoch is set 10. We the Adam optimizer with learning rate of 2e Weight decay is 0.03. We set the filtering threshold to 0.9. When the number of iterations is 1, only one alignment set sampling probability 0.7. When the number of iterations more than
sampling probability Pota to 0.7 and Pnew to 1.0. The sampling rounds is defaults to 5.
Probs
En-De
En-Fa
En-Hi
En-Ro
Rounds
En-De
En-Fa
En-Hi
En-Ro
0.5
17.6
25.9
34.8
26.1
17.4
26.2
35.1
26.5
0.7
17.3
25.8
34.5
26.1
5
17.3
25.8
34.5
25,8
0.9
17.7
26.2
35.2
26.5
8
17.3
25.7
34.3
25.8
Table 3:
AER with different sampling proba-
Table 4: The effect of the number of aug- mented sentences.
bilities.
4.4 RESULTS
As shown in Table report the Fl and AER scores the six language pairs. And Table also includes some widely used algorithms which is not based on pretrain mddels, such as Giza++ (Och & Ney 2003), fast-align (Dyer et al. 2013) and eflomal (Östling & Tiedemann 2016). It can

---

Under review as a conference paper at ICLR 2023
Layer
1
3
5
6
7
8
9
10
11
Baseline
53,4
58.5
61.9
57.4
49.7
45.9
40.4
40.2
39.4
42.2
44.4
46.7
En-Hi
Ours
53.5
55.1
59.0
52.9
45.8
41.2
36.0
35.7
34.5
35.9
37.0
39.1
Baseline
47.7
50.4
54.2
47.2
39.2
34.0
29.9
29.2
28.9
30.7
33.6
37.0
En-Fa
Ours
47.6
48.2
52.0
45.8
37.1
30.7
27.2
26.6
26.1
25.8
26.1
28.0
Table 5: Evaluation results different layers.
be observed that our method improve the performance three alignment methods on generating high-quality alignment pairs. And from section 3.2 the standard tusk-adaptive pretraining is equiv- alent to Iter O. With the increase of iteration rounds, our method can constantly improve model performance. This prove that iterative task-adaptive pretraining is effective. In addition, there is an impressive trends: The worse the initial performance, the more the improvement. For En-Cs, En-De and En-Fr, baseline methods perform well and the AER scores are generally lower than 20 percent. After three iterations, the AER scores are reduced by about 1.5 points on average. For En-Ro, En-Fa and En-Hi, which are relatively low-resource different morphological language- pairs, baseline methods perform poorly and AER scores are generally greater than 30 percent. After three iterations, the AER scores are reduced by about 4 ~ 5 points on average. This an exciting result, which doesn't 't need any additional parallel sentences and
En-Hi-aur
gold labels. In the following experiments, employ the alignment method Argmax for further analysis and explore the word aligmment across different layers. Figure and table show that our paradigm con- sistently improves baseline performance cross all layers. For both En-Hi and EN-Fa, the performance of last layer based on our methods consistently outperforms the best result of baseline models. So the Corollary in section 3.3 is valid and our assumption "this approximate similarity is sufficient induce word alignment" is supported by experiments.
60
En Hi-basallinee Er-Fa-baseline
50
45
35.
Layer
Figure 3: The comparison between ours and baseline
cross layers. Lower is better.
4.5 ANALYSIS
We further explore the effect of sampling probability and the number of augmented sentences model performance. We choose XIM-RoBERTa base for multilingual pretrained models and Argmax the alignment method. Given the large amount of possible experiments when considering language pairs, we do not present all scores for all languages and we will pick up four of them in most cases: En-De, an established and well-known dataset, En-Fa and En-Hi, two low-resource languages written in a different script and En-Ro. And only two rounds of iterative training are performed (t=0,1) and we only list the final results.
Table shows the Alignment Error Rate in the setting of different sampling probabilities. When the sampling probability of code-switching is too high or too low, the diversity of augmented sentences will decline, which may hurt model performance. Vable@3prove this point. Although the final scores of different sampling probabilities are close, the middle probability 0.7 achieve the best scores cross four languages. Table shows the effect of the number of augmented sentences. The sampling rounds proportional to the number augmented sentences. From this table, we can infer that the scores will increase in general when more augmented sentences becomes available. But gains continue to decay. At the same time, the cost of MLM training will increases with more augmented sentences. So set the sampling rounds to without special statements.
8

---

Under review as a conference paper at ICLR 2023
Languages
En-De
En-Fa
En-Hi
En-Ro
Baseline
19
29
39
29
NO-CS
18.6
27.1
37.1
27.5
Random
18.4
26.8
36.5
27.3
NO-Filter
17.8
26.4
34.9
26.0
Ours
17.4
25.8
33.8
25.4
Table 6:
Ablation study of code-switching strategy.
4.6 ABLATION STUDIES
For ablation study, we choose XLM-RoBENI base for multilingual pretrained models and Argmax the alignment method. We consider three kinds of ablation studies. Table lists the final iterative results. "NO-CS" means only origin monolingual parallel sentences and there are no code-switched sentences. "Random" means the pairs used for code-switching are randomly generated and they are not gold-pairs in most cases. Note that the dataset "Random" also includes origin monolingual parallel sentences. "NO-Filter" means that we don't 't use threshold to filter the pairs and all aligned pairs will be employed to augment code-switched sentences. The result of "NO-CS" indicates that without code-switched sentences, method can improve performance. fact, it is almost equivalent to standard task-adaptive pretraining and Iter 0 (section 3.2). 'hhe comparison of "Random" and "NO-CS" shows that the improvement of "Random" mainly comes from origin monolingual parallel sentences. And the randomly code-switched sentences only bring a very slight boost. The comparison of "NO-FE NNO-Filter" and "Ours" indicates the filtering the pairs with a threshold is beneficial to model performance.
5 CONCLUSION
Inspired by the fact that continued pretraining of pretrained models on the unlabeled data of a given task has been show to beneficial for task performance, we further design an iterative tasptidee pretraining paradigm for word alignment, in which task-adaptive pretraining will be performed not only before task but also after task. The multilingual models will be continuously finetuned on augmented code-switched dataset. The iterative process will promote each other. More accurate alignment results in nigher-quality code-switched sentences. And finetuning on higher-quality code- switched sentences will encourage pretrained LMs to align representations from source and target languages by mixing their context information. A better pretrained LMs will obviously improve the accuracy of alignment. Experimental results six language pairs and demonstrate that paradigm can consistently improve baseline methods.
We considering the paradigm to other token-level tasks such as such as Named Entity Recognition and Parts-of-speech tagging. And how establish connection between the downstream tasks and self-supervised tasks of pretraining stage is key point, especially for low-resources tasks and languages. In fact, the prompt- based methods in which downstream tasks are reformulated language modeling are alternative solutions. And we are trying to combine our iterative task-adaptive pretraining with prompt-basee based methods.
general paradigm about iterative task-adaptive pretraining and will apply
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA. USA. May 7-9, 2015, Conference Track Proceedings, 2015. TRLhhttp:/arrxtt org abs/ 0473
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311 1993. URL
http:/aaaaaattooooooorgrg////// org 2000
Soumya Chatterjee, Sunita Sarawagi, and Preethi Jyothi. Accurate online posterior alignments for principled lexieally-constraineed decoding. In Smaranda Muresan, Preslav Nakov, and Aline

---

Under review as a conference paper at ICLR 2023
Villavicencio (eds.), Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). ACL 6689. Association for Computational Linguistics, 2022. URL/https://acllatthoo acl-long. 460
ACL 2022, Dublin, Ireland, May 22-27, 2022. pp. 6675-
org /rgg/000 2022
Chi Chen, Maosong Sun, and Yang Liu. Mask-align: Self-supervised neural word alignment. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/UCNILP 2021, (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pp. 4781-4799 Association for Computational Linguistics, 2021. doi: N0.18653//1/22 acl--oogg.66999 369. URL URt.httsss http:///ooooorr/////// 10. 18653/v1/228/ acl -ong. 369
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, and Furu Wei. Improving pretrained gross-lingure language models via self-labeled word alignment. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing, ACI/IINNLP 2021, (Volume I: Long Papers). Virtual Event, August 1-6, 2021. pp. 3418-3438 Association for Computational Linguistics. 2021. doi: 10.18653/v1/ 2021 acl- 265, URL https: orgoorg//// 10. 18653/v1/2222 acl-lalllloon 265
Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 876-885, San Diego, California, /ttt/ttps
June 2016. Association for Computational Linguisties. doi: 10. 10.18865/1/111--1 1002.
//aclanthologyy.om -ttp:/ooooooogg/1-11111111 1102.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 8440 8451, Online, July 2020. Association for Computational Linguistics. doi: 10. awwp///....0maii.a7..45413..32 URL https://aclanthooogy.oogg 2020
www.:00..111-m.. mollmmain... 747
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4 4186. Association for Computational Linguistics, 2019a. doi: 10. 10.18653/1/119-14222 URL https://doi http://oo.orr///// 100 18653/v1/n19-1444
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volune 1 (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019b. Association for Computational Linguistics. doi: 10. 11111/6/1/11111443 1423. URL hhttps Taa.lannhhlogg..rggg..
1423.
Zi-Yi Dou and Graham Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 2112-2128. Association for Computational Linguistics, 2021. doi: 10.18653/w1/2221.ceom http:///222eeeelllle. wea.--maa..nnnmmnmm.mmm 181. URL http:ttts:///oo.ooo ira.ddoo.ooog//// 10. 18653/v1/2021. eacl2maiicmmmmmmm 181
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of IBM model 2. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings. June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pp. The Association for Computational Linguistics, 2013. URL hoLLhhtps::/ https:////aaathooogg -
10

---

Under review as a conference paper at ICLR 2023
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Language Processing and the 9th International Joint Conference on Natural Language Processing, F.MNIP-IJCNIL. 2019, Hong Kong, China, November 3-7, 2019, pp. 1452-4661 Association for Computational Linguistics, 2019. doi: 10.18653/1//D199 19-5344 URL http::tts////c //dp:i////..... org/10
Empirical Methods in Natural
18653/v1/D19-1453
Yuxian Gu. Zhengyan Zhang, Xiaozhi Wang. Zhiyuan Liu, and Maosong Sun. Train no evil: Selective masking for task-guided pre-training. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6966-6974, Association for Computational Linguistics, 2020. doi: 18663//112002..mmmpmmmiiii
URL https
//doi. 8653/v1/2020 mamlp-main.565 566
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't it stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/11/2200 acl-main.740, URL
htwpe//nnnnoooooooooo/oooo org/2020.00. amann 740
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Wen Li, Xiaokai Wei, Andrew O. Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI. 2022, Seattle, WA, United States, July 10- 15, 2022, pp. 4764-4780, Association for Computational Linguistics, 2022. doi: 10. 10.16633/1/20222 naacl-main.33 URL httt::///dooi. orwoorgg/// 10. 11653/
351
Minki Kang, Jinheon Back, and Sung Ju Hwang. KALA: knowledge-auggmenedd language model adaptation. In Marine Carpuat, Marie-Catee Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 5144-5167. Association for Computational Linguistics, 2022. doi: 2.118653/v/22022.nag htt.:/22naaaa-mmii333333 URL httpps//ddo..rrgg org/10. 18653/v1/2022 www.namaalllmaaaama 379
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. UDALM: Unsuper- vised domain adaptation through language modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2579-2590, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202 mmacl-main.203.
.og. 2021. wwa.c-mmannnn.mmmmmmmmmmm 203
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 2018, pp. 66-71. Association for Computational Linguistics, 2018. doi: 10. 10.18653/w1/11--202 URL https://doi. 000 18653/v1//18-200
Dohyeon Lee, Jacseong Lee, Gyewon Lee, Byung-Gor -oung-Gonoon Chun, and Seung-won Hwang. SCOPA: soft code-switching and pairwise alignment for zero-shot cross-lingual transfer. In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (eds.), CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November / - 5, 2021, pp. 3176-3180. ACM, 2021. doi: 10.1145/ 07996677348276.66 URL WtRp:t/tp:///do org 10. 145/ 3459637 3482176
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinform., 3644): 2020. doi: 10. 10.109//boinfommttissttzz. URL

---

Under review as a conference paper at ICLR 2023
Zihan Liu, Genta Indra Winata, and Pascale Fung. Continual mixed- -anguuge pre- pre-training for extremely low-resouree neural machine translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computationnl Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACI/UCCLL 2021 of Findings of ACI, pp. 2706- Linguistics, 2021. doi: 10. 10.66///11/2021 18655////22...nndnnnsaallll. findinns--ac1.239
2718. Association for Computational I.
URL https://doii http://..oorrr/1.//////////////0
findings-acl 239
David Marecek. Automatic alignment of tectogrammatical trees from czech-english parallel corpus.
Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. A supervised word alignment method based on eross-language span prediction using multilingual BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 555-565. Association for Computational Linguistics, 2020, doi: 10.18653/v (0/53//1/2020.mmllg main.41. URL https://doi. htt.://////...11///12222220 18653/v1/2020. 1863://1/200..mmmmmmaaa........ main.4
Kosuke Nishida, Kyosuke Nishida, and Sen Yoshida. Task-adaptive pre- pre-training of language models with word embedding regularization. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Findings of the Association for Computational Linguisties: ACL/IJCNPP 2021, Online Event, August 1-6, 2021, volume ACL/JJNNLP 2021 of Findings of ACL. pp. 4546-4553. Association for Computational Linguistics, 2021. doi: 10. 11611//1/222211.... 1 findings--cl1398.. URL
https::/doi.org// wtt.:////....11111122222220
findings-acl... 398
Franz Josef Och and Hermann Ney. Improved statistical alignment models. In 38th Annual Meeting of the Association for Computational Linguistics, Hong Kong, China, October 1-8, 2000, pp. 440 447.
ACL, 2000. doi: 10.31 10.31 1075574. URL https:://aaaaantoooogg..././//// hrtp://ooooo...r//0-1111555
Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Comput. Linguistics, 19-1), 2003. doi: 10. 10.1162088120000221337722 URL
https://doi orgoo0g///00 10. 1062/ 103321337221
Robert Östling and Jörg Tiedemann. Efficient word alignment with markov chain monte carlo.
Prague Bull. Math. Linguistics, 106:125-146, 2016. URL httth:/p///u/llmmfff wtt.:mfff...iiccpmmm/...
106/art-ostling int--osstiiintiiidmmmm.... pdf]
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. Cosda-ml: Multi-lingual code-switching
data augmentation for zero-shot cross-lingual NLP. In Christian Bessiere (ed.), Proceedings of the
Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3863-3860. ijcai.org. 2020. doi: 20024999/iccai.22220533 URL httttp::/ddoi..rgg/// 24963/ijcai 2020/ 533
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1627-1643. Association for Computational Linguistics. 2020. doi: 10.18653/v1/2020 findings-emmlp.144 147. BRLhhttps://doo / findings-emnip 1477
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Pa- pers. The Association for Computer Linguistics, 2016. doi: 10.18653/v1/p16- 1162. URL https:://oi..om org 10. 18653/v1/p16-1162
Elias Stengel-Eskin, Tzy Su, Matt Post, and Benjamin Van Durme. A discriminative neural model for cross-lingus word alignment. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMMN.P-IJCNIPP 20/9, Hong Kong, China, November 3-7, 2019, pp. 910-920. Association for Computational Linguisties, 2019. doi: 10..8863/1//11-1110 1084.
: htt.//ii..org/.... 0tw.1/1..86611111 18653/v1/019-1083
Leila Tavakoli and Heshaam Faili. Phrase alignments in parallel corpus using bootstrapping approach.
12

---

Under review as a conference paper at ICLR 2023
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pp.
4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 119-10-95546-34-4 URL httl/httss///aclatteee www.olaagoooooaaaaeeoooooooo 1ree//2022..1111111111 www.rrec1..... 494
Zhen Yang, Bojie Hu, Ambyera Han. Shen Huang. and Qi Ju. CSP: code-switching pre-training for neural machine translation. In Bonnie Webber, Trevor Cohn, Yulan Hc, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 2624-2636 Association for Computational Linguistics, 2020a. doi: 10. 10.186533v1/2020.o 10.11663//1202200mmlmmmaaaaa 208.
18653/
v1/2020, emnlp-main. 208
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. CSProde-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 2624-2636, Online, November 2020b. Association for Computational Linguistics. doi: 10. 1853// ww/20000.mmlpmmmmin20000 URL
https://aclanthooooo.oorgg org/200 emnlp-main. 208
Thomas Zenkel, Joern Wuebker, and John DeNero. Adding interpretable attention to neural translation models improves word alignment. CoRR, abs/1901. 1359,, 2019. URL http://arxiv http://aaxxiv..o///bbbbbb org/abs/ 1901 11359
Thomas Zenkel, Joern Wuebker, and John DeNero. End-to-end neural word alignment outperforms GIZA++. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computaaional Linguistics, ACL 2020, Online, July 5-10, 2020. pp. 1605-1617. Association for Computational Linguistics, 2020a. doi: 10.18653/ v1/2020.ac1-mann.14 146. URL https://doi. org/10. 18653/v66vv// 00653//1/22200..2 0 146
Thomas Zenkel, Joern Wuebker, and John DeNero. End-to-end neural word alignment outperforms GIZA++. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingmistics, pp. 1605-1617, Online, July 2020b. Association for Computational Linguistics. doi: 10.18663/v1/22220ac--m acl-mman.. 146. URL / wrg. 220./202.222 acl-main. 146
Wenxuan Zhang, Ruidan He, Haiyun Peng. Lidong Bing, and Wai Lam. Cross-lingual aspect-based sentiment analysis with aspect term code-switching. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 9220-9230, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.186534 v1 v122021 .en2lpeenppaaeee wwnnlmmmain.2211 727. URL httppp://aclannhooooo.orrg org 2021 emnlp-main.. 727
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 A PROOF OF MLM ON CODE-SWITCHED DATASET
Proposition Training model with standard masked language modeling on source- lancea-a target- language and source-target code-switched sentences is approximately optimizing:
dgh
Ex.
Ey:
Cy
where cx represents the contextualized embedding of token in source sentence and c. represents the contextualized embedding of token y/ in target sentence. And Ex, and ey. are word embeddings of token æy and yi in vocabulary.
Note Under the existing conditions, we can not derive a strict bound but an approximate conclusion. b, if the projection components of and b onto another vector e are the same: a c = b.c. c. We think this is approximately reasonable for word alignment task, because for word alignments method two words are aligned as long their similarity is higher than other words in two parallel sentences and doesn't It need to exceed a fixed number. And in section 3.3, we give a corollary. Subsequent experiments prove that our assumption reasonable.
Given R", b E IR" .c E R assume
Proof We denote the embeddings of the corresponding original tokens as e1, e2, eL. The MLM objective (MLM(x) can be formulated as:
Σ log exp (m. e.))
Σ log Σ exp (m; ex m; e;)
(6)
[M
TEM Σxm exp (m; ex) M
iEM k=1
where M denotes the of masked tokens and |V] is the size of vocabulary V. m, is hidden state of the last layer at the masked position, and can be regarded as fusion of contextualized representations of surrounding tokens. Given two sen- tences: one source- sentence x = (Æ1, 24-1, Iz, "441, ,In) of length and its code-switched sentence x' (x1,-". ; Ub. 1,+1,"" ,Ta), where (1,, is aligned pair. If we only mask in the x and in the x', then Xmask (r .I:-1:< < mask 2,44115' [,)= , < 21-1,
mask 24411 .n) x mask> the
loss function can be written as
CMLM Lx +xx
1 (log
exp (m ex ex,, log
exp (m ek ey,)) (7)
This inequality below is easily proved.
max fII,..., Fn) I log Σ
s s max (2,, logn
(8)
i=0
So for
M
Lx
log
exp (m Ex - m Ex:)
(9)
We have:
m7 En-
m Ex,
m- Eo - m Ez,
m Cx.
Ex:
Ex,-1 -1 Ex;
max
0
<
log
m m.
< max
+log n
Czi+1
Czi
1=1
Cx,+1
Ezi
m ejvl m Ex:
m E|1a - m Exi
(10)
In Ineq 10 is fixed value. So when training model with this loss function, model is optimized to [ In other words, ex Cxy, Vk E [V). When k y/.
learn eg
ex; < 0, Vk
14

---

Under review as a conference paper at ICLR 2023
have Cy, I Cz,- Similarly, for Lx', have Ez; model with loss function LMLM Lx + Lx, model will be optimized to lear m Ez. = m Cy. This equation can't 't ensure ez, Ey, but ez, Ey. to some extent. For standard masked language modeling, there is probability that the original token will not be masked and we use c to represent the hidden state of the last layer, which is the contextualized embedding of token z; So have c ek S. cg; Ez, vk E [V]. Obviously, Ca, c. Similarly, if we consider target language sentence y = (y,," ... Wi-1. 1i+ ... Mn., we have Ey; ~ c So training model with masked language modeling on source- languge, target- target-language and tarreettarget code-switched sentences is approximately optimizing:
Cy. So when training
arc Cx, Ey 91
15


Reviews:
[{"review_id": "OyRbWySJa8t", "reviewer": "Reviewer_KcR5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies unsupervised word alignment using pre-trained multilingual LMs. The paper proposes an iterative method that repeats two steps: (1) generate word alignment using a LM, (2) using the word alignment, generate synthesis code-switched parallel text to fine-tune the LM. Empirically, this iterative method reduces alignment error rate for six language pairs.", "summary_of_the_review": "The paper proposes a novel method for unsupervised word alignment and shows good results. However, the paper does not discuss time complexity, which could be a key weakness of the method. Overall, I weakly recommend acceptance.", "strength_and_weaknesses": "Strength:\n- The proposed method is well-motivated and clear.\n- Empirically, the method can effectively improve alignment error rate.\n\nWeakness: \n- The biggest weakness of the proposed method is that it can be slow. Practitioners will need to decide whether using the method is worth the computation time and how to choose hyperparameters. Therefore, it would be very helpful if the paper can discuss computation time and add relevant metrics in the experiment result tables.\n- The introduction suggests that the proposed method (or framework) can be applied to other NLP tasks, but the paper is restricted to only one task (word alignment). Ideally, the paper should have other tasks to show that this framework can really generalize to other tasks.\n- The paper only experiments with up to 3 iterations. I am curious if the method can continually improve AER. Does the method converge after 3 iterations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EvlHBFhjHtI", "reviewer": "Reviewer_mPaq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a challenging scenario for unsupervised word alignment where   human labeled word alignment data is not available and only a bilingual dataset for testing is given. Technically, it extends the alignment model proposed by Sabet et al. (2020) and iteratively updates the alignment model defined on top of a multi-lingual pre-trained language model. Specifically, it iteratively performs the follow two steps: 1. it employs the updated alignment model to generate some word alignment dataset with pseudo alignments for the given bilingual test dataset; 2. it fine-tunes the multi-lingual pre-trained language model through code-switch to updates the word alignment model. In this way, the multi-lingual pre-trianed model can be optimized for the word alignment task. Experiments on five datasets demonstrate the proposed approach delivers consistent improvements over the baseline Sabet et al. (2020).", "summary_of_the_review": "Please strengths and weaknesses.", "strength_and_weaknesses": "Strengths:\n\n1. The proposed approach is appealing and easy to implement.\n2. The experiment results shows that the proposed approach achieves good performance compared with the baseline.\n\n\nWeaknesses:\n\n\n1. Although the key idea for the proposed is explained clearly, some important details are unknown such that I have some key concerns on the proposed approach. Specifically, is the equation (4) updated at the sentence level or updated on the entire dataset? By the sentence level, I mean that the model is updated only for each sentence x and then the updated model is applied for inference on this sentence only. I guess it is updated on the entire dataset and then the natural question is how this approach works on a very small test dataset, for example, including one or two sentences. In addition, how many sentences does this method need in the test dataset to achieve good performance? These questions are important to show the superiority of the proposed method, but this paper does not study them.\n\n2. In essence, the technical contribution of the proposed approach is limited in my opinion. At the high level, this approach is similar to Dou and Neubig (2021): both of them are extensions of Sabet et al. (2020); both employ the multi-lingual pretrained language model as the initialization of the word alignment model, and require a bilingual dataset to optimize another loss function which is aware of word alignment. The difference is that this paper only requires a relative small bilingual dataset (i.e., test dataset) whereas Dou and Neubig (2021) uses a large bilingual dataset. Theoretically, it is possible to use the similar idea of Dou and Neubig (2021) on the same scenario as a baseline. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C9cHMtleRJ", "reviewer": "Reviewer_hkhx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This proposes an  iterative self-supervised task-adaptive pretraining  framework with a focus on the task of word alignment. In other words, task adaptative pretraining and word alignments are done iteratively. The method works without labeled data.\nExperiments shows a reduction of the AER with an XLM-R model.\n", "summary_of_the_review": "In my opinion the proposed method lacks of originality. In short, I would described it as an application of task-adaptative pre-training to the task of word alignments. An extensive analysis and better motivation for code switiching would improve the paper, but it would still lack of substance for a top-tier conference. ", "strength_and_weaknesses": "Strengths:\n- the paper is well-written and illustrated\n- the idea is very simple\n- the application to word alignment is novel\n- the method is somewhat reproducible \n\nWeaknesses:\n- the paper lacks of substance. To me, it only sticks together some pieces from previous work (task-adaptative pre-training) and applies it to a particular task. \n- the need for code switching is never properly motivated in the paper. It makes sense to do code switching, but since this is a crucial point of the method it should be extensively motivated. Why does the method use code-switching? Code-switching is introduced very early, but briefly, in the paper, then there is a section about it in related work, and then it is presented as a component of the method. The relationship between task-adaptative pretraining and word alignments is clearly given and motivated, but we miss the links with code-switching.\n- the analysis doesn't answer the questions that I expect the readers will have:\n- How does this method increase the computational cost of the training? (it looks like a huge increase)\n- How does this method compare to just pre-training XLM-R for a few more epochs? (I have no idea)\n- What are the limits of the method?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B5QnHNNORCw", "reviewer": "Reviewer_j4s3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work propose a simple fine tuning method for unsupervised word alignment based on multilingual contextualized pre-trained language model, e.g., BERT. This method is based on the similarity score of the word representations in two languages and employs fine tuning by 1) iteratively augmenting code-switched training data based on the previously predicted word alignment and 2) masking inputs. The proposed method allows a model to focus more on the paired word representation considering context. Experimental results show that the proposed method achieved gains when compared to the prior baselines.", "summary_of_the_review": "This work is interesting in employing code-switching during fine-tuning combined with masking. Experiments show gains when compared with baselines. However, it is not clear how the hyperparamters were tuned.", "strength_and_weaknesses": "Strength\n\n* It is a simple yet effective method to improve word alignment quality without any supervisions. \n\nWeakness\n\n* It is not clear why the proposed method works. The author tries to prove it by the similarity of code-switched and masked sentences. However, the proposed method might prone to poor local optimum problem in the similar manner as done by an EM algorithm.\n\n* The experimental procedure is a bit unclear in that that there exists no training data, but a model is simply fine-tuned on test data. Thus, it is not clear how the hyper-parameters are tuned, e.g., sampling probability, for a fair comparison.\n\n* Given the iterative procedure for fine-tuning, it is not clear how the word alignment will be revised, e.g., whether the proposed method is trying to concentrate on more confident pairs or not. Probably it would be good to analyze the word alignment confidence in each iterations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject