Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation
Authors: Xuan-Phi Nguyen
Hongyu Gong
Yun Tang
Changhan Wang
Philipp Koehn
Shafiq Joty
Venue: ICLR 2022 Poster (2022)
Keywords: machine translation
unsupervised machine translation
pseudo-parallel data
contrastive clustering
pretraining

Abstract: Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative back-translation. However, there may exist small amount of actual parallel data hidden in the sea of unlabeled data, which has not been exploited. We develop a new fine-tuning objective, called Language-Agnostic Constraint for SwAV loss, or LAgSwAV, which enables a pre-trained model to extract such pseudo-parallel data from the monolingual corpora in a fully unsupervised manner. We then propose an effective strategy to utilize the obtained synthetic data to augment unsupervised machine translation. Our method achieves the state of the art in the WMT'14 English-French, WMT'16 German-English and English-Romanian bilingual unsupervised translation tasks, with 40.2, 36.8, 37.0 BLEU, respectively. We also achieve substantial improvements in the FLoRes low-resource English-Nepali and English-Sinhala unsupervised tasks with 5.3 and 5.4 BLEU, respectively.


Published as a conference paper at ICLR 2022
CONTRASTIVE CLUSTERING TO MINE PSEUDO PAR- ALLEL DATA FOR UNSUPERVISED TRANSLATION
Xuan-Phi Nguyen'; Hongyu Gongl, Yun Tang', Changhan Wang Philipp Koehn
& Shafiq Joty!
Meta AI
#Nnyyang Technological University
JJohns Hopkins University
nguyenxu0028e. nt edu.
ABSTRACT
Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative back- translation. However, there may exist small amount of actual parallel data hid- den in the sea of unlabeled data, which has not been exploited. We develop a new fine-tuning objective, called Laaguage-Agnoseee
Constraint for SwAV loss, or LAgSwAV, which enables pre-trained model to extract such caruuddlpallllllll data from the monolingual corpora in a fully unsupervised manner. We then propose an effective strategy utilize the obtained synthetic data augment unsupervised machine translation. Our method achieves the state of the art in the WMT'14 English-French, WMT' 16 German-English and English-Romantee supervised translation tasks, with 40.2, 36.8, and 37.0 BLEU, respectively. We also achieve substantial improvements in the FLoRes low-rce oowrresource English-Nephn and Inglish-Sinhale unsupervised tasks with 5.3 and 5,4 BLEU, respectively.
bilingual un-
INTRODUCTION
The quest build a fully unsupervised machine translation (UMT) system, where only unlabeled monolingual corpora are available, has received increasing attention in recent years. Profoundly, Lample et al. (201 8a c) introduced the general principles for UMT that include cross-lingual initial-
ization, language modeling and iterative tuck-translation. Although various UMT variants (Lample cct al. 2018a Conncau & Lample 2019; Song et al. 2019 Liu et al. 2020 Nguyen et al. (021) applied these principles differently, they ultimately train their models on noisy synthetic parallel data generated by the models themselves or some randomization processes, which may cause harm the generated synthetic data is often noisy and low-qualing especially at the beginning of training- However, these methods may have missed out a potential that some parallel (or comparable) data may exist in the monolingual corpora, which be effectively mined augment the UMT training.
This paper is motivated to explore mining of -aauddl-arrllell data for UMT tasks. While there have been limited research in unsupervised mining of such data (Wu et al. 2019a b). there have been several studies on bitext mining from sentence embeddings in semi-supervised translation or zero- shot transfer setups (Zweigenbaum et al. 2018 Schwenk 2018: Artetxe & Schwenk, 2019a b). However, these methods require the models to be pre-trained on massive multilingual parallel data, which renders them inapplicable and incomparable
the fully unsupervised setup. Furthermore, such models may not behave well off-the-shell on low-resource languages that were not in the pre- training data, such as Nepali and Sinhala (Guzmán et al.) 2019). Meanwhile, applying these mining algorithms directly to sentence embeddings produced by fully self-supervised models (Conneau & Lample| 2019) leads to a significant amount of misalignments, which hurts the UMT models as shown later in experiments ($4).
Most of work done during an internship Meta AT.

---

Published as a conference paper at ICLR 2022
Business (Ro)
Sports (Ro)
Sports
Animal (Ro)
Sports (En)
Business
Animal (En)
Business (En)
Animal
Placca esto un mamier.. [Ro]
Cat mammall [En]
Cat mammal. [Enn]
Pisica mamiler [Po]
(a) Contrastive but not tanguagc-agnostie
(b) Contrastive and langunge-agnostic
Figure 1: Illustrations of semantically contrastive and Non-language-agnostt (Ia vs. language- agnostic (Ib) representations of English (red) and Romanian (cyan) sentences. For semantically contrastive models, sentences of the same topic will be clustered the same region. If the models also language-agnostic, the semantic regions will not differentiate sentences by language.
In this paper, we propose novel modification to SwAV loss (Caron et al. 2020), called Language- Agnostic Constraint for SwAV (LAgSwAVI) which is used to fine-tune existing pre-trained self- supervised models (Conneau & Lample 2019; Liu et al., 2020), that they become semantically contrastive and gonguage-agnostie Specifically, semantic contrastiveness is defined. in this paper, as the model's ability to cluster sentences of similar semantic contents into similar groups. Meanwhile, iinguage-agnostic property means that the embedding space of sentences from different languages are indistinguishably mixed together. Figure further illustrates the two properties. We argue that these two properties combined can help the model mine more accurate ascudo-paralled data, since language-agnosticism reduces the distances between cross-lingual embeddings while semantic contrastiveness helps bring sentences of similar contents closer together even if they are in different languages. To achieve the semantically contrastive property, we adopt the SwAV loss (Caron et al.) 2020), which is designed to classify input data into different clusters that reflect certain semantic characteristics. However, since Sw is not designed to be language-agnostic, we introduce a new aanguage-agnosti constraint for it enable the latter property. Analysis experiments in $ show that our constraint significantly outperforms vanilla SWAV loss as well as other baselines in mining pseudo-parallel data from monolingual corpora.
After fine-tuning pre-trained self-supervised models with LAgSwAV loss, we sample sentences from both languages and pass them the model to obtain the SwAV cluster assignment probabilities, which are treated as sentence embeddings. The soft cluster assignments are then matched by margin-based algorithm (Artetxe & Schwenk, (019b) with additional filtering criteria ($3.2). To augment UMT training with the obtained pseudo- parallel data, we propose a simple ranking-based weighted cross-entropy loss to train on this data along with the standard iterative back-trane mack-translation ($3.3). We also adopt dynamic weight to the augmentation loss to alleviate the overfitting effect due to the limited augmentation data.
We tested our method on the standard bilingual WMT' English-French, WMT' German- English and English- -olisi---n unsupervised translation tasks and achieved the state the art with 40.2, 36.8, and 37.0 BLEU, respectively; which is up 2 BLEU gain from the baselines. In the FLo- Res low-resource unsupervised setups (Guzmán et al. 2019), our method also outperforms mBART baseline (Liu et al. 2020) by up 2.8 BLEU in the English-Nepali and English-Sinhal. tasks. We conducted series of analyses to demonstrate the outperforming ability of LAgSwAV in min- ing pseudo-parallel data. The method is also found outperform supervised LASER (Artetxe & Schwenk 2019a), which can be attributed to LASER's 's lack of language specifiers.
BACKGROUND
2.1
UNSUPERVISED MACHINE TRANSLATION
Lample et al. 8a) and Artetxe et al. (2018) were among the first to employ iterative back- translation in fully unsupervised machine translation (UMT). Lample et al. (2018c) later formu-
lated the three- three-prinet three-principles principles to UMT: initialization, language modeling and iterative back-translation Sennrich et al. 2016a). Initialization can takes form of MUSE bilingual dictionary (Lample et al.
Code: http:///hthh..oommmm///////aasss
2

---

Published as a conference paper at ICLR 2022
2018b a), cross-linguat masked language model (XLM) (Cnnneau & Lample 2019) Sea2Seg
model (Song et al. 2019). Language modeling often takes form of denoising autoencoding (Lam- ple et al. 20] is omitted entirely (Song et al. 2019). Apart from these principles, Nguyen et al. ) use two distinet UMT teachers to distill the student in a two-stage back-translation process. Despite being different, most existing methods commonly employ the same iterative back- translation process. Multilinguality is also proved to be useful as Liu et al. (2020) and Garcia et al. (202I) utilize extra training data of more than two languages. These methods, however, do not leverage the potential pseudo- pseudo-parallel data that may exist in the monolingual data to improve UMT.
2.2
SEFUDO-PARALIEE DATA MINING
Parallel data mining for machine translation has been an active field of research (Zweigenbaum et al, 2018 Schwenk 2018 Shinea-Rioss et al. 2017). Artetxe & Schwenk (2019b) suggested an effective margin-basee algorithm mine parallel data using sentence embeddings, which we also use in our method. More recently, LASER (Artetxe & Schwenk 2019a) was trained on massive multilingual parallel data from 93 languages. Nonetheless, most of the effort has been invested largely in supervised setup where the training data is parallel. In terms of UMT, Wu et al. (201 2019a) proposed an extract-edit approach to mine relevant data and edit for use, while Wu et al. (2019b) mine weakly-pairt -pared documents, 'Tran et al. (2020) suggested to train on massive multilingual data from 25 languages and mine data from 180 translation directions, which may not be applicable in bilingual setup, where monolingual data only for the two languages under consideration are available.
2.3 SWAPPING ASSIGNMENTS BETWEEN VIEWS (SwAV Loss)
Caron et al. 2020) introduced Swapping Assignments between Views (SwAV) loss means
to conduct self-supervised training in computer vision because masking- baaed training methods applied in language modeling
intractable in images. Although SwAV is type of contrastive learning, it differs from its sisters (Chen et al. 2020; He et al. 2020 Wu et al. 2018) in that it is an online clustering-bast -based method. This means the model is trained assign cluster labels, or "code" vector to an input image with encoded latent features In particular, SwAV loss seeks to enforce consistency between codes of different augmentations (or views) of the same image, In other words, different augmentations of the same image should have almost the same semantic content as the original, and thus the cluster assignments of them should also be consistently the same.
Formally, let 21, E Rd be latent features of two distinct views X1, X2 of the same image X after passing them through an encoder fe, we compute the corresponding codes qı. 92 E [0, 1 by passing 21, 22 into the prototype layer, which consists of prototype weights C [cl,..., CK] with C E Rd. We then proceed to swap the codes Q Q2 and use them as labels for 22, 21, respectively:
SSwAv (Z1, 1(21, 42) + l(z2, 91)
The loss function 1(=, q) in Equation code q. and is defined as:
O measures the similarity distance between features and
1(2 (0.))==> = Σ qt logp", with x² exp($2T<ax) EN exp(= cw)
(2)
&
where 7 is temperature hyper-parameter and k is the index for e& row in C. Figure 2a illustrates how SwAV loss works. While the above formulations involve two different augmentations X1, X₂ of the same image for brevity purpose, multiple augmentations are used in practice. We refer the reader (Caron al. 2020) for further details on SwAV loss.
METHOD
In this section, we present our three main contributions: the language-agnostic constraint for SwAV loss or LAgSwAV (3.I). the pseudo-paralle data mining with filter suite ($3.2), and finally the rank- cross-entropy loss for UMT training with augmented data ($3.).
3

---

Published as a conference paper at ICLR 2022
English
Imbolanced
Balanoed
Proto
Kej 12)
NaswiN
2
Proto
41 I(, 92)
Proto
Kzj vi)
SWAV
Imbalanoed
Balanoed
22-
Proto
92
Kzz. 91)
Romanian
Proto
Kejo 1.l)
S
Proto
Kez. vp)
(a) Vanilla SwAV loss
(b) LAgSwAV loss
Figure 2: Comparison between vanilla SWAV (a and LAgSwAV 2bb losses. The "Proto" block represents the prototype layer C. Unlike SwAV, our LV loss differentiates which language sentence belongs to. As in Figure 2b] the cluster assignments of English sentences (i.e., qi g are skewed towards left clusters (blue histogram), and those of Romanian (i.e., qi a are skewed towards right clusters (red dotted histogram), causing imbalance. The Ω block rebalances them into more evenly distributed assignments of, Vg2 of and 2z
3.1 LANGUAGE-AGNOSSS CONSTRAINT FOR SWAV LOSS IN LANGUAGE MODEL
The SwAV loss was developed to conduct self-supervised training in vision due to infeasibility of applying masked language modeling (MLM) to images (Caron et al. 2020). Thus, for language model pre-training, we stick to MLM and use our proposed L.ASSAA to fine-tune pre-trained LM to achieve semantic contrastiveness and language agnosticism.
= [XI, Xp] and B [Y,,.., Yn] be the batches of B sentences from language 8 and t, respectively. We proceed to augment them into B [X], "i......J,,, X1]] and R B = - Y by various noising techniques. such as token swapping, masking and delet- ing. Next, denote Fo as the Transformer encoder (Conneau & Lample, 2019), which outputs the sentence embedding z given an input sentence. We compute the respective latent features z z Zt. z². R B for B . 13 13.2 using Fe, described in $2.3 Subse-
Specifically, in the presence of sentences from two languages, let B ======-
and 5 [Y,] ..... Y.. and
quently, we compute the code cluster assignments Q Q=2 Q Qt. E [0, 1] as:
1 Z 2. Zt,, 2 CT
Q Q²2 Qu: 2 sinkhorneexp
where [ is the www-wise concatenation function and sinkhorn( is the iterative Sinkhorn-Knopp algorithm (Cuturi 2013), a batch-wise renormalization process to compute the cluster assignment probabilities, and is syper-parameter to control the smoothness of the distribution.
The above formulation in fact indifferent from the vanilla SwAV loss (2.).. However, intuitively, if the model Fe is not aanguage-agnostic (i.e., and are distributionally separate), the resulting cluster assignments Qad and Q are likely to be distributed unevenly for and f. In other words, some clusters c; C' will be frequently assigned to the sentences of language but rarely to the sentences of t, and vice versa. Figure 2bb illustrates the imbalance phenomenon. We argue that is possible to enforce the language-agnostic property indirectly by fixing this imbalance phenomenon. Therefore, the objective of our proposed LAgSwAV is to measure the stochastic imbalance in clus- ter assignments Q* Qu and rebalance them into V..j Via E [0, 1 such that clusters in V*.,, Vtù are stochastically assigned evenly regardless of the language of the sentences. Then, we compute the total LAgSwAV loss as usual for both and & as:
CASSWAV. = 1(2,1) 0 2)= + 1(2, 0.2 ,0," C'gSSWAV. = [(x] (1a(",et?)) + 1(x) 1(212,0f1))
(4)
LLAgSWAY = ECASSWAN + Σ CAASSAAV.
(5)
We formalize the transformation from Q to V using the language-agnoste rebalancing function Ω which operates at the batch level to capture the statistics on cluster assignments. Specifi- cally, given the Q matrices specified above, we compute the per-language rebalance weight vectors un w K such that when multiplying them with prototype outputs, they try to suppress the
4

---

Published as a conference paper at ICLR 2022
magnitudes of frequently-assignedd clusters and promote those of rarely- ansly-assigned clusters, resulting in more balanced cluster assignments. More formally, the weight vectors are computed as:
2
f onehot ; 10
Σ
onehot :
(6)
K
w 1.0 (f a000 (f
1.0
where onehot( is row-wise one-hot function which returns a one-hot vector by turning the max- imum entry in the to 1.0 and the rest to 0.0. We then the weight vectors un w to compute the rebalanced cluster assignments V*J, ViJ E RBxK by modifying Equation 3 as:
(8) where a @ B element-wise multiplication operation of vector a applied each of B, and the rows Vs.j, v represent the v... , vectors in Equation 44. Note that there is gradient flow for the computation of w" and w', as well as the cluster assignments Q and V.
[V+,,; V Vil V*22] sinkhorneeppf [20" 1Z = w 1Z 2 (1CT)T
3.2 ASEUDO-PARALIER DATA MINING WITH FILTER SUITE
To mine pscudo-parallel data, we sample sentences from two monolingual corpora and and compute their cluster assignments using our LV model. The cluster assignment probabil- ities are treated as sentence embeddings in the next step to mine the data using the margin-based algorithm proposed by Artetxe & Schwenk (2019b). The resulting dataset is decreasingly sorted by the alignment scores. However, we empirically found that the obtained data are considerably noisy. Therefore, we propose filter suite, which is set of filtering criteria to filter out low-quality samples from the initial mined dataset, X = [ 3/1,,.., (mm- ymm]]. resulting in the final mined data X,_, [[11, 31), .... (In, yx)]. While details and motivations for these criteria given in the Appendix, we briefly introduce them as:
1. Minimum and maximum sentence lengths of Lmia and Lmux for both source and target.
2. Source-target length ratio μ with μ 1.0; i..,, j
(en(m:)
(en(
S p.
3. Subword overlap ratio "/ between source and target is larger than the corpus mean value, where
"t' is defined as len( 2: ++)=:)+ i.e., 7 > Σ 22.
len( (en(
4. Accept only top 9% of pairs based on the aligument scores. This is equivalent to threshold filtering (Artetxe & Schwenk 2019b), but it does not depend absolute threshold value.
5. UMT filtering with agreement BLEU > 3. Specifically, after training UMT model with the mined data from steps 1-4, use the model itself translate each source sentence of the mined data and measure the BLEU score between its translation and the target. We then filter out any pair with BLEU < B. This step drastically reduces the size of the mined dataset.
3.3 RANK-BASH BBSSE CROSS-ENTROPE ENTROPY FOR UMT INTEGRATION
It is trivial to compute a supervised loss on the mined pseudo-parallel data and add it to the standard UMT necktrransltionn loss. However, we found that (i) the mined data still contains, especially at the bottom of the ranked list, certain amount of misalignments, and (if) the data is too small (<100K sentence pairs) not cause overfitting. These issues motivate us to develop Rank-based Cross-Entropy loss, or Rank-XF, to be applied to the mined data. Specifically, given mined dataset x__ = (I1, (1),, (1), (En:yr)) which decreasingly sorted by their alignment scores, we assign a weight a; 1.0 - (i 1)/n for each pair (z,, y1). is designed to assign higher weights to the pairs at the top and lower to the bottom ones, which effectively reduces the influence of low-ranked pairs to the loss function. The loss for the mined (or augmented) data is then defined as:
LAUG Eo;; log Po (1(
which is then added to the standard iterative taacktrransltionn loss LBT to compute the total loss: L - Ler + LAuG
(10)
where the weight Aang is set to 1 at the beginning and gradually decreases to 0 avoid overfitting.
5

---

Published as a conference paper at ICLR 2022
Table BLEU scores on the bilingual WMT" English-French (En-Fr), WMT' English-German (En-De) and WMT 16 English-Romanian (En-Ro) UMT tasks. Note that mBART is incomparable as it was trained on multilingual CC25 data, 45 times large than the bilingual NewsCrawl data used in our experiments. The numbers in subscripts are the corresponding sacrebleu scores.
Method
En-Fr
Fr-En
En-De
De- En
En-Ro
Ro-En
NMT
(Lample ct al.
2018c
25.1
24.2
17.2
21.0
21.1
19.4
PBSM
[Lample
2018c
27.8
27.2
17.7
22.6
21.3
23.0
XLM
Conneau & Lamp Lample
ple
2019
33.4
33.3
26.4
34.3
33.3
31.8
MASS
(Song et al.
2019
37.536.4
34.934.5
28.328.3
35.235.1
35.235.3
33.132.7
CBD
guyen et al.
2021
38.237.2
35.535.1
30.1
36.336.2
36.335.4
33.813.5
mBART
Liu et al
2020
iincompurable
29.8
34.0
35.0
30.5
CRISS
ran
2020
incomparaable
38.3
36.3
32.
37.1
35.1
37.6
LAgSwAV (MASS)
40.2 39.0
37.6
29.5 29.7
36.8 36.7
37.0
34.9 34.7
4
EXPERIMENTS
In this section, we first report the performance of our method in comparison with the established baselines on the standard WMT and FLoRes low-resource UMT tasks. Then, we experimentally show how LAgSwAV is able to mine better pseudo-paree parallel data. We also compare our method with related approaches and conduct other analyses. To optimize for the most informative content within the limited page real estate, we leave some of experimental details to the Appendix.
4.1 WMT BILINGUAL UNSUPERVISED MACHINE TRANSLATION TASKS
Setup. For the WMT' 14 English-French (En-Frr, WMT 16 English-German (En-De) and WMT 16 English-Ron Romanian (En-Ro) bilingual UMT tasks, we follow the established predecessors (Lample
ct al. 2018c; Conneau & Lample, 2019; Song et al. 2019 Nguyen et al. 2021) to use only the monolingual data from 2007-2017 WMT News Crawl datasets of the two languages for each task.
Note that no data from a third language is used. We fine-tune the pre-trained XLMM (Conneau & Lample 2019) with proposed LAgSwAV loss and follow the process in $3.2 mine pseudo- parallel data, which are then used to augment MASS UMT models (Song et 2019)). Following all established previous work, we evaluate the models with tokenized BLEU mulrii-beu.pper script
(Koehn et al. 2007). In addition, we also report the corresponding sacrebleu scores (Post (2018) in the subscript for our method as well as the baselines that we reproduced in our experiments.
Results. From the results reported in Table we notice that our method outperforms MASS baseline (Song et al., 2019) by 2.7, 2.1, 1.2, 1.6, 1.8 and 1.8 BLEU for the WMT En-Fr, Fr-En, En-De, De-En, En-Ro and Ro-En, respectively. It also surpasses CBD) (Nguyen et al. (202]) by up BLEU 5 out of 6 language pairs. The results demonstrate that the aseudo-paralles data mined by our method is valuable for the UMT models.
4.2 Low- COW-RESOURCE UNSUPERVISED TRANSLATION
For the FLoRes low-resource UMT tasks Guzmánn et al. 2019). we follow the same setup mBART (Liu et al. 2020). Specifically, we reuse their published pre-trained model to fine-tune with our LAgSwAV loss and mine pseudo-parallel data. Table 2 reports the re- sults on Ne-En and Si-En for our method, compared with XLM (Conneau & Lample 2019). MASS (Song) al. 2019) and mBART (Liu et al. (220). It can be shown that our method outperforms mBART by up to
Table 2: Performance the low-
resource FLoRes Nepali-English (Ne-En) and Sinhala- Sinhala-English (Si-En) UMT tasks.
Method
Ne-En En-Ne Si-En En-Si
XLM
0.5
0.1
0.1
0.1
MASS
0.7
0.3
0.4
0.1
mBART
10.0
4.4
8.2
3.9
LAgSwAV
12.8
5.3
9.4
5.4
2.8 BLEU in these UMT tasks.
4.3 UNDERSTANDING LANGUAGE AGNOSTIC CONSTRAINT FOR SWAV LOSS
To understand how LV behaves in terms of the aforementioned language agnostic and mantically contrastive properties, we conduct a series of evaluations with manual visualization as well as automatic metrics measure the focused properties.

---

Published as a conference paper at ICLR 2022
Er-Fr
Em-D
XLM
(b) XLM+SWAV
(c) XLM+LAgSWAV
Figure 3: T-SNE visualizations of sentence embeddings produced by XLM, XLM+SWAV and XLM+LAgSWAV for En-Fr and En-De. Visualizations for En-Ro is given in the Appendix.
4.3.1 T-SNE VISUALIZATIONS
We sample 50K sentences of both languages from a held-out monolingual data for each language pair and use T-SNE (Van der Maaten & Hinton 2008) to visualize the sentence embeddings pro- duced by XLM, XLM fine-tuned with SWAV (XLM+SwAV) and XLM fine-tuned with LAgSwAV (XLM+LAgSWAV), From the visualizations shown in Figure 3 it can be seen that the English em- beddings (red) are scattered separately and distinguishable from those of French and German (cyan) in XLM and SwAV XLM. Among them, the visualizations for German fare better than those of French. However, after fine-tuning with LA loss, the sentence embeddings are mixed indis- tinguishably, indicating a high degree of language-agnosticism.
4.3.2 QUANTITATIVE MEASURES
Although $4.3.I provides qualitative sense of how the models exhibit in terms of the two examined properties, necessary measure the properties in quantifiable manner. We propose three metrics to evaluate the models: Tatoeba Accuracy, Global Accuracy and Cluster Word Distribution. We compare different models on these metrics in Table In the Appendix, we also provide sentence samples of different clusters to show that they are semantically related.
Tatoeba Accuracy. Artetxe & Schwenk (2019a) proposed use parts of the Tatoeba corpus as test sets for the Tatoeba similarity search task, which can be used to measure how good model is at mining parallel data. Specifically, for each source sentence, we use cosine similarity to search for the closest target sentence based on the sentence embeddings to check if the obtained target sentence is indeed the correct target the given source, from which we compute the Tatocba accuracy score.
Global Accuracy. This metric uses validation set of parallel sentences to measure how language- agnostic the models are, based on the perception that non-agnostic models exhibit a separation of latent spaces for different languages. Specifically, if there is separation by language, each sentence of language+ s would be closer to all other sentence of the same language than any sentence of language L. Therefore, search for the target sentence in search space that includes not just the target sentences, but both source and target sentences combined. If the model non-agnostic, the search will always pick incorrect intra-lingual sentences, driving down the accuracy significantly.
measure Global Accuracy in the same way Tatoeba except that
Word Distribution Score. This metric uses a held-out monolingual data measure how semanti- cally contrastive the models are in terms of content and topics. One of the ways to measure this is to detect certain words that appear frequently in only one cluster and rarely in others. Intuitively, if a cluster contains certain key-words dominantly, it is likely the sentences in that cluster are of similar semantic topic or content. From this notion, the word distribution score computed based on the unique word appearance frequency for each cluster. Further details are given in the Appendix.
Results. Table 3 shows the results in terms of the three metrics for the XLM (baseline), XLM+SwAV, and XLM+LAgSWAV models along with their respective BLEU scores achieved us- ing the mined data obtained by the models after going through the Filter Suite ($3.2). We observe consistent improvements across all three measures for our LAgSwAV model. In particular, the global accuracy scores reflect closely the degree of wagguage-agnosticiin as observed in Figure 3 For example, the global accuracy values for French and Romanian particularly low for the base-
7

---

Published as a conference paper at ICLR 2022
Table 3: Tatoeba, Global Accuracy (GlobAce) and Word Distribution scores (WordDis) of baseline XLM and XLM fine-tuned with SwAV and LAgSwAV. We also include the corresponding down- stream UMT performance (BLEU) of models trained with the respective augmentation data.
En-Fr
En-De
En-Ro
Model
Tatoeba GlobAce WordDis BLEU Tatoeba GlobAcc WordDis BLEU Tatoeba GlobAce WordDis BLEU
Unsupervised models
Baseline
30.5
25.6
0.30
35.9
59.1
74.1
0.35
25.5
11.5
13.2
0.22
34.1
SWAV
34,2
30.4
0.28
36.1
62.8
76.2
0.34
26.1
11.9
24.3
0.18
34.0
LAgSWAV 91.2
88.2
0.38
40.2
93.0
87.2
0.39
29.5
80.3
84.6
0.30
37.0
Supervised models
LASER
96.2
93.8
0.43
38.3
99.1
93.9
0,46
28.2
98.0
96.8
0.35
33.3
Table 4: Comparison of our method LAgSwAV versus supervised LASER (Artetxe Schwenk, 2019a), along with the study the effect of different components of our method: Rank-based Cross Entropy (Rank-XE) and Dynamic Xnn ($33) and Filter Suite ($3.).
Method
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
Extract-Edit (Wu et al.
2019a
27.6
26.9
19.5
23.3
23.3
21.6
Weakly-paird Does Wu 2019b
24.2
30.3
30.1
27.6
MASS (Song ct al. 2019
37.5
34.9
28.3
35.2
35.2
33.1
LASER (Artetxe Schwenk 2019a
36.2
34.5
25.6
27.5
26.5
+ LASER + Rank Bank-XEB & Auug
37.4
35.0
27.0
33.6
32.9
32.0
LASER Rank-XE & Amng Filter Suite
38.3
35.7
28.2
34.1
36.0
33.3
MASS
+ LAgSwAV Only
35.9
34.7
26.0
31.2
26.9
26.4
AV Rank-XE K & Aung
37.5
35.0
28.0
35,0
35.4
33.0
LAgSwAV Rank-XE& & Auug Filter Suite
40.2
37.6
29.5
36.8
37.0
34.9
lines, while that German is already much higher. These improvements also translate well to the performance of our LAgSwAV AV models on the downstream UMT tasks.
4.4 COMPARISON AND ABLATION STUDY
As unsupervised bitext mining method, it is noteworthy to compare our method with existing methods (Wu et al., 2019a b). We also report the results for LASER (Artetxe & Schwenk 2019a). Note that LASER is not directly comparable to our setup it was trained on parallel data from 93 languages. Specifically, we use the pre-trained LASER model to mine sauudd--aallllll data from the same monolingual sources used in LAgSwAV. We then use the mined data to augment UMT models with and without the Filter Suite ($3.2) and rank- based loss ($33)) Plus, we also conducted ablation study examine the contributions of different components of our method. We train the UMT models with the mined data from our LAgSwAV AV models without any filtering and rank-based loss, and then subsequently add these components to how they contribute the performance.
Ablation Study. From the results in Table it is noticeable that without a sufficient filtering process, the mined data by LASER as well LAgSwAV cause significant performance drops across all language pairs. This is because the majority of sentence pairs ranked at the bottom of the mined data are noisy, causing incorrect loss terms. The performances gradually improve as add rank- based cross entropy and dynamic Asng weight to the framework, which essentially reduce the impact of low-ranked pairs. However. the most performance jumps are observed when we add the Filter Suite, where we considerably filter the mined data and retain the top R 5% of the mined data. These observations are true for both LASER and our LAgSwAV fine-tuned XLMM models.
Comparison with LASER. A more surprising observation we have from the results in Table 4 that LASER underperforms our method, even with the Filter Suite. This is despite the fact that LASER was trained on much larger parallel data from 93 languages. After investigation, we discover that LASER does not differentiate languages and the monolingual data contains sentences from the other language. For instance, the English corpus contains some French sentences arti- facts; LASER simply picks these artifacts up and pair them with their identical matches found in the French corpus. Since these Fr-Fr pairs almost identical, they (wrongly) ranked top of
8

---

Published as a conference paper at ICLR 2022
Table 5: Agreement BLEU between the targets of the augmentation data mined by different pre- trained models and their translations produced by fixed baseline MT model. Higher scores means the targets are close to the sources according to the baseline model.
Method
En-Fr
Fr-En
En-De
De-En
En-Ro
Ro-En
XLM
3.5
2.4
1.5
1.4
2.3
1.8
XLM + SWAV
3.4
1.7
1.9
1.3
2.5
2.0
LASER
24.3
24.7
13.4
15.6
17.5
17.6
XLM + LAgSwAV
37.9
36.9
21.2
24.4
26.3
27.4
the mined dataset. Our LAgSwAV, meanwhile, differentiates languages thanks XLM's language embedding layer. Therefore, when it encounters French sentence that is 'marked' as English, the model will infer it as erratic and avoid selecting during mining. Table 3 this observation.
4.5 FURTHER ANALYSIS
Quality of Mined Data. To gain further insights into the quality of the augmentation dataset mined by our method, we use a supervised NMT model (Ott et al. 2018) to translate the source side of the mined data and compute the agreement BLEU scores between its translations and the dataset's target side. The higher the agreement BLEU, the more likely the source and target of the mined dataset are semantically and meaningfully aligned, according to the NMT model. We use this metric compare the quality of mined datasets produced by XLM, supervised LASER (Artetxe & Schwenk 2019a), AV and XLM+LAgSWAV As shown in Table 5 XLM and XLM+SWAV fail completely at mining pseudo-parallel data as their agreement BLEU scores are significantly low. Meanwhile, our method outperforms even supervised LASER, which aligns well with the results in Table
Percentage p%. Among the hyper-parameteri specified in the Filter Suite 3.22 we found that the p% plays the paramount role. As shown in Figure for the WMT En-Ro task, we found only the top 10% of the mined data is valuable to improve UMT performance. As we gradually include more data from the bottom of the mined data by increasing p. the performance degrades substantially.
BLEU
36.5 $
LAgSwAV
En-De
Baseline
LAgSwAV
Baseline
35.5
X + Y* 24.9
26.3
35
X Y
25.5
28.5
510
20
30
40
60
80
90
100
X + 32.9
35.5
Figure 4: Performances of our method on En-Ro
Table 6: Translationese analysis En-De unsu-
w.r.t filtering p%.
pervised MT models
Translationese Effect.. Edunov et al. (0220) found that back-translation only benefits translationese- source sentences but does not improve for natural-
texts. To investigate whether our method is affected by the translationese effect, we use the test sets provided by [Edunov et al. (2020) and repli- cate the translationese analysis in three setups: (f) Natural-source - Translationese- targee X*),)), (ii) Translationese-sourc > Natural-target (X*+YY) and (ii) Translationese
translationese of source Translationese- target +Y*). The results in Table 6 show that our method improves the performance in De-En and En-De UMT tasks for both natural-source and translationese-soure setups. This demonstrates that method is not affected by the translationese effect.
CONCLUSION
We have proposed a new language-agnostic SwAV loss, which is used to fine-tune pre-trained lan- guage models to improve their ability to mine pseudo-parallel data in an unsupervised manner. We have also introduced additional modules, such as Filter Suite and dynamic loss weight, improve the performance of UMT systems. The method achieves significant improvements over the baselines in the WMT English-French. German-Engie English and English- Romanian bilingual UMT tasks. It also outperforms mBART in the FLoRes Nepali-English and Sinhala- Englishh UMT tasks. Our analysis also shows that the method outperforms the baselines in mining pseudo-parallel data.
English-Romamian

---

Published as a conference paper at ICLR 2022
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics. The paper presents research on unsupervised machine translation where experiments were conducted on publicly available datasets, such as the NewsCrawl and CommonCrawl datasets, which have been used in various established publications. Thus, believe the research and experi- ments do not contain any questionable concern regarding bias, discrimination, fairesss, privacy, etc. Nonetheless, in case of application production systems, we advise practitioners use solutions with caution, as our mining method may still align source-target pairs that are close but exact, such as "I am from the US" versus from California", These inexact pairs may then influence the downstream translation model to make incorrect predictions.
Reproducibility. Apart from the experimental explanations in the main paper, Appendix ap- pendix | provides further experimental details on the WMT and FLoRes unsupervised translation tasks, as well as other analysis experiments. In addition, the source codes provided above contain further technical instructions and code explanations to reproduce the results.
ACKNOWLEDGMENTS
We deeply appreciate the efforts of our anonymous reviewers and meta-reviewer in examining and giving us feedback on our paper. We also thank Juan Pino, Paco Guzman, Naman Goyal, Chau Tran and Xian Li from Meta Al for their valuable advice and feedback for our work.
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingun transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, March 2019a. doi: 10.1162/tacl 00288. URL /ttphtttps://accaattooool ooyy org/ 019-1038
Mikel Artetxe and Holger Schwenk. Margin-based parallel corpus mining with multilingual sen- tence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 3197-3203, Florence, Italy, July 2019b. Association for Compu- tational Linguistics. doi: 10.18653/11/19-11000 URL bttps://cclanthooogy org
219-1309
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representations, 2018. URL https:
//openreview. (lopee/eeveee.nst/formieeooeeeee wttp://eeeeooommoooss/eeeeeeeeee
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Mara Chinea-Ríos, Álvaro Peris, and Francisco Casacuberta. Adapting neural machine translation with parallel synthetic data. In Proceedings the Second Conference on Machine Translation, pp- 138-147, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 11118655/1//11744744 htR.hhttps:://ccaaaa ogy.org//1111117010
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'lché Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7069-7069. Curran Associates, Inc., 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
10

---

Published as a conference paper at ICLR 2022
Sergey Edunov, Myle Ott, Mare' Aurelio Ranzato, and Michael Auli. On the evaluation of machine translation systems trained with back translation. In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics, pp. 2836-2846, Online, July 2020. Association URL https://w//t
for Computational Linguisties. doi: 10.
acl helleeb. wee..rrg/attee antho http///tttoollyyyy////// accl
253
Xavier Garcia, Aditya Siddhant, Orhan Firat, and Ankur Parikh. Harnessing multilinguality in unsupervised machine translation for rare languages. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 1126-1137, Online, June 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/200 wtw2222..naacllannna .nacc1-main88)
URL https: ttttps:/aalaanthllogooo wtwpogyy.rgg/220..
naacl-ma in. 89
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino. Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. The FL.ORRS evaluation datasets for low- resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6098-6111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ D19-1632,
URL https: www aclweb. org/ anthhlooyoooo--111111
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9978, 2020.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceed- ings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 388- 388- 395, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:
//aclanthologe www.errg///oo WD4 3250
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, al. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the as- sociation for computational linguistics companion volume proceedings of the demo and poster sessions, pp. 177-180, 2007.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.0663 062266 2018.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 201 8a. /RL/itttss:///peeereeet wttp:viwwwweeeeeeeeeeeeeeeeeeeee nett/forum2i==xXT11
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018b. URL URRhhtts:///ppeeeee/ ittp:errreeeeeeeeeteooooooiiiiaa
net www.fooruuruuioooo
96sains
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 5039-5049, Brussels, Bel- gium, October-November 2018c. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1548 1549. URL httpt::ts////...tt.. www aclweb wwt.:/wwwwww...ttoooooooeeeeeeee wttp///nthoolyy//1-1155555
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.088 .0.0000..82210 10, 2020.
Xuan-Ph Phi Nguyen. Shafiq Joty, Thanh- Tung Nguyen, Wu Kui, and Ai Ti Aw. Cross-model back- translated distillation for unsupervised machine translation. In 38th International Conference on Machine Learning, 2021.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT), 2018.
11

---

Published as a conference paper at ICLR 2022
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, Pp. 186-191, Brussels, Belgium, Octo- ber 2018. Association for Computational Linguistics. doi: 10.18653/11/118-6331 URL
httpss: / wwwwwwwwooom ac wwb. wag.atooooooyyy/a.o web.orrg/atthhoogg///////
8 1
Holger Schwenk. Filtering and mining parallel data in joint multilingual space. arXiv preprint arXiv:1805.09822 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 86-96, Berlin, Germany, Au- gust 2016a. Association for Computational Linguistics. doi: 1.18653/v1/P16-1000 URL https: www wwwwwb.wwa......... org/ httpa/aaooooyy/o-11//0/0/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 1715-1725. Association for Computational Linguis- tics, 2016b. doi: 10.18653/v1/P16-1111 11622. URL http: www. aclweb. org wwwggantgtoggyy.ooooooooooooooot P1 1163 62
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence sequence pre-training for language generation. arXiv preprint arXiv: 1eX5:v:905.024555 /90500240,, 2019.
Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. Cross-lingual retrieval for iterative self-supervised training. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 2207-2219. Curran Asso- ciates, Inc., 2020. URL https: /ttpps://proceeddnngg htwp://eeeeiiio..eeiiis.e.ssssss neurips........... cc/paper/2220/Eiee/ 1763ea5a7e 7 wtt.aa/eeeeeeeeeee333222........ 2ttp:4/073-22222/////////--
pdf
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Sergei Vassilvitskii and David Arthur. k-means++: The advantages of careful seeding. In Proceed- ings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pp. 1027-1035, 2006.
Jiawei Wu, Xin Wang, and William Yang Wang. Extract and edit: An alternative to back-translatien for unsupervised neural machine translation. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 1173-1183, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. 00.18653/1/NN1--1120 URL
log.. 1arg/N11911222
Lijun Wu, Jinhua Zhu, Di He, Fei Gao, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Machine translation with weakly paired documents. In Proceedings of the 2019 Conference Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 4375-4388 Hong Kong, China, Novem- ber 2019b. Association for Computational Linguistics. doi: 10.18653/1//D19-1444 URL
httts::/ htt////aaaaattohoooo org/D19-- 1446
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of / Ith Workshop on Building and Using Comparable Corpora, pp. 39-42, 2018.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
In the Appendix, we provide further details and explanations about the inner-workings of our method. Plus, we also show experimental configurations as well other supplementary ex- periments.
A.1 FURTHER EXPLANATION FOR OUR METHOD
In this section, we first explain the Sinkhorn algorithm and more details about the Filter Suite that we use to filter pseudo-paralle data mined from our LAgSwAV model. Then, we lay out the complete procedure used to integrate the mined data with standard back-translation for unsupervised machine translation models.
A.1.1 LANGUAGE LANGUAGE-AGNOSTIC AGNOSTIC CONSTRAINT FOR SWAV LOss IN LANGUAGE MODEL
In addition to the details explained in the main paper, we illustrate the batch-wise sinkhorn algorithm (Cuturi 2013) Algorithm| Further details about this procedure and the SwAV loss are also provided in Caron et al. (2020).
Algorithm Sinkhorn: Given matrix Z BBxK which represents the after-exponential latent representations batches of samples, and n number of iterations; return the sinkhorn prototype output Q _xx
1: Q - z"
2: ++ E,E,in
3: U.R.C zeros( ( ones(K)/K ones(
4: for 0...n do
5:
U U+ Q
6:
Q Q+
(R/U)
7: Q Q+
Q (C/C,, Qi.))
8: end for
9:
(0,
10: return Q
A. A.1.2 PSEUDO-PARALLEL DATA MINING WITH FILTER SUITE
In addition to explanation given in the main paper, we further clarify certain note-worthy details of the Filter Suite. Firstly, we sample sentences from two monolingual corpora and and compute their cluster assignments using our LAgSwAV model with soflmax operation instead of sinkhorn because is unreliable to conduct batch-wise operation during inference. Specifically, we compute the mining sentence embeddings ex of given sentence X of any language
ex softmax C Fo()))
(11)
is then used by the margin-based algorithm proposed by Artetxe & Schwenk (2019b) to mine pseudo-parallel data. Secondly, as the proposed Filter Suite consists of a set of different filtering criteria,, we specify the motivations for each of them follow:
1. Minimum and maximum sentence lengths of Lmin and Lmax: short sentences may not be useful, while long sentences may be more likely mis-aligned.
2. Source-target length ratio μr: source and target sentences should be of comparable length, the variation may depend on the specific language pair.
3. Subword overlap ratio 7 between source and target: for relatively close language pairs (En, Fr, De, Ro), parallel sentence pairs of the same meanings are likely to share certain amount of subword: so we accept aligned pairs whose 7/8 is higher than the corpus average. In addition, also cap this value up to maximum, such 0.35, to avoid the exact match scenario, where source and target are exactly the same. Note that we do not use this criterion for low-resource languages, such as Nepali and Sinhala.
13

---

Published as a conference paper at ICLR 2022
English
Imbalanced
Balanced
Proto
gi
Vi Inl
Mzjss 15)
Transformer
nolog
Encoder
AAgSWAV
Proto
Mezz vi)
Imbalanoed
Romanian
noioe
Proto
Kzj, 1g)
Y
noloe
Transformer
Ω
CAgsWAY
Encoder
Proto
1[2g, vi)
Figure 5: Complete diagram for our LAgSwAV. The diagram shows the paths each English (X) and Romanian (Y) sentences take compute the final LAgSwAV loss.
4. Accept only top p% of pairs: sentence pairs are ranked based on alignment scores, as a result, pairs with low scores are ranked at the bottom and are often low-quality. Thus, we filter them out.
5. UMT filtering: once the UMT model has been trained to be competent enough, it can act as filterer too. Therefore, we it to translate the mined data verify whether the source and target are sufficiently aligned.
A.1.3 RANK-BASER CROSS-ENTROS FOR UMT INTEGRATION
In addition to the total loss L formulation for the rank-based cross-entropy loss and the UMT in- tegration strategy in the main paper, we further provide the overall UMT integration of the mined dataset procedures in Algorithm
Algorithm Augmented Iterative Back-translltion: Given monolingual data X, of languages s, pseudo parallel data x,- pre-trained model Å. Target side omitted for clarity.
i
2: while until convergence do
3: Sample batches X, from X
4: Sample batches (X, Y) from X,- where X = [r1,..., Fn), Y [V1, .... Vn
5: LBT - log Po X2 (ta[ttaallaee(XX.))
Standard BT loss)
6: LAUG E. -;, log Po(vi(z;;
{ (Augmentation loss)
7:
L LBT + Xang LAUG
8:
0 8 VoLo
9: end while
return
A.2 COMPLETE LAGSWAV MODEL
Figure 5 illustrates the complete structure of our LAgSwAV model. Specifically, on the left side, we have input data, which are English (X) and Romanian (Y) sentences. These inputs perturbed into four different augmentations (X,, X2, Yi and Y2). These augmentations are then passed to the Transformer encoder and then the prototype layer, well as the rebalancing block Ω to compute the final LV loss.
A.3 EXPERIMENTAL DETAILS
WMT bilingual unsupervised tasks. For the WMT' 14 English-French (En-Fr), WMT' 16 English- Genman (En-De) and WMT' 16 English-Ronn Romanian (En-Ro -o) bilingual unsupervised translation tasks, follow established predecessors (Lample al. 2018c Conneau & Lample) 2019, Song al. 2019 Nguyen et al., 2021) to use the all the monolingual data from 2007-2017 WMT News Crawl datasets of the two languages for each task, which include 190M, 78M, 309M and 3M sentences
14

---

Published as a conference paper at ICLR 2022
for English (En), French (Fr), German (De) and Romanian (Ro) respectively. We fine-tune the pre- trained XLM (Conneau & Lample 2019) provided by the authors with our proposed LAgSwAV loss. For each language pair, we also use the respective provided bilingual dictionary of 60K sub- word units (Sennrich et af.] 2016b). Regarding the mining process, we sample up to 20M sentences from the monolingual data explained ahove for each language and our LAgSwAV model to mine pseudo-parallel data with Filter Suite explained in $3.2 We set mininum, maximum lengths of Lmio 5 and Lmax = 300; source/target length ratio μ 1.5; maximum overlap ratio Ti 0.35 and accept only the top p = 5% of mined pairs. The agreement BLEU threshold is 3 = 30 We use the mined data to augment the MASS (Song et al.] 2019) UMT models. Following all established previous work, we evaluate the models using tokenized BLEU with Moses mulii-bleu, (Kohhn et al. 2007). addition, we also report corresponding sacrebleu scores (Post 2018 in the subscript of Table for our method as well as baselines that we can reproduce.
script
In addition, we also use bootstrap sampling (Koehn) 2004) to test for statistical significance for the WMT bilingual unsupervised tasks, As a result, round that our method's improvements are significant with p-value < 0.05 for En-Fr, En-De, and En-Ro (both directions). Plus, we also ran our experiments 3 times and found the standard deviations are approximately 0.1, 0.2, 0.1, 0.3, 0.2, and 0.3 BLEU for En-Fr, Fr-En, En-De, De-En, En-Ro and Ro-En, respectively.
FLoRes low-resource tasks. For the FLoRes low-resouree unsupervised translation tasks (Guzmán et al. 2019). we follow the same setup as mBART (Liu et al. 2020). Specifically, we reuse their published pre-trained mBART model to fine-tune LAgSwAV loss well as initial model for UMT back-translation training. We also use the same sentencepiece tokenizer model (Kudo & Richardson 2018) as mBART. We sample up to 20M sentences of the English, Nepali and Sinhala corpora of the CC25 pre-training data to mine pseudo-parallel data and train the UMT model. To avoid high-rank mis-alignments in the mined data, we filter the mining monolingual data to include only sentences with unicode Ap [IInevvanagari and IInIinhhala) word counts more than 80% the sentence length for Nepali and Sinhala respectively
T-SNE Visualization. We sample 50K sentences of both languages from a held-out monolin- gual data for each language pair and use T-SNE (Van der Maaten & Hinton 2008) to visualize the sentence embeddings of those sentences produced by the baseline XLM (encoder output of '[eos]' token) and the Sw as well as LAgSwAV fine-tuned XLM models (softmax representations of pro- totype outputs softmax( CT Z)). These visualizations are shown in Figure 6 As it can be seen, the sentence embeddings of English sentences (red) are scattered separately and distinguishable from those of French, German and Romanian (cyan) in XLM and SWAV XLM. Among these, the visu- alizations for German fare better than French's and Romanian's. ss However, after fine-tuning with LAgSwAV loss, the sentence embeddings are mixed indistinguishably, indicating a high degree of language-agnostic property.
Tatoeba Accuracy. Artetxe & Schwenk (2019a) proposed use parts of the Tatoeba corpus as test sets for the Tatoeba similarity search task, which can be used to measure how good a model is at mining parallel data. Specifically, for each source sentence, we use cosine similarity to search for the closest target sentence based on the sentence embeddings to check if the obtained target sen- tence is indeed the correct target of the given source, from which we compute the Tatoeba accuracy score. In other words, given a parallel test set [(X1, Yi), (X...., Ya)] and a nearest neighbor search function / operating on the sentence embeddings of the sentences, we compute the Tatoeba accuracy as:
ACCTatoeba
1 2n
Σ (V =Y]] + Σ
X.I)
(12)
Global Accuracy. This metric also uses a valid parallel set to measure how lang-agnostic the models are, based on the perception that aon-agnottic models exhibit separation of latent spaces for different languages, like in Figure 3 Specifically, if there is a separation by language, each sentence of language s would be closer to all other sentence of the same language than any sentence of the other language t. Therefore, we measure Global Accuracy in the same way as Alignment Accuracy, except that we conduct search for the target sentence among a search space of not just the target side, but both source and target side combined. If the model is non-agnostic, the search will always
IpIinDevanagari and Ap [PIInSinaala] are unicode blocks dedicated for characters in Nepali and Sinhala.
15

---

Published as a conference paper at ICLR 2022
NAA VET
(a) En-Fr
(b) En-De
(c) En-Ro
Figure 6: T-SNE visualizations of XLM, XLM+SWAV and XLM+LAgSWAV for En-Fr, En-De and En-Ro. For each chart, we sample 50K sentences from the respective dataset and compute T-SNE on the embeddings produced by the models.
pick incorrect intra-lingual sentences, driving down the accuracy significantly. Formally, with the same notations as above, Global Accuracy is computed as:
AcCglob
Σ 1I Y1 1). =Y] Σ 1[V [X1.... .i....)) X,)) (13)
2n
Word Distribution Score. This metric uses held-out monolingual data measure how semanti- cally classifiable the models are in terms content and topics. One of the ways to measure this is detect certain words that appear frequently in only one cluster and rarely in others. The intuition is that if a cluster contains certain key-words dominantly, it is likely the sentences in such cluster of similar semantic topic content. Specifically, given same data sampled in $4.3.1 we use the K-means clustering (Vassilvitskii & Arthur (006) to cluster uhthe sentences into N = 20 clus- ters. Then, for each unique word w; V in the data, we count its appearance frequency Fúl for each cluster j. After divide it by the total occurance of the word w; in all clusters and compute the normalized mean scores for each word. Formally, the Word Distribution Score computed as:
WoDis
(FL
Σ F
(14)
N
W.EV
Percentage 5% and Translationese effect.
In addition to analyses of percentage p% for En-Ro and translationese effect for En-De, we also report the similar analyses for Ro-En and De-En, respectively. The results are shown Figure we observe that the performance of our method for unsupervised Ro En task reduces gradually with the percentage p%. As shown similarly in Table 7 our method is also not affected by the translationese effect for De-En direction.
BLEUU
LAgSwAV Bascline
De-En
Baseline
LAgSwAV
33
X + Y*
24.7
25.2
X - Y
34.2
38.5
5
20
30
40
50
60
70
80
90
100
x** + Y*
32.0
32.6
Figure 7: Performances of method Ro-En w.r.t filtering p%.
Table 7: Translationese analysis De-En unsu- pervised MT models
16

---

Published as a conference paper at ICLR 2022
A.3.1 CLUSTER SAMPLES
Semantic contrastiveness reflects the model's ability classify sentences of similar topies and con- tents into the same cluster group. To demonstrate this capability of our LAgSwAV model, we ran- domly sample monolingual sentences from the NewsCrawl English-Romanian Romanian corpora and compute their cluster assignments using our LAgSwAV model. Then, we use a simple K-means clustering method (Vassilvitskii Arthur, 2006) to cluster the sentences into clusters. After that, we extract randomly some sentences from these clusters and report them in Table g As it can be seen in the next page, the sentences in each cluster are closely related semantically. Specifically, sentences of cluster talk about football, while those of cluster 5 talk about airlines and aviation.
17

---

Published as a conference paper at ICLR 2022
Table 8: Sample English sentences grouped by a K-means clustering method applying on their cluster assignments produced by our LAgSwAV model.
Closter
Senteners
Mallorca's Femaado Navarro and Liverpool defender Alvaro Arbelca are also included the lirst time. but there so place the squad veteran
Real Madrid forward Raol.
The former Bolton boss explained that a sepurate groin injury would already miee Owen out for three gumes, mouning he would miss England 's two
upcoming Earopean qualifying gimes.
David Nugent was the only substitue to score during his entire tenure with England.
It 's litll bit more acceptable if well Loch Loasond.
back from down away from home derby yame and get draw.
always
might ? 1892282017
try something differeut, " said.
This article hopp 7/ theseatsman seatamin
cousse know that they have good individual players Some have socused him of acting in desperation after two lacklustre perforaacces against Namibia and Georgis victories by 32-17 and 14-10 respectively- -pt the pressure on Ireland
""I "'e playal together with Cristiane one and half years in club foodball,
TThe Edinburgh outfit missed the upportunity
top of the Clydesdale Bank Premier Leugue last weekend when Motherwell ell brought their impressive
sequence without loss to and.
United have been
this year competition and with impressive depth. look better- placod European football yet unlikely they will do without Romaldo reprduucing his
top prize than at any time
their previous successs
cague
form in continental competition.
The flodd sinuation his wer unreenod, Assam state 's relief minister Rhumidhar Raman said, addiing thousands of villages had been inundiated
and 20 of 27 state distrees had been affacted.
Taiwan stopped the milttary parades 16 years ago as sought to improve relttions with communist ruled China following the island 's tronsformation to demeeracy..
Relatives of the 2,000 member 13th MEU, most of whom have known mure than munth that the unit coming home. are collectively bit confused the inclasion the 13th MEU in annouceeeen tropp called General Musharraf eattier the week, the Pakistuni leader ansounzed that national elections would be held by Feb. 15,
angry.
After Presideat month laier than plansed
CAIRO. Egypt [ (AP) ) Osama, bis Laden wamed Ira 's Sanni Arabs against lighting al Quida and vowed to expund the terur group 's holy war to 2. Isrsel in new uudiooaae Saturday, threatening" " bloud for blood destruction for destruction.
Palestinian leaders are increasingly promooting the idea of swap provided they comparable land in Lsmell even though implies recogninion that large lewish semrements will romain in place
EIT in Tiasukia
east of the
city Gowahat when bomb Hindu temple killed three people offieer Bhoskar Jyoti
Mahunia said.
Thesday was carried out by -oottnllll Tragi army unit at Restamiyah milliury seidemy.
Syed Munawar Hasaa. a leader of the Islamist Muitshida Majlis-e-Ami ur United Acticn Forum. said its lawmakers would resign from the National Assembly Parliament 's lower house, but were keeping their options open until after the Supreme Coust verdict on whether to resign from
the four provincial assenablies.
RRppoeers without Boeders usid Suleiman, in letters sent from prison, had complained of being handeuffed and besten then put into on isnlation cell where received very little food water
KKelly Oshoume made triumphant debut Meeday night prisce matron Mana Manton.
that dosinntted Hollywood 50 years ago bui today is mide only when podigreed actor like Hirris (or Kevin Costner Tommy Lee Jones ) insiste on it.
The thing about the picture. which Hamis also directed co-wrote. is respect for the streagths of a
Pianist Brian Kellock will make several uppearances- with singer Shella Jurdan (1 November ). in his BK3 collaboration with New York Chris
Lighscup and Malt Wilson 25 October) and bringing us neall back to Denmurk- with guiaa maesiro Jacob Fischer (28 November ).
NEW YOKK AP )- Dragg queens comival kings Big Bird and John Travolta are all a day work Jon Coles, whose company has cuttitted all them with feathers.
-Minttes before Rest Comedy announced,
Rock " Alse Baldwin- who inexplicably lost Best Actor Ricky Gerenis for his shor-livedd ducked past to get cigurelle outside the theaser back door.
IIBO comedy" Extras
-( And why the wig from the 1940 for play set in 1974.
Says the theater: It s genderless story with multi generational appeal : plus, it 's nice and carkk . just time for Hallowwen. $15 $20 students. $10
and the acting for that mattr- bring to mind Vincent Price and Victorian-era full moons... The Massnchusetts-ing hom Goulet who spent much his youth in Canada, gained stanlom in 1960 with " "Camolor ," "hee Lemer and Loewe musical
The
starned Richard
Arthur Julie Andhows his Guenevvre.
Montemezz Incantesimo. "the second of one act operas on fimame-sellnng thestes performed concert at Avery Fisher Hall on Tuesday evening. but was appizently not the cards.
popular suprano Aprile Millo was scholuled to appearin
But don 'I wail lou long to gel stanee:: : The protection didn 'kick in until the women had cuten less fat for four years and couning.
Bised on our experienes. the fict he moving so well. so carly after such a estastrophic injory meins he will walk agsin, ." "sid Dr. Borth Green chaimmnn of the depurtment of neurologacal surgery at the University of Miami school of medicine..
Saavia 's role brain chemistry therefore neads more research, Mr Schitano
Variants of DIAJA crank out high low levels Hap
and suthor Mario Formazi Purdue University said catechins are relutively unstable non-seidic envinements, such the intestines, and less
than 20 percent the sotal ramains after digestion..
And De. Berelowizz said the study had been accepted by the British Journal of Corliology and would soon be published.
hh Company 's first thraansttic product is PF-401, a 'Super EPO "( "erythropoictnn dimer protein drag for the treatmeat of anemia in renal dialysis patienis ( with end stage renal discase ).
Indeed, Dr Mottin has already used the New Zealand group to show how violent family uphringing and different versions of another neurologically
important gime interact proluce
and less violen people.
's
Andrew Jacono, reconstrsctvve plasic sargoon North Shone University Houpital
We must
the whole patieni experience including psychological support which putient often need.
LLOUSS. Oct. 19/ PRNewssire FiruCall/ The Boeing Company (YSEE RA ) today annoumced that Uuh will henefit from an extimated
direct and indirect jobs it selected to build the U.S. Air Furce 5 new lanker aircralt fleet.
Entroprenzur Sir Richard Bransom is backing a bid to break the world recurd for cincomnavigasing the globe a powerbust run om 100 renewable
fuel.
To fuel and other
Maxjet selects its departure dates and times masimize the number of passengers, company officials said.
The Delta pllor made nosedive and missed plate about 400 feet, the Federal Aviation Administration said. Business seats will cost
do other Singapone Airlines planes compurable routes because of suhetantial smount extra real extate" ddvored these sests said.
15
percent
Along with its twin TC-2. TC-1 was the finst satellite buill and operated by the Chinese National Space Administration in -She was on the 78th floor. waiting for an express express elevator to laave the south tower. she said, wien the second plane struck Details were also given of how insrumestation developed for the Bengle 2 and Rosetta missions being tumee into cost effective tool for diagnosing tnherculosis.
cooperation with the ESA.
Chang
scheduled sean the lunar surface from Wednesday prepacation an unmanned velicle planned for 2012 manned
hading within 15 years.
site in the village Dongchang di appears to be designed luunch" a bugger sized missile or satellite projectile "than rockets deployed from
the North is cast const facilit..
18


Reviews:
[{"review_id": "wu_SjClxc-8", "reviewer": "Reviewer_QiHo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clearly written in that I have no problem reading it and understanding the concepts. The experiments are well structured and provide reasonable depth in evaluation. The authors are also clear on the motivations, which is a good thing since it highlights the real contributions of this paper, which is enhancing the multi-lingual encoder's property of being language-agnostic, while making it more robust to noise.\n\nA minor point that concerns me is the fact that it seems semantic contrastive. I can understand that artificial noise might not completely alter the semantic representation of the sentence to the extent that it's general context/topic becomes completely different, but I am a bit conservative when it comes to whether such improvement in robustness could really be interpreted as increase in semantic contrast. Topic and word distribution alone isn't enough to model sentiment, let alone semantics. This is a minor point in the use of terminology there, but I do wish the authors would clarify their point-of-view on the subject.", "summary_of_the_paper": "The paper proposes a fine-tuning method for multi-lingual sentence representations trained on monolingual data, such that the final representation is more language agnostic, as well as more contrastive in the sense that artificially infused noise do not break the clustering of these representations.\nThe authors then experiments on using these fine-tuned sentence encoders to mine parallel samples from monolingual data, which is then integrated into UMT training, and show visible improvements.", "summary_of_the_review": "This paper proposes empirically useful methods to fine-tune multi-lingual sentence encoder for mining data for Unsupervised NMT augmentation. I believe that this approach shows novelty, and the paper itself good motivation and inspiration from previous work, making it a pleasant read. I have only minor concerns regarding terminologies and interpretations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sHBQYkRBJm1", "reviewer": "Reviewer_3VMA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths :\n- The authors introduce a novel language agnostic approach to mine synthetic parallel data from existing monolingual corpus of different languages\n- Adapting SwAV loss from computer vision into natural language processing as a fine-tuning step on top of masked language models.\n- Re-balanced weighting technique to force a balanced cluster assignment independent of the language itself (in case of using separate masked language models) and thus a mixed embedding space\n- Language agnostic embedding space  is clearly shown in their tsne plots, sentences with similar content and from different languages are close to each other in the embedding space\n- The proposed approach doesn’t require pretraining on massive multilingual parallel data\n\nWeaknesses:\n- The author proposed a set of filtering criteria to filter out low quality samples from the initial-mined dataset. The filtering criteria rely more on hand-crafted thresholds like source-target length ration, and sub-word overlap which would require per language pair tuning to get a decent result. A meta-classifier on top of these features can be trained using a small clean source-target dataset to avoid per-language tuning.\n- gradients are not flowing through re-balanced weights vector and the weights are computed based on per batch statistics, wouldn't it be better to optimize the re-balanced weights to avoid local bias within an already imbalanced batch (what if that batch contains sentences from source only), it would be a good idea to do an ablation study on an imbalance set of batches to see the cluster assignment behavior and its effect on the data mining.\n- The target of the paper is to mine parallel data for machine translation, it makes sense to evaluate training an NMT network from scratch on mined parallel data and compare it to baselines[1, 2] mined data trained also from scratch.\n- In background section pseudo-parallel data mining for machine translation, the authors missed citing some related papers that generated synthetic data for neural machine translation for example:  [4, 5]\n\nReferences:\n\n[1] Wu, Jiawei, Xin Wang, and William Yang Wang. \"Extract and edit: An alternative to back-translation for unsupervised neural machine translation.\" arXiv preprint arXiv:1904.02331 (2019).\n\n[2] Wu, Lijun, et al. \"Machine translation with weakly paired documents.\" Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.\n\n[3] Song, Kaitao, et al. \"Mass: Masked sequence to sequence pre-training for language generation.\" arXiv preprint arXiv:1905.02450 (2019).\n\n[4] Sennrich, Rico, Barry Haddow, and Alexandra Birch. \"Improving neural machine translation models with monolingual data.\" arXiv preprint arXiv:1511.06709 (2015).\n\n[5] Chinea-Rios, Mara, Alvaro Peris, and Francisco Casacuberta. \"Adapting neural machine translation with parallel synthetic data.\" Proceedings of the Second Conference on Machine Translation. 2017.", "summary_of_the_paper": "The authors propose a language agnostic SwAV loss function to mine pseudo parallel data from unlabeled monolingual corpus from various languages. They start with a pretrained masked language model and then fine-tune it using their proposed language agnostic SwAV. The self supervised model predicts cluster code based on representation in latent space created by swapping predicted codes between different views of the same sentence. Furthermore, they extend that proposed loss with a weighting term to re-balance predictions coming from different embedding spaces resulting in a homogeneous space between source and target languages. They experiment on both Wmt14 dataset and low resource FLoRes dataset achieving state of the art bleu score compared to existing unsupervised machine translation.", "summary_of_the_review": "Overall, the authors’ proposed approach is novel and their empirical results show the effectiveness of their approach on several datasets. I accept the paper in its current form, but with some weaknesses including the requirement to hand tune thresholds for the filter suite used on the mined data, and the paper is missing the rationale behind using per batch statistics to re-balance cluster assignment without flowing gradients through them. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qC7roy49BIu", "reviewer": "Reviewer_NzbT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- this work is well-movited\n- attempt to introduce CV SwAV loss in NLP is interesting\n- the analysis is very detailed and cover most questions that the reader may have\n\nWeaknesses:\n- I found the paper extremely difficult to read. Many equations that are not necessary in the paper (maybe better in appendix) with various notations that are difficult to connect with each other. \n- Table 1 is wrong. Almost none of these scores are comparable (use of different post-processing, BLEU implementations, and/or data). The paper cannot be published as long as it makes claim based on this table. Easy fix: remove all the multibleu scores.\n- statistical significance testing is never performed on the results. As far as we know, they may all be coincidental.\n- experiments settings are unclear (hyper-parameters, dataset pre-processing, etc.). Also I could not understand what baselines are reproduced by the authors. I am confident that I cannot reproduce the results.\n\n\nQuestions/comments:\n- introduction: the SwAV loss is mentioned the first time without any explanation of what it is. Acronym should at least be spelt out.\n- introduction can be reduced significantly. I think too many details are given on the approach that are then given again latter in the paper.\n- Section 3.1; \"applying masked language... to images\" sounds strange. I am not sure why applying a language model to image would make sense.\n- I suggest to draw a complete figure illustrating all the framework\n- one of the assumption of the paper is that regular UMT may not exploit the potential parallel inside the monolingual data. it would be very interesting to check this assumption (for instance by finding and removing the parallel data from the monolingual data and retrain UMT). I actually think your assumption is wrong: UMT do exploit these parallel data and it is one of the explanation behind the very high BLEU scores obtained for en-fr. \n- I found \"LAgSwAV\" very difficult to read. Maybe it is just the case for me, but I would recommend to lowercase more letters or just change the name to something more readable.", "summary_of_the_paper": "This paper presents improvements of unsupervised machine translation (UMT).\nIt introduces a modified SwAV loss to cluster sentences from monolingual data that are then exploited during training of UMT.\nEvaluation is performed on standard UMT tasks and an in-depth analysis is provided.", "summary_of_the_review": "A good idea overall but with many details and choices difficult to understand. The paper would benefit from being restructured and simplified.\nThe evaluation is very questionable and prevent a publication of the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CO6RKGNqxe2", "reviewer": "Reviewer_ARKs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe idea of rebalancing cluster assignment between language makes sense and the improvement in Table 3 is very strong, showing why the modification to the loss is important. The improvement in Table 4 is also very interesting, showing that rank-based entropy and filter suite are important to make the mining effective. Overall I found the work admirable: It is novel and results are strong\n\nWeakness:\n\nThe method itself is fairly complicate. First, SwAV itself is already complicated, its root is from computer vision, where SwAV is popular I guess because it not straightforward to do masking in computer vision. Bringing SwAV into NLP is a very nice attempt, yet I am concerning whether there is a simpler method to learn meaningful multilingual sentence embedding in an unsupervised manner so that we can mine parallel sentences. Second, extra gradients are needed, including tweaking different hyper-parameters from filter-suite and rank-based cross entropy integration (lambda hyper-parameter). This not only makes the framework complicate to scale, but also makes it hard to compare with other approaches (because we need to tune many things for the method itself but also other methods properly).\n\nQuestion: Table 4 is too good to be true because LAgSwAV is very close to LASER, a supervised method. I guess I am not satisfied with this and more explanation why it is the case could help.\n\n\n\n\n", "summary_of_the_paper": "This work provides a nice extension to the SwAV proposed by Caron. Specifically, it proposes a variant that can be used to build meaningful cluster assignment for different languages at the same time in an unsupervised manner. The key to their modification to SwAv is the so called language-agnostic rebalancing function, which is a simple modification that encourages the cluster assignments to sentence in different languages balance. Experiments show that their modification improves mining quality (i.e. resulting higher BLEU score) than using SwAV. Despite that, the work shows that using their model is not enough, and we need extra steps to make the mined data usuable (i.e. using filter suite to filter data and rank-based cross entropy to train the model with a better use of the data).", "summary_of_the_review": "An admirable work on mining parallel sentences from building meaningful sentence cluster assignment. Tweaks including rank-based entropy and filter suite are important to make it work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, reviewers are positive. The majority praised the approach as novel and viewed the results as quite strong. Further, reviewers valued the provided ablations and analysis that helped motivate the proposed method. A few concerns were raised about overall clarity (though some reviewers praised the clarity of presentation), the use of hand-crafted filters, and certain experimental comparisons. The majority of these concerns have been adequately addressed in author response.
Decision: Accept (Poster)