Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation
Authors: Wen Lai
Alexandra Chronopoulou
Alexander Fraser
Venue: EMNLP 2023 Findings (2023)
Keywords: Data Imbalance
Representation Degeneration
Multilingual Machine Translation
TL;DR: We propose a novel approach to mitigate the data imbalance and representation degeneration issues in multilingual machine translation which only requires target-side monolingual data and a bilingual dictionary.

Abstract: Despite advances in multilingual neural machine translation (MNMT), we argue that there are still two major challenges in this area: data imbalance and representation degeneration. The data imbalance problem refers to the imbalance in the amount of parallel corpora for all language pairs, especially for long-tail languages (i.e., very low-resource languages). The representation degeneration problem refers to the problem of encoded tokens tending to appear only in a small subspace of the full space available to the MNMT model. To solve these two issues, we propose Bi-ACL, a framework which only requires target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We define two modules, named bidirectional autoencoder and bidirectional contrastive learning, which we combine with an online constrained beam search and a curriculum learning sampling strategy. Extensive experiments show that our proposed method is more effective than strong baselines both in long-tail languages and in high-resource languages. We also demonstrate that our approach is capable of transferring knowledge between domains and languages in zero-shot scenarios.

Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation
Wen Lai!?,, Alexandra Chronopoulou 1.2 Alexander Fraser
CCntter for Information and Language Processing, LMU Munich, Germany Munich Center for Machine Learning, Germany {lavine, achron, fraser)@cis. 1mu de
Abstract
+A HM
paid
toda @iBinntee
:loup
Decodar
Despite advances multilingual neural ma- chine translation (MNMT), argue that there are still two major challenges in this area: data imbalance and representation degenera- tion. The data imbalance problem refers to the imbalance in the amount of parallel corpora for all language pairs, especially for long-tail languages (i.e., very low-resource languages). The representation degeneration problem refers the problem of encoded tokens tending to ap- pear only in small subspace of the full space available to the MNMT model. solve these two issues, propose Bi-ACL, framework which only requires target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We define two modules, named bidirectional autoencoder and bidirectional contrastive learning, which we combine with an online constrained beam search and a curriculum learning sampling strat- egy. Extensive experiments show that our pro- posed method is more effective than strong baselines both long-tail languages and in high-resource languages. We also demonstrate that our approach is capable of transferring knowledge between domains and languages in zero-shot !
Teghts Texil
happs
Uidrectasal
Bitrectand
Constraind
Bceas Scarch
Dus Qpiieee
Piroder
Method Overview. Our approach mainly consists of three parts: online constrained beam search, bidirectional autoencoder and bidirectional contrastive learning. Our approach explores the scenarios of using only target-side monolingual data and bilingual die- tionary to simultaneously alleviate the data imbalance and representation degeneration issues in large-scale scale MNMT model.
Figure I:
only one unified model to translate between mul- tiple language pairs, which saves on training and deployment costs.
Although significant improvements have been made recently, we argue that there are still two major challenges to be addressed: i) MNMT mod- els suffer from poor performance on long-tail lan- guages (i.e., very low-resource languages), for which parallel corpora are insufficient or non- existing. We call this the data imbalance problem. For instance 21% of the language pairs in the m2m_ 100 model (Fan al., 2021) have BLEU score of less than 1 and more than 50% have a BLEU score of less than Only 13% have a BLEU score over 20. For example, the average BLEU score for the language pairs with Irish as the target language is only 0.09. ii) Degeneration of MNMT models stems from the anisotropic distribution of token representations, i.e., their representations side in a narrow subset of the entire space (Zhang et al., 2020b). This is called the representation de- generation problem. It can lead a prevalent issue in large-scale MNMT: the model copies sentences from the source sentence or translates them into the wrong language (off-targed problem; Zhang et al.,
1 Introduction
Multilingual neural machine translation (MNM') makes possible train single model that sup- ports translation from multiple source languages into multiple target languages. This has attracted a lot of attention in the field of machine transla- tion (Johnson et al., 2017; Aharoni et al., 2019; Fan et al., 2021). MNMT is appealing for two rea- sons: first, it can transfer the knowledge learned by the model from high-resource to low-resource languages, especially in zero-shor shot scenarios (Gu al., 2019; Zhang et al., 2020a); second, uses
'Our source code is available at https: htte:://ggttuuuocoe comm lavine- Davineelmm/Bi-ee ACLL
"Plaase refer to Figure of Appendix D.I for more details.

---

2020a).
lem. iii) We show that our method demonstrates zero-shot domain transfer and language transfer capability. iv) We also show that our method is an effective solution for both the repetition (Fu et al., 2021) and the off-target (Zhang et al., 2020a) problems in large-scale MNMT models.
To address the data imbalance problem, prior work has attempted to improve the performance of a machine translation model without using any par- allel data. On the one hand, unsupervised machine translation (Lample et al., 2018a,b) 8), attempts to learn models relying only on monolingual data. On the other hand, bilingual dictionaries have shown to be helpful for machine translation models (Duan al., 2020; Wang et al., 2022). What these ap- proaches have in common is that they only require data that is both more accessible and cheaper than parallel data. As an example, 70% of the languages in the world have bilingual lexicons or word lists available et al., 2022).
2 Related Work
Multilingual Neural Machine Translation. MNMT is rapidly moving towards developing large models that enable translation between an increas- ing number of language pairs. Fan et al. (2021) proposed m2m_100 100 model that enables translation between 100 languages. Siddhant et al. (2022) and Costa-jussà et al. (2022) extend the current MNMT models support translation between more than 200 languages using supervised and sule--upe learning methods.
Representation degeneration is prevalent prob-
lem in text generation (Gao et al., 2018) and ma- chine translation models (Kudugunta et al., 2019). Contrastive learning (Hadsell et al., 2006) aims to bring similar sentences in the model close together and dissimilar sentences far from each other in the representation space. This is an effective solution to the representation problem in machine transla- tion (Pan et al., 2021; Li et al., 2022). However, the naïve contrastive learning framework that utilizes random non-target sequences negative examples is suboptimal, because they are easily distinguish- able from the correct output (Lee et al., 2020).
Autoencoder. An autoencoder (AE) is a generative model that is able to generate its own input. There are many variants of AE that can be useful for ma- chine translation. Zhang et al. (2016) and Eikema and Aziz (2019) propose using variational au- toencoder to improve the performance of machine translation models. A variant of the same genera- tive model is the denoising autoencoder, which is an important component of unsupervised machine translation models (Lample et al., 2018a). 8a). How- ever, the utility of autoencoders has not been fully explored for MNMT. To the best of our knowledge, are the first to propose training autoencoder using only target-side monolingual data and a bilin- gual dictionary improve low-resource MNMT. Contrastive Learning. Contrastive learning is a technique that clusters similar data together in a representation space while it simultaneously sepa- rates the representation of dissimilar sentences. it useful for many natural language processing tasks (Zhang et al., 2022). Recently, Pan et al. (2021) and Vamvas and Sennrich (2021) used contrastive learn- ing to improve machine translation and obtained promising results. However, these methods use the random replacing technique to construct the neg- ative examples, which often leads to a significant divergence between the semantically similar sen- tences and the ground-truth sentence in the model representation space. This large changes makes the model more difficult to distinguish correct sentence from incorrect ones. We use small perturbations to construct negative examples, ensuring their proxim- ity to the ground-truie truth sentence within the semantic
To address both problems mentioned above, present a novel multilingual NMT approach which leverages plentiful data sources: target-side mono- lingual data and a bilingual dictionary. Specifically, we start by using constrained beam search (Post and Vilar, 2018) to construct pseudo-parallae data in an online mode. To overcome the data imbal- ance problem, we propose training a bidirectional autoencoder, while address representation degen- eration, we use bidirectional contrastive learning. Finally, we use a curriculum learning (Bengio et al., 2009) sampling strategy. This uses the score given by token coverage in the bilingual dictionary to rearrange the order of training examples, such that sentences with more tokens in the dictionary are seen earlier and more frequently during training.
In summary, we make the following contribu- tions: i) We propose novel approach that uses only target-side monolingual data and bilingual dictio- nary to improve MNT performance. ii) We define two modules, bidirectional autoencoder and bidi- rectional contrastive learning, to address the data imbalance and representation degeneration prob-

---

space, which significantly mitigates the aforemen- tioned issue.
3.2.1 Backward Autoencoder
After we obtain X from Eq. 1, have the pseudo-parallel pairs (X .x-1) E Dyse. Then, we feed X to the MNMT model to get the contextual output embedding Zikdd Formally, the encoder generates a contextual embedding H given X, and l as input, which is in turn given as input the decoder (together with (3) to generate bkd
3 Method
Our goal is to overcome the data imbalance and representation degeneration issues in the MNMT model. We aim to improve the performance of MNMT without using parallel data, instead re- lying only target-side monolingual data and a bilingual dictionary. Our approach contains three parts: online pseudo-pare paridolla data construc- tion (Section 3.1), bidirectional autoencoder (Sec- tion 3.2) and bidirectional contrastive learning (Sec- tion 3.3). Figure illustrates the overview of our method. The architectures of the bidirectional au- toencoder (left) and bidirectional contrastive learn- ing (right) are presented in Figure 2.
H Encoder (X,: (e),
(2)
Z bkd = Decoder H
Finally, the backward autoencoder loss is formu- lated follows:
CAR_ bkd = --
Σ log Po (x/² |x Z
3.1 Online Pseudo-Parallel Data Construction
3.2.2 Forward Autoencoder
Let us assume that we want to improve perfor- mance when translating from source language f, to target language &, We start with a monolingual set of sentences from the target language, denoted as D Dinonoo bilingual dictionary, denoted as Dfi-tt, diet and target monolingual sentence with tt tokens, denoted as X { Zut), X E D mono. We use lexically constrained decoding (i.c., con- strained beam search; Post and Vilar, 2018) to gen- erate pseudo source language sentence X { [11, ... } in an online mode:
Given Z hkd from Eq. 2, we feed the MNMT model and get the contextual output denoted as Zfwd
H Encoder (Z Es),
(4)
Z fwd Decoder(H), 4))
The forward auto-encoder loss given by:
AAE_ =
Σ log Pa
(x Zoka Zrinal
(5)
X gengnng x 0, x Dao
3.3 Bidirectional Contrastive Learning
where gen(-) is the lexically constrained beam search function and 0 denotes the parameters of the model. It is worth noting that parameters 0 will not be updated during the generation process, but will be updated in the following steps (Section 3.2 and Section 3.3).
The main challenge in contrastive learning is to con- struct the positive and negative examples. Naive contrastive learning (Pan et al., 2021) uses ground- truth target sentences positive examples and ran- dom non-target sentences as negative examples. When pretrained MNMT model used, nega- tive examples are initially located far from positive examples in the embedding space.
3.2 Bidirectional Autoencoder
An autoencoder (Vincent al., 2008) first aims to learn how to efficiently compress and encode data, then to reconstruct the data back from the reduced encoded representation to representation that is close as possible to the original input.
Motivated by Lee et al. (2020), we automatically generate negative and positive examples, such that both kinds of examples are difficult for the model to classify correctly. This arguably motivates learning meaningful representations. Different from Lee et al. (2020), we construct negative examples in the source-side sentence" and positive examples in the target-side sentence, instead of constructing
We propose performing autoencoding using only target-side monolingual data. This is different from prior work UNMT, which uses both source and target-side data (Lample al., 2018a). Our bidirec- tional autoencoder contains two parts: backward autoencoder (Section 3.2.1) and forward autoen- coder (Section 3.2.2).
"The source-side sentence here denotes the source-side in both directions and is not a sentence in the source language. This is also the case for the target sentence.

---

Cola bldd
chi fwd
Z bkdd
3r
T
Decoder
Decoder
h1
Beyt
02
8m
61+
8
Encoder
h
m
222
2
Encoder
22g
Figure 2: Model architecture: bidirectional autoencoder (left) and bidirectional contrastive learning (right). The symbols consistent with the description in Section 3.
both the target-side sentence. Specifically, to generate negative example, we add a small pertur- bation 6; = {ō1- år} to H;, which is the hidden representation of the source-side sentence. We con- struct positive examples by adding perturbation G (G ... (r) H,. which is the hidden state of the larget-side sentence. Different from Pan al. (2021) who use a random replacing technique that may result mcaningless negative examples (al- ready well-discriminated in the embedding space), we add the perturbation to ensure that the resulting embedding space is not already in a close proxim- ity or far apart from the original embedding space. More details how generate the perturbation of 8g and & can be found in Appendix A.
ample H is generated by adding another small
(t") perturbation Gokd
H Encoder X8, Ee),
Hp H
(P))
bkd!
(6)
C bkd Decoder H 4 Ls),
H Decoder Che (s) +
(3)) Sokd
Finally, the backward contrastive learning loss is formulated as follows:
sim (H/r 2//
LCL_bkd =
Σ log
Σ L sim (HL ///
3.3.1 Backward Contrastive Learning
Given pseudo-paralla pairs (X xxe E D pse from Eq. 1, we first feed X to the MNMT model generate the contextual embedding H.. Then, we add a small perturbation off)) after H to form the negative example, denoted as H 44 Finally, the contextual output of the decoder C is generated by feeding H., to the decoder, and the positive ex-
3.3.2 Forward Contrastive Learning
After we get Cle bkd from Eq. 6, we feed Chi and the small perturbation ((P)) to the MNMT model to obtain the contextual output denoted C³a Ciwd and the negative example H.," Then, we feed C and another small perturbation (t') to generate a positive example denoted as H

---

ar, ca, ga, bs, ko, ka, ir, af. hi, jv, ml. In the multi- lingual setting, we aim to improve the performance on long-tail language pairs, which share the same target language. We randomly select 10 languages where the average BLEU score on the language pairs with the same target language is less than 2.5, For the languages not covered from news-crawl, use the monolingual data from CCAligned (EI- Kishky et al., 2020). The languages hy, ka, be, kk, az, mn, gu. For the high-resource setting, we aim validate whether our proposed method also works for high-resource languages. We randomly select 6 language pairs that cover the following language codes: en, de, fr. cs.
H = Encoder hidd Ex),
Hp l H + ((1) fwdd
(8)
C
fwd Decoder H. 6o lt);
H Decoder( (C "wdd: 4))
(8)
ftud:
Finally, the forward contrastive learning loss is given by the following equation:
la,
sim (H.,H,
CCL_fvd
Σ log
sim (H17 ///1
Σ 2Hp e
Dictionaries. We extract bilingual dictionaries us- ing the wiktextextract' tool. For pairs not involving English, we pivot through English. Given source language l, and target language &,. the intersec- tion of the two respective bilingual dictionaries with English creates a bilingual dictionary Di,- from l, to l. The statistics of the dictionaries can be seen in Appendix D.I.
3.4 Curriculum Learning
Curriculum Learning (Bengio et al., 2009) suggests starting with easier tasks and progressively gaining experience to process more complex tasks, which has been proved to be useful in machine transla- tion (Stojanovski and Fraser, 2019; Zhang al.. 2019; Lai et al., 2022b). In our training process, we first compute token coverage for each monolin- gual sentence using the bilingual dictionary. This score is used to determine curriculum to sample the sentences for each batch, so that higher-scores sentences are selected early on during training.
diet
Data Preprocessing. For the monolingual data, we first language detection tool (langid) to filter out sentences with mixed language. We proceed to remove the sentences containing least 50% punctuation and filter out duplicated sentences. To control the influence of corpus size on our exper- imental results, we limit the monolingual data of all languages to 1M. For dictionaries, we also use langid to filter out the wrong languages both on the source and target side. Baselines.
3.5 Training Objective
The model can be trained by minimizing compos- ite loss from Eq. 3, 5, 7 and 9 as follows:
L. =A(CAE bld + fmn) (1 (I-A)[CC + (wd)
We compare methods to the fol- lowing baselines: i) m2m: Using the original m2m_ 100 model (Fan al., 2021) generate trans- lations. ii) pivot_en en: Using English as a pivot language, we leverage m2m_100 to translate target- side monolingual data to English and then translate English to the source language. Following this method, we finetune the m2m 100 model using the pseudo- parallel data. iii) BT: Back- Translate (Sen- nrich al., 2016) target-side monolingual data us- ing m2m_ 100 model to generate the pseudo source- target parallel dataset, then finetune the m2m__100 model using this data. iv) wbw Im: Use a bilingual dictionary, cross-lingual word embeddings and a target-side language model to translate word-by- word and then improve the translation through a
(10)
Where À is the balancing factor between the au- toencoder and contrastive learning component.
4 Experiments
Datasets. We conduct three group of experiments: bilingual setting, multilingual setting and high- resource setting. In the bilingual setting, we fo- cus on improving the performance on specific long-tail language pair. We choose 10 language pairs random that have BLEU < 2.5 in the orig- inal m2m__10 model and considerable amount of monolingual data in the target language in news- crawl. The language pairs cover the following languages (ISO 639-1 language code"): en, ta, kk,
"https://opus.n wwwp.eee/CCAligeedd....
'https://data. htt:://ddta.sstamtt. statmt, braamttorg/news-cren
/https://github. com/ ctmm//aaaulloeeee/iikttttttrrr "https://fasttex" http//astteetttctceceteeeeeeeeee cc/dacs/en/ ieguuageiidetiiicatioo html
'https://en.u htt////wwwkippdia.c www.ppaia..rg/kkkiiittt.ffoo 639-1_codes

---

Models
Bilingual Setting
en-tta
en-*kk
en
ar-Ata
ca-Ata
ga ba-+bs
kk-
ka-tar
ta-ttr
af->ta
hi-tkk
m2m
2.12
0.26
0.34
1.75
0.51
0.85
2.14
1.41
1.46
0,84
pivot_en
0.30
0.74
0.00
0.27
0.15
1.38
1.00
0.22
BT
0,76
0.67
0.60
1.13
0.63
0.97
0.06
2.051
0.72
0.43
whw Im
2.76
0.36
0.87
0.68
0.36
0.07
2.861
2.261
1.47
0.04
syn sen_lexicon
1.33
0.14
0.72
2,071
0.93
1.101
0.85
0.57
2.071
0.89
Bi-ACL w/ Curriculum
4.57:
1,351
1.767
1.811
3.07
3.92
4.181
3.151
1.531
Bi-ACL (ours)
5.14
2,591
2,32"
3,50
2,371
3.61
4.76
4,971
3.681
2,471
A
+3.02
+2.33
+1.98
+1.75
+1.86
+3.03
+2.62
+3.56
+2,22
+1.63
Multilingual Setting
hy
ka
be
kk
gu
my
ga
m2m
1.46
1.69
0.52
0.67
2.32
0.26
0.24
0.09
Bi-ACL A
2.54
3,17
2.38
3.12
1.44
3,28
1.95
1.18
1.94
1.37
+1.08
+1.48
+1.86
+1.17
+0.77
+0.96
+0.83
+0.92
+1.70
+1.28
Multilingual Setting (specifie language pair)
en-*ta"
ar-
ca-Pta"
af-
el->ta
en-*kk"
hi-skk"
Ta fa-kkk
-kk
ml ml-*kk >kk
m2m
2.12
0.34
1,75
1.21
0.26
0.54
1.77
0.69
Bi-ACL
5.37
2.81
3.82-
4.16
3.24
2.94-
2.91
2.87
3,73/
3,29-
A
+3.25
+2.47
+2.07
+2.70
+2.03
+2.68
+2.07
+2.33
+1.96
+2.60
+
+0.23
+0.49
+0.32
+0.48
+0.35
+0.44
Table 1: Main Results: BLEU scores for low-resource language pairs in the bilingual, multilingual setting, and
10 randomly selected language pairs in the multilingual setting. Language pairs with in the multilingual setting are covered by the bilingual setting. denotes improvement over the original m2m_ 100 model, while shows improvement over the bilingual setting. Best results shown in bold. and denotes significant over original m2m 100 model at 0.05/0.01, evaluated by bootstrap resampling (Koehn, 2004).
target-side denoising model (Kim al., 2018). v) syn_seeeeee _e__exxcon: Replace the words in the target mono- lingual sentence with the corresponding source language words a bilingual dictionary and use the ggeudo-parallel data to finetune the m2m_100 model (Wang et al., 2022).
isotropy measures from Wang al. (2019), with I (W) E [0, 1] and I2(W) 0, where W is the model matrix from the whole model parameter 0. Larger I I (W) and smaller I2( () indicate a more isotropic embedding space in the MNMT model. Please refer to Appendix B for more details on /i and 12.
Implementation. We use m2m, released in the HuggingFace repository" (Wolf et al., 2020). For the wbw_Jm baseline, monolingual word embed- dings are directly obtained from the fasttext web- site and cross-lingual embeddings are trained us- supervision signal. We set 0.7 in all our experiments (the effect of different can be find in Appendix E.5).
Results
Table shows main results on low-resource lan- guage pairs in a bilingual and multilingual setting. Table shows results on high-resouree language- pairs in a bilingual setting, while Table 2 presents an isotropic embedding space analysis for the bilin- gual setting.
ing bilingual dictionary
Evaluation. We measure case-sensitive detok- enized BLEU and statistical significant testing as implemented in SacreBI.EU1) All results com- puted on the devtest dataset of Flores101 (Goyal al., 2022). To evaluate the 1 of the MNMT model, we adopt the I and Iz
Low-Resource Language Pairs in Bilingual Setting. As shown in Table 1, the baselines per- form poorly and several of them are worse than the original m2m__00 model. This can be attributed to the fact that their performance depends on the translation quality in the direction of source lan- guage to English and English target language (pivol_en), the quality in the reverse direction (BT), the quality of cross-lingual word-embeddings (wbw_lm) and the token coverage bilingual dic- tionary (syn sey__eexioon.. Our method outperforms the baselines across all language pairs, even when
"gi. htttts///faateett.c//oo///////// html
12 "git wwt. ccmm//aaeeboorreeercc/fffeeceee
"hhe representation in MNMT model is not uniformly distributed in all directions but instead occupying narrow cone in the semantic space, we call this 'anisotropy'.
13

---

ta-tr
de-frr
Encoder
Decoder
Encoder
Decoder
Encoder
Decoder
f f
12
1
1z
4 f
4
4
J
4
4g 4
1
1g
m2m
0.042
20.017
0.012
26.639
0.036
20.408
0.006
26.901
0.058
16.521
0.016
24.695
pivot_en
0.034
22.852
0.008
24.472
0.019
22.889
0.007
25.977
0.056
16.843
0.016
24.763
BT wbw_lm
0.011
25.825
0.007
25.797
0.028
22.009
0.009
27.492
0.074
14.774
0.015
24.878
0.023
23.485
0.015
24.746
0.038
19.389
0.010
26.320
0.037
19.099
0.015
24.935
syn lexicon
0.059
17.513
0.015
25.694
0.028
20.640
0.013
26.475
0.020
23.859
0.014
24.137
Bi-ACL w/o Curriculum
0.074
16.174
0.017
24.176
0.039
19.139
0.018
24.712
0.078
14.165
0,017
24.128
Bi-ACL (ours)
0.086
15.714
0.020
23.251
0.043
18.672
0.021
22.716
0.086
13.666
0.017
24.067
Table Main Results: Isotopicc embedding space analysis in ar- and ta-ttr translation task. The definitions of /1 and 12 can be found in Appendix B.
Mudels
en-ide
ea-+fr
en-ics
de-fr
de-ies
fr-ses
terestingly, our findings reveal that back translation
22.79
32.50
21.65
20.13
20.30
does not yield optimal results in both low and high
pi
11.68
7.09
BT
24.08
23.71
21.52
19:45
17:41
17:00
resource settings. In fow-resource languages, the
9.04
9.44
10.15
syn Jesicon
5.35
12.56
10.90
5.90
3.39
8.89
performance of the language pair and its reverse direction in the original m2m 100 model is signif- icantly poor (i.e., nearly zero). Consequently, the results in performance that
BI-ACL w/o Curnculum
25.17
35.52
23.91
29-43
22.71
22.04
Bi-ACE (own))
27.76
23.56
14,97
15.34
14.24
42.13
-3.07
13.26
use of back-translate is inferior to that of m2m_ 100. For high-resource languages, the language pairs already exhibit strong performance in the original m2m_100 model. This makes it challenging to demonstrate that the incor- poration of additional pseudo-parallel data can out- perform the non con-utilization of the gseudo-corpus Another potential concern is that the large amount of monolingual data employ, coupled with the substantial amount of aseudo-parallle data derived from back translation, may disrupt the pre-trained model. This observation aligns with the findings of Liao et al. (2021) and Lai al. (2021).
Table 3: Main Results: BLEU scores for high- high-resource language pairs in the bilingual setting.
the performance of the language pair is poor in the original m2m_ 100 model. addition, using the curriculum learning sampling strategy further improves our model's ' performance.
Low-Resource Language Pairs in a Multilingual Setting. In the middle part of Table 1, we show the average BLEU scores of all language pairs with the same target language. Our approach consis- tently shows promising results across all languages. Based on the results shown in the lower part of the same Table, we notice that the BLEU scores obtained in the multilingual setting on a specific language pair outperform the scores obtained in the bilingual setting. For example, we get 3.68 BLEU points for af-xta in the bilingual setting, while we get 4.16 in the multilingual setting. This confirms our intuition that knowledge transfer between dif- ferent languages in the MNMT model when using a multilingual setting beneficial (see more details in Section 6.2).
Statistical Significance Tests. The use of BLEU in isolation as the single metric for evaluating the quality of method has recently received criti- cism (Kocmi et al., 2021). Therefore, we conduct statistical significance testing in the low-resource setting demonstrate the difference well as the superiority of our method over other baseline sys- tems. As can be seen in Table 1, our method outper- forms the baseline by significant diffcrences, which is even more evident in the case study in Table 10. This is because the baseline system faces the seri- ous problems of generating duplicate words (repeat problem) and translating to the wrong language (off toff-target problem), while our method avoids these two problems.
High-Resouree Language Pairs in a Bilingual Setting. As shown in Table 3, baseline systems do not perform well on all high-resource pairs due to the same reasons as in the long-tail languages setting. Our approach outperforms the baselines on all high-resource pairs. In addition, curriculum learning takes full advantage of the original model in the high-resource setting, with stronger gains in performance than in the low-resource setting. In-
Isotropy Analysis. It is clear from Table 2 that the embedding space on the encoder side is more isotropic than on the decoder side. This is because we only use the target-side monolingual data to

---

ta-ttr
en-
fod
DGL
Jood
BLEU
11
1/
BLEU
11
12
BLEU
1 1
1/ L
2.51
0.005
30.737
3.34
0.004
32.378
23.14
0.011
24.876
V
3.27
0.006
29.299
3.96
0.006
29.373
23.82
0.011
24.651
#1
X
X
V
2.39
0.008
26.562
2.69
0.007
28.663
22.57
0.013
24.872
X
X
V
2.36
0.009
26.541
2.65
0.007
27.155
22.89
0.013
24.367
V
V
4.03
0.009
27.147
4.12
0.011
27.039
25.62
0.014
24.075
V
2.36
0.012
26.782
3.64
0.010
27.636
24.84
0.013
24.513
2.50
0.014
26.007
3.37
0.012
26.881
24.36
0.012
24.841
3.54
0.012
26.964
3.89
0.011
27.175
24.59
0.012
24.761
3.81
0.019
25.597
4,03
0,015
26.460
25.17
0,014
24.025
2.53
0.013
28.459
3.61
0.012
27. 67.639
24.73
0.012
24.723
3.85
0.020
24.732
3.73
0.016
26. 26.197
25.43
0.014
24.137
4.31
0.028
23.861
4,29
0.019
25.573
26.44
0.015
24.019
2.82
0.023
24.352
3.77
0.018
25.852
25.63
0.014
24.257
3.83
0.025
24.173
4.05
0.015
26.447
26.17
0.015
14.192
#4
V V
5.14
0.031
22.392
4.97
0.022
24.175
27.76
0.016
23.951
Table 4: Ablation study of four loss functions on en- and ta->ar translation task. "v" means the loss function is included in the training objective while "x" means is not. Both I and Iz score are computed in the decoder side.
improve the decoder of the MNMT model. Com- pared to other baseline systems, we get a higher Ii and lower I2 score, which shows more isotropic embedding space in methods. An interesting finding that the difference in isotropic space be- tween high-resource language pairs not signif- icant. This phenomenon because the original 100 model already performs very well on high-resource language pairs and the representation degeneration is not substantial for those language pairs. In addition, the phenomenon is consistent with the findings in Table 4.
tasks to understand the contribution of each loss function. The experiments in Table 4 are divided into four groups, each group representing the num- ber of loss functions. We have the following three findings: i) #1 clearly shows that the bidirectional autoencoder losses (CAF_bka bkdd and CAR_fwa) play a more critical role than the bidirectional contrastive learning losses (LCL_bbd and LCL_fwd) in terms of BLEU score. However, bidirectional contrastive losses are more important than bidirectional au- toencoder losses in terms of Iı and 12 score. This could be the case because contrastive learning aims to improve the MNMT model's isotropic embed- ding space rather than the translation from source language to target language. ii) Using forward di- rection losses results in a better translation quality compared to backward direction losses (#1). This is because our goal is to improve the performance from source language target language, which is the forward direction in the loss functions. iii) The more loss functions there are, the better the perfor- mance. The combination of all four loss functions yields the best performance. iv) We show that the I and Iz scores in high-resource language pairs (en+tde) do not have a significant change as the original embedding space is already isotropic.
6 Analysis
In this section, we conduct additional experiments to better understand the strengths of our proposed methods. We first investigate the impact of four components the results through ablation study (Section 6.1). Then, we evaluate the zero- shot domaín transfer ability and language transfer ability of our method (Section 6.2). Finally, we evaluate some impact factors (the quality bilin- gual dictionary and the amount of monolingual data) on our proposed method (Section 6.3) and present case study to show the strengths of our approach in solving the repetition problem and off- target issues in MNMT model.
6.2 Domain Transfer and Language Transfer
6.1 Ablation Study
Motivated by recent work on the domain and lan- guage transfer ability MNMT models (Lai et al., 2022a), we conduct a number of experiments with extensive analysis to validate the zero-shot domain
Our training objective function, shown in Eq. 10, contains four loss functions. We perform an abla- tion study on en tta, ta ta->ar >ar and en en- translation

---

transfer ability, as well as the language transfer abil- ity of our proposed method. We have the following findings: i) Our proposed method works well not only on the F0ores101 datasets (domains similar to training data of the original m2m_100 model), but also on other domains. This supports the domain transfer ability of our proposed method. ii) We show that the transfer ability is more obvious in the multilingual setting than in the bilingual setting, which is consistent with the conclusion from Ta- ble 6 in the multilingual setting. More details can be found in Appendix E.3 and E.4.
on other NLP tasks, such as text generation or ques- tion answering task, because our framework only depends on the bilingual dictionary and monolin- gual data, which can be easily found on the internet for many language pairs. ii) We only evaluated Bi- ACL using m2m_moomom ___0000 as a pretrained model. How- ever, we believe that our approach would also work with other pretrained models, such mT5 (Xue et al., 2021) and mBART (Liu et al., 2020). Be- cause the two components (bidirectional autoen- coder and bidirectional contrastive learning) we proposed can be seen as plugins, they could be easily added to any pretrained model.
6.3 Further Investigation
Acknowledgement
To investigate two other important factors in our proposed methods, conducted additional exper- iments to evaluate the impact of the quality of the dictionary and the amount of monolingual data. In general, we observe that better performance can be obtained by utilizing a high-quality bilingual dictionary. In addition. the size of the monolin- gual data used is not proportional to the perfor- mance improvement. More details can be found in Appendix E.1 and E.2. Also, compared with the baseline models, our method has strengths in solv- ing repetition and off targett problems, which are two common issues in large-scale MNMT models. More details can be found in Appendix E.6.
This work was supported by funding from China Scholarship Council (CSC). This work has received funding from the European Research Council under the European Union's 's Horizon 2020 research and innovation program (grant agreement #640550). This work was also supported by the DFG (grant FR 2829/1-1).
References
Roee Aharoni. Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
7 Conclusion
To address the data imbalance and representation degeneration problem in MNMT, we present a framework named Bi-ACL which improves the per- formance of MNMT models using only target-side monolingual data and bilingual dictionary. We employ bidirectional autoencoder and bidirec- tional contrastive learning, which prove be effec- tive both on long-tail languages and high-resource languages. We also find that Bi-ACL shows lan- guage transfer and domain transfer ability in zero- shot scenarios. In addition, Bi-ACL provides a paradigm that an inexpensive bilingual lexicon and monolingual data should be fully exploited when there are bilingual parallel corpora, which believe more researchers in the community should be aware of.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international confer- ence on machine learning. pages 41 41-48.
Marta R Costa-jussa James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.0667
Xiangyu Duan, Baijun Ji, Hao Jia, Min Tan, Min Zhang, Boxing Chen, Weihua Luo, and Yue Zhang, 2020. Bilingual dictionary based neural machine translation without using parallel sentences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1570-1579, Online. Association for Computational Linguistics.
8 Limitations
Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceed- ings of the 4th Workshop on Representation Learn- ing.for NLP (RepLANLP-2019) 2219), pages 124-141, Flo-
This work has two main limitations. i) We only evaluated the proposed method on the machine translation task, however, should work well

---

rence, Italy, Association for Computational Linguis- tics.
multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Ahmed EI-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Kochn. 2020, CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5960- 11660-596,0000 Online. Association for Computational Linguistics.
Yunsu Kim, Jiahui Geng, and Hermann Ney. 2018. Im- proving unsupervised word-by-word translation with language model and denoising autoencoder. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 862- 868, Brussels, Belgium. Association for Computational Linguistics.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low- now-resource neural machine translation. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 567-573, Vancouver, Canada. Association for Computational Linguistics.
Tom Koemi, Christian Federmann, Roman Grund- kiewicz, Marcin Junczys-Downunt, Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 494, Online. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric mul- tilingual machine translation. Journal of Machine Learning Research, 22(107)::-48.
Philipp Koehn. 2004. Statistical significance tests for
machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Sncha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1565-1575, Hong Kong, China. Association for Computational Linguistics.
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A theoretical analysis of the repetition problem text generation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages -1886..
Jun Gao. Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2018. Representation degeneration prob- lem in training natural language generation models. In International Conference on Learning Representa- tions.
Wen Lai, Alexandra Chronopoulou, and Alexander Fraser. 2022a. m adaater:: Multilingual multi- domain adaptation for machine translation with a meta-adapter. arXiv preprint arXiv:2210. 11912.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-534 -388.
Wen Lai, Jindfich Libovický, and Alexander Fraser. 2021. The LMU Munich system for the WMT 2021 large-scale multilingual machine translation shared task. In Proceedings of the Sixth Conference on Ma- chine Tronslation, pages 412-417, Online. Associa- tion for Computational Linguistics.
Wen Lai, Jindfich Libovický, and Alexander Fraser. 2022b. Improving both domain robustness and do- main adaptability in machine translation. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 5191-5204, Gyeongju. Republic of Korea. International Committee Com- putational Linguistics.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Vic- tor O.K. Li. 2019. Improved zero-shot neural ma- chine translation via ignoring spurious correlations. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 1258- 1268, Florence, Italy. Association for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEFE Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR'06), volume 2. pages 1735-1744 1742. IEEE
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018b. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
Melvin Johnson, Mike Schuster, Quoc V. Le. Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's

---

5039-5049, Brussels, Belgium. Association for Com- putational Linguistics.
Dario Stojanovski and Alexander Fraser. 2019. Improv- ing anaphora resolution in neural machine transla- tion using curriculum learning. In Proceedings of Machine Translation Summit XVII: Research Track, pages 140-150, Dublin, Ireland. European Associa- tion for Machine Translation.
Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. 2020. Contrastive learning with adversarial perturbations for conditional text generation. International Con- ference on Learning Representutions.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation (LREC'12), 12), pages 2214-2218, Istanbul, Turkey, European Language Resources Association (ELRA).
Yaoyiran Li, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulié. 2022. Improving word translation via two-stage contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computaaional Linguistics (Volume I: Long Papers), pages 4353-4374, Dublin, Ireland. Association for Computational Linguistics.
Jannis Vamvas and Rico Sennrich. 2021. Contrastive conditioning for assessing disambiguation in MT: A case study of distilled bias. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10246-10265, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Baohao Liao, Shahram Khadivi, and Sanjika Hewavitha- rana. 2021. Back-translation for large-scale multilin- gual machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 418 424, Online. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics,& 8:726-742.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. 2008. Extracting and com- posing robust features with denoising autoencoders. In Proceedings of the 25th international conference Machine learning, pages 1096-1103.
Ilya Loshchilov and Frank Hutter. 2018 Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Lingxiao Wang, Jing Huang, Kevin Huang. Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2019. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representa- tions.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 244-258, Online. Asso- ciation for Computational Linguistics.
Xinyi Wang. Sebastian Ruder, and Graham Neubig. 2022. Expanding pretrained models to thousands more languages via exicon-based adaptation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 863-877, Dublin, Ireland. Association for Computational Linguistics.
Matt Post and David Vilar. 2018 Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pu- pers), pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Lin- guistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5; A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao. Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2022. Towards the next 1000 languages in multilin- gual machine translation: Exploring the synergy be- tween supervised and self-supervised learning. arXiv preprint arXivv2201.0311 03.10.

---

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020a. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine trans- lation. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Process- ing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
Rui Zhang, Yangfeng Ji, Yue Zhang, and Rebecca J. Passonneau. 2022. Contrastive data and learning for natural language processing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, pages 39-47, Seattle, United States. Association for Computational Linguistics.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curricuum learning for domain adaptation in neu- ral machine translation. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 1903-1915, Minneapolis, Minnesota. Association for Computational Linguisties.
Proceedings of the 2019
Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao, Qinli Yang. and Jumming Shao. 2020b. Revisit- ing representation degeneration problem in language modeling. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 518 527,, Online. Association for Computational Linguistics.

---

A Contrastive Learning
Our approach is different from traditional con- trastive learning, which takes a ground-trunt sen- tence pair as a positive example and a random non- target sentence pair in the same batch nega- tive example. Motivated by Lee et al. (2020), we construct positive and negative examples automati- cally.
1540
300
343
K2 +20
415
+22
+22
A.1 Negative Example Formulation
BLDU
Figure 3: BLEU score statistics of the m2m_100 model (Fan et al., 2021) on Flores101 dataset (Goyal et al., 2022) for 102 101 10302 language pairs. Each bar denotes the number of language pairs in the interval of the BLEU score.
As described in Section 3.3, to generate nega- tive example, we add small perturbation d; to the H₁, which is the hidden represen- tation of the source-side sentence. As seen in Eq. 6, the negative example is denoted as H and is formulated as the sum of the original contextual em- bedding H of the target language sentence X 8(1")
space.
and the perturbation g((") bkd" Finally, bkd is formu- lated as the conditional log likelihood with respect
((W) min ((s)
max,cs Z(s)
to 8. 8(6") is semantically very dissimilar to x but very close the hidden representation H.." & in the embedding space.
12(W) Eees (Z((s) - Z())
[S|2(s)2
H -Encoder 4t),
where ((s) - - Ei= exp (s" (s"wm)
is close to some constant with high probability for all unit vectors s if the embedding matrix W is isotropic (wy W). S' is the set of eigenvectors of W W. I2 is the sample standard deviation of ((s) normalized by its average (s)). We have (W) E [0, and (W) 2 O. Larger I (W) and smaller I2 (W) indicate more isotropic for word embedding space. In this work, we randomly select 128 sentences from Flores101 benchmark to compute these two criteria. The results are shown in Table 2.
w.)
H H iated
((0) arg min log pa bled
X
+
where E (0,1] is parameter that controls the perturbation and denotes the parameters of the MNMT model.
A.2 Positive Example Formulation
As shown in Eq. 6, we create a positive example of the target sentence by adding a perturbation G - (r to H;, which is the hidden state of the target-side sentence. The objective of the perturbation Sitd is to minimize the KL divergence between the perturbed conditional distribution and the original conditional distribution as follows:
C Model Configuration
We use the m2m__00 L00 model with 418MB param- eters implemented in Huggingface. In our experi- ments, we use the AdamW (Loshchilov and Hutter, 2018) optimizer and the learning rate are initial 2e - with a dropout probability 0.1. 1. We trained our models on one machine with NVIDIA V100 GPUs. The batch size is set to per GPU during training. To have a fair comparison, all experiments are trained for 3 epochs.
((t))
Saee arg min DEL
pg-
(x/+
x
(11)
where 0* is the copy of the model parameter 0. As result, the positive example is semantically similar to X and dissimilar to the contextual embedding of the target sentence in the embedding space.
D
Statisties
B Evaluation of Isotropy
D.1 Statistics of BLEU scores in m2m 100
We use I1 and 12 scores from Wang et al. (2019) to characterize the isotropy of the output embedding
Figure 3 shows the BLEU scores the m2m_100 model on all 10302 supported language pairs. We

---

ka
ta-xtr
Flores
TED
QED
KDE
Flores
TED
QED
KDE
Flores
TED
QED
KDE
m2m
2.12
2.19
1.16
0.57
2.14
4.47
1.00
13.35
1.41
2.03
1.32
8.10
en
0.15
0.21
0.08
0.53
1.38
1.34
0.86
3,93
BT
0.76
0.26
0.01
0.36
0.06
0.24
0.09
0.64
2.05
0.89
0.63
whw_in Im
2.76
1.97
0.29
0.67
2.86
0.17
0.12
6.24
2.26
1.53
1.19
8.70
lexicon
1.33
1.19
0.08
0.20
0.85
1.65
0.45
10.64
0.57
0.41
0.55
7.08
Bi-ACL w/o Curiculum
4.57
2.62
1.24
0.75
3.92
4.61
1,20
13.82
4.18
2.39
1.68
11.75
Bi-ACL
5.14
2.84
1,41
1.05
4.76
4.93
1,57
14.62
4,97
2.81
1.96
12.74
A
+3.02
+0.65
+0.25
+0.48
+2.62
+0.46
+0.57
+1.27
+3.56
+0.78
+0.64
+4.64
Table 5: Domain transfer: BLEU scores -t, ka >ar, >tr in different domains.
Bilingual Setting
Language Pair
#Size
Language Pair
#Size
ar2ta=>en2ta en2ta=>ar2ta
en-sta
8,376
af-tta
5,268
m2m
0.34
2.12
en en-*kk
11,323
-kk
24,762
pivot_en
0.34
ar-
26,768
en-*de
68,029
BT
0.28
0.55
ca-
18,757
en-
78,837
wbw Im
1.07
2.68
ga-+hs
125,336
en-+cs
35,879
syn lexicon
0.43
1.86
kk-
38,710
de-+fr
207,831
ka-tar
25,825
de-+os
125,909
Bi-ACL w/o curriculum
2.08
4.53
14,169
fr-xes
104,510
Bi-ACL. transfer
2.74
5.78
+0.42
+0.64
Table 7:
Statistics of bilingual dictionaries.
Multilingual Setting
ta=
be=tta
m22
1,95
1.46
en-+ta
ca-
ga
ba-bbs
ta
>tr
Bi-ACL
2.73
2.31
m2m
2.12
1.75
0.51
1.41
Bi-ACL transfer
3.42
panlex
3.19
2.47
0.67
3.41
+0.55
+0.88
wiktionary
5.14
3.50
2.37
4.97
Table 6: Language transfer: BLEU scores langauge (pair) transfer ablity both on bilingual setting and multi- lingual setting. 'A=B'' means from language (pair) transfer to language (pair) B. & denotes the improve- ment on our proposed methods.
Table 8: The effect of bilingual dictionary quality on experimental performance in terms of BLEU score.
guages. We evaluate the performance on en en->ta, ca-*ta, ga-tbs and ta->tr translation tasks.
As in Table using the dictionary mined from wikitionary results in a better performance than using the panlex dictionary. The reason for this is that, while Panlex supports bilingual dic- tionaries for many language pairs, we discovered that the quality of them is quite low, especially when English is not one of the two languages in the language pair.
see that 21% % of the langauge pairs have a BLEU score of almost and more than 50% have BLEU score of less than 5.
D.2 Statistics of Bilingual Dictionaries
Table 7 shows the size of the bilingual dictionaries used in bilingual setting. For the multilingual setting, we will publish code generate the bilingual dictionary for any language pair.
E.2 Amount of Monolingual Data
As described in Section 3.4, we use the bilingual dictionary coverage & as the curriculum to order the training batch. In this section, we aim to inves- tigate how the number of monolingual data affects the experimental results. A smaller Φ means larger number of monolingual data. We conduct experiments on en-+ta, en-*kk, ar-tta and ca-+ta translation tasks with a different
E
Further Analysis
E.1 Quality of the Bilingual Dictionary
To investigate whether the quality of bilingual dic- tionary affects the performance of our method, we conduct additional experiments using the Panlex dictionary big dataset that covers 5,700 lan-
"https: / //tp//x..anelrrgeeeeeeee
As seen Table 9, we observe that the amount

---

35
cnzta
5.0
ta2tr
de2fr
30-
29
21g6
3.5
3.0
25
25
240
0.0
0.2
0.4
0.6
0.8
A
(a) Long-Tai Tail Language Pair
(b) Hlihh-Resoune Language Pair
Figure 4: The effect of different X
publicly available datasets from OPUS (Tiede- mann, 2012) and the Flores101 benchmark. The results is shown in Table 5.
en- ta
en en->kk
ar- ta
ca ca->ta ta
m2m
2. 2.12
0.26
0.34
1.75
0.5
3.70
0.84
1.22
2.28
0.6
3.60
0.75
1.07
2.26
According to Table 5, the performance of the baseline systems is even worse than the original m2m_ model, which suggests that they do not show domain robustness nor domain transfer ability due poor performance (see Table 1). In contrast, our proposed method works well not only the Flores datasets (domains similar to training data of the original m2m__10 model), but also on other domains.
0.7
3.64
0.23
1.15
2.24
0.8
3.82
0.46
1.28
2.52
0.9
5.14
2.59
2.32
3.50
1.0
3.27
0.07
1.01
2.15
Table 9: The effect of monolingual corpus size in exper- imental results in lerms BLEU score. The smaller value of (bilingual dictionary coverage) the larger the monolingual corpus.
E.4 Language Transfer
To investigate the language transfer ability, we use the model trained on a specific language (pair) to generate text for another language (pair) both in the bilingual and multilingual settings. For the bilingual setting,
of monolingual data is proportional the perimental performance. This is because a large percentage of words in a sentence are not covered by lexicons in the bilingual dictionaries, the per- formance constrained beam search is limited.
experiments to assess the language transfer ability between en-tta and ar-+ta translation tasks. For the multilingual setting, we focus on translation scores between ta and be. The results is shown in Table 6.
This phenomenon is consistent with the conclusion that the effect of the size of the pseudo parallel corpus in data augmentation (Fadaee et al., 2017) and back-translan -tttssaallation (Sennrich et al., 2016) on the experimental results, i.e., that the performance of machine translation is not proportional to the size the pseudo parallel corpus.
As indicated in Table 6, we observe that the performance in our method outperforms the other baselines both in the bilingual and in the multi- lingual setting. We also discover that the transfer ability is more obvious the multilingual setting than the bilingual setting. This phenomenon is consistent with the conclusion from Table in the multilingual setting. We believe that this can be at- tributed to the fact that in multilingual setting, the language is used for all language pairs that share
E.3 Domain Transfer
To investigate the domain transfer ability of our approach, we first conduct experiments on en-tta, ka >ar, ta >tr translation tasks, then evaluate the performance in a zero-shot setting on three dif- ferent domains (TED, QED and KDE) which are
Wtttps: httpss:////uuunieee

---

Example 1 (Repetition Problem)
Sourcw (English:: 'We now lame 4-mon/hookd that we not--iiteeric that usd be dadaiit; Ta atdded
Reforence (Kaaakh) ekgopp diyg gratemen apaрсои dipax uolpp graterl Hor 4 aitua To Taursauggo Sape Jin KoOM on.
mmm
<ieetizee yoer vockaupaoo aMorriie yav veckangrps, oiterfox vocc mootor paps, dHentiee yrec sockenaaak
pHot_on
Mo raap Nevicas diare abauk Xieroox katenti zwobern Olare r0laaatreuuun Hervoos groberiic" Anal noccaus.
Cospaiios,, mags
speip wombo siksaa Myscrisss Giose saciiii pesfienii Giow ruquuseeeeee sompmmaees sparfen die"" youú Rocaae
syn lerizon
Gei spepp finy 4- as Sypienes swdaa ian Gany useit sjoelones 1am0 apocess flnn quethirie" on apaay.
BI-ACL Curnculum
диeßerneн seckangpp,
(ours)
pxebeenne wockanppp. con kaspp aktux wox aRturi Kastenoppan:
Example 2 (Off-targes Problem)
Source (Englisht: "hher themmal behaworr not ae steecy large coves on Earch that otten maintain to restntt temperature, but conslstenc with deep holes the ground Glen Gusting Unted States Geolog col Survey Astrogcoiczy Team of Nonthom Artzona University Flagstff, rooona
Reference (Tamit: adlanfaad cursamontiii aunnisnne agi unfis Llss sun&sment ait cundegu Qui Qui anggnnuu
Gun@eanf طoggp curéaammt ugKCsuc uuó susCuunuuh @Cean) uhongs Gaftfeo #LLb Gatg åands4nar &cJeniuuss
4yrlv8oost.0e @ians Single Orgasm, Mulipl Orgaom mndtggub du Migiggntssst
pwot
Guggbugga @ein'vn.@@mmäi énon Bib Clen Cualing the United Sanas Caslogical Survey (USCB) Auroganiogy Toum and of Norhhmm
Faggzet
Their tharaal behavior is not as steady as lorge cave on untts dlgs suraanent 485 curógú #bB GurGgo Qut a. a.angennuu
Gie Gueling of the Unted State Gusingoal Survay (USGS) Adroualloy Toam of Adizna Univassly incated in Rappallt Arieana
wmw.Im
with these baing daap hakes the graand ungnstret Canar #LLb Cumn sc (CunneeesssssS @uann (768 eapaes
bolavior maacy linga Lunenn 4658 sursanant sliggu at:
syn Jerion
themmll behostor
ass 6ay, ugmofita thk Bupken ORitufomne, ayanmb geitons
@6g Br5d ground;
States Gcolog cal Survey (USGS) Astrogooioyy Teom Nontom Arizons University located
Ussgrstt Aricona
Bi-ACL w/o Curriculum
aiStsffissh sunssmmmfiiin candsrent eti untés dhgss cureament ads sundeu Quann elii ags auL @rant uoggg u eibs cunéos. Uhangs Calad) süiih, BLaugasssser s(jawuiiee QeLaiu066gff1g
(ouns)
aiStnffish 468 cursemmnt) awnLinennd ags dpps, cundamant atbs carégú Quiiuuloo a.sepeuu Gun@cunt wógo suréaanntt ugGai uy uobjyá anduupptg arts sandes wüLth BL donssmmen (pmg)uu6ggt g056 suppaa.
Table 10: Case study
the same target language, which can be seen as common information for all language pairs.
In contrast, our method does not have repetition problem, most likely because we enhanced the rep- resentation of the decoder through a bidirectional contrastive loss. For the second example, we show that while the off-target problem is prevalent in baseline systems, our method seems to provide an effective solution to it.
E.5 The effect of À
In Section 3.5, we set À to balance the impor- tance of both autoencoding and contrastive loss to model. From Figure 4, show that the au- toencoding loss plays a more important role than contrastive loss terms of BLEU. When 0.7, we got the best performance both in long-tail lan- guage pair and high-resource langauge pair.
E.6
Case Study
We now present qualitative results how our method addresses the repetition and off-target prob- lems. For the first example in Figure 10, we find that other bascline systems suffer from a severe rep- etition problem. This is attributed to a poor decoder.


Reviews:
[{"review_id": "8EXUfQ75ll", "reviewer": "Reviewer_G7Tr", "summary": "", "questions": "- Could the authors add statistical significance tests to the BLEU tables? Because the BLEU scores in Table 1 are very low. It is not very clear how much the BLEU improvements imply.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helpful comments.\n\nQ1: add statistical significance tests to the BLEU tables?\n\nA1: Thank you for your valuable feedback. We agree that it is important to add statistical significance tests to our experiments. We used the sacrebleu tool for significance testing and the results show that the difference between the baseline systems is not large, as the BLEU score between them is almost 0 and has strong randomness. However, the difference between our method and the baseline system is large, indicating that our improvement is meaningful despite the low BLEU score. Additionally, our case study (Figure 5 in the appendix) found that the baseline system exhibits a serious repetition phenomenon, where the generated translation repeats previously generated words. Our method successfully addresses this problem, further confirming that our approach is quite different from the baseline method. Although we are unable to add experimental results during the rebuttal stage, we will ensure that this part of the significance test will be included in the camera-ready version to address any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the proposed approach is novel;\n- the paper is well written;", "reasons_to_reject": "- the BLEU scores reported in Table 1 are very low, so it may not be very clear how much the BLEU improvements really imply;", "questions_to_authors": "- Could the authors add statistical significance tests to the BLEU tables? Because the BLEU scores in Table 1 are very low. It is not very clear how much the BLEU improvements imply.", "paper_topic_and_main_contributions": "The paper proposes a new approach called  Bi-ACL to fix two problems in multilingual NMT models, including data imbalance and representation degeneration.\n\nMain contributions:\n- a novel approach that uses only target-side monolingual data and a bilingual dictionary to improve MNMT translation quality;\n- designed bidirectional autoencoder and bidirectional contrastive learning, to address the data imbalance and representation degeneration problems;\n-  the proposed approach has zero-shot domain transfer and language transfer capabilities;", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 112: \"MNT\";"}, "extra_text": {}}, {"review_id": "m9k1hsfYye", "reviewer": "Reviewer_LrBM", "summary": "", "questions": "- Q1: I don't understand how the contrastive learning works in your case. Given batches of pseduoparallel pairs, did you calculate both losses at the same time (contrastive and autoencoding)? I need a bit more clarity here.\n- Q2: In the introduction the (real) low resource dictionary seems available. This might be very useful given dictionary construction often requires parallel text that is hard to find. Did you do any experiment using real world dictionary?\n- Q3: I want to understand the effectiveness of the method when we have more data. In a real world scenario, you often combat the low resource language just by collecting more data. How much data do we need so that standard cross entropy training is better than your proposed method?\n- Q4: Why we need to use \\lambda and (1-\\lambda) for Equation 10. Do these losses behave like probability? Why can't we use two hyperparameters for them?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and suggestions.\n\nQ1: How does contrastive learning work?\n\nA1: Yes, the four losses presented in this paper are calculated simultaneously in each batch.\n\nQ2: Did you do any experiment using a real world dictionary?\n\nA2: Yes, we ran experiments using a real world dictionary. While it is possible to extract bilingual dictionaries from most language pairs in Panlex, the dictionaries are often of low quality or contain fewer than 100 pairs. For this reason, we choose to use higher quality Wiktionary dictionaries and obtain bilingual dictionaries between non-English pairs by using English as a pivot language. For the experiments using a real world dictionary (e.g., Panlex), you can refer to Table 8 in Appendix.\n\nQ3: How much data do we need so that standard cross entropy training is better than your proposed method?\n\nA3: We appreciate your suggestion and thank you for bringing this to our attention. While we haven't run the experiment you propose for this paper, we will make sure to explore it in future work.\n\nQ4: Why can't we use two hyperparameters for them in Eq. 10?\n\nA4: Yes, the weights in our model can be interpreted as probabilities. We are aware of related work that dynamically adjusts these weights and plan to explore the application of dynamic weights to our work in the future to see if it leads to further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Although BLEU score is low, but this is a very low resource language, where baseline model BLEU is close to 0. Experiments showed respectable gains across low, medium, and high resource language. This showed robustness of the method.\n- Works on tail very low resource language is not common. The paper raises the bar of these languages.", "reasons_to_reject": "- The intuition behind contrastive learning is hard to follow. The paper jumps into explaining the technical details of the method before explaining the intution behind it.", "questions_to_authors": "- Q1: I don't understand how the contrastive learning works in your case. Given batches of pseduoparallel pairs, did you calculate both losses at the same time (contrastive and autoencoding)? I need a bit more clarity here.\n- Q2: In the introduction the (real) low resource dictionary seems available. This might be very useful given dictionary construction often requires parallel text that is hard to find. Did you do any experiment using real world dictionary?\n- Q3: I want to understand the effectiveness of the method when we have more data. In a real world scenario, you often combat the low resource language just by collecting more data. How much data do we need so that standard cross entropy training is better than your proposed method?\n- Q4: Why we need to use \\lambda and (1-\\lambda) for Equation 10. Do these losses behave like probability? Why can't we use two hyperparameters for them?", "paper_topic_and_main_contributions": "This paper addresses the issue of translating into low resources language for multilingual machine translation which are data imbalance and representation degeneration problems using Bi-Acl (Bidirectional autoencoder and contrastive learning) framework that can leverage monolingual corpus and dictionary. BiACL works by first creating online pseudocorpus using constrained decoding with dictionary as helper. This corpus is then used for training the forward and backward autoencoder (src->trg, and trg->src). Contrastive loss is added on top of the autoencoder for the model to distingusih between positive and negative examples.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The notations are hard to follow, notably indices are too small to read and quite cluttered. These are my suggestion:\n- l_{s} can be changed to 'f' (french or source language, this is common convention)\n- l_{t} can be changed to 'e' (english or target language, this is common convention)\n- fwd, bkd can be changed to use arrowhead.\n- tt and ss can be changed to T and S or |F| and |E|\n- Using + and - for marking positive and negative examples are quite confusing because those can be interprated as unary operator for exponent.\n- Equation 7 and 9 can use the exp() function, even better we can use log exp(...) - log sum exp(...) because it is fraction inside logarithm for clarity.\n- Figure 2 needs more captions and explanation. It's not obvious for the readers to understand what's going on there.\n"}}, {"review_id": "twIB49FJil", "reviewer": "Reviewer_G5qY", "summary": "", "questions": "Question A: In line 272, it says H^{+l_t} while in the last formula of equation 6, it is H^{+l_s} instead. Meanwhile, no h^{+l_s} is in Figure 2 (right part). By h^{+l_s}, do you mean c^{+l_s} in the figure?  I also don't understand the Encoder part in the last formula in equation 6 because in the figure c^{+l_s} = c^{l_s}+perturbation and no Encoder is involved. I have similar confusion about the forward contrastive learning part.\n\nQuestion B: Probably because of my confusion with equation 6, I also don't understand the logic behind your positive example construction. In lines 267 to 273, adding the first perturbation to form a negative example is straightforward. But why feeding H^{-l_t} into the decoder and then adding another small perturbation make it a positive example? \n\nQuestion C: I find it quite surprising that BT (back-translation) is not helping in most cases for both high-resource and low-resource languages. Do you have any intuition or observation for this?\n\nQuestion D: I find one ablation experiment a bit hard to interpret: in Table 4, the 2nd ablation of #1 that uses AE_fwd only. One side note: the name, backward (or forward) autoencoder, is a bit confusing because they are not real autoencoders. For example, for your backward autoencoder, its input is X_{l_t}, and its output is X_{l_t}. For a real \"autoencoder\", it needs to reconstruct its input. So putting your backward and forward autoencoder together forms one real \"autoencoder\". Your backward autoencoder is the \"encoder\" and your forward autoencoder is the \"decoder\". From my understanding, if you train the model with AE_fwd loss only, it will soon learn to copy input (target sentence). Hence, the AE_bkd is critical for learning meaningful hidden presentation Z_{bkd}. But in Table 4, using AE_fwd only surprisingly works quite well. \n\nQuestion E: In lines 882 to 884, it says 128 sentences are randomly sampled to compute I1 and I2 scores. I wonder why only 128 examples are used and how different the results will be if you sample two different sets of 128 examples. \n\nQuestion F: In Figure 5, Example 1 shows copy problem. My understanding of \"copying\" is copying words from the source sentence. However, none of the highlighted red errors are copied English tokens from the source. Do you mean other \"copying\" problems?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback.\n\nQ1: Questions in Eq. 6?\n\nA1: We apologize for any confusion caused by our description of the formula. In the camera-ready version, we will provide a more detailed explanation to avoid any misunderstanding. We have also noticed some typos, such as H^{+l_t} in main text should be changed to H^{+l_s} as shown in Eq. 6. Since we are translating from the target language to the source language, our positive example is the source language. Additionally, in the last formula of Eq. 6, it should be ‘Decoder’ instead of ‘Encoder’. We will also map the formula to Figure 2 to provide a better understanding of our method.\n\nQ2: Questions on the positive example construction?\n\nA2: Due to space constraints, we have provided more details on the construction of positive and negative examples in Appendix A. The traditional contrastive learning method [Pan et al., 2021] uses ground-truth examples as positive examples. However, in our scenario, the ground-truth examples are noisy due to the absence of a parallel corpus as a supervised signal. To minimize the impact of this noise on our model, we have added perturbations on the decoder side to ensure that the constructed positive examples are closely aligned in the representation space.\n\nQ3: Explain the reasons why BT does not improve the performance both in high-resource and low-resource languages?\n\nA3: In low-resource languages, the performance of the language pair and its reverse direction in the original m2m_100 model is extremely poor, nearly zero. As a result, using back-translation (BT) leads to inferior performance compared to m2m_100. For high-resource languages, the language pairs already perform well in the original m2m_100 model, making it difficult to demonstrate that incorporating additional pseudo-parallel data will yield better results than not using the pseudo-corpus at all. Another potential issue is that the large amount of monolingual data we use, and the substantial amount of pseudo-parallel data we obtain from BT, may interfere with the pre-trained model. This is consistent with the findings of Liao et al., 2021 [Back-translation for Large-Scale Multilingual Machine Translation].\n\nQ4: Explain Table 4 ?\n\nA4: Yes, combining the backward and forward processes results in a traditional autoencoder. We have split it into two directions to provide a better understanding of our method. However, the backward part is also a complete encoder-decoder process, as detailed in Eq. 2 and Eq. 4, as well as in Figure 2. Therefore, using only AE_bkd will also work.\n\nQ5: Why only 128 examples are used in computing the I_{1} and I_{2} score?\n\nA5: We have experimented with larger datasets and obtained similar results. We chose to use 128 samples to expedite the calculation of these two values, as they are performed on the CPU. In the camera-ready version, we will update the appendix with results from different sample sizes.\n\nQ6: Copying problem in Figure 5?\n\nA6: We apologize for any confusion and thank you for bringing this to our attention. Yes, we were referring to the repetition problem, as described by Fu et al., 2021 [A Theoretical Analysis of the Repetition Problem in Text Generation], not the copy problem. We will make the necessary corrections in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Data imbalance and representation degeneration are two important problems to address for multilingual translation.\n\n2. Experiments are thoroughly done in this paper and demonstrate the effectiveness of Bi-ACL framework.", "reasons_to_reject": "1. The method section (Section 3) is not easy to follow. In particular, I have a hard time understanding how contrastive learning is performed and why the positive (or negative) example is considered positive (or negative). The notations are confusing. Please refer to my questions for more details.\n", "questions_to_authors": "Question A: In line 272, it says H^{+l_t} while in the last formula of equation 6, it is H^{+l_s} instead. Meanwhile, no h^{+l_s} is in Figure 2 (right part). By h^{+l_s}, do you mean c^{+l_s} in the figure?  I also don't understand the Encoder part in the last formula in equation 6 because in the figure c^{+l_s} = c^{l_s}+perturbation and no Encoder is involved. I have similar confusion about the forward contrastive learning part.\n\nQuestion B: Probably because of my confusion with equation 6, I also don't understand the logic behind your positive example construction. In lines 267 to 273, adding the first perturbation to form a negative example is straightforward. But why feeding H^{-l_t} into the decoder and then adding another small perturbation make it a positive example? \n\nQuestion C: I find it quite surprising that BT (back-translation) is not helping in most cases for both high-resource and low-resource languages. Do you have any intuition or observation for this?\n\nQuestion D: I find one ablation experiment a bit hard to interpret: in Table 4, the 2nd ablation of #1 that uses AE_fwd only. One side note: the name, backward (or forward) autoencoder, is a bit confusing because they are not real autoencoders. For example, for your backward autoencoder, its input is X_{l_t}, and its output is X_{l_t}. For a real \"autoencoder\", it needs to reconstruct its input. So putting your backward and forward autoencoder together forms one real \"autoencoder\". Your backward autoencoder is the \"encoder\" and your forward autoencoder is the \"decoder\". From my understanding, if you train the model with AE_fwd loss only, it will soon learn to copy input (target sentence). Hence, the AE_bkd is critical for learning meaningful hidden presentation Z_{bkd}. But in Table 4, using AE_fwd only surprisingly works quite well. \n\nQuestion E: In lines 882 to 884, it says 128 sentences are randomly sampled to compute I1 and I2 scores. I wonder why only 128 examples are used and how different the results will be if you sample two different sets of 128 examples. \n\nQuestion F: In Figure 5, Example 1 shows copy problem. My understanding of \"copying\" is copying words from the source sentence. However, none of the highlighted red errors are copied English tokens from the source. Do you mean other \"copying\" problems?", "paper_topic_and_main_contributions": "This paper proposes to mitigate two problems in multilingual machine translation: data imbalance and representation degeneration. To this end, they propose a Bi-ACL framework that contains a bidirectional autoencoder to make use of monolingual data as well as a bidirectional contrastive learning technique to help the model learn hidden representations better. \n\nThe contributions of this paper are mainly made toward the NLP engineering experiment direction. Their Bi-ACL framework demonstrates improved translation performance for low-resource languages in both bilingual and multilingual settings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The related work section (section 2) misses relevant work about addressing data imbalance and representation degeneration.\n\n2. It would be better to provide more details about how the bilingual dictionary is used in constrained decoding.\n\n3. It would be better to provide more details about how curriculum learning is performed, e.g., how did you implement it so that higher-score sentences are selected early.\n\n4. Only the ISO codes are mentioned in the paper. It is better to mention the full names of these languages.\n\n5. The limitation section does not sound like a limitation. One limitation missed is that your method requires a decent amount of monolingual data for the target language which is probably not available for extremely low-resource or endangered languages. "}}]

Author rebuttal:

Meta-review: The paper introduces a novel framework named Bi-ACL, aimed at addressing two key issues in multilingual Neural Machine Translation (MNMT), specifically data imbalance and representation degeneration.

Pros:
- Combine several popular (and easy to implement) techniques into a novel framework for enhancing MNMT. These improvements demonstrate consistency across various data conditions.
- Utilizes readily available resources that are more accessible than parallel data, namely target-side monolingual corpora and bilingual dictionaries.
- Addresses the challenge of extremely low-resource translation directions (alongside with mid- and well-resource).
- The paper is presented in a clear and comprehensible manner.

Cons:
- The rationale behind the combination of these techniques is not sufficiently elucidated. The individual techniques themselves are not novel.
- Some minor ambiguities/errors in the paper require clarification/correction, which was addressed during the rebuttal phase thanks to the reviewers' feedbacks.
Decision: Accept-Findings