Pseudo-label Training and Model Inertia in Neural Machine Translation
Authors: Benjamin Hsu
Anna Currey
Xing Niu
Maria Nadejde
Georgiana Dinu
Venue: ICLR 2023 poster (2023)
Keywords: knowledge distillation
semi-supervised learning
self-training
forward translation
stability
robustness
machine translation

Abstract: Like many other machine learning applications, neural machine translation (NMT) benefits from over-parameterized deep neural models. However, these models have been observed to be brittle: NMT model predictions are sensitive to small input changes and can show significant variation across re-training or incremental model updates. This work studies a frequently used method in NMT, pseudo-label training (PLT), which is common to the related techniques of forward-translation (or self-training) and sequence-level knowledge distillation. While the effect of PLT on quality is well-documented, we highlight a lesser-known effect: PLT can enhance a model's stability to model updates and input perturbations, a set of properties we call \textit{model inertia}. We study inertia effects under different training settings and we identify distribution simplification as a mechanism behind the observed results. 

Published as a conference paper at ICLR 2023
PSEUDO-LABEL TRAINING AND MODEL INERTIA IN NEURAL MACHINE TRANSLATION
Benjamin Hsu, Anna Currey, Xing Niu, Maria Nädejde and Georgiana Dinu AWS AI Labs
benhsu amazon. com
ABSTRACT
Like many other machine learning applications, neural machine translation (NMT) benefits from over-parameterized deep neural models. However, these models have been observed to be brittle: NMT model predictions are sensi- tive to small input changes and can show significant variation across re-training incremental model updates. This work studies a frequently used method in NMT, pseudo-label training (PLT), which is common to the related techniques forward-translation (or self-training) and sequence-level knowledge distillation. While the effect of PLT on quality is well-documented, we highlight lesser- known effect: PLT can enhance a model's stability to model updates and input perturbations, set of properties we call model inertia. We study inertia effects under different training settings and we identify distribution simplification as a mechanism behind the observed results.
INTRODUCTION
Self-training (Fralick 1967 Amini al., 2022) is a popular semi-supervised technique used to boost the performance of neural machine translation (NMT) models. In self-training for NMT, also known as tranaadtttannatiinn an initial model is used to translate monolingual data: this data is then concatenated with the original training data in subsequent training step (Zhang & Zong 2016)] Marie et al. 2020 Edunov et al.
2020: Wang et al. 2021). Self-training believed to be effective through inducing input smoothness and leading to better learning of decision boundaries from the addition of unlabeled data (Chapelle al., 2006 He et al. 20200] Wei et al.
202T). has also been
observed to effectively diversify the training distribution (Wang et al. 2021
Nguyen et al. 2020).
closely related technique is that of knowledge distillation (Hinton al.
2015 Gou et al. 2021)
particularly sequence-level knowledge distillation (SKD), which uses hard targets in training and reduces to pseudo-labeled data augmentation (Kim & Rush 2016). In NMT, knowledge distillation is effective through knowledge transfer from ensembles
larger-capacity models and as data augmentation method (Freitag et al.] 2017 Gordon & Duh 2019 Tan et al. 2019 Currey et al., 2020). In non-autoregressive trans lation, Zhou et al. (2020) explored the effect of SKD on training data complexity and showed that simpler training data from distillation is crucial for the performance of non-autoregressive MT models.
This paper examines the component that is common to these techniques, the introduction of pseudo- labeled training (PLT) data. We focus the common autoregressive NMT formulation and show that in addition to the known quality gains, PLT has large impact on model brittleness in that increases smoothness as well stability across model re-training. Our main contributions are:
We focus set of stability properties in NMT models, which unify under the umbrella term inertia, and show that PLT increases model inertia. We further show that both the quality gains and the improved inertia are not properties of any one specific technique such as self-training or knowledge distillation, but are common to the use of labule-labeled data in training.
We investigate the hypothesis that the observed properties correlate with a training data simpli- fication mechanism, similarly to the observations made in Zhou et al. (2020). We compare with other popular semi-supervised techniques to investigate if the model quality and inertia properties hold when distribution simplification effects are not present.

---

Published as a conference paper at ICLR 2023
Based on our findings, recommend incorporating PLT into NMT training whenever inertia (e.g., stability to input perturbations and across incremental model updates) is important, as increases inertia without sacrificing quality.
2 RELATED WORK
Neural network models are known to be sensitive to input variations, i.e., lacking in smoothness. This can make them brittle or open adversarial attacks, a property observed across many applica- tion domains (Goodfellow et al., 2014; Szegedy et al. 2014 Dia & Liang] 2017). Neural machine translation models are similarly prone to robustness issues and can be affected by both synthetic and natural noise, leading to lower translation quality (Belinkov & Bisk 2018 Li et al. [0199] Niu et al.)
2020 Fadaee & Monz 2020). In MT, earlier works have found noisy data augmentation (Belinkov) & Bisk 2018) and subword regularization (Kudo, 2018 Provilkov et al. (0200) to be among the most simple yet effective methods for addressing instability to input perturbations.
In addition to smoothness, neural models are known be sensitive to the various sources of ran- domness in training, such as initialization or dropout (Bengio) [012]] Reimers & Gurevych| 2017] Madhyastha & Jain 2019). This instability negatively impacts end-usen in the form of spurious differences in outputs between model updates, or more acutely, as quality regressions on specific data points, also known negative flips (Shen et al. 2020, Xie et al., 202T Yan et al. 2021). In NLP, Cai et al.] (022) focus on a set of structured prediction tasks and show that when random initialization changes, up to 30% of all errors can be regression errors, and that improved accuracy does not always mean reduced regressions. While negative flips are more difficult to measure in MT as multiple translations can be valid, the lack of consistency across re-training is a known problem: in our experiments ~080% of the translations change due to different model random initialization alone. Despite this, to the best of our knowledge, minimizing regressions or improving stability across incremental model updates or re-trainings has not yet been addressed in MT.
This paper examines pseudo-label training in NMT and effect on stability to both input variations and incremental model updates, which we group under the term inertia. Earlier work pseudo- label training in MT focused on measuring quality alone and did not shed light on stability-
properties (Wang et al. 2021 He et al. 20200 Wei et al. [021 Yuan et al. 2020). In terms of stability to input variations, or smoothness, our findings are related to the work of Papernot et al. (2015),
where authors introduce defensive distillation and show that (self- )distillation increased smoothness
when tested on digit and object recognition tasks. They show that the effect is one of reducing the amplitude of the network gradients. Unlike our work, they do not test pseudo-label training. but soft-target distillation, where a student is trained using the prediction probabilities of a teacher.
we hypothesize that PLT techniques are able increase model inertia based on their distri- bution simplification properties. Earlier works have explored the distribution simplification property PLT methods in terms of model performance. In non-autoregressive NMT. Zhou al. (2020) and Xu et al. (2021) explored the effect of SKD on training data complexity and its correlation with model performance. As in previous work, they hypothesized that SKD alleviates the multiple modes
Finally,
problem, i.e., the existence of multiple alternative translations (Gu et al. (018). Similarly to Zhou
jet al.) (2020), we measure training data complexity when adding pseudo-labed labeled data and use the entropy of conditional word-level alignment complexity metric.
TRAINING WITH PSEUDO-LABELS IN NMT
Neural machine translation (NMT) We use the autoregressive formulation of NMT, where given parallel data containing source and target sequences, model 0 is learned using the following objec- tive:
J
V
C -
Hw = log kky k,
(I)
where [x1, xy] and y Jy...... YJ] the source/target sequences respectively, I and J are the source/target length, and V is the size of the vocabulary. Unless otherwise stated, we use beam search with fixed number of hypotheses in order generate translation from this model.
2

---

Published as a conference paper at ICLR 2023
Pseudo-label training (PLT) In this paper, introduce the term pseudo-label training (PLT) to refer to the general technique adding pseudo-labeled data during training, where the labels are obtained using previously trained NMT model. Specifically, consider two-step PLT. In first stage we estimate a teacher model 0* trained with supervised loss on samples drawn from p, the empirical distribution of the original training data:
--- (z) log pe- (y/ar
a second step we estimate the final student model 0, combining the supervised loss with a PL (pseudo-label) loss L + LPI, where:
Lpc= SpC=-Banp"(()) log po(y'
In this case the targets y' are given by the teacher distribution po- and the samples are drawn from a second distribution, p which varies in the experiments below.
Related techniques As discussed earlier, PLT is common feature of several widely used tech- niques in NMT such as self-training (a.k.a. serward-translation) and sequence-level knowledge dis- tillation. This paper opts for the term pseudo-label training (PLT) in order to avoid confusion with additional assumptions made by these techniques. Specifically:
PLT does not necessarily imply semi-supervision, as self-training does.
PLT is more specific than KD in that it is restricted to hard labels (as opposed to training on soft targets as in [iinonn et al. 2015), but more generic as does not assume model compression.
Another technique introducing synthetic data the use of back backttranslation (BT), where target segments are translated into source segments (Sennrich et al. 201 Hoang et al. 2018 Edunov ctt al. 2020). PLT does not include BT since the latter does not introduce synthetic targets or labels.
Lastly, note that self-training is closely related to entropy minimization (Grandvalet & Bengio 2004), semi-supervised technique that encourages high-confidence predictions on unlabeled data. When reducing this objective to its mode, it becomes identical to Lpc above, also observed in He
ct al. (020))
4 MODEL INERTIA
This section introduces set of desired stability-related MT properties that group under the term inertia. All our metrics are closed-box (based on user-observed model behaviour alone) and we investigate types of model inertia: (1) robustness input perturbations (or smoothness) and (2) stability across incremental model updates.
4.1 INPUT SMOOTHNESS
Robustness to input variations is important in MT models, which have been shown to be negatively affected by misspellings and other small variations in input (Belinkov & Bisk, 2018). Niu et al. (220) introduced metrics that contrast translations of noisy input with those of their clean counter- parts in order to disentangle robustness from generic quality changes. We evaluate model robust- ness and consistency input changes following the definitions introduced in Nuu et al. (2020): Rebustness measures degradation in translation quality when small variations are present in the input, while Consistency is reference-free metric for changes in translation output alone. Specifically:
Consistency = H(BLEU(Y', Y), BLEU(Y, Y"))
Robustness = BLEURT(Y', Yref) BLEURT(Y, Yref)
where Yrey stands for reference translations, Y, are translations of a clean/noisy versions of the test set (e.g., with introduced misspellings) and H(..)) )) stands for the harmonic mean. In this paper, we expand these definitions to consider robustness not only to synthetic misspellings, but also to natural grammatical emors.

---

Published as a conference paper at ICLR 2023
4.2 STABILITY TO MODEL UPDATES
Unlike smoothness metrics, stability metrics functions of two models: an original one (e.g.. one that is deployed and available to users) and an update of this model which implements an incremental change. We denote model update as a pair (0, D. A),. (8, D, A))+1, where 0 are the model parameters obtained when training using data D and algorithm A. While many incremental updates are possible, this work we keep the model size and architectures intact and vary the random parameter initialization in training, following Xie et al.] (2021) and Cai et al.] (2022). We define stability as a measure of similarity between model outputs, irrespective of quality changes, while regressions (negative flips) measure output changes that result in lower quality on given input segment CCai et al. 2022 Xie et al. 2021; Shen et al. 2020 Yan et al. 2021).
string similarity between the different outputs Y, Y We use a symmetric BLEU-based metric, the harmonic mean between BLEU(Y, Y++1) and 1111 Yi), where Y, and Yi+1 are translations obtained with models 0g and 0 respectively.
STABILITY Stability is measured
NFR Similarly to earlier works (Cai et al. 2022 Xie et al. 2021 Yan et al. 2021), we measure re- gressions Negative Flip Rate, the number of sentences for which the trans laanntaionn degrades between model updates over the total number of segments. We consider degradations in terms of both overall quality and targeted translation error category. Unlike other tasks, NMT lacks reliable automatic segment-level quality metric (Kocmi et al. 2021 Mathur et al.] 2020); we use human evaluations for this reason. Having an additional targeted error category allows to measure segment-level regression automatically. In this work, we adopt gender translation accuracy as the targeted error category.
NFI Following Cai et al. (022)), ) we also measure regressions in terms of Negative Flip Impact. NFI defined as the proportion of negative flips to the total number of errors made by the new model. Note that in NMT, error is less well-defined for quality since it is not a categorical concept. This is not the case with targeted translation error categories.
EXPERIMENTS
We perform experiments across 6 language pairs (LPs): English (en)e>German (de), Russian (ru), and Japanese (ja). We adapt the Transformer- architecture (Vaswani al., 2017] to 20 encoder layers and 2 decoder layers (denoted 20:2) as recommended by Domhan et al. (2020) and SSRU
decoder layers for faster decoding (Kim al.. (019). The deep-encoder-shalloowdeer
configu-
ration is widely used (Miceli Barone et al. 201 Kim et al. 2019, Kasai et al., (02T), and the 20:2 model was found by [Domhan et (2020) to yield comparable quality to the and 10:10 models
while significantly decreasing latency. Unless otherwise noted, we use beam decoding with beam size of (further details in Appendix(A)
Experiments are carried out with the WMT21 dataset (Akhbardeh et al., 2021 For en<+de we use 286M parallel segments, for en++ja we use 17.2M parallel segments, and for en<-ru we use WMT newstest datasets from earlier years (see Appendix B for more details on datasets used). We evaluate quality using BLEU") (Papineni et al, 2002) and BLEURT (Sellam al. (020)) the WMT21 newstest sets (Akhbardeh et al.) (021), l). We use only source-original test sets in order to avoid misestimating model performance due to translationese input (Marie et al. 2020).
34M parallel segments. For development,
We train PLT-augmented models using a mix of the original training data and pseudo-labeled data in joint training setting following Zhang & Zong (2016); Gordon & Duh (2019). Based on recom- mendations by He et al. (2020), we use dropout for all the models, set to 0.1. We do not tune the trade-off between the two losses L and LPC (we use an equal amount of original and PLT data) or the number of incremental applications of the PLT augmentation.
'Specifically, using sacreBIFU Post 2018) with signature: nrefs: ccas::m mi xed ef no took:
13a smooth eep versi :n: 0.0 except for en-+ja where we the ja ja-mecab tokenizer.
4

---

Published as a conference paper at ICLR 2023
Thousands of people aree guven drug and thousands of others given placebo..
BASELINE
Tausende von Menschen erhalten Guven ein Medikament und Tausende von anderen erhalten ein Placebo.
PLT(TRAIN)
Tausende Menschen erhalten eine Droge und Tausende von anderen erhalten ein Placebo.
SRC
Can yo put cites on those?
BASELINE PLT(TRAIN)
Können Sie Zitate darauf setzen?
Kannst du diese zitieren?
Table 1: Example translations from BASELINE and PLT(TRAIN) on the synthetic misspellings and GMEG test sets. In the first example (synthetic misspelling), the baseline invents the word Guven as translation of the original miss-spelled word, guven(given). PLT translates the second example (English learner error) Can you cite these? using the informal register, while the Baseline translates literally as Can you put citations on these? (formal register).
5.1 QUALITY AND INERTIA USING PSEUDO-LABELES DATA
This section evaluates PLT for both generic model quality and for inertia. Unless otherwise noted, student models share the same architecture as the teacher and are trained using the same parallel data with the addition of pseudo-labeled data. PLT can be implemented by sampling and labeling data from different source distributions p' the original training data (as in KD) unseen monolingual data (i.e. sumi-supervised). This section tests both: to that end, teacher models are trained on half of the available parallel data, while the other half is reserved source of unlabeled monolingual data. Specifically, we compare:
BASELINE: Model trained on half the available data without any data augmentation.
PLT(TRAIN): Data used in PLT augmentation is sampled from the training data.
PLT(UL): Data used in PLT augmentation is sampled from unused parallel data.
ALLDATA: Finally, to account for the differences in training data size, we also compare against model trained all available parallel data without any PLT.
5.1.1 INPUT SMOOTHNESS
For each of these models compute newstest quality (BLEU score) well model smoothness (robustness and consistency). We measure robustness and consistency as defined in Section 4 with the following sources of input variations:
Synthetic misspellings: We introduce misspellings proposed by Niu et al. (2020) into the new- stest set. Each word misspelled with probability of 0.1, and the strategy is randomly chosen from single-character deletion, insertion, and substitution (Karpukhin et al. 2019).
GMEG: The GMEG corpus (Napoles et al. 2019) contains data with natural errors made by En- glish language learners (grammatical misuse, misspellings, etc.). We compute consistency using the noisy input and a reference correction made by a professional annotator. We report the average consistency over the four provided reference corrections]
Example translations and results show in Tables 1 and respectively. Across all LPs, translation quality improves when pseudo-labeled data is used in training, irrespective of the source of the data added. However, sampling from unseen data does not bring additional improvements seen data for PLT. Similarly, using all parallel data vs. only half is not beneficial across the board, suggesting limitations of the training data w.r.t. the test domain.
using
PLT shows significantly higher model consistency on both synthetic misspellings and the GMEG test sets Unlike Niu et al. (2020), however, we find that robustness scores (translation quality changes relative to input changes) are not as well correlated with consistency scores, suggesting that while translations are more stable under noisy conditions they may not necessarily be better. In the context
II is not possible to compute robustness scores for GMEG as this set does not contain reference translations. Thhe noisy datasets (synthetic misspellings and GMEG) do not cover the ja ja-ten translation direction, so this direction not included in Table
5

---

Published as a conference paper at ICLR 2023
Misspellings
GMEG
LP
Setting
Teacher
Student (Original + PL.
BLEU
BLEURT
Const
Const
DAT.
286M
_1.
0..33220.03
+0.725
73.9
83.1
BASELINE
143M
27.62
0..16
-0.68440.04
73.1
82.8
en on-de
PRI(IRAIN)
143M
143M 143M (Train)
27.86-11
-0.115 2000
-0.802
76.5
85.1
PLT(UL)
143M
143M 143M (UL)
28.07
-0.118-0.0
-0.779+0.04
76.8
85.3
ALTDATA
34M
25.71
0.027 20.05
-0.77420.05
66.421.1
79.3
BASELINE
I7M +
25.08-1.1
0.035
-0.893+0.06
64.9+1.0
78.4
en-
PLI(TRAINN
17M
17M + 17M (Train)
26.16
0.044
-0.90120.05
70.2.g1.0
81.5
PLT(UL)
17M
17M 17M (UL)
25.87=1.1
0.044=0.05
=0.920
70.1
ALLDATA
17.2M
23.62..0.6
-0.289
-0.834+0.06
59.3
72.6
BASELINE
8.6M +0
22.82_00.
00.366
-0.844
59.3
71.5
en-aja
PLT(TRAIN)
8.6M
8.6M + 8.6M (Train)
24.63 =0.9
-0.280 20.00
4,9559
64.5
75.9
PLT(UL)
8.6M
8.6M 8.6M (UL)
24.59 .0.9
-0.271 20.00
+0.905
64.5
75.9
ALTDATA
286M 0
32.32_1.1
0.393 0.393
-1.202
77.
BASELINE
143M
32.36_1.1
2.394
-1.24620.06
76.9
de-+en
PLI(TRAIN)
143M
143M 143M (Truin)
32.93
0.397=0.05
-1.300
80.0
PLT(UL)
143M
143M 143M (UL.)
32.50
0.400 20.03
-1.32020.06
80.2
ALLDATA
34M
36,39_12
0.324=0.00
-0.36320.04
85.6+a.8
BASELINE
17M +0
36.26_112
0.321
-0.44220.04
84.2+0.9
nu-ton
PLI(TRAIN)
17M
17M 17M (Train)
37,00_1.2
0.337
-0,391
87.6
PLT(UL) 17M
17M + 17M (UL)
36.85
0.329 29.3
-0.430
87.0
Table 2: Training data sizes and performance scores for PLT/Baseline models. Quality is measured with BLEU and BLEURT on the WMT21 newstest set. Smoothness is measured as robustness and consistency to synthetic (Misspellings) and natural (GMEG) noise. GMEG scores computed as the average over four reference corrections. Robustness measures changes in translation quality w.r.t. input variations, while consistency measures translation changes alone.
en en-jja
ja-ten
Tu--en
Setting
St.
EM
St.
EM
St.
EM
St.
EM
St.
EM
St.
EM
ALLDATA
15.6%
2.1%
44.73
63.45
BASELINE
72.48
14.2%
77.95
30.4%
53.64
2.1%
40.78
3.6%
60.48
7.4%
67.01
11.3%
PLT-R(STUDENT)
82.03
27.9%
86.34
46.3%
63.12
5.6%
56.89
8.4%
72.93
14.3%
77.11
24.8%
PLT-A(TEACHER)
81.45
26.3%
85.04
42.4%
63.25
5.5%
54.75
5.6%
69.97
11.3%
74.81
20.6%
DISTIL.
75.44
16.2%
80.57
35.3%
44.73
4.6%
44.54
3.5%
64.51
7.0%
70.20
15.0%
Table 3: Stability (St.) to model updates, in this case re-training with different random seed. Exact match (EM) is the percent of outputs that stay identical across the two models. For Distillation (Distil), the second model trained mimic the first model.
of semi-supervised learning. it has been hypothesized that self-training has the effect of making models smoother through the addition of new data (He et al.) (2000) Wei et al., [202]. Our results suggest that this is not necessarily the case, as smoothness results are similar irrespective of the use of new unlabeled (monolingual) data (i.e., PLT(TRAIN) and PLT(UL) have similar smoothness).
5.1.2 STABILITY TO MODEL UPDATES
Next we investigate stability properties with respect to model updates when PLT is used in train- ing. We fix the source of the pseudo-labeled data to be the training data (i.c., we consider only PLT(TRAIN)) and compare translation changes when re-training model. Recall, model update consists of a pair (9, D. A)1, (8, D. A)2, where are the model parameters obtained when training using data D and algorithm A. In these experiments, keep the network architecture identical and hold A = A2, modulo the random seed used in initialization. We contrast several settings:
Models are trained and re- re-trained with half the original data (D1 = D2), and no labeled data is used. As above, we also evaluate the case where all of the original data is used (ALLDATA). We vary the random seed, leading to 0 # 62.
BASELINE:
PLT-EES) This tests the hypothesis that using PLT leads to more stable models that behave similarly when varying minor training conditions. We consider identical setup as the baseline (D, D2), except that the data is augmented to contain PLT data
PLT-Ó(TEACHER) In this setting, the two models 01 and fg use PLT data in training; however, two different teachers are used to create it (the teachers are trained with different random seeds). This simulates realistic setting where the teachers used to create pseudo-labels are not likely to

---

Published as a conference paper at ICLR 2023
WMT21
WinoMT
cn-ddc
cn-+ja
Cn-IU
cD-
CU-HU
Setting
NFR
NFR
NFR
NFR
NFI
NFR
NFI
ALLDATA
18.1%
16.5%
14.1%
4.7%
14.3%
5.4%
9.2%
BASELINE
18.0%
14.9%
16.1%
5.2%
17.0%
5.4%
9.0%
PLT-6(STUDENT)
7.4%
15.0%
10.2%
3.2%
9.8%
3.2%
5.2%
PLT-A(TRACHER)
7.8%
13.6%
11.1%
3.6%
10.8%
3.2%
5.2%
DISTIL.
14.4%
19.1%
10.9%
4.7%
14.8%
4.4%
7.2%
Table 4: Negative flip rate (NFR) and negative flip impact (NFI) on WMT21 (assessed by human annotators) and WinoMT (using the automatic gender translation accuracy metric).
stay constant. Note however that this is not a direct comparison to the baseline and PLT methods: the models do not vary in random seed alone, but also the contents of the training data (D1 # D2). DISTILLATION: In this setting D2 is obtained from D1 using pseudo-labeled data obtained with model 0. The training data D, is re-translated and merged with the original D1 to create D2- This setting is standard distillation approach for minimizing regressions, where 0g is trained to explicitly mimic 01* 's predictions (Yan et al. 2021 Cai et al. 2022).
Stability and regression metrics are averaged over (01, 02) and (0g, 01) scores since random initial- ization changes are not directional model updates. Tables 3 and 4 show stability and regression metries respectively (regression results are discussed in the next section).
First, we observe that a striking number of translations change when changing random initialization: only 15% of outputs remain identical for en-xde, and 8% and 2% remain identical for the lower- resource en en-tr and en- pairs respectively. Doubling the amount of training data (ALL.DTTA) improves stability, but not by a large margin. Across all I.ss tested, PLT improves stability relative to the baseline models and nearly doubles the percentage of segments translated identically. In- terestingly, PLT also improves stability relative to the system trained on all available parallel data, once again indicating that inertia effects do not simply stem from more data. This result is particu- larly surprising for the (LLTÖ(TEACHER setting: unlike the baseline, the two models compared are trained on different data on the target side, yet their outputs are more similar each other than the baseline outputs are to each other. This suggests that the high translation variability of the original data (a. multiple modes in Gu al. 2018) is an issue with auto-regressive MT well, and that pseudo-labeled data alleviates it even when created with different models.
Finally, we also find that distillation, where new model is explicitly trained to mimic the previ- ous model, increases stability between teacher and student, confirming earlier observations on text classification Cai al. (002)). However, this improvement is modest our experiments.
5.1.3 NEGATIVE F1.IPS
Next, we assess PLT in terms of negative flips (NFs) described in Section 4 We evaluate regres- sions in terms of overall quality (human evaluations on the WMT21 newstest set) and on a targeted error category (gender translation accuracy). For human evaluations, notators who assigned scores on a scale of 1 to 6 with 0.2 increments, where 6 indicates a perfect translation. A NF is defined both annotators agreeing that there is a degradation. Since quality is evaluated on a scale, and not as a binary score, the concept of NFI is ambiguous. We therefore com- pute negative flip rate (NFR) alone. We evaluate on en -a.eaaaa due to availability of annotators.
used two professional an-
For gender translation accuracy, which aggregates categorical measurements, we evaluate both NFR and NFI. We use the WinoMT benchmark (Stanovsky et al., 2019), a gender accuracy benchmark with a reliable segment-level metric suitable for automatic measurements of negative flips. The dataset consists of English source segments containing a profession whose gender is ambiguous at the lexical level but disambiguated in the sentential context, along with automatic morphology- based classifier that evaluates the gender accuracy when translated. We evaluate on the two of our language pairs that covered by WinoMT, en en- and en->ru.
Results are shown in Table First, we observe that, like other NLP tasks, regressions are also an issue for NMT: WMT21. 15%-20% of the test set samples are judged as having degraded in quality according to both human annotators. NER is lower for the gender translation accuracy task; however, NFs still amount to 10%-15% of the total number of emrrs, as measured by NFI. Mirroring
7

---

Published as a conference paper at ICLR 2023
Teacher
PLT
I.P
Arch.
BLEU
BLEURT
BLEL
BLEURT
Slability
Coss(GMEG)
Roh(Missp)
ConstrMissp)
25.12
-0.136
62,20
77.37
-0.807
65.77
20:1
24.08
-0.185
25.54
-0.083
72.67
79.93
-0.876
69.72
en-+X
20:2
25.12
-0.136
26.12
-0.043
74.10
80.72
-0.868
70.38
20:4
25.63
-0.136
26.27
-0.044
73.68
80.87
-0.886
70.05
28.12
0.076
61.91
-0.44
80.53
20:1
26. 74
-0.047
28.30
0.029
73.15
-0.825
83.56
X xen
20:2
28.12
12
0.076
29.56
0.108
73.96
-0.875
83,80
20:4
28.40
0.095
29.94
0.113
73,86
-0.808
83,99
Table PLT models using teachers of varying quality (averages over the three language pairs in each direction). We find that teacher quality correlates with quality of PLT; however, weaker teachers can still improve student quality. terms of inertia properties, these are preserved regardless of teacher quality. Note X->en averages for robustness and consistency to misspelling include only de,ru->en
stability results, both PLT models have significantly fewer segment-level regressions than baseline models. For quality, this is most pronounced for en en->de,ru (~50%--100% relative NFR reduction). In contrast, the effect of distillation is not consistent across the language pairs or the two test sets.
5.2 TEACHER QUALITY
In previous sections, we found that quality and model inertia improved when using PLT regardless of the source of the data. In this section, we examine another dimension which distinguishes different flavors of PLT, namely teacher quality. Stronger teachers (teachers with larger capacity than the student) are more common in KD applications whereas identical teacher and student models are the norm in self- tean-rraiiin/oorwarrd translation. Specifically, we vary the base 20:2 teacher architecture by decreasing the number of decoder layers to (weaker teacher) and increasing to 4 (stronger teacher). We keep the student architecture identical at 20:2 layers and fix the source of pseudo- labeled data to the training set (referred to as PLT(TRAIN) in earlier sections).
Interestingly, find that teacher quality does not play large role in model stability (Table 5 There are small improvements in stability and robustness when stronger teachers are used, but gains are in range for all teacher models considered, even for weak teachers. Stronger teachers, however, are responsible for better performing student models. Most surprisingly, we found quality improve- ments over the baseline corroborates other work suggesting that the mechanism behind PLT is not simply that of compress-
when the teacher is of worse quality than the baseline model. This
ing better performing (sets of) models (Furlanello et al. 2018 Hahn & Choi 2019 Yuan et al. 2020
DISTRIBUTION SIMPLIFICATION
The previous section showed that PLT increases both quality and model inertia under different mono- lingual data and teacher quality settings. We hypothesize that the increased inertia observed is cor- related with a distribution simplification mechanism: PLT leads to simpler training data, resulting in models that are less brittle. We test this by comparing PLT with other techniques used to improve quality and smoothness, but that may not have distribution simplification effect. Below, we fix the source of pseudo-labeled data to the training data and test:
BT: tracsstrans,,,,n, a commonly used semi-supervised method that adds parallel data obtained through translation of target data with reverse-directio MT model.
BPE-DROPOUT: A regularization method that has been shown to improve robustness to noise (Provilkov et al. 2020). We used dropout rate of 0.1 as recommended by the authors.
PLT(SAMPLE): A variant of PLT where vary the decoding strategy and perform sampling de- coding which leads to more complex data and weaker student models (Zhou et al.) 2020). Specif- ically, sampled the top-8 hypotheses.
In previous work, Zhou et al. (2020) proposed conditional entropy measure of training data com- plexity and showed that non non-autoregressive translation performance dependent on simpler training data distributions, such those obtained with SKD. Here, we use the same entropy-basee measure.
8

---

Published as a conference paper at ICLR 2023
LP
Setting
C(d)
BLEU
BLEURT
Stability
(OMM(()MEC))
Rob(Missp)
Const(Missp))
BASELINE
3.74
25.12
-0.136
62.20
71.57
-0.807
65.77
BT
3.64
25.96
-0. -0.140
65.36
77. 16
-0.934
64,83
>X
BPE-DROPOUT
3,90
24.82
-0.154
61.70
78.69
-0.547
71.86
PLT(SAMPLE)
3.56
25.96
-0.119
71.97
80.41
-0.865
69.87
PLI(TRAIN)
3,54
26.12
40.117
74.10
80.73
-0.868
70.38
BASELINE
3,12
28.12
0.076
61.91
-0.844
80.53
3.01
64.91
X-ten
BPE-DROPOUT
3.41
28.19
0.075
61.73
-0.591
84,96
PLT(SAMPLE)
2,98
72.69
83.70
PLT(TRAIN)
2.94
29.56
0.108
73.96
-0.875
83.80
Table 6: Quality and model inertia with PLT versus other methods (averages over the three language pairs in each direction). Stability to model updates is computed w.r.t. to random seed variation in student models. X-+en averages for robustness and consistency to misspellings involve de
For each setting, we (1) compute an alignment model on the training data using fast_alige ign (Dyer al. 2013), (2) use align sample of the training corpus, and (3) compute the entropy of the aligned data, leading to: C(d)
[V. Erev, Eylzzagn.. (v)(v)z), where y is the sequence of train- ing data tokens and Falign the sequence of source-side tokens that y tokens are aligned to. Lower entropy indicates that the data is explained by a simpler word-to-word translation model that uses similar word translations inrepecctive of context.
Results are shown in Table 6 First, we observe that the complexity scores confirm the results reported by Zhou et al. 2220 with smaller-scale differences due the fact that mix both original data and pseudo- labeled data. BPE-DROPOUS performs best on smoothness w.r.t. synthetic noise: it outperforms all methods by large margin on robustness, and by smaller margin on consistency. This is not the case on data with natural noise (GMEG), where the increased consistency effect is smaller w.r.t. the BASELINE model. On other metrics, DROPPOTT has no effect on quality (BLEURT) and minor negative effect on stability across re-training. BPE-DROFOUT is not only the only method that lowers stability, but also the only method that increases the complexity of the data compared to the baseline.
BT shows data simplification effect, mirrored by increased stability when re-training. However, BT has detrimental effect on robustness and consistency metrics. These results indicate that while back-translation and forward translation are typically seen as very similar methods, they have dif- ferent properties. PI.T( PLTT(SMPPLE performs very similarly to PLI(TRRAIN:: when compared with PLT(TRAIN), leads to slightly more complex data, and slightly worse quality and inertia scores. PLT(TRAIN) shows the lowest complexity scores and the highest stability.
While stability and complexity correlate, not all methods that simplify the data improve smoothness; conversely, smoothness to synthetic noise can be improved significantly with complementary meth- ods such as 3PE-DROPOUT We corroborate Niu et al. (2020) and find that synthetic and natural noise are different in nature and not all methods are equally effective on both types of noise.
CONCLUSION
This paper investigates pseudo-label training, a technique common to a number of methods for boosting NMT performance. We show that in addition to well-studied gains in generic translation quality, pseudo-label training induces several desirable stability-related properties, which we group under the term inertia. Empirically, these improvements are not tied to the use of unlabeled data (as in self-training) or the use of stronger teacher models (as in knowledge distillation) but are a consequence of the use pseudo-labeled data itself. When compared with other methods designed to improve robustness in NMT, we observed that the effect on stability over re- re-training occurs only for those methods that simplify the training data. Based on these findings, we recommend using PLT with unlabeled data (à la self-training) when developing NMT models where inertia is important due to its benefits to model inertia and its use in addressing potential language coverage bias (Wang et al.) 2021). In future work, we plan to investigate the interplay between PLT and different formulations of NMT (auto- vs. non non-autorrgressive MT) as well as potential negative side effects such as bias amplification (Renduchintala et al., 2021). Finally, developing automatic metrics to detect negative flips in NMT is an important task that has yet to be examined extensively and can help guide PLT techniques.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Yi Zhang and Elman Mansimov for discussions on negative flips and Miguel Ballesteros, Surafel Lakew, Cuong Hoang, and anonymous reviewers for their comments and suggestions.
REFERENCES
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondfej Bojar, Rajen Chatter- jee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsab-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi. Philipp Kochn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pp. 1-88, Online, November 2021, Association for Compu- tational Linguisties. URL httpp:::/accaatthooottt///
2021. wnt 11.
Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Emilie Devijver, and Yury Maximov. Self- training: A survey, ArXiv, abs/2202.120400 2022.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas. Leopoldo Pla Sempere, Gema Sancrez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. ParaCrawl: Web-scale acquisition of parallel cor- pora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4555, Online, July 2020. Association for Computational Linguistics. doi: 10. 18.531///v1 2020.acl- m17n.417. URL httpt:ttt///ett wwwaaccaanntoooooooooooooo wtt.:org//22c....ooo main. 417
Loic Barrault, Ondfej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 con- ference on machine translation (WMT19). In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, Au- gust 2019. Association for Computational Linguistics. doi: 10. 10.18663/11111955301.. URL https: ttt.ps://aclanthooo ogy .rg.org.... 9-1913011
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations, 2018. URL https:
//openrevi ew. net. htt.ee/ooormmm..d..a.ee.ee.
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. CoRR, abs/1206.5533, 2012. URL
5533
Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (WMT18). In Pro- ceedings of the Third Conference Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10. 18653/1// W18-640) URL htt////aaaaaaccaarrr////// /B 6401
Machine Translation: Shared Task Papers, pp. 272-303,
Deng Cai, Elman Mansimov, Yi-An Lai, Yixuan Su, Lei Shu, and Zhang. Measuring and reducing model update regression in structured prediction for NLP. CoRR, abs/2202.02976, 2022. URL
/200.. 02976
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.). Semi-Supervised Learning. The MIT Press, 2006. ISBN 9780262033589, URL
books/ lect ons Cw/CCC222000 html
Anna Currey, Prashant Mathur, and Georgiana Dinu. Distilling multiple domains for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 4500 4511, Online, November 2020. Association for Computational
10

---

Published as a conference paper at ICLR 2023
Linguistics. doi: 10.188553/1122220eemm http:///222220mmmmmmmmmm.mmm.6.. URL BtLLhtsss//aalaathoooooyy/
2020. www.p0mammpppm..mmmmm.mm.mmmma.m 364
Tobias Domhan, Michael Denkowski, David Vilar, Xing Niu, Felix Hieber, and Kenneth Heafield. The sockeye 2 neural machine translation toolkit at AMTA 2020. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume Research Track). pp. 110-115, Virtual, October 2020. Association for Machine Translation in the Americas. URL httips:/acclantholoyy.ooo retp://or.o.......maeeeeeeeeee researchheeehhh
2020
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparame- terization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644 648, Atlanta, Georgia, June 2013. Association for Computational Linguisties. URL http:///w..mttoooooo.oor////---0
Sergey Edunov, Myle Ott, Marc' Aurelio Ranzato, and Michael Auli. On the evaluation of ma- chine translation systems trained with back-translation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 2836 2846, Online, July 2020. As- sociation for Computational Linguistics. doi: 10. 10.18653//1/2020..oo acaiammmmm2...mmmmm.2.......3 253. URL https:
//aclanthology http:/hhollly...g//200.0...0 http::/....r//2...lalmmme.....m. 253
Marzich Fadaee and Christof Monz. The unreasonable volatility of neural machine translation mod- els. In Proceedings of the Fourth Workshop Neural Generation and Translation, pp. 88-96, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18663/ URL
Stanley C. Fralick. Learning to recognize patterns without teacher. IEEE Trans. Inf. Theory, 13: 57-64, 1967.
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural ma- chine translation. ArXiv, abs/1702.01 802, 2017.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Born- agg-a neural networks. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1602- .ww.mngsmmmmrrrrr.mr..m...m. mlr wwt...aww...e..ee.ee....mmmme.ee
1611. PMLR, 2018. html
wwwps/veev00S0 fur Eurlaneliol8a.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014. WRLhhtps:: hnttps://arxivv wrg 6672
Mitchell A. Gordon and Kevin Duh. Explaining sequence-level knowledge distillation as data- augmentation for neural machine translation, 2019.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision I 1789 mar 2021. doi: 10.1007/ s1 11123-021-01453-22 URL UtLLIttps::/dbb.oo////13 htwp::///..111111111111000011111
Yves Grandvalet and Yoshua Bengio. Semi-superrvised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL /ttps://proceeeiiggs.ott newriess cc/pppper/204//Elle/
pdf
Jiatao Gu, James Bradbury, Caiming Xiong, Victor Li, and Richard Socher. Non-autoregressivve
neural machine translation. In International Conference on Learning Representations, 2018. URL
Sangchul Hahn and Heeyoul Choi. Self-knowlledee distillation in natural language processing, 2019.
URL
Junxian He, Jiatao Gu, Jiajun Shen, and Mare Aureloo Ranzato. Revisiting self self-training for neural sequence generation. In Proceedings of ICLR, 2020. URL httpp:://peerevveew....
Forrum?:d==SJddnAkDD

---

Published as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL htttpt//p//axi org/ /a////b//aaa 1503. 02531
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back- translation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Ma- chine Translation and Generation, pp. 18-24, Melbourne, Australia, July 2018, Association for ogy.
Computational Linguistics. doi: 10. 12118//1//118827000 URL.
arg/w18-2700
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 2021-2033 Copenhagen, Denmark, Septemher 2017. Association for Computational I Linguistics. doi: (0.18653/11/D17-111
URL
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. Training on syn- thetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy Ussergeenerate Text (W-NUT 2019). pp. 42-47, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DI 1/D19///0 URL
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep encoder, shal- low decoder: Reevaluating non-autoregressive machine translation. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16 1139. URL http:s:://ccaaaanolooo..ooooooet
1139
Young Jin Kim, Marcin Lunczys-Dowmunt Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Ro- man Grundkiewiez, and Nikolay Bogoychev. From research to production and back: Ludicrously fast neural machine translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 280-288, Hong Kong, November 2019. Association for Computational Linguis- tics. doi: 10.18633v1/D19-10
URL
Tom Koemi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dommunt Hitokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Transla- tion, pp. 478 494, Online, November 2021. Association for Computational Linguistics. URL
https: aacllaaiii aataaaathhooyo www.oogg..ooooooooo.o..o.......o 2021 wii 11.7
Taku Kudo. Subword regularization: Improving neural network translation models with multi- ple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1/853//11118-1100000 URL https: Crc.lanthology.ogg/P11-3
Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Miguel Pino, and Hassan Sajjad. Findings of the first shared task on machine translation robustness. CoRR, abs/1906. 1943, 2019. URL http://
wwxiv.org/abs///e www.aa.ss9999... 11943
Pranava Madhyastha and Rishabh Jain. On model stability as function of random seed, 2019. URL bttps::/aaxiv..rog//
10447
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. Tagged back-translation revisited: Why does really work? In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 5990-5997, Online. July 2020. Association for Computational Linguis- URL https: /ttps://aalaatthooooog..o orgp:/0000...//220.0
tics. doi: 10.18653/v
aac aal-maing 1..main.53..
12

---

Published as a conference paper at ICLR 2023
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Transla- tion, pp. 688-725, Online, November 2020. Association for Computational Linguistics. URL https: a/cl antho ogw.ooog................r..gg wmt 1.77
wmt
Antonio Valerio Miceli Barone, Jindřich Helel, Rico Sennrich, Barry Haddow, and Alexandra Birch. Deep architectures for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pp. 99-107, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/11/11--7700, URL. httphtstssccclaaaee
ogy.
org/R17-4710
Makoto Morishita, Jun Suzuki, and Masaaki Nagata. JParaCrawl: A large scale web-based English- Japanese parallel corpus. In Proceedings the 12th Language Resources and Evaluation Confer- ence, pp. 3603-3609, Marseille, France, May 2020. European Language Resources Association. ISBN 179-144-5546-3444 URL
http://tttttoooooooo/oeeeeeeeeee 200. 1rec 1ereccc-.4440 443
Courtney Napoles, Maria Nädejde, and Joel Tetreault. Enabling robust grammatical error cor- rection in new domains: Data sets, metrics, and analyses. Transactions of the Association for Computational Linguistics, 7:551-566, 2019. doi: 10.1 a 00282. https:
Xuan-Phi Phi Nguyen, Shafiq Joty, Kui Wu, and Ai Ti Aw: Data diversification: A simple strategy for neural machine translation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10018-10029, Cur- ran Associates, Inc., 2020. htLL/tttps:/prrceeeeee mneurpi
fi
Xing Niu, Prashant Mathur, Georgiana Dinu, and Yaser Al-Onaizan. Evaluating robustness to input perturbations for neural machine translation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 5538-854,, Online, July 2020, As- URL https:
sociation for Computational Linguistics. doi: //aclanthology. ctgp acl-main. 755
1/20200//
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. CoRR, abs/1511.0445 2015. URL http:: //blpp. uni-trier. /blp: de/db/ http://ere..eeee//rraaaa////rro PapernotMMUS15
html#
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3 10:3115/107333331073333 URL https:
//aclantholoogy.oom www.ooy...../o00/////00
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo- ber 2018. Association for Computational Linguisties. doi: 10. 10.18653//1//118-3319 URL
https::/aclLatthloos
org/ W18-6319
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguisties. URL httptttt////aaaahoo.oooo wwwp:eoy......11ooeeee. (eorg/EE7-2200
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout: Simple and effective sub- word regularization. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pp. 1882-1892, Online, July 2020. Association for Computational Linguis- tics. doi: 10. 12033//1/200.acmmmaaaa 170. URL https: / antho logy. or2/2020. acl acl-main. 170
R. Pryzant, Y. Chung, D. Jurafsky, and D. Britz. JESC: Japanese-Englishe Subtitle Corpus. Language Resources and Evaluation Conference (LREC), 2018.
13

---

Published as a conference paper at ICLR 2023
Nils Reimers and Iryna Gurevych. Reporting score distributions makes difference: Performance study of sstt-networss for sequence tagging. In EMNLP, 2017.
Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, and Mona Diab. Gender bias amplification during speed-quality optimization in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference Natural Language Processing (Volume 2: Short Papers). As- sociation for Computational Linguistics, 2021. doi: 10.18653/v1/2021 aatp///222..lllsshooa 15...shor..15. URL
https: waa.li.org//000 1865382Fv &2F2021 acl short.orrro 15
Holger Schwenk, Vishrav Chaudhary. Shuo Sun, Hongyu Gong, and Francisco Guzmán. WikiMa- trix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguis- tics: Main Volume, pp. 1351-1361, Online, April 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2011 wttp://2221eeeelllleee leacl- moce-mmai.11555
URL https: / /acl anthe org/2021.
eacl ma mmmii... 115
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguis- URL https: //cc aotthelogy.org/20000
tics. doi: 10. ac mmmim.ann..m 704
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86-96, Berlin, Germany, Au- gust 2016a. Association for Computational Linguistics. doi: 10. 10.18653/1//11--009 URL
pettps:/accantton
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016b. Association for Computational Linguistics. doi: 10. (0.18653/11/116-1162, URL https:
//aclanthology: wwtp://-....P1111....1111
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compaatbble rep- resentation learning. In CVPR, pp. 6367-6376. IEEE, 2020. ISBN 978-1-7281-7166-5. URL
http::/dblp.unt undbpp.unnii. commi-traie trrierreeeeeeerr /ttp:///////////////////////////
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pp. 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics. URL /tRp:ttps://// http://allaanoool...roooo//////
doi: 10.18553/v
1164
Christian Szegedy. Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http:: arxiv. org/ abs / 1112. 61
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie- Ran Liu. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations, 2019. URL
https:://ppereviiee
net forummid=dde (issiddd=sggssoo0xX
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 5998-6008, 2017. URL http::///pppeeeee......e.....sos #FF5e2243447dee www.5301114444553333333
pdd
Shuo Wang, Zhaopeng Tu, Zhixing Tan, Shuming Shi, Maosong Sun, and Yang Liu. On the language coverage bias for neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 4778-4790 Online, August 2021. Associ- ation for Computational Linguistics. doi: 10.18653/w1/200
aadinngs-acl.422 URL https:
7/acl //Aclannhhllogy.orr//0000 findi ngs .22
14

---

Published as a conference paper at ICLR 2023
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self- self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2021.
Yuqing Xie, Yi-An Lai, Yuanjun Xiong, Yi Zhang, and Stefano Soatto. Regression bugs are in your model! measuring, reducing and analyzing regressions in NLP model updates. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6589-6602, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021
URL httphppppp//.laaattooooo.oooooo.
Weijia Xu, Shuming Ma, Dongdong Zhang, and Marine Carpuat. How does distilled data complexity impact the quality and confidence of -uo-aetoreee machine translation? In Findings of the
Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4392-4400, Online, August 2021. Association for Computational I Linguistics. doi: 10. 18653/ 1201.3////0... findinngs-all.385. URL
findings acl. 385
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Ste- fano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14299- 14308, June 2021.
Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge dis- tillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 3902- 3910. IEEE, 2020. ISBN 978-1-7281-7168-5 doi: 10.1 (D00CCVPRR2200022200000999 URL https: /tt::///doi.oo org/10 1109/11000 www.0///VVR22200.0 2020. 00396
Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language
Processing, pp. 1535,1545, Austin, Texas, November 2016. Association for Computational Lin-
guistics. doi: 1/66533/11D11111660 1160. URL:/tttps//aellatthoooooooo
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020.
URL htttps://openreviest net forum So-rrummidd-SKkHuCVVVD..
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non-
autoregressive machine translation. In 8th International Conference on Learning Representattions ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ner, 2020. URL https:
//openrevies ew. wwt.//e.toutee iisorummid=Byye http://oommmaaayyaa///aaaaa d=
Michat Ziemski, Marein Junczys-Dowmunt, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pp. 3530-3534, Portorož, Slovenia, May 2016. European Language Re- sources Association (ELRA), URL https: https://aclamhhologgg oww.ogg.....ogoeeee 1561
A TRAINING PARAMETERS
All models used in our experiments utilized the following of hyperparameters. Training and development data was tokenized using the Sacremoses tokenizer Words were segmented using BPE (Sennrich et al. 2016b) with 32K operations. Source and target subwords shared the same vocabulary. Training segments longer than 95 tokens were removed.
The source embeddings, target embeddings, and the output layer's weight matrix are tied (Press & Wolf 2017). Training is done on 8 GPUs with Sockeye 3's large batch training. It has an effec- tive batch size of 327,680 tokens, learning rate of 0.001 13 with 2000 warmup steps and a reduce
15

---

Published as a conference paper at ICLR 2023
rate of 0.9, checkpoint interval of 125 steps, and learning rate reduction after 8 checkpoints with- out improvement. After an extended plateau of 60 checkpoints, the checkpoints with the lowest validation perplexity averaged to produce the final model parameters.
Parameters for standard training:
Teaamning_ratt__shhhulleeee_____ : 'inv-sqrt-decay', 'keep_lase
10, 'upcate_intterval': 16, transformer_modessseee (512, 512), "transfemmee_psssprrossss ('dr', 'dr'), "Learning_rate_aarnnn 2000, "transfocmer_ctopouttetttt (0.1, 0.1), 'max imax_num_cheeeeeeemm www__nnn_ccccciiinncc hts____mm_ccccppppooooooo_iiiepe ttramnsffomre__eeeeeeooaarrrrrrr horrrurroorarrnnmmmiddddddddd (2048, 2048),
aaaiiittavee__aettteaaa 60, "cache_strategy" 'best',
'avg', 'optimized_metric'' 'perplexity',
'num 'num_layers': (20, 2), "use_cpu': False,
0.001, 'device_ids's [-1), 8, 'initial_learne "nttiall_eerrmnngrateee (,ch ppap::::ppppppxxxxppppe 'perplexity',
0.06325, 'seed: 1, 'cache_metric'.
gradiient__cllppinngteee"" 'abs',
8,
"weight_init__scaee' 3.0, 'dtype': 'float32', d 500, max
1036800, 'amp': True, 'keep 'keep inteiaaizaaai True, 'transformer saapouutprrepott'. (0.1, 0.1),
'sro_trg_softmax' h (8, 8), 'weight_ Beiing_type'.
0.9, 'loss'; "cross-entropy', 'horovod': Truc, 'num__mbed': (512, 512), 'mbee_ddropout's (0.0, 0.0), "transformer_proprocce""
('n', 'n'),
'encoder': 'transformer', 'loglevel sogllvee__eccondryy_oooeerrs 'ERROR', '1abel_smocthing'. 0.1, 'batch_size!" 2500, seaemiiggrraae___ccllll 1.0, 'batch_type': 'max-word', 'eptimizer': 'adam',
(0.1, 0.11, 'decoder';
"ssru_transformee wwwiiimmmneepnaaaeeaaaaaa nww_apcchsss.saeemmom 1, beaeckooint_intervvaee 500,
type'; 'tixed', "Lock_dix" 1/dat'', 'gradient__llppinnghhhhhhooee -1.0, 'weight_init': 'xavier', 8,
'no_yyrrddzztton'' False, parcchssentenee_mltiii "rransormer_aaciitttion ('relu', 'relu')
B DATASET
We trained enen<>de models on Paracrawl (Bañón et al. 2020), WikiMatrix (Schwenk et al.
2021), WikiTitles (Bojar et al. 2018), and news commentary datasets (Barrault et al. 2019). For en<+ru we additionally added the UN v1.0 dataset (Ziemski et al. 2016). For en<+ja we used JParaCrawl (Morishita et al.] (020) instead of ParaCrawl v9 and additionally added the Japanese- English subtitles dataset (Pryzant et al. 2018).
LP Datasets
parallel 286
en--de en++ja
Paracrawl v9, WikiMatrix, WikiTitles, news commentary
JParacrawl v2. WikiMatrix. WikiTidles, news commentary, Japanese-Englishh subtitles
17.2 M
ent+ru
Paracrawl, WikiMatrix WikiTitles, news commenatry, UN v1.0
34 M
Table 7: We trained our models on subset datasets from the WMT21 news task. Specifically, used Paracrawl v9 (Bañón et al. 2020), WikiMatrix (Schwenk et al. 2021). WikiTitles (Bojar
et al. 2018), news commentary, UN v1.0 dataset (Ziemski et al.) 2016), JParCCrawl (Morishita al. 2020 and the Japanese-English subtitles datasets (Pryzant al. 2018
LP
Years
parallel
en++>de
2017-2020
9k
en++ja
2020
2k
en++ru
2017-2020
9k
Table 8: We used the WMT news test datasets from previous years our development set.
16

---

Published as a conference paper at ICLR 2023
C TRAINING CURVES
Here, we compare pseudo-label training with back-translation (BT). We find that pseudo-label train- ing regularizes the models by controlling for over fitting. BT also regularizes the model, but it does not simplify the distribution to the extent PLT does, implying that controlling over-fitting is not main factor for stability. Comparisons with other methods (i.e. BPE-dropout and PLT(sample)) show similar trends.
Er-XX
X-4Ea
Exadits
330
205
Figure 1: Comparisons of PLT((rain) validation (solid lines) and training curves (dashed lines) against back-translation and baseline models. We find that in comparison, PLT is able to control over fitting the training data. Back-translation also regularizes the model, but does not simplify the distribution to the extent PLT does, implying that controlling over-fitting is not main factor for stability.
D BERTSCORE
We provide quality scores using BERTScore (Zhang et al. 2020). In terms of generic quality, PLT provides improvements in quality consistent with earlier results using BLEU and BLEURT metrics (see Tables 5 5nd We also computed robustness metrics BERTScore:
Robustness = BERTScoreeY's Yres) BERTScore Yref)
where Yrey stands for reference translations and Y, Y' are translations of a clean/noisy versions of the test set (e.g., with introduced misspellings).
17

---

Published as a conference paper at ICLR 2023
Misspellings
GMEG
LP
Setting
Teacher
Student (Original PL)
BLEU
BERTScore
Rob
Coust
Const
AATA
286M +
27.83
0.N60
-0.016
73.9
83.1
BASELINE
143M O
27.62
0.860
-0.016
73.120.9
82.8
CD-rde
PLT(TRAIN)
143M
143M 143M (Truin)
27.86
0..002
-0.017 20.001
76.5
85.1
PLT(UL)
143M
143M + 143M (UL.)
28.07
0.861 40.022
0,018
76.8.0.5
85.3
ALL.DATA
34M +
25.71
0.856
0..18
66.421.;
79.3
17
25.08
0.854
-0.019
20.001
64.9
78.4
PLT(TRAIN)
17M
17M 17M (Train)
26.16:1.1
0.857
-0.017
=0.017
70.2
81.5
PLT(UL)
17M
17M + 17M (UL)
25.87
0..07
-0.017 #0.001
70.1
82.2
ALLDATA
17.2M O
23. 23.62
0.836
=0.020
1
72.6
BASELINE
8.6M O
22.82.20.9
0.833
=0.020
59.3
71.5
-ja
PLT(TRAIN)
8.6M
8.6M 8.GM (Train)
24.63
0.839 1.839
1.000200.00
64.5
75.9
PLT(UL)
8.6M
8.6M 8.6M (UL)
24.59 #0.9
0.839 +0.001
-0.021 #0.001
64.5
75.9
ALLDATA
286M + O
32.32
0.953+0.001
-0.012
77. 77.4g1o
de de-
BASELINE
143M
32.36
0.953+n.001
=0.012=0.001
76.9
PLT(TRAIN)
143M
143M 143M (Train)
32,93
0.953
g0.01
80.0
PLT(UL)
143M
143M + (UL)
32.50
0.953
00013
$0.2
ALLDATA
34M O
36.39
0.950
20.000000000
85,620.8
BASELINE
17M +
36.2611.2
0.956.0.001
-0.005
84,2
nu-ten
PLT(TRAIN)
17M
17M + 17M (Train)
37.00
0.957
20.00000000
87.6 87.6
PLT(UL)
17M
+ (UL)
36.85
0.907
-0.004.0.000
87.0.0.7
Table 9: Training data sizes and performance scores for PLT/BBaseiiee models. Quality is measured with BLEU and BERTScore on the WMT21 news test set. Smoothness is measured as robustness and consistency to synthetic (Misspellings) and natural (GMEG) noise. GMEG scores are computed as the average over four reference corrections. Robustness measures changes in translation quality w.r.t input variations, while Consistency measures translation changes alone.
Teacher
1.P
Arch.
BLEU
BERTScore
BLEU
BERTScore
Stability
Const(GMEG)
Robi
Consi Missp)
25.12
0.845
62.20
77.57
-0.018
65.77
20:1
24.08
0.845
25.54
0,850
72.67
79.93
-0.018
69.72
en-+X
20:2
25.12
0.849
26.12
0.852
74.10
80.72
-0.018
70.38
25.63
73.68
28.12
0.937
61.91
-0.009
80.53
20:1
26.74
0.932
28.30
0.936
73.15
-0.008
83,56
X-ven
20:2
28. 12
0.937
29.56
0.939
73.96
-0.009
83.80
20:4
28.40
0.937
29.94
73.86
-0.008
83,99
Table 10: PLT models using teachers of varying quality as measured by BERTScore (averages over the three LPs in each direction). We find that teacher quality correlates with quality of PLT; however, weaker teachers can still improve student quality. In terms of inertia properties, these are preserved regardless of teacher quality. Note X->en averages for robustness and consistency to misspelling include only de.ru-ten.
LP
Setting
C(d)
BLEU
BERTSScoee
Stuiliity
Const(GMEG)
Rob(Missp)
Consi
BASELINE
3.74
25.12
0,849
62.20
77.57
-0.018
65.77
BT
3.64
25.96
0.851
65.36
77.16
-0.021
64.83
e
BPE-DROPOUT
3.90
24.82
0.847
61.70
78.69
40,011
71.86
PLT(SAMPLE)
3.56
25.96
0.852
71.97
80.41
-0.018
69.87
PLT(TRAIN)
3.54
26.12
0.852
74.10
80.73
-0.018
70.38
BASELINE
3.12
28.12
0.936
61.91
-0.009
80.33
BT
28.88
0.938
64.91
-0.010
80.00
X->en
BRE-POOPOUT
3.41
28.19
0.937
61.73
40.006
84.96
PLT(SAMPLE)
2.98
29.41
0.939
72.69
-0.009
83.70
PLT(TRAIN)
2.94
29,56
0,939
73.96
-0.009
83.80
Table 11: Performance and model inertia with PLT versus other methods (averages over the three LPs in each direction). Stability to model updates computed w.r.t. random seed variation in student models. X-+en averages for robustness and consistency misspellings involve de.ru-ven.
18


Reviews:
[{"review_id": "irvNC7tY6ip", "reviewer": "Reviewer_ZSXu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors do research on pseudo-label training that is also known as knowledge distillation or teacher-student training, which transfers task knowledge learned from a larger model to a smaller model by letting the smaller model mimic the teacher's outputs rather the gold labels during the training. They study \"model inertia effects\" of the pseudo-label training and discuss how to address the robustness training in NMT models. The NMT models are trained on six different language directions and assessed in a variety of automation metrics to capture robustness and consistency, in addition to the n-gram match accuracy like BLEU scores.", "summary_of_the_review": "The paper provides substantial experimental results and discuss inertia effect in the different MT task settings. The paper came up with a lots of observation and suggestions in a carefully designed experiments, however, it does not end up with any novel approaches or concrete conclusions for further improvement. We probably would like to have more theoretical proof or explanations behind the observations. ", "strength_and_weaknesses": "Strengths\n- Extensive experimental results to assess the NMT models and pseudo-label training in different scores\n- These analytic report would be useful to understand the impact of pseudo-label training\n\nWeaknesses\n- Lots of useful observation from the extensive experiments, however, there is few proposed approaches. Or, theoretical proof would be appreciated. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RMXMfN8U3_C", "reviewer": "Reviewer_Ypio", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper essentially reframes NMT models’ stability to model updates and input perturbations as *model inertia* and conduct various analysis experiments on it in the  pseudo-label training setups, aka forward-translation distillation. The paper shows that quality gains in different methods are due to the use of pseudo-labeled data in training.", "summary_of_the_review": "Overall, I think the paper is written with clear presentation. However, I think the analysis of model inertia is largely expected, and does not provide key learnings as there is no suggestions or conclusions on how to improve NMT model with this knowledge.", "strength_and_weaknesses": "Strength\n\n* The paper seems quite clear and easy to understand.\n* The experiment results are decent\n\nWeakness\n\n* Lack various comparisons with related methods that try to do the same, such as robustness training against adversarial attack.\n* The paper doesn't conclude or provide any suggestions or implication from the results so that it is hard to find the key takeaways from the paper, and it's difficult to imagine what we can do about the knowledge from the paper. For example, the paper should show experiments that key learning from analysis of model inertia helps us develop better model that perform better. I do not see any of such indication in the paper. And thus, I find it difficult to find this useful.\n* Missing citations of some pseudo-label training forward translation methods, such as data diversification (Nguyen et al., 2020, NeuIPS)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GV5t1_rkPMb", "reviewer": "Reviewer_ixEF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper broadens the understanding of pseudo-label training (PLT) in neural machine translation (NMT). It finds that PLT can not only improve model performance but also enhance model stability. ", "summary_of_the_review": "This is an interesting paper that investigates the relationship between PLT and model inertia. However, it does not provide enough insights to guide future research. Therefore, I recommend rejection at this moment.", "strength_and_weaknesses": "Strength:   \nOverall, PLT is a simple and effective method for improving the model performance of NMT. The understanding of PLT is an important topic. This paper gives some insights that might be useful.\n\nWeaknesses:\n1. The main weakness is that the analysis of the model inertia just focuses on the evaluation of model results but ignores the analysis of the pseudo-labels themselves. Which kind of characteristics of the pseudo-labels affect model inertia? For example, previous works have shown that NMT models are more likely to make frequent predictions, monotonic translations, and so on. In other words, why simplifying the training data can bring such benefits? The core questions are not well answered.\n2. After carefully reading the paper, I still cannot find how can the paper findings guild future research. Could we design better PLT methods based on the findings, and how? Or could we better train NMT models with PLT?\n3. I strongly suggest the authors include a comparison of non-autoregressive models, which might attract more attention from the community.\n4. In 4.1, COMET or BERTScore could be a better choice for evaluating the robustness or consistency.\n\n---\nThank you for the response that has alleviated my part concerns. I would like to increase my score a bit. However, similar to Reviewers ypio and zsxu, how can the paper's findings guild future research is still unclear which limits me from giving a higher rating.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-edVVJ3p_s", "reviewer": "Reviewer_fgwu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Pseudo-labels are widely used in practical NMT, either in a classical (sequence-level) KD scenario or for forward translation. The paper demonstrates that both forms of pseudo-labels improve the sanity of NMT by showing that models are both more robust against input perturbations and less prone to variations between training runs due to random initialization. Experiments on 6 language pairs using existing robustness measures and new model stability metrics as well as human evals support their claims convincingly.", "summary_of_the_review": "A solid and sober paper, but not the most exciting one.", "strength_and_weaknesses": "The paper makes a pretty straightforward point: Pseudo-labels improve the robustness of NMT inference and the stability of NMT training. This point may not be overly surprising to MT practioners. But that doesn't mean that it is a useful contribution to demonstrate this as thoroughly and clearly as this paper.\n\nI don't see any major flaws, but here are some minor points:\n- Where applicable, significance tests in Sec. 5.1 would be good.\n- I'd encourage the authors to add some citations on faithfulness / robustness in NMT. Things like https://arxiv.org/pdf/2005.12398.pdf also come to mind, as well as the renewed popularity of the lottery ticket hypothesis.\n- \"NMT lacks a reliable automatic segment-level quality metric\" There is evidence that neural MT metrics (BLEURT, COMET, ...) are more reliable on the sentence level than BLEU.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster