Revisiting Self-Training for Neural Sequence Generation
Authors: Junxian He
Jiatao Gu
Jiajun Shen
Marc'Aurelio Ranzato
Venue:  (2020)
Keywords: self-training
semi-supervised learning
neural sequence generatioin

Abstract: Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that self-training is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a noisy version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

Published as a conference paper at ICLR 2020
REVISITING SELF-TRAININ FOR NEURAL SEQUENCE GENERATION
Junxian He Carnegie Mellon University junxianhe cs cmu edu
Jiatao Gu', Jiajuun Shen, Marc'Aurees Ranzato Facebook Al Research, New York, NY {igu, jia ren.ata)aaa.accccccoomm
ABSTRACT
Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the eveudo-paralle, data). While self-training has been ex- tensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self self-training works due the com- positionality of the target space. In this work, we first empirically show that self- training is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) critical for self-training to benefit from the aseudo-paralle, data, which acts as regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions courage this mechanism, we propose to inject noise to the input space, resulting in "noisy" version of self-training. Empirical study standard machine trans- lation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin
unlabeled data. To further en-
INTRODUCTION
Deep neural networks often require large amounts of labeled data to achieve good performance. However, acquiring labels is a costly process, which motivates research on methods that can ef- fectively utilize unlabeled data to improve performance. Towards this goal, semi-supervised learn- ing (Chapelle et al. 2009) methods that take advantage of both labeled and unlabeled data are a natural starting point. In the context of sequence generation problems, semi-supervised approaches have been shown to work well in some cases. For example, back-translation (Sennrich et al. 2015) makes of the monolingual data the target side to improve machine translation systems, latent variable models (Kingma et al.) 2014) are employed to incorporate unlabeled source data to facilitate sentence compression (Miao & Blunsom 2016) or code generation (Yin al. 2018).
In this work, we revisit a much older and simpler semi-supervised method, self-training (ST, Scudder (1965)), where base model trained with labeled data data, which is then used to augment the original small training set. Then, a "student" model is trained with this new training set to yield the final model. Originally designed for classification problems, common wisdom suggests that this method may be effective only when good fraction of the predictions on unlabeled samples are correct, otherwise mistakes are going to be reinforced (Zhu & Goldberg 2009). In the field of natural language processing, some early work have successfully applied self-training to word sense disambiguation (Yarowsky) 1995) and parsing (McClosky et al. 2006 Reichart & Rappoport 2007
"teacher" label the unannotated
Huang & Harper 2009).
However, self-training has not been studied extensively when the target output is natural language.
This is partially because in language generation applications (e.g. machine translation) hypotheses are often very far from the ground-truth target, especially in low-resource settings. It is natural to
"Equal Contribution. Most of the work is done during Junxian's 's internship at FAIR.
Code available at http:://ghaabb.oo////////raaaann

---

Published as a conference paper at ICLR 2020
Algorithm 1 Classic Self-training
1: Train a base model So on = (mi, y 1=
2: repeat
3: Apply fe to the unlabeled instances U
4: Select a subset S C {(x, fx(x))( U}
5: Train a new model fe on SUL
6: until convergence maximum iterations are reached
ask whether self-training can be useful at all in this case. While Ueffing 2006) and Zhang & Zong (2016) explored self-training in statistical and neural machine translation, only relatively limited gains were reported and, to the best of our knowledge, it is still unclear what makes self-training work. Moreover, Zhang & Zong] (2016) did not update the decoder parameters when using pseudo parallel data noting that "synthetic target parts may negatively inffuence the decoder model of NMT"
In this paper, we aim answer two questions: (1) How does self-training perform in sequence generation tasks like machine translation and text summarization? Are "bad" pseudo targets indeed catastrophic for self-training?
(2) If self-training helps improving the baseline, what contributes to its success? What are the important ingredients to make it work?
Towards this end, we first evaluate self-training on a small-scale machine translation task and em- pirically observe significant performance gains over the supervised baseline (33.2). then we perform a comprehensive ablation analysis to understand the key factors that contribute to its success (33.3) We find that the decoding method to generate pscudo targets accounts for part of the improvement, but more importantly, the perturbation of hidden states dropout (Hinton et al. 2012) turns out to be crucial ingredient to prevent self-training from falling into the same local optimum as the base model, and this is responsible for most of the gains. To understand the role of such noise in self-training, we use toy experiment analyze how noise effectively propagates labels nearby inputs, sometimes helping correct incorrect predictions ($4.1). Motivated by this analysis, we pro- pose to inject additional noise by perturbing also the input. Comprehensive experiments on machine translation and text summarization tasks demonstrate the effectiveness of noisy self seff-traiiing.
2 SELF-TRAINING
Formally. in conditional sequence generation tasks like machine translation, we have a parallel dataset L = {æ., y. and a large unlabeled dataset U = {æ; y ) where |U] > |L] in most cases. As shown in Algorithm [] classic self-training starts from a base model trained with parallel data L. and iteratively applies the current model obtain predictions on unlabeled instances U, then it incorporates subset of the pseudo parallel data S to update the current model.
There are two key factors: (1) Selection of the subset S. S is usually selected based on some confidence scores (e.g. log probability) (Yarowsky 1995) but it is also possible for S to be the
whole pseudo parallel data (Zhu & Goldberg 2009). (2) Combination of real and pseudo paral- lel data. A new model is often trained on the two datasets jointly as in laac-,,,nnlation, but this introduces an additional hyper-parameter to weigh the importance of the parallel data relative to the pseudo data (Edunov et al.] 2018). Another way is to treat them separately first we train the model on pseudo parallel data S. and then fine-tune it on real data L. In our preliminary ex- periments, we find that the separate training strategy with the whole pscudo parallel dataset (i.c. S = (xx, (x E U)) produces better or equal performance for neural sequence generation while being simpler. Therefore, in the remainder of this paper clude quantitative comparison regarding joint training, separate training, and filtering in Appendix
this simpler setting. We in-
data B where separate training is able match (or surpass) the performance of
joint training.
In self-training, the unsupervised loss Lu from unlaheled instances is defined as:
LU = - Ee-p(w) E My-ro- (yl=) log Pe (y)z))
where p(æ) is the empirical data distribution approximated with samples from S, (o(y) is the conditional distribution defined by the model. 0² is the parameter from the last iteration (initially it
2

---

Published as a conference paper at ICLR 2020
23.0
22.0
NST [eavd] peaddolraing
NST tread): fautane
KST [prnpheoi) pos tostriiniinn
NST [prophrod) faotase
Methods
PT
FT
20.4
20.0
baseline
15.6
19.0
18.7
ST (scratch)
16.8
17.9
ST (baseline)
16.5
17.5
18.0
17.9
Table 1: Test tokenized BLEU on WMT100K. Self-training re- sults are from the first itera- tion. "Seratch" denotes that the system is initialized randomly and trained from scratch, while "baseline" means it is initialized with the baseline model.
16.0
15.6 15.6
15.5
15.0
14.0
baseling
Iteration
Iteration
fteration
Figure 1: BLEU on WMT100K dataset from the supervised base- line and different self-training variants. We plot the results over 3 iterations. "ST" denotes self-training while "NST" denotes noisy self training.
set the parameter of the supervised baseline), and fixed within the current iteration. Eq. re- veals the connection between self-training and entropy regularization (Grandvaler & Bengio 2005 In the context of classification, self-training tion [Lee|20133) which favors low-density separation between classes, commonly assumed prior for semi-supervised learning (Chapelle & Zien 2005).
be understood from the view of entropy regulariza-
3 A CASE STUDY ON MACHINE TRANSLATION
To examine the effectiveness of self-training on neural sequence generation, we start by analyzing machine translation task. We then perform ablation analysis to understand the contributing factors of the performance gains.
3.1 SETUP
We work with the standard WMT 2014 English-German dataset consisting of about 3.9 million training sentence pairs after filtering long and imbalanced pairs. Sentences are encoded using 40K byte-pair codes (Sennrich et al.) 2016). As preliminary experiment, sentences from the training set to train the model and use the remaining English sentences as the unlabeled monolingual data. For convenience, setting allows us to have high-quality unlabeled data to verify the performance of self-training. We train with the Base Transformer architecture (Vaswani et al., 2017) and dropout rate at 0.3. Full training and optimization parameters can be found in Appendix A.I All experiments throughout based the fairseq toolkit (Ott al., 2019),
randomly sample 100K
refer to this dataset WMT100K. Such synthetic
this paper including the transformer implementation and all results are in terms of case- sese-sensitive tokenized BLEU (Papineni et al., (2002). We use beam search decoding (beam size 5) create the pseudo targets and report BLEU on test set.
3.2 OBSERVATIONS
In Figure I we use green bars to show the result of applying self-training for three iterations. We include both (1) pseudo-training (PT): the first step of self-training where we train a new model (from scratch) using only the pseudo parallel data generated by the current model, and (2) fine- tuning (FT): the fine-tuned system using real parallel data based on the pretrained model from the PT step. Note that in the fine-tuning step the system is re-initialized from scratch. Surprisingly, find that the pseudo-training step at the first iteration able to improve BLEU even if the model is only trained its predictions, and fine-tuning further boosts the performance. The test BLEU keeps improving over the first three iterations, until convergence to outperform the initial baseline by 3 BLEU points.

---

Published as a conference paper at ICLR 2020
Methods
PT
FT
baseline
15.6
baseline (w/o dropout)
5.2
ST (beam search, w/ dropout)
16.5
17.5
ST (sampling, w/ dropout) ST (beam search, w/o dropout)
16.1
17.0
15.8
16.3
ST (sampling, w/o dropout)
15.5
16.0
Noisy ST (beam search, w/o dropout)
15.8
17.9
Noisy ST (beam search, w/ dropout)
16.6
19.3
Table 2: Ablation study on WMT100K data. For ST and noisy ST, we initialize the model with the baseline and results from one single iteration. Dropout is varied only in the PT step, while dropout is always applied in FT step. Different decoding methods refer to the strategy used to create the pseudo target. At test time we use beam search decoding for all models.
This behaviour is unexpected because no new information seems to be injected during this iterative process target sentences of the monolingual data are from the base model's predictions, thus trans- lation errors are likely to remain, not magnified. This is different from pack-translation where new knowledge may originate from additional backward translation model and real monolingual targets may help the decoder generate more fluent sentences.
One straightforward hypothesis is that the added parudlllarallell data might implicitly change the training trajectory towards a (somehow) better local optimum, given that we train a new model from scratch at each iteration. To rule out this hypothesis, we perform ablation experiment and initialize from the last iteration (i.e. 0*). Formally, based on Eq. we have:
VeLulezo- = -Paappaa Ve Ryr-po (w/x) log (e(yy))))ee=p
O.
(yyk)) (z) matches the underlying data because the conditional log likelihood is maximized when pa distribution pe- (z) Therefore, the parameter O should not (at least not significantly) change if we initialize it with 0 from the last iteration.
Table the comparison results of these two initialization schemes the first iteration. Surpris- ingly, continuing training from the baseline model also yields an improvement of 1.9 BLEU points, comparable initializing from random. While stochastic optimization introduces randomness in the training process, it is startling that continuing training gives such a non-trivial improvement. Next, we investigate the underlying reasons for this.
3.3 THE SECRET BEHIND SELF-TRAINING
To understand why continuing training contradicts Eq. 2 and improves translation performance, we examine possible discrepancies between our assumptions and the actual implementation, and formulate two new hypotheses:
HI. Decoding Strategy. According to this hypothesis, the gains come from the use of beam search for decoding unlabeled data. Since focus is sequence generation task, we decode y with beam search to approximate the expectation in Eey<pe-(yles log Pa (y]z), yielding a biased es- timate, while sampling decoding would result an unbiased Monte Carlo estimator. The results in Taboestdem that the performance drops by 0.5 BLEU when we change the decoding strategy sampling, which implies that beam search does contribute bit to the performance gains. This phenomenon makes sense intuitively since beam search tends to generate higher-qualin pseudo tar- gets than sampling, and the subsequent cross-entropy training might benefit from implicitly learning the decoding process. However, the decoding strategy hypothesis does not fully explain it, as still observe gain of 1.4 BLEU points over the baseline from sampling decoding with dropout.
H2. Dropout HHinton et al. 2012). Eq. [ and Eq. 2 implicitly ignore (seemingly) small dif- ference between the model used to produce the pseudo targets and the model used for training: at test/decoding time the model does not use dropout while training time dropout noise is injected in the model hidden states. At training time, the model is forced to produce the same (pseudo) targets given the same set of inputs and the same parameter but various noisy versions of the
4

---

Published as a conference paper at ICLR 2020
hidden states. The conjecture is that the additional expectation over dropout noise renders Eq. false. To verify this, we remove dropout in the pseudo training step" The results in Table 2 indicate that without dropout the performance of beam search decoding drops by 1.2 BLEU, just 0.7 BLEU higher than the baseline. Moreover, the pseudo-training performance of sampling without dropout almost the same the baseline, which finally agrees with our intuitions from Eq.
In summary, Table 2 suggests that beam-search decoding contributes only partially to the perfor- mance gains, while the implicit perturbation dropout accounts for most of it. However, it is still mysterious why such perturbation results in such large performance gains. If dropout is meant to avoid overfitting and fit the target distribution better the seeudo-training step, why does it bring advantages over the baseline given that the target distribution is from the baseline model itself? ? This the subject of the investigation in the next section.
4 NOISE IN SELF-TRAINING
4.1 THE ROLE OF NOISE
One hypothesis as why noise (perturbation) is beneficial for self-training, is that it enforces lo- cal smoothness for this task, that is, semantically similar inputs are mapped to the same simi- lar targets. Since the assumption that similar in- put should ideally produce similar target largely holds for most tasks in practice, this smooth- ing effect of pseudo-training step may provide favorable regularization for the subsequent fine- tniing step. Unlike standard regularization in pervised training which is local to the real parallel data, self-training smooths the data space covered by the additional and much larger monolingual data.
Methods
smoothness
symmetric
error
baseline
9.1
9.8
7.6
ST
8.2
9.0
6.2
noisy ST
7.3
8.2
4.5
Table 3: Results on the toy sum dataset. For ST and noisy ST, smoothness (L) and symmetric (1) results are from the pseudo-training step, while test errors (4) are from fine-tuning, all at the first iteration.
To verify this hypothesis more easily, we work with the toy task of summing two integers in the range to 99. We concatenate the two integers and view them as a sequence of digits, the sum is also predicted at the digit level, thus this is still a sequence to sequence task. There are 10000 possible data points the entire space, and we randomly sample 250 instances for training 100 for validation, 5000 for test, and 4000 as the unlabeled data. Test errors are computed the absolute difference between the predicted integer and the ground-truth integer. We use an LSTM model to tackle this task. We perform self-training for one iteration this toy dataset and initialize the model with the base model to rule out differences due to the initialization. Setup details are in Appendix A.
For any integer pair (x1, we measure local smoothness as the standard deviation of the predic- tions in 3 x 3 neighborhood of 22). These values are averaged over all the 10000 points to obtain the overall smoothness. We compare smoothness between baseline and ST pseudo-training in Table 3 To demonstrate the effect of smoothing on the fine-tuning step, we also report test er- rors after fine-tuning. We observe that ST pseudo-training attains better smoothness, which helps reducing test errors the subsequent fine-tuning step.
One natural question is whether we could further improve performance by encouraging even lower smoothness value, although there clear trade-off, as totally smooth model that outputs a con- stant value is also bad predictor. One way to decrease smoothness is by increasing the dropout probability in the pseudo-training step, but large dropout (like 0.5) makes the model too unstable and slow converging. Therefore, we consider simple model-agnostic perturbation process perturbing the input, which we refer to as noisy self-training (noisy ST).
During finetuning, we still use dropout. We choose 250 instances since we find that 500 training samples already yields perfect performance on this task. However, we to mimic real seq2seq tasks where the supervised models often far from perfect.
5

---

Published as a conference paper at ICLR 2020
X
3
3
6
71 buswlin
65
65 pawuado-traiiiigg
prouddo-taaiingg
Figure 2: Two examples of error heat map on the toy sum dataset that shows the effect of smoothness. The left panel of each composition is from the baseline, and the right one is from the pseudo-training step the first iteration. x and y axes represent the two input integers. Deeper color represent larger errors.
4.2 NOISY SELF-TRAINING
If we perturb the input during the pseudo-training step, then Eq. I would be modified to:
Cu -.-E'' (g(m) m-pp( By--p (y/m) log po (y)l)))
(3)
where g(x) is perturbation function. Note that apply both input perturbation and dropout in the pseudo-traininn step for noisy ST throughout the paper, but include ablation analysis in 44.3) We first validate noisy ST in the toy task. We shuffle the two integers in the input the perturbation function. Such perturbation is suitable for this task since would help the model learn the commutative law as well. To check that, we also measure the symmetry of the output space. Specifically, for any point (2), we compute (f( - f(r2, x1) and average it over all the points. Both smoothness and symmetry values are reported in Table While we do not explicitly perturb the input at nearby integers, the shuffling perturbation greatly improves the smoothness metric as well. Furthermore, predictions are more symmetric and test errors are reduced.
In order to illustrate the effect of smoothness, in Figure we show two examples of error heat map When a point with large error surrounded by points with small errors, the labels might propagate due to smoothing and its enror is likely to become smaller, resulting in a "selffcorreeting" behaviour, as demonstrated in the left example Figure 2 However, the prediction of some points might become worse due the opposite phenomenon too, shown in the right example of Figure 2 Therefore, the smoothing effect by itself does not guarantee a performance gain the pseudo- training step, but fine-tuning benefits from it and seems to consistently improve the baseline in all datasets we experiment with.
4.3 OBSERVATIONS ON MACHINE TRANSLATION
Next, we apply noisy self-training to the more realistic WMT100 translation task. We try two dif- ferent perturbation functions: (1) Synthetic noise as used in unsupervised MT (Lample al.. 2018), where the input tokens are randomly dropped, masked, and shuffled. We use the default noising parameters in unsupervised MT but study the influence of noise level in {5.4 (2) Paraphrase. We translate the source English sentences to German and translate it back to obtain paraphrase as the perturbation. Figure I shows the results over three iterations. Noisy ST (NST) greatly outperforms the supervised baseline by over 6 BLEU points and normal ST by 3 BLEU points, while synthetic noise does not exhibit much difference from paraphrasing. Since synthetic noise much simpler and more general, in the remaining experiments we use synthetic noise unless otherwise specified.
Next, we report an ablation analysis of noisy ST when removing dropout the pseudo-training step in Table 2 Noisy ST without dropout improves the baseline by 2.3 BLEU points and is comparable normal ST with dropout. When combined together, noisy ST with dropout produces another 1.4 BLEU improvement, indicating that the two perturbations are complementary.
EError heat map for the entire space be found in Appendix

---

Published as a conference paper at ICLR 2020
WMT English-Germin
FloRes English-Nepali
Methods
100K (+3. 8M mono) 3.9M (+20M mono)
En-Origin
Ne-Origin
Overall
baseline
15.6
28.3
6.7
2.3
4.8
BT
20.5
8.2
4.5
6.5
noisy ST
21.4
29.3
8.9
3.5
6.5
Table 4: Results on two machine translation datasets. For WMT100K, we use the remaining 3.8M English and German sentences from training data as unlabeled data for noisy ST and BT, respec- tively.
EXPERIMENTS
Our experiments below are designed to examine whether the noisy self self-training is generally useful across different sequence generation tasks and resource settings. To this end, we conduct experi- ments on two machine translation datasets and one text summarization dataset to test the effective- ness under both high-resource and low-resource settings.
5.1 GENERAL SETUP
We run noisy self-training for three iterations or until performance converges. The model is trained from scratch in the pseudo-training step at each iteration since we found this strategy work slightly better empirically. Full model and training details for all the experiments can be found in Ap- pendix A. some settings, we also include fack-translation (BT. [Sennrich et al., 2015) as reference point, since this is probably the most successful semi-supervised learning method for ma- chine translation. However, we want to emphasize that BT not directly comparable to ST since they use different resources (ST utilizes the unlabeled data on the source side while BT leverages target monolingual data) and use cases. For example, BT is not very effective when we translate English extremely low-resource languages where there is almost no in-domain target monolingual data available. We follow the practice in (Edunov et al. 2018) to implement BT where we use unre- stricted sampling translate the target data back to the source. Then, train the real and pseudo parallel data jointly and tune the upsampling ratio of real parallel data.
5.2 MACHINE TRANSLATION
We test the proposed noisy self-training on a high-resource translation benchmark: WMT14 English-- German and low-resource translation benchmark: FloRes English-Nepali.
WMT14 English-German: In addition WMT100K, also report results with all 3.9M training examples. For WMT100K we use the Base Transformer architecture, and the remaining parallel data as the monolingual data. For the full setting, ture (Vaswani et al. (017) and randomly sample 20M English sentences from the News Crawl corpus for noisy ST.
the Big Transformer architec-
FloRes -nelish-Nepali: We evaluate noisy self-training on low-resource machine translation dataset FloRes (Guzmán et al.] 2019) from English (en) to Nepali (ne), where we have 560K train- ing pairs and very weak supervised system that attains BLEU smaller than 5 points. For this dataset we have 3.6M Nepali monolingual instances in total (for BT) but 68M English Wikipedia sentences We randomly sample 5M English sentences for noisy ST. We use the same trans- former architecture in (Guzmán et al. 2019).
The overall results are shown in Table 4 For almost all cases in both datasets, the noisy ST outper- forms the baselines by large margin ~ 5 BLEU scores), and we see that noisy ST still improves the baseline even when this is very weak.
Effeet of Domain Mismatch. Test sets of the FloRes benchmark were built with mixed original- translationese some sentences are from English sources and some from Nepali sources. In- tuitively, English monolingual data should be more in-domain with English-origin sentences and
http://wwed statmt. .rrpetttttrgg/mmom wwt./wtt////......oo.o/.//.....t httg:/mt19/paree. 9/para/llllaae httpppaallellcooom hatp

---

Published as a conference paper at ICLR 2020
100K
(+3.7M mono)
640K
(+3.2M mono)
3.8M
(+4M mono)
Methods
RI
R2
RL
RI
R2
RL
R1
R2
RL
MASS (Song et al. 2019
38.7
19.7
36.0
baseline
30.4
12.4
27.8
35.8
17.0
33.2
37.9
19.0
35.2
BT
32.2
13.8
29.6
37.3
18.4
34.6
noisy
15.6
31.4
36.6
18.2
33.9
38.6
19.5
Table 5: Rouge scores on Gigaword datasets. For the 100K setting we use the remaining 3.7M training data unlabeled instances for noisy ST and BT. In the 3.8M setring 4M unlaheled data for noisy ST. Stared entry (+) denotes that the system uses a much larger dataset for pretraining.
30
noisy ST
baseline
26:2
28.3
29.3
19
noisy ST
19:3
19.2
19.2
25
19
20
20:4
23.2
18
18.7
27.4
BLEU
15.6
17
10
16.6
18
17.9
17.9
16
16.6
nolsy
2.0
10K
100K
640K
3.9M
0
100K 1:5M 3.8M 20M
5T
0.2
0.4
0.6
0.8
(a) BLEU v.s. parallel data
(b) BLEU v.s. monolingual data
(c) BLEU v.s. noise level
Figure 3: Analysis of noisy self-training on WMT English-German dataset, demonstrating the effect of parallel data size, monolingual data size, and noise level.
Nepali monolingual data should help for Nepali-origin sentences. To demonstrate this possi- ble domain-mismatch effect, in Table 4 we report BLEU on the two different test sets separately As expected, ST very effective when the source sentences originate from English.
Comparison to Back- Translation. Table that noisy ST is able to beat BT on WMT100K and the en-origin test of FloRes. In contrast, BT more effective on the ne-origin set according to BLEU, which is not surprising as the ne-origin test is likely to benefit more from Nepali than English monolingual data.
5.3 TEXT SUMMARIZATION
We further evaluate noisy self-training on the Gigaword summarization dataset (Rush et al. 2015) that has 3.8M training sentences. We encode the data with 30K byte-pair codes and use the Base Transformer architecture. Similar to the setting of WMT100K, for Gigaword we create two settings where sample 100K or 640K training examples and use the remaining as unlabeled data to compare with BT. We also consider the setting where all the 3.8M parallel samples are used and we mine in-domain monolingual data by revisiting the original preprocessing procedure and using the 44M samples that Rush et al. (2015) disregarded because they had low-quality targets. We report ROUGE scores [Lin 2004) Table 5 Noisy ST consistently outperforms the baseline in all settings, sometimes by large margin (100K and 640K). It outperforms BT with 100K parallel data but underperforms with 640K parallel data. We conjecture that BT still effective in this case because the task is still somewhat symmetric Gigaword mostly contains short sentences and their compressed summaries. Notably, noisy ST in the full setting approaches the performance of state-of-the-a systems which much larger datasets for pretraining (Song al. 2019).
5.4 ANALYSIS
In this section, we focus on the WMT English- German dataset to examine the effect three factors on noisy self-training: the size of the parallel dataset, the size of the monolingual dataset, and the noise level. All the noisy ST results are after the fine-tuning step.
TTsst set split is obtained through personal communication with the authors. https:://i thub. com/ faw.bookarchiieeeeeoooooooo
8

---

Published as a conference paper at ICLR 2020
Parallel data size. We fix the monolingual data size as 20M from News Crawl dataset, and vary the parallel data size as shown in Figure (aa We use a small LSTM model for 10K, Base Transformer for 100K/640K, and Big Transformer for 3.9M Noisy ST is repeated for three iterations. We see that in all cases noisy ST is able to improve upon the baseline, while the performance gain is larger for intermediate value of the size of the parallel dataset, as expected.
Monolingual data size. We fix the parallel data size to 100K samples, and use the rest 3.8M English sentences from the parallel data as monolingual data. We sample from this set 100K, 500K, 1.5M, and 3.8M sentences. We also include another point that uses 20M monolingual sentences from a subset of News Crawl dataset. We report performance at the first iteration of noisy ST. that the performance keeps improving as the monolingual data size increases, albeit with diminishing returns.
Figure
Noise level. We have shown that noisy ST outperforms ST, but intuitively larger noise must not always be better since at some point it may destroy all the information present in the input. We adopt the WMT100K setting with 100K parallel data and 3.8M monolingual data, and set the word blanking probability in the synthetic noise (Lample et al. 2018 to 0.2 (default number), 0.4, 0.6, and 0.8. We also include the baseline ST without any synthetic noise. Figure (3() demonstrates that performance is quite sensitive to noise level, and that intermediate values work best. It still unclear how to select the noise level a priori, besides the usual prper-parameter search maximize BLEU on the validation set.
5.5 NOISE PROCESS ON PARALLEL DATA ONLY
In this section, we justify whether the proposed noisy self-training process would help the supervised baseline alone without the help of any monolingual data. Similar to the training process on the monolingual data, we first train the model on the noisy source data (pseudo-training), and then fine- tune it on clean parallel data. Different from using monolingual data, there are two variations here in the "pseudo-training" step: we can either train with the fake target predicted by the model as on monolingual data, or train with the real target paired with noisy source. We denote them as "parallel fake target" and "parallel + real target" respectively, and report the performance on WMT100K in Table 6 We use the same synthetic noise as used in previous experiments.
When applying the same noise process parallel data using fake target, the smoothing effect is not significant since it restricted into the limited parallel data space, producing marginal improvement over the baseline (+0.4 BLEU). As a comparison, 100K monolingual data produces +1.0 BLEU and the effect is enhanced when we increase the monolingual data to 3.8M, which leads to +3.7 BLEU. Interestingly, pairing the noisy source with real target results in much worse performance than the baseline (-4.3 BLEU), which implies that the use of fake target predicted by the model (i.e. distillation) instead of real target is important for the success of noisy self-training, at least in the case where parallel data size is small. Intuitively, the distilled fake target is simpler and relatively easy for the model to fit, but the real target paired with noisy source makes learning even harder than training with real target and real source, which might lead to bad starting point for fine-tuning. This issue would be particularly severe when the parallel data size is small, in that case the model would have difficulties to fit real target even with clean source.
6 RELATED WORK
Self- trainiriing belongs to a broader class of "pseudo-label" semi-supervised learning approaches. These approaches all learn from pseudo labels assigned to unlabelled data, with different methods on how to assign such labels. For instance, co-training (Blum & Mitchell 1998) learns models on two independent feature sets of the same data, and assigns confident labels to unlabeled data from one of the models. Co-training reduces modeling bias by taking into account confidence scores from two models. In the same spirit, democratic co-training (Zhou & Goldman 2004) or tri-training (Zhou & Li 2005) trains multiple models with different configurations the same data feature set, and subset of the models act as teachers for others.
Thhee architectures are selected based on validation loss.
9

---

Published as a conference paper at ICLR 2020
Methods
PT
FT
parallel baseline
15.6
noisy ST, 100K
fake target
10.2
16.6
noisy ST, 3.8M mono + fake target
16.6
19.3
noisy ST, 100K parallel real target noisy ST, 100K parallel fake target
6.7
11.3
10.4
16.0
Table 6: Results on WMT100K data. All results are from one single iteration. "Parallel real/fake target" denotes the noise process applied on parallel data but using real/fake target in the "pseudo- training" step. "Mono fake target" is the normal noisy self-training process described in previous sections.
Another line of more recent work perturb the input or feature space of the student's inputs as data augmentation techniques. Self-training with dropout or noisy self-training can be viewed an in- stantiation of this. These approaches have been very successful on classification tasks (Rasmus) al., 2015 Miyato al.) 2017]
Aila 2017 Miyato et al. 22118 Xie et al. 2019) given that reasonable amount of predictions of unlabeled led data (at least the ones with high confidence) are correct, but their effect on language generation tasks is largely unknown and poorly understood be- cause the pseudo language targets are often very different from the ground-truth labels. Recent work on sequence generation employs auxiliary decoders (Clark et al. 2018 when processing unlabeled data, overall showing rather limited gains.
7 CONCLUSION
In this paper revisit self-training for neural sequence generation, and show that it can be an effective method to improve generalization, particularly when labeled data is scarce. Through a comprehensive ablation analysis and synthetic experiments, we identify that noise injected during self-training plays critical role for its success due to its smoothing effect. To encourage this be- haviour, we explicitly perturb the input to obtaín new variant of self-training, dubbed noisy self- training. Experiments on machine translation and text summarization demonstrate the effectiveness of this approach in both low and high resource settings.
ACKNOWLEDGEMENTS
We want to thank Peng-Jen Chen for helping set up the FloRes experiments, and Michael Auli, Kyunghyun Cho, and Graham Neubig for insightful discussion about this project.
REFERENCES
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. Proceed- ingy of the eleventh annual conference Computational learning theory, pp. 92-100. Citeseer, 1998.
Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In Proceedings of AISTATS, 2005.
Olivier Chapelle, Bernhard Scholkopf. and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b06) reviews). IEEE Transactions Neural Networks, 20(3):542-542, 2009.
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. Semi-supervised quence modeling with cross-view training. In Proceedings of EMNLP, 2018.
Sergey Edunov, Myle Ou, Michael Auli, and David Grangier. Understanding back-translation at scale. Proceedings of EMNLP, 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Pro- ceedings of NeurlPS, 2005.
10

---

Published as a conference paper at ICLR 2020
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. The FLoRes evaluation datasets for low-resource machine translation: Nepali-english and sinbala-english. In Proceedings of EMNLP, 2019.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing ad-apptttiionn of feature detectors. arXiv preprint arXiv:/207.05500 2012.
Zhongqiang Huang and Mary Harper. Self-training pcfg grammars with latent annotations across languages. In Proceedings of EMNLP, 2009.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Proceedings of NeurIPS, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proceedings of ICLR, 2017.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. Phrase-based & neural unsupervised machine translation. In Proceedings of EMNLP, 2018.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, 2013.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of NAACL, 2006.
Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sen- tence compression. In Proceedings of EMNLP, 2016.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In Proceedings of ICLR, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervipe learning. IEEE transactions on pattern analysis and machine intelligence, 1979) 2018
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL (Demo Track), 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proceedings of ACL, 2002.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Hami Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Proceedings of NewrlPS, 2015.
Roi Reichart and Ari Rappoport. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. Proceedings of ACL,, 2007.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, 2015.
Scudder. Probability of error of some adaptive recoenn-rooogiitionn machines. IEEE Transactions on Information Theory, (3):3333377, 1965.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 86-96, 2015.
11

---

Published as a conference paper at ICLR 2020
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACL,, 2016.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of ICML, 2019.
Nicola Ueffing. Using monolingual source-language data to improve mt performance. In /WSLT, 2006.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv:190411244,, 2019.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Pro- ceedings of ACL, 1995.
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. StructVAE: Tree-structured latent variable models for semi-supervise semantic parsing. In Proceedings of EMNLP, 2018.
Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural machine translation. In Proceedings of EMNLP. 2016.
Yan Zhou and Sally Goldman. Democratic co-learning. In 16th IEEE International Conference on Tools with Artificial Intelligence, pp. 602. IEFE, 2004.
Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions Knowledge & Data Engincering, (11):1529-1541 2005.
Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
12

---

Published as a conference paper at ICLR 2020
A EXPERIMENTS DETAILS
A.1 SETUP DETAILS
For all experiments, we optimize with Adam (Kingma & Ba 2014) using 31 0.9, 32 0.98,
le - 8. All implementations are based on fairseq (Ott et al., 2019), and we basically use the same learning rate schedule and label smoothing in fairseq examples to train the transformers Except for the toy sum dataset which we runs on a single GPU and each batch contains 32 examples, all other experiments
run 8 GPUs with effective batch size of 33K tokens. All experiments validated with loss on the validation set. For self-training or noisy self-training, the pseudo-training takes 300K synchronous updates while the fine-tuning step takes 100K steps.
We use the downloading and preprocessing scripts in fairseq to obtain the WMT 2014 English- German dataset 10 which hold out small fraction of the original training data as the validation set.
The model architecture for the toy sum dataset single-layer LSTM with word embedding size 32, hidden state size 32, and dropout rate 0.3. The model architecture of WMTI0K baseline in Figure M[a] single layer LSTM with word embeddings size 256, hidden state size 256, and dropout rate 0.3.
A.2 JUSTIFICATION OF THE WMT100K BASELINE
We provide more details and evidence to show that our baseline model on WMT100K dataset is trained properly. In all the experiments on WMT100K dataset including baseline and self-training ones, use Adam optimizer with learning rate 0.0005, which is defaulted in fairseq. We do not use early stop during training but select the best model terms of the validation loss. We train with 30K update steps for the baseline model and (300K eseudo-training + 100K fine-tuning) update steps for self-training. In both cases we verified that the models are trained sufficiently to fully converge through observing the increase of validation loss. Figure 4 shows the validation curve of the baseline model. Note that the model starts to overfit, and we select the model checkpoint at the lowest point. We also varied the learning rate hyperparameter as 0.0002, 0.0005, and 0.001, which produced BLEU score 15.0, 15.6 (reported in the paper), and 15.5 respectively our baseline model previous sections obtained the best performance.
validabo
5000
10000
15000
20000
25000
30000
step
Figure 4: Validation loss v.s. number of update steps, for the baseline model WMT100K dataset.
13

---

Published as a conference paper at ICLR 2020
B COMPARISON REGARDING SEPARATE TRAINING, JOINT TRAINING, AND FILTERING
In the paper we perform self-training with separate pseudo-training and fine-tuning steps and always use all monolingual data. However, there are other variants such as joint training or iteratively adding confident examples. Here we compare these variants on WMT100K dataset, noisy self-training uses paraphrase the perturbation function. For joint training, tune the upsampling ratio of parallel data just as bock-translation (Edunov et al. 2018). We perform noisy self-training for 3 iterations, and for filtering experiments iteratively use the most confident 2.5M, 3M, and 3.8M monolingual data respectively in these 3 iterations. Thole that the filtering process helps joint training but still underperforms separate-training methods by over 1.5 BLEU points. Within separate training filtering produces comparable results to using all data. Since separate training with all data is the simplest method and produces the best performance, we stick this version the paper.
Methods
BLEU
baseline
15.6
noisy ST (separate training, all data)
21.8
noisy ST (separate training, filtering)
21.6
noisy ST (joint training, all data)
18.8
noisy ST (joint training, filtering)
20.0
Table Ablation analysis on WMT100K dataset.
C ADDITIONAL RESULTS ON THE TOY SUM DATASET
We additionally show the error heat maps of the entire data space on the toy sum datasets for the first two iterations. Here the model at aseudo-training step initialized as the model from last iteration to clearly examine how the decodings change due to injected noise. As shown in Figure 5 for each iteration the pseudo-training step smooths the space and fine-tuning step benefits from and greatly reduces the errors
14

---

Published as a conference paper at ICLR 2020
(a) baseline
(b) noisy ST (PT, iter=1)
(c) noisy ST (FT, iter=l)
(d) noisy ST (PT, iter=2)
(c) noisy ST (FT, iter=2)
Figure 5: Error heat maps on the toy sum dataset over the first two iterations. Deeper color represent larger errors.
15


Reviews:
[{"review_id": "rke6Rf_pKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates why self-training helps in machine translation and text summarization tasks, identifying that auxiliary noise can amplify the benefits of this process. The paper is well-structured and clearly written, and conducts a fairly thorough analysis of the issue at hand.\n\nComments:\n\n- The authors argue self-training enhances smoothness, but I would like to see this explained mathematically/conceptually in greater detail. It is not immediately clear to me why this would be, particularly in the case of discrete text data.\n\n- Why not evaluate smoothness on the actual MT task instead of just the toy task?\nThe authors could measure the L2 norm between encodings of source sentences for neighboring sentences (eg. based on edit distance or word-movers) vs very different sentences. And then compare the base model vs the one obtained from self-supervised training.\n\n- If the primary beneficial effect of self-supervised training is smoothness as the authors claim, then they should try enforcing smoothness in alternative ways to see if performance improves.  Some options here could be (using dropout in all of them): 1) add your same noise process to the original labeled dataset to create augmented examples, 2) rather than the self-supervised objective, use an auxiliary training objective which says the predictions on each unlabeled datapoint should be similar to the predictions on noised versions of this datapoint, 3) some form of virtual adversarial training [VAT]. In fact, the authors should discuss [VAT] a bit more, as this paper also presents a smoothness-regularizer that is highly useful for semi-supervised learning.\n\n[VAT] Miyato et al. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. https://arxiv.org/abs/1704.03976\n\n\n- As the authors write, self-training can be viewed as a form of entropy-regularization. Likewise, the input perturbation+dropout process also seems like it should affect the conditional entropies. Can the authors expound upon this connection a bit further?  Some mathematical analysis would be nice to have here as well.\n\n\n- \"Another way is to treat them separately – first we train the model on pseudo parallel data S, and then fine-tune it on real data L.\"\n\nThe authors should clarify the overall training process with a more precise explanation. I assume is actually: \n1) train initial model M on (limited) real data L\n2) use M to generate pseudo-targets for unlabeled data in S\n3) train M on this pseudo-dataset S\n4)fine-tune M on real data L\n\nIs this correct? And it should be clarified whether M in step (3) is fine-tuned from the M in step (1) or re-initialized from scratch before training begins (Based on later text, it seems like the latter, but this should be clarified early on).\n\n- Baseline in Figure 1 should be described a bit more clearly.\n\n- Since the BLEU score dropped from 3 to 1.9 when the authors continued training from the baseline model (Sec 3.2), isn't the optimum hypothesis not ruled out?  I don't think the authors should make this claim, and rather state that the initialization does seem to play some role, but does not fully explain the benefits of self-training.  \n\n- \"We use a small LSTM model for 10K, Base Transformer for 100K/640K, and Big Transformer for 3.9M\" \n\nIs this because these are the best performing models on these respective datasets? The authors should explain these decisions.\n\n\n- \"We include quantitative comparison regarding joint training, separate training, and pseudo-parallel data filtering in Appendix B\"\n\nShould clarify here (in main text) that separate training matches the performance of joint training.\n\n- typos: \"joint traing\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hygxm9Q7Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an interesting study that tries to explain why conditional text generation models with autoregressive decoders benefit from self-training on pseudo labels created from the same model. The paper introduces and verifies two hypotheses: 1) Decoding strategy: Since beam search is a biased estimator sampling using it doesn't reflect the learned distribution from the model and hence variations happen that benefit learning. (this partially help).\nAuthors verify that by replacing the beam search decoder by an unbiased sampling method for decoding. 2) Additional noise during training: The use of dropout adds discrepancy between training and inference which creates favourable regularization. Through smoothing of the latent spaces so similar semantic inputs get mapped to similar outputs.  Authors verify that using a toy task of summing two numbers. \n\nFollowing this intuition, authors introduce a new method of noisy self-training that even enforces hypothesis #2 (rather than increasing dropout which isn't practical to train). Authors experiment with word perturbation from (Lample et al. 18) and paraphrasing through translation back and forth.\n\nAuthors provide a thorough experimental section evaluating the noisy ST on Machine translation, low resource MT and Summarization the latter two latter are especially interesting to use ST rather than Back translation as target side documents might be hard to find (low resource MT) or quite challenging to recover the source side from (in case of summarization ).\n\nAdditional analysis experiments were performed to show the effect noisy self-training with regard to, increasing noise, number of available parallel data, size of ST samples generated from monolingual data.\n\nI am in favour of this work acceptance, overall the paper introduces very interesting insights to explain the usefulness of self-training for auto-regressive generation tasks and could inspire future work along this line in designing better ST algorithms and/or adoption of self-training in low-resourced generation tasks. \n\n\nQuestions to authors: \nAs explained in section 2, you preferred to model the ST task as an iterative process between pseudo training and fine-tuning was mainly chose for simplicity while providing equal results as shown in Table 6 (appendix). I wonder why joint training does provide lower results than separate training? I doubt this might be due to that it is given less time to converge compared to the 3 iterations of self-training. Can you provide more details about how this is done?\n\nFigure 4 (appendix), I was wondering if there's an intuition about the artefacts in the error heat maps of the toy task ( the patterns with -45 degrees slope)\n\n\nSuggestions to enhance readability:\nA lot of abbreviations finish by \"T\" this makes it quite hard to follow, I would suggest authors to remove \"PT\" and \"FT\" since they haven't been used much in the paper. \n\nLabels in Figure 1: especially early in the paper becomes hard to grasp, would be nice if you can add description to what each abbreviation means in the figure caption. \n\ntypo table 6 appendix: joint \"traing\"\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bke-eZJ0YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a self-training approach for improving sequence-to-sequence tasks. As a preliminary experiment, this study randomly sampled 100k sentences from WMT 2014 English-German dataset (WMT100K, hereafter), trained a baseline (Transformer) model on WMT100K, and applied self-training methods on the remaining English sentences as the unlabeled monolingual data. After exploring different procedures for self-training, this study uses the fine-tuning strategy: train a model on the supervision data; build pseudo parallel data by predicting translations for all unlabeled data using the trained model; train a new model on the pseudo parallel data; and fine-tune the new model on the supervision data. This strategy alone gave a 3 points improvement of BLEU.\n\nThis paper hypothesized the reasons behind the performance improvements: beam-search decoding (+0.5 BLEU) and dropout (+1.2 BLEU). The authors argue that the beam-search decoding contributes partially to the performance gains, while the dropout accounts for the most. The authors infer that the dropout causes an implicit perturbation. Exploring different perturbation strategies, synthetic noise (e.g., input tokens are randomly dropped, masked, and shuffled) and paraphrase (round-trip translation, e.g., English-German-English), the authors reported no significant difference between these two strategies. Finally, this paper reports empirical results (on WMT 2014 English-German, FloRes English-Nepali, and English Gigaword (summarization task)) of self-training strategies presented in this paper. This paper concluded that self-training can be an effective method to improve generalization and that the noise injected during self-training plays a critical role for its success.\n\nThis paper is well structured and well written. It is interesting to see the improvements of the sequence-to-sequence tasks by using the self-training approach. This paper is interesting because it has a close connection with the back-translation approach that has been popular in recent years.\n\nAlthough the hypotheses presented in this paper are interesting, they were not fully validated after all. The analyses on the loss functions, ablation tests, and experiments on the toy task can only bring indirect explanations about why we could observe the performance improvements. This impression is also demonstrated by the fact that this paper uses 'might' five times when explaining interpretations of the experimental results. Having said, I tend to agree that identifying the exact reason is difficult.\n\nHowever, I have two other questions before recommending this paper: (1) whether the baseline model was trained sufficiently, and (2) whether this paper is about self-training strategies or regularization methods. \n\n(1) In order to accept the experimental results that the self-training approach can improve the performance, we need to make sure that the baseline model was trained sufficiently. However, the appendix explains, \"we basically use the same learning rate schedule and label smoothing as in fairseq examples.\" I'm not sure whether this training procedure was fair among different models because the baseline model and self-supervised model received totally different number of training instances (100k vs 3.9M). This claim would be stronger if this paper could show an evidence that the baseline model was trained properly by, for example, explaining the stopping criteria for iterations, tuning hyper-parameters (e.g., learning rate) individually for the baseline and self-trained models, showing the mean and variance of BLEU scores with different initializations, and showing the training curve of the baseline model.\n\n(2) We can view the self-training strategies presented in this paper as regularization methods. For this reason, I am wondering whether the self-training strategies presented in this paper are only for self-training or general to pure supervised setting. We can easily guess that the performance would drop if we removed the dropout from the baseline method. In contrast, I would like to see whether the synthetic noise could improve the performance of the baseline method alone, behaving as a regularization method. It would be useful to see the performance of the baseline method without the dropout and with the synthetic noise to highlight the effect of the presented strategies under the self-training scenario.\n\nMinor comments:\n\nIt would be useful to see the number of unlabeled instances in Section 3.1.\n\nSectoin 3.2: \"This is different from back-translation where new knowledge may originate from an additional backward translation model.\"\nI'm not sure whether a backward translation model can introduce new knowledge because the supervision data are usually the same between forward and backward directions.\n\nSection 3.3: \"at test/decoding time the model does not use dropout\"\nTo be precise, the weights are scaled by the dropout rate (p) in the decoding time.\n\nReference: Lample et al. 2018 should be replaced with another paper:\nGuillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato. Unsupervised Machine Translation Using Monolingual Corpora Only. ICLR 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes self-training for sequence-to-sequence models and proposes a noisy version of self training. An empirical study shows the proposed noisy version improves results for machine translation and summarization tasks.

All reviewers appreciate the interesting contributions of the research, as well as clear writing. They also offer several comments for the revision of the paper. 

We look forward to seeing this paper presented at the conference!
Decision: Accept (Poster)