Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generation
Authors: Xiang Lin
Prathyusha Jwalapuram
Shafiq Joty
Venue: Submitted to ICLR 2023 (2023)
Keywords: exposure bias
text generation

Abstract: State-of-the-art neural text generation models are typically trained to maximize the likelihood of each token in the ground-truth sequence conditioned on the previous target tokens. However, during inference, the model needs to make a prediction conditioned on the tokens generated by itself. This train-test discrepancy is referred to as exposure bias. Scheduled sampling is a curriculum learning strategy that gradually exposes the model to its own predictions during training to mitigate this bias. Most of the proposed approaches design a scheduler based on training steps, which generally requires careful tuning depending on the training setup. In this work, we introduce Dynamic Scheduled Sampling with Imitation Loss (DySI), which maintains the schedule based solely on the training time accuracy, while enhancing the curriculum learning by introducing an imitation loss, which attempts to make the behavior of the decoder indistinguishable from the behavior of a teacher-forced decoder. DySI is universally applicable across training setups with minimal tuning.  Extensive experiments and analysis show that DySI not only achieves notable improvements on standard machine translation benchmarks, but also significantly improves the robustness of other text generation models. 

Under review as a conference paper at ICLR 2023
DYNAMIC SCHEDULED SAMPLING WITH IMITATION Loss FOR NEURAL TEXT GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-arr neural text generation models are typically trained maximize the likelihood of each token in the ground-truth sequence conditioned on the previous target tokens. However, during inference, the model needs to make a prediction conditioned on the tokens generated by itself. This train-test discrepancy is referred to as exposure bias. Scheduled sampling is a curriculum learning strategy that gradually exposes the model to its own predictions during training to mitigate this bias. Most of the proposed approaches design a scheduler based on training steps, which generally requires careful tuning depending the training setup. In this work, we introduce Dynamic Scheduled Sampling with Imitation Loss (DYSI), which maintains the schedule based solely the training time accuracy, while enhancing the curriculum learning by introducing an imitation loss, which attempts to make the behavior of the decoder indistinguishable from the behavior of a teacher-forced decoder. DYSI is universally applicable across training setups with minimal tuning. Extensive experiments and analysis show that DySI not only achieves notable improvements on standard machine translation benchmarks, but also significantly improves the robustness of other text generation models. Our code is available at:
INTRODUCTION
Advances in deep learning have led to great achievements in neural text generation tasks including machine translation (Vaswani et al. 2017 Wu al. 2019), summarization (Zhang et al. 2019a Lewis iet al. (020) and language modeling Radford et al. 2019 Brown et al. (020). The dominant approach to date generates the output sequence with a decoder in an autoregressive manner (Bahdanau et al.
2014; Vaswani et al. 2017). To realize the autoregressive formulation, most of the text generation models are trained to maximize the likelihood of each token in the ground- trutt sequence conditioned on the previous target tokens with Maximum Likelihood Estimation (MLE). In particular, Teacher Forcing (Williams & Zipser 1989) has been the de facto strategy to help stabilize and speed up the training, where the decoder takes the ground-trute -ruth token from the previous time step as the conditioning input for generating the next token. At inference time, however, the decoder does not have access to the previous ground-truth tokens when it is predicting the next token. Thus, the decoder has to instead make a prediction conditioned on the tokens generated by itself so far, resulting in a train-test discrepancy, often referred to as exposure bias (Bengio al., 2015). This discrepancy can lead to error accumulation over time steps as the model might encounter unexpected (though not necessarily wrong) tokens that has never been exposed during training.
The methods proposed to combat exposure bias can be categorized into two groups: Non-MLE-based approaches (Goyal et al. 2016 Yu et al. 2017 Lin et al.
2017 Nie et al. 2019 and MLE-based
approaches (Bengio et al., 2015 Song et al.) 2021 Liu et al. (021b). Most non-MLE- E-saed approaches take advantage of generative adversarial networks (GAN) (Goodfellow et al.) 2014) and/or reinforcement learning methods to avoid teacher forcing. However, the advantages of these approaches often come with the price of training instability and difficulty, and empirically they still struggle to outperform the MLE baseline (He et al. (022I). On the other hand, MLE-based approaches typically apply curriculum learning (Bengio et al. 2009) strategy to gently bridge the gap between training and inference. These methods often consist of a scheduler, e.g.. based training steps, which controls the extent to which the model should be exposed to its own predictions during training. Intuitively, the model should be exposed more of its own outputs as the training proceeds.

---

Under review as a conference paper at ICLR 2023
MLE- approaches are inherently more efficient and parallelizable as the models do not need to generate the full sequence in inference mode to compute the training loss. Also, MLE has been the mainstream method for training deep neural models. Our work in this paper thus concerns ML.E-based training. [Bengio et al. (2015) propose scheduled sampling to alleviate exposure bias, where the decoder uses the ground-truth previous token input with probability 6, and uses its own prediction with probability (1 <). The probability is controlled by a scheduler to decay based on the training steps. Such curriculum learning strategy allows the model to use ground-truth previous tokens at the initial stage of the training and gradually exposes the model to more and more of its own predictions. ZZhng et al.] (2019b) modify the scheduled sampling of Bf et al. (2015) by allowing the model to sample from a set of oracle tokens (e.g., synonym of the target token) as the previous token to simulate the model's output at inference time. Goodman et al. (2020) use a stack of N temporal decoders trained to decode along a secondary time axis that allows updating model parameters based on N prediction steps with N being a hyper-parameter Each decoder gets a predicted previous token as input from its prior decoder, but the training cost increases linearly with N. [Song et al. (202T) incorporate an error correction mechanism with two decoders. A query stream decoder having access to only positional information first predicts intermediate results, which is then corrected by a content stream decoder. The inference requires running through both decoders, which lowers the decoding efficiency. More recently, Liu al. (2021b) propose to use a scheduler based on both training and decoding steps. Intuitively, the later decoding steps usually have higher error rates during inference due to еттот accumulation. Therefore, the model output should be sampled as input with higher chance for the later decoding steps during training.
As discussed, MLE-based approaches maintain a schedule to decide how much the model should be exposed to its own predictions, which often needs a proxy to estimate the training progress. A schedule (linear or nonlinear) based on training steps usually requires careful design for the specific problem setup as different batch sizes may lead to different training speeds and different tasks may have different convergence rates. This limits the applicability of these approaches to new tasks and datasets. In this work, we introduce Dynamic Scheduled Sampling with Imitation loss (DySI). First, we propose a scheduler that solely depends on training time accuracy. By tracking training progress though accuracy, avoid having to perform costly heuristic search to find suitable scheduler for each different problem setup. In addition, we use an imitation loss to enforce the condition that the generative behavior should match teacher-forced behavior as closely possible, core idea in professor forcing (Goyal et al., 2016). Our imitation loss uses the decoder in teacher-forcing mode as the expert to regularize/guide the decoder's behavior when takes self-generated tokens as input.
We first conduct experiments on machine translation (MT) to demonstrate how our approach performs in various aspects such as generalization and degeneration. Results show that training with DYSI achieves notable improvements on standard MT benchmarks. We then introduce a novel framework for evaluating the robustness of language model (LM) when exposed to perturbed data, using auto-completion as test bed. We find that current pre-trained LMs, when trained with standard teacher forcing, are quite sensitive to erroneous contexts that are typical for LM generation such as repetitions. Analysis shows SI, by reducing exposure bias, yields significantly more robust LM across various kinds of perturbations, and overall produces better quality text.
BACKGROUND
Text generation. Typical neural text generation models use an autoregressive factorization of the joint probability over the target sequence. An autoregressive decoder trained with maximum likelihood
estimation (MLE) learns to assign probability to a target sequence y = (" ,yr) containing T tokens by factorizing the joint probability using the chain rule:
T
LMLE =
Σ
log P(ylB<c...)
(I)
where x is a source input for conditional text generation (e.g., machine translation) and for unconditional generation (e.g., language modeling), and yer (y1...., .Vb1)... denotes tokens before the current step f. To train autoregressive models, teacher forcing (Williams & Zipser 1989) is commonly used for faster convergence and training stability. In this method, ground-trus tokens from the previous steps are used as input to predict the current token y.. However, it also causes the train-test discrepancy or exposure bias as the target tokens are not available inference time.
2

---

Under review as a conference paper at ICLR 2023
MLE Loss
Expert Distribution
Leamnr Distribution
1
P
P3
P4
P
P
Imitation Loss
P
Ps
Decoder
Decoder
arg mAX
Mo
M1
M2
U4
Ms
Mu
Dynamic Scheduler
Mo
91
M2
1/3
94
96
Teacher-Foree Foreirg Mode
Preparation
Operative Mode
Figure Illustration Dynamic Scheduled Sampling with Imitation Loss (DYSI). The teacher- forced decoder allows the computation of training accuracy that is directly used in our dynamic scheduler, and also provides the expert distribution as a supervision signal to the operative decoder. The decoder parameters only get updated when it is in operative mode.
Scheduled sampling. Scheduled sampling (Bengio et al.. 2013) is a curriculum learning approach to alleviate exposure bias. Instead of conditioning only the ground-trute context, the model conditions on a sequence ÿct that mixes tokens from the ground-truth sequence ye and model's previous predictions P-r-- Essentially, each decoding step, a ground-truth token is used input with probability of and the previous prediction is used with probability of (1 e) as:
Ve with probability 51
3t with probability (1
The method maintains decay schedule for the probability based on training steps such that the model is exposed to more self- self-generated tokens the later stage of the training. The model is trained with the standard MLE loss:
Lss
log P(y (())).)).)))
3 METHODOLOGY
Figure [ shows an illustration DYSI. Different from traditional MLE training, it consists of a dynamic scheduler and imitation module. The decoder is first run in teacher-forcing mode (henceforth referred to as teacher-forced decoder) to obtain an expert distribution, and later in operative mode (henceforth referred to as operative decoder) during training. The dynamic scheduler determines the sequence containing target and model-generated tokens that is provided as input to the operative decoder to perform training. The imitation loss constrains the operative decoder behavior to match the teacher-forced decoder's behavior.
3.1 DYNAMIC SCHEDULED SAMPLING
Most of the proposed variants scheduled sampling perform sampling differently, i.e., they use different sampling strategies decide which decoding positions should take the model-generated previous token as input. For example, [Bengio et al. (2015); Zhang et al. (2019b) uniformly sample e), where decays with training steps. Liu et al. (2021a) la) propose to select the positions where the model has high prediction confidence, egg. p())) 0.9. while Liu al. (2021b) propose to sample the positions based on both training and decoding steps with joint probability distribution function] Instead of proposing a new sampling method, we propose new scheduler that does not rely on training steps, but instead uses the model's performance directly to keep track of the training progress.
the decoding positions from sequence with a probability
Training progress can highly depend on the task, dataset, and experiment setup. For instance, Vaswani et al. (2017) report good performance on WMT" 14 En-Fr translation task with s 300K updates, while Ott et al. (2018b) need only xe 90K updates to get better results with the same model due to a larger batch size. A scheduler based training steps will inevitably require heuristic-based tuning
See Appendix E for approaches that do not adopt sampling strategies based training steps.
3

---

Under review as a conference paper at ICLR 2023
for different experimental conditions, which could be expensive. Moreover, such scheduler makes the assumption that all the training instances in a batch have the same training progress.
In light of this, we propose to use training time accuracy for the schedule. as training accuracy gives more direct feedback about the learning progress. As shown in Figure IT given a target sequence y (y1, (y1..... yr), first run the teacher-forced decoder obtain sequence of distributions over the vocabulary, (PI,..., Pr).
We then (greedily) sample the distributions to obtain the predictions (0) (0I...... ................................ (T) where l/2 arg mP. Vt E [L, .1,....23. ,T We can compute the training time accuracy for a sequence as Acc ÿ) = (Ei=1 Lyy = Tt)) The scheduler then decides the number of positions (NV) in the ground-trutie sequence to be replaced with tokens generated by the teacher-forced decoder as follows:
N ~ 3 (1, Acc(y, ().
where UI denotes a uniform distribution and B E [0, 1] is a hyper-parameter that provides further control on the sampling strength in addition to the inherent dynamic control according to training accuracy. Notice that N changes dynamically based on the training accuracy of each instance and is agnostic to training steps. As a sampling strategy, we choose N positions in the sequence randomly and uniformly. A random selection opposed to selection based on high confidence avoids confirmation bias (Tarvainen & Valpola| 2017) where the model accumulates its own errors, and potentially exposes the model to more varied input-output samples, which in turn helps the behavior cloning, as discussed in the next section. We view our method, dynamic scheduled sampling and imitation loss, as unified approach where the the random selection in sampling also contributes to the imitation process.
Ultimately, the output of the dynamic scheduler is a sequence that mixes the tokens generated by the teacher-forced decoder (i.e., tokens from ÿ) and the tokens from the ground-truth sequence y. This sequence ý is then used as the input for training the decoder in its operative mode. Note that with Transformer architecture (Vaswani et al. 2017), all the decoding steps can be trained in parallel (within layer) by masking out the future tokens.
3.2 IMITATION LOSS
In order to mitigate exposure bias entirely, the decoder should behave indistinguisaably whether is fed with ground-truth or a self-generated token input. Such a property would allow the model to generalize beyond the context sees in training. This has been proven effective by Goyal et al. (2016), where they borrow the idea of GAN (Goodfellow et al. 2014) to use a discriminator to distinguish between the hidden states of the decoder in teacher-forcing mode and inference mode. However, putting the decoder in inference mode during training makes the training slow as it requires sampling the full sequence in an autoregressive manner (Lee,, no parallelization). Also, training GANs for texts can be challenging as it requires the generator and the discriminator to be on par with each other.
We instead propose to close the gap between the teacher-forced decoder behavior and the operative decoder behavior in the MLE-based setup. To match the operative decoder behavior the teacher- forced decoder behavior, we bring the intuition of imitation learning. In particular, the operative decoder can be seen as a learner, which tries to imitate the behavior of an expert at each decoding step. This is also known as behaviour cloning.
Expert. As shown in Figure the expert, in our case, the teacher-foreed decoder, which provides demonstrations to the learner in the form of sequences of distributions over actions (tokens in the vocabulary). At each decoding step f, the expert takes the previous target token y/-1 as the observation, and maps its state 8 to the action distribution based on its policy Ta(s;) e IR Vi, where V is the vocabulary and the subscript tf stands for teacher-forcing. More formally,
Ti (*)) soft max(s:) Pa, (vel (vll)rr,)) x)
The expert-generated action distribution is regarded as the supervision signal to guide the learner.
Learner. The learner is the decoder runming in operative mode in Figure I Unlike the expert, it will not always take previous ground- truth tokens input, instead it will also use the predicted tokens from the expert (see arg max in Figure I according to the dynamic schedule ($3.I). Specifically, for an observed sequence ý comprising of both ground-truth and model-generatee tokens, the learner
4

---

Under review as a conference paper at ICLR 2023
generates an action distribution at every step as:
( = Po.p (UnlDer
where op denotes the operative decoder. Notice that the predicted tokens in ÿ provide new demon- strations (unseen in the original training data) for the learner comprising states that it may experience during inference. Since the learner and expert share the same parameters, also simulates the mistakes that the learner may make during inference. Overall, once trained, the operative decoder is expected to behave more robustly under different decoding conditions.
Learning from expert demonstration. To match the learner's 's policy with the expert's, ss we minimize the Setbaack-Leiblee divergence (Kullback & Leibler, 1951) (Dxl) between the two policies to guide the operative decoder behavior so that it better matches the teacher-forced behavior, considering the latter fixed:
T Σ
T
(Oop)
DEL (s)))
Σ DKL(Poo (Poy (3cl *) (Br 1t, x))
Imposing an auxiliary loss learn from the output distribution of the sorchr--oored decoder has another advantage. Although a teacher-forced decoder may fail to predict the exact target token in some positions, it (after being trained enough) often assigns a higher probability mass to the plausible translations in the context, such as synonyms of the target token Li & Lu (2021). Arguably, the soft output distribution contains much more information compared to the one-hot target, which helps the learning of the operative decoder (Furlancllo et al. 2018). In Appendix [D] we justify why we call the above learning process imitation rather than knowledge distillation (Hinton et al.) 2015).
3.3 OVERALL TRAINING OBJECTIVE
The generation model trained with combination of an MLE loss and the imitation loss:
[(Oop)
Σ
log Pa., (Jeli x) +a Σ DRL (Ra PP (vr (Brlii
(8)
MLE
Imitation
where a is hyper-parameter to control the relative weight and Our = Oop.
4 EXPERIMENTS ON MACHINE TRANSLATION
4.1 EXPERIMENTAL SETTINGS
Datasets & metric. We evaluate our model on two standard neural machine translation (NMT) benchmarks: WMT' 14 English-German Germann (En+Dee) and English-French (En->Fr). The training datasets contain about 4.5M and 35M sentence pairs respectively. We use newstest2013 and newstest2014 4 as the validation and test sets respectively. The sentences are encoded with joint Byte-Pair Encoding (BPE) (Sennrich et al. 2016) with 40K operations. For performance measure, following previous work, we report the tokenized BLEUU (Papineni et al.) (002). We also report other popular translation metrics, e.g., Bac (Post 2018). in Table 2 We use the Transformer big (Vaswani et al., 2017) as our backbone NMT model (refered to simply as Transformer henceforth). The learning rate warms up to a peak of 0.0008 with 4, 000 steps, and then decays with the inverse square-root schedule. The value of a in Eq. g and / in Eq. are set to 0.5 on both datasets. We use 0.1 habel-smoothing in training and beam size at inference. We train the models with DYSI and teacher forcing for the same number of updates. Appendix A gives further details about the setup.
We also compare against other MLE-based approaches: Correction (Song et al. (0221) SS SS-Steps (Liu et al. (2221b) and Scheduled Sampling (Bengio et al.) 2015). For SS-Steps, we run their publicly available code using the provided optimized parameters since that produced the best results. For Scheduled Sampling, we adopted the Exponential and Linear decay schemes from Bengio et al. (2015) and tuned the hyper-parameter based on the validation set performance. Note that the scheduled sampling (Bengio et al. 2015) that originally proposed for a recurrent architecture samples from the model generated tokens step by step, which is highly inefficient for training large models due to its inability to parallelize. The current paradigm, where
DynamieCony (Wu al., 2019 Error
5

---

Under review as a conference paper at ICLR 2023
the fracher-foreed outputs sampled as previous outputs, can be approximation and has been widely used in Transformer- based models (Duckworth et al. 2019 Mihaylova & Martins 2019 Liu et al. 2021a b).
4.2 TRANSLATION PERFORMANCE
We present the tokenized BLEU scores on WMT newstest2014 in Table I We observe that our method, i.e,, Transformer big trained with DYSI achieves 0,9 and 0.5 BLEU improvement on En-xDe and En respectively, over the the same model trained with standard teacher forcing. Our method also outperforms other MLEE-hased approaches that deal with exposure bias in NMT, such Scheduled Sampling (SS) and SS- SS-Steps. We also see that training with only Dynamic Scheduled Sampling performs better than or on par with Scheduled Sampling while requiring significantly less tuning, which demonstrates its contribution.
There have been claims that BLEU is not adequate for measuring MT performance and it correlates poorly with human judge- ments (Chaganty af.) (2018] Post, 2018). In addition to tokenized BLEU, we also report three other popular metries: deto- kenized SacreBLEU Lacre 2018), which tries to solve the problem of difficulty in comparing tokenized BLEU due to differ- ences in tokenization; BLEURT (Sellam
Table 1: Tokenized BLEU scores on newstest2014 for WMT' En->De and En->Fr translation tasks.
Models
En Rn=
En En-+Fr
Transformer (Ott et al. 2018b
29.3
43.2
DynamicConv (wu ct al. 2019
29.7
43.2
Error Correction (Song
2021
29.2
SS-Steps (Liu et al. 2021b
29.6
42.8
Our Implementation Transformer
al. 2020), which is learned evaluation
29.2
42.9
measure based on BERT and is trained with
+ SS (Bengio et al. 2015
29.5
43.0
human judgments; COMET (Rei al.
Dynammc SS
29.6
43.0
DYSI
30.1
43,4
2020). which leverages cross-lingual pre-
trained language modeling and exploits information from both the source and the reference translation to more accurately predict MT quality. From the results in Table we can observe that our training approach SY leads to improvement across all three metrics on both En-Dee and En En-*rr tasks. Our model can outperform the baselines by even larger margins when measured by COMET, which has a high correlation with human judgements.
Table 2: Additional metrics for NMT. ** and denote significantly better than the Scheduled Sam- pling with < 0.005 and 0.05, respectively.
Table 3: Ablation study a and / on WMT 14 En En->ee development set when the other set to the default 0.5.
Models
SacreBLEU
BLEURT
COMET
8
0.0
0.25
0.5
0.75
1.0
En-De
BLEU
26.70
27.05
27.14
27.11
27.06
Transforner
28.6
58.6
35.8
Scheduled Sampling
28.9
58.7
36.4
a
0.0
0.25
0.5
0.75
1.0
DYSI
29.4*
59.1
37.3"
BLEU
26.84
26.97
27.14
27.06
27.09
En-Fr
Transforner
41.0
54.2
59.8
Scheduled Sampling
41.1
54.2
60.0
DYSI
41,4*
54.5*
60.7"
4.3
ABLATION STUDY
Effeet of and o. We conduct ablation studies for the two hyper-parameters in DYST, Lee., B in Eq. Mand a in Eq. 8 to understand how they impact the performance. Intuitively, the larger / is, the more positions will be sampled and thus the operative decoder gets to see more of its own predictions during training. Table 3 shows that the performance on the validation set is generally robust to different values of 3 as long as is larger than certain value, e.g., 0.25. With a fairly small value of B, e.g., 0, the model deteriorates to standard teacher forcing the scheduler will not sample any positions for using model-generated tokens. On the other hand, a controls the extent which the operative decoder should imitate the teacher-forced behavior in addition to the original one-hot
LaC signature: nrefs: case mixxdd| eff noo cok:134 snooth:exp version:2 momn-obberaa-aa
We BLEUUT-200-11 We use

---

Under review as a conference paper at ICLR 2023
target. When 0, DYSI simply becomes another variant of vanilla scheduled sampling. From Table we observe that when a is small, there is a clear gap between model performance and the best result the validation set, which renders it necessary to include the imitation loss to further boost the performance. We also study two different training initialization strategies in Appendix B
4.4 ANALYSIS
As translation performance alone may not be sufficient understand how DYSI helps mitigate the exposure bias problem, we conduct further analysis to get more insights.
Multiple reference translation. We examine how the model performs against multiple semantically equivalent references. Particularly, we use the dataset from Ott et al. (2018a). which contains 10 additional human translations for 500 sentences from the WMT En En-*De testset. We present the results in T Oracle score computes BLEU for every hypothesis w.e.t. its best matching reference and averages it over all hypotheses. The oracle scores Table 4 indicate that the best scores that the models can achieve w.E.f multiple references are comparable. However, the higher corpus score, which measures the BLEU score with all the human references, means that our model has potentially produced more diverse translations, thus having higher coverage over different translations.
Table 4: Corpus BLEU and Oracle Sentence BLEU on WMT14 En->De test set with 10 additional references.
Table 5: Zero-shot translation perfor- mance on 1M Robustness En->rr task and IWLST' 14 En->De test set.
Multiple Ref.
Model
Single Ref.
Corpus
Oracle
Model
WMT' Rob.
IWSLT
Transformer
28.6
74.0
83.4
Transformer DYSI
29.2
DYSI
29.4
74.8
83.4
29.8
We conjecture that DySI prevents the model from being over-confident and makes the prediction distribution more spread out such that the model tends to diverse words. To confirm this property, we compute the entropy of the model generation distribution over the vocabulary as -Ewey P(y= w) log P(ye w),, and average it over all the decoding steps. The entropy values over the 1M En testset for our model and the baseline are 2.22 and 1.79, respectively, confirming our hypothesis.
Previous work (Ott et 20188a) points out that excessive spread of the generation distribution may be an indication of over-smoothing, which could happen due the application of label smoothing. However, unlike the standard label smoothing, where all the classes get smoothed uniformly, the imitation loss in our approach allows the model learn from the expert distribution through DKL. Learning from soft target distribution can be seen as an adaptive version of label smoothing, which in turn keeps improving for better model regularization and calibration (Müller et al. 2019).
Robustness. As mentioned before, exposure bias is closely related to generalization. Wang & Sennrich (2020) also indicate that exposure bias can be more problematic under domain shift. We thus test if DYSI I can lead to improvements under a distribution shift. For this, we use the models trained on the WMT' 14 dataset (news domain) to perform zero-shot translation on IWSLT' 14 En De testset (Cettolo et al.) 2012). consisting of transcribed and translated TED talks (spoken language text). In addition, we use the WMT 19 MT Robustness dataset (Michel Neubig 2018) (En->Fr) to investigate how DY performs both with domain shift and non- standaaddddd, noisy text. As shown in Table consistent improvements on both tasks indicate that the model trained with DY is able to deliver more robust performance compared to the baseline.
TOWARDS ROBUST TEXT GENERATION
has been observed that even with large pre-trained language models (LM), high frequency tokens largely dominate generated text (Welleck et al. 2020; Holtzman et al.] 2020 Lin et al. 2021). Repetition, both at a single token and at higher n-gram levels, is a well known problem in neural text generation. In addition, oversensitiviry (Jia & Liang 2017) is common issue, in which models produce significantly different outputs for very similar inputs, even when the changes preserve semantics (Ribeiro al. 2018).
7

---

Under review as a conference paper at ICLR 2023
30.45
53.3
10.91
$3.221
5.17
2.22
2:000 251.15
TINLE lass Uuvvat
3.50
1/0
1.221
4.5
3.570
0.44
(a) A 1-Gram Rep.]
A 2-Gram Rep.1
(c) 3-Gram Rep.]
Figure 2: First row shows results for Last Word Repetition. The x-axis plots the number of times
m 3, 5, 7, 10 that the last word repeated. Second row shows results for n- n-gram Repetition. The x-axis plots the various n- n-gram sizes that we repeat.
Table 6: Examples showing the robustness of GPT-2 trained with DYSI to various perturbations.
DYSI is significantly robust to repetition, even in cases of extreme perturbation that causes the baseline to fail irreparaby.. Text highlighted in red indicates changes due to perturbation, and orange indicates repetition induced by perturbation. (condensed for brevity; best viewed in color).
Perturhation
Perturbed Prompt
LM Generations
Last Word Repetition
By 2012, she was was was was was was
MLE: was, the Australian National Equestrian Champion, the Australian National Champion
DYSI: competition were underway, was placed third
u-gram
staming Bells
Doctor heginning
Repetition
starting with 'The Bells with "The Bells
and the Doctor
DYSI:".
would
We conduct experiments to test if training with DY can produce model that is more robust to these kinds of errors. We first fine-tune GPT-2 (Radford et al.] (019) on the WikiText- 103 (Merity al., 2017) training set with MLE loss (standard teacher forcing), Scheduled Sampling (SS) and DISI. We prompt the trained models with texts extracted from the WikiText-100 test set to perform the nuto-completion task. We then perturb these input prompts in an effort to instigate the models to commit repetition errors.
We report MAUVE (Pillutaa et al. 2021) scores to compare the machine generated texts with human produced text. MAUVE calculates the under divergence curves for text distributions and produces a score 5l where a higher number indicates closer text distributions. MAUVE has been shown to have high correlations with human judgments for open-ended text generation. We the n-gram repetition ratio difference to evaluate the variations that the perturbations cause. The n-gram repetition ratio measures how unique the n-grams in given text are, where a higher score indicates higher repetition and lower diversity. We report the difference between the n-gram repetition ratios of two texts for various n, which indicates if given text is more repetitive w.t. to another. robust model should produce diverse and consistent output even when the prompt is perturbed.
AUTO-COMPLETION
Fine-tuning LM. We fine-tune GPT-2 Medium on WikiText- 103 (Merity et al. (017) with sequence length of 300 tokens and initial learning rate of 0.0005. Each model trained for maximum of 35K iterations and evaluated based on the perplexity on the validation set after every IK iterations. The perplexity scores the test set for MIE, SS and DYSI are 13.4, 14.3 and 13.9, respectively.
We report scores scaled between 0-100 for readability.
8

---

Under review as a conference paper at ICLR 2023
Prompts for auto-completion. We use the test set from WikiText- 103,, which is extracted from set of verified good and featured articles on Wikipedia. We extract the paragraphs from the document and use the first 50 words as the imput prompt to the trained language models. The models need to generate a continuation of 200 BPE tokens based on the prompt. We apply nucleus sampling (Holtzman et al., 2020) (p- 0.8) as the decoding strategy since it leads high-quality and human-like generated text.
Comparison to human. We compare the texts generated by the baseline MLE and the DYSI model to the original human continuations using MAUVE. We sample continuation text three times from each model and report the average and standard deviation. MLE achieves a MAUVE score of 71.88 + 9.48, SS achieves MAUVE score of 72.46 2.76, while DYSI achieves score of 73.08 + 3.64. DySI has higher MAUVE score, showing that it produces text that is closer to human- text. In addition, the baseline MLE model has a significantly higher standard deviation compared to S$ and DYSI, showing that models trained with methods that alleviate exposure bias are also more consistent across multiple samplings.
5.2 PERTURBATION EXPERIMENTS
We use various strategies to perturb the prompts and compare the n-gram repetition ratio differences of model outputs for the perturbed prompts to the model outputs for the original prompts. For these experiments, we sample each model twice for both the original and perturbed prompts, and report the average of all 4 combinations. We also include a comparison of their MAUVE scores in Appendix F
Last word repetition. To test the robustness of the models to repetition, we repeat the last word of the prompt m 3, 5, 7, 10 times, and plot the difference in 1, 2, and 3-gram repetition ratios of the generated text with respect to the text generated for the original prompt. From Figure (row 1), we see that the repetition ratio changes increase significantly with m for the two baseline models. However, DYSI is much more robust. It produces a significantly lower repetition ratio difference.
n-gram repetition. In this setup, we repeat the last n words of the prompt to test whether DYŞI is also robust to repetitions involving longer sequences of words instead of only a single repeated word. We experiment with repeating the last n = 3, 5, 7, 10 words and plot the 1, 2, and 3-gram repetition ratio difference of the generated text with respect to the text generated for the original prompt. Interestingly, we see in Figure (row 2) that repeating a longer sequence of words leads to an increase in the repetition ratios for higher order n-grams for the MLE baseline. In contrast to both MLE and SS, DYSI maintains a low repetition ratio difference.
Table shows examples of the outputs generated by the baseline MLE and DYSI models for various perturbed prompts. DYSI produces reasonable outputs even with extreme perturbations, and is remarkably robust repetition perturbations. [e et al. (202T) show that LMs trained with teacher forcing have the ability to self-recover. Our experiments demonstrate that DY can significantly enhance the model's self-recovery ability from erroneous generation, especially for repetition errors. We also conduct experiments to trigger oversensitivity by perturbing prompts through word replace- ment. The full experiment setup and results are given in Appendix [F] Overall, we find that LMs are generally robust to this, with both SS and DYSI doing better than MLE.
Experiments on more settings. We examine if the abovementioned findings are consistent with larger model size. From the results in Appendix G we see that even with a larger model size (774M), the model trained with the standard teacher forcing is still very sensitive to repetition errors, leading to significant text distribution changes when the repetition error happens. On the other hand, DYSI consistently leads to a more robust language model regardless of the model size. Appendix H further shows the above observations still hold when different decoding algorithms, e.g., top-k. top-p sampling, are applied.
CONCLUSION
We have introduced Dynamic Scheduled Sampling with Imitation Loss (DYSI) to combat one of the most well-known problems in autoregressive text generation, exposure bias. It consists of dynamic sampling scheduler, which keeps track of training progress based on the training accuracy, and imitation loss, which enables the model learn from the expert's 's behavior. DY achieves consistent improvement on several translation tasks and experiments. Furthermore, extensive analysis demonstrates that it can yield significantly more robust text generation model.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URL http:: org/ abs/1409, 04 Seiv:1149.04473oom
cite
Accepted at ICLR 2015 as oral presentation.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1171-1177 Cambridge, MA, USA, 2015. MIT Press.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 41 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374.55555
URL /tt.::::://....///11.... 111.11111111555555 1553380
Tom B. Brown, Benjamin Mann. Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Mauro Ccttolo, Christian Girardi, and Marcello Federico. Wit3: Web inventory of transcribed and translated talks. In EAMT, 2012.
Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 643-653, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P18/ 1060. URL https? hrw.la/..oooooy.orr/o111-------.
Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. Parallel scheduled sampling. 2019.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anand- kumar. Born-again neural networks. In ICML, pp. 1602-1611, 2018. URL http://
proceedi ngs ml 1 orr.eeeddcng.m....rrssss//////e
1ol.a. Ba. html
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Sebastian Goodman, Nan Ding, and Radu Soricut. TeaForN: Teacher-forcing with n-grams. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Processing (EMNLP), pp. 8704-8717, Online, November 2020. Association for Computational Linguis-
tics. doi: 1/2.66//111/222.mmmmmm... 702. URL UBLL:ttees////llaaccooooo..oo
Con.lp-mas main. 702
Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang. Aaron Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4608-461 6, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Exposure bias versus self-recovery: Are distortions really incremental for autoregressive text generation? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5087-5102, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18865340 wt2.22.eeellpp1eeeeeeee www.e-mmainnnnn....... 415. URL https:
emm.pp main. emn.p-main. 415
Geoffrey E. Hinton. Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, 503.02333.. 2015.
10

---

Under review as a conference paper at ICLR 2023
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https?
net/ htt.:/orrmmmmmdmm0 /
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. ArXiv, wttpab//....311211 2017.
S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Statist,, 22(1):79-
86, 03 1951. doi: 1/1111//////1177////0 URL https:://do/ htttps:///oi.orr/100 12mm/
1177729694]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7888 Online, July 2020. Association for Computational Linguistics. doi: 10. a0l18663///22000aaa---aaaa 703- URL http:s://aclantho..o
wto. 200gg/2220ccom aci.mmnn.lmmmin.n.... 703
Haoran Li and Wei Lu. Mixed cross entropy loss for neural machine translation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 1425-6436. PMLR, 18-24 Jul ngs ml press/pssseeo.......... http//l31/11111 htmml
2021. URL htttt::tt///prtt.t
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 3158-3168 Red Hook, NY, USA, 2017. Curran Associates Inc.
ISBN 9781510860964.
Xiang Lin, Simeng Han, and Shafiq Joty. Straight to the gradient: Learning to use novel tokens for neural text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6642 6653. PMLR, 18-24 Jul 2021. URL httpps://proceeedgggs mlr press/v139/1s 1ttp99/111211b..mm html
Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Confidence-aware scheduled sampling for neural machine translation. In Findings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021. pp. 2327-2337, Online, August 2021a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2211
findings-acl.205.. URL phttps://acleeee https://acaaaatctc www.alnnoollyyy /rg/
2021. fl nd ngs- 2005
Yijin Liu, Fandong Meng. Yufeng Chen, Jinan Xu, and Jie Zhou Scheduled sampling based on decoding steps for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3285-3296, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 10.18653/v1/200 org/2021
emnlp-main.264.
emnlp main. 264
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1197. 1692, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017. URL http:tttss/:/peeeetttt http:e/weee.//p....o---yyya
Paul Michel and Graham Neubig. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Mtnt: testbed for machine translation of noisy text. In
Tsvetomila Mihaylova and André Martins. Scheduled sampling for transformers. In Proceedings ACL SRW, 2019.
Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When Does Label Smoothing Help? Curran Associates Inc., Red Hook, NY, USA, 2019.
11

---

Under review as a conference paper at ICLR 2023
Weili Nie, Nina Narodytska, and Ankit Patel. RelGAN: Relational generative adversarial networks for text generation. In International Conference on Learning Representations, 2019. URL
https:://ppeneeiiet
net/ htt.raaeeeeeeeeeeeoouuorraddada Eorummi
Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning mlr
Research, pp. 3956-3965, PMLR, 10-15 Jul 2018a. URL
gress/v80/o0 press/ /t0///000///// ott 8a htm.mmml
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation ( (WMT), 2018b.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In International Conference Learning Representations, 2021. [RLLsttpss//pperrevii.w
forum?
-id=Rovx-QQ13
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Amnual Meeting of Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association URL https: //www/wwwww...../ ac weww
for Computational Linguistics. doi:
hrr/aanthology/PO22
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In NeurlPS, 2021.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, October 2018.
Association for Computational Linguistics. doi: 10. 11118663/1111--631999 URL https:/7
www. www.accweb..o wwwp:allveb..rg/aattolooooo wwt.:r///aattoooooo111--36633
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Open-Al Blog, 2019.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, Online, November 2020. Association for Computational Linguistics. doi: 10. 8653/v /20000e0mmpmmmmii....... URL https: //acl http///claattoooy....oooo org 2020. emn emnlpmnama i 213
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In ACI, 2018.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 11885//11/111-- 1162. URL httttttss///ccannn Srg/PP6-1166)
Kaitao Song, Xu Tan, and Jianfeng Lu. Neural machine translation with eтrоr correction. In Proceedings the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20, 2021. ISBN 9780999241165.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models:
consistency targets improve cemi-supervised deep learning results. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 1195 1204, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978151086096
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 3998-6008, 2017.
Chaojun Wang and Rico Semnrich. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3544-33 -3542, Online, July 2020. Association for Computational Linguisties. doi: 10.18653/w1/2020.cce acl- m26n.326. URL ttpps//acllaattooorrryoooooo wwtp://ooooooo../o//2///// acl -ci-main
326
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference Learning Representations, 2020. URL
net/ www.ee/oooueeeeeeeeeeeee..e.
R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Compulation, 1(2):270-280, 1989.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019. URL/ittpss//pperreveee...cc
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference Artificial Intelligence, 17, pp. 2852-2858. AAAI Press, 2017.
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019a.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4334-4343, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653.4 18653/v 1/P19-1429 URL https: https:://aaaaatthooccoo
13

---

Under review as a conference paper at ICLR 2023
A HYPER-PARAMETERS FOR NMT EXPERIMENT
1M En->De. The learning rate warms up to a peak of 0.0008 with 4, 000 steps, and decays with the inverse square-root schedule. We apply 0.3 dropout rate, 0.1 label smoothing and 0.0 weight-decay. We use Adam optimizer with & being (0.9, 0.98). The training batch size is around 350K tokens. For the random initialization, initialization, we first the train the model with teacher forcing for 15K updates and use the checkpoint to start retraining (reset the learning rate scheduler) with DYSI or teacher forcing (baseline) for 35K updates. We save the checkpoints every 500 updates and we average the last 10 checkpoints to obtain the final model. During inference, we use a beam size of 5 and a length normalization factor of 0.2, which is tuned the validation set.
train the model for 35K updates. For the hot start
14 En-*Fr. The learning rate warms up to a peak of 0.0008 with 4, 000 steps, and decays with the inverse square-root schedule. We apply 0.1 dropout rate, 0.1 label smoothing and 0.0 weight-decay. We use Adam optimizer with 3 being (0.9, 0.98). The training batch size is around 550K tokens. For the random initialization, initialization, we first the train the model with teacher forcing for 30K updates and use the checkpoint to start retraining (reset the learning rate scheduler) with DY or teacher forcing (bascline) for 50K updates. We save the checkpoints every 500 updates and we average the last 10 checkpoints to obtain the final model. During inference, use beam size of 5 and a length normalization factor of 0.8, which is tuned on the validation set.
train the model for 50K updates. For the hot start
Computation hardware. We conduct the experiment on machine with CPU Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz, and 8x GPU Quadro RTX 6000.
B INITIALIZATION STRATEGIES
Hot start. We investigated two strategies to initialize the training with DYSI, namely, random and hot start. In random, all the model parameters are randomly initialized. The other initialization strategy is to first train the model with the standard teacher forcing as it stabilizes training in the initial stage, thus referred to as hot start. Specifically, we first pre-train the model with teacher forcing in the standard setup ($4.1) I for 15K and 30K updates on WMT' 14 En-De and En-FFr, respectively, and then use the checkpoint to start training with DYSI using the setup described in $4.1
Table 7: Ablation study of initialization strategies on WMT" 14 En DDe development set.
Config
Random
Hot Start
Transformer
26.6
26.7
DYSI
27.0
27.1
Table shoows the performance on the WMT" 14 En De validation set for the strategies. We additionally include the results for the baseline model (teacher forcing) trained with hot start for comparison. Specifically, we take the same pre-trained checkpoint as above for initialization and start training the model with the same training method used to train from scratch. We can see that hot start can generally result in a slightly better performance. We thus adopt the hot start strategy for the NMT experiments. Note that pre-trate tre-trannng the models with standard teacher forcing is not exclusive for our approach. In fact, earlier work have adopted the same strategy (Liu et al. 2021a b).
C PERFORMANCE WITH VARIED LENGTHS
We present translation performance with different reference lengths in Table 8 We see that model is able to consistently outperform the baseline w.r.t. all the length buckets. In particular, when the length is longer than 10, the performance gap becomes more significant.
14

---

Under review as a conference paper at ICLR 2023
Table 8: Translation performance in BLEU on WMT' 14 datasets with varied reference lengths.
Length
(0,10)
[10,20)
[20,30)
2 30
Transformer
22.86
28.35
29.22
29,89
DYSI
23.22
29.33
30.18
30.42
No. of Sent.
357
1153
838
655
D WHY CALL IT IMITATION
Why call it imitation. We call the above learning process imitation rather than knowledge distillation or KD (Hinton et al. 2013) for two reasons. First, training an autoregressive generation model can be naturally seen as learning a policy as the model learns make a sequence of decisions (Pang He 2021). Second, KD generally seeks to make student model learn better from the knowledge extracted from a teacher model (e.g.. the prediction distribution) conditioned on the same input. However, in our case, the learner may be exposed to a different observation sequence compared to the expert (y vs. ÿ). In other words, the operative decoder is supposed to imitate the behavior of teacher-forced decoder at each step regardless of the different input.
E RELATED WORK
We discuss another line of related work that does not require scheduler based training steps. TeaForN (Goodman et al. 2020) uses a stack of N temporal decoders trained to decode along secondary time axis that allows updating model parameters based on N prediction steps, with N being a hyper-parameter. Each decoder gets a predicted previous token as input from its prior decoder except for the first decoder, but the training cost increases linearly with N. Although does not indeed require curriculum learning, the main difference is that it requires the loss to be backpropagated all N decoders, while our approach only requires backpropagation to one decoder. SS-Conf (Liu et al.] 22221a) decides decoder positions that will receive model outputs based on model confidence, which also does not require sampling based on training steps. However, using a model's confidence is generally not robust as the confidence is closely related to model calibration.
TeaForN achieved 0.1 and 0.4 SacreBLEU improvements on WMT' 14 En->De and En-xFr with Transformer Big compared to vanilla Transformer with teacher-foreing, On the other hand, DySI achieved 0.8 and 0.4 SacreBLEU improvement on WMT' 14 En->De and En-FFr, respectively. As for SS-Conf, compare to its latest following work SS-Steps (Liu et al. 202 in the main paper.
F MORE AUTO-COMPLETION RESULTS
We present more results for auto-completion experiment in this section.
Word replacement. To trigger oversensitivity in the language models, we randomly replace content words (i.e., nouns, verbs, adjectives and adverbs) in the original prompt with another word. To find reasonable replacement, we mask the words and use a trained RoBERTa base (Liu et al. 2019) model to generate the replacement word. We experiment with replacing k 5, 10,20 words and plot the difference in 1, 2, and 3-gram repetition ratios of the generated text with respect to the text generated for the original prompt. We both SS and SY doing better than MLE.
Figure that LMS are generally robust to word replacement, with
MAUVE scores. We present the MAUVE scores between the generated texts from different models prompted by the original prompts and the perturbed prompts in Figure 9 We can observe that similar to prior results in 2 the model trained with DySI significantly more robust to last word repetition and n-gram repetition problems compared to other baselines.
Ablation for DYSI. We provide the ablation results for proposed dynamic scheduled sampling (DS) and imitation loss. In Table I0, the numbers (except PPL) are the average of 1, 2 and 3-gram repetition changes when prompting the trained model with the original and the perturbed text. The best scores between vanilla scheduled sampling (SS) and DS marked hold and the overall best
15

---

Under review as a conference paper at ICLR 2023
0.5
04-
0.32
0:51 04
0.25
(0:33
0.770s
6/0
10
(a) LA 1-Gram Rep.]
(b) A 2-Gram Rep.
(c) | 3-Gram Rep.
Figure The results for Word Replacement. The x-axis plots the number of words replaced,
k = 5, 10, 20.
Table MAUVE scores between continuations generated by the same model with the original and perturbed prompts.
Replacement
Last Word Rep.
n-gram Rep.
Models
20
3
5
7
10
MLE
96.20
96.54
95.00
95.72
85.71
73.45
34.76
95.58
95.87
95.56
94.65
SS
96.46
96.40
96.82
96.52
95.95
85.31
57.06
96.75
95.94
95.94
95.70
DysI
96.49
97.09
96.54
97.20
96.69
95.13
92.56
96.77
96.87
97.25
97.01
performances are marked with pink. Compared SS, DS has similar overall performance but is much more robust to last word repetition (also refer to the NMT ablation above). see that DYSI outperforms others notably in almost all cases due to the imitation loss.
Table 10: Ablation study for auto-completion experiment.
Replacement
Last Word Rep.
7-gram Rep.
Models
PPL.
5
10
20
3
5
10
3
5
7
10
SS
14.3
0.08
0.23
0.26
0.16
1.80
7.43
19.02
0.45
0.75
1.44
2.52
DS
14.7
0.38
0.44
0.23
0.27
1.72
4.24
11.20
0.49
1.01
1.43
3.46
MLE
13.4
0.20
0.32
0.63
0.19
7.82
12.88
30.57
0.41
0.98
1.50
3.30
DYSI
13.9
0.15
0.12
0.29
0.19
0.45
0.93
3.69
0.29
0.08
0.44
1.32
Full examples. Table 11 shows more examples in auto-completion experiments.
G PERTURBATION EXPERIMENTS WITH LARGER MODEL SIZE
We replicate the perturbation experiments with GPT-2 Large (774M). Specifically, we experimented with the perturbation setup of Last Word repetition with = 10 (the last word in the prompt is repeated 10 times), and n- n-gram repetition with n 10 (last words are repeated). We choose this task to demonstrate that in settings with the maximum perturbation, both the bigger and smaller margin differences between the baseline and DY still hold even with GPT2-L We sampled the outputs twice from the models trained with standard teacher forcing and DI and report the n-gram repetition changes over combinations.
From the results in Table 12 we that even with larger size, the model trained with the standard teacher forcing is still very sensitive to repetition errors, leading to significant text distribution changes when the repetition error happens. On the other hand, SI consistently leads to more robust language model regardless of the model size. This experiment shows that SY can be effective even with larger model size, since larger model size does not necessarily resolve exposure bias.
16

---

Under review as a conference paper at ICLR 2023
Table 11: Examples showing the robustness of GPT-2 trained with DY to various perturbations.
DYSI is significantly robust to repetition, even in cases of extreme perturbation that causes the baseline to fail irreparably. Further, output of the baseline MLE training changes completely even when a single token is replaced, but DYSI output maintains the general semantics. Text highlighted in blue indicates original text, red indicates changes due to perturbation, and orange indicates repetition induced by perturbation. (condensed for brevity; best viewed in color).
Perturbation
Perturbed Prompt
LM Generations
Last Word
By 2012, was
MLE: was, Australian National Equestrian Champion
Repetition
was was was was
the Australian National Champion
DYSI: competition were underway. was placed third
Last Word Repetition 2
He was assigned to its developinenal promotion Ohio Valley Wrestling ( (VW)
MLE: (
DVSI: 2002. Lesnar was involved in the successfull submission
)))3
wrestligg a eventually become a member of the WWE
n-gram Repetition
character that would travel with the Doctor
MLEE:
and the Doctor starting with " "The Bells and the Doctor
stariing with "The Bells with "The Bells
beginning with The Bells and the Doctor
DVSI." It was announced that the third series would be "Clara"
n-gram
the Reconnaissance Company particularly distinguished himself. He mounted himself
MLEE: the right flank of the 1st and 2nd Companies and he mounted
Repetition
the right flank of the 3ed and 4th Companies. He mounted his left
He mounted
DYSI: the tank, hold with
in the sir, and lifted the pole.
Word
After some deliberation Nerva adopted Trajan,
MLE given the title of Caesar 85.
Nervo attempted light
Replacement
young
wealthy and popular general
DYSI:
his heir.
succeed as emperoe:
Word
reaching 88 at the crater floor
MI.E: the freezing temperature of the atmosphere
the salt water
Replacement
at the sea level Under these conditions,
in the craters forms a "fossil" record
2
DYSI: the shudow temperature is would
any significant temperattures
Table 12: Results for perturbation experiment with different model sizes.
A 1-gram
A 2-gram
A 3-gram
Last Word Repetition
MLE (Medium)
20.69
33.50
37.53
DYSMMedi)m)
2.72
3.73
4.63
MLE (Large)
16.21
26.20
28.99
DYSI(Large)
2.16
3.03
3.37
n-gran Repetition
MLE (Medium)
2.29
3.80
3.82
DY S((Mediuum)
1.12
1.44
1.40
MLE (Large)
2.34
3.52
3.36
DY (ILarge)
1.20
1.41
1.18
H PERTURBATION EXPERIMENTS WITH DIFFERENT DECODING METHODS
In this section, we conduct experiments on the perturbation task using two of the most popular decoding methods: top-k and top-p sampling with different values of and p. Although we used top- p sampling in our paper with p = 0.8, here we show more configurations for comparison. Specifically, we experimented with perturbing the original prompt by repeating the last word 10 times and report the 1, 2, and 3-gram repetition changes (before and after) of the generated text with respect to the text generated for the original prompt. Following the same setup as our main experiment, we sample each model twice for both the original and perturbed prompts obtain the generated text. The results are averaged over 2 generated texts.
From the results in Table 13 we can observe that the model trained with standard teacher forcing is still sensitive to the repetition errors despite different decoding strategies and different sampling strengths. For example, the generated text has 3- 3-gram repetition ratio of 0.185, which is significantly higher than natural text (0.033), even with a very strong sampling strength (Top-p with = 0.95). If compare the outputs with the perturbed prompts to the outputs with the original prompt using the
17

---

Under review as a conference paper at ICLR 2023
Table 13: Results for perturbation experiment with different decoding algorithms.
A 1 1-gram (OP/PP)
A 2-gram (OP/PP)
A 3-gram (OP/PP)
Humans
0.362
0.089
0.033
Top-k
Teacher foreing (k=40)
0.469/0.529
0.166/0.255
0.082/0.17
DySI (k=40)
0.472/0.492
0.153/0.163
0.068/0.073
Teacher forcing (k=90)
0.446/0.501
0.150/0.230
0.073/0.15
DySI (k=90)
0.442/0.456
0.129/0. 134
0.054/0.058
Top-p
Teacher forcing (p=0.4)
0.660/0.87
0.444/0.811
0.340/0.77
DySI (p=0.4)
0.649/0.710
0.274/0.501
0.298/0.044
Teacher foreing (p=0.95)
0.400/0.486
0.120/0. 245
0.057/0. 185
DySI (p=0.95)
0.392/0.391
0.102/0. 100
0.043/0.045
same decoding configuration, we see that DY is generally very robust across different decoding strategies and different sampling strengths.
18


Reviews:
[{"review_id": "Uvy2AYg3QSX", "reviewer": "Reviewer_MTWV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Dynamic Scheduled Sampling with Imitation Loss (DySI), which combines knowledge distillation and scheduled sampling for text generation.", "summary_of_the_review": "This paper applies KD to dynamic schedule sampling but lacks novelty.", "strength_and_weaknesses": "## Strengths\n1. This paper proposes a new decoding training method that combines scheduled sampling with knowledge distillation. \n1. The performance looks good and the authors also run a significance test.\n1. The method is well-motivated and reasonable.\n\n## Weaknesses\n1. Isn't imitation loss knowledge distillation? I see the explanation in Appendix D (probably it's added because it was once asked by reviewers of another venue) -- I'm not convinced by the explanation there. It makes people confused since it hints about the use of reinforcement learning, which is not the case. It simply adds a KD loss to learn from the teacher-forcing distribution.\n2. Applying KD to decoding is nothing new. Non-autoregressive decoding uses knowledge distillation to learn from autoregressive decoding. This is very similar to what's proposed in this paper while there isn't a discussion about it. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "PW-NT3U-JGU", "reviewer": "Reviewer_tF2J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a simple modification of scheduled sampling for transformers that allows them to get better performance on a variety of tasks, including NMT and text generation. Importantly, they analyze why exactly their method does better and produce pretty conclusive quantitative results of this.", "summary_of_the_review": "Authors apply a minor tweak to scheduled sampling and it works a little better than before.", "strength_and_weaknesses": "This paper does a good job of analyzing why using DySI lets the model produce better, more robust output. They perform the requisite ablations of their method to show it helps. One thing they don't do is check whether repetitions occur with the same frequency as when using regular SS. That is, is their method just a training time optimization (specifically the KL term addition), or does it produce qualitatively different results when you use DySI.\n\nI don't think the NMT results are very impressive. The BLEU scores are < 1 point better, and not even an improvement on the state of the art but rather a Transformer baseline. In my opinion these results are not good enough to publish on.\n\nThe abstract has a broken link to the code, so I couldn't review it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "M2a4-2Ao8V", "reviewer": "Reviewer_8LdF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents two techniques to solve the exposure bias. The first one, dynamic scheduled sampling, is to sample the generated token conditioned on the accuracy. The second one, imitation loss, is to regularize the training. The authors conduct experiments on machine translation and robust text generation to evaluate their methods.", "summary_of_the_review": "Overall, this work tries to solve an important problem of sequence generation and has lots of potential. However, I still have some concerns about the design.\n\n1. Low accuracy comes from two cases: (1) the model is not well trained (e.g., in early stage), (2) the output is actually correct but different from the reference. In case (1) it is reasonable to not sample too much from $\\tilde{y}$, but not in the case (2). \n\n2. The accuracy is computed without any alignment. Therefore, we will have a low accuracy when there is just one token deleted/added.\n\n3. The accuracy is only based on single token, not considering the longer sub-sequences such a bi-grams or tri-grams.\n\n4. The accuracy is not considered during sampling. Therefore, when the accuracy is high, there is a high chance that the sampled tokens are identical to reference, making the method less effective.\n\nTherefore, I suggest the authors conduct more theoretical and empirical studies of this part, to make the work more persuasive.\n", "strength_and_weaknesses": "Strength:\n\n- The paper is well written and easy to follow.\n\n- The experiments are comprehensive, and the results are promising\n\n- The idea that using the signal of training process is kind of novel. \n\nWeakness:\n\n- The design of the sampling strategy, especially the Eq. (4), is heuristic. More motivation and analysis of Eq. (4) is required.\n\n- Forwarding a decoder again with modified sequences then computing a loss is not novel because it’s already been used before in similar tasks (e.g., contrastive loss in [1] and consistency loss in [2]). The authors should discuss the differences with them.\n\n[1] BRIO: Bringing Order to Abstractive Summarization, ACL 2022\n\n[2] Target-Side Input Augmentation for Sequence to Sequence Generation, ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject