Parallel Scheduled Sampling
Authors: Daniel Duckworth
Arvind Neelakantan
Ben Goodrich
Lukasz Kaiser
Samy Bengio
Venue:  (2020)
Keywords: deep learning
generative models
teacher forcing
scheduled sampling

Abstract: Auto-regressive models are widely used in sequence generation problems.  The output sequence is typically generated in a predetermined order, one discrete unit(pixel or word or character) at a time. The models are trained by teacher-forcing where ground-truth history is fed to the model as input,  which at test time is replaced by the model prediction. Scheduled Sampling (Bengio et al., 2015) aimsto mitigate this discrepancy between train and test time by randomly replacing some discrete units in the history with the model’s prediction. While teacher-forced training works well with ML accelerators as the computation can be parallelized across time, Scheduled Sampling involves undesirable sequential processing. In this paper, we introduce a simple technique to parallelize Scheduled Sampling across time. Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forced training.  n dialog response generation task,Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over teacher-forcing while in image generation it achieves 20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score (IS) respectively. Further, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.

Under review as a conference paper at ICLR 2020
PARALLEL SCHEDULED SAMPLING
Anonymous authors
Paper under double-blind review
ABSTRACT
Auto-regressive model architectures are the workhorse of sequence generation tasks in text, images, audio, and more. These models are predominantly trained with teacher-forcing, wherein the model is encouraged to predict the next ground-truth token conditioned on all previous ground-truth tokens in the sequence. At test time, the model does not have access to ground-truth tokens and must instead condition on its own prior predictions, resulting in a train-test discrepancy known exposure bias (Ranzato al., 2015). Scheduled Sampling (Bengio et al.) (013)) aims mitigate exposure bias by randomly replacing some tokens with model predictions during the course of training. Unfortunately, this technique inherently sequential and unlike teacher-forcie foreing, cannot be trivially parallelized across time. In this paper, we introduce a simple technique to parallelize Scheduled Sampling, significantly increasing training throughput. Experimentally.
find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forcing. In dialog response generation, Parallel Scheduled Sampling achieves a 1.6 BLEU score (1 (11.5% improvement over teacher-forcing with more than 250 times higher throughput than Scheduled Sampling. In image generation, our approach achieves 20% and 13.8% improvement in Fréchet Inception Distance (FID) and Inception Score (IS) respectively. Finally, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.
INTRODUCTION
Auto-regressive models are popular choice for generating sequences of any kind including audio (van den Oord et al.) 20T6b), images (yan den Oord et al. 2016a), and text (Sutskever al., 2014) Cho et al. 2014). Here, the joint probability of the sequence is factorized in pre-determined order during train and test time, For example, auto-regressive models for text generation factorize the joint probability left-to-right. The text sequence is generated by a decoder network left-to-right, one token (word or word-piece or character) at a time and are widely used in text generation tasks such as summarization (Liu et al., 2018), machine translation (Sutskever et al.) 2014] and dialog response generation (Budzianowski et al. 2018) in the encoder-decoder (Cho et al. 2014, SStskkverr et al. 2014) setting. Such models are typically trained by leacher-forcing (Williams and Zipser 1989) where ground-truth history is fed to the model as input, which at test time is replaced by the model prediction. Auto-regressive models applied in any domain suffer from this train-test time discrepancy,
Scheduled Sampling (Bengio et al. 2015) aims to mitigate the discrepancy between train and test time in teacher-forcing by randomly replacing some tokens in the history with the model's prediction. More concretely, at given time step in generating the output sequence, the model is conditioned either on troun or model prediction from the previous time-step with some probability. The probability of selecting model predicted token is gradually increased as training progresses. This procedure potentially allows the model to recover from its own errors, and [Bengio et al.] (2015) observe better empirical performance in natural language parsing, image captioning, and speech recognition compared to reacher-forced training. Scheduled Sampling has also been used to get better performance in other tasks such video prediction (Finn al. (016)), knowledge base completion MMwa and Bansal 2016) and piano music transcription (Sigtia et al. 2016)..
A key bottleneck in training models with Scheduled Sampling is its inherently sequential nature. Unlike feacher-foreing, tokens must be processed one time-step at a time. The sequential procedure makes Scheduled Sampling impractical for training neural networks, particularly problems

---

Under review as a conference paper at ICLR 2020
K=1
K-2
K-3
Target
Lass
Semple
Logits
Condition
Figure 1: Parallel Scheduled Sampling for K 3 passes. On each pass, the model is conditioned on the previous pass's output. New tokens are sampled according to the model's logits, which are then mixed with the gold sequence. After the final pass, the loss is calculated with respect to the gold tokens.
involving long sequence generation. In this work, we describe a simple technique parallelize Scheduled Sampling. Given an input example, we first generate an entire model prediction sequence parallel by conditioning on ground-truth history (equivalent to forward-pas -aaadd-passsae of teacher-forcing). Then, we employ (parallel) mixing step where we generate a new sequence whose token at every time step is either from the model prediction or the ground-truth. Finally, we perform training as in teacher-forcing by conditioning the sequence obtained from mixing. In Section 2.3 we show that by performing multiple passes parallel prediction and mixing, we obtain a conditioning sequence that converges to sample decode from the model.
Our key contributions are,
A novel, fully parallelizable approach to reducing train-test discrepancy of auto-regressive models without modification. We believe this to be the first approach that does not require full-sequence decoding within the training loop. We find the method produces the same empirical benefits of Scheduled Sampling while using as little as 0.3% of the training time.
We prove equivalence of the proposed approach to Scheduled Sampling under certain choices of hyperparameters. This recovers the clear interpolation between training-ting time teacher-forcing and test-time decoding described in Bengio et al. (0015).
We extensively evaluate our approach on four auto-regressive tasks in text and image domains. We find that Parallel Scheduled Sampling matches Scheduled Sampling's benefits, takes massively less compute time, and significantly improves performance compared to teacher-forcing most tasks. In dialog response generation task, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over fracher-forcing while in image generation it achieves 20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score (IS) respectively.
2
METHOD
Our proposed technique can applied to both conditional and unconditional auto-regressive genera- tive models. For notational simplicity, we consider the task of conditional sequence generation. The training set is given in terms of N input-output sequences ((x², )² where x² is the input and target y² is the desired output. The target y" is variable-length sequence of T; tokens (or pixels), (vf- ......, ... vr, whereas x' may be variable-length (as in translation) or fixed-length
in image
captioning). The goal is to learn model that accurately predicts y given x². We use fı to denote the sequence of tokens (y), 32....,
2

---

Under review as a conference paper at ICLR 2020
2.1 TEACHER-FOR AND DECODING
Given an input x and target the log-probability of the target can be decomposed autoregressivell:
P(u[=) II P(uelvn=-1;3))
Auto-regressive sequence generation models learn assign high likelihood to token y/ given previous
target tokens Vis--1 (V1..... 1/t-1) and inputs a via a learned likelihood model Pg- Neural
language models such as RNNs (Mikolov al. 2010) and Transformer (Vaswani al.
2017) adopt
left-to-right decomposition while image generation models (van den Oord et al. 20 ba 2018) adopt raster-scan order.
Parmar et al.
Such models are typically trained with facher-forcing (Williams and Zipser forcing, the log likelihood of the training set is directly maximized,
1989 In teacher-
T. log po (oioooii
argmax Li((0) = argmax
Importantly, forcing conditions on gold target prefixes fı--1, enabling backpropagation through all timesteps with single pass of inference.
At inference time, beam or sample decoding is often used to generate candidate target j' this time while conditioning previously-generated tokens.
regime, target tokens are generated
Úr ~ Po Po
or 9r = argmax pa 0 01
y
A potential failure mode for feacher-forcing-raee trained models is in conditioning previously unob- served target prefixes Vi=--1. As the model has not conditioned on these prefixes at training time, may generate bland, repetitive, or nonsensical candidate targets (Holtzman et al. 2019).
2.2 SCHEDULED SAMPLING
Scheduled Sampling [Bengio et al. 2015). hereafter Sequential Scheduled Sampling is training technique designed to bridge the gap between teacher-forcing and sample decoding. its simplest form, Sequential Scheduled Sampling generates tokens ý1:t and conditions on these target prefixes during training. Sequential Scheduled Sampling uses the same objective function as teacher-forcing (Equation I except the conditioning tokens V1 are random mixture of gold tokens fı.r and sampled tokens U1:t instead of gold tokens U1:1- See Algorithm for implementation.
Algorithm Sequential Scheduled Sampling (single example)
for all timesteps = 1....... T. do
Sample ÿt ~ pe (āe -1.-111 æ) Choose next conditioning token,
ye
with probability
with probability
end for return Accumulate loss Σ, log Pe (vve (1.
As + 0, we condition on V1--1 as in forciig, and as p + 1, we condition on Í as in sample decoding. Typically schedule will be used to gradually increase over the course of training. As illustrated in Bengio et al. (2015). Scheduled Sampling leads to performance improvement in variety of language generation tasks.

---

Under review as a conference paper at ICLR 2020
In spite of its benefits, Sequential Scheduled Sampling is inherently sequential algorithm: choosing conditioning token Yt requires conditioning autoregressively on tokens V1:1-1. While this is natural for sequential architectures such RNNs and LSTMs, it is poorly suited to self-attending ferwrforward models such as Transformer where inference for multiple timesteps can be carried out simultaneously.
2.3 PARALLEL SCHEDULED SAMPLING
We propose natural extension to Sequential Scheduled Sampling called Parallel Scheduled Sampling. Whereas Sequential Scheduled Sampling selects conditioning tokens one after another, we propose generating conditioning tokens for all timesteps in parallel over the course of one or more passes. While this technique requires strictly more operations than Sequential Scheduled Sampling, it better suited to hardware accelerators such as GPUs and TPUs (Jouppi al. 2017). Moreover, we find in our experiments that only modest number of passes is necessary for improving model performance.
Parallel Scheduled Sampling generates conditioning tokens for all timesteps simultaneously. The procedure consists of multiple passes, each pass consisting of parallel sampling and mixing steps (Figure D. In the first pass, the algorithm conditions on gold tokens 1/:t. generating tokens f1: i.i.d. according to Po 1, Sampling tokens the first pass is equivalent to the forward-pan of teacher-foreing. The sampled tokens, U1:t, are mixed (in parallel) with gold tokens, V/:t, to produce conditioning tokens for the next pass, V1:t-
We now describe the multiple-pass procedure. Let 1s and UK, 1st denote sampled and mixed tokens respectively on pass k. The mixed tokens from pass k. j/k.1:t. are used for conditioning on pass k + 1 in place of gold tokens y1.2. Finally, the loss is calculated as before, conditioning on the final mixture of gold and sampled tokens 1K, See Algorithm for implementation.
Algorithm 2 Parallel Scheduled Sampling (single example)
Set No. = V/-
for all passes k = ...... K do for all timesteps in parallel do Sample Ük,t pe ( 1,8-1, if then
Copy Uk. 5k-111
else
Ut
with probability 1 - p
Sample UN..
with probability p
end if
end for
end for return Accumulate loss
Σ, log Po 0K,14-1,
Finally, we prove that by running the sampling and mixing steps for multiple passes as described in Algorithm 2 the final sample from Parallel Scheduled Sampling converges to a random sample decode from the model when p and K > T.
Theorem 2.1. Consider sequence of tokens = (21, 22...., 2T) of length T. Letp and K 2 T be fixed. Then the likelihood of 21:7 under Parallel Scheduled Sampling's 's proposal distributior over conditioning tokens on pass K, dø (Z1.T), is identical to random sample decoding's, s, po(21.T) d (#1.T) (o
Proof. We begin by establishing notation. Let (o(2).)) be the likelihood of a sequence 21.7 according to random sample decoding. Let of (21.e) be the likelihood of the same according to Parallel Scheduled Sampling's proposal distribution on pass K.
The proof proceeds by induction. First we show that the proposal distribution for the first token matches random sampling's on the first pass, (21) po(21). Then we show that if g (#1a) = p0(21:)) holds for some K, it also holds for all K' > K. Finally, we show that if the previous
We drop conditioning on x: in the following for conciseness
4

---

Under review as a conference paper at ICLR 2020
statement holds, it also holds for tokens 21:4+1 pass K + 1. Thus, it follows that the proposal distribution matches random sampling's s for all tokens so long as K 2 T.
Base Case: Consider the proposal distribution for the first token the first pass, 21. As p the first token is sampled from po(2)) by construction. Thus,
(21) pa (=(21)
Induction over K: Suppose that the proposal distribution for tokens Og (21:) p0(21:0) some K 2 t. Then the equality also hold for the proposal distribution on pass + 1. This follows trivially as tokens 21 are "copied" from pass K K and thus their likelihood is unchanged,
K+1 (z1a) 98
90 (z1:) (=(
Induction over t: Suppose that the proposal distribution matches random sample decoding's for the first t tokens for = K; that is, K (21:1) = p0(21:). We show that the statement holds for pass K + 1 for tokens 21:4+1-- First, recall that by construction the proposal distribution for token 24+1 given previous tokens Zi.e is the same as random sampling's when / 2 K,
K +1 (2+41l21:)) po(24t1/21.) 90
Note that this only holds when = 1. Then,
K+ (21441) g8
Oø K+I (2441 (2441/210088 [1:)% K+1 + (21a)
K+ (2441/21a)g) (2:))0 K (21:)
qo
po (2141/21111) (p(
(21:411)
Where we use the chain rule, induction over K for 21:1, the inductive assumption for q (21:), and definition of dg +1 (2141/211)) when 2 K.
3 RELATED WORK
A variety of approaches have been introduced since Bengio et al. (2015) to eliminate the train-test discrepancy also known as exposure bias. Goyal et al. (2017) 7) integrate a diffcrentiable approximation to argmax into the Scheduled Sampling procedure. Ranzato et al. (2013) propose MIXER, which finetunes a model trained with teacher-forcing with REINFORCE to maximize BLEU on a validation set. Zhang et al. (2019) propose procedure similar to Scheduled Sampling but with alternative method for selecting the model's next-token prediction. Both works inherently require full-sequence decoding within the training loop, making them impractical to apply in domains with long generations.
Concurrently, Mihaylova and Martins (019) proposed variant of Scheduled Sampling similar to the approach introduced in this work. In their work, only single pass K 1 applied with the first pass conditioning on a mixture of input embeddings. Unlike this work, the proposed approach is not equivalent Scheduled Sampling under any choice of hyperparameters. [Lmbb al. (2016) introduce an alternative approach with similar motivation. A discriminator network is trained jointly with the generator distinguish between generator's hidden states produced by conditioning on ground-trutt and model prediction sample. The generator, apart from maximizing the likelihood of the data, is also trained to fool the discriminator (Goodfellow et al.) 2014). With this new objective, the dynamies of the generator would be the same for conditioning both ground-truth and model prediction. Our parallel sampling approach is orthogonal to professor forcing and can be potentially applicd their framework.
Another approach with similar motivation is proposed in Collins and Roark (2004). Rather than a maximum likelihood objective, the authors apply variant of the Perceptron algorithm (Rosenblati 1958) which penalizes the model's most likely decode if it doesn't 't exactly match the target. In the time-series modeling domain, Venkatraman al. (0155 propose DAD, wherein the training
5

---

Under review as a conference paper at ICLR 2020
set for Markov dynamics model is augmented by pairing model predictions with ground- -ruth states. This method is similar in spirit to the approaches above except in its Markov assumptions. Further, methods with similar motivation have also been studied in sequential decision making and reinforcement learning setting (Daumé et al. 2009: Ross et al. ).
While focus is on accelerating mitigation strategies for exposure bias, another line of work aims to accelerate sequential decoding. [Stern et al. (2019) generate successively longer decodes by inserting tokens between pairs of already-decoded tokens. |Roy et al. (2018) and Kaiser al. (2018) generate tokens in parallel by additionally conditioning on a short sequence of discrete latent variables. [eee al. (2018) apply an iterative refinement technique by performing multiple passes of parallel decoding.
EXPERIMENTS
We evaluate our proposed technique on image and text domains. In the text domain, we evaluate Parallel Scheduled Sampling on text summarization (Liu et al. 2018), task-oriented dialog response generation (Budzianowski et al. 2018), and machine translation (Sutskever et al.) 2014 Vaswani al., 2017) and compare it to teacher-foreed training. We further evaluate the proposed technique image generation on CIFAR-10. 10. Since our procedure is intended to generate better sequences, we evaluate it at the sequence level and not the word token pixel level. We compare
method to Sequential Scheduled Sampling only on the dialog task (Budzianowski et al.] 2018) as we find runtime infeasible on all other tasks. We use the Tensor2Tensor framework for all experiments (Vaswani et al. 2018).
4.1 DIALOG RESPONSE GENERATION
We evaluate our method on dialog response generation task using MultiWOZ (Budzianowski et al., 2018), a task-oriented dialog dataset. Here, we consider the problem of mapping conversation history consisting of alternating user and assistant turns to single turn of assistant response. We use Transformer model containing approximately one million parameters for this study as the dataset is much smaller (approximately 100k training examples) than those in other experiments. We truncate the length of the input and output to 512, and train all the models for 50k steps. As both model and dataset are small, we are able to empirically compare our method to Sequential Scheduled Sampling (such experiments are infeasible in larger models). Table I summarizes results for all experiments on the MultiWOZ dataset.
Both Sequential Scheduled Sampling and Parallel Scheduled Sampling (with just one pass) achieve better results than teacher-forced trained models. However, as can be seen in Table [] Parallel Scheduled Sampling and teacher-forcing are both two orders of magnitude faster to train than Sequential Scheduled Sampling. single pass of Parallel Scheduled Sampling is approximately 25% slower than facchrr-orred training while producing the benefits of Sequential Scheduled Sampling. TablecTalso shows the impact of mixing probability, number of passes, warm-up steps, and the mixing probability schedule (Bengio et al.] 2013) model performance. Overall, we find single pass with 50% gold/sampled mixing probability sufficient for improving performance. In the best setting, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over Seacher-forcing
4.2 SUMMARIZATION
Liu et al. (2018) propose of Wikipedia article given its references and other related documents. The dataset has close to 1.9 million training examples, and 230,000 test examples. We use Transformer seq2seq model for this settings: a base model with 60 million parameters and large model with 210 million parameters. For the base model, we restrict the maximum length of input and output to be 500, while for the large model the maximum length is set to 1500.
summarization task, where the task is generate the text
task in two
Table 2ishow the results of training base and large Transformer models for the summarization task. The base and large models were trained for 250k steps and 500k steps respectively. We use teacher- forcing for the first 50% of training steps in Parallel Scheduled Sampling as warm-up steps. The mixing probability is 50% and we perform single pass of sampling and mixing (Algorithm 2)

---

Under review as a conference paper at ICLR 2020
Training Method
Mixing Prob
Num Passes
Warm-up Steps
Schedule
Mean BLEU
Max BLEU
Training Steps/Sec
Teacher-Forcing
14.11
14.62
47
Sequential SS
0.25
25k
exp
14.35
0.13
Sequential SS
0.50
25k
exp
13.84
0.13
Sequential S$
0.75
25k
exp
12.97
0.13
Sequential SS
1.00
25k
exp
3.95
0.13
Parallel SS
0.25
I
25k
exp
14.13
14.50
35
Parallel SS
0.50
I
25k
exp
14.55
14.74
35
Parallel SS
0.75
1
25k
exp
14.06
14.32
35
Parallel SS
1.00
1
25k
exp
5.63
6.24
35
Parallel SS
0,5
2
25k
exp
14.60
14.75
27
Parallel S$
0.5
25k
exp
14.32
14.73
23
Parallel SS
0.5
5
25k
exp
14.33
14.70
17
Parallel S$
0.5
25k
exp
14.56
15.07
Parallel SS
0.5
10
25k
exp
14.55
15.21
10
Parallel SS
0.5
1
10k
exp
14.24
14.88
35
Parallel SS
0.5
I
15k
exp
14.56
14.98
35
Parallel SS
0.5
20k
exp
14.52
15.01
35
Parallel SS
0,5
30k
exp
14.66
14.98
35
Parallel SS
0.5
35k
exp
14.56
15.11
35
Parallel SS
0.5
1
40k
exp
14.73
15.38
35
Parallel SS
0.5
1
25k
linear
14.49
14.76
35
Parallel SS
0.5
1
25k
sigmoid
14.30
14.66
35
Table 1: Results from models trained with teacher-forcing. Sequential Scheduled Sampling, and Parallel Scheduled Sampling on dialog response generation. We report mean BLEU and maximum BLEU over random restarts for each configuration except Sequential Scheduled Sampling, for which we report a single run. We provide results by varying different hyperparameters for both variants of Scheduled Sampling. We also provide training steps per second for the different training algorithms. In the best setting, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over reacher-forcing.
Model Size
Max Length
Training Method
Decoding Method
ROUGE-2
ROUGE-L
Base
500
Teacher-Foree Forcing
Beam Search
24.74
34.42
Base
500
Parallel SS
Beam Search
25.19
34.76
Base
500
Teacher-Forcing
Greedy
20.18
28.66
Base
500
Parallel SS
Greedy
22.09
31.16
Large
1500
Teacher-Forcing
Beam Search
30.98
39.83
Large
1500
Parallel SS
Beam Search
30.35
39.09
Large 1500
Teacher- Teacher-Forcing
Greedy
29.25
37.91
Large 1500
Parallel SS
Greedy
29.42
38.35
Table 2: Performance on the summarization task using base and large Transformer when trained with teacher-forcing and Parallel Scheduled Sampling. We consider both beam search and greedy decoding. We adopt the widely-used ROUGE score as the evaluation metric (higher the better).
7

---

Under review as a conference paper at ICLR 2020
Training Method
Mixing Prob
Num Passes
Warm-up Steps
Schedule
FID (4)
(t)
Ground Truth
0.0
11.31
Teacher-Forcing
52.39
5.64
Parallel SS
0.10
I
100k
cxp
64.88
5.36
Parallel SS
0.25
1
100K
exp
48.85
6.08
Parallel SS
0.50
I
100k
exp
41.47
6.42
Parallel SS
0.75
1
100k
exp
43.38
6,48
Parallel SS
1.00
1
100k
exp
98.38
3.97
Table 3: Empirical results on CIFAR-10. We use Frechet Inception Distance (FID) (Heusel al.
2017) and Inception Score (IS) (Salimans et al. 2016) metrics to evaluate the quality of the samples from macher-forcing and Parallel Scheduled Sampling. We provide upper-bound scores by computing the metric on the ground-truth data.
With the base model, Parallel Scheduled Sampling obtains better performance than forchng with both beam search and greedy decoding while it performs better only with greedy decoding when the large model used. Since we use models that are much bigger than the ones in the dialog task discussed before, it is runtime infeasible to apply Sequential Schedule Sampling here.
4.3 IMAGE GENERATION
In the image domain, we evaluate our method for image generation on the CIFAR-10 dataset. We compare class-conditional Image Transformer (Parmar et al. 2018) trained with Parallel Scheduled Sampling. After training, we decode total of 50,000 randomly-sampee images conditioned classes drawn from the training set. In additional to baseline model, compare all metries to ground truth samples from the CIFAR-10 training set. We evaluate the image samples using Frechet Inception Distance (FID) (Heusel al. 2017) and Inception Score (IS) (Salimans et al. 2016) metrics. These metrics have been widely used to evaluate the quality of image samples from GANs (Lucic et al. 2018 Miyato et al.) 2018 Karras al., 2018 Zhang al. 2018).
and
Table 3 compares teacher-forcing with Parallel Scheduled Sampling on image generation. We train the 50 million parameter Image Transformer model for 200K steps both the cases. We find that Parallel Scheduled Sampling with single pass and mixing probability of 50% significantly decreases FID by 20% and increases IS by 13.8% compared to the baseline. Similarly our summarization experiment, it is runtime infeasible to apply Sequential Schedule Sampling here.
4.4 MACHINE TRANSLATION
We evaluate our method on the WMT 2014 English-German task which consists of approximately 4.5 million training sentences. We experiment with the large Transformer model that contains approximately 210 million parameters. We did not see performance improvements by using Parallel Scheduled Sampling. The model trained with feacher-forcing for 500k steps gets 28. 74 BLEU, The same model trained with 250k warm-up up steps using feacher-forcing and the next 250k steps trained with Parallel Scheduled Sampling with mixing probability set to 50% and a single pass of sampling and mixing (Algorithm ( obtains 28.57 BLEU. probability did not improve performance. We hypothesize the lack of performance improvement may be due to the fact that the summarization, dialog response generation and image generation tasks have much longer output sequences than in machine translation, though further investigation required.
tuning of warm-up steps and mixing
5 CONCLUSION
We introduce a simple technique to parallelize Scheduled Sampling that allows Schedule Sampling to be applied for training models with hundreds of millions of parameters large datasets. The technique potentially mitigates discrepancy between train and test time in auto-regressive sequence generation models. We find that in most cases technique leads to better empirical performance on
8

---

Under review as a conference paper at ICLR 2020
summarization, dialog generation, and image generation compared to teacher-foreed training. Our empirical results indicate that Parallel Scheduled Sampling can potentially improve the performance of auttorrgressivve sequence generation models particularly tasks containing long sequences.
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. NeurIPS, 2015.
Pawel Budzianowski, Tsung- Hsien Wen, Bo- Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. EMNLP. 2018.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP, 2014.
Michael Collins and Brian Roark. Incremental parsing with the perceptron algorithm. ACL, 2004.
Hal Daumé, Iii, John Langford, and Daniel Marcu. Search- structured prediction. Machine Learning, 2009.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. NeurIPS, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick, Differentiable scheduled sampling for credit assignment. arXiv preprint 1rXiv:170..06970 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degenera- tion. arXiv, 2019.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), 2017.
Lukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. arxiv, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018.
Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: new algorithm for training recurrent networks. NeurlPS, 2016.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv, 2018.
Peter J. Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. ICLR, 2018.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. NeurIPS, 2018.
Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers. arXiv preprint arXiv: 0xXiv:996.007655 2019.

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Martin Karafíát, Lukás Burget, Jan Honza Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. INTERSPEECH, 2010.
Makoto Miwa and Mohit Bansal. End-to-end relation extraction using Istms on sequences and tree structures, ACL, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Niki J. Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. Image transformer. ICML, 2018.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint 1rXiv:/511.0675: 2015.
Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological Review, 1958.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. NeurlPS, 2016.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for poly- phonic piano music transcription. JEEE/ACM Truns. Audio, Speech and Lang. Proc., 2016.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. CoRR, abs/1902.03249, 2019. URL http:/7
wtx.v.orr//bbs//ii 03249
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. NeurlPS, 2014.
Aäron van den Oord, Nal Kalchhrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. ICML, 2016a.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. Arxiv, 201
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, 2018.
Arun Venkatraman, Martial Hebert, and Andrew Bagnell. Improving multi-step prediction of learned time series models. In TwentyNintt AAAI Conference on Artificial Intelligence, 2015.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1989.
Han Zhang, Ian J. Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML, 2018.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. arXiv preprint arXiv: 1xXiv:9906.02448, 2019.
10


Reviews:
[{"review_id": "ryebilJ0tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a parallelized version of Scheduled Sampling, which is proposed initially to mitigate Exposure Bias. Exposure Bias occurs when sequence generation models are trained by teacher-forcing, where the t-th token, $y_t$, is forced to be correctly estimated given ground truths, $y_1, ..., y_{t-1}$. Scheduled Sampling randomly selects the previous sequence from generated one or ground truth. Since sequential Scheduled Sampling requires a long time to train, the authors propose a way to parallelize it without degrading the performance. Experimental results for dialog response generation, summarization, and image generation (as a grammar model on images) demonstrate that the proposed method is faster than the original Scheduled Sampling. Additionally, the performance gain is shown in comparison to baselines trained by teacher-forcing.\n\nMy first concern is that several papers are not cited, although they also address Exposure Bias:\n- Venkatraman et al., Improving multi-step prediction of learned time series models. AAAI, 2015.\n- Ranzato et al., Sequence Level Training with Recurrent Neural Networks. ICLR, 2016.\n- Zhang et al., Bridging the Gap between Training and Inference for Neural Machine Translation. ACL, 2019.\nThe second paper proposes MIXER, which is a method based on reinforcement learning. The training step can be performed without sequential sampling from the ground truth and predicted tokens. The third paper also randomly selects from a predicted sequence and the ground truth, but the selection is performed in a sentence-wise manner. \n\nRelated to the first concern, secondly, Scheduled Sampling itself is no longer the state-of-the-art method to solve Exposure Bias. It is unclear if the proposed method is competitive with the methods above. In addition, the empirical benefit in terms of training time seems to be small in comparison to them. The authors evaluated the proposed method using diverse tasks. Although it is good to see if Parallelized Scheduled Sampling is versatile, lack of comparisons to the existing methods except for the original Scheduled Sampling remains the superiority of Parallelized Scheduled Sampling unclear.\n\nFinally, the technical contribution is not so fascinating. Although parallelizing the original Scheduled Sampling is practically essential, the proposed solution seems to be a straightforward way. Theoretical arguments and proof also look obvious.\n\nNow I lean to reject this paper because of the concerns above. Since Exposure Bias is a fundamental problem for sequence generation problems as described in this paper, I would like the authors to revise their paper and submit it to another conference.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxcEjWH5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Scheduled Sampling aims to overcome the problems that come with the discrepancy between train and test\nwhen training sequence generation models using teacher-forcing. However one of the drawbacks scheduled sampling is that is it is hard to parallelize this is simply because some of the decode input tokens aren't known beforehand and need to be inferred i.e. loss of some tokens are dependent on some previous generations. \nAuthors introduce a method to perform parallel scheduled sampling by iteratively train using teacher forcing however after each iteration the gold references are mixed with the model predictions with a probability p, this allows parallelization as all decoder input tokens can be known beforehand. The final loss is calculated as before, conditioning on the final mixture.\n\nIn theorem 2.1 Authors proof that parallel scheduled sampling can converge to scheduled sampling when P is set to 1 and the number of iterations is larger than the sentence length. \n\nIn experiments over summarization, dialogue response generation and image generation authors show that the parallel scheduled sampling can achieve comparable performance to scheduled sampling with high-performance gains reaching 300 times faster and very comparable to teacher forcing. \n\npros:\nThe work presents a simple idea that is presented neatly with sufficient experiments. One of the outcomes of this method, that might not been stressed enough in the paper, is a neat way of doing scheduled sampling for the transformers architecture which wasn't straight forward before. \nCurrent proposals include some architecture modifications such as in Mihaylova et al. 2019 https://arxiv.org/pdf/1906.07651.pdf\n\ncons:\nThe only drawback I can find in this paper is it is lacking content. while the idea is interesting and supported by experiments, I find the content is slightly below the amount of content in average ICLR papers. \n\nThe explanation of the proposed scheduled sampling can be much simplified. The idea is quite simple however for example I find the pseudo code in Algorithm 2 is quite hard to grasp maybe due to some typos\n\nQuestions to authors: \n- We only see performance comparison in dialogue response generation experiments. There are other factors that can affect performance or make parallelization effective.  I wonder what are the performance gains of parallel scheduled sampling on normal scheduled sampling with respect to avg. number of tokens / sentence or batch size. \n\n- I had a hard time grasping Algorithm 2 although I understood the corresponding text could you please verify there aren't any typos. What it says is in the first iteration (k=1) scheduled sampling will be performed which doesn't make sense. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gYK5zaYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a technique for scheduled sampling that can be parallelized. At a high level, the method is to do teacher forcing, followed by passes of sampling from the model. During each round, the output of each pass are mixed with some probability. The time complexity then scales with the number of passes as opposed to the length of the sequence. The authors show results on dialogue generation, image generation, and machine translation. The proposed method generally obtains performance on par with scheduled sampling, but is much faster. The paper is well-written and the method clearly explained. I am not super excited about the technical contribution of this work, so my score is for weak acceptance.\n\nSome feedback\n- It would be nice to put the task in the caption for Table 1\n- Figure 1 is underwhelming and doesn't really help illustrate how the method works, though its caption does.\n- Because the main selling point of this is efficiency, I would like to see how it compares to work on non-autoregressive sequence generation (e.g. https://arxiv.org/pdf/1902.01370.pdf), which should probably be in the related works as well.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a parallelization approach for speeding up scheduled sampling, and show significant improvement over the original.  The approach is simple and a clear improvement over vanilla schedule sampling.  However, the reviewers point out that there are more recent methods to compare against or combine with, and that the paper is a bit thin on content and could have addressed this.  The proposed approach may well combine well with newer techniques, but I tend to agree that this should be tested.
Decision: Reject