Depth-Adaptive Transformer
Authors: Maha Elbayad
Jiatao Gu
Edouard Grave
Michael Auli
Venue:  (2020)
Keywords: Deep learning
natural language processing
sequence modeling

Abstract: State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

Published as a conference paper at ICLR 2020
DEPTH-ADAPTING TRANSFORMER
Maha Elbayad'
Jiatao Gu, Edouard Grave, Michael Auli Facebook Al Research
Univ. Grenoble Alpes
ABSTRACT
State of the art requence-to-sequenee models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy hard to process. In this paper, we train Transformer models which can make out- put predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT Bermann-Englis trans- lation our approach matches the accuracy of well tuned baseline Transformer while using less than a quarter of the decoder layers.
INTRODUCTION
The size of modern neural sequence models (Gehring al., 2017 Vaswani et al., 2017 Devlin et al.) 2019) can amount to billions of parameters (Radford et al. 2019). For example, the winning entry of the WMT 19 news machine translation task in English-Germal used an ensemble totaling two billion parameters (Ng al.) 2019). While large models are required to do better hard examples, small models are likely to perform as well on easy ones, e.g., the aforementioned ensemble prob- ably not required to translate short phrase such "Thank you". However, current models apply the same amount of computation regardless of whether the input is easy or hard.
In this paper, propose Transformers which adapt the number of layers to each input in order to achieve a good speed-accuracy trade off at inference time. We extend Graves (2016; ACT) who apply different
introduced dynamic computation recurrent neural networks in several ways: layers at each stage, we investigate range of designs and training targets for the halting module and we explicitly supervise through simple oracles achieve good performance on large-scale tasks.
Universal Transformers (UT) rely on ACT for dynamic computation and repeatedly apply the same layer (Dehghani et al. 2018). Our work considers variety of mechanisms to estimate the network depth and applies a different layer at each step. Moreover, Dehghani et al. (2018) fix the number of steps for large-scale machine translation whereas we vary the number of steps to demonstrate substantial improvements in speed at no loss in accuracy. UT uses layer which contains as many weights as an entire standard Transformer and this layer is applied several times which impacts speed. Our approach does not increase the size of individual layers. We also extend the resource efficient object classification work of Huang et al. (2017) and [Bolkbasi et al. (2017) to structured prediction where dynamic computation decisions impact future computation. Related work from computer vision includes Teerapittayanon et al. (2016); Figurnov et al. (011) and Wang et al. (2018) who explored the idea of dynamic routing either by exiting early by skipping layers.
We encode the input sequence using a standard Transformer encoder to generate the output sequence with varying amount of computation in the decoder network. Dynamic computation poses chal- lenge for self-attention because omitted layers in prior time-steps may be required in the future. We experiment with two approaches to address this and show that simple approach works well (2)). Next, we investigate different mechanisms to control the amount of computation in the de- coder network, either for the entire sequence or on a per-token basis. This includes multinomial and binomial classifiers supervised by the model likelihood or whether the argmax is already correct as well as simply thresholding the model score ($ Experiments on IWSLT14 German-Fnnliin
Work done during an intersship Facebook AT Research.

---

Published as a conference paper at ICLR 2020
State
Copied state
Classifier
Copy
Decoder depth
'6'
M
Decoding step
Decoding step
(a) Aligned training
(b) Mixed training
Figure 1: Training regimes for decoder networks able emit outputs at any layer. Aligned training optimizes all output classifiers 6, simultancously assuming all previous hidden states for the current layer are available. Mixed training samples M paths of random exits at which the model is assumed to have exited; missing previous hidden states are copied from below.
translation (Cettolo al. 2014) well WMT 14 English-Frenchs translation show that match the performance of well tuned baseline models at up to 76% less computation ($4).
ANYTIME STRUCTURED PREDICTION
We first present a model that can make predictions at different layers. This known as anytime prediction for computer vision models (Huang et al. 2017) and we extend it to structured prediction.
2.1 TRANSFORMER WITH MULTIPLE OUTPUT CLASSIFIERS
We base our approach the Transformer peruuence-t-sequeeee model /Vawwani et al. 2017). Both encoder and decoder networks contain N stacked blocks where each has several sub-blocks sur- rounded by residual skip-connections. The first sub-block multi-hean dot-product self-attention and the second position-wist fully connected additional sub-block after the self-attention to add source context via another multi-head attention.
network. For the decoder, there is an
Given pair of source-target sequences (x, y). is processed with the encoder to give representa- tions (81,..... ). Next, the decoder generates y step-by-step. For every new token y, input the decoder time t, the N decoder blocks process it to yield hidden states (hę) ((f)) ssnssi
h. ey.eddy.), h block,, (h" (h=T), *),
where block, is the mapping associated with the nh block and embed is lookup table.
The output distribution for predicting the next token computed by feeding the activations of last decoder layer ly" into softmax normalized output classifier W:
p (h,) softmax
(2)
Standard Transformers have single output classifier attached the top of the decoder network. However, for dynamic computation we need to be able to make predictions at different stages of the network. To achieve this, we attach output classifiers C, parameterized by W,, to the output hy of each of the N decoder blocks:
Vn, p( (b:) softmax(W..hh)
The classifiers can be parameterized independently or we can share the weights across the N blocks.
2.2 TRAINING MULTIPLE OUTPUT CLASSIFIERS
Dynamic computation enables the model any of the N exit classifiers instead of just the final one. Some of our models can choose a different output classifier at each time-step which results in exponential number of possible output classifier combinations in the sequence length.
2

---

Published as a conference paper at ICLR 2020
We consider two possible ways to train the decoder network (Figure I Aligned training optimizes all classifiers simultaneously and assumes all previous hidden states required by the self-attention available. However, at test time this is often not the case when choose different exit for every token which leads to misaligned states. Instead, mixed training samples several sequences of exits for given sentence and exposes the model to hidden states from different layers.
Generally, for a given output sequence y, we have a sequence of chosen exits (M1,...., 72 vn) and we denote the block which we exit time Ti.-
2.2.1 ALIGNED TRAINING
Aligned training assumes all hidden states h4-1 ..... . available in order to compute self- attention and it optimizes loss terms, one for each exit (Figure
LL logp(yolh"-1), LL" Σ LLS.
Caee y) En Wn
Wn LL"
(4)
The compound loss (2, y) is a weighted average N terms w.r.t. to (w1,...WN WN). We found that uniform weights achieve better BLEU compared to other weighing schemes (c.f. Appendix A). At inference time, not all time-steps will have hidden states for the current layer since the model exited early. In this case, simply copy the last computed state all upper layers, similar to mixed training ($2 2.2.2p. However, we do apply layer-specific key and value projections to the copied state.
2.2.2 MIXED TRAINING
Aligned training assumes that all hidden states the previous time-steps are available but this as- sumption is unrcalistic since early exit may have been chosen previously. This creates mismatch between training and testing. Mixed training reduces the mismatch by training the model to use hid- den states from different blocks of previous time-steps for self-attention. We sample M different
exit sequences (n1 (m)) ....(()) (m))
and evaluate the following loss:
vi ISmSM
lyl
M Σ m=
LL( , mp)
log p(u)l))))) Laec(æ y)
[L
(m)) n
(5)
M
When n4 < N, we copy the last evaluated hidden state hā to the subsequent layers so that the self-attention of future time steps can function usual (see Figure Ibp.
ADAPTIVE DEPTH ESTIMATION
We present variety of mechanisms to predict the decoder block which the model will stop and output the next token, or when should exit to achieve good speed-accuracy trade-off. We consider two approaches: sequence-specific depth decodes all output tokens using the same block ($3.)) while token-specific depth determines a separate exit for each individual token ($3.2).
We model the distribution of exiting at time-step I with a parametric distribution t where ge(n) is the probability of computing block,,... block,, and then emitting a prediction with C, The parameters of ge are optimized to match an oracle distribution of with cross-entropy:
Lesil (x. y) Σ H (ge (z, ()))(()))
The exit loss (Cexit) is back-propagated to the encoder-decoder parameters. We simultaneously optimize the decoding loss (Eq. (4) and the exit loss (Eq. (6)) balanced by a hyper-parameter a to ensure that the model maintains good generation accuracy. The final loss takes the form:
C(x, y) = (ate (x. y) + (2 y).
In the following we describe for each approach how the exit distribution g. is modeled (illustrated in Figure 2 and how the oracle distribution of is inferred.
3

---

Published as a conference paper at ICLR 2020
State
Copied state
O
Halting decision
Classifier
Copy
Docoder depth
g
Decoding step
Decoding stap
Decoding stsp
(a) Sequence-speciifg depth
(b) Token-specific
Multinomial
(c) Token-
Geometric-lin
Figure 2: Variants of the adaptive depth prediction classifiers. Sequence-speciffc depth uses a multi- nomial classifier choose an exit for the entire output sequence based on the encoder output (2a) It then outputs token at this depth with classifier E,, The token-specific multinomial classifier determines the exit after the first block and proceeds up to the predicted depth before outputting the next token (2b). The token geometric-like classifier (2c) makes a binary decision after every block dictate whether continue (C) to the next block or stop (S) and emit an output distribution.
3.1 SEQUENCE-SPECIFE DEPTH:
For sequence-specifer depth, the exit distribution and the oracle distribution qº are independent of the time-step we drop subscript t. We condition the exit on the source sequence by feeding the average s of the encoder outputs to a multinomial classifier:
((n]2r) softmax(Was bb) RRV.
St:
where W, and by are the weights and biases of the halting mechanism. We consider two oracles to determine which of the N blocks should be chosen. The first is based on the sequence likelihood and the second looks at aggregate of the correctly predicted tokens at each block.
Likelihood-
This oracle is based on the likelihood the entire sequence after each block and optimize it with the Dirac delta centered around the exit with the highest sequence likelihood.
gº (æ, y) 5(arg max LL").
We add regularization term to encourage lower exits that achieve good likelihood:
q" (x, y) 6(arg max LL" -An).
Correctness-baseet Likelihood ignores whether the model already assigns the highest score to the correct target. Instead, this oracle chooses the lowest block that assigns the largest score to the correct prediction. For each block, we count the number of correctly predicted tokens over the sequence and choose the block with the most number of correct tokens. A regularization term controls the trade-off between speed and accuracy.
C
# {t
ye arg max p(ylh? 9" (x, v) ō(arg max C An).
(10)
Oracles based on test metrics such as BLEU are feasible but expensive to compute since we would need to decode every training sentence N times. We leave this for future work.
3.2 TOKEN-SPECIHI DEPTH:
The sskeifcp approach can choose a different exit at every time-step. We consider two options for the exit distribution di at time-step multinomial with classifier conditioned the first decoder hidden state h and a geometric-like where an exit probability X is estimated after each block based the activations of the current block hg
4

---

Published as a conference paper at ICLR 2020
Multinomial q&:
(e(n yc)) = softmax(Wa) + bn), The most probable exit arg max gr (r(n x. yer) is selected at inference.
(1I)
Geometric-like 9::
Vn E -11, Xt sigmoid(wj, he +bb),
(12)
XF xt'),
n<
(o(nl,yy<l)
(13)
(1 Xt otherwise R'CN
II
where, d is the dimension of the decoder states, W, RRxx and wa E Rd the weights of the halting mechanisms, and b, their biases. During inference the decoder exits when the halting signal X exceeds threshold Tn which tune the valid achieve better accuracy-speed trade-off. If thresholds (Ta)Ign<N have not been exceeded, then we default to exiting at block N.
The classifiers are trained minimize the cross-entropy with respect to either one the following oracle distributions:
Likelihood-basee At each time-step L, we choose the block whose exit classifier has the highest likelihood plus a regularization term weighted by À to encourage lower exits.
q. xx, y) = 8 arg max LL," -An)
(14)
This oracle ignores the impact the current decision on the future time-steps and we therefore consider smoothing the likelihoods with RBF kernel.
k(t,,t)) =
LL
k(t, t(t,,))LL,
(x, y) = 8(arg max An), (15)
where we control the size of the surrounding context with a the kernel width. We refer to this oracle as LL(o, X) including the case where we only look at the likelihood of the current token with 0.
Correctness-based:: Similar to the likelihood-basee oracle we can look at the correctness of the prediction time-step well surrounding positions. We define the target of as follows:
C = L = arg max m(y plylppyyni111 11-,)>,
CT
x(t,t') C..
(16)
qe (x, y) = 6(arg max CT An).
(17)
i.c., exit threshold Tn- This does not require training and the thresholds T = (TI...., 1N- are simply tuned on the valid set to maximize BLEU. Concretely, for 10k iterations, we sample sequence of thresholds ~ (0, 1)x-1, decode the valid set with the sampled thresholds and then evaluate the BLEU score and computational cost achieved with this choice of T. After 10k evaluations we pick the best performing thresholds, that is T with the highest BLEU in each cost segment.
Confidence thresholding Finally, we consider thresholding the model predictions ($2)
when the maximum score of the current output classifier p( (h(') exceeds
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate several benchmarks and measure tokenized BLEU (Papineni et al. 2002):
IWSLT'14 German to English (De-En). We use the setup of Edunov et al. (2018) and train on 160K sentence pairs. We use N 6 blocks, beed-forward network (ffn) of intermediate-dmmette
5

---

Published as a conference paper at ICLR 2020
Uniform
n
n
T
5
Average
Baseline
34.2
35.3
35.6
35.7
35.6
35.9
35.4
Aligned (wh, =
35.5
34.1
35.5
35.8
36.1
36.1
36.2
35,6
Mixed M 1
34.1
32.9
34.3
34.5
34.5
34.6
34.5
34.2
Mixed M =
35.1
33.9
35.2
35.4
35.5
35.5
35.5
35.2
Mixed M 6
35.3
34.2
35.4
35.8
35.9
35.8
35.9
35.5
Table 1: Aligned vs. mixed training on IWSLT De-En. We report valid BLEU for uniformly sampled exit n ~ (l([1..6) at each token, a fixed exit n E [1..6] for all tokens, as well as the average BLEU over the fixed exits. As baseline show six standard Transformer models with 1-6 blocks.
1024, heads, dropout 0.3, embedding dimension denc = 512 for the encoder and dac = 256 for the decoder. Embeddings untied with 6 different output classifiers. We evaluate with single checkpoint and beam of width 5.
WM English to French (En-Fr). We also experiment the much larger WMT English- French task comprising 35.5m training sentence pairs. We develop on 26k held out pairs and test newstest14. The vocabulary consists of 44k joint BPE types (Sennrich et al., 2016). We use a Transformer big architecture and tie the embeddings of the encoder, the decoder and the output classifiers (Ww)) 12.1). We average the last ten checkpoints and use beam width 4.
Models are implemented in fairseq (Ott et al., 2019) and are trained with Adam (Kingma & Ba 2015). We train for 50k updates on 128 GPUs with a batch size of 460k tokens for WMT En-Fr and on 2 GPUs with 8k tokens per batch for IWSLT" 14 De-En. To stabilize training, we re-normalize the gradients if the norm exceeds Ielip 3.
For models with adaptive exits, we first train without exit prediction (a = 0 in Eq. using the aligned mode (c.f. 12.2.I) for 50k updates and then continue training with a of until convergence. The exit prediction classifiers are parameterized by a single linear layer (Eq. (8) with the same input dimension as the embedding dimension, e.g., 1024 for a big Transformer; the output dimension N
for multinomial classifier or one for geometric-like. We exit when Xt.n 0.5 for geometric-like classifiers.
4.2 TRAINING MULTIPLE OUTPUT CLASSIFIERS
We first compare the two training regimes for our model ($ 22.2 Aligned training performs self- attention aligned states ($ 2.2.1 and mixed training exposes self-attention to hidden states from different blocks ($2.22).
We compare the two training modes when choosing either uniformly sampled exit fixed exit n 1,..., 6 at inference time for every time-step. The sampled exit experiment tests the robustness mixed hidden states and the fixed exit setup simulates an ideal setting where all previous states are available. As baselines we show six separate standard Transformers with N E [1..6] decoder blocks. All models trained with an equal number of updates and mixed training with M paths is most comparable to aligned training since the number of losses sample is identical.
Table Tsoos that aligned training outperforms mixed training both for fixed exits as well for ran- domly sampled exits. The latter is surprising since aligned training never exposes the self attentioo mechanism to hidden states from other blocks. We suspect that this due the residual connections which copy features from lower blocks subsequent layers and which are ubiquitous in Transformer models 2D. Aligned training also performs very competitively to the individual baseline models.
Aligned training is conceptually simple and fast. We can process training example with N exits single forward/backward pass while M passes are needed for mixed training. In remaining paper, we use the aligned mode to train our models. Appendix reports experiments with weighing the various output classifiers differently but we found that a uniform weighting scheme worked well. On our largest setup. WMT" 14 English-French, the training time of an aligned model with six output classifiers increases only marginally by about 1 1 compared to baseline with a single output classifier keeping everything else equal.

---

Published as a conference paper at ICLR 2020
(a) Token-specfic
(b) Sequence-specilee depth
(c) Confidence thresholding
35.5
35.0
35.0
35.0
BLEC
34.5
BLEU
34.5
BLEU
34.5
34.0
Aligned
34.0
34.0
Baselite
Tok: C Mutinomial
Baselliee Aligned Soppa.
Aligued
A
Multinomal
Toll Geometnic like
33.5
33.5
Tol: LL Geometiic Mhe
Seq C
Coufidence threhholiing
Average exit (AF)
Avergge exit (AF)
Average exit (AF)
Figure 3: Trade-off between speed (average exit or AF) and accuracy (BLEU) for depth-adaptive methods on the IWSLT14 De-En test set.
Average exit (AE)
3
Average (AEE
3
4.5
0.01
0.05
0
2
Regularization parameter
RBF kernel width
(b) Effect a on AE
(a) Effect of on AE
Figure 4: Effect of the hyper-parameters a and À the average exit (AE) measured the valid set
of 14 De-En.
4.3 ADAPTIVE DEPTH ESTIMATION
Next, train models with aligned states and compare adaptive depth classifiers in terms of BLEU as well as computational effort. We measure the latter as the average exit per output token (AE),
As baselines classifier. We also measure the performance of the aligned mode trained model for fixed exits n E [1..6]. For the adaptive depth token-specifie models (Tok), we train four combinations: likelihood- based oracle (LL) + geometric-like like, likelihood-basee oracle (LL) multinomial. correctness based oracle (C) + geometric-like and correctness- based oracle (C) + multinomial. Sequence-specinf models (Seq) are trained with the correctness oracle (C) and the likelihood oracle (L...) with different values for the regularization weight X. All parameters are tuned on the valid and we report results on the test set for range of average exits.
again six separate standard Transformers with N E [1..6] with a single output
Figure 3 shows that the aligned model (blue line) can match the accuracy standard 6-block Transformer (black line) at half the number of layers (n = 3) by always exiting the third block. The aligned model outperforms the baseline for n = ......
For token specific halting mechanisms (Figure 3a the geometric-like classifiers achieves better speed-accuracy trade-off than the multinomial classifiers (filled vs. empty triangles). For geometric- like classifiers, the correctness oracle outperforms the likelihood oracle (Tok-C geometric-like Tok-LL geometric-lin like) but the trend is less clear for multinomial classifiers. At the sequence-leved likelihood is the better oracle (Figure 3b).
The rightmost Tok-C geometric-like point (a = 0, = 0.1) achieves 34.73 BLEU at AE 1.42 which corresponds to similar accuracy the N baseline at 76% fewer decoding blocks.

---

Published as a conference paper at ICLR 2020
(a) BLEU AE (test)
(b) BLEU vs. FLOPs (test)
44.0
41.5
43.5
47.0
43.3
BLEU
42.5
Aligned
Aligued
Sapd.1
Supdl
Tukk-C
42.0
Tuk-CC Puisson
Tok-l I. Peesson
Canfidlena: terrshaldnng
Cunfiderez: ihresholdingg
41.5
41.5
5
Average exit (AR)
Average FLOPs
100
Figure 5: Speed and accuracy on the WMT' 14 English-French benchmark (cf.. Figure 3).
The best accuracy of the aligned model is 34.95 BLEU at exit 5 and the best comparable Tok-C geometric-like configuration achieves 34.99 BLEU at AE 1.97, 61% fewer decoding blocks. When fixing the budget to two decoder blocks, Tok-C geometric-like with AE = 1.97 achieves BLEU 35, 0.64 BLEU improvement over the baseline (N 2) and aligned which both achieve BLEU 34.35.
Confidence thresholding (Figure 3cb performs very well but cannot outperform Tok-C geometric- like.
Ablation of hyper-parameters In this section, we look the effect of the main hyper- parameters on IWSLT 14 De-En: À the regularization scale Eq. (9)) and the RBF kernel width used smooth the scores (c.f. Eq. (15)). We train Tok-LL Geometric-like models and evaluate them with their default thresholds (exit if X > 0.5). Figure Aagure|4aaa 4a shows that higher values of X lead to lower exits. Figure 4 shows the effect for two values X. both curves, we see that wider kernels favor higher exits.
4.4 SCALING THE ADAPTIVE-DEPII MODELS
Finally, we take the best performing models form the IWSLT benchmark and test them on the large WMT" 14 Boggish-Freene benchmark. Results on the test set (Figure (Sa) show that adaptive depth still shows improvements but that they are diminished in this very large-scale setup. Confidence thresholding works very well and sequence-specific depth approaches improve only marginally over the baseline. Tok-LL geometric-like can match the best baseline result of BLEU 43.4 (N 6) by using only AE 2.40 which corresponds to 40% of the decoder blocks; the best aligned result of BLEU 43.6 can be matched with AE 3.25. In this setup, Tok-LL geometric-like slightly outperforms the Tok-C counterpart.
Confidence thresholding matches the accuracy of the N=6 baseline with AE 2.5 or 59% fewer de- coding blocks. However, confidence thresholding requires computing the output classifier at each block to determine whether to halt or continue. This is a large overhead since output classifiers pre- dict 44k types for this benchmark T$..T). To better account of FLOPs per output token (details in Appendix BB. Figure 5b) shows that the Tok-LL gcometric-like approach provides better trade-off when the overhead the output classifiers is considered.
this, we measure the average number
4.5 QUALITATIVE RESULTS
The exit distribution for given sample can give insights into what Depth-Adaptive Transformer decoder considers to be a difficult task. In this section, for each hypothesis y. we will look at the sequence of selected exits ( mlp) and the probability scores (PI,... P(y)) with Pt p(ū. (/==)) i.e. the confidence of the model in the sampled token at the selected exit.
Figures and 7 show hypotheses from the WMT' 14 En-Fr and IWSLT' 14 De-En test sets, respec- tively. For each hypothesis we state the exits and the probability scores. In Figure 6a predicting
8

---

Published as a conference paper at ICLR 2020
0.8
change mentts
0.8
Chi (
Premier
ministre
présent
0.4
passagers
devraient
attendre
immédiat
0.4
Scone
lc
était
0.2
Mais
les
ne
pas
des
0.2
(a) Sre: Chie@rac the Prime Minister, was there.
Src: But passengers G0dd expect changes
Ref: Chi Premier ministre est là.
happen immediately.
Ref: Mais les passagers ne devraient pas s' attendre des changements immédiats.
Figure 6: Examples from the WMT" En-Fr test set (newstest14) with Tok-LL geometric- depth estimation. Token exits in blue and confidence scores in gray. The "00' are due to BPE subword tokenization. For each example the source (Sre) and the reference (Ref) are provided in the caption.
0.8
neighbb @
Sco0
you
can
per @ @
Form
this
trick
to
your
friends
and
ors
thank
you
0.4
(a) Src: diesen trick können sie ihren freunden und nachharn vortiüfüürrn danke.
Ref: there is trick you can do for your friends and neighh&Qüoors thanks.
Figure 7: Example from the IWSLT' 14 De-En test set with Tok-LL geometric- depth estimation.
See Figure for more details.
'présent' (meaning 'present') hard. A straightforward translation is 'était là' but the model chooses 'present' which is also appropriate. In Figure 6b the model uses more computation to predict the definite article 'les" since the source has omitted article for 'passengers'.
A clear trend in both benchmarks is that the model requires less computation near the end of de- coding to generate the end of sequence marker 
 the preceding full-stop when relevant. In Figure 8 we show the distribution of the exits the beginning and near the end of test set hypothe- ses. We consider the beginning of a sequence to be the first 10% of tokens and the end as the last 10% of tokens. The exit distributions are shown for three models on WMT' 14 En-Fr: Model, has an average exit of AF = 2.53, Modelz exits at A = 3.79 on average and Model, with AF = 4.68. Within the same models, deep exits late used the heginning of the sequence and early exits selected near the end. For heavily regularized models such as Model, with AE = 2.53, the disparity between beginning and end is less severe as the model exits early most of the time. Model, and Modely are less regularized (higher AE) and tend to use late exits at the beginning of the sequence and early exits near the end. On the other hand, the more regularized Model, with AE 2.53 exits
Model
Modelz
Modelg
Beginning
Frequeniyy
0.4
0.2
2
3
4
5
6
3
4
6
1
2
4
5
Exit
Figure 8: WMT" En-Fr test set: exit distributions in the beginning (relative-position: rpos<0.1) and near the end (rpos>0.9) of the hypotheses of three models.

---

Published as a conference paper at ICLR 2020
Model
Model2
Modelg
0.9-1.0
0.8-0.9
0.7-0.8
Confidennee
0.6-0.7
0.5-0.6
0.4-0.5
0.3-0.4
0.2-0.3
0.1-0.2
0.0-0.1
2
2 3
4 5
Exit
Figure 9: Joint histogram of the exits and the confidence scores for Tok-LL geometric-like models on newstest14.
early most of the time. There also correlation between the model probability and the amount of computation, particularly in models with low AE. Figure 9 shows the joint histogram of the scores and the selected exit. For both Model, and Modely, low exits (n < 2) are used the high confidence range [0.8 1] and high exits (n 2 4) are used in the low-confidence range [0 0.5]. Modely has a high average exit (AE = 4.68) so most tokens exit late, however, low confidence ranges the model does not exit earlier than 5.
CONCLUSION
We extended anytime prediction to the structured prediction setting and introduced simple but effec- tive methods to equip sequence models to make predictions at different points in the network. We compared number of different mechanisms predict the required network depth and find that simple correctness based geometric-like classifier obtains the best trade-off between speed and ac- curacy. Results show that the number of decoder layers can be reduced by more than three quarters at no loss in accuracy compared to well tuned Transformer baseline.
ACKNOWLEDGMENTS
We thank Laurens van der Maaten for fruitful comments and suggestions.
REFERENCES
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proc. of ICML, 2017.
M. Cettolo, Niehues. Stüker, Bentivogli, and M. Federico. Report on the 11th iwslt evaluation campaign. In /WSLT, 2014.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In Proc. of ICLR, 2018.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACI, 2019.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Classical
structured prediction losses for sequence to sequence learning. In Proc. of NAACL, 2018.
Michael Figurnov, Artem Sobolev, and Dmitry P. Vetrov. Probabilistic adaptive computation time. In ArXiv preprint, 2017.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning, In Proc. of ICML, 2017.
Alex Graves. Adaptive computation time for recurrent neural networks. In ArXiv preprint, 2016.
10

---

Published as a conference paper at ICLR 2020
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In Proc. of ICLR, 2017.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Nathan Ng, Kyra Yee, Alexci Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. In Proc. of WMT, 2019.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL, 2019.
K. Papineni, S. Roukos, T. Ward, and W.-J Zhu. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL,, 2002.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In Technical report, OpenAl., 2019.
R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.
Surat Teerapitayannon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In ICPR, 2016.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NeurIPS, 2017.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dy- namic routing convolutional networks. In Proc. of ECCV, 2018.

---

Published as a conference paper at ICLR 2020
APPENDIX A LOSS SCALING
In this section experiment with different weights for scaling the output classifier losses. Instead of uniform weighting, we bias towards specific output classifiers by assigning higher weights to their losses. Table shows that weighing the classifiers equally provides good results.
Uniform
2
5
6
Average
Baseline
34.2
35.3
35.6
35.7
35.6
35,9
35,4
Win 35.5
34.1
35.5
35.8
36.1
36.1
36.2
35.6
Wn 35.3
32.2
35.0
35.8
36.0
36.2
36.3
35.2
Wip = vn
35.4
33.3
35.2
35.8
35.9
36.1
36.1
35.4
Wn 1/ VV
35.6
34.5
35.4
35.7
35.8
35.8
35.9
35.5
We = 1/n
35.3
34.7
35.3
35.5
35.7
35.8
35.8
35.5
(a) IWSLT De-En Valid
Uniform
=
π 2
3
Y 5
6
Average
Baseline
33.7
34.6
34.6
34.6
34.6
34.8
34.5
Wip 34.4
33.2
34.4
34.8
34.9
35.0
34.9
34.5
Wn = n
34.2
31.4
33.8
34.7
34.8
34.8
34.9
34.1
Wip vn
34.4
32.5
34.1
34.8
34.9
35.0
35.1
34.4
wn =I/vn 34.6
33.7
34.3
34.6
34.8
34.8
34.9
34.5
Wn 1/n
34.2
33.8
34.3
34.5
34.6
34.7
34.7
34.4
(b) IWSLT De-En Test
Table 2: Aligned training with different weights (wa) on IWSLT De-En. For each model we report BLEU on the dev set evaluated with uniformly sampled exit n ~ U{[1..6) for each token and fixed exit [1..6) throughout the sequence. The average corresponds the average BLEU the fixed exits.
Gradient scaling Adding intermediate supervision at different levels of the decoder results in richer gradients for lower blocks compared upper blocks. This because earlier layers affect more loss terms in the compound loss of Eq. (4). To balance the gradients of each block in the decoder, scale up the gradients of each loss term LL,,, when it is updating the parameters of its associated block (block, with parameters 02) and revert it back to its normal scale before back prokkgatpggg
the previous blocks. [ Algorithm Iillustrate this gradient scaling procedure. The 0, are updated with ammpliii gradients from the block' 's supervision and (N -n) gradients from the subsequent blocks. We choose 7n = >N n) to control the ratio 7/1 as the ratio of the block supervision to the subsequent blocks supervisions.
Table shows that gradient scaling can benefit the lowest layer at the expense of higher layers. However, no scaling generally works very well.
Nh LLn
Yo+1 LLaat1
7o42 V LLn+2
NN V LLN
blockn; On
block +1; On41
blockn. +2; 0o42
blockN; ON
V Llba1
v LLL
LLx42
LLn
Figure 10: Illustration of gradient scaling.
12

---

Published as a conference paper at ICLR 2020
Algorithm Pseudo-code for gradient scaling (illustrated for a single step ()
1: for 1... do
2: hr block, (N
3:
P(1++1 [b
soft mal,max
4: P(u++1 ((g) SCALE BRAEEGGAAADDNPUHHHHHHECC (b)); 7n)
5: if n < N then h = GEALFEGGRADEEN((1, 7n+1
6: end for
7:
function SCALE_ CRAALE_GGRDDENTTTeeRSTT scale ?)
8: return y + (1 L (RADIENT(
9: STOP GRADIENT in Py Torch with r.detach().
10: end function
Uniform
n =
7
Y
3
T
4
"
Average
Baseline
34.2
35.3
35.6
35.7
35.6
35.9
35.4
0
35.5
34.1
35.5
35,8
36.1
36.1
36.2
35.6
y = 0.3
35.1
33.7
34.7
35.3
35.7
35.8
36.0
35.2
7 0.5
35.4
34.8
35.4
35.6
35.6
35.7
35.6
35.4
7 = 0.7
34.9
34.6
35.1
35.1
35.2
35.4
35.3
35.1
7 0.9
34.9
34.8
35.3
35.3
35.3
35.4
35.5
35.3
7 1.1
35.1
34.9
35.2
35.3
35.3
35.3
35.3
35.2
(a) IWSLT De-En Valid
Uniform
n=
n
n
3
n =
4
n =
n = 6
Average
Baseline
33.7
34.6
34.6
34.6
34.6
34.8
34.5
0
34.4
33.2
34.4
34.8
34.9
35.0
34.9
34.5
7 = 0.3
34.2
32.8
33.9
34.3
34.6
34.8
35.0
34.2
7 0.5
34.5
33.8
34.2
34.6
34.5
34.7
34.7
34.6
7 = 0.7
34.0
33.7
34.2
34.3
34.3
34.3
34.3
34.2
7 0.9
34.1
34.0
34.2
34.3
34.4
34.4
34.4
34.3
7 = 1.1
34.2
34.0
34.3
34.3
34.3
34.3
34.2
34.2
(b) IWSLT De-En Test
Table 3: Aligned training with different gradient scaling ratios 7 on IWSLT' De-En. For each model we report the BLEU4 score evaluated with uniformly sampled exit ~ (((1..))) for each token and fixed exit n E [1..6]. The average corresponds average BLEU4 of all fixed exits.
13

---

Published as a conference paper at ICLR 2020
APPENDIX B FLOPS APPROXIMATION
This section details the computation of the FLOPS we report. The per token FLOPS are for the decoder network only since we use an encoder of the same size for all models. We breakdown the FLOPS of every operation in Algorithm (blue front of the algorithmic statement). We omit non-linearities, normalizations and residual connections. The main operations we account for are dot-pre dot-products and by extension matrix- vector products since those represent the vast majority of FLOPS (we assume batch size one to simplify the calculation).
Parameters
da decoder embedding dimension. encoder embedding dimension.
Operation
FLOPS
de
dy
The feed-forward network dimension.
Dot-product (d) Linear din + dout
2d -
|
source length.
2d dout
Current time-estep (t 2 1).
V
output vocabulary size.
Table FLOPS of basic operations, key parameters and variables for the FLOPS estimation.
With this breakdown, the total computational cost at time-step t of decoder block that we actually go through, denoted with FC, is:
FC( 12d 4djda 4tda + 4/x//a + [irsstCalll]a (d/ade.
where the cost of mapping the source" keys and values is incurred the first time the block is called (flagged with FirstCall). This occurs at for the baseline model but it is input-dependent with depth adaptive estimation and may never occur if all tokens exit early.
skipped, a block still has to compute the keys and value of the self-attention block so the self- attention of future time-steps can function. We will denote this cost with FS and we have FS = 4d3.
Depending on the halting mechanism, an exit prediction cost, denoted wit FP, is added:
Sequence-speciffe depth:
Ft g(t) 2 1J.Nda
Token-specific Multinomial:
FP q(t)) = 2NNd
Token- Token-speeific Geometric-like:
g((t.g(t) 2ddg((t
Confidence thresholding:
FP((,g(t)) = 2 da
For set of source sequences {a (4) (Hez and generated hypotheses (N the average flops per token is:
Baseline (N blocks):
Σ; Σ Dt=1
(8 FNF + +VVd)
Σ.
Adaptive depth:
Σ, 2t=1
( (1(1)FC(x() () + (N ((t))FS + FP(t, + 2Vdz)
In the case of confidence thresholding the final output prediction cost (2) is already accounted for in the exit prediction cost FP.
14

---

Published as a conference paper at ICLR 2020
Algorithm 2 Adaptive decoding with Tok-geometric-like
1: Input: source codes 8, incremental state
2: Initialization: = 1. 1/ <8>
3: for 1...NN N do
4:
FirstCall[n]] = True. > A flag signaling the source' keys and values should be evaluated.
5: end for
6: while * 
 do
7: Embed the last output token y/-
8: for E 1...NN do
9:
Self-attention.
10:
Map the input into a key (k) and value (v). FLOPS=4d
Map the input into query q. FLOPS-2d
12:
Score the memory keys with to get the attention weights a. FLOPS=4td
Map the attention output. FL.PPS-22
14:
Encoder-Decoder interaction. FirstCall[n] then
15:
16:
Map the source states into keys and values for the nth block. PLOPS=4xxlddd FirstCall[n] = False
17:
18:
end if
19:
Map the input into a query FLOPS-2d
20:
Score the memory keys with @ get the attention weights FLOPS=1|a/)d
21:
Map the attention output. FL.PS=2dd
Fred-forward network. FLOPS=4dady
23:
Estimate the halting probability Xe.n FLOPS=2d4
if Xt.n > 0.5 then
25:
Exit the loop (Line 8)
26:
end end for
27:
28: if N then
29:
Skipped blocks.
30:
for n. E n 11...... do
31:
Copy and map the copied state into key (k) and value (t). FLOPS=402
32:
end for
33:
end if
34:
Project the final state and sample a new output token. FLOPS=2V da
35:
+++
36:
end while
15


Reviews:
[{"review_id": "S1lBy4S0tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies using dynamic computation to alter the number of Transformer decoding layers each token uses to translate a given sentence. The paper considers using two variants of losses:  aligned training - same layer wise prediction loss for all tokens, and mixed training: loss on the output of a random layer for each token. For sampling the different layers the paper considers different distributions based on likelihood and the prediction rate.\n\nThe paper experiments these different training strategies on IWLST and WMT datasets. Training with token level sampling based on likelihood results in models that have smaller average exit (number of layers used in prediction) while preserving the BLEU scores of the standard Transformer training. Overall I believe the problem considered is interesting and the paper did a good job in setting up the problem and explaining the experimental setup results.  \n\nThe paper mainly needs to improve in discussing existing work. Universal transformers also study dynamic computation based on input tokens. While the training setup is different here, without the large shared Transformer layers, and this paper mainly focuses on the dynamic halting strategies, it is important to discuss these differences in detail in the paper.\n\nIntroducing a classifier in each layer (W_n) increases the number of parameters and compute by N. How do you handle this? \n\nThe exit loss (eqn 3) is a cross entropy loss, which will have trouble when q^* and q have different supports. Doesn’t a dirac delta q^* cause problems here?\n\n\nMinor:\n\nIntro: first line of second paragraph needs to be rewritten.\n\nThere is a conflict in writing in both abstract and introduction as some sentences say that current models use the same computation irrespective of hardness of the input, followed by discussion of Universal Transformers, which do adaptive computation based on input hardness. The writing flow needs to be fixed in both abstract and intro.\n\n** Post response update **\n\nAfter reading other reviews I think my initial rating 8 might have been a bit higher and am adjusting to 6. Fixing the discussion about related works , as other reviewers also mentioned, will improve the paper.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJgkmBE_uB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Post-reponse EDIT]: After reading the authors' response, and taking into account the other reviewers' opinions, I am willing to suggest acceptance of this paper on the condition that ***the authors will add the experimental results to address issue #7 and #8 of my original review (as they promised in their rebuttal)***. Before that, I'm keeping the current score, while noting that a more accurate rating should be \"5 - Marginally below the acceptance threshold\" (which is not available in ICLR this year). I want to thank the authors for addressing the other issues/questions I raised.\n\n----------------------------------------- Original Review Below -----------------------------------------\n\nSummary: The author proposes a new way to improve the supervision of training encoder-decoder Transformers so that it can make flexible, depth-adaptive predictions at inference time. The paper compares multiple dynamic computation schemes to investigate the effectiveness of these different approaches (e.g., at both sequence and token levels). Relatively extensive experiments were conducted in different settings (on a small and a large MT dataset).\n\nWhile the dynamic halting on Transformers has been previously explored in the Universal Transformers paper, this technique has not been applied to standard (encoder-)decoder Transformers. I think this is an interesting and exciting topic to pursue, as the authors seem to demonstrate (at least on the small IWSLT'14 DE-EN dataset) that we may not need as many layers at the decoding phase as we thought (and thus # of FLOPs).  \n\nHowever, I think the paper can be further improved in terms of both its organization/clarity and its experimental study (see details following).\n\n--------------------------------------------\n\nQuestions/comments:\n\n1. Please add a Related Work section. While using dynamic halting on standard Transformers may be novel, general dynamic routing on deep networks (which also aims to reduce the computation) is not. For example, SkipNet [1] uses an input-based dynamic skipping to bypass certain layers of a DNN. It's a good scholarship to give proper credit to prior related work(s).\n\n2. I think Poisson binomial (PB) may be a misnomer for the second qt modeling technique. PB is a probability distribution on the __sum__ of N (unequal) Bernoulli trials [2], which doesn't make sense here. What you are looking for, I think, is a learned \"coin flip\" at every layer of the network to decide whether to continue to the next layer (i.e., you are looking for the first \"head\" in the coin flips, instead of the sum of heads). And indeed, in Eq. (9), the definition for qt(n|...) is a \"geometric-alike\" distribution with success probability \\chi_{t,n} at the n^th flip. (Which brings to a side point: for the \"otherwise\" case in Eq. (9), I think you mean \\prod_n (1-\\chi_{t, n}); otherwise you can't guarantee \\sum_n qt(n|...) = 1 :-) )\n\n3. Why and how did you pick the exit threshold \\chi_{t, n} > 0.5 for your experiments? According to your definition of \\chi_{t,n} and q_t in Eq. (9), isn't the correct thing to do to sample from \\chi_{t, n}? My major concern is that, given different \\sigma (which you didn't specify, but I guess is something like a sigmoid), your \\chi_{t, n} may land in very different ranges. What if you have the following case: \\chi_{t, 1} = 0.501, \\chi_{t, 2} = 0.1, \\chi_{t,3} = 0.999, \\chi_{t,4} = 0.95? Doesn't this mean it may worth it to stop at the 3rd layer instead? (Using the coin flip analogy, having a coin with head probability 0.501 doesn't necessarily mean you \"must\" get a \"head\" at the first flip.) Moreover, if I use $\\sigma = sigmoid(x)/2 + 0.5$, won't I always get a \\chi > 0.5?\n\n4. In the token-specific likelihood-based method, you used an RBF kernel to model the influence of a time step t on its neighboring time steps. Two questions: 1) When you do \\sum_{t'}, did you also include t'<t? 2) Have you tried any other kernels, and how does this choice affect the performance?\n\n5. The aligned training is actually a type of intermediate auxiliary loss (or deep supervision, as the computer vision community probably more often calls it), which makes it not surprising that the unaligned Transformer could perform slightly better than the baseline. I find it interesting that the mixed training strategy doesn't work.\n\n6. Why did you use different training schemes for WMT'14 and IWSLT'14 (e.g., freezing the parameters, etc.)?\n\n7. One of the major concerns I have is that the empirical results don't seem that impressive. First, to demonstrate the effectiveness of the proposed adaptive depth estimation methodology, besides comparing to the baseline (black line) in Figures 3 & 4, you should also compare with the blue line (which adds these auxiliary losses to the baseline, but doesn't use the adaptive strategy that involves qt(n|...) at all) for fairness. From Figures 3 & 4, it seems that the adaptive-depth predictions are usually in the close neighborhood of the blue lines when at the same value of AE (instead of substantial, consistent improvement). In addition, while Tok-LL Poisson did well in Figure 4(b), it didn't seem to make a difference in Figure 4(a), which is where one is supposed to tune the hyperparameters on (such as \\lambda, \\sigma; see #8 below). This makes me a bit dubious about how much help the adaptive module brought, compared to just using the aligned model.\n\n8. For the WMT'14 experiments (Figure 4), why is there only one run of the \"Tok-LL Poisson\"? My understanding is that, if you are to use the proposed approach on a new dataset, you would want to try different settings of (\\sigma, \\lambda) on the validation set and pick the best one to use for testing. In Figure 4(a), for instance, for Seq-LL I would probably pick the (\\sigma, \\lambda) setting corresponding to the top-right yellow square--- which turns out to be slightly worse than the blue line on the test set (Figure 4(b)). It'd be useful to plot more \"red triangles\" in Figure 4 to evaluate how much the adaptive-depth methodology contributes to the performance.\n\n========================\n\nSome minor errors that don't have much impact on the score:\n\n9. The beginning sentence of the 4th paragraph of Section 1 is a bit strange (grammatically).\n\n10. Not all equations are numbered! See section 2.\n\n11. In the first equation of Section 2, do you mean h_{\\leq t}^{n-1} rather than h_{< t}^{n-1}? I think h_t from the previous layer is also used?\n\n12. In the 3rd equation of Eq. (12), you should have m somewhere within the summation.\n\n13. Equation (7) missing a right parenthesis.\n\n14. In Eq. (8) W_h is a matrix, whereas in Eq. (9) W_h is a vector (if I'm not mistaken)? Maybe use a lowercase w.\n\n15. Inconsistent notations. For instance, as I described in #3 above, you didn't say in the paper what \\sigma means in Eq. (9), but later re-used the same letter for a different meaning in Eq. (11) for the RBF kernel. Another example is the usage of \\theta_n in the \"Confidence thresholding\" paragraph for the threshold value; you used the same letter again in the \"Gradient scaling\" section of the appendix but with a different meaning (learnable parameters). \n\n16. In the last sentence of the Likelihood-based token specific depth, I'd suggest \\sigma \\rightarrow 0 instead of \\sigma=0, which would otherwise make \\frac{...}{\\sigma} in Eq. (11) undefined.\n\n17. I don't think you specified the \\theta_n used in your experiments (and how you tune them).\n\n18. What is the FS in Appendix B? Also, for \"ffn\" in Table 4, it's best to write its full name before referring to it with acronyms.\n\n19. Just curious: how did you implement the gradient scaling described in Appendix A (e.g., ensuring the \\gamma_n only applies to block n, but not blocks < n)?\n\n========================\n\nI think the current shape of the paper is marginally below the acceptance threshold. (Note that while the rating is \"3 - Weak reject\", I don't mean the score, but only the \"weak reject\" part. I'm still excited about the idea of applying dynamic computation in Transformer.) But I'm happy to consider adjusting the score if my concerns above can be satisfactorily addressed.\n\n\n[1] https://arxiv.org/abs/1711.09485\n[2] https://en.wikipedia.org/wiki/Poisson_binomial_distribution", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1ldAbEpYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study depth-adaptive Transformer models and show that they can perform well even under fairly basic strategies to stop the Transformer early. The paper is well written and the results convincing. There is not a lot of novelty but the results hold well, so it is a clear contribution. One main issue that prevents this reviewer from increasing the rating is the measure of speed the authors use: counting exit at every token. This is a fine measure for inference time, but during training the model proceeds on the whole sequence, right? Could you provide speed numbers for the training step? Is there any improvement over a baseline Transformer or is this technique solely for inference? (Which is still a contribution, but it should be made clear in the paper.)\n\nI thank the authors for the response and clarification. I stand by my score in the light of it.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an adaptive computation time method for reducing the average-case inference time of a transformer sequence-to-sequence model. 

The reviewers reached a rough consensus: This paper makes a proposes a novel method for an important problem, and offers reasonably compelling evidence for that method. However, the experiments aren't *quite* sufficient to isolate the cause of the observed improvements, and the discussion of related work could be clearer.

I acknowledge that this paper is borderline (and thank R3 for an extremely thorough discussion, both in public and privately), but I lean toward acceptance: The paper doesn't have any fatal flaws, and it brings some fresh ideas to an area where further work would be valuable.
Decision: Accept (Poster)