IOT: Instance-wise Layer Reordering for Transformer Structures
Authors: Jinhua Zhu
Lijun Wu
Yingce Xia
Shufang Xie
Tao Qin
Wengang Zhou
Houqiang Li
Tie-Yan Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Layer order
Transformers
Instance-wise Learning

Abstract: With sequentially stacked self-attention, (optional) encoder-decoder attention, and feed-forward layers, Transformer achieves big success in natural language processing (NLP), and many variants have been proposed. Currently, almost all these models assume that the \emph{layer order} is fixed and kept the same across data samples. We observe that different data samples actually favor different orders of the layers. Based on this observation, in this work, we break the assumption of the fixed layer order in Transformer and introduce instance-wise layer reordering into model structure. Our Instance-wise Ordered Transformer (IOT) can model variant functions by reordered layers, which enables each sample to select the better one to improve the model performance under the constraint of almost same number of parameters. To achieve this, we introduce a light predictor with negligible parameter and inference cost to decide the most capable and favorable layer order for any input sequence. Experiments on $3$ tasks (neural machine translation, abstractive summarization, and code generation) and $9$ datasets demonstrate consistent improvements of our method. We further show that our method can also be applied to other architectures beyond Transformer. Our code is released at Github\footnote{\url{https://github.com/instance-wise-ordered-transformer/IOT}}.

Published as a conference paper at ICLR 2021
IOT: INSTANCE-WISE LAYER REORDERING FOR TRANSFORMER STRUCTURES
Jinhua Zhu Lijun Wu?.:, Yingce Xia², Shufang Xie", Tao Qin', Wengang Zhou', Hougiang Li', Tie-Yan Liu?
UUniversity Science and Technology of China; Microsoft Research;
"teslazhu@mar usto. edu.euu.uu. en, fzhwg, wwtggehitttteeee edu. en, 2 [Li jun. Wu, Yingce. Xia, shufxi, taoufiittoqqiin ty liu htw.::ttyyyymmmmrrssoottooooo
ABSTRACT
With sequentially stacked self-attention. (optional) encoder-decoder attention, and feed-forward layers, Transformer achieves big success natural language pro- cessing (NIP), and many variants have been proposed. Currently, almost all these models assume that the layer order is fixed and kept the same across data samples. We observe that different data samples actually favor different orders of the layers. Based on this observation, in this work, we break the assumption of the fixed layer order in Transformer and introduce instance-wise layer reordering into model struc- ture. Our Instance- wise Ordered Transformer (IOT) can model variant functions by reordered layers, which enables each sample select the better one improve the model performance under the constraint of almost same number of parameters. To achieve this, we introduce light predictor with negligible parameter and inference cost to decide the most capable and favorable layer order for any input sequence. Experiments on tasks (ncural machine translation, abstractive summarization, and code generation) and datasets demonstrate consistent improvements of our method. We further show that our method can also be applied to other architectures beyond Transformer. Our code is released at Github
INTRODUCTION
Transformer Vaswani al.
2017) has been the dominant architecture in deep learning models
2019 Carion et al. 2020 Radford et al. [219 Dai et al. 2019 Lee Yang et al.) 2019 Cai & Lam| 2019). A Transformer model is stacked by several identical blocks, and each block consists of sequentially ordered layers: the self-attention (SA). encoder-decoder attention (ED) (decoder only) and feed-forward (FF) layer. Recently, various modifications have been proposed, where the focus is on replacing or inserting some components (e.g., attention layer/layer norm/position encoding) in standard Transformer (Wu et al. 2019
(Hassan et al. [g et al.
al. 2019 Devlin et al. 2018
et al.
2019 Shaw al. 2018
So et al. 2019: Ahmed et al., 2017).
Despite these Transformer alternatives have achieved improved performances, one critical element is almost neglected in cur- rent models, which how to arrange the components within Transformer network. i.c., the layer order also matters. As pointed by He et al. (2016b), different orders ReLU, batch normalization and residual connection significantly af- fect the performance of ResNet (He et al. 201 Therefore, we ask: What if we reorder the sequential layers in Trans- former (e.g., SA FF or FF-+SA of encoder, SA-FF-ED FF-ED-+AA of decoder)? What the best order for these different layers?
Order
De- En
Ratio
1:SA->ED-FFF
34.64
17.9%
34.60
17.9%
3:ED-+FF+4A
34.75
16.5%
4:ED-SAA+FF
34.67
17.5%
5:SA->FF
34.76
14.1%
6:FF->ED +SA
34.78
16.1%
Variance
0.0045
Table 1: Results for different de- coder orders on IWSLT14 Dn translation.
"Equal contribution and corresponding authors. htttps:////t/ htwp::/ow.oooo...aaoaeeeeeeeeeer

---

Published as a conference paper at ICLR 2021
Reference
and just like that the iceberg shows you a different side of its personality.
BLEU
TER.
Order 1 Trans
and just so. the iceberg shows different side of its personality
77.11
18.75
Order 2 Trans
and just like that, the iceberg shows you a different side of its personality.
100.00
0.00
Order 3 Trans
and just so. , the iceberg gives you another side of his personality.
0.00
37.50
Order 4 Trans
and just like this the ioeberg gives you unother side its personality.
38.71
25.00
Order Trans
ans so simply
iceberg shows another side and just like this the iceberg shows you another side of his personality.
personality.
30.33
50.00
Order 6 Trans
36.61
25.00
Table 2: Translations (Trans) from all ordered decoders of Transformer for one example sentence.
We first conduct preliminary experiments. We vary the three layers in decoder with all six vari- ants (each with unique order of the three layers) and train these models. Results on IWSLT14 German German-+Englisi +English translation are reported in Table T and no one is outstanding. The corpus BLEU variance is only 0.0045, which means that simply reordering the layers and training over the whole corpus impacts little. Press et al. (2019) also reported this for machine translation, but they stopped here.
As we can see, their performances are similar
This seems to be a negative answer. However, we take a further step and ask one more question: Does different data favor different ordered layers? That is, investigate whether each specific data has its own preference for one particular order. Intuitively, putting various data patterns in one order should not be the best choice. For example, harder samples may favor particular order while easier ones favor another one. Thus, for each order, we count the ratio of samples that achieve the best score with that order. In Table D we find they almost lie on uniform distribution (e.g.. 17.9% samples achieve the best BLEU with order SA-ED--FP Besides, we calculate the BLEU variance for each sample, and average all these variances, the result 114.76, which is much larger than above corpus variance (0.0045). These both mean the data indeed has its own preference to different orders. In Table 2 we present translations from all decoders for on example with BLEU and TER score to give an evidence.
Motivated by above observations, in this work, we present mnstancewwise Ordered Transformer (IOT), in which the layer order is determined by the specific data through instance-wis learning. To achieve this, we utilize light predictor to predict the confidence for each order, given the corresponding classification losses as training signals. However, directly training the predictor with conventional (i.e., NMT) loss tends to quickly converge to bad order, and ignore explorations we introduce an exploration loss and an exploitation loss to make an effective training while keeping an unambiguous prediction for each data so that the best order be decided during inference.
others. Thus,
We evaluate our approach on 3 sequence generation tasks, including neural machine translation (NMT), abstractive summarization (ABS) and code generation (CG). For NMT, we work 8 IWSLT and 2 WMT tasks, both on low-resource and rich-resource scenarios. Our method can consistently obtain 1.0 BELU score improvements over Transformer. For ABS, IOT also outperforms Transformer and other baselines on Gigaword dataset. For CG tasks, the results on 2 large- scales datasets (Java and Python) collected from Github surpass the state-of-the-an art performances. These all demonstrate the effectiveness of our 1OT. Furthermore, we provide detailed studies to verify that the instance-we wise learning and order selection make a reasonable and necessary modeling.
code
The contributions of this work can be summarized as follows:
We are the first to leverage instance-wise learning for layer order selection in a Transformer model (with shared parameters), and we demonstrate the instance-wise learning critical.
We demonstrate our learning approach can be universally applied to other structures beside Transformer (e.g., Dynamic Convolutions), as long as there are multiple different layers.
Experiments on 3 sequence generation tasks and datasets verify the effectiveness of 1OT with consistent performance improvements.
2 RELATED WORK
Architecture Exploration Inventing novel architectures by human designing or automatic searching plays an important role in deep learning. Specific to Transformer structures, various modifications have been proposed. For example, human knowledge powered designs include DynamicCony (Wul
et al. 2019). Macaron Network (Lu et 2019), Reformer (Kitaey et al.) (020) and others (Fonollosa Shaw et al. 2018). As for automatic searching, neural architecture
al.
2019 Ahmed et al. 2017
2

---

Published as a conference paper at ICLR 2021
X1
Salf Attention
Feed Forward
Salf Attension
Enc Dec Attention
Feed Forward
V/2
X2
Pred
Weight Tylng
Prod
Weight Tying
X
Feed Fonward
Soff Attennion
Sell Atlention
Attantion
Input
Encoder
Decoder
Output
The IOT framework. Pred means the light predictor introduced in 3.1 for order selection. We show two ordered encoders/decoders here. After taking X1, X2, X3. the selected order for Y2, Y3 is the lower encoder and upper decoder, while for Y1 is the upper encoder and lower decoder.
Figure 1:
search can discover networks with state-of tece--of-hheeaaee art performances but always with complicated com- putation, i.e., Evolved Transformer (So et al. 2019). The underlying principle is to add or replace some components of Transformer. For instance, Wu et al. (2019) replace self-attention with dynamic convolution, ,So al. (2019) add separate convolution layer in new branch. Different from them, we, instead, only focus on the selection of layer orders for each data sample so as to improve the model performance, without heavy modification. Besides, our approach is structure agnostic, which can be universally applied to other structures, only if multiple different layers exist.
Instance-wise Learning Deep learning models are trained over large-scale datasets, and data samples are often treated equally without modeling the difference between them. Some works attempt to
weight each data with different importance (Ren et al. 2018 | Hu al., 2019; Chang et al., 2017) or
feed data with curriculum learning according to its diffiulty (Bengio et al. 2009 Fan et al. [00 8). However, they often explicitly manipulate the data during training only, while no distinction exists in inference, and under one fixed model. Elbayad al. (2020) take step further and propose the depth-adaptive Transformer, which can forecast different depths of the network by predicting the required computation for particular data. Similarly, [Lu et al. (020)) propose sample-wist adaptive mechanism to dynamically calculate the number of required layers. They both aim at reducing the
computation cost and speed up the inference. [Schwartz et al. (2020), Bappa et al. (2020) and [Shazeer et al. (2017) all leverage conditional computation for each sample to control the computation and accuracy tradeoff during inference. Instead, we pay attention to the variant modeling functions and perform instance-wise order selection in order to boost the Transformer performance.
The most related work is Press al. (2019), which manually generates randomly ordered Transformer encoders and finds the Sandwich Transformer can slightly reduce the perplexity of language modeling. However, they find that Sandwich Transformer pattern has no effect on NMT task. Besides, it still performs over the whole corpus without considering each specific data. We, instead, investigate on various seueeneeet---eqeenee generation tasks and greatly improve the task performances through instance-wise learning, so as to discover the optimal ordered Transformer for each particular data.
3 INSTANCE-WISE ORDERED TRANSFORMER
The overall framework of 1OT is presented in Figure I In comparison with the standard Transformer, IOT only incorporates light-weighted predictors and reorders the encoder/decoder with weight tying, under the constraint of almost same number of parameters and exempt from heavy modifications. In this section, we introduce the details of IOT, including training, inference and discussions.
Notations Sequence-to-sequeee another sequence y = [y1, 3/2 .... yr where F,, J/y denotes the i-th and j-th token of and y, Tz and T, are the corresponding lengths. Given one sentence pair (ar, y) and learning model M, we can define the training objective as minimizing the cross-entropy loss LM = Σ log P Ve 2)) Besides, DEL (Pl Q)) denotes the Kullback-Leible (KI.) divergence between distributions IP and Q.
learning aims to map one sequence z [I1, I2..... TT. into
3.1 INSTANCE- WISE ENCODER/DECODER
IOT intends break the fixed order of layers in Transformer. As shown in introduction, simply reordering the layers w.r.t the whole corpus impacts little, while each data has its own preference to orders. Therefore, 1OT incorporates instance-wise learning to adjust the favorable order for each data.
3

---

Published as a conference paper at ICLR 2021
As shown in Figure [ both encoder and decoder in IOT consist of several blocks of SA, ED, FF layer with dynamic order, and we assume there are M (e.g., M = 2) ordered encoders and N (e.g.. N 6) ordered decoders (with shared weights). Inspired by the fact that lower training loss implies the higher proficiency confidence for candidate orders, we utilize the cross-entropy loss as signals to learn the confidence. That is, we calculate confidence "/m and X, for each encoder en decoder decn (resulted model MM".,, and use them to weight the training loss LM m. To calculate the confidence, we add simple and light predictor help distinguish the orders.
Training Given one source sequence x [xi, ,2, "T,, we first map each token into word embedding& [e1, 62, .... er,], where e; E Rd and then apply one light encoder predictor Tene to
predict the confidence of encoder orders using sentence embedding 8 = T. Σi C₁. Concretely, Tene takes Se as input and predicts T' for enom by Gumbel-softmax (Jang et al. 2016):
exp (log (Tana... gm) Te) Im M exp (llog (Tanca) + gx)) /Te)
Tene softmax (s.WW),
(1)
where Im is sampled from Gumbel distribution: Im log(- log Um), )2 Um Uniform(O,, 1), We E dddx is the weight matrix, Te is a constant temperature to control the distribution to be identical approximation with categorical distribution. Simultaneously, the token embeddings will feed to the encoders to get hidden states h = [M,, h2,..., lYTz), then we can calculate decoder order confidence An by one predictor Tiea in the same way Tanc:
exp (llog (Tdeo,,) + 9n) Ta)
Am
Wdac softmax (saWa),
Ek exp (llg (Teeoy) + gk) /Ta)
where 82 T. EE= h4 and Wa is the weight matrix. For each ordered path through encm and m..
deen. we can obtain the training loss "M"".,, and the final cross-entropy loss is weighted by confidence 7m and An with LM m.n, formulately as:
M Σ
Le
Σ
(Ym Ln) m.n.
Inference During inference, we directly replace the Gumbel-softmax used in training with argmax, order to choose the most capable encoder and decoder for each sequence a:
enc argmax (s. W dec argmax (s.
Discussion The decoding process is almost the same as standard Transformer, with only little overhead for order predictions. One may concern the training cost increased through our training. As we present in Section 5.1 the cost is actually affordable with fast convergence. Currently, we rcorder the layers of the encoder/decoder block and stack the same ordered block L times (see Figure D. A complex extension is to reorder all L blocks of encoder/decoder and we take it as future work.
3.2 AUXILIARY LOSSES
As we can see, the predictors are trained an unsupervised way, and we observe they lean to he lazy so that all samples quickly converge one same order during training, without senseful lcarning. Thus, to make an effective training and inference, we introduce exploration and exploitation losses.
(1) Exploration: first, we explore the diverse capability of all orders with help of loss Co to encourage all orders to participate in training. The spirit is the same as to encourage exploration in reinforcement learning. The expected softmax probability Ez [Tu] (encoder/decoder) from the predictor should approximate the uniform distribution Q = I ...... N. (e.g.. decoder orders), and we achieve this by minimizing Kl.-divergence between the statistical average Ezz [m.]] and Q:
N
CD = D E. (T)) =
Σ
log( (TT),,)) - log N,
(5)
N
where (Tz),, is the probability of n-th decoder order for data x. For encoder order, it is (Tz) processed in same way decoder.
4

---

Published as a conference paper at ICLR 2021
(2) Exploitation: different from LD to keep all orders effectively trained, during inference, the output distribution πı for each data should be able to make an unambiguous argmax selection. We then introduce another loss Ls to constrain each πy to be far away from the uniform distribution Q Concretely, we maximize the Kl.-divergence between each probability xx and Q:
N
Ls - -E, 1xx = EE,
log(T=)n log N
(6)
Note that we clamp the value of probability тa since the KL value is theoretically unbounded. With above auxiliary losses, the final training objective is to minimize:
L = Le + ciD + c2Ls.
where e1 and 02 are coefficients make a trade-off between Lp and Cs. In this way, we can achieve effective training, while keeping the ability to distinguish the favorable order for each data.
Discussion CD and Ls aim to keep effective training and unambiguous inference. There are several alternatives. The first is to simply decay the temperature T in Equation ①T) and (2) and remove the auxiliary losses. However, we do not notice obvious gain. Second is to linearly decay e only and remove Ls. which is able to fully train all orders at the beginning and loose this constraint gradually. find this also beneficial, but our two losses method performs better.
EXPERIMENTS
We conduct experiments on sequence generation tasks: neural machine translation (both low- resource and rich-resource), code generation and abstractive summarization. The main settings of each experiment are introduced here, and more details can be found Appendix A
4.1 DATASET
Neural Machine Translation For the low-resource scenario, conduct experiments IWSLT14 Englishe+German (En++De), English++Spanish (En++Es), IWSLT17 English++French (En++Fr), English+-Chinese (En++Zh) translations. The training data includes 160k, 183k. 236k, 235k sen- tence pairs for each language pair respectively. For the rich-resource scenario, we work on WMT14 En-+ee and WMT16 Comanian-+English (Ro+Enn) translations. For WMT14 En-De, we filter out 4.5.M sentence pairs for training and concatenate newstest2012 and newstest2013 as dev set, newstest2014 as test set. For WMT16 Ro-+En, we concatenate the 0.6.M bilingual pairs and 2.0M back translated data for training,
serve as dev/test set.
Code Generation Code generation aims to map natural language sentences to programming language code. We work on one Java (Hu et al. 2018) and one Python dataset (Wan et al. (008), following Wei et al. (2019) to process the two datasets. The Java dataset is collected from Java projects on Github, and the Python dataset is collected by Barone & Sennrich (2017). We split each dataset with ratio 0.8 0.1 0.1 as training, dev and test set.
Abstractive Summarization Abstractive summarization is summarize one long sentence into short one. The dataset we utilized is a widely acknowledged one: Gigaword summarization, which is constructed from subset of Gigaword corpus (Graff al. 2003) and first used by Rush et al. (201)). The training data consists of 3.8M article-headline pairs, while the dev and test set consist of 190%: and 2k pairs respectively.
4.2 MODEL AND OPTIMIZATION
For IWSLT translation tasks, use t transformes iwslt de setting model configuration. The number of block, embedding size and feed-forward network (FFN) size are 6, 512, 1024. WMT tasks trans wttp://www...wa....t.......m
configuration, with blocks, embedding size 1024 and FFN size 4096. Optimization and learning scheduler are the default settings in Vaswani et al. (017)) For code generation, block number/embeddiing size/FFN size are 3, 256, 1024

---

Published as a conference paper at ICLR 2021
Table 3: Preliminary results of varied orders on IWSLT14 De-+En task.
(a) Results of encoder, decoder orders, and their combinations.
(b) Results of varied number of decoder orders.
De-En
De-En
Transformer
34.64
Transformer
34.64
Encoder (M = 2,
35.18
1OT (M 1.N 6)
35.60
Decoder (M = 1, =
35.60
1OT (M 1, N.N== 5)
35.65
Encoderx Decoder (M N 6)
35.24
IOT (M = 1,N 4)
35.62
IOT (M I,N 3)
35.58
Eacoderx Decoder (M = 2, N 4)
35.30
IOT (M 1,
35.32
Encoderx Decoder (M = 2, N
35.25
Table 4: BLEU scores of IOT on eight IWSLT low-resource translation tasks.
En->De
De-+En
En-+Fr
Fr-=En
En-+Zh
Zh- EEn
En-+Bs
Es->En
Transformer
28.57
34.64
35.9
36.1
18.4
39.0
40.6
IOT
29.52
35.62
37.2
37.8
27.2
19.3
40.1
41.7
respectively. Others are the same as NMT. For summarization, we take transforrmt _wwtterrwww............... www.wwmteeee.eeme. with 6 blocks, embedding size 512 and FFN size 2048. Dropout (Srivastava et al. 2014) is set to be 0.3. Other settings are also the same as NMT task. Implementation is developed on Fairseq (Ott et al.
2019). We first grid search C1, C2 IWSLT14 De->En dev set, and then apply them other tasks. The best setting is C₁ = 0.1, C2 = 0.01, and the importance study of C₁, C2 is shown in Appendix [.1
4.3 EVALUATION
We use multi iibb-bbuu perl peerlto to evaluate IWSLT14 De and all WMT tasks for a fair comparison with previous works. For other NMT tasks, we follow Vaswani et al. (017) to use beam size 4 and length penalty 0.6 for WMT14 En->De beam size and penalty 1.0 for other tasks. For code generation, the evaluation is based two metrics, the sentence BLEU computes the n-gram precision of a candidate sequence to the reference, and the percentage of valid code (PoV) that can be parsed into abstract syntax (AST), As for summarization, the generated summarization is evaluated by ROUGE- ROUGE-1/2/L. F1 score Lin 2004)
sacre-ba -bere for evaluation. During inference,
4.4 MAIN RESULTS
Encoder/Decoder Orders Encoder block only contains SA and FF layers, the resulted max number of encoder layer orders M is 2, while for decoder, the max order variants N is Therefore, we first evaluate the utilization of encoder orders, decoders orders, and both orders on IWSLT14 De-+En translation, order see the impacts of different number of order candidates and their combinations. In Table 3 (a). we can see that 2 ordered encoders improve the result, and 6 ordered decoders achieve more gain. This meets our expectation, since the search space limited when there are only 2 ordered encoders. However, if we train both encoder and decoder orders (e.g., M = 2, N 6). the results (e.g., 35.30) can not surpass the decoders only (35.60). We suspect the search space is too large so that training becomes hard, and decoder orders play more important role than encoder orders for sequence generation. Therefore, we turn to investigate different decoder order candidates (refer to Appendix A.3 for detailed combinations) in Table 3 (b). Results show that N 4, 5, 6 achieve similar strong performances (results on other tasks/datasets are in Appendix A.4). Thus, considering the efficiency and improvements, utilize N 4 ordered decoders (order 1, 2, 4, in Table I to reduce training cost in later experiments.
NMT Results BLEU scores 8 IWSLT' low-resource tasks shown in Table As 1OT achieves more than 1.0 BLEU points improvement on all tasks (e.g.. 1.7 on Fr-xEn). The consistent gains on various language pairs well demonstrate the generalization and effectiveness of our method. We then present comparison with other works on IWSLT14 De-IEn task in Table 5 (a), and IOT is also better than several human designed networks. The results of WMT14 En->ee and WMT16 Ro are reported in Table 6 We also compare with existing works, such as the unsupervised Ro-+En based on pre-trained cross-lingual language model (Lample & Conneau 2019).
see,

---

Published as a conference paper at ICLR 2021
Table 5: Results on IWSLT14 De-tEn translation task (a), Java and Python code generation tasks (b).
(a) Results on IWSLT14 De translation.
(b) Results Java and Python code generations.
Method
BLEU
Java
Python
Transformer
34.64
Method
BLBU
PoV
BLEU
PoV
1OT
35.62
Transformer
24.58
74.44%
13.20
61.89%
Adversarial MLE Wang ct 9019a
35.18
10T
25.51
77.30%
14.05
63.14%
DynamicCony (Wh el al. 2019
35.20
Wei (011
17.17
27.4%
12.09
51.9%
Macaron Network (Lu at 2019
35.40
MADL Wang ct al. 2019c
35.56
Similarly, our method outperforms them and shows our framework can work well on rich-resource scenario.
Method
En >De
Transformer"
29.12
Code Generation Results The results are shown in Table 5()). We can observe that Transformer obtains better result than the LSTM-based work (Wei et al., 2019). Compared with Transformer, IOT can further improve the quality of generated code. Specifi- cally, IOT boosts Transformer with 0.93 BLEU/2.869. PoV gain on Java generation and 0.75 BLEU/T 25%% PoV gain on Python spectively. Again, these results well demonstrate the effectiveness of our method.
10T
Shaw al.
2018
29.20
OIL a.
Wu
2019
29.70
St
2019
29.80
Method
Ro EEn
Transformer"
37.73
1OT
38. 83
Senarich al. 2016
33.90
Abstractive Summarization Results The IOT performances on summarization task are shown in Table 7 From the results, we can see IOT achieves 0.8, 0.7 and 1.0 scores gain of ROUGE-I, ROUGE-2 and ROUGE-L metrics over standard Transformer on Gigaword summarization. IOT also surpasses other works such as reinforcement learning based method (Wang et al. 2018), which again verifies our approach is simple yet effective.
Lample Conneau 2019
38.50
Table 6: WMT14 En-+De and WMT16 Ro-En translation re- sults. *stands for our reproduced result.
Method
ROUGE-1
ROUGE-2
ROUGE-L
Transformer (Vaswani et al. 2017
35.59
17.74
32.98
1OT
36.37
18.46
33.89
RNNScarch+MRT (Ayana et al. 2016
36.54
16.59
33.44
Concept pointer+DS Wang 2019h
37.01
17.10
34.87
RNNSeanchtselecttM MatmmbhhsseleeeMT--D-AAA
(Lel al. 2018
35.
17.27
33. 13
CGU [Lin al. 2018
18.00
33.80
Reinforeca topic Conv--CovvS2 Conv
Wang et 2018
36.92
18.29
34.58
Table ROUGE-1/2/L F1 scores for Gigaword summarization.
5 STUDY AND ANALYSIS
5.1
INFERENCE/TRAINING COST
As discussed before, our approach only increases negligible parameters and inference time cost. Here we compare the detailed inference time and model size of our framework the standard Transformer. The detailed parameter numbers and inference time on IWSLT14 En test set are shown in Table & Since we only add one linear layer and softmax layer as the predictor, the number of extra parameters is M x hidden. size (encoder predictor) or N x hidden size (decoder predictor), which negligible compared to other model parameters. Therefore, IOT introduces more model diversity and improves the performance, but under the constraint of almost same number of parameters. As for the inference time, the only difference from the one-pass order prediction and the cost is extremely low compared with heavy autoregressive generation process, which can be seen from Table 8
Apart from the inference cost, one may concern about the training cost since IOT trains multiple orders in one model. To the influence, provide several statistics here. Specifically, on the four IWSLT En x translation tasks, we analyze the cost by counting the training time for each epoch, the epoch number when model convergences, and the corresponding total training time. The numbers
7

---

Published as a conference paper at ICLR 2021
En De
De >n
Method
Inference time (8)
#parameters
Inference time (8)
#parameters
Transformer
1487.19
36741120
1432.03
36741120
IOT (N
1496.70
367421444
1429.26
36742144
1OT (N 3)
1470.40
36742656
1480.83
30742656
1OT (N 4)
1505.83
36743168
1444.72 74
36743168
1OT (N
1491-10
367433680
1424.10
367 367436880
1OT (N
1479.20
36744192
1464-00
36744192
Table 8: Inference time and model parameters counted for Transformer and our framework on IWSLT14 En The study is performed single Tesla P100 GPU card.
Transformer
IOT (N 2)
IOT (N 3)
Epoch Time Epoch Number
277.1
475.6(1.72x)
685.4(2.47x)
En->De
67
42(0.63x)
39(0.58x)
Total Time (8)
18565.7
19075.2(1.08x)
267.0.6(1.44x))
Epoch Time Epoch Number
410.5
715.1(1.73x)
(2.4.1().49x)
En FFr
55
37(0.67
33(0.60
Total Time
22577.5
26418.7(L.17x))
33795.3(1-49x)
Epoch Time Epoch Number
349.9
612.3(1.75x)
931.(2.666x)
En -Zh
59
34(0.58x)
30(0.51x)
Total Time
20644.1
20818.2(L.01x)
27942.0((.35x))
Epoch Time 310.1
538.0(1.77
784.4(2.53x)
En-=Es
Epoch Number
69
43(0.62x)
30(0.43x)
Total Time
21369.9
23134.0(1.8x)
23532.0(1.10x))
Table Training cost analysis for Transformer and our IOT on four IWSLT translation tasks. The study is performed on single Tesla P100 GPU card.
presented in Table 9 and we can have several observations. Take IWSLT14 En-+De translation as an example, (1) jointly optimizing different orders indeed introduces more training cost for each epoch. Transformer baseline costs 277.1s per epoch training, while IOT costs 475.6s and 685, 4s with = 2 and = 3 orders respectively, the increased cost ratio is about 1.72 and 2.47 x (but less than 2.0 and 3.0). (2) However,
find that with the shared parameters between these orders, the model convergence also becomes faster. Transformer needs 67 epochs when converge, while IOT only needs 42(0.63 and 39(0.58x) epochs for N and N 3 orders, much fewer than Transformer. (3) The total training cost actually is not increased much. IOT (N = 2) and IOT (N 3) about 1.08x and 1.44x training time compared with Transformer baseline (the ratio for IOT (N 3) is only 1.10 on IWSLT17 En-=ss.. From these observations, we can see that the increased training cost is affordable due to the fast convergence.
5.2 CASE VERIFICATION
We perform study with N = 3 to verify that IOT has made necessary instance-wirg order selection. We first split IWSLT14 Ent+De dev set into 3 subsets according to the prediction Taen. and then decode each subset all 3 ordered decoders, and report the BLEU results. As shown in Figure 2. each subset indeed achieves the best score on the corresponding predicted order (outperforms other orders by 0.2-0.4 BLEU). We also do the same study on the test set, and the predicted order outperforms others by 0.7-0.8 BLEU. These well prove that IOT makes a reasonable prediction.
Besides, find that the predicted orders correlate different sentence difficulties. In our case, the set sentences belong to decoder 1 achieve highest BLEU than other sets, which means set is relatively simple translate, and vice versa for samples in set 2. These imply that different difficulty sentences have different structure preferences. We provide statistics and examples in Appendix B.2
5.3 APPLY ON ANOTHER STRUCTURE (DYNAMICCONY)
As we discussed, our instance-wise layer reordering is structure agnostic. In this subsection, we evaluate this by applying our approach DynamicConv network (Wu al. 2019) beyond standard
8

---

Published as a conference paper at ICLR 2021
32.60
32.44
32.42
32
39.10
38.85
38.87
38.4
31
37.6
set 27.75
28.13
27.88
30
34.45
34.78
34.37
36.8
36.0
29
set 3
35.82
36.01
36.11
35.2
20
decoder
decoder
decoder
decoder
decoder
decoder
34.4
(a) BLEU scores IWSLT14 En- (b) BLEU scores IWSLT14 De
Figure 2: BLEU scores of three subsets (divided by predicted decoder) on all decoders.
Transformer, which replaces the self-attention with dynamic convolution. We train layer ordered DynamicCony on N 2, 3, decoders and test the performances. The BLEU score of standard DynamicConv is 35.20, and with our instance- wise order learning, we achieve 35.60, 35.82, 35.87 for N = 2, 3, 4 ordered decoders respectively (near 0.7 point gain). Therefore, this study verifies our claim that our approach can be applied to other structures, long multiple different layers exist.
5.4 DISCUSSIONS
Ensemble Since our framework involves multiple orders (with shared parameters), which is also done in ensemble framework, make comparison with ensemble. The ensemble method trains multiple models with different parameters separately in independent way. While work trains orders in a joint way with an intention to make them more diverse. More importantly, from the view of time and memory cost, the ensemble frame- work increases N times which totally different from ours. In this sense, method can be combined with ensemble to fur- ther boost performance. The competitive results on IWSTL14 Ent+De test are shown in Table I0 We can clearly conclude that IOT and ensemble are complementary to each other.
Ensemble Models
En->De
De-FEn En
1-model (standard)
28.57
34.67
2-model (standard)
29.71
35.92
3-model (standard)
30.08
36.40
4-model (standard)
30.18
36.54
I-model (IOT)
29.52
35.62
2-model (IOT)
30.38
36.80
3-model (IOT)
30.93
37.25
4-model (IOT)
31.02
37.38
Table 10: Ensemble performances
of standard Transformer and IOT.
Regularization IOT consists different ordered blocks weight tying method, which may looks like parameter regularization to some extent. However, we show that IOT is more than regularization and can be complementary with other regularization methods. Setting (1): We first train a Transformer model on IWSLL114 De->En task with all shared decoder orders, but without instance-wizg wise learning, and test the performance with each order. We find the BLEU scores on test set are near 34.80 for each order, much worse than IOT, which means that simply regularizing the shared parameters for different orders is not the main contribution to performance improvement, and instance-uu wise learning is critical. Setting (2): Another experiment that train Transformer with LayerDrop (Fan et al., 2019). a dropout technique to regularize the layer parameters. The test BLEU is 35.40, which achieves about 0.8 score improvement over Transformer. After applying 1OT with LayerDrop, we obtain further gains than IOT only (35.62) to reach a BLEU score 36.13. 13. Therefore, this demonstrates IOT is not only regularization and can be smoothly integrated with other regularization methods. More details and experiments on other tasks are shown in Appendix B.3
CONCLUSION
this work, we propose Instance-wise Ordered Transformer, which leverages instance-wise learning to reorder the layers in Transformer for each data. Compared with standard Transformer, IOT only introduces slightly increased time cost. Experiments on 3 sequence generation tasks and datasets demonstrate the effectiveness of IOT. We also verify that approach can be universally applied to other structures, such as DynamicCony, In future, we plan to work on more complicated reordering in each block, well other tasks such multi-lingual translation and text classification.

---

Published as a conference paper at ICLR 2021
REFERENCES
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arXiv preprint vrXiv:1711.02133. 2017.
Shiqi Shen Ayana, Zhiyuan Liu, and Maosong Sun. Neural headline generation with minimum risk training. arXiv preprint 4r019:460..01904, 2016.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. arXiv preprint SoXiv:2002.07106 2020.
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv: 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009,
Deng Cai and Wai Lam. Graph transformer for graph-to-sequence learning. arXiv preprint 1rXiv:1911.07470, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint arXiv:2005.1287 2020.
Haw-Shiuan Chang, Erik kearned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, pp. 1002-1012, 2017.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:1810.04805 2018.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In
International Conference Learning Representations, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In International Conference on Learning Representations, 2019.
Yang Fan, Fei Tian, Tao Qin, Xiang- Li, and Tie-Yan Liu. Learning to teach. In International Conference on Learning Representations, 2018.
José AR Fonollosa, Noe Casas, and Marta R Costa-jussà. Joint source-target self attention with locality constraints. arXiv preprint arXiv:1905.063: 06596, 2019.
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34, 2003.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dow Downms, William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint 1803 05567, 2018.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 645. Springer, 2016b.
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. 2018.
10

---

Published as a conference paper at ICLR 2021
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, pp. 15738-157444 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint /r1.v:/611.01 144, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:
2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining, arXiv preprint arXin:1901.07291, 2019.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Scungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention- basod permutation-invant neural networks. In International Conference Machine Learning, pp. 3744 3753. PMLR, 2019.
Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. Proceedings of the 27th International Conference on Computational Linguistics, pp. 1430-1441, 2018
Chin- New Lin. Rouge: A packagefor automatic evaluation of summaries. In Proceedingss/WWrrsshpp on Text Summarization Branches Out, Post2Conference Workshop of ACL, 2004.
Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. Global encoding for abstractive summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 163-169, 2018.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Haotang Deng, and Qi Ju. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02 178, 2020.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong. Tao Qin, Liwei Wang, and Tie- Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv: 19990002772. 2019.
Nathan Ng. Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp- 314-319, 2019.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. arXiv preprint /91 .03864, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference Machine Learning, pp. 4334 4343, 2018.
Alexander M Rush, SEAS Harvard, Sumit Chopra, and Jason Weston. A neural attention model for sentence summarization. In ACLWeb. Proceedings of the 2015 conference on empirical methods in natural language processing, 2017.
Roy Schwartz, Gabi Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A Smith. The right tool for the job: Matching model and instance complexities. arXiv preprint arXiv:2004.07453 2020.
11

---

Published as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1XXV:19508.0799, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv: wr0vv:160.00281,, 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of -ofturreff-experts layer. arXiv preprint arXiv
2017.
David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pp. 5877-5886, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 597-403 2018.
Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adversarial training. In International Conference on Machine Learning, pp. 6555-6565, 2019a.
Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. reinforced topic-aware convolutional sequence-to to--tt--ssss to-sequence model for abstractive text summarization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 4453-4460, 2018
Wenbo Wang, Yang Gao, He-Yan Huang, and Yuxiang Zhou. Concept pointer network for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3067-3076, 2019b.
Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, Cheng Xiang Zhai and Tie Yan Liu. Multi- agent dual learning. In 7th International Conference on Learning Representations, ICLR 2019, 2019c.
Bolin Wei, Ge Li. Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation as a dual task of code summariza- tion. In Advances in Neural Information Processing Systems, pp. 6559-6569, 2019.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoc Le.
Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754-5764, 2019.
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SETTINGS AND MORE RESULTS
A.1 DETAILED DATA SETTINGS
Neural Machine Translation Following the common practice (Ott et al. 2019) we lowercase all words for IWSLT14 En+>De. For IWSLT14 En+>De, En++Es, IWSLT17 En Fr, we use joint source and target vocabulary with 10k syte-pair-encoding (BPE) (Sennrich al., 2015) operations, and for IWSLT17 En++Zh, use seperate source and target vocabulary. For all WMT tasks, sentences are encoded by a joint source and target vocabulary of 32k tokens.
Code Generation In the Java dataset, the numbers of training, validation and test sequences are 69, 708, 8, 714 and 8, 714 respectively, and the corresponding numbers for Python are 55, 538, 18, 505 and 18, 502. All samples are tokenized. We use the downloaded Java dataset without further processing, and use Python standard AST module to further process the python code. The source and target vocabulary sizes in natural language Java code generation are 27% and 50k, and those for natural language to Python code generation are 18k and 50k. In this case, following Wei et al. (2019) do not apply subword tokenization like BPE to the sequences.
Abstractive Summarization The Gigaword corpus represents for a headline generation task, each source article contains about 31.4 tokens on average, while the larget headline contains near 8.3 tokens per sentence. The training data consists of 3.8M article-headlie and test set consist of 190k and 2k pairs respectively. We preprocess the dataset same way as NMT task. The words in the source article and target headline are concatenated to make joint BPE vocabulary. After preprocessing, there are 29k subword tokens in the vocabulary.
pairs, while the validation
A.2 DETAILED MODEL/TRAINING CONFIGURATIONS
Model Configuration The detailed model configurations are as follows:
frransformer_is'/ lt_____ssss_deee____e__________e httpsslt__e__e__eeeee _e_e_nn___________e_________e___ setting: 6 blocks in encoder and decoder, embedding size 512, feed-forward size 1024, attention heads 4, dropout value 0.3, weight decay 0.0001.
transformer wwwswani.aaaaaa...a....a.. wwtpwwniwwtteeee wmt_en_dde big setting: 6 blocks in encoder and decoder, embedding size 1024, feed-forwad size 4096, attention heads 16, dropout value 0.3, atten- tion dropout 0.1, relu dropout 0.1.
transformen wwwperrwwtteeeeeeeeeeeeeeeeeeeee _eiieieegggg..... setting: 6 blocks in encoder and decoder, embedding size 0124, feed-forward size 4096, attention heads 16, dropout value 0.3.
Optimization We adopt the default optimization setting in Vaswani et al. (017). Adam (Kingma & Ba 2014) optimizer with 81 0.9, 32 0.98 and
10 ? The learning rate scheduler is inverse sqrt with warmup steps 4, 000, default learning rate is 0.0005. Label smoothing
(Szegedy al. (2066) is used with value 0.1. As introduced, to learn the predictors, we clamp the softmax output with value 0.05.
A.3 RESULTS OF ORDER COMBINATIONS
We show the paper that different number of orders (e.g., or N = 5) have varied perfor- mances. Therefore, necessary point is about the different combinations of these N decoders. Here, we work on N IOT model to show the results of different order candidates.
We first present each ordered decoder in Table I again (same in Table I.. For the N - ordered decoders with 1OT model, we show the performances with 5 combined orders selected from all six variants dev set of IWSLT14 De and En En translations. The results reported in Table We can see the different combinations achieve similar strong performances, which shows that our
12
Code
2
3
4
5
6
Order FF SA-ED->FF
+ED
ED-
ED-
SA-
FF-ED- SSA
Table 11: Each numbered code for specific ordered decoder.
13

---

Published as a conference paper at ICLR 2021
10T (N 5)
12345
12346
12356
12456
13456
23456
En-De
30.64
30.65
30.53
30.58
30.70
30.67
De EEn
36.71
36.67
36.72
36.79
36.74
36.75
Table 12: BLEU scores for IWSTL14 De-+En and En-*De translations on dev set. '12345' represents the combinations of order 2, 3, 4, decoders.
approach is robust towards different order combinations. This also demonstrate that the importance of IOT is the diversity among order candidates that can help each data distinguish them. For other N ordered decoders, the patterns are similar. Therefore, here we only report the N combinations used for IOT experiments the paper as follows: IOT (N 2) is combined by order 4, (ED+SAA+P and FF- and IOT (N 3) is order 1, 1. 6. IOT (N 4) is order 1,2, 4, 6, and IOT (N 5) is order 1, 4, 5,
A.4 RESULTS OF DIFFERENT NUMBER OF DECODERS
The results of N = 4 ordered decoders (order 1, 1, 4, 6) are mainly reported in the paper. Here, we also show results of other N decoders for all tasks, along with the Transformer baseline.
The results of different N decoders for WMT14 En and WMT16 Rn translations. code generation task, and Gigaword summarization are reported in Table [4] Table [5] and Table [6] respectively. As we can see, more ordered decoders can bring better performance, which supports the effectiveness of our framework and demonstrates the data has own favor towards different orders. Considering the efficiency, we do not perform experiments with more than 4 decoders for these tasks.
Model
De
De-+En
En-+Fr
Fr-+En
En +Zh
Zh 7h++En
En En-+Es
Es
Fnn
Transformer
28.57
34.64
35.9
36.1
26.3
18.4
39.0
40.6
IOT (N 6)
29.48
35.60
37.4
37.7
27.1
19.2
40.2
41.5
IOT (N 5)
29.51
35.65
37.2
37.6
27.2
19.2
40.2
41.9
1OT (N = 4)
29.52
35.62
37.2
37.8
27.2
19.3
40.1
41.7
IOT (N 3)
29.43
35.58
37.0
37.6
27.0
19.1
39.7
41.5
IOT (N- 2)
29.18
35.32
36.6
37.1
26.8
18.9
39.6
41.0
Table 13: BLEU scores on 8 IWSLT tasks with different N ordered decoders (with shared weights).
Model
WMT14 En-DDe
WMT16 Ro-EEn
Transformer
29.12
37.73
IOT (N 2)
29.71
38.57
1OT (N = 3)
29.89
38.79
IOT (N 4)
30.03
38.83
Table 14: BLEU scores for WMT14 En->ee and WMT16 Ro-=n translation tasks different N ordered decoders in 1OT.
Java
Python
Model
BLEU
PoV
BLEU
PoV
Transformer
24.58
74.44%
13.20
61.89%
IOT (N 2) 25.44
77.88%
13.97
62.22%
IOT (N = 3)
25.51
75.83%
14.00
62.04%
1OT (N 4)
25.51
77.30%
14.05
63. 63.14%%
Table 15: BLEU and PoV scores for Java and Python code generation results of different N ordered decoders in IOT.
14

---

Published as a conference paper at ICLR 2021
Model
ROUGE-1
ROUGE-2
ROUGE-L
Transformer
35.59
17.87
32.98
IOT (N 2)
36.14
18.37
33.61
1OT (N 3)
36.15
18.48
33.81
1OT (N 4)
36.37
18.46
33.89
Table 16: ROUGE Fl scores for Gigaword abstractive summarization results of different N ordered decoders in IOT.
c1/02
0.0
0.005
0.01
0.05
0.0
35.97
36.36
36.46
36.42
0.05
36.35
36.58
10T (NV 2)
0.1
36.31
36.56
36.60
36.54
0.5
36.39
36.40
0.0
36.16
36.44
36.66
36.51
0.05
36.47
36.47
10T (N = 3)
0.1
36.35
36.48
36.79
36.57
0.5
36.44
36.62
Table 17: BLEU scores for IWSLT14 De-+En dev set. The performances are varied by different weighted auxiliary losses controlled by ei and 02 value.
B MORE STUDIES
B. IMPACT OF WEIGHTED AUXILIARY LOSSES
We conduct another study IWSLT14 De dev set to investigate the impact of proposed auxiliary losses controlled by weight and c2. The values of and 0 are varied between 10.0, 0.05, 0.5) and [0.0, 0.005, 0.01, 0.05] respectively, and the results are presented in Table 17 It can be seen that the best configuration is = 0.1 and
0.01. Therefore, we report the leading results the paper with c 0.1, C2 = 0.01. The results also clearly demonstrate that the two additional losses are necessary make our framework effective.
B.2 DATA EXAMPLES VERIFICATION
the data split by the corresponding predicted order in different pattern. For example, the difficulty of each is different. We therefore analyze the split data and calculate some statistics among these subsets. Specifically, we first count the sentence number S, the tokens T. and the distinct vocabulary D; each subset. We show these numbers Table [8] along with corresponding averaged BLEU score (see Figure 2). We can see that the vocabulary size of set is the smallest, and set is the largest, which means there are more distinct words in set 2. This leads the generation of set 2 to be harder than set 1, which maps the BLEU score ranking among these sets. Besides, we also calculate the token frequency fig for token j in each own subset i, and sum the frequency of top 20 tokens in each subset, F, = Σø=1 fiy. to give another evidence. The results also show that Fi is the highest, which means the tokens in set 1 contains most frequent words to make an easy learning, while set is harder since F22 small.
As discussed in Section 5.2
Set
S
T;
D;
BLEU Aug
2, 404
56, 036
4, 557
0.3693
32.49
2, 093
49, 077
4, 897
0.3234
27.92
i =
3
2, 786
66, 226
4, 699
0.3668
30.28
Table 18: Statistics of each English valid subset IWSLT14 En En translation. S is the number of sentences in set Conrespondingly, T; the token number, D; is the vocabulary size. F, = Jas is the sum of the frequency of top 20 tokens in set where fij is the frequency for token
15

---

Published as a conference paper at ICLR 2021
i will come it later wasn &apos; very good at reading things.
Set
this is very little known fact about the two countries. &apos;s why they leave lights around the house.
Set 2
and how can we do that ?
Set
&apos;s our faith, and we will he lo@ @ y @ al to it.
Table 19: Samples of each English valid subset on IWSLT14 En En->De translation.
We further take look at the data and find the sentences in set are mostly "simple sentences", and set contains many "emphatic sentences", while set 3 somehow mixed. In Table [9] we provide some sentence examples belong each subset give more clarifications.
B.3 REGULARIZATION
In Section 5.4 we have provided an example of regularization experiments on IWLST14 De translation, which demonstrates that our 1OT is not only regularization and can be smoothly inte- grated with other regularization methods. To give more evidences and more details, we extend the regularization experiments on all IWSLT translation tasks (IWSLT14 En IWSLT14 En++Es, IWSLT17 En++Zh, IWSLT17 En++Fr), and WMT16 Ro-En translation. The two specific settings of the experiments are as follows. Setting (1): The first experiment is "ordered Transformer" without instance awareness. That is, all the reordered architectures are trained the whole same corpus with equal weights, and the parameters for these reordered architectures are shared. More specif- ically, the decoder block has different ways order SA, ED, and FF layers (e.g., FF-SSA>ED, SA-ED--FF ctc), but the parameters for the reordered blocks are shared. Mathematically, the loss function is: L >n (A,, Lx"") where LM" is the model loss function for n-th or- dered decoder. Compared with Eqn (3). the weight An is fixed to be here. At inference, we first find out the best order according the dev performance and apply it on the test set. We cannot use instance-we wise reordered model in this setting, while our proposed IOT can. The exper- iments are conducted with trans former iwslt de configuration for IWSLT translations, and t rans former vaswani wmt en de big configuration for WMT16 Ro->F translation. Setting (2): We integrate another regularization technique 'LayerDroop' (Fan al. 2019) into both the Transformer baseline and our IOT (N 4) method, while other settings remain unchanged. The study results of these two settings are represented in Table 20
From the results, we have same conclusions as discussed in Section 5.4 Simply sharing the parameters of different decoders regularization cannot boost the model performance ("Transformer (1)" in while our IOT can further improve the performance with other regularization methods.
Table 20
B.4 ROBUSTNESS
An impact of IOT training besides performance gain is that the model can be more robust compared to order only. In Table 21] provide one example to prove the robustness. We train one Transformer model by decoder order /, and to decode the sentences with all orders in inference. Obviously, only decoding with order leads good performance, while other orders can not achieve reasonable scores since the layer order is changed and the feature exaction becomes incorrect. As for IOT, the generated sequences remain stable and high results for each order.
B.5 VISUALIZATION
To better understand the difference between IOT and standard Transformer, investigate the training process and provide visualization results about model optimization and performance improvements. Specifically, we plot the curve of training loss, validation loss, well the validation BLEU score and test BLEU score along the training epochs on IWSLT14 De->En translation dataset. The loss curves are visualized in Figure and the BLEU curves are presented in Figure 4
From the validation loss curves of Figure ((b) and 3(c) we can first see that our IOT (N = 3) training converges faster than Transformer baseline and shows the advantage of IOT, which is consistent to
16

---

Published as a conference paper at ICLR 2021
Model
En DD
De->En
Fir+Frr
Fr Er->En
Transformer IOT (N = 1)
28.57
34.64
35.9
36.1
29.52
35.62
37.2
37.8
Transformer + (1)
28.35 + 0.12
34.68 0.05
35.9 = 0.23
36.7 + 0.14
Transformer + (2)
29.01
35.40
36.2
36.8
IOT (N = 4) (2)
29.94
36.13
37.4
38.1
Model
En -Zh
Zh Zh+EEn
En-EEs
Es Es
Ro-En
Transformer 1OT (N 4)
26.3
18.4
39.0
40.6
37.73
27.2
19.3
40.1
41.7
38.83
Transformer + (1)
26.4 + 0.05
18.8 0.17
37.7
0.28
39.6 + 0.11
37.82 + 0.05
Transformer + (2)
26.8
19.0
39.1
40.8
38.33
IOT (N = 4) + (2)
27.3
19.5
40.6
42.5
38.98
Table 20: Regularization study experiments on IWSLT translation tasks and WMTI6 Ro-EEn translation. We study both setting (1): train Transformer model with all shared decoders but without instance-wise learning, and setting (2): add LayerDrop (Fan et al. 2019) regularization technique experiments these tasks.
Order
Transformer
1OT
1*
35.84
36.42
27.96
36.45
8.05
36.47
5.11
36.38
1.32
36.46
6
0.35
36.41
Table 21: Robustness study on IWSTL14 De->En translation task dev * is the order trained by Transformer.
our analysis Section 5.1 The converged (smallest) validation loss value seems to similar to Transformer bascline, but please note that the loss computation of IOT is different from Transformer baseline. As shown in Eqn (3), the loss function of IOT a weighted sum loss values for each order, while for Transformer, is only one order loss. Therefore, when turn to the comparison of validation BLEU score, the superiority of our IOT can be clearly verified. From the BLEU score curves in Figure is obvious that IOT achieves better BLEU score than standard Transformer along the whole training epochs, on both validation and test sets. These visualized results well demonstrate the effectiveness of our IOT approach.
4.10
10
Boentine
Validation Loss
Valldbation Loss.
-an Lan
Training
4.00
1/
m
3.05
20
40
60
80
20
40
60
80
3.90
20
40
60
80
100
Epoch
Epoch
Epoch
Training loss
(b) Validation loss (y-axis: 1.0-9.0Xc) Validation loss (y-axis: 3.9-1.1)
Figure 3: Comparison of training/validatinn loss curves along the model training IWSLT Dn translation. 'IOT' is IOT (N 3) and 'Baseline" is Transformer. Figure 3(0) is the same curve as 3(b) except the value of y-axis in Figure 3(c) is between 3.9-4.1, while 4.9-9.0 for Figure 3(b)
17

---

Published as a conference paper at ICLR 2021
36
Validalion BLEU
30
Test BLEU
Test BIEU
-5
MM
10
34
10T
33
100
Epoch
(a) Validation BLEU
(b) Test BLEU (y-axis: 0-36)
(c) Test BLEU (y-axis: 33-36)
Figure 4: Comparison of validation/test BLEU curves along the model training on IWSLT De->En translation. 'IOT' is IOT (N = 3) and 'Baseline' is Transformer. Figure 4() is the same curve as 4(b) except the value of y-axis in Figure is between 33-36, while 0-36 for Figure 4(b)
18


Reviews:
[{"review_id": "v7cQFreGuGi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work explores instance-wise layer re-ordering in transformers. The key idea is to incorporate classifiers that predict the ordering of sub-layers (self-attention, cross-attention, feed-forward) from the averaged input sequence representation, one classifier each for the encoder and the decoder. During training the model uses a soft Gumbel-noised output of the classifier to combine the outputs from stacks with differently ordered sub-layers. During inference the argmax of the classifier prediction is used to generate the output sequence. The model is trained with two auxiliary losses: (i) A loss to ensure the expected output of the classifiers is uniform and (ii) A loss to ensure the classifier output for each individual sample is distant from uniform.\nExperiments are performed on 8 IWSLT translation tasks, code generation and Gigaword summarization. The proposed approach results in significant gains over a standard transformer on all the reported tasks.\n\nStrengths:\n1. Significant gains on a variety of sequence-to-sequence NLP tasks.\n2. The authors conduct several experiments trying to uncover the source of the quality gains, comparing against ensembling approaches, layer re-ordering based regularization and layerdrop on the IWSLT En-De task. The approach is also demonstrated to improve the quality of a DynamicConv based Machine Translation model (although less effective than in the Transformer setting).\n\nWeaknesses:\n1. While the authors do compare against regularization (layer re-ordering), the settings of these experiments are not very clear and the results are reported only on the IWSLT De->En task. A more careful comparison on all the reported tasks to counter against the regularization hypothesis would significantly strengthen the claims in the paper.\n\nOther questions for authors:\n1. On page 2 in the introduction, the reported average Bleu variance per sample (114.76) seems way too high, given that Bleu $\\in [0, 100]$.\n2. Do the predictor outputs approach the argmax? If not, is there a quality loss incurred by choosing the argmax instead of using the expectation like during training?\n3. Did the authors try using a single predictor for both the encoder and decoder (i.e. a single classifier predicting $M\\times N$ classes)?\n4. Are the layer orders preferred by a particular input (or the subset of input examples that prefer the same layer order) consistent across randomly initialized training runs? This analysis might help justify the claim that certain layer orders are better for harder/easier inputs.\n5. Other relevant work on instance-level adaptation in NLP: [1,2,3]\n\nRecommendation: I think there are several interesting ideas in the paper, and would recommend (weak) acceptance. There is still some doubt as to whether the quality gains are arising from a regularization effect or the instance-based layer re-ordering. I would be willing to update my recommendation given additional evidence to address this concern.\n\nReferences:\n\n[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER, Shazeer et al.\n\n[2] Controlling Computation versus Quality for Neural Sequence Models, Bapna et al.\n\n[3] The Right Tool for the Job: Matching Model and Instance Complexities, Schwartz et al.\n\nEDIT: Updated recommendation to acceptance (7) following author response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SI7UWwaSOPi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the influence of the arrangement order for the internal structure in a single-layer Transformer (they named it as layer order) on the performance. It makes a hypothesis that different layer order has an impact on the performance of the model, and the hypothesis is verified by experiments. Based on this hypothesis, a lightweight layer order predictor is designed to predict an input-related layer order, and through reinforcement learning with two auxiliary loss, the model can not only be trained by diverse layer order, but also make unambiguous layer order prediction as far as possible. The IOT structure proposed in this paper has been evaluated on several datasets of machine translation, abstract summarization and code generation. Compared with the traditional transformer structure, it has been improved consistently, which shows the effectiveness of the proposed structure.\n\n\n\nMy concerns are as the following,\n\n1. Training time: it seems that this paper only reports the time of inference. However, compared with the standard Transformer, the IOT proposed in this paper only changes the order inside the Transformer layer and it does not increase the additional inference time consumption. In addition, too simple predictor will have little impact over time cost. However, for training, due to the additional loss, additional order exploration, etc., the convergence time of the model may be changed. Therefore, I expect to see comparison of IOT with Transformer.\n\n\n2. Internal order: since the mentioned internal structure order of Transformer will affect the results, meanwhile, according to the previous work, the order of LayerNorm may also affect the performance of the model. I am curious if such a structure can be also considered and compared.\n\n3. Model capacity: There is a claim that the capacity of the model has been improved due to the introduction of IOT, while there is no direct evidence from experiments that can support such a claim. Since the parameters of the model have not changed, I feel it hard to regard the improved model capacity. Besides, it is not clear on the definition of the model capacity.\n\n4. Different input and different structures: because this paper claims that different inputs should bring different structures, but according to the current training method with mini-batch for neural networks, it seems that it is impossible to give a completely different structure for each sentence. I expect to see some obvious enough evidence on different structures in sequence-level or batch-level from such training.\n\nMinor comments,\n\nIt would be better if the ablation experiments are done on a larger dataset such as WMT 14.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GYliPA4drwL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the general idea behind this paper is very exciting: architectures mutating in an instance-based way.\n\nI was hoping to see that the authors had achieved this whilst adding only a small overhead of parameters and training FLOPs such that we could be sure any gains aren't due to augmenting these two quantities (i.e. instance-based re-ordering is the key ingredient) and to be sure this could be applied in the large-model setting and theoretically improve production translation systems.\n\nHowever the proposed architecture is essentially training an ensemble of three, or n_reorder, weight-shared models and then at inference time, using a hard-threshold. The paper really tries to brush over this fact, stating there is negligible additional cost in the abstract and in several other parts of the text. There is only this one line to acknowledge the fact that we are training 3x (or even 4x) models:  \"One may concern the training cost is increased through our training. As we present in Appendix B.1, the cost is actually acceptable with a fast convergence.\" and then this section in the appendix continues to mostly discuss inference cost, but finally admits the n_reorder flops increase stating that it is acceptable because the model converges faster. It doesn't seem likely the model will converge 3x faster in general, especially if it learns to mostly select only one re-ordering after some training.\n\nMy general sense is the proposed architecture sits in a very tenuous space. If the researcher has 3-4x space and flops to train a model 3-4x larger, then they should clearly do so. This will get much better performance. If the researcher has 3-4x space to train a larger model but wants faster inference, perhaps this approach could be useful but I would suspect training large and then pruning or distilling to be much more effective. If the architecture trained with hard re-ordering decisions and thus used the same number of flops, but still outperformed the baseline it would be a clear win.\n\nSuggestions:\n* Compare training flops and eval performance to the approach of training a 3x larger model that is then pruned or distilled.\n* Be up-front about the training wall-clock time in the main text, instead of stating everything has negligible cost and only reporting inference speed.\n* Consider using RL or another hard-decision approach.\n* Consider comparing to a model that has the same number of parameters but 3x the compute (e.g. a transformer with 3x depth but shared weights for each of the three layers).", "extra_scores": {}, "extra_text": {}}, {"review_id": "1j76PJ57v7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at improving the transformer architecture by reordering the encoder/decoder layers. Improving the performance of transformer is an active area of research and has numerous applications. The authors first show that a single layers' ordering is not better than all others and further demonstrate that per instance layer ordering with parameter sharing consistently improve overall performance (which is quite surprising). Other works considered per instance early return (depth) but I am not aware of per instance layer reordering. \n\n#### Strong points\n+ The layer order decision is made on simple sentence embeddings which does not add significant computation compared to the transformer. \n+ The weights are shared between layers of different reordering which does not increase the number of parameters.\n+ The authors conduct extensive evaluation on Neural Machine Translation, Code Generation and Summarization and show consistent improvement of their method (IOT).\n+ They show that this method is not a form of ensembling/experts (which was my main concern), IOT is orthogonal to ensembling (Section 5.3).\n+ The same reordering trick can be applied to other multi-layer architectures.\n\n#### Weak points\n\n- The different layer orders process the samples differently and require to split a batch into smaller batches for each ordering.\n- Why some samples are better processed by a specific layer ordering remains not understood.\n\n#### Decision\n\nI tend to accept this paper as the method is novel and the results are good. The method can be applied without strong drawbacks in term of computation or number of parameters.\n\n#### Questions\n\n- Table 15: I am surprised that inference time is not affected by the batching per order (see weak point), did you apply some tricks like masking to be more GPU friendly?\n- Can you show the proportion of each ordering? (for instance IOT (N=6) on IWSLT14) I am curious if it is balanced and if the decisions made by the classifier are consistent with the best performing order in Table 1 when each transformer order is trained separately.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the order of Transformer modules and its effect on performance. The proposed approach IOT, consists of several pre-defined encoder and decoders with different orderings (and weight sharing), along with a light predictor which is trained to choose the best configuration per instance.  

Most reviewers found the general idea of predicting the order of Transformer modules at instance-level quite intriguing. Other strengths included wide range of evaluation tasks, major empirical gains, and novelty.   
 R1 and R4 raise valid and important concerns on validity of results when the model size and training time are controlled.  
However, after carefully reading the author response and the revised paper, I feel that this issue is resolved.  
The authors provide comparison with larger models, ensemble models, and models trained longer, and in all cases the gains are still obvious.  
 Overall, I feel that the general idea behind this paper is very exciting and could inspire more research in this direction. Therefore I recommend accept. 
Decision: Accept (Poster)