Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: I think the general idea behind this paper is very exciting: architectures mutating in an instance-based way.

I was hoping to see that the authors had achieved this whilst adding only a small overhead of parameters and training FLOPs such that we could be sure any gains aren't due to augmenting these two quantities (i.e. instance-based re-ordering is the key ingredient) and to be sure this could be applied in the large-model setting and theoretically improve production translation systems.

However the proposed architecture is essentially training an ensemble of three, or n_reorder, weight-shared models and then at inference time, using a hard-threshold. The paper really tries to brush over this fact, stating there is negligible additional cost in the abstract and in several other parts of the text. There is only this one line to acknowledge the fact that we are training 3x (or even 4x) models:  "One may concern the training cost is increased through our training. As we present in Appendix B.1, the cost is actually acceptable with a fast convergence." and then this section in the appendix continues to mostly discuss inference cost, but finally admits the n_reorder flops increase stating that it is acceptable because the model converges faster. It doesn't seem likely the model will converge 3x faster in general, especially if it learns to mostly select only one re-ordering after some training.

My general sense is the proposed architecture sits in a very tenuous space. If the researcher has 3-4x space and flops to train a model 3-4x larger, then they should clearly do so. This will get much better performance. If the researcher has 3-4x space to train a larger model but wants faster inference, perhaps this approach could be useful but I would suspect training large and then pruning or distilling to be much more effective. If the architecture trained with hard re-ordering decisions and thus used the same number of flops, but still outperformed the baseline it would be a clear win.

Suggestions:
* Compare training flops and eval performance to the approach of training a 3x larger model that is then pruned or distilled.
* Be up-front about the training wall-clock time in the main text, instead of stating everything has negligible cost and only reporting inference speed.
* Consider using RL or another hard-decision approach.
* Consider comparing to a model that has the same number of parameters but 3x the compute (e.g. a transformer with 3x depth but shared weights for each of the three layers).
review_title: Does this increase the training flops by a factor of n_reorder?

final_score: rating: 7: Good paper, accept
review_content: Summary: This work explores instance-wise layer re-ordering in transformers. The key idea is to incorporate classifiers that predict the ordering of sub-layers (self-attention, cross-attention, feed-forward) from the averaged input sequence representation, one classifier each for the encoder and the decoder. During training the model uses a soft Gumbel-noised output of the classifier to combine the outputs from stacks with differently ordered sub-layers. During inference the argmax of the classifier prediction is used to generate the output sequence. The model is trained with two auxiliary losses: (i) A loss to ensure the expected output of the classifiers is uniform and (ii) A loss to ensure the classifier output for each individual sample is distant from uniform.
Experiments are performed on 8 IWSLT translation tasks, code generation and Gigaword summarization. The proposed approach results in significant gains over a standard transformer on all the reported tasks.

Strengths:
1. Significant gains on a variety of sequence-to-sequence NLP tasks.
2. The authors conduct several experiments trying to uncover the source of the quality gains, comparing against ensembling approaches, layer re-ordering based regularization and layerdrop on the IWSLT En-De task. The approach is also demonstrated to improve the quality of a DynamicConv based Machine Translation model (although less effective than in the Transformer setting).

Weaknesses:
1. While the authors do compare against regularization (layer re-ordering), the settings of these experiments are not very clear and the results are reported only on the IWSLT De->En task. A more careful comparison on all the reported tasks to counter against the regularization hypothesis would significantly strengthen the claims in the paper.

Other questions for authors:
1. On page 2 in the introduction, the reported average Bleu variance per sample (114.76) seems way too high, given that Bleu $\in [0, 100]$.
2. Do the predictor outputs approach the argmax? If not, is there a quality loss incurred by choosing the argmax instead of using the expectation like during training?
3. Did the authors try using a single predictor for both the encoder and decoder (i.e. a single classifier predicting $M\times N$ classes)?
4. Are the layer orders preferred by a particular input (or the subset of input examples that prefer the same layer order) consistent across randomly initialized training runs? This analysis might help justify the claim that certain layer orders are better for harder/easier inputs.
5. Other relevant work on instance-level adaptation in NLP: [1,2,3]

Recommendation: I think there are several interesting ideas in the paper, and would recommend (weak) acceptance. There is still some doubt as to whether the quality gains are arising from a regularization effect or the instance-based layer re-ordering. I would be willing to update my recommendation given additional evidence to address this concern.

References:

[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER, Shazeer et al.

[2] Controlling Computation versus Quality for Neural Sequence Models, Bapna et al.

[3] The Right Tool for the Job: Matching Model and Instance Complexities, Schwartz et al.

EDIT: Updated recommendation to acceptance (7) following author response.
review_title: Good paper, regularization experiments could be more thorough

final_score: rating: 7: Good paper, accept
review_content: This paper aims at improving the transformer architecture by reordering the encoder/decoder layers. Improving the performance of transformer is an active area of research and has numerous applications. The authors first show that a single layers' ordering is not better than all others and further demonstrate that per instance layer ordering with parameter sharing consistently improve overall performance (which is quite surprising). Other works considered per instance early return (depth) but I am not aware of per instance layer reordering. 

#### Strong points
+ The layer order decision is made on simple sentence embeddings which does not add significant computation compared to the transformer. 
+ The weights are shared between layers of different reordering which does not increase the number of parameters.
+ The authors conduct extensive evaluation on Neural Machine Translation, Code Generation and Summarization and show consistent improvement of their method (IOT).
+ They show that this method is not a form of ensembling/experts (which was my main concern), IOT is orthogonal to ensembling (Section 5.3).
+ The same reordering trick can be applied to other multi-layer architectures.

#### Weak points

- The different layer orders process the samples differently and require to split a batch into smaller batches for each ordering.
- Why some samples are better processed by a specific layer ordering remains not understood.

#### Decision

I tend to accept this paper as the method is novel and the results are good. The method can be applied without strong drawbacks in term of computation or number of parameters.

#### Questions

- Table 15: I am surprised that inference time is not affected by the batching per order (see weak point), did you apply some tricks like masking to be more GPU friendly?
- Can you show the proportion of each ordering? (for instance IOT (N=6) on IWSLT14) I am curious if it is balanced and if the decisions made by the classifier are consistent with the best performing order in Table 1 when each transformer order is trained separately.
review_title: A simple modification of Transformer that consistently improve its performance without parameters/computation drawback.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the influence of the arrangement order for the internal structure in a single-layer Transformer (they named it as layer order) on the performance. It makes a hypothesis that different layer order has an impact on the performance of the model, and the hypothesis is verified by experiments. Based on this hypothesis, a lightweight layer order predictor is designed to predict an input-related layer order, and through reinforcement learning with two auxiliary loss, the model can not only be trained by diverse layer order, but also make unambiguous layer order prediction as far as possible. The IOT structure proposed in this paper has been evaluated on several datasets of machine translation, abstract summarization and code generation. Compared with the traditional transformer structure, it has been improved consistently, which shows the effectiveness of the proposed structure.



My concerns are as the following,

1. Training time: it seems that this paper only reports the time of inference. However, compared with the standard Transformer, the IOT proposed in this paper only changes the order inside the Transformer layer and it does not increase the additional inference time consumption. In addition, too simple predictor will have little impact over time cost. However, for training, due to the additional loss, additional order exploration, etc., the convergence time of the model may be changed. Therefore, I expect to see comparison of IOT with Transformer.


2. Internal order: since the mentioned internal structure order of Transformer will affect the results, meanwhile, according to the previous work, the order of LayerNorm may also affect the performance of the model. I am curious if such a structure can be also considered and compared.

3. Model capacity: There is a claim that the capacity of the model has been improved due to the introduction of IOT, while there is no direct evidence from experiments that can support such a claim. Since the parameters of the model have not changed, I feel it hard to regard the improved model capacity. Besides, it is not clear on the definition of the model capacity.

4. Different input and different structures: because this paper claims that different inputs should bring different structures, but according to the current training method with mini-batch for neural networks, it seems that it is impossible to give a completely different structure for each sentence. I expect to see some obvious enough evidence on different structures in sequence-level or batch-level from such training.

Minor comments,

It would be better if the ablation experiments are done on a larger dataset such as WMT 14.

review_title: overclaim

Meta-review: The paper investigates the order of Transformer modules and its effect on performance. The proposed approach IOT, consists of several pre-defined encoder and decoders with different orderings (and weight sharing), along with a light predictor which is trained to choose the best configuration per instance.  

Most reviewers found the general idea of predicting the order of Transformer modules at instance-level quite intriguing. Other strengths included wide range of evaluation tasks, major empirical gains, and novelty.   
 R1 and R4 raise valid and important concerns on validity of results when the model size and training time are controlled.  
However, after carefully reading the author response and the revised paper, I feel that this issue is resolved.  
The authors provide comparison with larger models, ensemble models, and models trained longer, and in all cases the gains are still obvious.  
 Overall, I feel that the general idea behind this paper is very exciting and could inspire more research in this direction. Therefore I recommend accept. 
Decision: Accept (Poster)