Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new transformer model called DeLighT. DeLighT differs from the original transformer in the following ways:
- a DeLighT transformation is performed before the self-attention 
- the self-attention is performed in a vector space with fewer dimensions and with only 1 head
- the MLP’s hidden layer is 16 times smaller 
- The DeLighT transformation, as far as I understand, is a sequence of block-sparse linear layers and permutations. The number of intermediate layers in DeLightT transformations is higher in the upper layers of the whole model. 

The key claims of the paper are that DeLight performs as well as or better than Transformer, while being deep, having less parameters and involving fewer floating point operations. With regard to depth, I don’t think that depth can be the goal in and of itself, it is the means for achieving good performance. I think the paper puts too much emphasis on DeLight depth. I also find it rather controversial that the depth in the paper counts all linear layers, and not just the number of nonlinearities between the input and the output.

DeLighT does seem to have a consistent edge over the original Transformer model in terms of both performance and memory footprint. Compared to TransformerXL and EvolvedTransformer, DeLighT performs on par, but has less parameters. The set of experiments performed to assert DeLighT strengths is quite extensive.  The experimental side of paper appears technically sound. 

It is important to build more efficient and compact models, but I find the paper’s analysis of models computational demands rather incomplete. If computational efficiency is the main focus of the paper (and it appears to be), it would be great to see some analysis of how much time and memory is required for model training and inference. Is the model size the dominating factor here? 

Overall, the architecture ends up being quite complicated. The supplementary material contains an extensive set of ablations justifying many of the complexities, but I find that a few straight-forward remain unanswered:
- how does DeLighT compare to a Transformer with one head
- how does DeLighT compare to a Transformer with a narrow MLP
- what about both above modifications? 
I think it is important for the paper to justify the use of the DeLighT layer. 

With regard to writing, the paper is generally quite clear and accurately done. A few places raised my eyebrows: 
- “Since the DeLighT block learns wider representations of the input using the DeLighT transformation, it enables us to replace multi-head attention with single-head attention” - I think the causation here is not obvious and needs to be proven. Or the wording should be changed to illustrate that this is just your intuition.
- “This is likely because scaling model width and depth allocates parameters uniformly across blocks, which may lead to learning redundant parameters.” - same, I think it is important to state that this is a hypothesis
- I think the “input mixer” layer and how it widens the state should be discussed in the main text. Otherwise the role of w_m is completely unclear. 

The paper’s novelty appears quite limited. DeLightT layers seem similar to the DeFINE layer. The difference is not sufficiently explained neither in the main text nor in the appendix. Overall it seems that the paper takes existing components and does a lot of tweaking to yield a better model with good performance. 

It is hard to make a call in the case of this paper. 
Pros:
- the final moment model does deliver some improvement
- the execution is quite accurate
Cons:
- the paper feels very incremental 
- the papers leaves an impression that its key goals could be achieved in a simpler way
- compute time and runtime memory footprint are not discussed. 

UPD: the score was updated after the rebuttal stage.
review_title: limited contribution, but the claimed goal is achieved; very borderline

final_score: rating: 7: Good paper, accept
review_content: This paper presents a variant of Transformer where low-dimension matrix multiplications and single-head attention are used. Stacked group-linear-transformation (GLT) are applied on input of each layer to perform dimension growth and then reduction. The paper is well-written and easy to follow. Experiments demonstrate the propose architecture matches or improves the performance of baseline Transformers with fewer parameters.

Although the proposed architecture has fewer MACs, it would be interesting to know the real decoding time. Deep models could reduce model size or improve performance, but layers have to be executed consecutively and may slow down decoding in practice especially when using deep decoders. I’m also curious about the performance when keeping decoders unchanged.


review_title: Impact on  decoding in practice

final_score: rating: 6: Marginally above acceptance threshold
review_content: The original Transformer includes many design heuristics, e.g., the arrangement of layers and how layer widths are tied ($d_{ffn} = 4 \cdot d_{model}$, $d_k=d_q=d_v$, etc). While much effort has been spent on up-scaling, it is also important to question these basic design heuristics.

This work does just that and presents a more parameter-efficient Transformer variant by combining existing techniques (GLT, Mehta et al, 2018; Channel Shuffling, Zhang et al. 2018) and exploring some known trade-offs (feedforward net ratio, nheads vs dhead) under the new architecture. The resulting model, DeLighT, while more complicated to describe, performs on par with the original Transformer with far fewer trainable parameters. The ablation studies seem thorough and most design choices are more or less accounted for.

My main concern is the complete absence of any discussions on time-efficiency, or latency. The only reference I could find is in the conclusion: “we expect a dedicated CUDA kernel for DeLighT units to be much more efficient, both in terms of speed as well as memory during forward and backward passes.” This implies that the current implementation might not be efficient speed-wise. However, even after excluding the benefits from CUDA kernels, this new architecture appears to be less parallelizable as it is significantly deeper than other Transformer variants. This is crucial as neural network accelerators are becoming increasingly available, even in edge devices. Before fully recommending this paper for acceptance, I’d like to see more discussions on how the proposed changes affect latency, a latency benchmark against other relevant Transformer variants, especially with parallelism, and some concrete statements regarding latency in the abstract and the conclusion. I would still consider this work valuable even if the result on latency is not strongly positive, but the work seems incomplete without any discussions of it.

Minor issues:
1)     In table 4, Delight has a context length of 480. Why not 640 for a more direct comparison?
2)     There’s little detail on how the hyperparameters are selected for the ablation study. Are they swept to account for the change in optimal hyperparameters with architectural change?
review_title: A thorough empirical investigation on parameter-efficient Transformers, though missing discussions on latency

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper replaces the standard MLP block with a novel building block called DeLight. DeLight is a deeper MLP of partially group-wise linear layers, which leads to parameter and potential compute savings. The authors show that their approach outperforms other transformer-like architectures with more parameters on machine translation and language modeling.

The results of the paper look encouraging at first glance but I have a couple of major concerns with this paper as it is right now: 

1) The paper's motivation is that parameter rich, wide and shallow models are problematic to train due their requirement of large training data or strong regularization. However, the authors use the same training setup for both delight and standard transformers on the same datasets, and results for standard transformers are just fine, though lacking a bit behind delight. In particular, the authors do not show that using delight significantly alleviates any of the mentioned problems. They do not show that it needs less data, nor that it is "easier" to regularize.

2) The paper seems to suggest that using less parameters is better per-se, which I strongly disagree with. Especially with huge amounts of data the main driving factors for performance are compute + number of parameters. Making networks wider (instead of deep) has the advantage of massive parallel computation, which is always faster than deeper networks. That's one of the main reasons why RNNs are not used anymore in language. Furthermore, deeper models will have other issues such as storing many more activations which can result in memory bottlenecks.

3) The paper misses to discuss the most important factor which is train efficiency. In other words, how much faster can delight lead to better results than standard transformers? Note, that "larger models [can] train faster" [1], which is especially the case given large enough datasets which is definitely the case for language modeling. Unfortunately, this study optimizes for and normalizes by number of parameters. However, it should normalize by on-device training time or at least the idealized training time in flops or MACS. Right now there are some MACS comparisons but they are just for a single forward pass, not total training, which is a bit meaningless. I would like to see a proper comparison of performance to training time/flops. The best way to do this study would be to control for total training time/flops.

Finally, without showing a clear advantage in terms of training speed it is hard to judge the significance of this architectural block. I would also like to see how the differences progress when increasing the total compute equally for the vanilla transformer and delight. Will the gap between those widen or narrow the larger the models become? From my experience, the larger the models and datasets it is mostly the total compute that decides the performance, so I would actually expect the gap to become narrower.

Now I realize that the operations required by DeLight are not well supported by current accelarators (and they probably won't be in the near future), because memory shuffling is very expensive on these devices. And that's why most of the significance of this work would rely on custom kernels or even hardware. The results presented by the paper, though good, are not convincing enough for me to suggest that this customization is necessary.

[1] https://arxiv.org/abs/2002.11794

== UPDATE after rebuttal ==

I am willing to update my recommendation to weak accept after the rebuttal. Some of my main concerns remain, but added experiments, comparisons and discussions alleviate some of those. I still believe that there is too much focus on fairly useless metrics such as number of parameters which might lead to future work that will follow this trend, but the results are overall strong enough to warrant acceptance.

review_title: Less parameters is not necessarily better.

Meta-review: This paper presents some innovations to transformers allowing some significant reductions in parameter count. While some reviewers were concerned that the proposed innovations seem incremental and may not stand the test of time, all reviewers recommended acceptance after engaging in a rich and interactive author discussion. Given the clear importance of making transformers more efficient I think this paper will be of interest to the community and is worthy of acceptance at ICLR. 
Decision: Accept (Poster)