Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an approach to training large-scale sparse transformers by sparsifying the activations of linear layers in the network (feedforward and Q, K, V projections). The approach is particularly appealing because it doesn't appear to require any hardware-specific or low-level optimizations to be efficient since it sparsifies activations. A controller network produces a block-wise one-hot mask on the activations and weights for the subsequent layer are selected on the fly based on this mask. Gradients are propagated through this hard-selection operation via the straight-through Gumbel softmax estimator.

The authors are able to achieve fairly substantial inference time speedups via sparsification. They also apply the same approaches to sparsifying models that are tailored towards handling long sequences such as the Reformer with LSH attention.
main_review: Strengths:

1. The sparsification technique is simple and easy to implement and seems to work at scale.
2. The approach shows empirical results on multiple tasks and large-scale benchmarks.
3. Significant improvements in inference times.
4. The paper is well written and easy to understand.

Weaknesses:

1. The paper seems to lack comparisons to more traditional weight pruning-like strategies which also promise minimal loss in task performance while being faster at inference.
2. It would be interesting to see ablations that push for slightly stronger sparsity constraints and how it affects task performance.
3. While the proposed method doesn't require hardware-specific or low-level library optimizations, it appears that block sparsity or other kinds of structured sparsity may be optimized on specialized hardware in the near future such as in https://developer.nvidia.com/blog/exploiting-ampere-structured-sparsity-with-cusparselt/. A discussion about the compatibility of the kinds of block-wise activation sparsity used in this work and structured weight sparsity would be useful.

Overall comments:

- I'd encourage the authors to rethink the title of the paper since it isn't clear what "Enough" is in the context of model scaling or even along what axis you are scaling models. The title is potentially misleading into having readers think that it enables efficient training, while this is only helps at inference.

limitations_and_societal_impact: The authors do not discuss the limitations of their work and its potential negative societal impact. There is plenty of prior literature to engage with here since there has been plenty of discussions recently on the societal impact of large language models.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Update: I've moved my score from 7 to 8.

The paper first introduces a sparse feedforward layer, which uses the input at every timestep to decide how to zero out the hidden vector (the one right in the middle) of the MLP feedforward layer. It uses the gumbel-softmax trick and zeros out (N-1) out of every N activations. This zeroing out of each hidden vector allows us to not use some of the rows & column of the W_1 and W_2 matrices of the feedforward layer. 

The next method is the sparse QKV layer, where each input is compressed into a few smaller vectors that each do attention independently. The authors show that a convolutional layer can make this faster, without loss of accuracy, and also toss away the final output projection from their self-attention module.

The authors then apply a few off-the-shelf tricks to their transformer, such as reversible layers and replacing the enc/dec attention by attending to the concatenation of the enc/dec words.

The empirical results are very strong and show good performance on challenging tasks such as language modeling, summarization, and GLUE, while massively improving inference speed (training speed is not improved). 


main_review: Strengths:
The paper is well written, and the ideas presented are relatively simple to implement. 
The paper presents strong empirical results on hard tasks, with big inference speedups. 

Weaknesses:
I might be wrong about this, and I would be grateful if the authors corrected me in the author response period, but I believe that the methods shown in this paper mostly improve the speed of *unbatched* inference. If this is true than the authors should point it out in the paper, and if not then they should also show speed number of batched inference. 
If my assumption is correct it might slightly weaken the results of the paper, since for most models I believe batched inference is used in production. It's still a strong paper that I believe should be accepted, but this limitation should be pointed out, and maybe future papers will be able to address it. 


Missing Citations:
Line 222: "We therefore remove the encoder-decoder attention, but just concatenate the encoder representations before the decoder tokens". Authors must cite "Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation", He et al, which did exactly that. 

Nitpicks:
Line 27/100/and throughout the paper: "output layers"- consider referring to these as the 'output projection'. I feel like calling them "output layers" might confuse some readers. 
Line 103: " decoding speed is dominated by the execution cost of the feedforward block". Clarify that this is true only for shorter input sequences (the authors do mention this later in the paper but I feel it's important to also mention it in line 103). 
Table 2: Consider adding the final perplexity of each model to the table (I know you also have it in the figure but it would be nice to have it in the table).
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a variant of the Transformer layer that includes sparsified dense and attention layers. For FFN layers, the author introduces a controller layer that selects only a subset of hidden dimensions to be used in the computation. For the attention layer, The author proposes a multiplicative transformation to simulate the permutation of the input hidden states and applies a convolution layer on top of it. The author shows no accuracy drop with the proposed method and speedup in the decoding phase.


main_review: Overall, I believe the proposed method in the paper is practically useful and can speed up the Transformers. However, I still have some doubts about the motivation and rationale of the proposed methods. Detailed comments:

Strengths
- The paper is clear and easy to follow. I can understand the proposed method very easily.
- The author did thorough experiments to verify the effectiveness of the method, including end-to-end experiments on multiple datasets and ablation studies of each method.

Weaknesses
- The method used for the FFN layer is similar to the previous mixture-of-expert method. The argmax selection (i.e. controller) part in the proposed method can be analogized to the expert selection part in mixture-of-expert. The difference that the authors mentioned in the related work doesn’t convince me: In mixture-of-expert, all of the experts are also being trained and only a subset of experts will be activated during the decoding for each token.
- The convolution layer in the proposed attention layer doesn’t have any semantic meaning. The convolution over the sequence length dimension can be understood because neighboring words in a sentence often have a more semantic correlation. But the convolution over the S dimension doesn’t have this kind of semantic meaning. Please elaborate on the rationale for the convolution design choice here. In addition, the sequence length and S dimension are very different, why is the convolution window size for both dimensions set to be F?
I would like to see more results on the speedup. What’s the speedup on the training speed? What’s the speedup with inference on a sequence or a batch of sequences?

Other questions:
- I haven’t seen people use negative log perplexity before. Why don't use log perplexity or just perplexity?


limitations_and_societal_impact: Please refer to the weaknesses in main review section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work focuses on improving Transformer inference efficiency by redesigning the aforementioned architecture such that the matrix multiplications can be sparse at inference time. This sparsity is introduced in the dense layers that compute the query, key, and value, and in the feed-forward layer as well. This results in a much faster decoding speed when compared to dense transformers.
main_review: _Originality_: The methods proposed are novel, and are clearly distinguished from related work.

_Quality_: The changes to the Transformer architecture that are proposed in this paper are interesting and very useful. They allow for much faster decoding. However, it's not clear if there are differences in training time due to additional blocks, like the "Controller". Can the authors clarify this? Furthermore, I would like to see the proposed models compared to more baselines, for example, the Reformer architecture (in performance and decoding time).

_Clarity_: The paper is mostly well-written, although I found it a bit confusing at times, particularly in §3. For example, are C_1 and C_2 additional parameter matrices? The method described in §3.2 also deserves a more thorough explanation.

_Significance_: As mentioned, this work proposes a Transformer variant in which the decoding speed is greatly increased, at little cost in performance. This result is very useful for practioners and researchers with varying ammount of computing resources, making it a valuable contribution to the community. The authors mention in the conclusion that the proposed technique can help to fine-tune a large Transformer "a single workstation". Can the authors clarify? Isn't your method only faster at inference time?
limitations_and_societal_impact: The potential negative societal impact is adequately addressed.
review_title: null

Meta-review: The paper introduces a technique for improving the inference speed of transformers by sparsifying all the linear layers. During training, a controller module predicts a sparse mask for the activations (trained with a straight-through Gumbel softmax), and during decoding this mask is used to prune the weights for the linear layer. Reviewers appreciated that the method is simple, and dramatically improves the speed of decoding. Reviewer vUEu raises reasonable concerns about the lack of comparison with mixture-of-experts approaches (which improve both training and inference speed), however the authors argue that their method is more expressive, and include an empirical comparison in their response. I think the title and abstract should be clearer about that the paper is addressing purely unbatched decoding speed. Overall, I recommend acceptance.
Decision: Accept (Poster)