Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper experimentally discusses the emergence of sparse activations in the feed-forward layer of Transformer neural networks. The authors highlight that sparse activations emerge for a variety of tasks and architectures. They then discuss how sparsity can be taken advantage of to reduce FLOP count and the implications of sparsity for robustness under different sources of noise.
strength_and_weaknesses: Strengths:
- Many experiments emphasize that this phenomenon happens for a variety of architectures and tasks. 
- The appendix offers some very good insights. The relevance of the optimizer and the residual connections are the first candidates that fall in mind when first reading the paper and could be emphasized/briefly mentioned in the main text as well.
- It is nice to see that depending on the task and the rank of the representation required, different sparsity patterns with layer index can be observed. 

Weaknesses:
- As mentioned, time-wall performance is not guaranteed apart from unbatched greedy decoding for larger models.
- Would be beneficial to discuss more (dis)similarities with sparsity in the attention matrix. I suspect similar results to Table 1 would come out if the Top-k operator was applied to the attention matrix.

Questions/Remarks:
- Would be interesting to see what happens when residual connections weights are learned, as e.g. in ReZero [1].
- The Role of the optimizer is discussed extensively in the Appendix. [2] might offer an alternative to train Transformers via SGD, which would be a valuable comparison.
- It is not clearly described what the legend in Figure 5 means. I guess (Train) indicates that the model was trained from scratch with a fixed k? If Top-k Transformer models are trained from scratch, it would be interesting to know if more steps are required to reach convergence.
- Studying the relationship with ResNets is not fair, as ResNets have batch normalization layers before the activations. Comparison with MLP should also include residual connections to be fairer.

[1]: Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." Uncertainty in Artificial Intelligence. PMLR, 2021.
[2]: Noci, Lorenzo, et al. "Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse." arXiv preprint arXiv:2206.03126 (2022).
clarity,_quality,_novelty_and_reproducibility: The paper is well written. Appendix includes all details to reproduce the results.
summary_of_the_review: All in all the paper offers some valuable insights on the activation sparsity in (Transformer) networks. Plethora of experiments demonstrate that this phenomenon is present across architectures and tasks. Claims regarded FLOP efficiency are briefly mentioned, but not adequately backed. A new Top-k layer is proposed, with limited insights given regarding drops in expected accuracy and expected gains in wall time inference. Robustness claims are also linked to previous work. The authors do make attempts to explain possible causes (mainly in the appendix). Contributions are incremental and not totally clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study sparsity in the MLP portion of transformers as a means to speed up inference time computation. They also show that sparsity enables robustness to input noise. 

strength_and_weaknesses: + Reducing the number of operations in a dense matrix multiplication makes sense as a route to speedups.
+ The writing is easy to understand.
+ Exhaustive empirical evaluation

- I'm not convinced by reading the paper that sparsity it the solution to more computationally efficient models. Right above section 3.3, the authors claim a 10% wall-time reduction. Is this significant? I wonder if it's worth retraining a model to be sparse for only a 10% wall-time reduction.
- I'm curious why the authors mainly studied the activations of the MLP portion of transformers and not so much the activation maps prior to the MLP.
- Section 3.3 contains interesting information but feels rushed and too short. I was hoping the authors could expand on how much sparsity improves robustness to noise and confidence calibration.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but I'm not sold on the novelty. I don't know what the takeaway of this paper should be. The authors have run many experiments but it's hard to decipher a core message.

summary_of_the_review: I am on the fence about accepting this paper. It contains a lot of interesting experiments but is lacking a strong message. I hope the authors can address some of the weaknesses: I am open to increasing my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work shows that sparsity is prevalent in transformer models, including T5 model for NLP and ViT model for vision with extensive experiments. Such a discovery might be suggesting that the law of parsimony is playing a role in Transformers even though they are not explicitly designed so. Furthermore, a top-k transformer is proposed by using top-k thresholding, which achieves good generalization.
strength_and_weaknesses: Strength:

(1)  The authors show the natural emergence of sparse activation in commonly used Transformer models.

(2)  Inspired by the experiments on sparsity, a top-k transformer is proposed to obtain good performance and generalization.

(3)  Extensive experiments are conducted to support their arguments.


Weaknesses:

This work shows the emergence of sparse activation in Transformer models. I have some questions as follows. 

(1)  In Sec. 1.3, the authors state that they use ReLU as the activation function instead of GeLU.  However, what is the performance gap between ReLU and GeLU? If GeLU is employed, Does sparsity still exist? Furthermore, the authors also propose a top-k transformer. The authors should list the specific performance with the top-k transformer and the original GeLU for comparisons. 

(2)  The authors choose the top k value for the transformer. However, it might affect the data parallelism thereby reducing the inference speed. The authors should provide a reasonable comparison, e.g., throughput. 

clarity,_quality,_novelty_and_reproducibility: This work has high quality and high clarity. This work has originality in analyzing the sparsity of transformer models based on my knowledge. The results should be reproducible because the authors provide amounts of experiments.
summary_of_the_review: To sum up, the work shows the emergence of sparse activation in transformer models with amounts of experiments and proposes a top-k transformer based on this discovery, which is promising. Therefore, I tend to vote accept. Some experiments for reasonable comparisons should also be conducted.


-------- After rebuttal --------

I have read the authors' responses and other reviewers' comments. I keep my 'accept' rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study the role of sparsity in deep neural networks that are based on transformers. In particular, the authors show that for a trained transformer only a small percentage of the hidden neurons in Multi-Layer Perceptrons (MLP) are non-zero. Motivated by that they propose a variant of the transformer networks, for a MLP layer only the top k hidden neurons are kept. They conduct numerical experiments for two instances of transformer networks for images and one for natural language processing. When comparing their version with the base models, they show that this model achieves comparable metrics, when the top 128 neurons (k=128) are kept. They further investigate the robustness of this model by introducing noisy labels and adding noise to the inputs. Overall, this new model appears to be outperform the baseline in this noisy conditions.   
strength_and_weaknesses: Strengths:

(a) The work presents an interesting application that can be easily applied to all transformer based models.  
(b) The paper is in general well-written and easy to follow. 
(c) The authors conduct additional experiments on the robustness of the models. 

Weaknesses:

(a) From Fg. 6 I see that in quite a few cases using this "sparse" model does not lead to improvement in inference time. For instance for T5 Large all the numbers are negative. This undermines the main goal of this model, which was to improve running time. 
(b) On page 6, the authors discuss the possibility of using approximate algorithms for finding the top K values with sublinear running times in hidden dimension. However, it is not clear to me of the authors actually used this approximate method or not. 
(c) Minor comments/questions:
    (c1) Are the results in Fig. 1 for train set or test set? 
    (c2) Is dim in the legend of Fig. 4 the same as dff?
    (c3) What is dff in Fig. 5? 
    (c4) On page 9, change "many work..." to "many works..."


clarity,_quality,_novelty_and_reproducibility: The authors propose interesting insights into the sparsity patterns of  transformers. However, the contributions of the paper seems to be limited to only proposing to take the top k values in hidden layers of MLPs. In particular, from Fig. 6 we see that this does not necessarily lead to running time performance. Moreover, the in Fig. 5 we see that the baseline networks slightly outperform the proposed method in most cases. 
summary_of_the_review: Overall, the authors show intersting insights into the transformers sparsity patterns. However, I believe that the the current numerical results are not convincing to immediately adopt this variant.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work demonstrates the natural emergence of sparsity in commonly used Transformer models. 

This paper proposed  Top-k thresholding to enforce sparsity, which brings robustness of training with erroneous annotations, less sensitivity to input noise/perturbation, and better confidence calibration of the predictions.
strength_and_weaknesses: Strength:
(1)This paper observed the sparsity in Transformers and this sparsity is emergent without any explicit design.

(2) This paper found that sparsity in Transformers is a prevalent phenomenon, and can improve efficiency, robustness and calibration.

(3) This paper further presented Top-k Transformer, a simple modification to Transformer architecture that allows control on the sparsity level for all model inputs throughout training.

Weakness:
(1) This paper only shows sparsity on the classification task. Is the sparsity existing for downstream tasks? such as detection and segmentation.

(2) Tab 1 shows that Top-128 ViT is on par with ViT for natural accuracy while is significantly better for model robustness and calibration. How about the other Top-k results? (please move some experiments from the appendix). 

clarity,_quality,_novelty_and_reproducibility: This paper's motivation is based on statistical observation. The analysis is insightful and well supported by some experiments.
As T5 and ViT are public available models, this paper is reproducible.

summary_of_the_review: This paper observed the sparsity in Transformers and proposed Top-k thresholding to enforce sparsity. Based on the proposed method, the efficiency, robustness and calibration are improved. However, this paper only considered the classification task, and I'd like to see the sparsity existing in the downstream tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper experimentally discusses the emergence of sparse activations in the feed-forward layer of Transformer neural networks. This paper is mostly empirical and although it seems to have a limited degree of novelty, the reviewers do find some of the conclusions of the papers interesting. 

The various concerns raised by the reviewers were mostly addressed by the authors so overall, this is an interesting contribution and I recommend acceptance. Note however that there was little discussion among reviewers following the author's response. The decision was therefore mostly taken based on my own reading of the paper and reviews, and the feedback that was given by the authors (which I found to be mostly satisfying).

One aspect that I find lacking in the paper is the discussion of prior work. Some reviewers (including Reviewer urTS) provided some additional references that are relevant to the paper and the authors should discuss them in their revised manuscript. 
justification_for_why_not_higher_score: The novelty aspect of the contribution made in the paper is somewhat limited. This is mostly an empirical paper with some interesting results but it does not appear to be a breakthrough result that would justify an oral (or even a spotlight).
justification_for_why_not_lower_score: The reviewers all agree this paper is worth publishing.
Decision: Accept: poster