Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at reducing the complexity of BERT models to accelerate their inference by putting together a series of previously-introduced techniques. To this end, the authors propose to approximate softmax with a two-layer neural network, replace GELU with ReLU, fuse normalization layers with adjacent linear layers, and use knowledge distillation. It has been shown that the acceleration of 1.27x can be obtained on CPUs using the aforementioned four techniques while achieving a comparable accuracy performance w.r.t. the baseline model.
strength_and_weaknesses: Strengths: 

-- I believe this work is well-motivated. The paper targets a very important issue (i.e., reducing latency of inference) associated with large models such as BERT. 

-- The detection of the source of the latency overhead during inference is insightful as shown in Fig. 1. 

-- The amount of speedup during the inference on CPUs is interesting and shows the effectiveness of the proposed techniques.

-- The paper in general is well-written and easy to understand.

Weaknesses:
-- In general, the contribution of this paper is rather limited since each of those techniques have been previously-proposed.

-- It would have been great if the breakdown of cycles in Fig. 1 was shown for the sequence length of 128 to be compatible with experimental results in Table 2.

-- My major concern about this paper is the lack of comparison with prior works. First, there is no results for inference on GPUs. Second, there is no direct comparison with prior works such as Linformer? Where does the proposed method stand w.r.t. prior works? Finally, what speedup can be obtained when using MA-DistilBERT or MA-DistilRoBERTa?

-- The GELU and softmax functions can be approximated using polynomials similar to I-BERT with no impact on accuracy.


clarity,_quality,_novelty_and_reproducibility: This work puts together a series of previously-proposed works to reduce the complexity of BERT models. The paper is well-written, clear and easy to understand. I believe the results of this paper can be reproduced.
summary_of_the_review: In general, I believe the contribution of this paper is rather limited since the main method is a combination of previous works. On the other hand, the proposed method is simple and effective (when running on CPUs). My main concern is the lack of comparison with other works. It is not clear where this works stands w.r.t. prior works that also tries to reduce the complexity of BERT models such as Linformer and I-BERT. There is also no evaluation on GPUs while it has been mentioned as one of the motivation of this work in the introduction.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces 3 approximations to remove expensive non-linearities in transformers using cheaper matrix math and ReLU activation functions. Next, they perform a knowledge transfer between the trained BERT model and the approximated model to bring the accuracy up to standard in MA-BERT. The results show comparable accuracy while being cheaper to implement MA-BERT in hardware.
strength_and_weaknesses: Strengths:

* The techniques are simple to understand

Weaknesses

* The reasons as to why you can approximate the softmax using a 2-layer NN is unclear. For example, performer had clear justification why the projection was approximating the softmax. The argument in this paper is poor (universal approx. theory)
* No hardware results, but just software CPU implementation.

clarity,_quality,_novelty_and_reproducibility: Novelty: The techniques seem to be novel. However, I am not an expert in this area.

Clarity: The reasons why the approximations work is not clear. At least an intuitive explanation would have been good. Universal approx. theory based explanation is usually too sparse.
summary_of_the_review: I enjoyed the premise of the paper and Figure 1, which shows how much hardware cycles are spent in the non-linear computations. The techniques seem simple and easy to understand. However, it is unclear why they would work. The authors should explain why softmax can be approximated using a 2-layer FFN with ReLU. It is not clear to me. Citing universal approx. theory is not enough. For inspiration, look at performer or linformer papers.

I did not see ablations on the techniques suggested in the paper. Since, these approximations are not grounded in theory, I expect more ablations to show which technique mattered more and also why. 

I am not an expert in this area, however has a working knowledge base in it. The paper is simple, but lacks justifications for the choices made and hence my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Transformers are complex and require several compute-intensive operations. This paper proposes to simplify the execution of the BERT inference by using matrix arithmetic-only operations. The results show that the proposed method achieves considerable inference time reduction with relatively low accuracy loss compared to the baseline.
strength_and_weaknesses: Strengths:
1. The tackled problem is relevant to the ICLR community.
2. The contributions look solid.

Weaknesses:
1. It is recommended to evaluate the proposed method executed on AI accelerators, rather than only on CPUs.
2. The reason why the knowledge transfer is needed (instead of training from scratch the MA-BERT) is unclear. Please provide more details on it.
3. The results do not seem to provide a significant improvement compared to the related works.
4. It would be useful to provide the source code for reviewers' inspection during the rebuttal.
clarity,_quality,_novelty_and_reproducibility: Clarity: 5/10

Quality: 5/10

Novelty: 7/10

Reproducibility: 5/10
summary_of_the_review: Borderline paper where several concerns should be clarified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces an approach to improve inference speeds in transformer architectures, by distilling a trained model into an architecture with non-linearities removed. They find performance and speed benefits for MA-BERT compared to more traditional distillation approaches, and the authors have addressed the majority of author concerns. The paper will make a worthwhile addition to ICLR. The paper could be further strengthened by a more thorough comparison of MA-BERT versus standard distillation across different distilled model sizes, and by comparing MA-BERT to quantization (as suggested by 9tBM).
justification_for_why_not_higher_score: The paper could have benefitted from additional / stronger baselines.
justification_for_why_not_lower_score: Quoting: This paper introduces an approach to improve inference speeds in transformer architectures, by distilling a trained model into an architecture with non-linearities removed. They find performance and speed benefits for MA-BERT compared to more traditional distillation approaches, and the authors have addressed the majority of author concerns.
Decision: Accept: poster