Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose applying both grouped feed-forward layers and grouped convolutional layers to improve the computational efficiency of the transformer model.
main_review: The proposed architectural changes are simple but effective. The resulting model can be trained faster than the transformer baseline with the same MLM validation loss objective. However, the novelty of this paper is limited because many components such as GLU and convolutional layers have been proposed in the literature of efficient transformers. My major concerns are listed below and will consider raising the score if the major concerns can be addressed.

1. Comparison with the transformer baseline: As shown in Figure 1, it seems that the proposed GroupBERT is twice deeper than the transformer baseline. Since the transformer model scales very well with depth, it is also possible to reduce the width of the transformer and increase the depth to improve efficiency.
2. Empirical evaluation: There are no results on GLUE datasets. Moreover, since SQuAD v2 is more challenging than SQuAD v1.1, it is important to report the results on SQuAD v2 as well.
3. Comparison with prior arts: Figure 4a suggests that both prior arts achieve a worse computational cost and performance tradeoff compared to the transformer baseline. However, both papers claimed huge improvement over the BERT baseline. Why there is a gap between the results reported in this paper and the original paper. In addition, ConvBERT is a pre-work that should be compared to.
4. Number of groups: the authors claimed that group size 4 is optimal. Is it optimal for models with different sizes?

summary_of_the_review: In this paper, an efficient transformer structure is proposed to improve the efficiency of the transformer model. However, some empirical evaluations are missing and comparisons with prior art are not convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a set of modifications to the structure of the Transformer layer that improves FLOPS utilization. The proposed GroupBERT relies on grouped matrix multiplications and convolutions, being more efficient and superior in task performance and computation. 

The overall network architecture contains four modulesL a multi-head attention, a grouped convolution module, and two grouped feed-forward modules (GFFN). Even being twice as many modules, the total FLOP increases of about 60% when sparse grouped operations are used. 


main_review: Strengths:
- Evaluation on total training FLOPs vs. MLM validation loss looks reasonable. GroupedBERT is more efficient, that is agnostic to implementation or the hardware type. However, the training time vs. validation is not intuitive as grouped operation can hurt step time on a different hardware, such as a GPU or a TPU. 

Weaknesses:
- The ideals look trivial to the reviewer. Grouped FFN is not new and the structured sparsity formulation is not surprising. The hardware performance implication of the structured sparsity is not clear, as the paper is evaluated on a novel hardware accelerator. What are the performance implications when evaluating on GPUs?

- Grouped convolution has already been used in Evolved Transformer [1] and Primer [2]. The paper should cite the related work and compare with the related work. 

- This work demonstrates the performance on a unconventional Graphcore's Intelligence Processing Unit (IPU), which can benefit ops of lower computational intensity (like the Grouped FFN and convolution). The reviewer is not sure how useful the customized model is when the hardware becomes GPUs or TPUs. 

[1] The Evolved Transformer, https://arxiv.org/abs/1901.11117

[2] Primer: Searching for Efficient Transformers for Language Modeling, https://arxiv.org/abs/2109.08668

summary_of_the_review: Overall, the reviewer thinks the contributions are small. Grouped FFN and grouped convolution are not new. The paper lacks a comparison on MLM perplexity and downstream tasks with stronger baselines, such as evolved transformer and Primer where grouped ops are used. With respect to evaluation, only SQuAD is evaluated, which shows limited impact on generation tasks. The reviewer would suggest to evaluate on a wider set of down stream tasks such as GLUE and superGlue. 

A few more detailed comments:
- Try to add more stronger baselines of efficient transformers (Reformer, Linformer, Evolved Transformer, Synthesizer, etc.). 
- Add more tasks (GLUE or superGLUE)
- Target general accelerators such as GPUs. Then analyze training convergence time on a general processor.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new Transformer architecture. Compared the the traditional Transformer, the proposed architecture has two new features.:1) grouped FFN and 2) a convolution module. The grouped FFN is an efficient replacement of the original FFN. The convolution module is responsible for capturing local dependencies. Experiments demonstrate that the proposed variant achieves up to 2x efficiency gain in terms of both FLOPs and time-to-train.  
main_review: Strength:

The application of grouped FNN is well-motivated as two-thirds of the FLOPs are concentrated in the FFN module. It is natural to use grouped FFN to reduce the computation costs of FFN. 

The writing is clear and easy to follow. 

The proposed approaches achieve higher results than previous Transformer variants with grouped matrix multiplications.

Weakness:

Since previous approaches have applied grouped matrix multiplications in Transformer, the idea of GROUPBERT is not very novel to me. 

There are other efficient Transformer variants, like BERT quantification and tiny BERT with knowledge distillation. They can achieve massive speedups while the proposed approach achieves up to 2x speedups. I am wondering whether the proposed architecture can still boost the performance if applied on these extremely small networks. 

The comparison focuses on training/validation losses, rather than downstream results, which is not very convincing to me.  Previous methods usually report results on the GLEU benchmark to verify that their methods can achieve better performance on diverse NLP tasks. This work only reports results on SQUAD. It would be better to add more solid results on various downstream datasets. 
summary_of_the_review: The idea is well-motivated and easy to follow. It would be better to report more solid comparisons on popular GLUE datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Pre-trained language models have become a critical component in NLP applications. However, due to its large-scale pre-training computation, pre-trained models are usually too expensive to train. Therefore, in this paper, authors design GroupBERT, which introduces a set of modifications to the structure of the Transformer network. More specifically, they first apply grouped transformation to reduce the computations cost of dense feed-forward layers, and then use grouped convolution to enhance the learning of local and global interactions. Experimental results demonstrate the effectiveness of the proposed method. 
main_review: Strengths:
1. The writing of this paper is good and easy to follow.
2. This paper introduces an interesting grouped feed-forward network for efficient training.
3. Authors introduce the normalized position entropy to quantify the locality of an attention head is also interesting. 

Weaknesses: 
1. The novelty of this paper is limited. Apply convolution module into self-attention is not a novel idea, and it has been used in many works. For example, in NLP models, it has ConvBERT [a], and in CV tasks, it has [b][c]. What are the advantages or differences of GroupBERT when compared with these works? I think authors should provide these differences, not just compare original BERT. 
2. The comparison between GroupBERT and BERT is unfair. You should scale BERT to the same parameters (or the same computations in FLOPs) as GroupBERT for comparison. Otherwise, it cannot argue the contributions is from the proposed method or additional computation. And additionally, GroupBERT introduces too many hyper-parameters, like pre-norm and large learning rate.

Questions:
1. It seems the proposed architecture is hardware-specific. Is IPU necessary for model training and why not select GPU or CPU for testing? Specifically, since one of the targets in this paper is for efficiency (especially for server or embedded device deployment), it is necessary to measure the computations on CPU. 
2. Why not report the results on GLUE or SuperGLUE?
3. In addition to the comparison of MLM loss in the training stage, the results on downstream tasks are also very important, especially for pre-trained model. However, authors only compare BERT in downstream tasks (SQuAD v1.1), which lacks convincing. Some other works, including using CNN or for efficiency training, should be considered, like ConvBERT, SqueezeBERT and DeLight, not only the comparison in MLM loss. Besides, why not report SQuAD 2.0 results?
4. It seems the code link (https://github.com/neurips_authors/example) cannot be opened.
5. According to the ablation study in Table 1. it seems that the Conv contributes more benefits than grouped FFN modules. I am wondering how about the performance of the only transformer plus grouped convolution with the same parameters. 

[a] Improving BERT with Span-based Dynamic Convolution
[b] CvT: Introducing Convolutions to Vision Transformers
[c] Co-Scale Conv-Attentional Image Transformers
summary_of_the_review: This paper introduces GroupBERT, as an efficient architecture for pre-trained models. It designs grouped transformation and grouped convolution for efficient training. However, the experiments of this paper still lack convincing. Most of the experiments are only analyzed on MLM loss in the pre-training stage. However, the downstream task performance is an important metric to measure the quality of the pre-trained models, and this paper only selects SQuAD v1.1 to report performance, which lacks convincing. And the comparisons on downstream tasks only choose BERT as the baseline.
review_title: null

Meta-review: The authors propose modifications to the Transformer architecture in BERT by using grouped FFN and an additional convolution module.
The paper doesn't have all the results and comparison that should be done for a model that has seen similar architecture modification in the previous papers. While it is not necessary to show improvements on multiple hardware systems, it is important to see comparisons to more, stronger baselines and metrics on the full downstream GLUE eval rather than just Squad to establish improvements.
A reject.
Decision: Reject