GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
Authors: Ivan Chelombiev
Daniel Justus
Douglas Orr
Anastasia S. D. Dietrich
Frithjof Gressmann
Alexandros Koliousis
Carlo Luschi
Venue: ICLR 2022 Submitted (2022)
Keywords: Transformer
BERT
self-supervision
compute efficiency
sparsity
convolution
natural language processing

Abstract: Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requirements, due to long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. First, we rely on grouped transformations to reduce the computational cost of dense feed-forward layers, while preserving the expressivity of the model . Secondly, we add a grouped convolution module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-point operations (FLOPs) and time-to-train.

Under review as a conference paper at ICLR 2022
GROUPBERT: ENHANCED TRANSFORMER ARCHITEC- TURE WITH EFFICIENT GROUPED STRUCTURES
Anonymous authors Paper under double-blind review
ABSTRACT
Attention based language models have become critical component in state-of-the- art natural language processing systems. However, these models have significant computational requirements, due long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of Transformer layer, producing more efficient architecture. First, we rely on grouped transformations to reduce the computational cost of dense feed- forward layers, while preserving the expressivity of the model. Secondly, we add a grouped convolution module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-pots operations (FLOPs) and time-to-train.
INTRODUCTION
Deep neural networks have emerged as the leading solution enabling end-to-end language pro- cessing (Hochreiter & Schmidhuber 1997| Sutskever et al. 2014 Chung et al. (2014). Recently, the Transformer model based on the self-attention mechanism (Vaswani al.] (017)) has become the most promising architecture for language applications, especially when used as pre-trained foundation model (Devlin al., 2018 Radford et al. 2019 Brown et al. 2020 Bommasani et al.
2021). Attention based models are also increasingly showing promising results for established applications in domains different from natural language processing (Dosovitskiy et al. 2020).
Complementary to the Transformer's improved ability to model long-range dependencies in sequences is its superior potential to scale to larger sizes (Kaplan et al. 2020) and its suitability for execution on existing accelerators. This makes these models favoured over traditional recurrent language models. Given the increased computational demand of these models, there a growing and pressing interest to develop more efficient architectures (Strubell et al.) 2019). Some previous proposals were able to reduce the computational burden of the Transformer with improved task performance, but often with corresponding slower execution, will be discussed further in Section
We demonstrate a set of modifications to the the structure of the Transformer layer that improve FLOP utilization by the encoder stack. The proposed GroupBERT model relies on grouped matrix multiplications and convolutions, and delivers a more efficient version of BERT, superior in both task performance and computation. GroupBERT utilizes grouped operations in novel way which makes the model more FLOP efficient. However grouped operations are characterised by reduced compu- tational load for given memory access (Masters et al. 202T). i.e arithmetic intensity. This property would make them undesirable for traditional accelerators, which rely on large dense computations and reduced memory access. For this investigation we use the IPU (Jia et al., 2019) hardware, since its architecture uses on-chip SRAM for model execution, making it suitable to explore the potential of these efficient building blocks.
We achieve further task performance boost by increasing the depth of the model as GroupBERT extends each Transformer layer to contain four modules: one multi-head attention (MHA), one grouped convolution module, and two grouped ferd-forward modules (GFFN). The MHA and grouped convolution modules process token information along the sequence dimension, and each is followed by the general computation GFFN module. While there twice many modules in

---

Under review as a conference paper at ICLR 2022
GroupBERT
BERT
Lx
Convolution
Grouped FFN
Multi-Head Attention
Grouped FFN
Multi-Head Attention
LN
LN
1x1
1x1
1x1
Txt
4x
1x1
1x1
1xt
1x1
4%
1xt
OKV
2iet
GELU
DKV Core
GELU
1xt
1/4x
1xt
Grouped
Grouped 1xt
LN
1/4x
1x1
1x1
Tixt
Dense conv/matmul
Grouped conv/matmul
Non-conv operation
Figure 1: Schematic representations outlining the difference between GroupBERT and BERT struc- tures. A single GroupBERT layer has double the number of modules.
the proposed GroupBERT layer, the overall increase in computation is modest as we utilize sparse grouped operations, for total FLOP increase of about 60%.
Not only does GroupBERT deliver better performance per FLOP, but is also executed faster measured in total time-to-train. By employing both attention and convolution, the model has components dedicated to both short and long-range interactions, making more efficient use of the expensive attention mechanism. We also utilize the parameters of GroupBERT more efficiently during training, by discarding dropout for pre-training on a large corpus of text and by improving stability to higher learning rates. With all these innovations, GroupBERT Base is only slightly larger than BERT Base, yet it achieves better validation MLM loss than BERT Large using less than half of its FLOPs.
ARCHITECTURE
In this work, we propose an efficient modification of the Transformer encoder layer called GroupBERT. The original Transformer layer consists of modules: multi-head attention (MHA) and feed- forward network (FFN). Each of these modules also includes dropout, shortcut connection, and layer normalization (Srivastava et al.) 2014 He et al.) 2016 [Ba et al. 2016). GroupBERT includes four modules in every layer, as illustrated in Figure We add a convolution module in sequence with the MHA efficiently model local interactions between tokens and allow the attention mechanism to focus on long-range interactions. We then complement every sequence processing block with dedicated fully-connected module. For better efficiency, introduce grouped projections to the FLOPs intensive FFN module, making the layer structure more FL.OP efficient.
2.1 GROUPED FEED-FORW MODULES
The FFN module plays a crucial part in the unparalleled task performance Transformers (Dong et al. 202 Lee-Thorp et al. 2021). Although it is an essential complement to sequence processing modules it introduces a significant computational burden, as two thirds the FLOPs are concentrated in the FFN module. To make this integral part of the transformer more lightweight utilize structured sparsity in a form of sparsely grouped matrix multiplication. Consider a dense matrix multiplication of matrices H Baxb and W Roxc;
2

---

Under review as a conference paper at ICLR 2022
(HW), ky
hin
Wn.j
A sparsely grouped version of W corresponds to a block diagonal matrix W with G groups, matrix of similar dimension to W and a sparsity ratio of 1/ An equivalent altemaaive formulation of block-diagona matrix is grouped convolution for 1-dimensional eet al., (020). This reduces the number of stored parameters, and can be implemented efficiently without zero-multiplicttion
convolution (landola
h hin Wn.j
6/G Σ
HHW (TW(G)) Jass
Σ
w "+ " 1--1 E n 1-11 &J
We propose novel scheme to utilize grouped transformations in a transformer FFN module. Our first finding is that parameters in the expanding FFN matrix contribute more to task performance, and sparsity particularly damaging for these fan-out matrices. The second matrix is less sensitive to parameter reduction due to the sparse input and the reduction of projection dimension. Therefore, introducing sparsity in the second matrix results in Pareto efficient balance between compute and pes--eerformanee Our second finding is that the locality constraint of grouped projections the hidden dimension is detrimental to the model. However we remove this constraint by using an output linear projection. This is similar to the output projection matrix used in the MHA block, where each attention head acts only on a slice the hidden dimension. We find the optimal value for the number of groups be G 4, bringing the parameter count of GFFN be 75% of its dense counterpart, while also delivering a more Pareto efficient task performance (Fig 66. Alternative schemes of applying grouped transformations (landola al., 2020 Mehta et al., 2021 Jiang et al.)
(220) (Figure 2) in a transformer fail to outperform the baseline in setting of BERT pre-training
(see Section
BERT
DeLight
SqueezeBERT ConvBERT
GroupBERT
GELU
GELU
GELU
GELU
Figure 2:
Schematic diagram showing the comparison of FFN implementations of different trans- former variants that rely on grouped matrix multiplications.
2.2 CONVOLUTION BLOCK
Sequential locality plays an important role for contextualizing tokens in language models. At the same time, long-range interactions have proven to be vital for state-of-th -oee-o--thhtrrr art performance. Transformers inherently support long-range content-based based interactions via self-attention and usually incorporate form of positional encoding, allowing attention to also capture position-basee interactions (Dai al. 2019). Although this gives self-attention strong representational power, convolution is more efficient implementation of strictly local, position-based fusion. For this reason adopt dedicated convolutional module to improve overall efficiency.
The design of our convolution module is similar to Gulati et al. (2020), in which convolutions were in- troduced into speech recognition Transformer. We apply gate consisting of a pointwise convolution followed by Gated Linear Unit (GLU) that has been beneficial in language applications (Dauphin et al., 2017; Wu et al., 2019a] 2020). Unlike Gulati et al.) (2020), we use grouped convolutions in place of depthwise convolutions add representational capacity. We find that the best trade-off
3

---

Under review as a conference paper at ICLR 2022
0008
0.00
0.34
0.32
0.00
(a)
(b)
Figure Attention maps a single head from the first MHA module of BERT Base (a) and Base Conv (b) (Section 3.5), averaged over 10 validation sequences show the sequential locality preference of a typical attention head (see Appendix for all heads). Each row corresponds to a query token position, each column key position. Adding the convolution module encourages attention layers to learn long-range content-based interactions over short-range position-based interactions.
between task performance and computational cost is achieved by using a grouped convolution with group size 16 and kernel size 7, computed over the sequence dimension. The module also includes an additional layer normalization and a Swish activation (Ramachandran et al. (017).
With this module included, fewer attention heads show strong locality preference since such interactions are readily captured by convolutions. This effect is visible in the attention maps of Figure 3 showing weaker locality in the model that includes convolutions. To measure this effect quantitatively, we calculate the entropy across target positions for each head and source position. We then average, and normalize by the maximum possible value (see Appendix (C. For this measure, zero means that every head attends to a single position exclusively, while one means that every head position agnostic, although there could still be a joint position and content term. BERT Base has an average entropy ratio of 0. 05 and BERT Base + Cony has 0.92, indicating shift of positional fusion work from attention to convolution.
2.3 EFFICIENT PARAMETER UTILIZATION
In line with earlier research on the Transformer architecture (Wang et al., 2019 Liu et al. 2020 Xiong et al., (220)), we move layer normalization (Ba et al., (2066) from its position after the module's residual ("posinorm") ") to the first position within each residual block ("prenorm") While this modification does not directly improve task performance, it stabilizes training and allows the use of larger learning rate that would otherwise trigger the model with postnorm to diverge. We increase the learning rate by factor of 4x compared to the postnorm bascline.
Similarly to Lan et al. 2020), we find the use dropout to be detrimental to the pre-training stage. Due to the substantial size of the dataset, this kind of regularization is not required. While removing dropout yields improvements to the pre-tree mre-trannin loss, this does not apply to downstream tasks that include dropout only when fine-tuning on supervised
rely on smaller datasets. Consequently, tasks, that have smaller datasets than the pre-training corpus.
3 RESULTS
To evaluate the architecture modifications, we chose BERT (Devlin et al. 2018) pre-training and fine-tuning. The large dataset and challenging training objective mean that task performance improves consistently with model size (Lan et al.) 2020) and the risk of over-fitting is reduced. This makes it possible to clearly distinguish architecture modifications that benefit efficiency.
Our evaluation of GroupBERT for language representation learning shows that the architecture is:
1. Training Ff.OPeefficient across range of model sizes (Sections 3.3
2. Training time-efficient across range of compute budgets (Sections 3.3
3.4
4

---

Under review as a conference paper at ICLR 2022
3. Improved by each constituent part (Section 3.5
3.1 EXPERIMENTS
Each experiment consists of two pre-training phases and fine-tuning phase consisting of multiple training runs, started from the pre-trained model. All phases use the AdamW optimiser (Loshchiloy & Hutter 2019), with 81 0.9, .32 = 0.999, 10-6 The learning rate follows a linear warm-up decay schedule, whereby the warmup phase lasts for min(10", 0.1 total steps) steps, and the peak learning rate depends on the training phase and model size. The model defined over a vocabulary of 30522 WordPiece tokens (Wu et al. 2016). Weights are initialized using a truncated normal distribution of standard deviation 0.02. For all experiments we use Grapheore M2000 IPU systems.
Pre-training phase one optimises the Masked Language Model (MLM) and Next-Sentence Prediction (NSP) loss for corrupted sentence pairs. Masked and padded sequences of length 128 are grouped into batches of upproximately 512 sequences, with slight variations depending on the model size (see Appendix (A). The model trained for 10 epochs of Wikipedia BookCorpus (Zhu et al., 2015). corresponding to approximately 8. 8.10 optimisation steps. For all experiments with GroupBERT, baseline BERT and all other models tested, the learning rate is set to the value that produces the best validation loss. Pre-training phase two uses sequence length 384, 5 epochs, and approximately 2 2-10a optimisation steps.
SQuAD 1.1 fine-tuning (Rajpurkar et al., 2016 adds token span prediction layer and the whole model is fine-tuned to perform extractive question answering. Training uses target batch size 32 and we train for 2-3 epochs with various learning rates (Appendix B and report results for the best hyperparameters setting. We report F1 and Exact match scores, which show higher variance than MLM loss values. On the grounds of larger variance, we fine-tune each pre-training checkpoint five times using different seeds for every hyperparameter setting. Fine-tuning has been shown to be quite brittle process in recent studies (Dodge al. 2020; Zhang al., 202T] Mosbach al. 2021). In particular, many instabilities are caused by fine-tuning without using bias correction, an implementation that was adopted following the original experimental setup of BERT, This omission in the optimizer was observed to cause collapse of the training process. For this reason, we included a bias-correction term to the AdamW implementation for fine-tuning.
3.2 IMPLEMENTATION
In this work we leverage Graphcore's Intelligence Processing Unit (IPU) (Jia et al. 2019). as allows to examine wider variety of techniques, centered around improving computational efficiency. In particular the IPU can execute grouped operation efficiently. We train all models on Graphcore MK2 IPU clusters using a combination of pipeline model parallelism and data parallelism (Harlap et al. 2018). To make efficient use of compute resources, we increase cluster size and pipeline depth based on model size. We maximise the "compute" batch size executed by each node to achieve maximum throughput, and choose the pipeline gradient accumulation count to achieve a target "global" batch size (see Appendix A for more details).
All parameters, activations, and optimiser states are stored and processed in IEEE 754 half-precision floating-poinn point. This excludes the Adam weight update calculation and variance state, the loss calculation from logits and temporary variables that require higher precision. Higher precision temporary variables are used by layer normalization, softmax, and the accumulation of partial matrix multiplications. Our models are implemented in TensorFlow and are publicly available
3.3 PRE-TRAINING
BERT type models are notoriously difficult to evaluate as most of the computation is devoted to performing self-supervised pre-training on unlabeled corpora of text. As this process is the most costly one, we are primarily interested in evaluating solely this part of the model training separately. Taking example from other studies (Kaplan et al., 2020 Tay et al. 202T), we choose MLM loss on the validation dataset as the most salient criterion for task res-ppefformane generality.
captures language in its
Github: http:://hthhaa.cmm/morrr.a.tttte
5

---

Under review as a conference paper at ICLR 2022
2.6
2.2
GroupBERT
Medium
GroupBERT
Medium
Base Large
Delight
Large
2.0
SqueeZeBERT
MLM validation loss
2.2
MUM validation loss
2.0
1.8
1.6
Interovidd
Inbroved
0.49
1.4
1,4
5e+19
1e+20
2e+20
50
200
500
Total training FLOPs
Time (hours)
(a)
(b)
Figure 4: The GroupBERT model architecture is Pareto efficient on pre-training masked language modeling task. As well as comparing against the original BERT baseline, we tested transformers that use grouped transformations like SqueezeBERT and the uniform version of DeLight. For all models we tuned the learning rate to maximize their task performance. (a) Mean MLM validation loss + standard deviation plotted against the total pre-training FLOPs. (b) Mean MLM validation loss + standard deviation plotted against total time for pre-training.
In Figure we report the MLM loss measured the validation dataset for GroupBERT and range of other models of different sizes, including baseline BERT, DeLighT and SqueezeBERT. Evaluating task performance in the against FLOPs (Figure 4a) presents theoretical picture, which completely agnostic to implementation or type of hardware used. To complement the theoretical viewpoint of performance weighted against FLOPS, we also assess speed of execution, highlighting that GroupBERT additionally saves overall time-to-train (Figure (4b). Figure summarizes the practical benefits of using GroupBERT in terms of both FLOPs and training time to reach desired task performance. By plotting many model sizes of the same family, the efficiency of GroupBERT is clearly visible. The GroupBERT Base model outperforms baseline BERT Large, but uses only 47% of its resources and trains in half the time.
3.4 FINE-TUNING
Both the theoretical and the practical Pareto improvements achieved by GroupBERT on MLM evaluation loss translate the SQuAD fine-tuning task (Figures and Appendix Not considering
92
92
Toprowed
90
89
86
Small
Small
84
Base
Base
Large
Large
GroupBERT
GroupBERT
82
BERT
BERT
5c+19
Total training FLOPs
Time to train (hours)
(b)

---

Under review as a conference paper at ICLR 2022
84
85
84
SQUAD EM
SQUAD EM
83
80
82
Small
Medium
Medium
76
Large
GroupBERT
79
GroupBERT ERT
BERT
72
78
1e+19
2e+19
Se+19
1e+20
2e+20
20
50
100
200
500
(hours)
(c)
(d)
Figure 5: Median SQuAD and Exact Match scores + standard deviation of the GroupBERT and BERT model families plotted against the total pre-training FL.OPs (a,c) and against the total time pre-train (b.d).
the additional dropout in fine-tuning, the cost for training on downstream tasks for different model sizes is proportional to cost for pre-training, while few orders of magnitude smaller. Hence, FLOPs and time-to to-train measures of training cost are given for pre-training, as fine-tuning takes comparatively negligible amount of resources.
3.5 ABLATION STUDY
To verify the efficiency of the different model changes, we study their individual contribution the validation MLM loss improvement as well as their cost in terms of parameters and FLOPs (Table [].
Some of the ablation modifications that make add extra parameters that trigger divergence during model training when using the best learning rate. Therefore for the ablation study we use the BERT Base model with the next-best learning rate as the main comparison point, to allow the use of the same learning rate throughout.
We measure the performance of each modification with the Pareto improvement of the corresponding model, defined here as the difference in MLM loss with respect to the baseline models for the same FLOPs count. This is visually represented in Figure o by the vertical distance between the ablation point and the baseline interpolation line.
Table I: Ablation study of the contribution of different model modifications.
MODEL
PARAMETERS
Training FLOPS
LR
MLM LOSS
IMPROVEMENT
BERT Base
110.IM
6.1e19
2c-4
1.679
0.018
BERT Base
110.1M
6.1c19
1c-4
1.697
Prenorm
110.IM
6.1e19
8e-4
1.677
0.020
No Dropout
110.1M
6.1e19
le-4
1.651
0.046
Convolution
132.4M
7.3e19
lc-4
1.613
0.058
2 GFFNs
138.5M
7.6e19
le-4
1.660
0.005
2 GFFNs Conv
160.8M
8.8c19
1c-4
1.573
0.072
GroupBERT Base
160.8M
8.8c19
8c-4
1.516
0.126
By carrying out individual additive ablation, we are able to see the effect of every model modification in isolation. Furthermore, investigate the interaction of adding both the convolution module and two GEEN modules. We observe that every module that mixes token information gets significantly more efficient when used with matching GFFN module. While using two GFEN modules increases
7

---

Under review as a conference paper at ICLR 2022
the performance in line with the baseline model, using it in conjunction with convolution increases its Pareto improvement.
1.75
BERT Baseline
Ablation
Base
GroupBERT
1.70
Prenom
MLM validation loss
GFFN
No Dropout.
1.65
Cony,
1.60
Conv # 2k GFFN
Large
1.55
GroupBERT Base
1.50
4e+19
6e+19
8e+19
1e+20
1.Se+20
2e+20
Total training FLOPs
Figure 6: Ablation results showing the contribution of individual model modifications. Points where we changed the method of training have the same number of FLOPs as the original BERT Base model. Other improvement add some extra FLOPs. Ablation experiments were run with three different random seeds and all points include a standard deviation.
4 RELATED WORK
In language modelling, modern attention based architectures utilize most of their non-embedding parameters in dense operations, thus driving up the computational requirements of these models. Some studies have already considered the use of grouped operations in Transformer-X: (Dai et al. 2019): DeLighT (Mehta et al. 202T) and DeFINE (Mehta et al. 2020) use cascades of grouped transtormations in the encoder and the embedding, respectively, to lower the computational load of the Transformer. While sophisticated in their design, these models fail to generalize the setting of self-supervised learning massive amounts of data and do not provide compelling acceleration for the model execution. SqueezeBERT (Tandola et al.] 2020) replaces almost all dense multiplications with grouped transformations in Transformer layer. This approach shows significant speedup, but loses too much task performance if pre-trained without distillation from a dense parent model (Hinton) al. 2015). In this work, we present more balanced implementation of grouped transformations, delivering both performance and speed.
Grouped transformations have been prevalent in Convolutional Neural Networks (CNN), initially used for their parameter efficiency, starting with Alexnet (Krizhessky et al. 2012). They have been further popularized by the ResNeXt architecture (Xie al.) 2017). More recent CNN architectures implement depthwise separable convolutions, which are special case of grouped convolutions with group size equal to one (Howard al., 2017] Tan Le| 2019). In addition to being beneficial for resource utilization, the of grouped transformation convolutional neural network architectures (Ioannou et al. (007)).
found to be particularly well suited to
The Transformer model was initially introduced as parallelizable solution for supervised sequence-to- sequence learning in the field of natural language processing. These models have become ubiquitous with the of self-supervised approaches (Devlin et al.) [0088] Radford et al.) (019)). While attention based models have superior performance due to their ability to model long-range interactions, this characteristic is also what makes the attention mechanism costly, especially when used long
8

---

Under review as a conference paper at ICLR 2022
sequences. Many studies have successfully managed to make the attention mechanism more efficient (Tay et al., 2020] Fournier et al. 202I). However, its general use across the model can be redundant, as some attention heads in practice reduce to convolutions to model local interactions (Cordonnier et a 2020) and duplicate other attention heads, making them redundant (Michel et al. (019). Using convolutions directly inside Transformer has been shown to make attention more focused on long- range interactions. thus enabling more of its capacity to be used for that specific purpose (Wu et al., 2019b Jiang et al. 2020 Wu et al. 2020). While the incorporation of convolutions in Transformers was previously investigated only for smaller models with no more than 100M parameters, we look at scaling much wider range of model sizes, from 30M 500M parameters.
DISCUSSION
The improved practical efficiency of GroupBERT reduces the cost of model training. This provides access to better performing language models for broader range of users, both in the research community and for industrial applications. Moreover, the advantage of GroupBERT over the BERT baseline model family increases with model size, and therefore reduces the cost for training large and highly accurate language models. This observed scaling behaviour indicates the capability of GroupBERT to maintain material advantage for even larger language models. This can be valuable improvement for state-of-the-an research, relying on models that are increasing in size exponentially. With the latest breakthroughs using significant amounts of energy and hardware resources (Brown) 2020), efficiency improvements can translate into significant savings.
et al.
The BERT models represent not only step forward for the NL. research community, but also a tool heavily exploited in day-to-day operations of large corporate entities. Hence, improving the efficiency of these models that are used in production environments can potentially result in significant power savings worldwide, thus limiting greenhouse gas release into the atmosphere. Due to the same factor, all experiments that were carried out for this investigation were done using 100% renewable electricity, making this work carbon neutral.
In this paper we propose general technique for improving self-supervised Transformers. However, we only investigate its effectiveness on the English language, due the case of comparison with other studies using the same datasets. A potential ethical concern is the increasing dominance of language models trained on the English language, which may be detrimental to the efforts of preservation of endangered languages. Another limitation of this study is the use of the BookCorpus dataset as part of the pre-training process. Pre-training models this dataset was standard practice during the course of the investigation, but since then was found to be of questionable ethical value (Bandy & Vincent 1)
We have relied on structured weight sparsity to increase the efficiency of BERT and have created more diversity of building blocks within the supe-supe encoder by complementing the attention block with a convolution module. Our aim to further build upon these ideas, and construct models that even
general in their application different domains and that can be efficiently executed. To do so, we see potential to rely on other forms of sparsity, including dynamic weight sparsity (Evei al. 2019) and conditional activation sparsity, improving the capability handle multiple languages and data domains within the same architecture (Fedus et al. 2021).
6 CONCLUSION
In this study, we present GroupBERT: an enhanced Transformer architecture that is shown to be more efficient alternative, with up to 2.1x efficiency gain in terms of both FLOPs and time-to-train. We achieve these improvements by adding dedicated grouped convolution module to every layer and using grouped transformations reduce the density of fully connected layers. The proposed model achieves better results on both pre-training and fine-tuning tasks, and applies to a wide range of model scales.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: orXivv/607.000 0e.///07.065555 06450 [stat.MI.], 2016.
Jack Bandy and Nicholas Vincent. Addressing "Documentation Debt" in machine learning research: retrospective datasheet for BookCorpus. arXiv preprint arXiv:2105.0524) [cs. CLs, 2021.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu. Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gcoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.0722 2021. URL httts:://rxxv..or org/ wwt./b/////aooooooooooooooos /t.bo//2100 200.002500822
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165 CL), 2020.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 [cs.NE], 2014.
Jean- SanttBpt Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In 8th International Conference Learning Representations, ICLR 2020, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-Xi XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In 34th International Conference on Machine Learning, ICMI. 20/7 pp. 933-941, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11X.::1100000803 (cs.CL,, 2018.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprinl arXi::2002.06305 [cs. CLJ, 2020.
Yihe Dong, lean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103.03404 [cs.LG], 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.111 1999 [cs. CVJ, 2020.
Utku Evci. Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. arXiv preprint 19X1::1911.1113. [cs.LG], 2019. URL http://
arxiv. hwt.vorrgabbs.
10

---

Under review as a conference paper at ICLR 2022
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03661 [cs.LG], 2021.
Quentin Fournier, Gaétan Marceau Caron, and Daniel Aloise. A practical survey on faster and lighter transformers. arXiv preprini arXiv:2103. 14636 [cs.LG], 2021.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang. Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmentt transformer for speech recognition. arXiv preprint arXiv:2005.08/ 00000000000//.0 [eess.AS]. 2020.
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http://t:///axi org/abs/ 1503. 02531
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1780, 1997.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1eXXiv::004.000
CVs, 2017.
Forrest landola, Albert Shaw, Ravi Krishna, and Kurt Keutzer. SqueezeBERT: vision teach NLP about efficient neural networks? In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 124-135, Online, November 2020. Association for Computational Linguistics. doi: 10. 10018863/11/2222ssstttit 1.17. URL https
What can computer
//www. wwww.aclwwe.. httpa/////..aaaoor////////aaaaaa
Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving CNN efficiency with hierarchical filter groups. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 5977-5986, 2017.
Zhe Jia, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza. Dissecting the graphcore IPU architecture via microbenchmmarking arXiv preprint 191.02191 [cs.DC), 2019. URL http:// http:://aaxxv.coom wttp://.rrg/bb///11222
Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Con- vbert: Improving bert with span- based dynamic convolution. In Advances in Neural Information Processing Systems 33, NeurlPS 2020, pp. 12837-12848, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20X1:2000100866/ [cs.LG], 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1106-1111 1114, 2012.
Zhenzhong I Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. FNet: Mixing tokens with Fourier Transforms. arXiv preprint 1xXiv:2105.03822 [cs. CI.), 2021.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training Transformers. arXiv preprint 40X4:22004.08249 [cs.LG), 2020.
11

---

Under review as a conference paper at ICLR 2022
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL httph::tt/////t http:sss//ppeeeviee.eeeeeeee net. formm?formm??mm
Bkg6RiCqY7
Dominic Masters, Antoine Labatie, Zach Eaton-Rosen, and Carlo Luschi. Making EfficientNet more efficient: Exploring batch-independent normalization, group convolutions and reduced resolution training. arXiv preprint orXiv:2106.03640 [cs.LG], 2021.
Sachin Mehta, Rik
Mohammad Rastegari, and Hannaneh Hajishirzi. DeFINE: Deep factorized input token embeddings for neural sequence modeling. In 8th International Conference on Learning Representation, ICLR 2020, 2020.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. DeLight: Deep and light-weight transformer. In 9th International Conference Learning Representations, ICLR 2021, 2021.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32, NeurlPS 2019, 2019.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. In 9th International Conference Learning Representations, ICLR 2021, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. URI. wttpt:tt///..mm /ttp/aawvvv///////aa//e.e...aavr wttp://..wggba/b/bbbab// 05250
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 [cs.NE], 2017.
2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15()):1929-195, 2014.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 3645-3650, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks arXiv preprint arXiv: 1409.3215 [cs.CL]. 2014.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In 36th International Conference on Machine Learning, ICML 2019, pp. 6105-6114, 2019.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.061: 06732 [Cs. 2020.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint 77Xiv:1706.03762 [cs. CL), 2017.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint arXiv: 190.01906 [cs. (cs.CLL),001 2019.
12

---

Under review as a conference paper at ICLR 2022
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.104: [cs. CLS, 2019 9a.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In 7th International Conference on Learning Representa- tions, ICLR 2019, 2019b. URL https::////.......t. htttps://ppenreeeee.... /ttp///ppeeeeeeeeeeeeeeeeeeeeeee neteeewwwwwweeeeeeeee.e.. forum? 1id-skkkhhhhhc
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 08144 [cs. CL.), 2016.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite Transformer with long-short range attention. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pultern Recognition, CVPR 2017, pp. 1492-1500, 2017.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the Transformer architecture. In 37th International Conference Machine Learning, ICML 2020, pp. 10524-10533, 2020.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Weinberger, and Yoav Artzi. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations, ICLR 2021, 2021.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasu, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.0677 [cs. CVJ, 2015.
13

---

Under review as a conference paper at ICLR 2022
A EXECUTION SCHEME
In pipeline parallelism, a model is divided between multiple accelerators. Pipeline parallelism is form of model parallelism where execution is decomposed into a sequence of stages. At given instant, every stage is processing different and independent partition of the model, before passing it on the next stage. For the majority of our experiments we split the model between four chips, three for encoder layers and one for embedding, projection and loss. However GroupBERT Large has larger memory footprint, making an eight-chip pipeline the most efficient for execution. To create computational parity between all models, we replicate four-chip pipelines twice for data parallel training, resulting in eight chips being used for all experiments.
In the context of pipelined training, we distinguish between compute batch size and global batch size. The general formula for global batch size is:
Global batch size Replicas Accumulation factor Pipeline depth x Compute batch size
Here the compute batch size is the largest portion of the global batch at every pipeline stage, which is always maximized for efficient resource utilization. The pipeline depth given by the total number of stages in the forward and backward passes. Therefore, the accumulation factor is the only variable independent from the model structure and pipeline layout, which we can change to get a desired global batch size. We target the use of global batch size 500 + 20 for all experiments. This value is not exact and has to vary slightly for different model sizes, as the pipeline layout dictates the exact global batch size that can be used. However, we have not observed any significant effect these variations on task mass-performance.
Table 2: Overview of the different model sizes used in this study, with the number of parameters used for the BERT baseline model and our GroupBERT architecture.
MODEL SIZE
LAYERS
HIDDEN SIZE
PARAMS BERT
PARAMS GROUPBERT
Small
512
29.1M
37.6M
Medium
41.7M
56.9M
Base
768
110.1M
160.8M
Large
24
1024
336.2M
515.5M
Table 3: Pre-training hyperparameters for both BERT and GroupBERT model families.
FAMILY
HYPERPARAMETER
SMALL
MEDIUM
BASE
LARGE
Learning
4e-4
2e-4
2e-4
le-4
BERT
Batch size
512
480
480
512
Learning rate
3e-3
1.5e-3
8c-4
4e-4
GroupBERT
Batch size
512
480
480
480
14

---

Under review as a conference paper at ICLR 2022
B FINE-TUNING RESULTS AND HYPERPARAMETERS
Table 4 contains the SQuAD v1.1 fine-tuning results produced from the validation dataset. When fine-tuning GroupBERT models, we always used batch size 32 + 2 (for the reasons outlined in Appendix A). trained for either 2 or 3 epochs, with learning rates being one of the following: (le-4, 55-4, 2c-4, 3c-4, 4o-4). Each model required a sweep identify the best candidate. The sweep for the baseline BERT models was performed according to range of hyperparamters specified in Devlin
al. 2018).
Table 4: SQuAD v1.1 results, F1/Exact match %.
SCORE
SMALL
MEDIUM
LARGE
BERT
73.2 81.9
79.0/ 86.4
81.6 88.9
84.2 90.8
GroupBERT
76.8 84.7
80.9
83.5 90.2
85.5
C ATTENTION MAPS
To visualise the effect of introducing a convolution block into the model, we study the attention maps of each head. An attention map shows the softmax weight between every pair of positions in the sequence. In order to remove the effect of content and focus on position, we average the attention map over 10 validation sequences. We show these maps in Figures 7 g and generated after pre-training phase one.
To quantify the locality of attention head, we define the normalized positional entropy:
H(a) Log ==1
aij log Cij.
where dzy is the attention weight between source position i and destination and L is the sequence length. We average the normalized positional entropy over all layers and heads to get single metric describing a trained model. This measures to what extent attention maps are spread positions. Results for Base-sized models are given in Table S The models were trained for the ablation study of Section 3.5 showing the effect of individual components on entropy.
over different
Table Average normalized positional entropy for the models of Section 3.5 Models containing convolutions, Conv and GroupBERT, show higher entropy.
MODEL
NORMALIZED ENTROPY
BERT Base
0.75
Conv
0.92
GroupBERT Base
0.89
15

---

Under review as a conference paper at ICLR 2022
L1o
#7
97
57
7
3
ho
fn
hz
ha
h4
hs
h6
fn
hs
h9
h10
h1
Figure 7: BERT Base attention maps, for all layers L0-L11- Heads fo--t11 in each layer are ordered by positional entropy, Equation 3 Drawn with value range of [0, 0.1] and gamma of 1/3.
16

---

Under review as a conference paper at ICLR 2022
L1
L30
3
9
5
5
5
5
9
ho
fn
hz2
ha
ha
hs
h6
hn
h8
h9
h10
h1
Figure 8: BERT Base Conv attention maps, generated and drawn as per Figure
7
17

---

Under review as a conference paper at ICLR 2022
17
L1
3
5
5
LS
g
ho
fn
hz
ha
ha
hs
h6 hn
h8
h9
h10
h1
Figure 9: GroupBERT Base attention maps, generated and drawn as per Figure
7
18


Reviews:
[{"review_id": "stPn5TpQR4u", "reviewer": "Reviewer_EcxR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The writing of this paper is good and easy to follow.\n2. This paper introduces an interesting grouped feed-forward network for efficient training.\n3. Authors introduce the normalized position entropy to quantify the locality of an attention head is also interesting. \n\nWeaknesses: \n1. The novelty of this paper is limited. Apply convolution module into self-attention is not a novel idea, and it has been used in many works. For example, in NLP models, it has ConvBERT [a], and in CV tasks, it has [b][c]. What are the advantages or differences of GroupBERT when compared with these works? I think authors should provide these differences, not just compare original BERT. \n2. The comparison between GroupBERT and BERT is unfair. You should scale BERT to the same parameters (or the same computations in FLOPs) as GroupBERT for comparison. Otherwise, it cannot argue the contributions is from the proposed method or additional computation. And additionally, GroupBERT introduces too many hyper-parameters, like pre-norm and large learning rate.\n\nQuestions:\n1. It seems the proposed architecture is hardware-specific. Is IPU necessary for model training and why not select GPU or CPU for testing? Specifically, since one of the targets in this paper is for efficiency (especially for server or embedded device deployment), it is necessary to measure the computations on CPU. \n2. Why not report the results on GLUE or SuperGLUE?\n3. In addition to the comparison of MLM loss in the training stage, the results on downstream tasks are also very important, especially for pre-trained model. However, authors only compare BERT in downstream tasks (SQuAD v1.1), which lacks convincing. Some other works, including using CNN or for efficiency training, should be considered, like ConvBERT, SqueezeBERT and DeLight, not only the comparison in MLM loss. Besides, why not report SQuAD 2.0 results?\n4. It seems the code link (https://github.com/neurips_authors/example) cannot be opened.\n5. According to the ablation study in Table 1. it seems that the Conv contributes more benefits than grouped FFN modules. I am wondering how about the performance of the only transformer plus grouped convolution with the same parameters. \n\n[a] Improving BERT with Span-based Dynamic Convolution\n[b] CvT: Introducing Convolutions to Vision Transformers\n[c] Co-Scale Conv-Attentional Image Transformers", "summary_of_the_paper": "Pre-trained language models have become a critical component in NLP applications. However, due to its large-scale pre-training computation, pre-trained models are usually too expensive to train. Therefore, in this paper, authors design GroupBERT, which introduces a set of modifications to the structure of the Transformer network. More specifically, they first apply grouped transformation to reduce the computations cost of dense feed-forward layers, and then use grouped convolution to enhance the learning of local and global interactions. Experimental results demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "This paper introduces GroupBERT, as an efficient architecture for pre-trained models. It designs grouped transformation and grouped convolution for efficient training. However, the experiments of this paper still lack convincing. Most of the experiments are only analyzed on MLM loss in the pre-training stage. However, the downstream task performance is an important metric to measure the quality of the pre-trained models, and this paper only selects SQuAD v1.1 to report performance, which lacks convincing. And the comparisons on downstream tasks only choose BERT as the baseline. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lGRZvXy5vc", "reviewer": "Reviewer_s6Mm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed architectural changes are simple but effective. The resulting model can be trained faster than the transformer baseline with the same MLM validation loss objective. However, the novelty of this paper is limited because many components such as GLU and convolutional layers have been proposed in the literature of efficient transformers. My major concerns are listed below and will consider raising the score if the major concerns can be addressed.\n\n1. Comparison with the transformer baseline: As shown in Figure 1, it seems that the proposed GroupBERT is twice deeper than the transformer baseline. Since the transformer model scales very well with depth, it is also possible to reduce the width of the transformer and increase the depth to improve efficiency.\n2. Empirical evaluation: There are no results on GLUE datasets. Moreover, since SQuAD v2 is more challenging than SQuAD v1.1, it is important to report the results on SQuAD v2 as well.\n3. Comparison with prior arts: Figure 4a suggests that both prior arts achieve a worse computational cost and performance tradeoff compared to the transformer baseline. However, both papers claimed huge improvement over the BERT baseline. Why there is a gap between the results reported in this paper and the original paper. In addition, ConvBERT is a pre-work that should be compared to.\n4. Number of groups: the authors claimed that group size 4 is optimal. Is it optimal for models with different sizes?\n", "summary_of_the_paper": "In this paper, the authors propose applying both grouped feed-forward layers and grouped convolutional layers to improve the computational efficiency of the transformer model.", "summary_of_the_review": "In this paper, an efficient transformer structure is proposed to improve the efficiency of the transformer model. However, some empirical evaluations are missing and comparisons with prior art are not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gCxelUm7m-w", "reviewer": "Reviewer_14JM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe application of grouped FNN is well-motivated as two-thirds of the FLOPs are concentrated in the FFN module. It is natural to use grouped FFN to reduce the computation costs of FFN. \n\nThe writing is clear and easy to follow. \n\nThe proposed approaches achieve higher results than previous Transformer variants with grouped matrix multiplications.\n\nWeakness:\n\nSince previous approaches have applied grouped matrix multiplications in Transformer, the idea of GROUPBERT is not very novel to me. \n\nThere are other efficient Transformer variants, like BERT quantification and tiny BERT with knowledge distillation. They can achieve massive speedups while the proposed approach achieves up to 2x speedups. I am wondering whether the proposed architecture can still boost the performance if applied on these extremely small networks. \n\nThe comparison focuses on training/validation losses, rather than downstream results, which is not very convincing to me.  Previous methods usually report results on the GLEU benchmark to verify that their methods can achieve better performance on diverse NLP tasks. This work only reports results on SQUAD. It would be better to add more solid results on various downstream datasets. ", "summary_of_the_paper": "The authors propose a new Transformer architecture. Compared the the traditional Transformer, the proposed architecture has two new features.:1) grouped FFN and 2) a convolution module. The grouped FFN is an efficient replacement of the original FFN. The convolution module is responsible for capturing local dependencies. Experiments demonstrate that the proposed variant achieves up to 2x efficiency gain in terms of both FLOPs and time-to-train.  ", "summary_of_the_review": "The idea is well-motivated and easy to follow. It would be better to report more solid comparisons on popular GLUE datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LWJtErM4u-x", "reviewer": "Reviewer_8smw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Evaluation on total training FLOPs vs. MLM validation loss looks reasonable. GroupedBERT is more efficient, that is agnostic to implementation or the hardware type. However, the training time vs. validation is not intuitive as grouped operation can hurt step time on a different hardware, such as a GPU or a TPU. \n\nWeaknesses:\n- The ideals look trivial to the reviewer. Grouped FFN is not new and the structured sparsity formulation is not surprising. The hardware performance implication of the structured sparsity is not clear, as the paper is evaluated on a novel hardware accelerator. What are the performance implications when evaluating on GPUs?\n\n- Grouped convolution has already been used in Evolved Transformer [1] and Primer [2]. The paper should cite the related work and compare with the related work. \n\n- This work demonstrates the performance on a unconventional Graphcore's Intelligence Processing Unit (IPU), which can benefit ops of lower computational intensity (like the Grouped FFN and convolution). The reviewer is not sure how useful the customized model is when the hardware becomes GPUs or TPUs. \n\n[1] The Evolved Transformer, https://arxiv.org/abs/1901.11117\n\n[2] Primer: Searching for Efficient Transformers for Language Modeling, https://arxiv.org/abs/2109.08668\n", "summary_of_the_paper": "The paper proposed a set of modifications to the structure of the Transformer layer that improves FLOPS utilization. The proposed GroupBERT relies on grouped matrix multiplications and convolutions, being more efficient and superior in task performance and computation. \n\nThe overall network architecture contains four modulesL a multi-head attention, a grouped convolution module, and two grouped feed-forward modules (GFFN). Even being twice as many modules, the total FLOP increases of about 60% when sparse grouped operations are used. \n\n", "summary_of_the_review": "Overall, the reviewer thinks the contributions are small. Grouped FFN and grouped convolution are not new. The paper lacks a comparison on MLM perplexity and downstream tasks with stronger baselines, such as evolved transformer and Primer where grouped ops are used. With respect to evaluation, only SQuAD is evaluated, which shows limited impact on generation tasks. The reviewer would suggest to evaluate on a wider set of down stream tasks such as GLUE and superGlue. \n\nA few more detailed comments:\n- Try to add more stronger baselines of efficient transformers (Reformer, Linformer, Evolved Transformer, Synthesizer, etc.). \n- Add more tasks (GLUE or superGLUE)\n- Target general accelerators such as GPUs. Then analyze training convergence time on a general processor. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose modifications to the Transformer architecture in BERT by using grouped FFN and an additional convolution module.
The paper doesn't have all the results and comparison that should be done for a model that has seen similar architecture modification in the previous papers. While it is not necessary to show improvements on multiple hardware systems, it is important to see comparisons to more, stronger baselines and metrics on the full downstream GLUE eval rather than just Squad to establish improvements.
A reject.
Decision: Reject