Group-Transformer: Towards A Lightweight Character-level Language Model
Authors: Sungrae Park
Geewook Kim
Junyeop Lee
Junbum Cha
Ji-Hoon Kim Hwalsuk Lee
Venue:  (2020)
Keywords: Transformer
Lightweight model
Language Modeling
Character-level language modeling

Abstract: Character-level language modeling is an essential but challenging task in Natural Language Processing. 
Prior works have focused on identifying long-term dependencies between characters and have built deeper and wider networks for better performance. However, their models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources. In this paper, we propose a lightweight model, called Group-Transformer, that reduces the resource requirements for a Transformer, a promising method for modeling sequence with long-term dependencies. Specifically, the proposed method partitions linear operations to reduce the number of parameters and computational cost. As a result, Group-Transformer only uses 18.2\% of parameters compared to the best performing LSTM-based model, while providing better performance on two benchmark tasks, enwik8 and text8. When compared to Transformers with a comparable number of parameters and time complexity, the proposed model shows better performance. The implementation code will be available.

GROUP-TRANSFORMER: TOWARDS A LIGHTWEIGHT CHARACTER- LEVEL LANGUAGE MODEL
Anonymous authors
Paper under double-blind review
ABSTRACT
Character-level language modeling based Transformer has brought great success by alleviating limitation of recursive operation. However, existing Transformer-based models require substantial computational resources, which hinders the usability of character-level language models in applications with lim- ited resources. In this paper, we propose lightweight model, called Group- Transformer, that factorizes the calculation paths by grouped embedding oper- ators. Additionally, Group-Transformer employs inter-group linear operators prevent performance degradation from the group strategy. With comparison ex- periments on about five times larger and the best performing LSTM-based models and compatible parameter size of Transformer-baset models, we show that Group- Transformer has better performance on two benchmark tasks, enwik8 and text8. Further experiments including ablation studies and qualitative analysis revealed that the proposed work contributes to the effective lightweight model for practical application. The implementation code will be avail lable.
INTRODUCTION
Character-level language modeling has become a core task in the field of natural language processing (NLP) such as classification (Zhang et al., 2015). sequence tagging (Guo al. 2019a). question answering (He & Golub 2016). and recognition (Baek et al., 2019 Hwang & Sung 2016), with its simplicity on generating text and its adaptability other languages. Along with the development of deep learning in NLP, using recurrent neural networks (RNNs) have been standard way to solve problem for many years. Recently, however, a new architecture, Transformer (Vaswani et al,, 2017]. have shown promise in addressing this problem and have achieved breakthroughs in general language modeling (Al-Rfou et al. 2019 Dai et al. 2019).
Though this technique has achieved incredible successes, it has led to the huge size of Transformer- based models due to building deeper and wider networks. Transformer-X (Dai et al. 2019) and GPT-2 (Radford et al., 2019). for instance, contain 277M and 1542M parameters, respectively. This trend toward a large size model for performance is not suitable for edge device applications, which require small memory sizes, such optical character reader (OCR) and speech to text (STT), and for auto-correction and auto-completion applications that need fast real-time responsiveness.
To tackle this issue, choosing an appropriately efficient strategy becomes more crucial, especially in the real-world application which requires not only good performance but a lightweight model. In this paper, we introduce lightweight transformer for character-level language modeling. Our method is one of the factorization methods in that it separates the standard linear layer in trans- former architecture using group-wise linear operation and makes sparse connectivity between linear transformations. The proposed model is referred to as Group-Transfommer since it is inspired by the group convolution approaches (Zhang et al., 2018, Sandler et al.) 2018) that have effectively compressed huge image processing models for usability mobile devices.
While the group strategy reduces parameters and calculations in the proposed modules, its mutually exclusive calculation for the multiple groups compromises performance, caused by the information loss of inter inter-group correlations. To compensate for this problem, we added two inter-grous opera- tions that share common feature over groups for the group attention layer and linking features in different groups for the group feed-forward layer. By modeling the inter-group information flows, Group-Transin Transforrnf becomes performant as well as lightweight.

---

We conducted extensive experiments two benchmark datasets, enwik8 and tex18, and found that Group- Transformer with 6M parameters outperformed all LSTM- models with under 35M parameters. Furthermore, Group- Trausforrnofrmer shows better performance when compared against Transformers with a comparable number of parameters. We provide further analysis to identify the contributions of our proposed modules in detail. To the best of our knowledge, Group-Transformer is the first attempt to build a lightweight Transformer with the group strategy.
2 RELATED WORKS
2.1 TOWARDS A LIGHTWEIGHT TRANSFORMER
Since Transformer has become a promising model for diverse NLP tasks, there have been attempts to improve its efficiency with two majority approaches. The first is to restrict dependencies between input tokens to reduce superfluous pair-wise calculations (Child et al.) 2019 Guo et al.) 2019b] Sukhbaatar et al., 2019a). The approach provides time efficiency during inference, but does not address the heavy parameterization of Transformer. The second approach is to develop a lightweight network architecture while maintaining the properties of Transformer. For example, (Ta et al. (2019) utilize quaternion algebra to build lightweight modules for Transformer. They also use the factorize the components of the embedding layer, but the expression power can be limited by the connection of factorized components based on the quaternion principle. Another such approach (Sukhbaatar et al. 2019b) combined the multi-head attention and point- wise ferw-forward layer to devise unified module with fewer parameters. Despite these attempts architectural changes, their models still struggle to provide a lightweight language model with nearly still 30M parameters. In this work, we describe a lightweight transformer with less than 10M parameters, which is extremely small when compared against previous character-level language models.
2.2 GROUP STRATEGY
A group strategy has attracted much attention recently to compress many large and deep state-of-the- art convolutional neural networks (CNNs) (Krizhevsky et al.) 2012, Szegedy et al.) 2015 Chollet 2017). For example, when the group strategy is applied to standard linear layer with a weight W R. the feature map is partitioned into G groups. G small linear layers where each holds weight w [B(1G))((O/G) leading to a significant parameter reduction. Although intuitively appealing, it has been reported that applying the group strategy to the model often leads to huge performance degradation, since the features in different groups cannot interact with each other. To overcome this problem, ShuffleNer (Zhang et al. 2018) proposed channel shuffle operation to make interactions between different groups. This kind of consideration has also been applied to recurrent neural networks (RNNs). Kuchaiev & Ginsburg (2017) proposed group-wise RNV as a special form of ensembled RNNs. But, they did not consider the interactions between different groups. Inspired by the ShuffleNer, Gao et al. (2018) combined the shuffling idea into the group-wise RNN and achieved promising results. this work, we adopt the group strategy and build the group-wire wise operations suitable for Transformer architecture
a result, the layer is replaced by
GROUP-TRANSFORMEE
Figure Ta shows the overall architecture of Group-Transformer. It consists of a group embedding (bottom grey box), which embeds character into grouped features, a group attention (yellow box), which contains attention modules to identify dependencies in the time domain, and group feed- forward layer (green box). which re-configures the grouped features. As can be seen, when an input character is given, Troup-Trranffomme converts the input into multiple group representation (blue dots and red dots), processes and merges them to predict the next character. Figure [b and show group-wise information flow (blue and red arrows), and inter-group information flow (grey arrow) in the sub-modules. Without the inter-group information flows, the grouped features are processed independently. We observed that group modeling ensures that the groups become aware of the others and prevents different groups hold the same information. The following subsections describe architectural details of the sub-modules and their relations.
2

---

Output
Group
Group 2
Group
Group 2
Linear
Lincar
Linear
Linear
Group forw-forwad
ReLU
ReLU
Concat
Concat
Multi head
Multi-head attention
Group atsention
attention
Linear
Linear
Linear
Lincar
Shuftlc
Linear
Linear
Linear
Linear
Linear
Group embbedding
Inpus character
Group
Group 2
Group
Group
(a) Transforanerrmer
(b) Group attention
(c) Group feed-forward
Figure 1: Architecture overviews of Group-Transfore and its sub-modules when the number of groups is two. The gray arrows show the information flow across the entire groups, and the blue and red arrows indicate the information flow for each group.
3.1 GROUP EMBEDDING LAYER
Group embedding layer identifies a set of embeddings represent a token. The idea representing sentence, word or even character using set of vectors can widely be found in many NLP models that embed input tokens by concatenating (or summing) its embedding and its sub-units' embed- dings (Verwimp et al. 20077] BBojanoskii et al.] 2017: Kim et al. [219 /hou et al.] Similarly, we assume single character to be represented with vector representations of groups, that is, [ee ..., UcG] where Ucg E IR 1 g S G. When a character is given, the group embedding layer retrieves a corresponding set of vectors and passes to the following group attention layer. Through this paper, describe process single time step.
3.2 GROUP ATTENTION LAYER
The attention mechanism identifies dependencies between features in the time domain and com- bines the information of them. It contains three steps; (1) identifying queries, keys, and values, (2) retrieving relative features at different times, and (3) transforming the attended feature into the input domain (Vaswani et al. (017). The main focus of this paper is to apply group strategy to the feature space of Transformer. Thus, we let the second step be identical to those of the original Transformer and focused on the first and the third steps.
Figure Tb explains the architecture of group attention. The multi-head attention module represents the second step, the under operations identify the queries for the first step, and the upper operations transform the attention output for the third step. We note that we do not represent the key and value for the multi-head attention block in the figure because they are possible to come from another source domain. The group attention processes the grouped features with intra-gros group operations (white boxes) and inter group operations (grey boxes).
Grouped Queries, Keys and Values
Let [N,,..., X be of input vectors where X, TR. Doop for the group g. Since the multi-head attention contains H attention modules for a single group, group attention first calculates query 4gh for group g and its head h as the below,
q gh = X, Wer-intra gh
Σ
X
Wra-inter 9'h
(I)

---

gh E P are linear weights to describe an intra-group group (white boxes) and an inter-group (grey box) combinations, respectively. In the formula, the first term on the right-hand side identifies a specific feature for the head h in the group g and the second term determines head-wise features that allow the grouped features to share common expression retrieving other features in a different time. When comparing with the fully connected linear layer over the groups, the approach restricts the connection between the groups, requires fewer number of parameters and calculations.
where Wa-iutr Po (Pawe/ R) and Wait gh
For the key Kghh and the value Vgh. we use fully connected layers by using all group pairs g' and gi Kgh Eg Xg Wi s'gh and Ygh- = -g X, gighh where W g'gh E R Pe (Dgnp/H) BB) and Wy g'gh T. Danu x (Dpunp/ H) are linear weights. As we mentioned, since the keys and the values can be defindd from the other source domain, we use the same formula of the original Transformer, pursuing the universality of the proposed module.
Multi-head Attention
The identified headed elements are used for connecting features in the time domain. In this step, position encoding /Vaswani al. 2017) has important role for the features to be aware of their position in an input sequence. In this paper, we apply the relative positional encoding, which de- scribes a long-length character sequence effectively. By following Dai al., (2019), we define the attention score map with the relative positional information and the attention mechanism determines the attended feature agh of the head h in the group g.
Combination of Multiple Heads
The multiple heads [agl, o.y {g ] in the group 9 are combined as the below:
Og Σ (agh Wo-iinra +
Σ aghh g'h
where Warinira (D xxx Dauus and Wrorinter E (Damae/ x are linear weights for combining intra-group and inter-group information, respectively. As can be seen, the final output is determined with specific feature from its own group and shared feature from whole groups. This step utilizes the same mechanism used to identify the queries with the same objective spreading group-wiss information from multi-headed attention all groups. These intra sirrp and inter-group modelings mainly contribute to reducing the number of parameters and calculations. Finally, the inputs Xg are added into the output Og as xg Xg + Og for residual connection.
3.3 GROUP FEED-FORE FORDARD LAYER
Group feed-forward layer re-configures the outputs of the attention module, Xg. by applying group- wise operation each position. Figure Ic shows the architecture of the proposed module. As can be seen, the groups are shuffled (grey box) and support each other. The group-wise features are processed with two linear transformations and one non-linear activation. As the original module does, the linear layers in our module transpose the input feature into a high dimensional space with non-linear activation and transform the output back into the input space.
The group feed-forward layer can be formally explained as follows. Given G input features [&] ..., xg], group feed-forward layer transposes the grouped features into high dimensional space as follows;
yg Ng w-llinnta
Σ
(3)
where Willinata ER Poop De and Wii-inee E E Denss Do are linear weights for mapping intra-group and inter-group information into the De-dimensional space, relatively bigger than gruu dimen- sion. Here, we introduce a low-rank matrix approximation on the inter-group transformation ma- trix wfll-inter Modeling interactions between the groups requires the x G numbers of weights, well the multiple weights for the group 9. transpose the group into the high dimensional space for the target group g". If designing a fully connected weight for all groups like the orig- inal Transformer, the forwad layer still holds heavyweights and expensive caleulations. To
9'g
4

---

reduce the overburden, we factorize the matrix Willimter into two matrices, W E R Pemup
and g'g i E RRMx inspired by Sainath et al. 2013 and Novikov et al. (2015). The newly introduced dimension M is smaller than D group and thus the number of parameters and calculation reduced proportionally with the ratio between M and D Dymup- In this paper, set M as Damupp /G to control the dimension relatively with the number of the groups. Interestingly, such matrix fac- torization can be modeled efficiently with a group-wise linear transformation and shuffle trick as shown in Figure Ic. Please refer to the appendix for the detail of the shuffle trick.
Finally, group-wise linear transformation applied upon the high-dimensional feature as follow;
yg
ReLU(y, )
where W E is a linear weight. For a residual connection, each grouped input feature is added into the output of the group feed- forward layer; y, = 8g + yg
3.4 RESOURCE REQUIREMENT
Here, describe the efficiency of Trans-Trrnsformme in view of the number of parameters and required computational costs. When considering the original transformer, its required numbers of parameters are 4 * O model) for its attention module (query, key, value, and output linear) and 2 O Dmodet) model) for its beed-forward module, where Dmodet is a bottleneck dimension. Group- Transformer pursues to reduce the number of parameters by splitting the hidden state into multiple groups and processing them group-wisely. When set the total dimension over groups as Dondel = D Dgouup * G, the resource requirements Group- Trau--rranformer become (2+ & (DDande)) for group attention and 3 * O(D D mmodet) for group feed-forward module. The number of G groups increasing, the resources decreasing. Appendix B provides the detailed required re- sources of all sub-modules and comparisons with those of the original Transformer.
EXPERIMENTAL RESULTS
4.1 DATASET AND EXPERIMENTAL SETTINGS
We demonstrate the efficiency of the proposed Group-Transformer with two popular benchmark datasets, emwik8 and text8. The enwik8 dataset contains 100M of English Wikipedia texts with 204 unique characters including alphabets, non-Latin and special characters. In comparison, the tex18 dataset provides 100MB of pre-processed texts only with 27 unique characters by filtering super- fluous content, such as tables, citations, and punctuation, and by replacing the non-Latin characters with spelled-out equivalents (i.e., "15" to "one five"). For a fair comparison with previous works, we used the training/dev/test splits defined by Mahoney (20TT) for both enwik8 and tex18.
Most of the experimental settings follow those Dai et al. (2019), where the difference lies in the hyperparameters that influence the size of the model. We set the number of layers L as 9, and we fixed the total size of feature Mmodel for a single character as 256 and the total numbers of heads as 8 while the number of groups are explored in {2.4). For the regularization of the model, we applied layer normalization (Ba et al. 2016)) independently of the group attention and the group feed-forward layer with the probability = 0.1. The length of the feed sequence was 512 with the cached 51 512-length for the previous sequence (Dai al. 2019). We use the Adam optimizer with a learning rate of 2.5e-4, 31 of 0.9, f3 of 0.999, a batch size of 22, the number of iterations of 400,000, and the best model on the validation set is chosen. The implementation code will be available for the other details.
groups and dropout layers upon the outputs
4.2 COMPARISON AGAINST PRIOR CHARACCER-LEVED LANGUAGE MODELS
We compare the Group- TrompTTTaaffrmerr against existing character-level language models using under 50M parameters in Table | The prior models are grouped according to their methodologies, includ- ing "LSTM," and "Transformer". We observe that the Group- Transformer outperforms the LSTM models with under 30M parameters and that the 2 against all prior I. STM models on the enwik8 dataset. When compared to Transformers, we observe
attains the best performance
5

---

enwik8
text8
Category
Model
Params
bpc
Params
bpc
Generic TCN Bai et al.
2018
5M
1.45
LSTM 1800 units (Mujika ct small HyperLSTM (Ha RHN depth 10 (Zilly et al. small mLSTM (Krause al
2017
14M
1.40
19M
1.35
LSTMs
21M
1.30
2016
22M
1.28
20M
1.59
FS-LSTM-4 (Mujkk et al. 2017
27M
1.28
HM-LSTM (Chung
20166)
35M
1.32
35M
1.29
A All-attention (Sukhbaatar et al.
2019b
39M
1.01
38M
1.11
Transformers
Transformer-XI.- 12L (Dai 2019)
2019
41M
1.06
T12 (AI-Rfou
44M
1.11
44M
1.18
Group-Transseiie Transformer (Transformer-) XL 9L)
8M
1.18
8M
1.27
Group-Transforme Transformer 91
7M
1.19
7M
1.26
Ours (small)
Truup-rranffomer 9L
4M
1.22
4M
1.30
Group-Transfommn 9L
3M
1.27
3M
1.34
Table Comparison with the prior character-level language models on enwik8 and tex18. We report bit- bit-per-character (bpc) for test sets as well as the number of parameters.
the Trou--Transfommers provide lower performances but use extremely lower number of parameters. The next section describes a fair comparison with Transformer in a lightweight setting.
4.3 COMPARISONS AGAINST LIGHTWEIGHT TRANSFORMERS
Here, we compare Group- Transforrns against Transformers with comparable number of param- eters and time complexity. In this experiment, we have explored the model settings of Transformer- XL. to identify the Transformer models with under 8M parameters. All models are trained by Adam with 3 of the learning rate through 200,000 iterations and the length of feed sequence is set to 384.
Class
Model
(L, Dmotel, Hmode)
# Params
FLOPs
bpc
Base
Transformer-XL 9L
(9, 256, 8)
8.1M
9.4B
1.224
Transformer-X XL. 9L
(9, 232, 8)
6.6M
7.8B
1.239
Transformer-Xi XL. 7L
(7, 264, 8)
6.7M
7.7B
1.254
[6M-8M]
Transformer-XL SL Traus-Trrnsformer
(5, 304, 8)
6.5M
7.2B
1.263
groups
(9, 256,
6.8M
6.7B
1.221
Transformer-X XI. 9I.
192, 8)
4.4M
5.6B
1.286
Transformer-N XI. 7L,
27,21, 8)
4.6M
5.4B
1.289
[4M-6M]
Transformer-XL. SL
(5, 256, 8)
4.6M
5.2B
1.304
Graup-Tran with 4 groups
(9, 256, 8)
4.3M
4.7B
1.261
Transformer- XL. 9L
(9, 152, 8)
2.9M
3.7B
1.336
Transformer-X XL 7L
(7, 176,7 8)
3.1M
3.7B
1.339
[Under 4M]
Transformer-Xe XL SL
(5, 208, 8)
3.1M
3.6B
1.340
Group- Transe with groups
(9, 256, 8)
2.9M
3.7B
1.316
Table 2: Comparison of lightweight Transformers that use under 8M parameters. L, Hmedel and D modde indicate the number of layers and heads, the hidden dimension over the model. The FLOPs indicates the number of calculations to generate 512 length of character sequence. We used D Dgrup D G and H Hmmdee/ G for Transforansssrmers set the same number of the total heads in the attention module.
Table shows the comparison results, focusing both on the performance and efficiency. At each class, tuned Transformer-LL to have the similar number of parameters when the number of layers are 5, 7, and 9. When compared to Transformers at each category, Group- Transformer shows better performance than the baseline models. In particular, the Group-Traassorme shows its

---

efficiency by using 82.1% % parameters and 71.3% FLOPs of the base Transformer, but provides similar performance. The Group-Transforee outperforms the second-bes best model with a large margin of 0.025. This is the result of inherent computational efficiency enjoyed by group operation and our careful design of the inter-group connections.
1.200
1.225
2160110)
1.225
(31,2310101
20
1.250
1.250
$400
1.275
1.275
x0
640
1.325
1.325
1.350
Depth (laye)) reduction
1.350
Deyth (layes) reduction
1.375
Widt (fatuue sise) seduction
1.375
(faiiee sire) roduction
Feature grouping
8 1.400
Featur groppnn
Number of parameteess (M)
Number of FLOP (B)
Figure 2: Performance comparison of three reduction methods from model parameters such as the number of layers ("I.",, the hidden dimension ("D"), and the number of groups ("G"). The FLOPS indicates the number of calculations to generate 512 length of character sequence.
The number of groups can be interpreted as hyppe--paametter affecting the model size. Figure 2 shows the effectiveness of three hyper-parameten, such as the number of layers, the size of hidden dimension, and the number of groups The default model used Transformer-XL. (Dai et al., 2019) Dmedet. and then we reduced the
with 9. Hmodel 8, Doodel = 256, and Dmodet three hyper-parameters. When making the model thinner or shallower, the performances of the model become worse, but the required resources are getting lower. When comparing ours with two reduction methods, the group strategy shows better performances than the models requiring comparable resources. This experiment proved that the feature grouping methods, the main idea of this paper, is more efficient to reduce the model size and the time complexity than tuning other model parameters.
4.4 ABLATION STUDIES ON PROPOSED MODULES
Traup-Transfommrr includes two modules utilizing group-wise operations and inter-group modeling. We conduct ablation studies to identify the contributions of the proposed modules and inter-group operations.
2 groups
groups
8 groups
# Param.
FLOPss
bpc
Param.
FLOPs
bpc
# Param.
FLOPs
hpc
Atten.
FF.
(A Parm.))
(A FLOPSs
(A bpa)
(A Param.)
(A FLOPs)
(A hpc)
(A Param.)
(A FL.0Ps
(A bpc)
Original Original
8.1M
9.4B
1.224
8.1M
9.4B
1.224
8.1M
9.4B
1.224
Ours
Original
7.8M
9.1B
1.236
7.1M
8.3B
1.247
6.6M
1.246
(0,3M)
(4.33)
(+0.012)
(-LOM)
(-11B)
(+0,023)
(-1.5M)
(-1.4B)
(+0.018)
Original Ours
7.1M
7.0B
1.221
5.3M
5.8B
1.251
4.1M
5.2B
1.284
(-100M)
(-2.4B)
(-0.003)
(-2.MM
(-3.6B)
(+0.027)
(-4.0M)
(4.2B)
(40.060)
Ours
Ours
6.8M
6.7B
1.221
4.3M
4.7B
1.261
2.9M
3.7B
1.316
(-1.3M)
(-2.7B)
(-0.003)
(-3.8M)
(-4.7B)
(+0.037)
(-5.2M)
(-5.7B
(40.092)
Table 3: Ablation study on the proposed modules, group attention and group feed-forward layer.
Table shows the module-wise impact on the number of parameters and performance. For a fair comparison, we set the baseline model to reduced Transformer-XI. (Dai et al. (019) of less than 8M parameters, and can gradually reduce the model size by replacing the attention and the feed- forward layer with Group-Trane Transforrnnermmer module selectively. When replacing the feed-forward layer with Proop-Trranffomee module, we observe that the number of parameters in all cases decreases more efficiently than replacing the attention module. Interestingly, when replacing both modules, the degradation is lower than the sum of the individual performance losses, but the of the individuals' reduces the required resources. This result demonstrates more efficiency of concurrently using both group-wise modules.
7

---

Ht
H2
H3
H44
HS
H6
HB
HI
HD
H3
114
116
HR
HI
H2
14
H5
H6
H7
HB
HI
H3
134
115
H1
H7
HI
HI
0.125
G2
0.050
0.025
in
H8
(a) 2 groups
(b) 2 groups (w/o inter)
(c) 4 groups
(d) 4 groups (w/o inter)
Figure 3: Similarity matrices of multi-head attentions. The black box indicates high similarity of attention patterns and the white box does the opposite. The red boxes represent groups of the mul- tiple heads. The similarity is measured based on the euclidean distance between attention weights over test sequence.
In addition this, we also investigate the influence of inter- gror-group operations our model. When the inter-group operations are removed (grey boxes in Figure [T and Ic]. we observed the performance degradation on Grrup-Transformn
by 0.028 bpc and 4- 4-Group- Transformer by 0.051 bpc. These gaps are relatively huge when compared to the performance gap between Transformer-Xi and Group Transformers Table 3 The results re-emphasize the importance of inter-group group modeling in Group- Transformer. Figure 3 shows the similarity patterns between the multi-head attention of our models ((a) and (c)) and the ablation models without the inter-gun inter-group operations ((b) and (d)). As can be seen, the multi-head attention map from the model without inter-group operations shows high similarities among different groups, while the proposed model shows the opposite. These similarity patterns imply that the model cannot fully take advantage of multi-head attention, which is designed to attend multiple positions of content, without the proposed inter-group operation.
5 CONCLUSION
Recently, remarkable progress has been made in character-level language modeling by Transformer. The advantage of Transformer lies in its effectiveness in modeling long-term dependencies between characters. However, the models have been developed with huge number of parameters, and the inference of them has required expensive computational cost. We argue that big models cannot be used limited computational environment. Group-Transformer has been developed to prove the effectiveness of Transformer in lightweight setting. We have grouped features and proposed croup-wise operations to reduce the number of parameters and time complexity of Transformer. In addition, to fully realize the advantage of the original Transformer, we have connected the groups to interact with each other. When applying Group-Transfome that Group- Transformer only with 6M parameters achieves better performances than LSTM-based models holding over 30M parameters. Further analysis has proved the effectiveness of the group strategy reduce computational resources.
on enwik8 and tex18, we found
REFERENCES
Rami Al-Rfou Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-lewe lan- guage modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3159-3166. 2019.
Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.0644 2016.
Jeonghun Back. Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Scong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model compar- isons? dataset and model analysis. In 2019 IEEE International Conference on Computer Vision (ICCV), 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.0127,, 2018.
8

---

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Rewon Child, Scott Gray, Alee Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint
2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrett neural net-
works. arXiv preprini arXiv: 1609.01 2016.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Tra:sformer-XI: Attentive language models beyond fixed-length context. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2999999 2988, 2019.
Fei Gao, Lijun Wu, Li Zhao, Tao Qin, Xueqi Cheng, and Tie- Liu. Efficient sequence learning with group recurrent networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long Papers), pp. 799-808, 2018.
Jinxi Guo, Tara N Sainath, and Ron J Weiss. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5651-5655, 2019a.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. arXiv preprint 1902. 09111....11,,......1111111.111.1 2019b.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetorrss. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Xiaodong He and David Golub. Character- question answering with attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1598-1607, 2016.
Kyuyeon Hwang and Wonyong Sung. Character-level incremental speech recognition with recur- rent neural networks. In 2016 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 5335-5339. IEEE, 2016.
Geewook Kim, Kazuki Fukui, and Hidetoshi Shimodaira. Segmentation-free compositional n-gram embedding. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3207-3215, 2019.
Ben Krause, Liang Lu, lain Murray, and Steve Renals. Multiplicative 1stm for sequence modelling. arXiv preprint
2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for LSTM networks. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
Matt Mahoney. Large text compression benchmark. URL: http://www. mattmahoney. net/text/text. html, 2011.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 5915-5924, 2017.

---

Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in neural information processing systems. pp. 442 450, 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high- gigh-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655- 6659. IEEE, 2013.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4520, 2018.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:/905.0m 07799, 2019a.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Aug- menting self-attention with persistent memory. arXiv preprint
2019b.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. arXiv preprint arXiv: I 1906.04393, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008. 2017.
Lyan Verwimp, Joris Pelemans, Hugo Van hamme, and Patrick Wambacq. Character-word LSTM language models. In Proceedings of the 15th Conference of the European Chapter of the Associ- ationfor Computational Linguisties: Volume Long Papers, pp. 417-427 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Jianing Zhou, Jingkang Wang, and Gongshen Liu. Multiple character embeddings for Chinese word segmentation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. pp. 210-216, 2019.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189-4198, 2017.
A SHUFFLE TRICK FOR GROUP INTERACTION
Here, we describe the shuffle trick used for the inter-group interaction in the group feed-forward layer;
ginter Σ
xg
Wiltorl ( g'g
(5)
M are linear weights used in the low rank ma- trix factorization. To explain the relationship from the shuffle trick, we describes the operations
where Winter|ll g'g
Poup xd and Winterr221 g'g
10

---

in the group feed-fol forward layer in bottom-up way. When applying group wise linear opera- tions on the imput features [x ..... Xc], the outputs are formed [& Winter()) ..... & Wi where Winterll E R P (M++G) S g S G. By splitting each element into G groups and the shuffle operation perturbs the outputs as follows;
&; Wi & Wittrr
N1 Wi Ka Wimer
& Wan Witeen
& Was Witeet
R: Winee
& Witedl
& Witle
Shuffle(
...
& W m
& W (il
& Wird
& Wae
& GG
(6)
where the Shuffle operation transposes the first and second dimensions of G x M matrix and Witterll| E IR Pomp XM XM is linear weight describing information flow from the group g' to the group
g. Finally, linear transformation with weight Wrinter (gM+G))DDpv is applied at the column
& Winei
& 20
ginier Flatten
Wintte[21
Σ
Winter
Winter122
9'9
g's
S Windlll Ga
where the Flatten operation vectorizes G M matrix to G* M vector. Therefore, the outputs (yggler R goop, Vg). defined with multiple calculations with 2G matrices (Wiiterll) and Winner(2) Vg'g) g), can be formulated with only 2G matrices ( (Wimeerll] and Wiiter[2] Vgg) by applying the introduced shuffle trick.
B REQUIRED RESOURCES OF GROUP-TRANSFORNET
In this section, we compare Troup-Transformet from the original transformer in views the num- bers of parameters. For common expression, we denote M D H as the feature size, the filter size in the fordwfd layer and the number of heads for the original transformer, and we set the feature size of group D group Dmode/ G, the filter size Deroup Dusda/ G, the number of heads in a group as H Hgroup H/G for Stroup-Transfone In this calculation, we set the filter size four times bigger than Dmodel
B. ATTENTION MODULES
The multi-head attention of the original transformer uses 4D of parameters for the query, the key, the value, and the output. The feature size for the multiple head usually set as Dmodel/ H where H is the number of the heads. Therefore, all transformations in the module is conducted for a Dmoda--diensioon
input feature to identify a Dmmate-dimensionn feature.
In comparison, a group attention requires G² D2 grop +4GD group 2D modee & D of parameters, including 2GD2 group for we--ittaa and Wq-inter 2GD2 group for Wo-intra and Wro-inter and G² D2 group for the keys and the value transformations. As we mentioned, we set DG = Dmodel/ G and thus the total number of the parameters becomes 2D2 modee + & model When the number of groups is 2. the number of parameters of group attention is the same with those of the original transformer. However, when the number of groups increases to 4 or 8, the number of the parameters decreases 75% or 62.5% 5% of the original module.
B.2 FEED-FORWARD MODULES
A woint-wise feed-forward layer of the original transformer requires 8D2 D of parameters to map an input feature into mona -dmeenionnl space and transpose it back to the input space, Dmodel-
In comparison, a group feed-forward layer requires SGD2 group + 5G2 group of parameters, in-
cluding 4GD2 group from willintra MD from W fl-iieiettl 4G M goup from infl-itter2] and 9'9

---

4GD grou from W In our experiment, since we set M as D Dgroup/G /G, the total number of parameters group
becomes 13GD group C D model- When the number of groups is 2, the group feed-forward layer uses 81 % parameters of those of the original transformer. When increasing the number of groups to or 8, the number of the parameters decreases proportionally to 40.6% or 20.3%.
C CHARACTER SEQUENCE GENERATION
To see the effectiveness of our models on real-world applications, we have performed two generative tasks: word completion and sentence completion. The former is to generate word and the latter is to conclude a sentence when a given character sequence is in-complete.
Word completion task: T shows the generated top 20 words to conclude the in-complete char- acter sequence, "pr". Although our 6M and 4M (bpc) on the quantitative evaluations, as can be seen, the model still produces all real or plausible English words without a significant quality gap from the Transformer-Xe with 41M parameters.
showed relatively lower scores
Seed: mary was not permitted to see them speak in her (--(abbreviate) --proof of guilt if authenticc the inquiry reached the conclusion that nothing was proven from the start this could have been pr (Truth) predicted
Transformer-XL (41M, 1.17 bpc text8)
proven, proved. proof, presented, proposed, probably, prevented, preceded, predicted, presumed, praised,
preserved, problematic, preferred, present, previously, precisely, printed, produced, profound
2 GGroup-Transfowe (6M, 1.26 bpc tex18)
proven, proof. proved, proposed. present, previously, presented, preserved, printed, probubly, practically.
produced, prepared. prohibited, predicted, progressively, profound, primarily, problematic, pructical
4 Group-Transformer Transformer (4M, 1.30 bpc text8)
proven, present, proposed, preserved, presented, previously, proved, practiced, produced, prepared, printed, probably, practically, provided, properly, presumed, praised, presently, prevented, primarily
Table 4: Examples of word completions. The seed text is prepared from tex88 test dataset.
Sentence completion task: Table generate reasonable sentences.
5 shows the generated sentences. We observe that the all models
Seed: By age fifteen, he had already given concerts worldwide. After a short stay at the ([Leipzig Conservatory]], in [[1876]] went to study in [[Brussels]]. In [[1880]], he went
(Truth) //Budapest]] to study with [[Franz Lisal1, only to find out that Liszt was |I W/Weimrr, Germanyll.
Transformer-XL. (41M, 1.06 bpc on enwik8)
(Sample 1) the [[Laertes Medical School]] in [[London]], to join the [[Excellets Laertes Medical School]]. (Sample 2) study law, married in [[Landjeball]].
2 Group- Transformer (6M, 1.19 hpc enwik8) (Sample 1) /|France]] for 4 years.
(Sample 2) study, and concentrate the context his [[later] study, followed.
4 Group- Transformer (4M, 1.22 bpc on enwik8) (Sample 1) Scotland, to denounce as an (urccittcttre/eechiteeeell and analysed [[Leipzig Conservatory]]. for for
(Sample 2) [[England]]
school at the [[Battle of Engstado Coast Corps]].
Table 5: Examples of sentence completions. The seed text is prepared from enwik8 test dataset.
D WORD-LEVEL LANGUAGE MODEL
The proposed method is focused on developing character-level language models, but the model can be applied to other NLP tasks. When comes to the word-level language modeling, compressing the word embedding layer becomes the most important part for designing lightweight language model. Therefore, we set a embedding dimension as 500 and adjusted the number of layers and the hidden dimension for the models to have the same number of parameters (4.5M). Specifically,
12

---

set the bottleneck dimension times larger than the hidden dimension and follows other experimental settings of Dai et al. (2019). Table Table compprees scale-downed Transformer XL and the Group-Tramee Transformers. In all multiple settings, the Group- Tranffrranfssmmrrs show better performances
than the baselines.
Model
Params
Non-embedding params.
ppl
Transformer-Xi XL (6L, 224d)
139M
4.5M
37.3
Baselines
Transformer-X XL (5L, 248d)
139M
4.6M
37.3
Transformer-X (4L, 272d)
139M
4.5M
37.4
4 Trouu-Trannformer (6L, 320d)
139M
4.5M
36.6
Ours
Group-Transfore (5L, 352d)
139M
4.5M
36.6
4 Group-TTransfors (4L, 392d)
139M
4.5M
37.0
Table 6: Comparison with the prior word-level language models wikitext-103, We report perplex- ity (ppl) for test sets as well as the number of parameters.
E ABLATION STUDY ON THE NUMBER OF GROUPS
Table shows the performance variation Group-Transformer as the number of groups change.
Group- Transformer outperforms the baselines in all parameter size categories. The 4 Group-
Transformer performs the best models with [4M-5M] and [6M-7M] parameters and the Group- Transformer becomes the best in [2M-4M] parameters.
Model
Params
enwik8 bpc
Trouu-Traanformer base (91., 232D)
6.6M
1.239
[6M-7M]
StroupTrranffomnn (9L. 256D)
6.8M
1.221
Trensfrraasr (9L, 328D)
6.8M
1.218
8 Group-Transfoon Transformer (9L, 400D)
6.8M
1,230
Group- Transformer hase (9I., 192D)
4.4M
1.286
[4M-5M]
Group-Transteree Transformer (9L, 200D)
4.3M
1.267
4 Group- Transforerr (9L, 256D)
4.3M
1.263
8 Group- Trou-frranformerr (9L. 312D)
4.3M
1.278
Shrou--TranSforenn base (9L, 152D)
2.9M
1.336
[2M-4M]
Choup-Transfornnn (9L, 168D)
3.1M
1.313
Group-Transformer Transformer (9L, 208D)
2.9M
1.316
8 Group-Transformo Transformer (9L, 256D)
3.1M
1.316
Table 7: Performance comparison between the numbers of groups under the similar number of parameters. denote "L" and "D" as the number of layers and the hidden dimension, respectively.
F ABLATION STUDY: GROUP OPERATIONS ON QUERIES, KEYS, AND VALUES
We investigated the effects of all grouping methods group attention. Table shows that the benefit on the parameter size is marginal compared to the performance drop when the number of grouping operations is increased. On the other hand, there relatively small performance gap between grouping targets under the same number of grouping operations.
13

---

Group Operations
2 Group
4 Group
8 Group
Query
Key
Value
Param.
hpc
Param.
bpc
Param.
bpc
6.8M
1.221
4.3M
1.261
2.9M
1.316
0
6.8M
1.225
4.3M
1.266
2.9M
1.316
6.8M
1.223
4.3M
1.262
2.9M
1.318
0
6.8M
1.233
4.0M
1.283
2.5M
1.328
6.8M
1.231
4.0M
1.277
2.5M
1.334
6.8M
1.228
4.0M
1.275
2.5M
1.340
6.8M
1.237
3.7M
1.296
2.0M
1.378
Table 8: Ablation study in modeling query, key, and value with our group operations.
14


Reviews:
[{"review_id": "r1xkV0ATtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a way of reducing the number of parameters in Transformer by splitting some of the linear layers into multiple separate groups. Additional mixing linear layers are added so information can pass from one group to another. In the feedforward submodule, a bottleneck layer is also added to reduce the number of parameters. Small scale experiments on language modeling compared the proposed model to vanilla transformers.\n\nI think the motivation of the paper is good and important for real-world applications of Transformers. However there are several major problems with the paper.\n\nFirst, the proposed method is only marginally better than a vanilla transformer of similar size, and both are much worse than the current sota. Also, the baseline transformer experiment is done by the authors themselves, and it’s not clear how well they tuned it. From table2, it seems they simply reduced the hidden size, but that’s not the only way of reducing parameters. \n\nThe second problem is that the authors completely ignore the fact that the multi-head attention is already doing grouped attention. It splits the hidden states into multiple parts, and each head performs attention separately. In thinking this way, the proposed “group attention” feels more like multiplying the number of heads. This also means the authors should compare their model to a vanilla transformer with 2x more heads.\n\nAnother problem is section 3.4. Here the authors claim their model has O(D_m^2/G) parameters, but it’s not true because key and value projections are not grouped and their size is O(D_m^2). Also, the number of parameters in the first layer of the feedforward submodule depends on M rather than G (if I understand it correctly). Despite this, I can’t find the exact value of M in the paper. \n\nOther minor comments are:\n- I don’t understand the reasoning behind not grouping key and value projections because “they can come from other source domain”. What does it mean and why it prevents grouping? In any case, the experiments only use characters so why not group them as well?\n- The paper has many typos and weird english such as “natural language process model ...”, “increased training complexity issues ...”, “where also requires …”, “gets incredible achievements”, “... performance compare to Transformers’. ”, “... rational behinds.”, “heavy weights”, “... how high similarity ...”, “... since Transformer raises.”\n- Why a batch size of 22? It’s much smaller than Dai et.al, so shouldn’t the learning rate need to be adjusted accordingly?\n- The figure 1c is not exactly consistent with the text. According to eq3, there should be a summation before ReLU. I know a summation of two linear layers can be written as concat + linear, but it would be easier to understand if the figure was consistent with the text.\n- Maybe make it clear in the introduction that a bottleneck layer is also used. \n- The introduction suggests that Transformers only works with large sizes, which is bit misleading. Yes, one needs huge models for reaching SoTA, but there is nothing in the Transformer architecture that requires large sizes. In fact, the experiments in the paper show that a small vanilla transformer outperforms much larger recurrent networks.\n- The group embedding in section 3.1 doesn’t make sense. Embedding simply assigns a vector to each token, so grouping dimensions here doesn’t change anything. It’s the same as having a simple embedding, then splitting it into multiple parts.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Syx8D1tKFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a lightweight Transformer model (Grouped Transformer) for character level LM tasks. The key idea to reduce model complexity (in terms of the number of parameters) is the idea of grouped computation, i.e., splitting the embedding into groups, applying functions group-wise and then learning some inter-group aggregation. The end result is a model that reduces parameter cost by the number of groups.\n\nOverall, the idea is an incremental one, although interesting largely based on the fact that this works. It mainly involves the application of group-wise paradigm to Transformers which enables parameter savings in the attention and feed-forward layers. I like the direction that this work is pushing for and I feel that the development of efficient Transformers is indeed a compelling direction. I am voting for weak accept.\n\nThe perhaps most limiting factor in this work lies in the execution. Personally, I find the experiments a little lacking and it is particularly puzzling to me why the authors restricted the scope of this work to only character level LM tasks. It would be interesting to know how the proposed method works on the standard MT benchmarks or other tasks where Transformers are state-of-the-art. (I note that there are some negative results on word-level LM in the appendix section)\n\nAnother particularly peculiar point in comparison with the standard Transformer model. Are the experiments (Table 1) really fair? Why do the authors not compare with the Transformer-XL with the same setting, i.e., number of layers (9 in theirs)? The authors should provide a direct comparison (some form of \"1-Group Transformer\" without inter-group interactions). \n\nThe charts in section C of the appendix are highly confusing, it would be better to just observe the effect of certain direct hyperparameters (number of groups, layers etc), instead of being hidden behind the number of parameters. I would be happy to see a distinct table or chart for every hyperparameter. This is the appendix so I don’t think space is an issue. \n\nI have some additional follow up questions\n\n1)\tWhat happens beyond 2 and 4 groups? What is the maximum number of groups before performance degradation becomes too much?\n2)\tMy understanding is that each linear layer gets parameter saving relative to the number of groups. Is this correct? The overall parameter cost is divided by the number of groups? If so, the extent of parameter savings is very similar to the Quaternion Transformer paper already cited in this paper? The grouped Transformer also splits embeddings into multiple groups, which draws parallels with the component-wise splitting in Quaternion Transformers. Should this be discussed or compared with given the striking similarities?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1e_weqbKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a lightweight alternative to the design of self-attention based Transformers on character-level language modeling (LM). The approach was motivated by the similar technique that has been applied on group convolutions, but with a few notable differences too, such as inter-group mixing and low-rank approximation (which also appeared in ConvNets before, but this still strkes me as a difference in the Transformer context). Via experiments on two large-scale char-level LM datasets as well as a relatively extensive set of ablative experiments, the authors demonstrated the effectiveness of their approach. \n\nPros:\n+ A very well-written paper. Most of the math symbols in the paper come with clear dimensionalities, which make it very easy to follow. The description for the methodology is also pretty clear. \n+ Well-designed experiments. Enwik-8 and text8, while widely used to benchmark Transformers these days, are still very challenging large-scale tasks. The authors also provide a series of ablative studies comparing the group-transformer with the original transformer in Table 3.\n+ Table 2 and Figure 3 (in the Appendix) are pretty strong proof of the effectiveness of the approach (at least on character-level language modeling). \n\n================================\n\nA few questions/issues/comments:\n\n1. For the key/value computation, why did you still keep the \"more complex/expensive\" $D_\\text{model}^2$ design? You explained in the paper that they could \"come from other source domain\", but in the specific case of character-level language modeling (in which you are just using a decoder Transformer without encoder-decoder attention), I don't think this is a problem. Why not make $\\mathbf{k}_{gh}$ and $\\mathbf{v}_{gh}$ something similar to how you compute the query? Or alternatively, why don't you make them low-rank too, as in the feed-forward layer? This difference in design seems strange to me.\n\n2. In Section 3.4, you mentioned that the Group-Transformer (I'll call it GT for simplicity below) has resource complexity $O(D_\\text{model}^2/G)$ whereas the original Transformer has complexity $O(D_\\text{model}^2)$. However, this is not true by your design of the key/value module, and by your own analysis in Appendix B.1, where you still have a $2 D_\\text{model}^2$ term. Therefore, I suggest reworking on Section 3.4, as the big-O complexity of the parameter space should be the same. (This again makes me curious about question (1) above...)\n\n3. Section 4.1 says that you only explored group size from {2, 4}. How did you pick this number? Why not 8 groups or more? As the 2-group option only saves about 10%-15% of the parameters (according to your analysis in Appendix B), it's actually not a large difference. Meanwhile, it seems 2-group is always better than 4-group. While I guess the 8-group option would certain make the model size very small, I'm very curious to see how good/bad it is when you match the # of parameters of an 8-group GT with a {2,4}-group GT.\n\n4. As the \"lightweight\" property of GT is what you are focusing on, could you also show/approximate the number of FLOPs used by LSTMs in Table 1? While LSTMs use more parameters, they don't use as much computation as do the Transformers (which has needs to form a $O(L^2)$ matrix in the self-attention module, where $L$ is the sequence length). Also, I think it's important to show the actual (wall-clock) runtime comparison of GT with Transformer-XL and the best LSTM model(s).\n\n5. I find it a bit strange (and slightly disappointing) that this method does not generalize that well to word-level language modeling, as none of the designs introduced in the paper are specific to \"character\"-level modeling alone. How's the performance of GT if you forget about the word embedding compression for a while (e.g., use a large embedding size, such as 500 like in prior works)? Some recent work [1] seems to suggest that very small Transformer-XL (only 4M parameters + a normal embedding) can achieve a perplexity around 35, too.\n\n------------------------------------\n\nSome issues that did not really affect the score:\n\n6. In Secton 3.2 (currently at the bottom of page 3), maybe add the dimensionality of $\\mathbf{x}$ (which should be $D_\\text{model}$) just for clarity, as you are omitting the \"time\" dimension (of a sequence) and only considering a single time step.\n\n7. Right after Eq. (2), the first $\\mathbf{W}_{gh}^\\text{m-intra}$ should be $\\mathbf{W}_{gh}^\\text{o-intra}$.\n\n8. In Eq. (4) (and the sentence following it), $W_{hg}^\\text{f2}$ shouldn't have a reference to $h$, as the reference to heads should only be in the self-attention.\n\n9. Eq. (7) intra -> inter.\n\n10. Some descriptions in Appendix A are confusing. For instance, you didn't really define function $\\text{Shuffle}(\\cdot)$, and it took me a while to realize you mean transposing the 0th and 2nd dimension of a $G \\times M \\times G$ matrix. Similarly, the $\\text{Concat}(\\cdot)$ function in Eq. (7) is \"undefined\", in the sense that its input is already a $G \\times M$ matrix (each row is a $1 \\times M$ vector). I think what you want is to vectorize it to shape $1 \\times (M * G)$ , and $\\mathbf{W}_g^\\text{intra[2]}$ should have shape $(M * G) \\times \\bar{D}_\\text{group}$. I suggest you revise an clarify this part.\n\n\n6. I'm curious (and wonder if you've tried this): What if you increase the model size of the Group-Transformer to be as large as the original Transformer on enwik-8 and text8 (e.g., 40M)? How does the GT perform? While Table 3 is indeed convincing, the result obtained by GT is still far from the actual SOTA (e.g., obtained by Child et al. [2] with a much larger model). Would be interesting to compare how a model \"as large\" would do.\n\n------------------------------------\n\nOverall, I think this is a promising strategy that seems to work very well on character-level language modeling. My only major concerns are some of the specifics of the design of the methodology (e.g., the key/value part) and the failure of the approach to generalize to the very-relevant domain such as word-level LM.\n\n[1] https://arxiv.org/abs/1909.01377\n[2] https://arxiv.org/abs/1904.10509", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using a lightweight alternative to Transformer self-attention called Group-Transformer. This is proposed in order to overcome difficulties in modelling long-distance dependencies in character level language modelling. They take inspiration from  work on group convolutions. They experiment on two large-scale char-level LM datasets which show positive results, but experiments on word level tasks fail to show benefits. I think that this work, though promising, is still somewhat incremental and has not shown to be widely applicable, and therefore I recommend that it is not accepted. 
Decision: Reject