Title: GROUP-TRANSFORMER: TOWARDS A LIGHTWEIGHT CHARACTER-LEVEL LANGUAGE MODEL

Abstract: Character-level language modeling based on Transformer has brought great success by alleviating limitation of recursive operation. However, existing Transformer-based models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources. In this paper, we propose a lightweight model, called Group-Transformer , that factorizes the calculation paths by grouped embedding operators. Additionally, Group-Transformer employs inter-group linear operators to prevent performance degradation from the group strategy. With comparison experiments on about five times larger and the best performing LSTM-based models and compatible parameter size of Transformer-based models, we show that Group-Transformer has better performance on two benchmark tasks, enwik8 and text8. Further experiments including ablation studies and qualitative analysis revealed that the proposed work contributes to the effective lightweight model for practical application. The implementation code will be available.

1. INTRODUCTION:

Character-level language modeling has become a core task in the field of natural language processing (NLP) such as classification (Zhang et al., 2015) , sequence tagging (Guo et al., 2019a) , question answering (He & Golub, 2016) , and recognition (Baek et al., 2019; Hwang & Sung, 2016) , with its simplicity on generating text and its adaptability to other languages. Along with the development of deep learning in NLP, using recurrent neural networks (RNNs) have been a standard way to solve the problem for many years. Recently, however, a new architecture, Transformer (Vaswani et al., 2017) , have shown promise in addressing this problem and have achieved breakthroughs in general language modeling (Al-Rfou et al., 2019; Dai et al., 2019) .

Though this technique has achieved incredible successes, it has led to the huge size of Transformerbased models due to building deeper and wider networks. Transformer-XL (Dai et al., 2019) and GPT-2 (Radford et al., 2019) , for instance, contain 277M and 1542M parameters, respectively. This trend toward a large size model for performance is not suitable for edge device applications, which require small memory sizes, such as optical character reader (OCR) and speech to text (STT), and for auto-correction and auto-completion applications that need fast real-time responsiveness.

To tackle this issue, choosing an appropriately efficient strategy becomes more crucial, especially in the real-world application which requires not only good performance but a lightweight model. In this paper, we introduce a lightweight transformer for character-level language modeling. Our method is one of the factorization methods in that it separates the standard linear layer in transformer architecture using group-wise linear operation and makes sparse connectivity between linear transformations. The proposed model is referred to as Group-Transformer since it is inspired by the group convolution approaches (Zhang et al., 2018; Sandler et al., 2018) that have effectively compressed huge image processing models for usability on mobile devices.

While the group strategy reduces parameters and calculations in the proposed modules, its mutually exclusive calculation for the multiple groups compromises performance, caused by the information loss of inter-group correlations. To compensate for this problem, we added two inter-group operations that share a common feature over groups for the group attention layer and linking features in different groups for the group feed-forward layer. By modeling the inter-group information flows, Group-Transformer becomes performant as well as lightweight.

We conducted extensive experiments on two benchmark datasets, enwik8 and text8, and found that Group-Transformer with 6M parameters outperformed all LSTM-based models with under 35M parameters. Furthermore, Group-Transformer shows better performance when compared against Transformers with a comparable number of parameters. We provide further analysis to identify the contributions of our proposed modules in detail. To the best of our knowledge, Group-Transformer is the first attempt to build a lightweight Transformer with the group strategy.

2. RELATED WORKS:



2.1. TOWARDS A LIGHTWEIGHT TRANSFORMER:

Since Transformer has become a promising model for diverse NLP tasks, there have been attempts to improve its efficiency with two majority approaches. The first is to restrict dependencies between input tokens to reduce superfluous pair-wise calculations (Child et al., 2019; Guo et al., 2019b; Sukhbaatar et al., 2019a) . The approach provides time efficiency during inference, but it does not address the heavy parameterization of Transformer. The second approach is to develop a lightweight network architecture while maintaining the properties of Transformer. For example, Tay et al. (2019) utilize quaternion algebra to build lightweight modules for Transformer. They also use the factorize the components of the embedding layer, but the expression power can be limited by the connection of factorized components based on the quaternion principle. Another such approach (Sukhbaatar et al., 2019b) combined the multi-head attention and point-wise feed-forward layer to devise a unified module with fewer parameters. Despite these attempts on architectural changes, their models still struggle to provide a lightweight language model with nearly still 30M parameters. In this work, we describe a lightweight transformer with less than 10M parameters, which is extremely small when compared against previous character-level language models.

2.2. GROUP STRATEGY:

A group strategy has attracted much attention recently to compress many large and deep state-of-theart convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Szegedy et al., 2015; Chollet, 2017) . For example, when the group strategy is applied to a standard linear layer with a weight W ∈ R I×O , the feature map is partitioned into G groups. As a result, the layer is replaced by G small linear layers where each holds a weight W ∈ R (I/G)×(O/G) , leading to a significant parameter reduction. Although intuitively appealing, it has been reported that applying the group strategy to the model often leads to huge performance degradation, since the features in different groups cannot interact with each other. To overcome this problem, ShuffleNet (Zhang et al., 2018) proposed channel shuffle operation to make interactions between different groups. This kind of consideration has also been applied to recurrent neural networks (RNNs). Kuchaiev & Ginsburg (2017) proposed group-wise RNN as a special form of ensembled RNNs. But, they did not consider the interactions between different groups. Inspired by the ShuffleNet, Gao et al. (2018) combined the shuffling idea into the group-wise RNN and achieved promising results. In this work, we adopt the group strategy and build the group-wise operations suitable for Transformer architecture.

3. GROUP-TRANSFORMER:

Figure 1a shows the overall architecture of Group-Transformer. It consists of a group embedding (bottom grey box), which embeds a character into grouped features, a group attention (yellow box), which contains attention modules to identify dependencies in the time domain, and a group feedforward layer (green box), which re-configures the grouped features. As can be seen, when an input character is given, Group-Transformer converts the input into multiple group representation (blue dots and red dots), processes and merges them to predict the next character. Figure 1b and 1c show group-wise information flow (blue and red arrows), and inter-group information flow (grey arrow) in the sub-modules. Without the inter-group information flows, the grouped features are processed independently. We observed that inter-group modeling ensures that the groups become aware of the others and prevents different groups hold the same information. The following subsections describe architectural details of the sub-modules and their relations. 

3.1. GROUP EMBEDDING LAYER:

Group embedding layer identifies a set of embeddings to represent a token. The idea of representing a sentence, word or even character using a set of vectors can widely be found in many NLP models that embed input tokens by concatenating (or summing) its embedding and its sub-units' embeddings (Verwimp et al., 2017; Bojanowski et al., 2017; Kim et al., 2019; Zhou et al., 2019) . Similarly, we assume a single character c to be represented with G vector representations of groups, that is, [u c1 , ..., u cG ] where u cg ∈ R Dgroup , 1 ≤ g ≤ G. When a character is given, the group embedding layer retrieves a corresponding set of vectors and passes it to the following group attention layer. Through this paper, we describe a process at a single time step.

3.2. GROUP ATTENTION LAYER:

The attention mechanism identifies dependencies between features in the time domain and combines the information of them. It contains three steps; (1) identifying queries, keys, and values, (2) retrieving relative features at different times, and (3) transforming the attended feature into the input domain (Vaswani et al., 2017) . The main focus of this paper is to apply a group strategy to the feature space of Transformer. Thus, we let the second step be identical to those of the original Transformer and focused on the first and the third steps.

Figure 1b explains the architecture of group attention. The multi-head attention module represents the second step, the under operations identify the queries for the first step, and the upper operations transform the attention output for the third step. We note that we do not represent the key and value for the multi-head attention block in the figure because they are possible to come from another source domain. The group attention processes the grouped features with intra-group operations (white boxes) and inter-group operations (grey boxes).

Grouped Queries, Keys and Values:

Let x = [x 1 , ..., x G ] be a set of input vectors where x g ∈ R Dgroup for the group g. Since the multi-head attention contains H attention modules for a single group, group attention first calculates query q gh for a group g and its head h as the below,

EQUATION

where W q-intra gh ∈ R Dgroup×(Dgroup/H) and W q-inter gh ∈ R Dgroup×(Dgroup/H) are linear weights to describe an intra-group (white boxes) and an inter-group (grey box) combinations, respectively. In the formula, the first term on the right-hand side identifies a specific feature for the head h in the group g and the second term determines head-wise features that allow the grouped features to share a common expression retrieving other features in a different time. When comparing with the fully connected linear layer over the groups, the approach restricts the connection between the groups, so requires a fewer number of parameters and calculations.

For the key k gh and the value v gh , we use fully connected layers by using all group pairs g and g; k gh = g x g W k g gh and v gh = g x g W v g gh where W k g gh ∈ R Dgroup×(Dgroup/H) and W v g gh ∈ R Dgroup×(Dgroup/H) are linear weights. As we mentioned, since the keys and the values can be defined from the other source domain, we use the same formula of the original Transformer, pursuing the universality of the proposed module.

Multi-head Attention:

The identified headed elements are used for connecting features in the time domain. In this step, position encoding (Vaswani et al., 2017) has an important role for the features to be aware of their position in an input sequence. In this paper, we apply the relative positional encoding, which describes a long-length character sequence effectively. By following Dai et al. (2019) , we define the attention score map with the relative positional information and the attention mechanism determines the attended feature a gh of the head h in the group g.

Combination of Multiple Heads:

The multiple heads [a g1 , ..., a gH ] in the group g are combined as the below;

EQUATION

where W o-intra gh ∈ R (Dgroup/H)×Dgroup and W o-inter gh ∈ R (Dgroup/H)×Dgroup are linear weights for combining intra-group and inter-group information, respectively. As can be seen, the final output is determined with a specific feature from its own group and a shared feature from whole groups. This step utilizes the same mechanism used to identify the queries with the same objective spreading group-wise information from multi-headed attention to all groups. These intra-group and inter-group modelings mainly contribute to reducing the number of parameters and calculations. Finally, the inputs x g are added into the output o g as xg = x g + o g for a residual connection.

3.3. GROUP FEED-FORWARD LAYER:

Group feed-forward layer re-configures the outputs of the attention module, xg , by applying groupwise operation at each position. Figure 1c shows the architecture of the proposed module. As can be seen, the groups are shuffled (grey box) and support each other. The group-wise features are processed with two linear transformations and one non-linear activation. As the original module does, the linear layers in our module transpose the input feature into a high dimensional space with non-linear activation and transform the output back into the input space.

The group feed-forward layer can be formally explained as follows. Given G input features [x 1 , ..., xG ], group feed-forward layer transposes the grouped features into a high dimensional space as follows;

EQUATION

where W f1-intra g ∈ R Dgroup× DG and W f1-inter g g ∈ R Dgroup× DG are linear weights for mapping intra-group and inter-group information into the DG -dimensional space, relatively bigger than D group dimension. Here, we introduce a low-rank matrix approximation on the inter-group transformation matrix W f1-inter g g . Modeling interactions between the groups requires the G × G numbers of weights, as well as the multiple weights for the group g, transpose the group into the high dimensional space for the target group g . If designing a fully connected weight for all groups like the original Transformer, the feed-forward layer still holds heavyweights and expensive calculations. To reduce the overburden, we factorize the matrix W f1-inter Sainath et al. (2013) and Novikov et al. (2015) . The newly introduced dimension M is smaller than D group , and thus the number of parameters and calculation is reduced proportionally with the ratio between M and D group . In this paper, we set M as D group /G to control the dimension relatively with the number of the groups. Interestingly, such matrix factorization can be modeled efficiently with a group-wise linear transformation and a shuffle trick as shown in Figure 1c . Please refer to the appendix for the detail of the shuffle trick.

g g into two matrices, W f1-inter[1] g g ∈ R Dgroup×M and W f1-inter[2] g g ∈ R M × DG , inspired by

Finally, a group-wise linear transformation is applied upon the high-dimensional feature as follow;

EQUATION

where W f2 g ∈ R DG ×Dgroup is a linear weight. For a residual connection, each grouped input feature is added into the output of the group feed-forward layer; ŷg = xg + y g .

3.4. RESOURCE REQUIREMENT:

Here, we describe the efficiency of Group-Transformer in view of the number of parameters and required computational costs. When considering the original transformer, its required numbers of parameters are 4 * O(D 2 model ) for its attention module (query, key, value, and output linear) and 2 * O(D model Dmodel ) for its feed-forward module, where Dmodel is a bottleneck dimension. Group-Transformer pursues to reduce the number of parameters by splitting the hidden state into multiple groups and processing them group-wisely. When we set the total dimension over groups as 

D model = D group * G,

4. EXPERIMENTAL RESULTS:



4.1. DATASET AND EXPERIMENTAL SETTINGS:

We demonstrate the efficiency of the proposed Group-Transformer with two popular benchmark datasets, enwik8 and text8. The enwik8 dataset contains 100M of English Wikipedia texts with 204 unique characters including alphabets, non-Latin and special characters. In comparison, the text8 dataset provides 100MB of pre-processed texts only with 27 unique characters by filtering superfluous content, such as tables, citations, and punctuation, and by replacing the non-Latin characters with spelled-out equivalents (i.e., "15" to "one five"). For a fair comparison with previous works, we used the training/dev/test splits defined by Mahoney (2011) for both enwik8 and text8.

Most of the experimental settings follow those of Dai et al. (2019) , where the difference lies in the hyperparameters that influence the size of the model. We set the number of layers L as 9, and we fixed the total size of feature D model for a single character as 256 and the total numbers of heads as 8 while the number of groups are explored in {2,4}. For the regularization of the model, we applied layer normalization (Ba et al., 2016) independently over groups and dropout layers upon the outputs of the group attention and the group feed-forward layer with the probability p = 0.1. The length of the feed sequence was 512 with the cached 512-length for the previous sequence (Dai et al., 2019) . We use the Adam optimizer with a learning rate of 2.5e-4, β 1 of 0.9, β 2 of 0.999, a batch size of 22, the number of iterations of 400,000, and the best model on the validation set is chosen. The implementation code will be available for the other details.

4.2. COMPARISON AGAINST PRIOR CHARACTER-LEVEL LANGUAGE MODELS:

We compare the Group-Transformer against existing character-level language models using under 50M parameters in Table 1 . The prior models are grouped according to their methodologies, including "LSTM," and "Transformer". We observe that the Group-Transformer outperforms the LSTM models with under 30M parameters and that the 2 Group-Transformer attains the best performance against all prior LSTM models on the enwik8 dataset. When compared to Transformers, we observe Table 2 shows the comparison results, focusing both on the performance and efficiency. At each class, we tuned Transformer-XL to have the similar number of parameters when the number of layers are 5, 7, and 9. When compared to Transformers at each category, Group-Transformer shows better performance than the baseline models. In particular, the 2 Group-Transformer shows its efficiency by using 82.1% parameters and 71.3% FLOPs of the base Transformer, but provides similar performance. The 4 Group-Transformer outperforms the second-best model with a large margin of 0.025. This is the result of inherent computational efficiency enjoyed by group-wise operation and our careful design of the inter-group connections. The number of groups can be interpreted as a hyper-parameter affecting the model size. Figure 2 shows the effectiveness of three hyper-parameters such as the number of layers, the size of hidden dimension, and the number of groups. The default model used Transformer-XL (Dai et al., 2019) with L = 9, H model = 8, D model = 256, and Dmodel = 4 * D model , and then we reduced the three hyper-parameters. When making the model thinner or shallower, the performances of the model become worse, but the required resources are getting lower. When comparing ours with two reduction methods, the group strategy shows better performances than the models requiring comparable resources. This experiment proved that the feature grouping methods, the main idea of this paper, is more efficient to reduce the model size and the time complexity than tuning other model parameters.

4.4. ABLATION STUDIES ON PROPOSED MODULES:

Group-Transformer includes two modules utilizing group-wise operations and inter-group modeling.

We conduct ablation studies to identify the contributions of the proposed modules and inter-group operations. 

(-0.3M) (-0.3B) (+0.012) (-1.0M) (-1.1B) (+0.023) (-1.5M) (-1.4B) (+0.018) Original Ours 7.1M 7.0B 1.221 5.3M 5.8B 1.251 4.1M 5.2B 1.284 (-1.0M) (-2.4B) (-0.003) (-2.8M) (-3.6B) (+0.027) (-4.0M) (-4.2B) (+0.060) Ours Ours 6.8M 6.7B 1.221 4.3M 4.7B 1.261 2.9M 3.7B 1.316 (-1.3M) (-2.7B) (-0.003) (-3.8M) (-4.7B) (+0.037) (-5.2M) (-5.7B) (+0.092)

Table 3 : Ablation study on the proposed modules, group attention and group feed-forward layer.

Table 3 shows the module-wise impact on the number of parameters and performance. For a fair comparison, we set the baseline model to a reduced Transformer-XL (Dai et al., 2019) of less than 8M parameters, and can gradually reduce the model size by replacing the attention and the feedforward layer with Group-Transformer module selectively. When replacing the feed-forward layer with Group-Transformer module, we observe that the number of parameters in all cases decreases more efficiently than replacing the attention module. Interestingly, when replacing both modules, the degradation is lower than the sum of the individual performance losses, but the sum of the individuals' reduces the required resources. This result demonstrates more efficiency of concurrently using both group-wise modules. In addition to this, we also investigate the influence of inter-group operations in our model. When the inter-group operations are removed (grey boxes in Figure 1b and 1c ), we observed the performance degradation on 2-Group-Transformer by 0.028 bpc and 4-Group-Transformer by 0.051 bpc. These gaps are relatively huge when compared to the performance gap between Transformer-XL and Group Transformers in Table 3 . The results re-emphasize the importance of inter-group modeling in Group-Transformer. Figure 3 shows the similarity patterns between the multi-head attention of our models ((a) and (c)) and the ablation models without the inter-group operations ((b) and (d)). As can be seen, the multi-head attention map from the model without inter-group operations shows high similarities among different groups, while the proposed model shows the opposite. These similarity patterns imply that the model cannot fully take advantage of multi-head attention, which is designed to attend multiple positions of content, without the proposed inter-group operation.

5. CONCLUSION:

Recently, remarkable progress has been made in character-level language modeling by Transformer. The advantage of Transformer lies in its effectiveness in modeling long-term dependencies between characters. However, the models have been developed with a huge number of parameters, and the inference of them has required an expensive computational cost. We argue that big models cannot be used in a limited computational environment. Group-Transformer has been developed to prove the effectiveness of Transformer in a lightweight setting. We have grouped features and proposed group-wise operations to reduce the number of parameters and time complexity of Transformer. In addition, to fully realize the advantage of the original Transformer, we have connected the groups to interact with each other. When applying Group-Transformer on enwik8 and text8, we found that Group-Transformer only with 6M parameters achieves better performances than LSTM-based models holding over 30M parameters. Further analysis has proved the effectiveness of the group strategy to reduce computational resources.

A SHUFFLE TRICK FOR GROUP INTERACTION:

Here, we describe the shuffle trick used for the inter-group interaction in the group feed-forward layer;

EQUATION

where W inter [1] g g ∈ R Dgroup×M and W inter [2] g g ∈ R M × Dgroup are linear weights used in the low rank matrix factorization. To explain the relationship from the shuffle trick, we describes the operations in the group feed-forward layer in a bottom-up way. When we applying group-wise linear operations on the input features [x 1 , ..., xG ], the outputs are formed as

[x 1 W inter[1] 1 , ..., xG W inter[1] G ], where W inter[1] g ∈ R Dgroup×(M * G) , 1 ≤ g ≤ G.

By splitting each element into G groups and the shuffle operation perturbs the outputs as follows;

EQUATION

where the Shuffle operation transposes the first and second dimensions of G × G × M matrix and W inter [1] g g ∈ R Dgroup×M is a linear weight describing information flow from the group g to the group g. Finally, a linear transformation with a weight W inter [2] g ∈ R (M * G)× Dgroup is applied at the column g;

EQUATION

)

where the Flatten operation vectorizes G×M matrix to G * M vector. Therefore, the outputs (ȳ inter g ∈ R Dgroup , ∀g), defined with multiple calculations with 2G 2 matrices (W inter [1] g g and W inter [2] g g , ∀g g), can be formulated with only 2G matrices ( W inter [1] g and W inter [2] g , ∀g) by applying the introduced shuffle trick.

B REQUIRED RESOURCES OF GROUP-TRANSFORMER:

In this section, we compare Group-Transformer from the original transformer in views of the numbers of parameters. For a common expression, we denote D model , Dmodel , H as the feature size, the filter size in the feed-forward layer and the number of heads for the original transformer, and we set the feature size of a group as D group = D model /G, the filter size Dgroup = Dmodel /G, the number of heads in a group as H group = H/G for Group-Transformer. In this calculation, we set the filter size as four times bigger than D model .

B.1 ATTENTION MODULES:

The multi-head attention of the original transformer uses 4D 2 model of parameters for the query, the key, the value, and the output. The feature size for the multiple head is usually set as D model /H where H is the number of the heads. Therefore, all transformations in the module is conducted for a D model -dimensional input feature to identify a D model -dimensional feature.

In comparison, a group attention requires G 2 D 2 group +4GD 2 group = 2D 2 model + 4 G D 2 model of parameters, including 2GD 2 group for W q-intra g and W q-inter g , 2GD 2 group for W o-intra g and W o-inter g , and G 2 D 2 group for the keys and the value transformations. As we mentioned, we set D G = D model /G and thus the total number of the parameters becomes 2D 2 model + 4 G D 2 model . When the number of groups is 2, the number of parameters of group attention is the same with those of the original transformer. However, when the number of groups increases to 4 or 8, the number of the parameters decreases to 75% or 62.5% of the original module.

B.2 FEED-FORWARD MODULES:

A point-wise feed-forward layer of the original transformer requires 8D 2 model of parameters to map an input feature into 4D model -dimensional space and transpose it back to the input space, D model .

In comparison, a group feed-forward layer requires 8GD 2 group + 5G 2 M D group of parameters, including 4GD 2 group from W f1-intra g , G 2 M D group from W f1-inter [1] g g , 4G 2 M D group from W f1-inter [2] g g and 4GD 2 group from W f2 g . In our experiment, since we set M as D group /G, the total number of parameters becomes 13GD 2 group = 13 G D 2 model . When the number of groups is 2, the group feed-forward layer uses 81% parameters of those of the original transformer. When increasing the number of groups to 4 or 8, the number of the parameters decreases proportionally to 40.6% or 20.3%. 

:



Figures and Tables:

Figure 1: Architecture overviews of Group-Transformer and its sub-modules when the number of groups is two. The gray arrows show the information flow across the entire groups, and the blue and red arrows indicate the information flow for each group.

the resource requirements of Group-Transformer become (2+ 4 G ) * O(D 2 model ) for group attention and 3 G * O(D model Dmodel ) for group feed-forward module. The number of groups is increasing, the resources is decreasing. Appendix B provides the detailed required resources of all sub-modules and comparisons with those of the original Transformer.

Figure2: Performance comparison of three reduction methods from model parameters such as the number of layers ("L"), the hidden dimension ("D"), and the number of groups ("G"). The FLOPs indicates the number of calculations to generate 512 length of a character sequence.

.) (∆ FLOPs) (∆ bpc) (∆ Param.) (∆ FLOPs) (∆ bpc) (∆ Param.) (∆ FLOPs) (∆ bpc)

Figure 3: Similarity matrices of multi-head attentions. The black box indicates a high similarity of attention patterns and the white box does the opposite. The red boxes represent groups of the multiple heads. The similarity is measured based on the euclidean distance between attention weights over a test sequence.

Comparison with the prior character-level language models on enwik8 and text8. We report bit-per-character (bpc) for test sets as well as the number of parameters.

Table content: <table><tr><td>enwik8</td><td>text8</td></tr></table>

Comparison of lightweight Transformers that use under 8M parameters. L, H model and D model indicate the number of layers and heads, the hidden dimension over the model. The FLOPs indicates the number of calculations to generate 512 length of a character sequence. We used D group = D model /G and H = H model /G for Group-Transformers to set the same number of the total heads in the attention module.

Table content: <table/>

Ablation study in modeling query, key, and value with our group operations.

Table content: <table/>

Peer review (iclr2020):
Summary: This paper proposes a lightweight alternative to the design of self-attention based Transformers on character-level language modeling (LM). The approach was motivated by the similar technique that has been applied on group convolutions, but with a few notable differences too, such as inter-group mixing and low-rank approximation (which also appeared in ConvNets before, but this still strkes me as a difference in the Transformer context). Via experiments on two large-scale char-level LM datasets as well as a relatively extensive set of ablative experiments, the authors demonstrated the effectiveness of their approach. Pros: + A very well-written paper. Most of the math symbols in the paper come with clear dimensionalities, which make it very easy to follow. The description for the methodology is also pretty clear. + Well-designed experiments. Enwik-8 and text8, while widely used to benchmark Transformers these days, are still very challenging large-scale tasks. The authors also provide a series of ablative studies comparing the group-transformer with the original transformer in Table 3. + Table 2 and Figure 3 (in the Appendix) are pretty strong proof of the effectiveness of the approach (at least on character-level language modeling). ================================ A few questions/issues/comments: 1. For the key/value computation, why did you still keep the *more complex/expensive* design? You explained in the paper that they could *come from other source domain*, but in the specific case of character-level language modeling (in which you are just using a decoder Transformer without encoder-decoder attention), I don*t think this is a problem. Why not make and something similar to how you compute the query? Or alternatively, why don*t you make them low-rank too, as in the feed-forward layer? This difference in design seems strange to me. 2. In Section 3.4, you mentioned that the Group-Transformer (I*ll call it GT for simplicity below) has resource complexity whereas the original Transformer has complexity . However, this is not true by your design of the key/value module, and by your own analysis in Appendix B.1, where you still have a term. Therefore, I suggest reworking on Section 3.4, as the big-O complexity of the parameter space should be the same. (This again makes me curious about question (1) above...) 3. Section 4.1 says that you only explored group size from {2, 4}. How did you pick this number? Why not 8 groups or more? As the 2-group option only saves about 10%-15% of the parameters (according to your analysis in Appendix B), it*s actually not a large difference. Meanwhile, it seems 2-group is always better than 4-group. While I guess the 8-group option would certain make the model size very small, I*m very curious to see how good/bad it is when you match the # of parameters of an 8-group GT with a {2,4}-group GT. 4. As the *lightweight* property of GT is what you are focusing on, could you also show/approximate the number of FLOPs used by LSTMs in Table 1? While LSTMs use more parameters, they don*t use as much computation as do the Transformers (which has needs to form a matrix in the self-attention module, where is the sequence length). Also, I think it*s important to show the actual (wall-clock) runtime comparison of GT with Transformer-XL and the best LSTM model(s). 5. I find it a bit strange (and slightly disappointing) that this method does not generalize that well to word-level language modeling, as none of the designs introduced in the paper are specific to *character*-level modeling alone. How*s the performance of GT if you forget about the word embedding compression for a while (e.g., use a large embedding size, such as 500 like in prior works)? Some recent work [1] seems to suggest that very small Transformer-XL (only 4M parameters + a normal embedding) can achieve a perplexity around 35, too. ------------------------------------ Some issues that did not really affect the score: 6. In Secton 3.2 (currently at the bottom of page 3), maybe add the dimensionality of (which should be ) just for clarity, as you are omitting the *time* dimension (of a sequence) and only considering a single time step. 7. Right after Eq. (2), the first should be . 8. In Eq. (4) (and the sentence following it), shouldn*t have a reference to , as the reference to heads should only be in the self-attention. 9. Eq. (7) intra -> inter. 10. Some descriptions in Appendix A are confusing. For instance, you didn*t really define function , and it took me a while to realize you mean transposing the 0th and 2nd dimension of a matrix. Similarly, the function in Eq. (7) is *undefined*, in the sense that its input is already a matrix (each row is a vector). I think what you want is to vectorize it to shape , and should have shape . I suggest you revise an clarify this part. 6. I*m curious (and wonder if you*ve tried this): What if you increase the model size of the Group-Transformer to be as large as the original Transformer on enwik-8 and text8 (e.g., 40M)? How does the GT perform? While Table 3 is indeed convincing, the result obtained by GT is still far from the actual SOTA (e.g., obtained by Child et al. [2] with a much larger model). Would be interesting to compare how a model *as large* would do. ------------------------------------ Overall, I think this is a promising strategy that seems to work very well on character-level language modeling. My only major concerns are some of the specifics of the design of the methodology (e.g., the key/value part) and the failure of the approach to generalize to the very-relevant domain such as word-level LM. [1] https://arxiv.org/abs/1909.01377 [2] https://arxiv.org/abs/1904.10509