Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes a way of reducing the number of parameters in Transformer by splitting some of the linear layers into multiple separate groups. Additional mixing linear layers are added so information can pass from one group to another. In the feedforward submodule, a bottleneck layer is also added to reduce the number of parameters. Small scale experiments on language modeling compared the proposed model to vanilla transformers.

I think the motivation of the paper is good and important for real-world applications of Transformers. However there are several major problems with the paper.

First, the proposed method is only marginally better than a vanilla transformer of similar size, and both are much worse than the current sota. Also, the baseline transformer experiment is done by the authors themselves, and it’s not clear how well they tuned it. From table2, it seems they simply reduced the hidden size, but that’s not the only way of reducing parameters. 

The second problem is that the authors completely ignore the fact that the multi-head attention is already doing grouped attention. It splits the hidden states into multiple parts, and each head performs attention separately. In thinking this way, the proposed “group attention” feels more like multiplying the number of heads. This also means the authors should compare their model to a vanilla transformer with 2x more heads.

Another problem is section 3.4. Here the authors claim their model has O(D_m^2/G) parameters, but it’s not true because key and value projections are not grouped and their size is O(D_m^2). Also, the number of parameters in the first layer of the feedforward submodule depends on M rather than G (if I understand it correctly). Despite this, I can’t find the exact value of M in the paper. 

Other minor comments are:
- I don’t understand the reasoning behind not grouping key and value projections because “they can come from other source domain”. What does it mean and why it prevents grouping? In any case, the experiments only use characters so why not group them as well?
- The paper has many typos and weird english such as “natural language process model ...”, “increased training complexity issues ...”, “where also requires …”, “gets incredible achievements”, “... performance compare to Transformers’. ”, “... rational behinds.”, “heavy weights”, “... how high similarity ...”, “... since Transformer raises.”
- Why a batch size of 22? It’s much smaller than Dai et.al, so shouldn’t the learning rate need to be adjusted accordingly?
- The figure 1c is not exactly consistent with the text. According to eq3, there should be a summation before ReLU. I know a summation of two linear layers can be written as concat + linear, but it would be easier to understand if the figure was consistent with the text.
- Maybe make it clear in the introduction that a bottleneck layer is also used. 
- The introduction suggests that Transformers only works with large sizes, which is bit misleading. Yes, one needs huge models for reaching SoTA, but there is nothing in the Transformer architecture that requires large sizes. In fact, the experiments in the paper show that a small vanilla transformer outperforms much larger recurrent networks.
- The group embedding in section 3.1 doesn’t make sense. Embedding simply assigns a vector to each token, so grouping dimensions here doesn’t change anything. It’s the same as having a simple embedding, then splitting it into multiple parts.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a lightweight Transformer model (Grouped Transformer) for character level LM tasks. The key idea to reduce model complexity (in terms of the number of parameters) is the idea of grouped computation, i.e., splitting the embedding into groups, applying functions group-wise and then learning some inter-group aggregation. The end result is a model that reduces parameter cost by the number of groups.

Overall, the idea is an incremental one, although interesting largely based on the fact that this works. It mainly involves the application of group-wise paradigm to Transformers which enables parameter savings in the attention and feed-forward layers. I like the direction that this work is pushing for and I feel that the development of efficient Transformers is indeed a compelling direction. I am voting for weak accept.

The perhaps most limiting factor in this work lies in the execution. Personally, I find the experiments a little lacking and it is particularly puzzling to me why the authors restricted the scope of this work to only character level LM tasks. It would be interesting to know how the proposed method works on the standard MT benchmarks or other tasks where Transformers are state-of-the-art. (I note that there are some negative results on word-level LM in the appendix section)

Another particularly peculiar point in comparison with the standard Transformer model. Are the experiments (Table 1) really fair? Why do the authors not compare with the Transformer-XL with the same setting, i.e., number of layers (9 in theirs)? The authors should provide a direct comparison (some form of "1-Group Transformer" without inter-group interactions). 

The charts in section C of the appendix are highly confusing, it would be better to just observe the effect of certain direct hyperparameters (number of groups, layers etc), instead of being hidden behind the number of parameters. I would be happy to see a distinct table or chart for every hyperparameter. This is the appendix so I don’t think space is an issue. 

I have some additional follow up questions

1)	What happens beyond 2 and 4 groups? What is the maximum number of groups before performance degradation becomes too much?
2)	My understanding is that each linear layer gets parameter saving relative to the number of groups. Is this correct? The overall parameter cost is divided by the number of groups? If so, the extent of parameter savings is very similar to the Quaternion Transformer paper already cited in this paper? The grouped Transformer also splits embeddings into multiple groups, which draws parallels with the component-wise splitting in Quaternion Transformers. Should this be discussed or compared with given the striking similarities?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: This paper proposes a lightweight alternative to the design of self-attention based Transformers on character-level language modeling (LM). The approach was motivated by the similar technique that has been applied on group convolutions, but with a few notable differences too, such as inter-group mixing and low-rank approximation (which also appeared in ConvNets before, but this still strkes me as a difference in the Transformer context). Via experiments on two large-scale char-level LM datasets as well as a relatively extensive set of ablative experiments, the authors demonstrated the effectiveness of their approach. 

Pros:
+ A very well-written paper. Most of the math symbols in the paper come with clear dimensionalities, which make it very easy to follow. The description for the methodology is also pretty clear. 
+ Well-designed experiments. Enwik-8 and text8, while widely used to benchmark Transformers these days, are still very challenging large-scale tasks. The authors also provide a series of ablative studies comparing the group-transformer with the original transformer in Table 3.
+ Table 2 and Figure 3 (in the Appendix) are pretty strong proof of the effectiveness of the approach (at least on character-level language modeling). 

================================

A few questions/issues/comments:

1. For the key/value computation, why did you still keep the "more complex/expensive" $D_\text{model}^2$ design? You explained in the paper that they could "come from other source domain", but in the specific case of character-level language modeling (in which you are just using a decoder Transformer without encoder-decoder attention), I don't think this is a problem. Why not make $\mathbf{k}_{gh}$ and $\mathbf{v}_{gh}$ something similar to how you compute the query? Or alternatively, why don't you make them low-rank too, as in the feed-forward layer? This difference in design seems strange to me.

2. In Section 3.4, you mentioned that the Group-Transformer (I'll call it GT for simplicity below) has resource complexity $O(D_\text{model}^2/G)$ whereas the original Transformer has complexity $O(D_\text{model}^2)$. However, this is not true by your design of the key/value module, and by your own analysis in Appendix B.1, where you still have a $2 D_\text{model}^2$ term. Therefore, I suggest reworking on Section 3.4, as the big-O complexity of the parameter space should be the same. (This again makes me curious about question (1) above...)

3. Section 4.1 says that you only explored group size from {2, 4}. How did you pick this number? Why not 8 groups or more? As the 2-group option only saves about 10%-15% of the parameters (according to your analysis in Appendix B), it's actually not a large difference. Meanwhile, it seems 2-group is always better than 4-group. While I guess the 8-group option would certain make the model size very small, I'm very curious to see how good/bad it is when you match the # of parameters of an 8-group GT with a {2,4}-group GT.

4. As the "lightweight" property of GT is what you are focusing on, could you also show/approximate the number of FLOPs used by LSTMs in Table 1? While LSTMs use more parameters, they don't use as much computation as do the Transformers (which has needs to form a $O(L^2)$ matrix in the self-attention module, where $L$ is the sequence length). Also, I think it's important to show the actual (wall-clock) runtime comparison of GT with Transformer-XL and the best LSTM model(s).

5. I find it a bit strange (and slightly disappointing) that this method does not generalize that well to word-level language modeling, as none of the designs introduced in the paper are specific to "character"-level modeling alone. How's the performance of GT if you forget about the word embedding compression for a while (e.g., use a large embedding size, such as 500 like in prior works)? Some recent work [1] seems to suggest that very small Transformer-XL (only 4M parameters + a normal embedding) can achieve a perplexity around 35, too.

------------------------------------

Some issues that did not really affect the score:

6. In Secton 3.2 (currently at the bottom of page 3), maybe add the dimensionality of $\mathbf{x}$ (which should be $D_\text{model}$) just for clarity, as you are omitting the "time" dimension (of a sequence) and only considering a single time step.

7. Right after Eq. (2), the first $\mathbf{W}_{gh}^\text{m-intra}$ should be $\mathbf{W}_{gh}^\text{o-intra}$.

8. In Eq. (4) (and the sentence following it), $W_{hg}^\text{f2}$ shouldn't have a reference to $h$, as the reference to heads should only be in the self-attention.

9. Eq. (7) intra -> inter.

10. Some descriptions in Appendix A are confusing. For instance, you didn't really define function $\text{Shuffle}(\cdot)$, and it took me a while to realize you mean transposing the 0th and 2nd dimension of a $G \times M \times G$ matrix. Similarly, the $\text{Concat}(\cdot)$ function in Eq. (7) is "undefined", in the sense that its input is already a $G \times M$ matrix (each row is a $1 \times M$ vector). I think what you want is to vectorize it to shape $1 \times (M * G)$ , and $\mathbf{W}_g^\text{intra[2]}$ should have shape $(M * G) \times \bar{D}_\text{group}$. I suggest you revise an clarify this part.


6. I'm curious (and wonder if you've tried this): What if you increase the model size of the Group-Transformer to be as large as the original Transformer on enwik-8 and text8 (e.g., 40M)? How does the GT perform? While Table 3 is indeed convincing, the result obtained by GT is still far from the actual SOTA (e.g., obtained by Child et al. [2] with a much larger model). Would be interesting to compare how a model "as large" would do.

------------------------------------

Overall, I think this is a promising strategy that seems to work very well on character-level language modeling. My only major concerns are some of the specifics of the design of the methodology (e.g., the key/value part) and the failure of the approach to generalize to the very-relevant domain such as word-level LM.

[1] https://arxiv.org/abs/1909.01377
[2] https://arxiv.org/abs/1904.10509
review_title: Official Blind Review #1

Meta-review: This paper proposes using a lightweight alternative to Transformer self-attention called Group-Transformer. This is proposed in order to overcome difficulties in modelling long-distance dependencies in character level language modelling. They take inspiration from  work on group convolutions. They experiment on two large-scale char-level LM datasets which show positive results, but experiments on word level tasks fail to show benefits. I think that this work, though promising, is still somewhat incremental and has not shown to be widely applicable, and therefore I recommend that it is not accepted. 
Decision: Reject