Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a compression method for Transformer-based encoder-decoder or language models.
The key idea of the proposed method is to decompose the standard parameters into a much smaller shared parameter matrix and independent parameters for each original matrix.
Then, the method can approximately recover the original Transformer models by simple additions and multiplications.

The experiments are conducted on three MT tasks, one summarization task, and one language modeling task.
Experimental results show that the proposed method seems to reduce model sizes and computations successfully while preventing considerable performance degradation (in some cases, the proposed method appears to improve the performance).
main_review: The idea of the proposed method is interesting, but there are a few concerns in terms of the presentation.
Therefore, it is hard to judge whether this paper has enough contribution for publishing as the conference paper.

The following are my concerns in the current version.

### 1, Technical novelty
* The idea of the proposed method is interesting and might be effective.
However, the idea itself of sharing the parameter is not very innovative.
I think that sharing parameters for compressing DNNs is a standard technique nowadays.
Therefore, the authors need to clarify the contributions of the proposed method, such as the unique properties that previous similar compression methods cannot achieve.
Currently,  I do not find any strong properties in the proposed method.

* If my understanding is correct, the proposed method is a reconstruction method. 
Therefore, we need a trained model for applying the proposed method.
This means the proposed method requires additional computation.
I do not fully understand why this paper compares the computational cost with the standard Transformer.

### 2, Notation and equation
* The notations are incredibly messy and hard to understand.
The authors need to make notations much simpler for better understanding to readers.

### 3, L1 constraint
* If my understanding is correct, the relaxed L1 constraint does not guarantee to find the solution that satisfies the threshold of non-zero factors. This paper seems not to explain the way if such a situation occurs in the solution. ​

### 4, Typo or misconfiguration?
* In Table 1, it says the results for WMT De-En and WMT Fr-En.
However, at the beginning of Section 4, the experiments are conducted on WMT "En-De" and "En-Fr," which are not "De-En" and "Fr-En." 


### 5, Confirmation of model sizes 
* According to the original Transformer paper [1], the numbers of parameters of Transformer (base) and (large) are 64M and 213M, respectively.
However, in the experiments, the model size of the baseline Transformer is 3.6M (as shown in Table 1) for WMT En-De. 
Moreover, I checked the previous paper, such as the "Lite Transfomer" paper (Wu et al., 2020) and the "Pay less attention" paper (Wu et al., 2019).
However, I could not find the precise experimental settings used in this paper. 
I recommend clearly showing the model configurations and hyper-parameter settings for keeping reproducibility.
Otherwise, the reproducibility of the proposed method may not be sufficient.

[1] Vaswani et al., Attention Is All You Need, In Proc. of NIPS-2017.




### 6, Inconsistent results in Table 1 and 3
* I thought that the ablation study of Table 3 is based on the results (settings) of Table 1.
However, the numbers of parameters shown in Tables 1 and 3 differ entirely, so I do not understand the meaning of the ablation study in Table 3.
Please confirm it and clarify the configuration difference between Tables 1 and 3. 
Moreover, explain the results of the baseline Transformer and the proposed method corresponding to the ablation results in Table 3.

* Additionally, it seems that there is no description about what the "Improved-Embedded" is shown in Table 3.
If I miss the description, please let me know. If the paper lacks explanation, this can be an apparent problem for this paper in terms of completeness.













summary_of_the_review: The idea of the proposed method is interesting and might be effective.
However, the idea itself of sharing the parameter is not very innovative and rather incremental.
Experimental settings are ambiguous and seems to use very weak settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a technique for reducing the size and computation of a Transformer model by projecting and factoring weight matrices. Experiments on MT, summarization, and language modeling show improved results over competing techniques, and even over standard Transformers, despite using significantly fewer parameters and less computation.

main_review: The paper contains a lot of substance, but it is very dense and hard to follow. The core “dictionary” technique isn’t really explained at a high level before the paper plunges into the details. It seems to be something like the approach in [1] but it’s difficult to be sure (I gave up on section 3 after a while). The results in section 5 are very impressive, but some intuition about why a compressed approach like this could beat a much larger baseline on large data settings really need to be provided.

[1] Kaiser, Lukasz, et al. "Fast decoding in sequence models using discrete latent variables." International Conference on Machine Learning. PMLR, 2018.

Details:
- The “first line of research”: would be good to add a word or two saying how these papers reduce computational complexity.
- Figure 1 is really great, but you should say where these stats come from.
- Figures 2 and 3: captions crash into text.
- This is hard to understand: “	In this paper, the #Params omit word embedding size that would highly dependent on the sentence length and would significantly differ for various tasks. The total #Params in this paper includes the model size of word embedding.”
- It’s difficult to align table 3 with figure 5. You should include a line corresponding to the point in figure 5 with highest BLEU (higher than anything that appears in table 3).

summary_of_the_review: Potentially a great paper, but if so it deserves to be much better explained.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a modification of the original Transformer architecture by replacing attention layers and layers in its Feed-Forward Networks across all of its blocks with learned shared dictionaries. The proposed model, called DictFormer, has a smaller number of parameters and uses a smaller amount of computational operations when compared to the original Transformer and some of its variations. When evaluated against these models on popular machine translation, summarization, and language modeling benchmarks, DictFormer achieves comparable or better performance.
main_review: ### Strengths
- The proposed modification to the Transformer architecture reduces the number of model parameters and computational operations while sustaining competitive performance on various downstream tasks.
- To the best of my knowledge, the idea of replacing layers of the Transformer with shared dictionaries is novel.

### Room for Improvement

*Shared-dictionary Attention*
- I might be missing something but why is it stated that the unshared linear projection $\tilde{W_{i}^{Q_{j}}}$ is approximately equal to $W_{i}^{Q_{j}}$? My understanding is that this is not directly optimized for in the model.

*Group-wise Shared Dictionary FFN*
- The motivation behind dividing columns of the dictionary into groups is a bit unclear. What is meant by “high-quality performance” of the shared dictionary projection? Also, have the authors considered using a larger number of dictionary elements $m$ to increase the “flexibility” of the model?  
- How is the number of groups $G$ determined?

*Training the DictFormer*
- Since the sparse matrix Z is initialized using values in $C$, how are coefficients $C$ initialized? 

*Results tables*
- Missing confidence intervals. Were the experiments run with multiple seeds? 

*Suggested related work*
- How is this work related to work on Sparse Transformers (e.g. [1], [2]) or fixed attention such as [3], [4]? 

[1] Child R, Gray S, Radford A, Sutskever I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. 2019 Apr 23.

[2] Correia, G.M., Niculae, V. and Martins, A.F., 2019. Adaptively sparse transformers. arXiv preprint arXiv:1909.00015.

[3] You, W., Sun, S. and Iyyer, M., 2020. Hard-coded gaussian attention for neural machine translation. arXiv preprint arXiv:2005.00742.

[4] Raganato, A., Scherrer, Y. and Tiedemann, J., 2020. Fixed encoder self-attention patterns in transformer-based machine translation. arXiv preprint arXiv:2002.10260. 

*Additional questions*
- Is it necessary to have the dictionary size less than the embedding size, namely $m < d$? Would it not be feasible to have a large dictionary ($m > d$) but keep the number of selected components $t$ small (i.e. $t < d$) through a sparsity constraint? 
- Have the authors tracked whether all columns of the dictionaries are used in practice?
- Have the authors tracked what percentage of the $t$ coefficients are non-zero on average? 

*Nitpicks*

Typos:
- p. 2, first line: “few unshared linear projection*s*”
- p. 3, “Overview” paragraph: “given a*n* accuracy threshold”
- p. 4, paragraph starting with “The reason why...”: “C_{i}^{x}” - should not $x$ be capitalized? 
- p. 5, “Group-wise Shared-dictionary FFN” paragraph: “a $N d \times d$ weights” -> “$N$ weights of size $d \times d$”   
- p. 6, Figure 4: “training sparse coefficients” -> “we train sparse coefficients”
- p. 6, first sentence of “Training DictFormer via Constraints and Relaxation” paragraph: “linear projections of *a* dictionary” 
- p. 7, last paragraph of “Architecture and Evaluation” paragraph: switch first sentence to present tense; “total #Params *i*n...”
- p. 8, “Machine Translation” paragraph: “DictFormer obtain*s* more compact” 
- p. 8, “Sensitive Study” paragraph: rename to “Sensitivity Study”
- p. 9 , first paragraph: “coefficient size is fixed *to* 60”
- p. 9 , “Ablation” paragraph, first sentence: missing space after period
- p. 9, “We will release code and data...”: Is there data to be released? 


summary_of_the_review: The proposed modification to the Transformer architecture is novel and I believe would be interesting for the community but the methodology and motivation could be explained more clearly and provided with more context, including more details on the hyperparameter selection and on how the DictFormer is trained. The experimental results would be even more convincing if confidence intervals are provided.

#### Updates during paper discussion
Based on the author's responses to the reviewers' questions and updates to the manuscript (including clarifying some of their methodology and statements and including confidence intervals in the results section), I've decided to increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed an efficient transformer layer based on a dictionary of shared parameters instead of standard self-attention.  The goal is to reduce redundant parameters in transformer models.  The main contributions are: a lite transformer model, modification of the self-attention parameters, and evaluation on language dowstream tasks. The proposed transformer model outperforms related work on the machine translation and language modelling tasks.
main_review: Strengths
- Clear description of background knowledge.  
- Clear exposition of the proposed model.
- The authors perform a comprehensive comparison on different downstream tasks, such as, machine translation, summarization, and language modeling.
- The findings show that the proposed transformer model outperforms related work on the machine translation and language modeling tasks.

Weaknesses
- It is not clear how the initialisation of hyper-parameters affects model performance.  

Questions to the Authors.
Please address the following questions during the rebuttal:

- Does parameter initialization could affect model performance? A possible extra contribution is to perform multiple  random runs and report variance. However, how expensive could this exercise become? 
- Please speculate on how attention representations behave across layers. For example, in Abnar, and Zuidema Quantifying Attention Flow in Transformers or Voita, et al. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives 
- By using other pre-training objectives in the langgue modelling task (e.g. next sentence) would it change any finding or results? 

summary_of_the_review: I recommend acceptance given that the paper clearly describes related work, and proposed model. The authors proposed an efficient transformer model that can be trained with less resources. The authors perform an  evaluation of the proposed model with different language downstream tasks, and the model outperforms related work on machine translation and language modelling.
review_title: null

Meta-review: DictFormer is a method to reduce the redundancy in transformers so they can deployed on edge devices. In the method, a shared dictionary across layers and unshared coefficients are used in place of weight multiplications. The author proposed a l1 relaxation to train the non-differentiable objective  to achieve both higher performance and lower parameter counts.

All reviewers ended up giving the paper a score of 6 after increasing their scores during discussions. While the results are strong (better performance at much lower parameter counts), the paper is not clearly written. Several reviewers noted that the paper is difficult to understand and has a few unresolved points. For example, the method also ended up performing better than the base transformer model that DictFormer is supposed to compress. There seems to be a lack of understanding about what part of the model delivered the improvements. One reviewer said that this is potentially a great paper that deserves to be better explained. The basic concept of sharing a dictionary across layers should be simple enough to explain well and deserve a better explanation than eq 5. 

The authors promise to release the code, which would be necessary for a full dissemination of this work. I recommend accept.
Decision: Accept (Poster)