Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Transformer based bidirectional LMs pre-trained using Masked Language Model loss typically share input and output token embeddings. This paper makes an interesting investigation about decoupling input and output embeddings and gains which can be obtained out of this decoupling. In particular, this paper shows that the pre-training performance of transformers and the transferability of the learned representations can be improved by increasing the dimension of output embeddings while reducing the dimension of input embeddings. Better performance while pre-training and improved transferability further helps the performance while finetuning on downstream tasks. Parameters saved while reducing the dimensions of input embeddings can be re-invested into increasing the depth or width of the transformer layers. I believe that the findings in this paper are going to be useful in practice to the general NLP community working on Transformers and Multilingual problems.

Weak Points:
1. In table 3, (E_in=768,E_out=128) is just 0.1 point worse on average than (E_in=128, E_out=768).  Yet, authors draw some conclusions on Page 4 based on this table, without reporting any variance or statistical significance tests.
    E.g. 
    >  the model pretrained with a larger output embedding size slightly outperforms the comparison method on average despite 
    having 77M fewer parameters during fine-tuning

    > Reducing the input embedding dimension saves a significant number of parameters at a noticeably smaller cost to accuracy 
    than reducing the output embedding size.

I would request the authors to report the variance in Table 3 and also in Table 9. Otherwise, it’s hard to rely on conclusions drawn from such minor differences in performance.


Strong Points:
Strong empirical results.
1. Table 4 clearly shows that increasing the dimension of output embeddings improves transfer to downstream tasks while having the same number of trainable parameters during the finetuning stage.  
2. Table 6 shows that reinvesting the parameters saved from reducing the dimension of input embeddings into additional transformer layers yield improved performance. Further, RemBERT performs at par with XLM-R while having a comparable number of trainable parameters during the finetuning stage.
3. Sufficient amount of analysis in Section 6 to establish the usefulness of having higher dimensional output embeddings for improved pre-training and better transferability of the learned representations across tasks. 

Other comments/questions:
1. Shouldn’t #PT params in table 2 for the “Decoupled” method be more than #PT params for the “Coupled" method ?
2. More figures like Figure 1, on other tasks in addition to XNLI, would be really helpful in making the observations more conclusive.




review_title: Transformer based bidirectional LMs pre-trained using Masked Language Model loss typically share input and output token embeddings. This paper makes an interesting investigation about decoupling input and output embeddings and gains which can be obtained out of this decoupling.

final_score: rating: 7: Good paper, accept
review_content: Summary:
 
This work investigated the strategy of reallocating parameters of multilingual language models for improving their cross lingual transferability. Authors first decoupled the input and output embeddings and showed that the capacity of output embedding is more important than input embedding. Then,  they proposed a Rebalanced mBERT (RemBERT) model that reallocates the input embedding parameters of mBERT model to the output embedding and additional layers. Experimental results on XTREME benchmark showed that RemBERT significantly outperformed XLM-R with similar model size.

Pros:
 
- The paper is well written and easy to follow. The comparison of embeddings parameters ratios of different language models in Table 1 gives a very good motivation of reallocating the parameters of embeddings.
 
- Authors conducted several ablation studies to understand how the capacity of  different parts of the model (e.g., input and output embeddings, transformer layers) contributes to the final performance. 

Cons:

- Reallocating the parameters of input embedding to additional transformer layers might affect the pre-training and inference speed, it will be helpful to show the training and inference speed of different parameter reallocation strategies.
 
Questions:

- Why are the pre-training and fine-tuning parameters of coupled and decoupled models the same in Table 2?  Shouldn’t decoupling the input and output embeddings double the parameters of the embeddings?

review_title: Good Analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper systematically studies the impact of embedding coupling with multilingual language models.  The authors observe that while na¨ıvely decoupling the input and output embedding parameters does not consistently improve downstream evaluation metrics, decoupling their shapes comes with a host of benefits. Moreover, they achieve significantly improved performance by reinvesting saved parameters to the width and depth of the Transformer layers on the XTREME benchmark.

This paper is well-written and strongly motivated. The idea of decoupling embedding is novel, and the evaluation results are strong.  

Strength:

+ The systematical study of the impact of embedding coupling on state-of-the-art pre-trained language models.  This paper also thoroughly investigates reasons for the benefits of embedding decoupling and observes that an increased output embedding size enables a model to improve on the pre-training task, which correlates with downstream performance. They also find that it leads to Transformers that are more transferable across tasks and languages. Those empirical results will promote future model design and the understanding of the transformer for textual representation learning.
+ The paper proposes a method to reinvest saved parameters to the width and depth of the Transformer layers and achieve significantly improved performance on the XTREME benchmark over a strong mBERT.

Weakness:

- Some model details are missing. Although I know what the input and output embedding are, it is still a bit hard to follow in Section 4 and 5. I strongly recommend that the authors revise those parts and introduce the model details, such as decoupling and model architecture in your experiments.
- Lots of your model designs are empirical such as the embedding size, and it is a bit boring to optimize those hyperparameters, and sometimes we even do not know why it works.

Questions:

- Could you please introduce the model details of your decoupled model in Section 4?
review_title: A well-written paper and an exciting idea

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work studied the impact of embedding coupling in pre-trained language models, by taking a multilingual model as backbone. The major finding is that decoupling the input and output embedding shapes can bring benefits, and the output embedding plays an important role in the transferability of pre-trained representation. A rebalanced mBERT is designed by combing and scaling up the investigated techniques, achieving strong results on the XTREME benchmark.

This paper is well written, and the proposed strategy is simple yet effective for obtaining more transferable language representations. The insights of model design for more efficient fine-tuning are well supported by the analysis.

My concern is about the model efficiency and the true source of performance improvement. It seems that the number of the parameters is much more than those public ones. I am curious if it is a fair comparison in Table 7. 

The current approach does help in reducing the number of parameters in the fine-tuning stage by increasing E_{out}. Even in your all experiments, the optimal value of E_{out} appears to be 768 (similar in the baseline), where performance is quite similar to baseline. However, increasing E_{out} to a much larger value (i.e 3072) drastically increases the number of parameters in pre-training, even as compared to baseline, which seems like a trade-off between fine-tuning and pre-training. The performance with much larger E_{out} is a marginal improvement over baseline unless the saved parameters are reinvested. The reinvestment of parameters creates a larger improvement over baseline (Table 6). The authors must conclude with optimal values of E_{in} and E_{out}, otherwise the paper is merely a series of experiments with different values of E_{in}, E_{out}, and # of layers in the baseline.

Try referencing Table 7 in the main explanation of RemBERT rather than in the Appendix, as it is your major contribution. But again my concern is the same, the number of parameters in fine-tuning is larger than XLM-R plus the additional pre-training time (more than XLM-R). Yes, the performance increases but the main objective of the paper is not quite satisfied (reducing the number of parameters in FT which are actually larger in RemBERT than XML-R). Maybe you can alter the abstract accordingly and focus on reinvestment, which actually is helping in making your point in the experiments.



Minor comments:

Check the use of \citep{} and \citet{}, e.g., in the last sentence of Section 2 and Footnote 10.

Table 7 as main results was never cited in this paper (but only in appendix), which is not a proper organization way.

Page 3-> "It consists of 12 Transformer layers with a hidden size H of 768 and 12" the sentence seems incomplete.

Page 5 -> "increasing the number of Transformer layers L results in the best performance" I am wondering if reinvestment in L gains higher performance than in H because of the larger number of parameters (10M more). Maybe you can keep reinvestment the same in both H and L for fair comparison and better results projection.

Section 4: How about a proportional increase in E_{in} and E_{out} affects the performance (e.g increasing or decreasing both with the same proportion, especially decreasing which lines up with paper goal i.e providing flexibility with less number of parameters.)

Page 7 -> "For both E_{out} = 128 and E_{out} = 768, removing the last layer improves performance". Seems in disagree with Figure 1. Acc with 10 layers is lower than that with 12 layers for E_{out}=128. Similar is the case with E_{out}=768.

How about the extra pre-training and fine-tuning time compared with the baseline? 

For the final rebalanced mBERT described in page 5, how are the hyper-parameters decided? 


review_title: much larger model size?

Meta-review: This paper makes a thorough investigation on the idea of decoupling the input and output word embeddings for pre-trained language models.  The research shows that the decoupling can improve the performance of pre-trained LMs by reallocating the input word embedding parameters to the Transformer layers, while further improvements can be obtained by increasing the output embedding size.  Experiments were conducted on the XTREME benchmark over a strong mBERT.  R1&R2&R3 gave rather positive comments while R4 raised concerns on the model size.  The authors gave detailed response on these concerns but R4 still thought the paper is overclaimed because the experiments were only conducted in a multilingual scenario.  
Decision: Accept (Poster)