Kati8/ICLR_2023_2025 huggingface dataset:
authors: Tamir David Hay
authors: Lior Wolf
reviews: 
Comments: The paper proposes to share the weights across layers when training Transformer architecture from scratch. The proposed method utilizes a network Q, which is trained with Q-learning, to assign the sharing policy. Experiments on several text datasets show that the proposed method uses less trainable parameters, reaches comparable performance as the conventional training, and sometimes reaches better training speed.
Strengths: The paper studies the weight-sharing scheme of training Transformers, and this may be used to reduce the training cost of developing large models.
Weaknesses: There are two main weaknesses of the paper in my opinion.

1. The related work section is not complete and misses many relevant works and topics. I think several lines of research are very related to this paper but they are missing in the related work: 

(1) network pruning (also lottery ticket hypothesis), 

(2) weight sharing in Transformers, 

(3) parameter-efficient fine-tuning methods (PEFT methods). 

Network pruning is mentioned a bit in the introduction, but the paper doesn't mention the pros and cons of this paper against network pruning, making the contribution of this paper unclear. I also found that there are several shared-weight Transformers papers [1, 2, 3] that are not cited in the paper, and they should be discussed. The parameter-efficient methods (LoRA, Adapters, Prefix Tuning, Prompt-tuning, etc) save a significant amount of parameters (such as 1 - 5%) and are very relevant to this work.

[1] Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers

[2] Sharing Attention Weights for Fast Transformer

[3] Lessons on Parameter Sharing across Layers in Transformers

2. Some weaknesses in experiments:

(1) Several weight-sharing techniques should be also included in the comparison.

(2) I would suggest reporting the performance on downstream tasks for a complete comparison. Lower perplexity sometimes does not mean higher downstream performance. 

(3) The perplexity of the results seems very high compared to previous results. As far as I know, the perplexity of WikiText-2 is usually around 20 and the perplexity of 1-billion words is around 30. I would suggest finding the proper training recipe or training longer to let the model converge, or the current results may not be very convincing. The current better perplexity in Tables 1 and 2 might only be due to the faster convergence of the model with less trainable parameters.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The authors propose a new means for essentially doing architecture search on a transformer model. The available variation for the model are which layers should be tied together, and which should remain independent. 

They employ an RL algorithm to do the selection of what should be connected to what. The reward signal comes straight from the perplexity of a model after an action has been taken. The results are very exciting and significant. It is very possible that the authors have just opened the floodgates to a new research path on which many will walk in the near future. 

The resulting models demonstrate that a transformer model with 48 layers, may require between 7-10 independent layer weights to perform as well, or better than conventionally trained 48 layer models. This opens up a lot of questions related to modern artificial neural circuitry and may allow for interesting modularization research to take place.
Strengths: 1. Clear, concise and precise writing. 
2. The idea attempted itself is intriguing. 
3. The idea is executed very well, resulting in excellent outcomes. 
4. The results are very promising.
5. The results open up a lot of new questions, as well as motivate the need for research in the pathway the authors have opened up.
6. The paper represents what could be a seminal moment in architecture search methods, as well as understanding modularity in NNs.
Weaknesses: 1. The method section, while technically sound, suffers from a lack of clarity as to the method being presented. For such a significant contribution, it is important to ensure that one can understanding the fundamentals of the method without having to crunch through all of the equations presented and fill many of the gaps with their own imagination. Perhaps something like a functional diagram, a more intuitive algorithm, or just a page that lays out the ingredients one by one, as well as how they are optimized, followed by the algorithm page would serve clarity of communication better. 
2. It's not clear if the authors have tried additional rewards in addition to PPL after updates, or if it was the only one tried.
3. Figure 1 is hard to read and understand. A more intuitive approach could leverage colour-coding to indicate a particular layers weights and perhaps a sequential figure showing a layer by layer colour coding might be more helpful/clear.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper proposes a method to reduce the number of trainable parameters of Transformers by dynamically tying the weights of different layers using reinforcement learning during the training process of Transformer. Experimental results indicate that compared to the conventional training method, the proposed approach effectively reduces the number of trainable parameters and slightly improves performance (perplexity score).
Strengths: 1 By employing a simple deep reinforcement learning algorithm, the number of trainable parameters of GPT-2 and BERT has been significantly reduced. The algorithm is easy to implement and holds the potential for application in other Transformer-based neural networks.

2 The algorithm has been thoroughly analyzed, providing a detailed explanation of the reasons for its effectiveness.
Weaknesses: 1 The Methods section contains a considerable amount of redundant content, providing a step-by-step explanation of the algorithmic process.

2 The experimental results only show the comparison of the proposed method and the conventional training method, lacking comparison with other baselines and the related methods.

3 The related work is not adequately summarized. There is only one publication after 2020 shown in the section of Related Work, and there is a lack of research focusing on improving the Transformer neural network architecture.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 1 poor
reviews: 
Comments: In this paper, the authors focus on the problem of reducing the number of parameters in transformers using Reinforcement Learning, specifically a variant of Q-Learning, to dynamically select layers in transformers and tie them together. They evalutate their approach on language models such as GPT-2 and BERT. They demonstrate the performance of their approach both in terms of final model performance and the amount of parameters reduced. They also conduct an ablation analysis of their approach.
Strengths: 1. Large transformer models have demonstrated excellent performance, but they are often very expensive in terms of computational resources. The paper addresses a very timely problem: ensuring transformer models can be applied in a practical setting without excessive cost.

2. The proposed method, parameter tying via Q-Learning-is intuitive and sensible approach for this problem.

3. Quantitative results both on the axis of performance and computational resources is convincing.
Weaknesses: 1. The approach is very specific in its focus. The architecture in question are only transformers, and evaluation is only on transformers in the language domain. It is unclear if these results hold for transformers in other data domains (such as vision transformers). It is also unclear if this approach (specifically parameter tying) could work for other neural architectures.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good