Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an efficient fine-tuning method for pre-trained language models. The main idea is to fix the models' weights,  calculate the layer-wise activation, project the activation to lower dimension, and feed the downsampled activation into a smaller side network. During training, only the side network's parameters are updated so that the memory cost is small. The authors propose an initialization method for the side network that directly takes weights from the pre-trained models. Experiments show that the proposed method can memory-efficiently train large language models while preserving decent accuracy. 
strengths_and_weaknesses: - The memory saving is significant compared to other methods due to the extracted layer-wise representation and smaller side network. 
- The number of parameters is smaller than full finetuning which saves the cost of storing parameters. 
- Propose an initialization method for side networks that uses Fisher information and structured pruning. 
questions: - I am a little bit confused why the initialization method could help. For example, the activation after layer 1 will be fed into the first layer of the side network, which is also initialized by the partial weights of layer 1 in the full model. With that being said, the activations seem to be processed by the same layer twice (or partially twice), and I am not sure whether this is a good choice if I had understood correctly. Although the ablation experiments show the comparison between different pruning criteria, I think it would be better to compare methods like "offsetting" or "random initialization". 

- This paper emphasizes on training memory but I am curious about how would the inference memory of LST compare with other methods since inference memory may also be bottlenecks. For example, a device may be able to fit T5-base but not able to fit T5-large or T5-3B for inference. So even if we are able to train a T5-3B with smaller memory, it maybe also limit the practical usage. This is somehow related to the comparison in Table 1 where the authors train a larger network for comparison. 

- The experiments on LoRA seem to use a larger rank that most works suggested (for example most works use rank smaller than 16 which will lead to smaller than 1% trainable parameters), and original LoRA paper stated that they can save more than 2/3 VRAM on GPT-3 while in this paper the memory saving is fewer. 
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new parameter-efficient transfer learning (PETL) technique called Ladder Side-Tuning (LST). The proposed method trains a small and separate ladder side network that takes intermediate activations as input and output predictions by short-cut connections. During finetuning, the paper only requires backpropagation through the extra small ladder side network, and thus improves parameter and memory efficient transfer learning. The experiments on various models and learning tasks show the efficiency of the proposed method.
strengths_and_weaknesses: Strengths:
(1) The proposed LST only needs to compute backpropagation on small ladder side networks, which greatly improves the efficiency of PETL models.

(2) The experiments on various models and tasks (vision and language) show the efficiency of LST.

Weaknesses:

(1) The proposed LST shares a similar idea with side-tuning [53]. As claimed in side-tuning [53], “we propose a straightforward alternative: side-tuning. Side-tuning adapts a pre-trained network by training a lightweight “side” network that is fused with the (unchanged) pre-trained network via summation”. In Line 99-107, this paper claims LST there are differences in motivations, architecture designs, and applied tasks between [53] and LST. I summarize these three aspects as follows.

-Motivation. Side-tuning [53] focuses on incremental learning, an adaption task that extends n-category classification to (n+m)-category classification. The proposed LST aims to fine-tune large pre-trained models on downstream tasks, which is a general adaption task. Although the tasks are not extra the same, they are both adaptation tasks. Side-tuning [53] can be directly used to fine-tune large pre-trained models without major adjustment.

-Architecture designs. Side-tuning claimed that places no constraints on the structure of the base model or side network, allowing for the architecture and sizes to vary independently. It applied a residual on the top of the backbone in experiments. LST uses multiple layers as a side network of the backbone. I think the modification mainly depends on how large the gap between a source task and a target task. The main reason could be incremental learning has to maintain the source classification and only require a minor update. [53] used knowledge distillation for initialization and LST uses network pruning. I think both are available.

-Applied tasks. Side-tuning [53] focused on incremental learning and LST focuses on adaptation from a large pre-trained model to downstream tasks.

Therefore, compared with [53], the contribution of this paper is weak.

(2) In Line 295, the paper conducts some studies on w/ random and w/ network pruning. I wonder if knowledge distillation side-tuning also works, which is proposed by side-tuning[53]. As far as I know, both knowledge distillation and network pruning work in network compression. It would be nice if the paper studies why not directly use knowledge distillation as side-tuning does.

(3) In Line 301-309, the paper conducts some studies on side-tuning. I am interested in the analysis that we directly use side-tuning [53] for this task and simply use more intermediate layers.  Using more intermediate layers is widely used in deep neural networks, at least dating back to segmentation/classification/detection tasks in 2016.

questions: (1) More discussions/analyses about the key differences between side-tuning and LST.
(2) More experiments show the claims/advantages that LST differs from side-tuning.

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes Ladder Side-Tuning to improve the memory efficiency of transfer learning. The core idea is to freeze the backbone while adding an extra trainable side network. The side network takes multiple features extracted by the backbone and produces the output. This paper also shows that using the weight from the backbone to initialize the side network and layer dropping are helpful for memory-efficient transfer learning. 
strengths_and_weaknesses: Strengths:
1. Improving the memory efficiency of transfer learning is critical especially when we want to use large pre-trained models. 
2. The proposed method is simple and can be applied to different model architectures.

Weaknesses:
1. The overall idea of the proposed method is very similar to Side-Tuning [1]. While the authors argue that the proposed method has a different goal and focuses on different tasks, I still find the overall technical novelty of the proposed methods is a bit limited. 
2. In the related work section, I think the authors should add a section discussing the differences between the proposed method and previous memory-efficient transfer learning methods (e.g., [2,3]). In the experiment section, I think it is necessary to add direct comparisons between the proposed method and previous memory-efficient transfer learning methods. 
3. There are some general memory-saving techniques for deep learning, such as reversible neural networks [4,5,6], and checkpointing [7]. I think it is better to add discussions about these techniques.
4. For the experiment section, I think the pre-trained model should be the same for all compared methods to ensure fair comparisons. To get results under larger memory constraints, the authors can increase the size of side networks. 

[1] Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks, ECCV 2020

[2] Tinytl: Reduce memory, not parameters for efficient on-device learning, NeurIPS 2020

[3] Rep-Net: Efficient On-Device Learning via Feature Reprogramming, CVPR 2022

[4] The Reversible Residual Network: Backpropagation Without Storing Activations, NeurIPS 2017

[5] Reformer: The Efficient Transformer, ICLR 2020

[6] Reversible Vision Transformers, CVPR 2022

[7] Training Deep Nets with Sublinear Memory Cost
questions: See weaknesses. 
limitations: Limitations are discussed in the appendix. Discussions about negative social impact are missing.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes Ladder Side-Tuning (LST), a method for parameter-efficient transfer learning with pre-trained neural networks. LST is an extension of the concept of Side-Tuning [1], which is a previous work for avoiding full parameter updates by adding small target side networks. In contrast to Side-Tuning, LST fuses the source and target models in the intermediate layer outputs by weighted sum and trains the model with several engineering techniques such as initialization of the weight by structural pruning and layer-wise dropout. Experiments confirm that LST can achieve competitive performances to several previous parameter-efficient transfer learning methods while largely reducing memory usage during the training time.

```
[1] Zhang, J. O., Sax, A., Zamir, A., Guibas, L., & Malik, J. (2020, August). Side-tuning: a baseline for network adaptation via additive side networks. In European Conference on Computer Vision (pp. 698-714). Springer, Cham.
```
strengths_and_weaknesses: ### Strength
  * Reducing memory usage during training time in comparison to Adapters and Prompt Tuning.

### Weakness
  * Novelty is limited since the idea of fusing intermediate layers is overlapped with an existing work (LIT, [2])
  * The motivation and research questions are unclear.
  * Lack of comparisons with baselines such as knowledge distillation, which are important for assessing the practicality of the proposed method.
  * Rather increasing memory usage during inference time.
  * Unclearness of experimental settings in particular the experiments of Fig.8.


LST is certainly effective in reducing training parameters and memory usage, but its practicality can not be judged based on the current contents because there is no comparison with more simple and lightweight methods such as linear probing (a transfer learning method updating only the target-task-specific layers) and knowledge distillation. Furthermore, the main idea of LST which is to fuse intermediate layers of source and target models is not novel since LIT [2] has already proposed it at ICML2019. Considering the above, we believe that this paper is not yet ready for publication because the weaknesses outweigh the strengths.

```
[2] Koratana, A., Kang, D., Bailis, P. &amp; Zaharia, M.. (2019). LIT: Learned Intermediate Representation Training for Model Compression. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:3509-3518 Available from https://proceedings.mlr.press/v97/koratana19a.html.
```
questions: ### Q1. Why is memory usage during training important?
The motivation explained in the Introduction is not clear to us. As far as we know, parameter-efficient transfer learning methods using large pre-trained models are motivated not only by the reduction of update parameters and memory usage but also by the prevention of overfitting [1,3,4]. There is also a focus on speeding up inference time for classical parameter reduction methods such as knowledge distillation [5] and pruning [6,7]. On the other hand, this paper focuses only on the limited issue of memory usage during training and does not consider the other important motivations enumerated above. We also consider that even the problem of memory usage has not been completely solved, since the experiment in Table 1 when using a huge pre-training network (T5-3B), ended up exceeding 16 GB, which is the standard for memory-saving environments in other evaluations. More to the point, in business, the training phase has relatively usable resources compared to inference, so providing an A6000 with the 48GB of memory used in the experiments in this paper for training is not much of a cost. Rather, LST or other methods adding parameters increase the cost of inference and cannot be used in resource-sensitive environments during inference. In summary, the research question for this paper is unclear and thus we are unable to determine the impact or accomplishment of the reported results. Please give us a specific practical scenario where LST would be useful.

```
[3] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177–2190, Online. Association for Computational Linguistics.
[4] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565–576, Online. Association for Computational Linguistics.
[5] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 2.7 (2015).
[6] Li, Hao, et al. "Pruning filters for efficient convnets." International Conference on Learning Representation (2017).
[7] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.
```

### Q2. Why not compare LST to other parameter-efficient methods?
This is a related question to Q1.
One of the most popular methods for transfer learning with reduction of training parameters and amount of memory is distillation. A method particularly related to LST is FitNets[8], which minimizes the Kullback-Leibler divergence between the intermediate layer outputs of source and target models. The difference between LST and FitNets is whether the output of the source model is directly used as the output of the target model. In contrast to LST, the trained models with FitNets do not require the source model during inference time. Is LST more effective than simple and lightweight transfer learning methods like FitNets?
Furthermore, the more related study is LIT [2], a prior study that shares a source-target intermediate layer but also distillation. Is there any technical difference between LIT and LST other than the detailed learning techniques (initialization and Layer Dropout)?

```
[8] Romero, Adriana, et al. "Fitnets: Hints for thin deep nets." arXiv preprint arXiv:1412.6550 (2014).
```
### Q3. What is the definition of "network-pruning" in Fig. 8?
The definition of "network-pruning" in L301-L303 is ambiguous. In which strategy is this model pruned in Fig. 7? What is the reduction factor for pruning? Is the pruned model being fine-tuned?

limitations: Nothing to report.
review_title: null

Meta-review: metareview: The paper contributes a novel training methodology based on a ladder network for a scenario of fine-tuning under limited memory constraints.

Two out of three reviewers recommended rejecting the paper. Reviewer L9hL noted that the paper is related to distillation or network prunning and is not discussed sufficiently. The method is related but not the same as distillation or network prunning. Both of these techniques require an additional computation step. I agree with Reviewer tNTu that it would strengthen the paper to make a one-to-one comparison with representative distillation or network prunning methods, but I also feel it is not strictly necessary as they have (slightly) different use cases. Hence, I feel it is more of an issue with writing, than with the core contribution.

It was also brought up that Side Tuning is a similar method from '19. My understanding is that Figure 8 and the rebuttal properly discusses this point and shows that the design choices made by the Authors (e.g. injecting the representation at multiple stages) are crucial innovations, and do not diminish novelty in my opinion.

All in all, I believe this is a solid contribution that is another tool that will help democratize large-scale models. It is my pleasure to recommend acceptance. Please remember to address reviewers' remark, and please pay special attention to better contextualizing your work in the broader field of memory efficient training of neural networks.

confidence: Less certain
Decision: Accept