Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work studies the problem of parameter-efficient transfer learning in the paradigm of pretraining/finetuning. The proposed method, diff pruning, can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5 of the pretrained model's parameters per task.

I don't get the problem definition of this work. In other words, the authors need to better motivate and justify parameter efficiency. Given that all the parameters of a big model are used in downstream tasks, what are the benefits of only modifying a few parameters? Here are several possibilities.
1. Does it speed up the finetuning process by modifying a few parameters? 
2. Does it speed up inference for downstream tasks?
3. Another possibility is model size reduction, which leads to reduced storage costs. "This approach can become parameter-efficient as the number of tasks increases as it only requires storing the nonzero positions and weights of the diff vector for each task. The cost of storing the shared pretrained model remains constant and is amortized across multiple tasks." Unfortunately, in real-world scenarios, inference of  large models like BERT_base/large is usually conducted on servers or in cloud, where storage is not a big issue while latency (speed) is critical. Inference in edge devices like phones, IoT sensors cares about storage, speed, and power consumption. 
review_title: Concern about problem setting

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work combines model pruning with transfer learning/multi-task learning in NLP. Instead of finetuning pretrained models on each individual task, the author propose to learn 'residual' parameters with sparse masks for each task independently, hence reducing per-task parameter requirements. The evaluation on GLUE shows some promising results with better efficiency compared to adapter networks. However, I am concerned with the motivation and applicability of the proposed method, such that I am left with the impression that the method may be hard to use in practice.

Pros:
1. To the best of my knowledge, combining pruning on pretrained models to transfer learning in NLP is novel.
2. Empirical results show the method can performance comparably with adapter networks but with fewer parameters.

Cons (see below for detailed questions):
1. Although the method is proposed for 'multi-task' learning, it is not really evaluated on multi-task settings. The motivation and applicability are not very clear to me.
2. Compared to adapter networks, the proposed method contains more hyper-parameters and is harder to tune in practice.

Questions:
1. In section 2, the author introduced multi-task learning as the background. However, the proposed method is not really designed for multi-task learning (training multiple tasks simultaneously), but rather for transfer learning on multiple tasks independently. In practice, however, training some tasks together can improve performance (GLUE for example), but the proposed method is trained for each task independently and thus there is no positive transfer between tasks. So have you evaluated your method when trained multiple tasks simultaneously (where the shared parameters are jointly finetuned)? How does that work? This is an important extension that might be useful for the community.
2. Please point it out if I miss this information, but how did you select your hyper-parameters for your approach? Besides, how sensitive they are compared to adapter networks?
3. Prior work [1] has shown the resulting mask of pruning can be compared to evaluate task similarities. Did you have similar observations in your experiments?

Missing reference:
The idea of adding 'residual' parameters for new tasks is not new in lifelong learning. It is good to mention [2] and [3]. [1] is also related of comparing pruning masks for pretrained models.

Typo:
Page 4, last line: QQC -> QQP

[1] On negative interference in multilingual models: findings and a meta-learning treatment. Wang et al., EMNLP 2020.

[2] Progressive neural networks. Rusu et al., arxiv 2016

[3] BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. Wen et al., ICLR 2020.
review_title: Interesting results, but better motivation and more experiments can further improve

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present an interesting approach to learn task-specific models with only a few tunable parameters. They propose learning a diff vector with a sparsity constraint and then pruning the vector using magnitude pruning. They also impose a structured sparsity constraint by introducing a group penalty.

This work is interesting and important with the growing size of the pretrained models. It enables the model to learn a new task with relatively few parameters would be very beneficial. The experiments and analysis conducted by the authors is thorough.

A few questions/thoughts which could improve the paper: 
* It would be good to list the epochs for training for each of the different approaches? Authors mention 2,3,4,5 epochs for training but would good to highlight the cost or savings of fine tuning with their approach.
* In table 1, it would be also helpful to include some information about the finetuning steps/sec with the different approaches. This would help understand the tradeoff of memory vs compute for the proposed approach.
* What’s the intuition for structured pruning performing better than non unstructured? Typically, enforcing some structure should have worse accuracy. If the structure seems to help, then the model should be able to learn structure without the group constraint as well.

Overall, I recommend accepting the paper. As the size of pre-trained models is growing quite rapidly, research that investigates parameter sharing and adapting a pretrained model to a new task with few parameters is essential.  
review_title: Interesting approach to reduce task specific parameters in transfer learning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes diff pruning, an alternative paradigm for parameter-efficient transfer learning of pre-trained models. Similar to adapters, diff pruning leaves the body of the pre-trained model unchanged. Rather than inserting additional task-specific parameters into the pre-trained model, diff pruning adds reparameterizes the parameters of the transferred model $\theta_\tau$ by adding a diff vector $\delta_\tau$ to them: $\theta_\tau = \theta_{\text{pretrained}} + \delta_\tau$. Parameter efficiency is achieved by regularizing $\theta_\tau$ to be sparse. The authors achieve this by using a relaxed mask vector to approximate the $L_0$ norm. They also propose a way to control for a specific sparsity rate via projection onto the $L_0$ ball after training and to enforce group sparsity that takes the model's structure into account. The approach is evaluated on the GLUE benchmark where it achieves competitive performance to full fine-tuning a BERT Large model and adapters while being more parameter-efficient than both of them.

Pros:
1. The proposed method is intuitive and the different modelling choices are principled and well motivated.
2. The method achieves strong results. It is competitive with full fine-tuning and more parameter-efficient than adapters, the prevalent approach for parameter-efficient transfer learning.
3. The authors show how to effectively control the sparsity rate and incorporating structure via group sparsity brings further gains.
4. The authors conduct extensive analyses, which pre-empted many of my questions, such as the variation across different sparsity masks, sparsity patterns across different tasks, etc. Overall, the analyses shed additional light on the characteristics and preferences of different tasks in transfer learning.

Cons:
1. The approach is potentially more complicated than the baseline, so it is important that the authors open-source their code.
2. The diff vector is distributed over the entire set of parameters of the model rather than focused in a few layers. This makes it potentially harder to combine the diff vectors from different tasks as can be done with adapters (see e.g. https://arxiv.org/abs/2005.00247) and to compose multiple diff vectors. 

Questions:
1. Does a visualization of diff vectors of different tasks (such as using t-SNE) reveal any interesting patterns?
2. Are there any transfer settings that the addition of task-specific parameters can model but inserting layer-specific transformations via adapters cannot (or vice versa)? Adapters have been used to transfer across modalities such as languages (see e.g. Pfeiffer et al. (2020), https://arxiv.org/abs/2005.00052) and I am wondering whether the same would be possible by adding task-specific parameters.
3. How long does your approach take to converge in comparison to the baselines? How much longer do you need to fine-tune with non-zero masks for magnitude pruning for sparsity control? What is the performance benefit of this further fine-tuning?
review_title: Review

Meta-review: This paper studies a problem setup of parameter-efficient transfer learning for large-scale deep models. The approach consists of learning a diff vector with a sparsity constraint and then pruning the vector using magnitude pruning. A group penalty is also introduced to enhance structured sparsity. The main motivation is that for each new task, we only need to add a few parameters based on a pre-trained model without fine-tuning it.

The proposed approach possesses technical soundness and shows empirical efficacy for the studied problem setup. During the rebuttal and discussion phases, two of the reviewers raised two major concerns based on which they strongly disagreed with acceptance:
- The problem setup is not elaborated sufficiently and falls short of plausibility. An approach targeting at efficiency should either improve inference speed or reduce storage cost. Unfortunately, neither advantage has been well approached.
- The technical novelty is somewhat incremental, given the rich previous work on residual adapter, network re-parameterization, and network compression (pruning, sparsity etc.).

AC read the paper and agreed that, while the paper has some merit such as a better model for the particular problem setup, the reviewers' concerns are reasonable and need to be addressed in a more convincing way. For example, try to study a practical application in which the proposed approach is essential and useful for efficiency enhancement.
Decision: Reject