Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes methods to improve the efficiency of large pre-trained models (millions or billions of parameters) by combining multiple notions of _sparsity_ and/or _dimensionality reduction_ on the model weights. This is achieved via (1) a low-rank update matrix with fine-tunable parameters, and (2) pruning the original pre-trained weight matrix. These two sets of parameters are combined during inference. The authors validate their proposed method on several model architectures and datasets.
main_review: Strengths:
- The topic of this paper is timely. Models are growing to immense sizes (at faster and faster rates), and the community is starting to come to a reckoning with their computational (and storage) budget. Reducing the impact both from a memory footprint, ability to fine-tune, and perform efficient inference can have immense impact.
- As an empirical result, some of the findings in this paper points towards the proposed method being worthwhile to try in practice.

Weaknesses:
- In general, the motivation and exact contributions of this work are hard to follow. In fact, I found it hard to understand what exactly the proposed method (and inference mechanism) is without first reading LoRA, the paper on which this work is based. For example, how outputs are even calculated are not explicitly spelled out (via equations) until Section 3.3. Personally, I also find the notation of $\mathcal{W} + \Delta\mathcal{W}$ to be slightly confusing when also talking in the context of "sparse updates" etc: it makes me think that $\Delta\mathcal{W}$ is a repeated gradient update of $\mathcal{W}$.

- It seems like a core deviation of this work from an ensemble of prior techniques is the addition of the sparse residuals. As such, it seems appropriate that this should be highlighted more/expanded on in the main introduction for this work. On this note, I'm still not that convinced by the motivation, particularly on identifying the subspace $\Omega$ from the pre-trained $\mathcal{W}$. Can you provide more explanation for why you assume $\Delta\mathcal{W}$ shares this subspace? Or why $\Omega$ isn't computed from the _sparse_ version of $\mathcal{W}$? Compared to random (Fig. 2), the performance gap seems fairly small, and is inconsistent across scales.

- It's hard to judge the practical benefit of the proposed approaches simply by looking at the number of trainable parameters and the number of flops. Actually exploiting these reductions to the full extent is hard (as the authors noted as well). It's also not really clear where the frontier is on total parameters vs. trainable parameters vs. accuracy is. Is it possible to give some real efficiency (i.e., wall clock) times? This would help shed some more light on the actual benefits vs. remaining challenges (how to actually implement this real-time). 

- The benefit of $\mathcal{S}_2$ seems dubious. The differences are quite small, which makes me wonder if it is worth the extra complexity. 
summary_of_the_review: Overall, I think that the topic the paper tackles is important and timely. The paper proposes a compelling combination of existing techniques to increase various aspects of efficiency. That said, unfortunately, in its current version the paper is not so clear to follow (in motivation and implementation). In terms of technical novelty, it seems that the added components over LoRA might need a bit more work for their performance gains to be more convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: * This paper proposes a Dually Sparsity-Embedded Efficient Tuning (DSEE) which enforces sporty-aware weight updates on top of the pertained weights, and encourages a sparse weight structure towards the final fine-tuned model. On a diverse set of backbones and datasets, DSEE achieves 35% inference FLOPs savings with <0.1% trainable parameters without loss of accuracy.

main_review: Strength:
* Developing more parameter-efficient and resource-efficient NLP models is important
* Experiments show promising results

Weakness:
* Language and presentation of the paper can be improved. For example, after reading the introduction, it is still quite confused for me to understand the method and motivation because of confusing word usages — what exactly is the relationship and difference between “sparsity-aware weight update” and “sparse final weight”? 
* Experimental results can be better validated. Compared to LoRA, they claim the advantage of DSEE is that the “burden on resources such as storing the pertained weights are relieved”. To further validate this claim, it will be insightful to also compare with LoRA applied on sparse pertained model (i.e. first apply pruning on pertained model, then apply LoRA). There is some discussion on FLOPs in e.g. section 4.1, but it will be more clear to have a complete table on training/inference time and memory.
* Technical novelty is somewhat limited, or at least can be better justified.


summary_of_the_review: This paper addresses an important problem and shows some promising results to improve the efficiency of NLP models. However, the presentation of the paper can be improved and there should be more justification of the motivation and novelty of the method. In particular, it seems that the main difference between this method and LoRA is the sparsity of pertained models, which is achieved mostly with existing techniques, yet it seems that LoRA (and/or similar method) can also be applied on top of sparse pre-trained models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tries to solve both efficient fine-tuning and model size compression simultaneously. It adopts the previous framework to model W and \delta W together and use different existing approaches to learn 2 parts in order. 
main_review: Strength:

1. The paper is presented in an easy to read manner.

2. I believe it's an important research topic to work on which can have a broad impact.

Weakness:

1. The organization can be improved. I feel the main motivation is actually in section 3 (page 4) 

"However, as revealed experimentally by (Yu et al., 2017), a part of the important information in the trained weights will
also scatter outside......"

but authors didn't put it in the introduction. This makes me feel the method is not well motivated. And consequently I think the proposed method is a bit ad-hoc. There are many details of thinking not well explained so maybe it can be an interesting method but currently it reads to me that it's fairly incremental. I think to make me increase the score, the introduction should introduce more rationale of previous methods and why the proposed method can improve.

2. I believe the authors overlook an important work in the line of efficient fine-tuning (https://arxiv.org/abs/2101.00190). I didn't run the code myself but one of my collaborators told me this method in practice is fairly competitive without any effort of parameter tuning. I think the comparison to the work should be added.

3. I think listing #trainable is not a fair comparison. To best of my knowledge, storing the sparse coordinate information requires additional coding of the row/col of the scalar which is omitted by most sparse-pruning papers. But when it comes to comparison of low-rank methods, it will create an unfair situation as low-rank methods doesn't require additional memory usage so maybe other researchers doesn't put emphasis on this but I insisted the memory footprint information should be represented in the result tables.

4. Also, sparse method is in general slow without hardware accelerator. I think authors also know this as they wrote in the paper:

It applies fine-grained manipulation on each individual model weight, while the generated irregular sparse masks bring limited hardware speedup (Wen et al., 2016). 

But somehow I didn't see any actions to cope with this and they just listed FLOP reduced which is again unfair to other structured methods such as structure pruning and low-rank. I think it's ok to require extra hardware accelerator. But to really benefit the research community, it's important also to present the weakness of method and I believe it's important to show the real wall-clock inference time instead of just the simulated #FLOP reduced. It's also interesting to know what's the wall-clock time saving in efficient fine-tuning part. As the low-rank method in general is easy to train and the proposed method seems to have many steps. So it seems to me that it's possible low-rank methods will have a bit more parameters but converges faster so the real time saving is better than the proposed method.

In fact, I think the empirical experiments doesn't really show a significant improvement over previous methods. For example, in Table 4 LoRA with  0.39M outperforms DSEE 0.17M and honestly 0.39M really doesn't increase too many parameters compared to DSEE. So I think the justified the efficacy of each step in DSEE and frankly present strength/weakness is rather important.

5. This leads to my concern that each functionality of the method is not well discussed. In particular, the ablation study of solving eq 1 looks weird to me. As an apparent way to solve eq 1 is via iterative methods. Fix S2, UV can be optimally decided by SVD and fixing UV S2 can also be determined easily. So to me I think iteratively solving UV and S2 is the best candidate to compare but it's not appearing in the paper. And we don't even know what's inside the GreBsmo algorithm as authors didn't explain it at all. That confused me a lot. 

And if we look into the ablation study in Figure 2, there are some cases where other heuristics actually outperformed the DSEE. It's important to study why such cases happen but author didn't touch this part. I feel this then make the experiments not able to justify the proposed method.
 


summary_of_the_review: Overall, I think the paper present an easily understandable method to an important and interesting question. But it reads to me many details are hidden or omitted so the efficacy of the proposed claim cannot be justified properly. I didn't list all of my concerns but at least the above mentioned five points should be addressed in order for me to increase the ratings.


Post rebuttal:

After author's reply, I still cannot confirm there is significant technical contribution. In addition, despite authors posted some additional experiment as requested, the numbers cannot match the original paper so I am not convinced the empirical result as well. In sum, there is no enough argument that the proposed method should work or any explanation why it should work. On the contrary, the experiments suggested that initilization plays important roles which might imply the claimed contribution might not be the real meat. Empirical evaluation also don't support the claim so I decide to remain my original ratings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a principled way for achieving both parameter efficient finetuning and resource efficient inference. They combines the two steps of (i) computing a low-rank decomposition of parameter updates with a sparsity matrix (ii) further prune the trained (ie., finetuned) model. The first step aimed to achieve the parameter efficient finetuning but w/ the sparse residual components based on sparsity from the pretrained model to preserve the performance. The second step can reduce the memory of parameter storage. On three benchmarks with three different model architectures such as encoder-only, encoder-decoder they a large number of experiments with ablation studies. Studies show several a number of promising enhancement over other baselines.  
main_review: This paper presents an interesting direction of achieving two different goals simultaneously. Below is my comments on this. 

Strength:
1) Clearly defines the problem, goal and most part of the method
2) Although not fully novel to scratch, the present an interesting idea of achieving both parameter and resource efficiency. They uniquely studied the problem of this and presented a way of making it work.   
3) Multiple solid experimental results with a diverse set of models, benchmarks, and metrics showing the generalization of their approach. 

Weakness:
1) My major concern is the evaluation. It is difficult to combine the experiments to support the claims. The gains in performance are marginal in compare to LORA, where LORA is a simpler method. With the S1 matric in Table 6, it even became worse in comparison to LORA or the finetuned model. The main claim is to excel for the large models but as found, with large models like DeBERTA,  the gains in performance is even lower. The FLOPs results are extremely insufficient to stand as an evidence (only on SST-B are reported). 
2) It's not described why S1 is only applied to pretrained weights W 
3) Unclear when to use unstructured or structured sparsity. 
4) Definitely just having a small reduction in memory usage does not mean to a significant resource utilization as it may lead to performance drop or may take an increased amount to training time. Analogously here, searching the sparsity also needs addition overhead (e.g., more epoch to tune UV after pruning etc.) the corresponding gain in performance is incremental specially when compare to the baseline LORA approach. 

Typos and tips:
1) 1/3 heads from each attention head -> 1/3 heads from each attention leyer.
2) reset the values of S2 back to 0. During the update, we only update the elements of S2 whose indexes are in Ω -> this should be in correspondence to Algo-2. 
3) More details how UV is computed without seeing pretrained weights W should be discussed (not just referring previous works) 
4) FLOs on BERT -> FLOPs on BERT



summary_of_the_review: Results with initial success but rejected as further evaluation is needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to enforce weight sparsity constraints into a prior parameter-efficient tuning method (LoRA) to achieve both parameter-efficient tuning and resource-efficient inference. Specifically,  this paper (1) adds another sparse, learnable weight matrix to the original low-rank decomposition in LoRA; (2) incorporates unstructured and structured weight pruning techniques to save computation at inference time. Experiments are conducted on GLUE and several data2text generation benchmarks to demonstrate the effectiveness of the proposed method.

main_review: The strengths and weaknesses are detailed below:

*Strengths:*

1. Combining weight-pruning and parameter-efficient tuning is a valuable attempt to further improve resource efficiency especially at inference time

2. Experiments demonstrate the effectiveness of the proposed method. The analysis sections regarding DSEE are also appreciated.

*Weaknesses:*

1. (This is a minor sanity check question) On GLUE tasks, are the results from one run or several random runs? This detail seems not to be mentioned in the paper – it is common practice to reports results from several random runs since the GLUE results are known to be sensitive to initialization 

2. The necessity of adding a sparse matrix (i.e. S2 in the paper) into LoRA is not well supported. The justification experimental results in Table 1 and Table 2 are not very convincing to me because (1) the improvement from “+S2” is very small in most cases (and sometimes worse), for example, I do not think 0.008/0.003 are meaningful differences to claim in the paper; without significance test, most of the small differences may not be trustable; (2) In Table 1 and Table 2, many results from UV of 2x sizes (i.e. 589.8K in Table 1 and 0.39M from Table 2) are better than UV + S2. I know that this is not a fair comparison because of different models sizes, however, I suggest the authors double the size of UV + S2 for a fair comparison instead of downsizing UV – on the scale of 589.8K or 0.39M parameters, the number of parameters is not really a practical concern anymore. For example, your method can tune 1M less parameters than others, but such a 1M parameter saving may not lead to any meaningful differences in most practical settings.  
Therefore, the sparse S2 adds complexity to the method and requires additional algorithm 1 in the pipeline, yet it does not yield very effective improvement as argued above, thus I do not think this is a proper design.

3. Without considering the questionable contributions of S2 because of point 2, I think the remaining technical contributions of this paper are limited because it is a combination of the existing parameter-efficient tuning method (LoRA) and existing weight pruning methods. Such a combination is empirically valuable but technically trivial.

4. The improvement of DSEE over LoRA seems from sparsity in pretrained weights, it is better to explain this phenomenon in the paper why weight pruning leads to a significant performance boost.

5. There is a sentence above Section 4.2 “DeBERTa is a larger model so applying low-rank decomposition with our hyperparameters cannot reach the performance of fine-tuning.” This statement is misleading and contradicts the claims by other papers [1] [2], which conclude that a larger model typically requires a lower-dimensional parameter update space.

6. The are some minor presentation flaws:  
(1) In the first paragraph of the introduction section, citations to related papers should be included such as when mentioning BERT and GPT2.   
(2) Above section 4.2, the authors say that Table 5 uses SST-2 …. which are not present in Table 5


[1] Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021  
[2] Aghajanyan et al. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021

summary_of_the_review: Some core method design is not well-justified and the technical contributions are limited. There are some valuable empirical contributions though.
review_title: null

Meta-review: The paper integrates several dimensionality reduction and sparsity methods for improving the efficiency of large pre-trained models. Overall, the paper is interesting and discusses an important topic. However, it seems that it is not ready to be published at the current stage. I would encourage the authors to take reviewers comments into account and further improve the paper 

The pros and cons of the papers are summarized in the following: 

Pros: 
+ Improving the efficiency of large pre-trained models is an essential research issue. 
+ The idea is interesting although the technical novelty is a bit limited. 

Cons: 
- The key concern is that the technical and practical benefit of the proposed approach is not clear based on the results demonstrated in the experiments. 
- The writing of the paper can be further improved in general to make the motivation more clear.
Decision: Reject