Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes MARVEL, a parameter-efficient fine-tuning method. MARVEL parameterizes the incremental updates of the pre-trained weight matrices in the form of SVD for parameter-efficient fine-tuning. Besides, it allocates the parameter budget adaptively according to the importance of modules to improve the final performance. The authors conduct extensive experiments on Natural Language Understanding (NLU), Question Answering (QA) and Natural Language Generation (NLG) tasks. Results show that MARVEL outperforms existing approaches like BitFit, LoRA, etc.
strength_and_weaknesses: *Strengths*

1. The paper is well-organized and easy to follow.
2. The proposed algorithm is well-motivated. Specifically, previous work like LoRA treats all parameters equally, but actually some parameters contribute more to the final performance and thus should be updated in higher priority when the budget is limited. Therefore, the paper proposes to adaptively allocate the parameter budget among weight matrices according
to their importance score, which improves the model performance.
3. Experiments conducted on various tasks including NLU, QA, and NLG demonstrate that MARVEL can achieve better performance compared with baselines without adaptive parameter budget allocation.

*Weaknesses*

1. The experiments are only based on two backbones (DeBERTaV3-base, BART-large). It would be better to provide more results on other backbones, especially on large-scale models, e.g., T5-3B, to show the generalization of the proposed method to larger models.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: In general, the paper is well-written and techniquely-reasonable. Extensive experiments demonstrate the superiority of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for adaptively allocating fine-tuning parameters during transfer of pre-trained models to downstream tasks.  The authors propose an SVD inspired decomposition of the adapter matrices and develop various importance scores to assess which triplets in the SVD decomposition are removable. This allows adaptively tuning the ranks of the adapter matrices across layers. The authors demonstrate empirical gains from using their method.
strength_and_weaknesses: **Strengths**
1. The idea of adaptive computation budget allocation is interesting and highly relevant for using pre-trained models for downstream tasks in a memory efficient fashion
2. The experiments, though limited in some respect, are quite extensive in others. 

**Weaknesses**
1. Missing Baselines / Experiments
     1.  MAM [1] seems to be the current state of the art but this was not compared against though it is referenced in the paper.
     2.  What is the training time cost of  Marvel compared to the non-adaptive methods ? It seems that Marvel adds non-trivial overhead compared to the simpler non-adaptive methods.  If the training overhead is significant, it might hinder Marvel's applicability, just like the authors mention with Diff Pruning. 
2. Some Missing ablations 
    1. Impact of orthogonality constraint - would be good to see ablations with and without this constraint
    2. Is the success of the gradient-weight product (as a sensitivity measure) possibly coming from the exponential smoothing ? Was exponential smoothing applied to the simple singular value magnitude measure (?)
    3.  LoRA version as presented has no regularization applied to the learned AB matrices.  It is possible that the gain produced by the current method is coming from “effective regularization effect"  of the global budget scheduler ?  Could LoRA be run with varying regularization (say L2) on the AB matrices  to ensure that such a regularization effect isn't indeed the cause of gains.


**Nits**
1. Figure 1
    1. We are not giving the reference result for full fine-tuning so it’s hard to put this figure in context
    2. Would be good to also include results for when budget is split equally across all layers / types of parameters

[1] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards
a unified view of parameter-efficient transfer learning. In International Conference on Learning
Representations, 2022. URL https://openreview.net/forum?id=0RDcd5Axok.
clarity,_quality,_novelty_and_reproducibility: *Clarity* :  Method described is clear

*Originality* : Method is original in the context of previous work as far as I can tell but approach  could be considered incremental

*Writing Quality* :   Writing quality can be improved - especially introduction - too many details given in introduction which late feel repeated in later parts of the paper   

summary_of_the_review: I think the ideas in the paper are solid. However, there are several missing empirical elements that I would need to be fully convinced that the method works as advertised (significantly improves downstream performance at same overall budget compared to simpler non-adaptive methods).
Conditioned on adequate responses being made to the weaknesses outlined above, I would be willing to raise my score.

****** 

Score updated after responses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes MARVEL, a method for parameter-efficient fine-tuning on big pre-trained language models. MARVEL is largely based on LoRA, a method that fine-tunes low-rank decomposition on parameter matrices. MARVEL uses SVD and dynamically allocate rank to each layer as resources, and it outperforms LoRA consistently with both DeBERTa and BART on natural language understanding and natural language generation respectively.
strength_and_weaknesses: ## Strengths

1. Parameter-efficient fine-tuning is a topic of significance in the application of big pre-trained LMs to real-world problems.

2. MARVEL uses SVD and dynamically allocate rank to each layer as resources. It is a novel technical contribution.

3. The paper compares MARVEL with multiple baselines on very well-established NLP models (DeBERTa, BART) and benchmarks (GLUE, SQuAD, CNN/DM, XSum). In these experiments, MARVEL outperforms other methods, especially LoRA, consistently.

4. This work includes an analysis section, where various design choices of MARVEL are compared. It helps readers understand how MARVEL works. The figure in Appendix B is also provides great insight on how different layers of big LM contributes to its downstream performance.

## Weaknesses

1. The method, MARVEL, is based on LoRA. Most important comparisons in this paper is comparing MARVEL against LoRA. The contribution of MARVEL is mostly incremental and empirical.

2. MARVEL introduces many new hyperparameters compared with LoRA (see Section 4.1). How could you rule out the possibility that a **larger hyperparameter search space** is what leads to a better performance? Also, MARVEL is considerably more complicated than LoRA, so fine-tuning with MARVEL could be slower than fine-tuning with LoRA. However, the **training time** is not compared in this paper either. To address these two concerns, could you show the time required for each run (including full hyperparameter sweep) of both MARVEL and LoRA?
clarity,_quality,_novelty_and_reproducibility: This paper is clear. The quality is very high. The technical contribution is incremental, but the empirical contributions are significant and new. The codebase required to reproduce results are all provided in supplementary materials.
summary_of_the_review: This paper proposes MARVEL, a method for parameter-efficient fine-tuning on big pre-trained language models. The topic is of significance. The method is based on LoRA, so technical contribution is incremental. However, the paper includes very solid experiments showing that MARVEL outperforms LoRA and other baselines consistently. There is a minor soundness problem: some experiments could be unfair against LoRA because training time is not compared. This paper is overall solid so I would like to give a borderline accept recommendation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: * This paper proposes a novel parameter-efficient fine-tuning method called MARVEL, which adaptively allocates the parameter budget among weight matrices according to their importance score. Specifically, it proposes to use a learned approximation of SVD decomposition, and prunes out singular values of unimportant updates. Learned approximation of SVD circumvent intensive exact SVD computations, and importance of singular values is defined by a novel metric correlating to the contribution to the model performance. It additionally proposes a global budget scheduler to facilitate training process.

strength_and_weaknesses: * Strength
    * The evaluation is extensive. The paper compares with a comprehensive set of baselines and evaluates on a variety of tasks (language generation, GLUE, and question answering).
    * The paper achieves great experimental results, e.g. achieving 1.2% F1 improvement on Squad2. With less than 0.1% trainable parameters.  
    * This paper proposes a novel approach for parameter-efficient finetuning, and can have wide applications.  
* Weakness
    * It will be great to also provide numbers of training/inference memory/time. Even though it's faster than SVD, is the training time longer than other methods? What are some trade-offs?

clarity,_quality,_novelty_and_reproducibility: The paper is written in clarity. The method is novel -- it might be a bit niche. The paper seems to have great reproducibility, with detailed hyper-parameters both in the paper and appendices.
summary_of_the_review: This paper presents a novel method that achieves better efficiency without much loss of accuracy, compared to a comprehensive set of baselines, on a variety of tasks. However, the novelty of the method is somewhat limited and there can be potential drawbacks such as longer training time. So I recommend weak acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a parameter-efficient fine-tuning method for pre-trained language models. The method first approximates the gradient update of weight matrics by low-rank factorization and uses a sensitivity-based importance score to prune unimportant triplets. Experiments on NLU, QA, NLG tasks show the effectiveness of the proposed method compared to other methods with the same amount of fine-tuning parameters.
strength_and_weaknesses: Strength:
- The paper is well-written and easy to follow. The research problem is important as pre-training LMs are growing in scale, resulting in the difficulty in fine-tuning all model parameters. 
- The method is simple, straightforward, and effective. 

Weaknesses:
- This paper mainly compares methods under the same amount of fine-tuning parameters. There is no comparison in terms of FLOPS/training time.
- Besides, it's also unclear about the speedup gain v.s. the overhead of computing the sensitivity score w.r.t. the whole training process. 
clarity,_quality,_novelty_and_reproducibility: - Not all parameters take the same amount of FLOPS for updates. For example, computing the gradient of weights in the upper layers would be faster than computing those in the bottom layers (due to backpropagation). It's unclear how the proposed method speeds up the training in terms of FLOPS (or training time) compared to other methods.
- The main purpose of efficient tuning is to speed up the fine-tuning process and maintain comparable performance to full model tuning. Note that **controlling model parameters is just one way to do so but not the main target**. In Algorithm 1, we have to first update P_k and Q_k (Line 7), and then take a gradient update to get the diagonal matrix (Eq. 5), and finally prune by Eq. (6). Computing the gradient updates is the most time-consuming part. If we have already spent time computing the gradients of P_k, Q_k, \Lambda_k (Line 7-8), **what's the point of pruning?** Note that just controlling the no. of fine-tuning parameters is not the main focus, it does not justify the training efficiency purpose. The overhead of computing the sensitivity score (Line 4-6) and pruning (Eq. 6) may just make the whole training process slower compared to the training process only doing the gradient updates of P_k, Q_k, \Lambda_k.
- It is claimed that one of the advantages of MARVAL is to maintain the singular vectors for potential future recovery. In which situation should we recover these triplets? There is no support for this statement in the experiment.


summary_of_the_review: The motivation for adjusting the rank of weight matrics is not very clear. Experimental results in terms of FLOPS (or training time) are needed to show the efficiency of the proposed method.

------------------
The score is updated after the responses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a space-efficient method for finetuning large LMs. The proposed method is a simple extension of the LoRA method: (1) instead of decomposing the update matrix into general A*B, the authors use SVD matrices, which allows reducing the matrix size rather partially than all or none. (2) They propose an importance ranking method for deciding which SVD values to prune in which matrix. The overall method obtains very strong results: it outperforms the LoRA method, as well as several baselines, and even the baseline vanilla fine-tuning.

strength_and_weaknesses: Strengths:
- A simple and very space-efficient FT method. 
- Very strong empirical results. 
- The method is quite simple (though there seem to be a lot of important details)

Weaknesses: 
- An important ablation seems missing: running LoRA with the proposed SVD decomposition rather than the AB one. This would help tease apart the two proposed contributions (the SVD part and the importance part). 

- The writing can be repetitive at times, but still a bit cryptic in other places. In particular, I had to read the SVD part several times to understand the differences between the method and the LoRA method. I still have a few questions, see below. 
- The authors only report validation results as far as I understand. Reporting such results is important for reproducibility, but does not replace reporting test results. Given the large number of small decisions made in developing the proposed method, it is particularly important to evaluate it on a truly held-out set.

Questions:
- How much does the initial stage (Sec. 3.3) increase the cost of fine-tuning (e.g., in runtime)?

- "All the gains have passed significant tests with p < 0.05." -> Which statistical test was used?

- "such that the pruned triplets can still get updated within these intervals and possibly reactivated in futur iterations." -> maybe I am missing something: once the warmup stage is over, and the budget is fixed, do importance scores change? In my understanding, the importance of a given singular value cannot change since it is always pruned. Or am I missing something?

- Table 1: Why does the bitfit baseline use far fewer parameters than all the others? 

- What are the NLG tasks? Summarization? What are the evaluation measures? These are very important details, NLG is not a task.
clarity,_quality,_novelty_and_reproducibility: The writing is generally clear (though see above). The intro seems a bit long and could be shortened. E.g., the paragraph starting with "Adding more trainable parameters" is a bit repetitive. Same goes for the background, e.g., no need to formally define transformers, Eq.2 repeats Eq.1, etc.

Typos and such:

- the product of two *samll*

- we present *an* concrete example 

In terms of novelty, the proposed method is a novel (as far as I know) combination of known techniques.
summary_of_the_review: A simple yet effective extension of the LoRA method for space-efficient fine-tuning of LMs. The proposed method performs very well on several benchmarks. There are a few clarity issues and some methodological concerns, such as an important missing baseline. Overall I am supportive of accepting this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: 

This paper proposes a parameter efficient fine-tuning method which adaptively allocates the parameter budget among weight matrices.  It uses a SVD decomposition of the incremental updates of the pre-trained weight matrices, and prune unimportant triplets according to their importance scores.  Experiments are conducted on Natural Language Understanding, Question Answering and Natural Language Generation tasks which show the effectiveness of the proposed method compared to other methods with the same amount of parameters.

Strengths:

The research problem is important for applying big LMs in real applications. The proposed method is quite simple but effective and novel.  The experiments are conducted  with a comprehensive set of baselines and evaluates on a variety of tasks and show it can achieve better performance compared with baselines.   This work includes an analysis section.  The paper is well-organized and easy to follow.

Weaknesses:

The main concerns from the reviewers are on experiments, for example, more backbone models could be used and the computing resources and memory/time cost on training and inference should also be compared.  How are the hyperparameters be selected?  The writing could be improved in some parts (the SVD part for example).

justification_for_why_not_higher_score: The comparisons could be done with more backbone models and in more dimensions (time/memory/FLOPs...).
justification_for_why_not_lower_score: The paper addresses an important problem and proposed a novel yet simple solutions.  Experiments show its effectiveness.
Decision: Accept: poster