Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the efficiency issue with visual parameter-efficient tuning (VPT), such as prompt learning and adapters. In particular, the idea is to tune only those important parameters of a pre-trained model per task, under an assumption that the importance of parameters is task specific. The proposed method is based on identifying the sensitivity of each parameters w.r.t a specific task based on previous model pruning methods. For performance, the whole parameters are still updated subject to a low-rank constraint for parameter efficiency, and further merge them into the backbone after fine-tuning. 

strength_and_weaknesses: **Strength**

- The problem of VPT efficiency is valid and valuable to be solved

- The idea of measuring the parameter sensitivity using network pruning methods is sensible and also novel in the VPT context. Technically, using the first derivative of Taylor expansion is a good choice with efficiency

- The results are strong in comparison.

**Weakness**

- It is some strange that the proposed method still update the whole parameters when a high proportion of parameters are sensitive to a specific task. 

- Method: It is largely similar to LoRA except an introduction of a threshold based choice /branch. SPT w/o unstructured is equivalent to LoRA. So it is very incremental over LoRA. 

- Experiment: Table 6 lacks comparison to LoRA.

clarity,_quality,_novelty_and_reproducibility: The writing is reasonably clear and fine to understand.

The novelty is some limited as the new bit of the whole model is identification and update of top most sensitive parameters, which needs to base on LoRA to work. As a result, the overall quality is some less conniving. 

The details given should be good for reproducibility. 
summary_of_the_review: This paper aims to adapt a pertained model to downstream tasks at no additional inference cost. The key idea is to find out what are top sensitive parameters for a given task, on which the update should be focused on. Previous pruning methods can be used to measure such sensitivity, and the final model needs to combine with LoRA according to some threshold. The overall method is some incremental over LoRA.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper assumes that not all parameters contribute equally to the performance across different tasks. This paper proposes a criterion to measure the sensitivity of parameters, which is the magnitude of the gradient. Reasonable results are reported.
strength_and_weaknesses: Strengths

S1. The idea is simple and easy to follow.

S2. The experimental results are acceptable.

Weaknesses

W1. The writing can be significantly improved. It is difficult to get what is new and what is significant in this paper. For example, after mentioning the criterion (Page 2), the authors commented finetuning only the important parameters (referred to as unstructured tuning) is not enough. The authors proposed to use LoRA (referred to as structured tuning). In other words, the proposed method only works in some cases and should be combined with LoRA. This significantly undermines the proposed method.

W2. The proposed method actually increases memory usage (Table 6), so why is that called efficient finetuning?

W3. As seen in Table 2, the proposed method is not impressively effective.
clarity,_quality,_novelty_and_reproducibility: The clarity and reproducibility are generally acceptable. I am not satisfied with the novelty because the proposed method is neither theoretically insightful nor practically useful. Please see my comments above.
summary_of_the_review: I rate it a rejection because the novelty is not satisfying. To me, the proposed method seems yet another LoRA with trivial modifications (i.e., in some cases we should select some parameters and finetune them in an unstructured way).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to perform parameter-efficient adaption for vision tasks, where only a small number of parameters are trained to adapt to new tasks (image classification tasks in this paper). To address this problem, the authors propose to exploit the estimated sensitivity to decide which parameters to be tuned, and the sensitivity is measured according to the gradients. Based on the ratio of sensitive parameters in the weight matrix, the authors use structured or unstructured tuning. In the experiment, the authors show that their proposed SPT can perform better using fewer parameters than the previous parameter-efficient methods in VTAB-1k and FGVC experiments. Additionally, they show some analyses, including using self-supervised pretrained backbones, using or not using structured tuning, and other hyper-parameter tunings.


strength_and_weaknesses: **Strength**
- The paper is well written, and most of the proposed components are easy to understand. The figures with a clear explanation help the reader follow the idea. 

- The proposed idea is novel and can potentially adapt to different datasets with task-wise numbers of trainable parameters. Applying structured and unstructured tuning in parameter-efficient learning is new to the community to the best of my knowledge. 

- The authors show extensive results on different datasets and pretrained models, and the following analyses confirm the effectiveness of the proposed method.

**Weakness**
- If I understand correctly, one potential weakness is the computation time of two-stage training. The proposed SPT requires an additional stage to estimate the sensitivity of each parameter based on the gradient and decide which parameters are decomposed into low-rank matrices. Then the proposed method tunes these chosen parameters to adapt to new tasks. This might make the computation time longer than simple one-stage training methods. 

- For $\Delta_w$ in equation 6, it is confusing when the authors explain  $\Delta_w$ means in the following paragraphs. First $(i,j) \notin P$ is confused (it might mean that these parameters are not in the selected parameters set, but this notation is quite confusing). My guess for what the authors try to express in the second line of Eq. 6 is that the unstructured tuning is applied on these weight matrices where only a few parameters need to be tuned. I would suggest the authors clarify this part.

- I would suggest the authors report the variance of reported results since the parameter-efficient method might be sensitive to the randomness in the training. 

clarity,_quality,_novelty_and_reproducibility: - Clarity and Quality: the paper is well-written and easy to understand, while there is only one part I would suggest the authors further clarify. 

- Novelty: the proposed structured and unstructured tuning is new to the parameter-efficient method to the best of my knowledge. 

- Reproducibility: the authors seem not to provide the code implementation, but it might be reproducible given the provided implementation details. 
summary_of_the_review: This paper provides a new parameter-efficient method for visual tasks, and they provide extensive results on different datasets and show the improvements against prior works. Given the current status of the paper, I would recommend the score "marginally above the acceptance threshold".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to first recognize influential visual parameters by its averaged gradients and then tune those parameters. In order to improve adaptation capability, the paper also propose to switch to a low-rank reparameterization of the weight matrix that contains enough sensitive parameters. The paper evaluated on both FGVC and VTAB dataset and achieve even better results compared to fully fine-tuned model due to the small size of the fine-tuned dataset. 
strength_and_weaknesses: Strengths: (1) a simple way to identify sensitive visual parameters.
(2) a low-rank reparameterization approach to update the visual parameters
(3) good results on FGVC and VTAB datasets


Weakness: (1) The introduction on sensitive visual parameter identification is kind of verbal. The criteria is simply the first order gradient and  it would not be efficient to compute second-order hessian matrix. Introducing eq 2 and 3 would be helpful because you can not find optimal in anyways.
(2) Missing introduction of how to compute W_up and W_down

clarity,_quality,_novelty_and_reproducibility: The idea is novel but the clarity is a concern, hard to reproduce if the paper does not release code 
summary_of_the_review: The paper propose a pipeline for recognizing and tuning visual parameters. The idea is interesting but my concern is on the clarity part
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work mainly focuses on the lower inference speed and lacking effective configurations for trainable parameters tailored for each task. It proposed a simple but effective approach named Sensitivity-aware visual Parameter-efficient Tuning (SPT) for these challenges. It quickly identifies the important parameters of the given task in a data-dependent way before fine-tuning without the complex selection schedule. Also, low-rank reparameterization is employed to achieve a better trade-off between efficiency and accuracy. The experiments are conducted on 24 downstream tasks.
strength_and_weaknesses: *[Strength]*
1. The motivation that focuses on task-driven parameter tuning is contributed.
2. This work proposed a novel criterion to efficiently measure the sensitivity (importance) of the pretrained backbone parameters to a specific task.

*[Weakness]*
1. The paper claimed there are challenges about lower inference speed, but in the experiments, there are no related experiments with other SOA methods in Table 1 (only comparisons with the PROMPT-DEEP in Table 6).
2. The experiments are about the ratio of the tuned parameter to the total parameter, instead of the comparison between the total tuned parameter number. There is a concern about the smaller ratio of tuned to total coming from more total parameters.
3. Does such a strategy lead to higher training time (fine-tuning time), compared with other SOA in table 1?
4. In section 3.2, how to get $P$ from $S$?


clarity,_quality,_novelty_and_reproducibility: The paper clarified its contributions and approach most clearly. 
In general, the paper is novel, can be reproduced and is of fair quality.

summary_of_the_review: This paper proposed a novel criterion to efficiently measure the sensitivity (importance) of the pretrained backbone parameters, and proposed a parameter-efficient tuning approach SPT to tune parameters at task-specific important positions.
However, there are a few concerns about if the experiments validate the claimed contributions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission receives 4 negative reviews and 1 slightly positive review. The major raised issues include memory consumption and computational time, which are not sufficiently addressed by the reviewers. During the post-rebuttal discussion phase, the reviewer who gives the slightly positive review shows that the raised concerns are not well addressed and feels ok to downgrade. Overall, the AC feels all the reviewers are not positive about this submission. The authors shall take these suggestions into account to further improve the current work. Welcome to the next venue.
justification_for_why_not_higher_score: All the reviewers unanimously feel the current work needs improvement, and so does the AC.
Decision: Reject