reviews: 
Comments: The paper proposes a method to do a more targeted fine-tuning process where instead of updating every single layer/parameter in the architecture, the layer that has the strongest effect in the total loss is selected and updated on each iteration of the fine-tuning process.

Towards this goal the paper proposes a metric to select the layer to be updated and an algorithm for the fine-tuning of the selected layers.

Experiments on several datasets (CIFAR-100, CIFAR-C, CIFAR-Flip, Living-17 and ImageNet-C) including several related methods (full fine-tuning, LIFT, LISA, surgical Fine-tuning and Auto-RGN) provide evidence on the performance of the proposed method.
Strengths: - A theoretical analysis for the proposed method has been provided (Sec. 3 & Sec. A.1).
    
- The empirical evaluation of the proposed method includes a good variety of datasets and competing methods. This allows to assess the applicability of the proposed method in different contexts.
    
- The validation of the proposed method includes an ablation analysis this is effective towards obtaining insights on how the different components of the proposed method contribute to its performance and how it compares w.r.t. full fine-tuning.
    
- The paper adequately highlights the limitations of the proposed method (Sec. 5)
Weaknesses: - The originality of the paper is somewhat reduced. While the method put forward by the paper outperforms (in some cases) the considered baselines, as admitted by the paper (Sec. 4.1)  there are already methods in the literature that aim at a more targeted fine-tuning process.
    
- In its current form the paper does not feel self-contained, there are several aspects of the paper that are delegated to the appendix. For instance, the models considered in experiments of Sec. 4.2 are not explicitly indicated. In other cases, the protocols and motivations behind the conducted experiments are not clear. Here a proper balance must be achieved to ensure the paper provides sufficient details as to allow the reader to critically analyze the proposed method and its conducted validation.
    
- While pairing a method with its corresponding theoretical analysis is very desirable, the extend to which such analysis is currently provided (Sec. 3) is just too shallow as to be informative.  Perhaps it could be completely moved to the appendix in order to allocate space to further elaborate on other parts of the paper that are currently not detailed enough.
    
- Some statements seem to lack supporting evidence. For instance, in several parts of the paper (l.535) statements are made regarding to the reduced computational costs of the proposed method. The evaluation section is missing however a proper experiment addressing that aspect.
    
- The improvement put forward by the proposed method is not that outspoken. For instance, in Table 1 it is observed that the proposed method is better in only 2/4 considered settings.
    
- In Fig. 6, averaged results over different datasets are reported. This not only hinders the variations in performance across datasets, but also behavior observed for the different methods across the considered datasets.
    

- Weak positioning; Good part of the related work is centered around other not directly related aspects. For instance, Sec. 6.1 discusses large architectures, which has close no link w.r.t. the proposed method.  Similarly l.497-502 from Sec. 6.2 is related to prompt-based methods. Thus having a relatively weak link with the proposed method. This weak positioning w.r.t. related efforts becomes more evident when we consider that almost none of the related methods considered in the experimental section (Sec. 4) are covered in the related work section. In addition, I would suggest looking at the two references below which seem to be very related to the proposed method.
    
    - Youngmin Ro, Jin Young Choi, "AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks", AAAI 2021
    - Basel Barakat; Qiang Huang, "Enhancing Transfer Learning Reliability via Block-Wise Fine-Tuning",  ICMLA 2023
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2