Kati8/ICLR_2023_2025 huggingface dataset:
authors: Shiqi Jiang
authors: Tho Tran Huu
authors: Qianxiao Li
authors: Tan Minh Nguyen
reviews: 
Comments: The machine learning community has made remarkable progress with the advent of large-scale models, but these models become a significant obstacle by consuming large amounts of memory during training. In this paper, we propose a fine-tuning Method using the Method of Successive Approximations, L-MSA. Experiments on different data sets verify the effectiveness of L-MSA
Strengths: - The paper is well-written and easy to follow.
- The experimental results seem to be promising, surpassing Full Fine-tuning on part of the dataset
Weaknesses: - This paper emphasizes the advantages on large-scale models, but the datasets used seem to focus on extremely small datasets such as CIFAR, which show the worst performance on experimental fine-tuning on Imagenet-C. The only LIFT that seems to outperform is a paper that has not yet been published through peer review. This makes me worry about its practical prospects.
- Not enough experiments, this paper is motivated by the fact that large-scale models consume memory, but there is no comparison of memory overhead in the paper.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper explores layer-wise fine-tuning strategies for adapting large pretrained models. The authors found that fine-tuning specific layers can lead to varied performance outcomes, and selectively fine-tuning certain layers may enhance results. Building on these insights, they propose a novel layer-wise fine-tuning approach that leverages Pontryagin’s Maximum Principle to guide layer selection for fine-tuning. The effectiveness of this method is demonstrated through transfer learning from ImageNet to CIFAR100.
Strengths: 1. The task of parameter-efficient fine-tuning is crucial for practical applications of pretrained models.

2. The authors performed an in-depth analysis on the effectiveness of fine-tuning at the layer level.
Weaknesses: 1. The paper's novelty is somewhat limited, as the idea that fine-tuning different layers can lead to varying performance outcomes has already been widely explored in previous research. It is not clear what is the main contribution of the paper. 

2. The proposed method, which relies on the Method of Successive Approximation, lacks clear motivation. It’s unclear why this approach would lead to improved layer selection or how it compares favorably to other existing methods. In other words, what are the benefits of the proposed approach?

3. For the theoretical analysis, it is not clear what is the main contribution. The authors should also connect the theoretical analysis to the proposed method and discuss why the proposed method can lead to better layer selection and fine-tuning. 

4. The explanation based on PMP is interesting, but the main technical contribution is not clear. It would be helpful if the authors could clarify where the primary novelty lies.

Overall, while the paper presents a reasonable idea, it suffers from poor writing, insufficient motivation and unclear contribution.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 1