Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates how to fine-tune pre-trained vision models with SGD. It particularly focuses on out-of-distribution (OOD) tasks. It is shown that the original SGD leads to poor OOD performance compared with AdamW on ViT architectures. Authors explain the reason as the large gradients on embedding layers, and propose to fine-tune with freeze-embed SGD. A further improvement is to gradually unfreezing parameters. 
strength_and_weaknesses: Strength
1. This paper is clearly written. 
2. This paper investigates an important problem of fine-tuning vision models, with the goal of improving OOD accuracy. 
3. The proposed method is simple and effective. 

Weaknesses
1. As authors mentioned in Section 6, there are other transfer learning approaches also suggest freezing a part of parameters. But the differences and advantages with this work are not fully discussed. There is also not any experimental comparison.
2. Some questions need to be clarified. (1) What’s the embedding layer in ResNet? Is it the first convolutional layer or fist block? To my knowledge, the terminology “embedding” is particularly used in Transformer architectures. (2) Why are some numbers (Waterbirds:89.8, DomainNet:98.8) in Table 5 inconsistent with those in Table 1?
clarity,_quality,_novelty_and_reproducibility: Good clarity, quality and novelty. 
Codes are not provided for reproducing. 
summary_of_the_review: The key finding of this paper is informative and useful.
The major drawback is the lack of comparative studies. 

=== after rebuttal ===
I have read the authors' response and other reviewers' insightful comments. Though I still appreciate authors' explorations on the OOD problem, I agree with other reviewers that the paper lacks convincing analyses regarding the empirical observations. Authors' response solved some of my concerns, but the discussion with other similar approaches (i.e., freezing a part of the network) is still not comprehensive. Taking ResNet as an example, it is common in practice to freeze the lowest one of two blocks for efficient or robust fine-tuning. It is not clear whether the performance gain of the proposed method can be attributed to the particular role of the embedding/stem layer, or just generally freezing lower layers? It would be better if the analyses in Appendix F can be confirmed by solid evidences. Overall, I agree with other reviewers that this paper needs further improvements for publication, and I decrease my score to 5.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper argues that SGD with freezing the first embedding layer can perform competitively with the AdamW for modern vision models, by empirically tests the method on 5 finetuning benchmarks.

strength_and_weaknesses: ### Strengths
1. The SGD (freeze-embed) is simple but effective on all the models shown in the experiments.
2. The method performs well on both OOD and ID datasets.
3. The paper achieves SOTA results on multiple benchmarks.


### Weaknesses
1. The paper claims the first layer is an outlier of many existing models with embedding layer, by comparing the L2 norm of the gradient for each layer. It is pretty straightforward as the embedding layer has much more parameters than the others. It would be great if some analysis can be made for whether the average gradient per parameter has a difference for the first layer.
2. A deeper analysis of why the gradient norm affects the finetuning would be more interesting. For example,  it may be helpful to understand how the freezing part of the embedding layer will affect the performance.
3. It would be interesting to see if it will achieve better results for SGD using a smaller learning rate for the first layer instead of freezing it for non-ResNet models.
4. The paper claims the existing gradual unfreezing can perform better than the SGD (freeze-embed) proposed in the paper. It is unclear, why should people consider SGD (freeze-embed), instead of directly using gradual unfreezing.
5. The paper implies that SGD is more efficient during the training than AdamW. It would be great if the memory consumption for the experiments can be shown in the tables to empirically show the evidence.
clarity,_quality,_novelty_and_reproducibility: The experiments are well presented, but a detailed analysis would be appreciated.
summary_of_the_review: The paper shows interesting empirical results of improving SGD for fine-tuning vision models, but more analysis would be appreciated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to narrow the performance gap between SGD with momentum and AdamW when evaluating on the downstream tasks. The authors find that for the models where AdamW outperforms SGD, the gradients of the first layer are much larger than the gradients of the other layers. Therefore, they try to freeze embedding layer and propose a novel training technique to improve the performance of SGD with momentum and narrow the gap. They also conduct various experiments to verify the performance of the proposed method.
strength_and_weaknesses: Strength:

This paper focus on an important research topic. If we can narrow the performance gap between SGD and AdamW, the memory cost will be reduced during the process.
This paper also provides an interesting experimental finding: First-layer gradient is an outlier.
This paper is easy to follow. 

Weakness:

The novelty is limited. The first layer shows a different pattern and freezing embedding layer is not novel and has been proposed in some related work. [1] [2] 
The research area about fine-tuning vision models (in your title) is very large. The experiments in this paper mainly focus on data shift datasets. Maybe you can report the experiment results on more general datasets. (such ImageNet pretrain and CIFAR fine-tune).
I think the large memory cost of Adam is a very important motivation of your work, maybe you should also provide some results about that. 
The resolution of Figure 2 is too low for me. 

[1] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable
Features with Deep Adaptation Networks. ICML 2015
[2] Jeremy Howard, Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. 
clarity,_quality,_novelty_and_reproducibility: NA
summary_of_the_review: NA
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to simply freeze the embedding layer (0.7% of the parameters) of the large neural networks in computer vision, then apply the standard SGD optimizer to finetune the parameters on the out-of-distribution datasets. It is shown that such a simple setting contributes to competitive performance with the popular AdamW. The experimental results on five benchmark datasets further support the effectiveness of the proposed method.
strength_and_weaknesses: # Strength:
- The technique part of this submission is quite simple and easy to follow, which may help to reproduce the results reported in the paper.
- The motivation for freezing the first (“embedding”) layer has been clearly illustrated in Figure 2. It is interesting to see that ResNets are completely opposite to Vision Transformers and ConvNeXt. 
- The authors conduct thorough ablation studies on several datasets to verify the effectiveness of SGD Freeze-Embed, which should be encouraged.

# Weaknesses:
- ### Major Concerns:
  - According to the **ABSTRACT**, "when the two methods perform the same, SGD is preferable because it uses less memory and is more efficient than AdamW", however, I failed to find any comparisons on the training cost or the memory footprint between SGD Freeze-Embed and AdamW. Since activations commonly consume far more memory resources than weights, I doubt that the real memory saving could be very limited. The authors are encouraged to include more quantitative analyses of the efficiency of the proposed methods.
  - It would be better to evaluate SGD Freeze-Embed on more downstream tasks. The current manuscript mainly focuses on the out-of-distribution (OOD) setting, which seems to weaken the importance of this discovery. I am curious to know whether the phenomenon illustrated in Figure 2 widely exists in different tasks. If it is true, any theoretical or empirical analysis may shed some light on the further understanding of why vision transformer is more difficult to train and the design of Transformer-specific optimizers.
  - Though the authors present extensive experiments on OOD tasks, the reason why large gradients damage performance remains unsolved. One may easily clamp the gradients of the first layer with `clip_grad_norm_` to fix the outlier. Hence, I am not fully convinced that SGD Freeze-Embed should be the best choice.
- ### Minor Issues:
  - In the original ConvNeXt paper, they "replace the ResNet-style stem cell with a patchify layer implemented using a 4×4, stride 4 convolutional layer". I am not sure whether the "patchify layer" refers to the embedding layer. Could the authors further explain the architecture of the first layer in ConvNeXt?
  - What is the difference between SGD and SGD (Freeze-embed) when applied to ResNets, listed in Tables 1-2?  
  - In **Freezing layers**: I politely disagree that "The embedding layer only consists of a small fraction of the model parameters (e.g., 0.7% in a base-vision transformer), so a priori freezing the embedding layer is a very tiny tweak—and we would not expect a large change in accuracy." Note that a group of "Visual Prompt Tuning" methods aim at introducing/fine-tuning only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the other model backbone frozen [1,2,3].
  - Did SGD Freeze-Embed still apply weight decay to the first layers?

# Reference:
- [1] Visual Prompt Tuning. ECCV2022
- [2] Prompt-Matched Semantic Segmentation. arXiv2022
- [3] LPT: Long-tailed Prompt Tuning for Image Classification. arXiv2022
clarity,_quality,_novelty_and_reproducibility: Since the proposed methods are rather simple, the clarity and reproducibility seem ok. However, the quality and novelty are my major concerns. Given the current manuscript, it is hard to verify that SGD Freeze-embed can be generally effective on downstream tasks except for OOD. Besides, simple baseline methods such as gradient clipping may serve as a counterpart of this approach. Last but not the least, this paper seems to lack drive and motivation. Please refer to the *Major Concerns*.
summary_of_the_review: This paper contains some interesting observations, however, the novelty may not meet the standard of the main track.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four knowledgeable referees. The initial reviews raised concerns about the positioning of the contribution w.r.t. similar approaches, the limited novelty, and unconvincing quality of execution, which questions the validity of the observations beyond the OOD task and the impact of the seemingly minor improvements. The authors' rebuttal partially addresses the reviewers concerns. During the discussion phase, the reviewers reiterate on some of their initial concerns. In particular, the motivation and claims wrt the effectiveness of the method remain unconvincing. The insights from the presented analysis appear limited, and do not fully unveil why the proposed approach of freezing and finetuning with SGD should lead to the observed improvements. Results across different architectures (e.g. ResNet and ConvNext vs transformers) do not fully support the claims about the gradients. The reviewers also agree that the discussion on differences with other similar approaches remains limited after rebuttal. The AC agrees with the reviewers' assessment and therefore recommends to reject. The AC strongly encourages the authors to take into consideration the reviewers' feedback to improve future iterations of their work. 
justification_for_why_not_higher_score: The paper motivation remains unclear after discussion with the authors. The positioning of the proposed approach with respect to similar work in the literature is limited. The empirical improvements appear minor and the insights of the analysis do not seem to hold across different architectures.
Decision: Reject