Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is about Source-free Unsupervised Domain Adaptation (SF-UDA).  The authors provide an analysis of the impact of main design choices in SF-UDA through a large-scale empirical study with several CNN and ViT models, methods, and datasets for image classification. In particular, they observe the normalization approach, pretraining strategy, and backbone architecture are the most impactful design choices, and propose best practices for robust SF-UDA methods. The authors also seek to study the strengths and failure modes of SF-UDA methods and compare them with UDA approaches. Empirical results show that SF-UDA methods like SHOT can provide accuracy comparable with that of state-of-the-art UDA methods (at much lower data and computational cost, and better than standard benchmarks and backbone architectures. 
strength_and_weaknesses: Strengths:
+ The authors provide an in-depth empirical analysis of design choices for SF-UDA with several UDA and SF-UDA methods, backbone architectures, and datasets.  This SF-UDA  setting is particularly challenging and still remains understudied. I anticipate that this study and its results should be useful to researchers and practitioners working in this area.  
+ The methods and experimental methodology are described in enough detail to understand the paper.     
+ The conclusions of this study are interesting. For instance, it is useful to know that State-of-art SF-UDA methods, like SHOT, are competitive with state-of-the-art UDA methods, but can be implemented much more efficiently. 
+ The supplementary material provides additional algorithmic and implementation details on baseline methods, and experimental results that help support the paper.  

Weaknesses:
- Some definitions are not very clear at the beginning: DA, DG, etc., and how they relate to SOA methods for multi-task and multi-domain learning.
- The authors do not provide a critical analysis of methods, or highlight from the outset the key challenges facing practitioners or researchers working on SF-UDA methods and problems. 
- The empirical results are not fully convincing. The scope of this study is limited to standard image classification problems, using common datasets. There is no results or discussion on the results that would be obtained from real-world data captured in the wild.  The authors should explore the impact on the performance of, e.g., class imbalance, noisy image data, etc. 
- The authors should measure domain shift empirically in order to characterize the difficulty of an SF-UDA problem.  
- Results on FT+NRC should not be included in the results if it is trained using only trained on 15% of the DomainNet data.  
- Finally, the authors should asses and compare the computational complexity of different SF-UDA methods, and show the differences w.r.t. UDA methods. 

clarity,_quality,_novelty_and_reproducibility: + The paper is well organized and their proposed approach is generally well described.
+ This study seeks to clarify the impact of the main design choices in SF-UDA approaches, and investigate the strengths and failure modes of SF-UDA methods. This paper is a details empirical study with mostly pre-existing existing methods. The study approach is well-motivated, but as expected, there is limited conceptual innovation.
+ The code was not made available, although the methods are well described. Since the experimental data and code are not available (released upon acceptance), there is a concern that the results in this paper would be difficult for a reader to reproduce.   


summary_of_the_review: Overall this is good quality submission.  The proposed study is well-motivated, although the experimental validation could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data.
strength_and_weaknesses: This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data. Overall, in my opinion, this work is only an experimental report instead of a complete research paper due to two main reasons.

1. In Section “Method”, the current manuscript only provided many details of the existing SF-UDA work (SHOT) without description of new proposed algorithm. Thus, this current version lacks a complete novel algorithm.

2. Currently, there are a lot of explorations on SF-UDA topic [1, 2, 3]. However, the current work does not make fair comparisons with them. Thus, the experiments are not enough.

[1] Chu, Tong, et al. "Denoised maximum classifier discrepancy for source free unsupervised domain adaptation." Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22). Vol. 2. 2022.

[2] Ding, Ning, et al. "Source-Free Domain Adaptation via Distribution Estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Kundu, Jogendra Nath, et al. "Balancing discriminability and transferability for source-free domain adaptation." International Conference on Machine Learning. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: The description of new algorithm is not clear, and the writing and paper organization should be polished.
summary_of_the_review: This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data. Overall, in my opinion, this work is only an experimental report instead of a complete research paper due to two main reasons.

1. In Section “Method”, the current manuscript only provided many details of the existing SF-UDA work (SHOT) without description of new proposed algorithm. Thus, this current version lacks a complete novel algorithm.

2. Currently, there are a lot of explorations on SF-UDA topic [1, 2, 3]. However, the current work does not make fair comparisons with them. Thus, the experiments are not enough.

[1] Chu, Tong, et al. "Denoised maximum classifier discrepancy for source free unsupervised domain adaptation." Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22). Vol. 2. 2022.

[2] Ding, Ning, et al. "Source-Free Domain Adaptation via Distribution Estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Kundu, Jogendra Nath, et al. "Balancing discriminability and transferability for source-free domain adaptation." International Conference on Machine Learning. PMLR, 2022.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper reports the performance of SFDA methods under the new setting, called double-transfer, via a large-scale empirical evaluation. The double-transfer represents two-stage transfer learning setting in which the pre-trained model is firstly fine-tuned with a source domain dataset and is then adapted to the target domain. Under this setting, the results shown in this paper reveal several critical factors to achieve better performance after the adaptation. 
strength_and_weaknesses: 

<strength>

- The double-transfer setting studied in this paper seems to be quite practical but has not been well explored in the literature. I love the motivation of this work.
- The experiments cover a very diverse set of datasets as well as model architectures and are done on a large scale.

<weakness>

- Not so surprising results. Some findings in this study are closely related to those shown in the literature; for example, using a large-scale dataset is beneficial in pre-training [Dosovitskiy et al., 2020], and BN has a crucial role when conducting domain adaptation [Li et al., 2017].
- In the experiments, only supervised pre-training is considered. Since self-supervised representation learning has recently become a popular way to pre-train the model, it would be great if it is included in the comparison to reveal the best strategy of pre-training in double-transfer setting.

clarity,_quality,_novelty_and_reproducibility: <Clarity>

- Some terminology is confusing.
	- Does "DGen" mean evaluation of a target model under the domain shift setting? This terminology is quite confusing. (it seems to indicate a certain domain generalization method.)
	- Why does "Generalization LP" perform best in Fig. 1? Is it showing the accuracy at the source domain? If so, this is also confusing, because the other subfigures show the accuracy at the target domain.
	- "DGen -> X" in Fig. 2 is also confusing. It sounds like training the model with a certain domain generalization method at the source domain and then conducting X to adapt the model to the target domain (but it does not mean such a thing). 

<Quality>

The experimental setup seems to be reasonable, and the analysis is rigorously based on the results of large-scale experiments.

<Novelty>

Evaluating SFDA methods in double-transfer setting is somewhat new and should be important for practical applications. However, some findings in this study are closely related to those shown in the literature as stated in <weakness>.

<Reproducibility>

The authors state "Experimental data and code will be released upon acceptance."

summary_of_the_review: This paper reports some interesting findings obtained through the empirical studies with a new setting called double-transfer. However, I have several concerns on clarity and novelty as stated above, which makes my score conservative. I vote for "weak reject."
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the source-free unsupervised domain adaptation problem and analyzes the impact of several design choices (e.g., the pre-training dataset, the pre-trained network, the normalization strategy, and the adaptation strategy) through a large-scale empirical study.
strength_and_weaknesses: Pros

1. this studied problem is interesting and receives increasing attention in the transfer learning field, making a large-scale empirical study valuable

2. the empirical analysis includes more than 500 architectures (including both CNNs and Vision Transformers), 6 datasets for domain adaptation, which is very extensive

Cons

1. the selected source-free domain adaptation (SFDA) methods are relatively less, only a classifier adjustment method (SCA) and a feature encoder fine-tuning method (SHOT) is validated throughout the study. More SFDA methods like Li et al, CVPR 2020 could be used in an extensive study. 

2. generally speaking, the observations would not inspire more ideas in the SFDA topic as most conclusions have already been known.

3. to study SFDA, these choices (the pre-training dataset, the pre-trained network, the normalization strategy) are vital but the authors may focus more on the adaptation strategy, like whether to recover the source data and the robustness to different label distributions (e.g., open-set, partial-set) 

Some missing references:

1. Liang, Jian, et al. "Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer." IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

2. Huang, Jiaxing, et al. "Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data." Advances in Neural Information Processing Systems 34 (2021): 3635-3649.

3. Kurmi, Vinod K., Venkatesh K. Subramanian, and Vinay P. Namboodiri. "Domain impression: A source data free domain adaptation method." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.

4. Xia, Haifeng, Handong Zhao, and Zhengming Ding. "Adaptive adversarial network for source-free domain adaptation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Typos:
siCP -> CP in section 3.1
clarity,_quality,_novelty_and_reproducibility: This paper is very written and conducts an extensive study on the source data-free unsupervised domain adaptation problem. Such motivation is interesting and important.
summary_of_the_review: This paper conducts an extensive study on the source data-free unsupervised domain adaptation problem, which is interesting and important in this field. However, some more studies are required to make this study more valuable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies methods for source-free unsupervised domain adaptation focusing on the "double-transfer" setting and performs an empirical study on the factors affecting their performance. While reviewers appreciated that the experiments were extensive in terms of datasets, domain pairs and models evaluated, they had unresolved concerns that only 2 adaptation methods were evaluated (even after the authors' response), thus limiting the generality of the conclusions. They also had concerns that many of the findings were not surprising, partly as similar results have appeared in the literature. As such, the paper does not currently meet the bar for ICLR.

That said, the study has promise and can likely be publishable with further work. Beyond expanding the set of methods included in the study, it would be useful to discuss the settings under which the identified factors (e.g. architecture, pre-training strategy) can actually be controlled; in some applications it would not be possible to control the source-model being provided, in which case a greater focus on the factors affecting adaptation would be more relevant as alluded to by the reviewers. It could also be helpful to take reference from a related study in the NLP literature -- "A Comparison of Strategies for Source-Free Domain Adaptation", ACL 2022.
justification_for_why_not_higher_score: - Empirical study is limited to 2 methods, which raises questions about its generalizability
- Findings are not too surprising (e.g. architecture affects performance) and partly alluded to in the literature
Decision: Reject