Revisiting Source-Free Domain Adaptation: a New Perspective via Uncertainty Control

Authors: Gezheng Xu, Hui Guo, Li Yi, Charles Ling, Boyu Wang, Grace Yi
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Source-Free Domain Adaptation, Unsupervised Domain Adaptation

Abstract:
Source-Free Domain Adaptation (SFDA) seeks to adapt a pre-trained source model to the target domain using only unlabeled target data, without access to the original source data. While current state-of-the-art (SOTA) methods rely on leveraging weak supervision from the source model to extract reliable information for self-supervised adaptation, they often overlook the uncertainty that arises during the transfer process.  In this paper, we conduct a systematic and theoretical analysis of the uncertainty inherent in existing SFDA methods and demonstrate its impact on transfer performance through the lens of Distributionally Robust Optimization (DRO). Building upon the theoretical results, we propose a novel instance-dependent uncertainty control algorithm for SFDA.  Our method is designed to quantify and exploit the uncertainty during the adaptation process, significantly improving the model performance.  Extensive experiments on benchmark datasets and empirical analyses confirm the validity of our theoretical findings and the effectiveness of the proposed method. 
This work offers new insights into understanding and advancing SFDA performance.

Decision: accept (poster)
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730432568232

thread_type:
official_review

<thread_object>
creator: reviewer_1nwj

<content>
summary:
The current SFDA methods that use source models for self supervised adaptation neglects the uncertainties involved in the transfer process. The authors conduct a comprehensive analysis of the two types of uncertainties arising from both positive and negative samples, providing theoretical insights for both. Based on these theoretical insights, the authors propose a dispersion control method to mitigate the uncertainty introduced by noisy negative samples. For the uncertainty arising from positive samples, the authors suggest leveraging partial labels to fully utilize the predictive uncertainty. By managing these two types of uncertainties, the authors significantly improve the model's performance.

strengths:
Overall, this could be an important theoretical and algorithmic contribution. The paper comprehensively analyzes the two types of uncertainties caused by positive and negative samples during the adaptation process, proves their impact on model performance, and provides effective solutions.

weaknesses:
The method of the paper requires handling both positive and negative sample uncertainties, which increases the computational complexity and training difficulty of the model.The theoretical analysis section may be too complex and difficult for general readers to understand.

questions:
How are the weights λ-CL, λDC, and λPL for the different loss terms (L-CL, L-DC, L-PL) determined? What principles or guidelines were used to set these hyperparameter values?
How are the threshold and the number of retained labels for the partial label method determined?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662387805

thread_type:
official_review

<thread_object>
creator: reviewer_gdvl

<content>
summary:
This paper studies source-fee domain adaptation, with a specific focus on the uncertainty in the contrastive learning-based source-free domain adaptation (SFDA) solution. The authors comprehensively analyze two types of uncertainty including both negative and positive uncertainty through the lens of Distributionally Robust Optimization. Based on the theoretical framework the authors propose an uncertainty-control SFDA method (UCon-SFDA). Extensive experiments demonstrate the advantages of UCon-SFDA over existing SFDA approaches.

strengths:
(i) Significance. This source-free domain adaptation problem studied in the submission is significant for real-world applications of deep learning models. The practical assumption of no source access is meaningful for privacy-sensitive scenarios such as medical data. 

(ii) Novelty. This paper provides an in-depth theoretical analysis of the overlooked uncertainty problem in previous SFDA methods within a unified DRO framework.

(iii) Quality. Extensive experiments and comparisons demonstrate that the proposed approach UCon-SFDA outperforms existing SFDA methods.

(iv) Clarity. The presentation is clear with detailed theoretical analysis and illustrative figures.

weaknesses:
Although the approach is supported by in-depth theoretical analysis and impressive experimental results, I remain concerned about the hyperparameters setting.

Since source-free domain adaptation only has unlabeled target-domain data, it is quite difficult to accurately tune hyperparameters for the SFDA approach. Usually, one solution is to avoid involving many hyperparameters in the SFDA approach. However, the proposed UCon-SFDA necessitates the tuning of many hyperparameters including different $\lambda$ as shown in Table 6 in the Appendix. Moreover, Table 6 demonstrates that the authors adopt different hyperparameter values for different datasets. The challenge is that in practical applications, we would be given an unseen unlabeled target domain where existing hyperparameters can be worse. 

My specific questions are:

- How to tune various hyperparameters to ensure UCon-SFDA can work well in both this paper and other new SFDA tasks? 

- Hyperparameter sensitivity analysis is critical for understanding the method but missing in the paper.

questions:
Please see the question in Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730871663235

thread_type:
official_review

<thread_object>
creator: reviewer_bkmw

<content>
summary:
This paper presents a novel approach for source-free domain adaptation (SFDA), grounded in an uncertainty-guided theoretical analysis of contrastive learning-based SFDA methods. We introduce a distributionally robust optimization framework to elucidate the role of uncertainty. Additionally, the method incorporates augmentation-driven dispersion control and an optimal solution for partial label sets within a contrastive learning-based SFDA approach. The proposed method is thoroughly evaluated across four benchmark datasets.

strengths:
- This paper induces a new aspect to solve the source-free domain adaptation problem.
- It induces an efficient method based on the guide of the proposed theoretical analysis.
- The experimental results prove the advantage of the proposed method

weaknesses:
- Intuitively, applying cross-entropy loss on uncertain target samples with estimated pseudo labels can be problematic and may lead to negative transfer due to noisy pseudo label information, as certain samples are more likely to have accurate pseudo labels than uncertain ones..
- There are seven hyper-parameters, as shown in Table 6, which suggests that the proposed method requires specific parameter tuning to achieve promising results.
- The data augmentation alignment in Eq. 7 is, in fact, a consistency regularization between the weak and strong augmentations of the same data, which is not a novel technique as it has been well studied.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732227577700

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear reviewers,

Thank you for dedicating your valuable time and effort to reviewing our paper. We  appreciate your insightful feedback and constructive comments, which have greatly helped us enhance the presentation of our work. We are grateful for your recognition of the contributions and strengths of our paper, as outlined below:

* Our paper **addresses a significant and practical topic**, the source-free domain adaptation (SFDA) problem, which is critical for real-world applications (Reviewer GdvL).

* We provide a **comprehensive theoretical analysis** of the previously overlooked uncertainty problem in SFDA methods within a unified distributionally robust optimization (DRO) framework. The additional analysis examines both positive and negative sample uncertainties and guides the development of our uncertainty control algorithm for SFDA (Reviewers Bkmw, GdvL, 1NWJ).

* **Extensive experiments** demonstrate the effectiveness of our proposed approach, UCon-SFDA, which consistently outperforms existing SFDA methods. Our results underscore the practical advantages of the method derived from our theoretical insights (Reviewers Bkmw, GdvL, 1NWJ).

* The paper is **well-written**, with thorough theoretical analysis, illustrative figures, and a clear presentation that facilitates understanding (Reviewer GdvL).

We have carefully reviewed each of your queries, concerns, and remarks. In preparing a revised version, we have thoroughly addressed every single comment to improve the clarity and rigor of our paper. Below highlights the key changes we have done.

* We appreciate your insights on emphasizing the theoretical contributions and their practical relevance (Reviewers Bkmw, 1NWJ). In response, we have streamlined the theoretical analysis, added remarks connecting theory to practice, and included a notation table for clarity in the revised manuscript (see **Remarks 4.1-4.4** and **Appendix A.1**).

* We value your suggestions to better highlight the novelty of our approach, particularly the consistency loss (Reviewer Bkmw). We have now elaborated on how our theoretical findings guided the design of the consistency loss (see **Remarks 4.1 and 4.2**) and included additional experiments to demonstrate its advantages (see **Appendix C.6**).

* Addressing your comments on hyper-parameters (Reviewer Bkmw, GdvL, 1NWJ), we have clarified that most parameters were adopted from previous work or derived from theoretical insights, thus requiring lightweight tuning. For the few parameters needing adjustment, we employed a straightforward tuning strategy based on the common hyperparameter tuning pipeline in the prior studies, resulting in effective selections (see **Appendix B**). We have also provided new experimental results on hyper-parameter sensitivity analysis (see **Appendix C.5**).

* On computational complexity (Reviewer 1NWJ), we acknowledge the increased demands but emphasize that our approach remains comparable to existing state-of-the-art methods while achieving superior performance (see **Appendix C.7**).

We hope that our responses have sufficiently addressed all of the concerns raised. Should further details or clarifications are needed, please let us know and we would be happy to provide them. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper received three reviews with ratings of 5, 6, and 8. The reviewers acknowledged its contribution in providing an in-depth theoretical analysis of uncertainties in previous SFDA methods and developing an effective uncertainty control algorithm for SFDA. However, all reviewers expressed a common concern regarding the large number of hyperparameters and the challenges in determining their values for practical applications, particularly in the context of SFDA. It is essential for the authors to include a parameter sensitivity analysis and clarify the relationship between Eq.(7) and the consistency regularization loss commonly used in previous work with weak and strong augmentations.