Title: DUET: Dual-Perspective Pseudo Labeling and  Uncertainty-aware Exploration & Exploitation  Training for Source-Free Domain Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Source-Free Domain Adaptation

Pseudo Labeling

Uncertainty-Aware Learning

CLIP
Abstract: Source-free domain adaptation (SFDA) aims to adapt a pre-trained source model to an unlabeled target domain without requiring labeled source data. 
In a self supervised setting, relying on pseudo labels on target domain samples facilitates the domain adaptation performance providing strong supervision. 
However, a critical problem of this approach is the inherent instability of the pre-trained source model in the target domain, leading to unreliable pseudo labels for the target domain data. 
To tackle this, we propose a novel Dual-perspective pseudo labeling strategy that jointly leverages a task-specific perspective and a domain-invariant perspective, assigning pseudo labels only to target samples on which the target model’s predictions and CLIP’s predictions agree. 
To further enhance representation learning without introducing noisy supervision, we apply consistency training to uncertain samples.
Additionally, we introduce a Tsallis mutual information(TMI)-based vision optimization strategy guided by an Uncertainty-based adaptation index (UAI), which dynamically modulates entropy sensitivity based on the model’s adaptation uncertainty. 
The UAI-based training paradigm enables stable and adaptive domain alignment by effectively balancing exploration and exploitation processes during the optimization process. Our proposed method achieves state-of-the-art performance on domain adaptation benchmark datasets, improving adaptation accuracy by 1.6% on Office-Home, 1.4% on VisDA-C, and 2.9% on DomainNet-126, demonstrating its effectiveness in SFDA.
The code is publicly available at https://github.com/l3umblee/duet-sfda.

DUET: Dual-Perspective Pseudo Labeling and Uncertainty-aware Exploration & Exploitation Training for Source-Free Domain Adaptation

Abstract

Source-free domain adaptation (SFDA) aims to adapt a pre-trained source model to an unlabeled target domain without requiring labeled source data. In a selfsupervised setting, relying on pseudo labels on target domain samples facilitates the domain adaptation performance providing strong supervision. However, a critical problem of this approach is the inherent instability of the pre-trained source model in the target domain, leading to unreliable pseudo labels for the target domain data. To tackle this, we propose a novel Dual-perspective pseudo labeling strategy that jointly leverages a task-specific perspective and a domain-invariant perspective, assigning pseudo labels only to target samples on which the target model's predictions and CLIP's predictions agree. To further enhance representation learning without introducing noisy supervision, we apply consistency training to uncertain samples. Additionally, we introduce a Tsallis mutual information (TMI)-based vision optimization strategy guided by an Uncertainty-based adaptation index (UAI), which dynamically modulates entropy sensitivity based on the model's adaptation uncertainty. The UAI-based training paradigm enables stable and adaptive domain alignment by effectively balancing exploration and exploitation processes during the optimization process. Our proposed method achieves state-of-the-art performance on domain adaptation benchmark datasets, improving adaptation accuracy by 1.6% on Office-Home, 1.4% on VisDA-C, and 2.9% on DomainNet-126, demonstrating its effectiveness in SFDA. The code is publicly available at https://github.com/l3umblee/duet-sfda.

1 Introduction

Unsupervised domain adaptation (UDA)[26, 3, 15, 6] explores training strategies or model architectures that enable a model trained on labeled source domain data to achieve high task accuracy on the unlabeled target domain data. Traditionally, UDA assumes access to both labeled source domain data

2

3 ,

and unlabeled target domain data. However, in practical scenarios, the application of UDA can be limited due to privacy concerns, legal regulations, and data security issues associated with labeled source domain data.

To address these limitations, source-free domain adaptation (SFDA)[12, 1, 28, 29, 30, 38, 11, 13, 27] has been proposed, allowing domain adaptation without the use of labeled source domain data. SFDA relies solely on a pre-trained source model and unlabeled target domain data to achieve adaptation. SFDA can be broadly categorized into two main approaches. The databased approach[32, 10, 39, 8] focuses on constructing a virtual source domain using adversarial learning techniques such as GANs[4, 9, 14] or on extracting intrinsic feature information from existing data to facilitate adaptation to the target domain. In contrast, the model-driven

1.0

LabelRatio

Ours

1.0

Ours

0.9

DIFO

0.8

DIFO

0.8

0.6

0.7

seudo

0.4

0.6

P

0.2

(a)Pseudo label accuracy

2

3

(b) Pseudo labeled sample ratio

2

3

approach[12, 1, 29] operates in a self-supervised learning paradigm, where supervision is derived from the target domain data based on the predictions (pseudo labels) generated by the pre-trained source model. It generally outperforms data-based approaches and has become the most widely adopted paradigm in SFDA. A representative method of the model-driven approach is SHOT[12], which generates pseudo labels for target domain samples based on feature space analysis. These pseudo labels are subsequently utilized for entropy minimization, enhancing the pre-trained source model's performance in the target domain. AdaContrast[1] performs batch-wise pseudo label generation and refinement for test-time adaptation. It identifies the nearest neighbors of each target sample in the feature space based on cosine similarity and assigns pseudo labels by averaging the target model's predicted probabilities. The pre-trained source model is then trained using a combination of a supervised loss calculated by using pseudo labels and contrastive learning. These approaches provide strong supervision in unlabeled settings, facilitating domain adaptation. However, the pre-trained source model (i.e., target model ) exhibits instability in the target domain, which leads to unreliable pseudo labels in the early training stages and makes it difficult for the target model to establish a clear learning direction.

To mitigate this issue, DIFO[29] employs CLIP[23], a vision-language (ViL) model with strong zero-shot capabilities, to enhance the quality of pseudo labels. Additionally, it introduces a mutual information (MI)-based loss function that enables prompt learning in CLIP and knowledge distillation between CLIP and the target model. Despite achieving state-of-the-art performance on multiple benchmark datasets, DIFO still faces challenges in assigning accurate pseudo labels to target domain samples, particularly in the initial training stages. Moreover, while MI-based prompt learning facilitates CLIP's adaptation to the target domain, its effectiveness is inherently limited by the fixed vision encoder extracting low-quality image features that lack clear inter-class separability.

Fig. 1 compares the proposed method with DIFO in terms of pseudo label accuracy and the pseudolabeled sample assignment ratio on the Office-Home[34] dataset. As shown in Fig. 1, DIFO suffers

Toyexample

TMI

0.10

MI

SSOT

0.05

0.00

1.00

0.75

0.50

0.25

0.00

(a)

(b)

(c)

from low pseudo label accuracy across the entire set of training samples. This issue hinders model convergence and introduces instability into the training process. Moreover, since CLIP is not pretrained for specific tasks but rather exhibits task-agnostic properties, its predictions tend to reflect general and ambiguous patterns rather than task-specific distinctions. Therefore, a refinement process that accounts for this limitation is necessary. To this end, we propose two key strategies: (1) A learning framework that assigns pseudo labels only to samples whose predictions from the task-specific target adaptation model and the domain-invariant CLIP model agree. The remaining samples, which lack such agreement, are optimized using consistency-driven representation enhancement until sufficient feature quality is achieved. (2) A Tsallis mutual information (TMI)-based adaptation uncertaintyaware vision optimization strategy that dynamically adjusts training intensity based on the model's current adaptation uncertainty level, enabling robust adaptation even under large domain gaps.

In the first strategy, we refine the pseudo label generation process by assigning hard pseudo labels only to samples where the task-specific target adaptation model and domain-invariant CLIP agree in their predictions. This selective assignment allows us to extract highly reliable pseudo labels, combining task relevance and domain generalizability. Meanwhile, instead of assigning noisy labels to the remaining samples, we apply consistency training to boost the feature representation ability of the target model. This approach enables effective use of unlabeled data while minimizing label noise, thus improving overall adaptation performance.

In the second strategy, unlike DIFO-which adapts CLIP to downstream tasks via text prompt tuning-we directly fine-tune the CLIP vision encoder. Existing prompt tuning[24, 35, 42] can be limited when the image features extracted by the frozen vision encoder exhibit low clustering quality due to a large domain gap, as illustrated in Fig. 2(a). In this case, the text embeddings are forced to adapt to low-quality image representations, limiting performance. This problem becomes even more severe in SFDA, where no source data is available and the domain gap is large. To overcome this issue, we propose an alternative strategy: a TMI-based adaptation uncertainty-aware optimization method that enables effective feature alignment between the pre-trained CLIP knowledge and the target domain, thereby mitigating the limitations of prompt tuning. Specifically, existing prompt tuning is guided by a MI-based loss function between the target model and CLIP predictions. While this encourages aligned predictions between the two models, the standard MI loss does not sufficiently reflect the model's current training state. We demonstrate this limitation with a simple toy example in Fig. 2(c). This toy example simulates the training process of a model optimized using MI loss. In particular, we measure the MI between two probability vectors: one is a randomly generated probability vector (considered as ground truth, p ), and the other is a noisy version of this probability vector (considered as the model prediction, ˆ p ). The detailed settings are provided in Appendix A. As the noise is gradually reduced, we observe how MI changes between these two distributions. As observed in Fig. 2(c), MI struggles to capture the overall changes in the probability distribution as the noise level decreases, making flexible loss measurement challenging (i.e., MI shows little meaningful variation despite gradual noise reduction). To address this limitation, we propose an approach that adaptively considers the model's adaptation uncertainty during CLIP's vision optimization. Specifically, we introduce a TMI loss function based on Tsallis entropy [33], which leverages our uncertainty-based adaptation index (UAI) to dynamically control the entropy curvature depending on the model's adaptation uncertainty. This UAI is used to compute TMI, allowing the model to adjust the strength of training based on its current adaptation uncertainty level and guiding the optimization process in a stage-aware manner. As shown in Fig. 2(c), the UAI enables TMI to encourage the utilization of diverse samples during the early unstable training phase (exploration), and to shift toward a stable optimization strategy (exploitation) for SFDA scenarios with large domain gaps. Consequently, TMI provides an effective optimization strategy in SFDA scenarios where a significant domain gap exists, balancing exploration and exploitation to achieve robust domain adaptation. Our contributions are summarized as follows:

We propose a Dual-perspective pseudo labeling strategy that integrates a task-specific perspective, which emphasizes class-discriminative semantics, and a domain-invariant perspective, which focuses on robust generalization, thereby enabling more reliable supervision and structurally enriched representation learning.

Weintroduce a Tsallis mutual information -based vision optimization strategy that leverages an Uncertainty-based adaptation index to dynamically regulate the optimization process by separating it into exploration and exploitation stages depending on the model's adaptation uncertainty level.

Our method achieves performance improvements over state-of-the-art methods on OfficeHome (+1.6%), VisDA-C (+1.4%), and DomainNet (+2.9%) datasets.

2 Preliminary

Consistency training Consistency training[31, 25, 36, 20] is a widely used technique in semisupervised learning that ensures the model produces consistent predictions when different augmentations are applied to the same input sample. This technique is particularly effective in scenarios with limited labeled data, as it provides a strong regularization effect, thereby improving model performance. In source-free domain adaptation (SFDA), where adaptation must be performed on an unlabeled target domain without access to labeled source domain data, the reliance on pseudo labels can introduce instability during training. By progressively aligning the model's predictions on augmented target samples, this approach mitigates the impact of pseudo label noise and enhances the model's ability to generalize to the target distribution.

Tsallis entropy The conventional Shannon entropy-based model training process can lead to overconfidence in the presence of label noise[41, 17]. Tsallis entropy[33] offers a broader representational capacity in entropy computation compared to Shannon entropy, providing a potential solution to this issue. Tsallis entropy is defined as follows:

Here, ln q ( · ) and S q ( · ) denote Tsallis logarithm and Tsallis entropy, respectively. p i represents the probability of event i in a given discrete probability distribution. q is known as the entropy index, controlling the degree of non-extensivity in the entropy formulation. As the entropy index q approaches one, Tsallis entropy asymptotically converges to Shannon entropy. If q is greater than one, the variation in entropy values with respect to probability changes is smaller than in Shannon entropy. For an intuitive understanding of how different values of entropy index q affect the shape of the entropy function, please refer to the visualization in Appendix B. From a model training perspective, a Tsallis entropy-based loss function with q larger than one reduces the model's sensitivity to highconfidence predictions and promotes exploration of uncertain samples. Conversely, when q is less than one, the entropy values change more sharply with the probability variations, leading to increase sensitivity to confident predictions, encouraging exploitation. This characteristic highlights that a loss function utilizing Tsallis entropy can effectively regulate the model's learning pace. By adjusting the entropy index q , the model can emphasize exploratory learning under high uncertainty and shift toward stable optimization (exploitation) as confidence increases.

3 Methodology

Overview As shown in Fig. 3, the proposed method is built upon three core components: 1) Calibrated pseudo label generation (CPG), which generates two types of pseudo labels (soft and hard) for target domain samples whose predictions from the target model and CLIP agree. 2) Dynamic entropy-guided vision optimization (DVO), in which the soft pseudo labels produced by CPG are used within the proposed Tsallis mutual information (TMI) to optimize CLIP's vision encoder for the target domain. 3) Pseudo label matching (PLMatch) framework, which trains the target model by incorporating hard pseudo labels.

These core components are then devided into two hierarchical processes, a cycle-level process and an iteration-level process in consideration of computational cost and training stability. In the cycle-level process, components with high computational cost, such as CPG and DVO are executed, enabling the generation of reliable pseudo labels and the adaptation of CLIP's vision encoder. In contrast, the iteration-level process consists of PLMatch framework, which repeatedly trains the target model with fixed hard pseudo labels from the current cycle. Specifically, our training framework is organized into multiple cycles, each consisting of several iteration-level processes. Each cycle first generates pseudo labels and optimizes the CLIP vision encoder, followed by several iteration-level processes that train the target model with the fixed hard pseudo labels.

Cycle-level process

Iteration-level process

CPG

DVO

PLMatch

Prediction Mixture

Dual-perspective PL

Dynamic entropy-guidedvision optimization

PseudoLabelMatchingframework

α(cu)

Unlabeled data

→a(x)'spath

yed s,()←

→A(x)'s path

Classifier g

Target. ft,g*

α(xu)->

α(cu)

A

攀二

Lcon

ax(ru)

Text

→

T

LTMI

A(cu)

Task-specific Pred. p

prompt

ViL

Target.ft

Lsup

ViL

Mixedprediction.

CPGMixedpredictionp

Adaptation index q

LKL

Exploration

Exploitation

dapd

T

α(x)

ViL

↑

Encourage feature

oo

learning capability

ViL

α(cu)

Pseudolabeleddata

(c)

samples

learning for high-entropy

Training cycle

CPG

Freezing

Training

Q(cu) A(cu): Weak and strong augmented image, respectively

3.1 Cycle-level process

3.1.1 Calibrated pseudo label generation (CPG)

In the cycle-level process, CPG generates pseudo labels for training both the CLIP vision encoder and the target model (pre-trained source model) by combining their prediction probabilities. Note that we obtain two types of pseudo label ( ˆ p, ˆ y ) in CPG, performing prediction mixture (soft label, ˆ p ) and dual-perspective pseudo labeling (hard label, ˆ y ). Soft pseudo label obtained from prediction mixture will be used in DVO for performing vision optimization of CLIP. Hard pseudo label acquired from dual-perspective pseudo labeling will be employed in PLMatch which is represented in Fig. 3 for training the target model.

Prediction mixture To obtain the prediction mixture ( ˆ p ) which will be utilized in DVO as a target label, the prediction probabilities for a target domain sample from CLIP ( p clip ) and the target model ( p t ) can be obtained as follows:

where f V ( α ( x i )) and θ T ( ∈ R ( K × D ) ) represent the weak augmented image feature which is the output of CLIP vision encoder f V ( · ) and the text feature which is the output of CLIP's text encoder with a sentence in the format of 'a photo of [class]'[23], respectively. f t ( · ) and g t ( · ) are the feature encoder and classifier of the target model, respectively. ϕ ( · ) and sim ( · , · ) denote the softmax function and the cosine similarity function, respectively. τ is a pre-trained temperature parameter[23]. p clip i and p t i denote the prediction vector ( ∈ R K ) for the i -th weakly augmented target sample α ( x i ) from CLIP and target model, respectively. K is the number of classes and D is the latent dimension. Subsequently, the mixed prediction for a target domain sample is obtained by averaging the predictions as follows:

where ˆ p i ∈ R K represents a soft pseudo label. Note that the soft pseudo label ˆ p i is utilized in DVO, which is the cycle-level process.

Dual-perspective pseudo labeling To ensure high-confidence pseudo labels and mitigate the risk of noisy supervision, we employ a dual-perspective pseudo labeling strategy that selectively assigns pseudo labels only to samples where the predictions of CLIP ( p clip ) and the target model ( p t ) are aligned. Formally, we implement this selection mechanism using a binary mask m i , where samples meeting the alignment criteria are assigned to pseudo labels, while others remain unlabeled:

Here, δ ( · ) and 1 [ · ] are a one-hot encoding function and indicator function, respectively. m i = 1 if the predictions of both models are identical, and m i = 0 otherwise. Note that the hard pseudo label ˆ y i is utilized in PLMatch which is the iteration-level process.

3.1.2 Dynamic entropy-guided vision optimization (DVO)

Uncertainty-based adaptation index We define the uncertainty-based adaptation index ψ to dynamically control the entropy behavior of TMI in response to the model's adaptation uncertainty. The ψ allows balancing flexibility ( ψ > 1 ) and convergence ( ψ < 1 ) throughout training, and is directly incorporated into the computation of Tsallis mutual information. Specifically, likely with the entropy index q in Eq. 1, when the model exhibits high adaptation uncertainty, the adaptation index ψ is adjusted above one, resulting in a flatter curvature of the entropy function and encouraging learning from low-confidence samples (exploration process). Conversely, as the model's uncertainty decreases, the adaptation index ψ is reduced below one, which sharpens the curvature of the entropy function and promotes stable learning centered on high-confidence samples (exploitation process). For this, to comprehensively reflect the model's uncertainty, the batch-wise normalized entropy ˜ H (˜ p clip ) is firstly computed as follows:

where ˜ p clip denotes the averaged prediction vector of CLIP across a batch of size N . ˜ p clip j denotes the j -th class probability of the averaged prediction vector ˜ p clip , and K is the number of classes. By quantifying the model's predictive uncertainty through this approach, we utilize EMA to dynamically adjust the adaptation index ψ . During the early training phase, when a higher learning rate is beneficial, ψ is set to be greater than one to encourage exploration. As training progresses, the model requires more stable optimization, leading adaptation index ψ to gradually decrease below one, with its variations being smoothly regulated:

where µ acts as a momentum parameter that combines the previously computed ψ with the current ˜ H (˜ p clip ) .

Tsallis mutual information (TMI) Subsequently, the standard MI loss[7] is replaced with Tsallis mutual information (TMI), where the adaptation index ψ dynamically adjusts the entropy behavior based on the model's uncertainty. To optimize the CLIP vision encoder, the TMI I ψ ( p clip , ˆ p ) between the CLIP prediction p clip and the soft pseudo label ˆ p is defined as follows:

where P kk ′ denotes the joint probability of the k -th class probability from CLIP prediction p clip , and the k ′ -th class from the soft pseudo label ˆ p . P k and P k ′ are the marginal probabilities, representing the probability vectors of p clip and ˆ p , respectively. For details on how the joint and marginal probabilities are computed for TMI, please refer to Appendix C.

CLIP vision encoder optimization Tsallis mutual information (negative term) is used as a loss function to optimize the CLIP vision encoder, encouraging the model to maximize the mutual information between the CLIP prediction ( p clip ) and the soft pseudo label ( ˆ p ). This is formulated as:

By minimizing L TMI , the CLIP vision encoder is trained to enhance the alignment between the CLIP prediction p clip and the soft pseudo label ˆ p , while simultaneously adapting the adaptation index ψ to regulate the balance between exploration and exploitation during training.

3.2 Iteration-level process

3.2.1 Pseudo label matching framework (PLMatch)

PLMatch, which belongs to the iteration level process effectively (1) performs supervised learning using hard pseudo labels, (2) leverages unlabeled samples through consistency training, and (3) conducts CLIP-guided knowledge distillation.

Pseudo supervision The target model is trained by minimizing the cross-entropy loss over the unlabeled set, using the hard pseudo label ˆ y i and the mask m i obtained from dual-perspective pseudo labeling in CPG (Eq. 4). The corresponding formulation is as follows:

where z t i is the logit output of the target model for α ( x i ) . N l is the number of pseudo labeled samples.

Consistency training In addition to pseudo supervision, we apply consistency training to all target domain samples encouraging the target model to learn more robust feature representations rather than being overly confident in misclassifying samples that have not been assigned pseudo labels. As illustrated in Fig. 3, consistency training leverages both weakly augmented ( α ( x i ) ) and strongly augmented ( A ( x i ) ) versions of the same image ( x i ). The training process is formulated as follows:

where p t ( α ( x i )) and p t ( A ( x i )) represent the prediction vectors of α ( x i ) and A ( x i ) through the target model. KL ( ·||· ) denotes the Kullback-Leibler (KL) divergence between the two input distributions.

CLIP-guided knowledge distillation We further perform consistency training by minimizing the KL divergence between the prediction vectors from CLIP ( p clip ) and the target model ( p t ). The loss function for aligning the predictions of both models is formulated as follows:

This alignment stabilizes the target model's feature representation, leading to more reliable predictions and improved generalization in the target domain.

Target model training The three previously described loss functions are utilized for training the target model. These loss functions are combined with different weighting factors to formulate the final loss function, as defined below:

Here, α , β , and γ control the balance between losses, as further described in the experiments.

4 Experiments

Datasets To evaluate the effectiveness of our proposed method, we conducted experiments on three widely used benchmark datasets for source-free domain adaptation: Office-Home[34], VisDA-C[22], and DomainNet-126[21]. These datasets encompass diverse domains and category scales. Detailed descriptions of each dataset are provided in Appendix D.1.

Implementation details For reproducibility and clarity, detailed implementation settings-including backbone architectures, optimizer configurations, data augmentation strategies, and hyperparameters for each dataset-are thoroughly described in Appendix D.2.

4.1 Main results

As shown in Tabs. 1 and 2, our method consistently surpasses prior state-of-the-art approaches across all benchmarks, with improvements of 1.6% on Office-Home, 1.4% on VisDA-C, and 3.0% on DomainNet-126. The gains are especially notable in challenging domains such as Clipart and Sketch, where large domain gaps and stylistic differences hinder existing methods. These results highlight the robustness of our method in handling severe domain shifts and improving generalizable feature adaptation.

4.2 Contribution of investigated components

The ablation study presented in Tab. 3 evaluates the effectiveness of the proposed components in our framework, incorporating results on the Office-Home and VisDA-C datasets. First, when using only the pseudo supervision loss ( L sup ) with hard pseudo labels, the proposed model achieves 79.2% average accuracy on the Office-Home dataset, already surpassing existing source-free methods except DIFO[29]. On VisDA-C, our model achieves 90.2% accuracy, outperforming the state-of-the-art method DIFO by +0.2%. This strong performance of pseudo supervision highlights the superiority of

the proposed pseudo labeling strategy, and a more detailed analysis of our pseudo labeling strategy is provided in Appendix D.3.

Incorporating L con and L KL boosts accuracy by 5.5% (84.7%), highlighting their role in enhancing adaptation stability. Additionally, optimizing the CLIP vision encoder with Tsallis mutual information (TMI) results in the best overall performance, outperforming both KL-based and mutual information (MI)-based approaches. Notably, TMI-based vision optimization yields a larger performance gain in the Clipart domain ( → C ), indicating its effectiveness in mitigating domain shift when adapting to visually diverse domains.

4.3 Variations depending on the loss weighting

We conducted an ablation study on the Office-Home and VisDA-C datasets to investigate the effect of weights α , β , and γ corresponding to L sup , L con , and L KL , respectively. As shown in Tab. 3, the optimal configuration ( α = 0 . 4 , β = 0 . 2 , γ = 0 . 4 ) achieved the best overall accuracy of 84.7% on Office-Home and 91.4% on VisDA-C. Notably, the model maintains robust performance despite variations in loss weight settings. This setting also showed strong performance on → R (91.0%), but slightly reduced accuracy was observed when β was lowered or γ was increased, indicating the importance of consistency regularization and balanced loss combination.

4.4 Analysis of Tsallis mutual information

Figure 4 analyzes the optimization dynamics of Tsallis mutual information (TMI) and standard mutual information (MI) losses during domain adaptation from Art to Clipart in the Office-Home dataset. The top plot shows the adaptation index ψ decreasing over iterations, adaptively adjusting the sharpness of the TMI. The bottom plot compares TMI and MI losses before and after the adaptation index ψ approaches one. The shaded regions in both plots divide the training process into two stages: the early stage encourages exploration, allowing the model to learn from diverse and uncertain samples, while the later stage favors exploitation, focusing on confident and stable predictions. As shown in Fig. 4, TMI effectively regulates loss oscillations, likely due to its entropy indexcontrolled structure. The loss patterns indicate that TMI promotes an effective exploration-exploitation balance, assigning greater importance to uncertain samples early in training. This structured learning improves model stability and makes DVO a more robust solution for source-free do-

1.05

Index

Index y

1.00

Exploration

Exploitation

0.5

TMILoss

MILoss

-1.0

1.5

TMIstd:0.17!

MI std:0.13

MI std:0.17

Iteration

main adaptation than standard MI-based training. The impact of TMI-based CLIP vision optimization on our framework is further analyzed in detail in AppendixD.3.

4.5 Analysis of Pseudo label matching framework

In all cases, PLMatch reduces the gap between confidence and classification accuracy more effectively than DIFO. Even in challenging domain shifts like R to A, PLMatch maintains relatively low expected calibration error (ECE)[19] (0.02) compared to DIFO (0.10). The superior calibration of PLMatch can be attributed to its dual-perspective pseudo labeling strategy, which prioritizes high-confidence samples initially and progressively refines pseudo labels over time. Additionally, consistency training enforces alignment between different augmented views, preventing the model from becoming overconfident in noisy pseudo labels. Additional results for other adaptation scenarios are provided in Appendix D.5.

ECE=0.02

ECE=0.03

ECE=0.10

ECE=0.06

1.00

1.00

1.00

1.00

Gap

Gap

Gap

Gap

0.75

0.75

Accurac

0.75

0.75

pected

0.50

0.50

xpected

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.25

0.50

0.75

1.00

0.00

8.00

0.25

0.50

0.75

1.00

0.00

8.00

0.25

0.50

0.75

1.00

0.00

8.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(a) Ours (R→A)

(b) Ours (R→P)

(c)DIFO (R→A)

(d) DIFO (R-→P)

5 Conclusion and limitations

To address pseudo label noise in source-free domain adaptation (SFDA) and facilitate ViL model adaptation to the target domain, we introduce a novel pseudo labeling strategy that integrates task-specific perspectives and domain-invariant characteristics, along with an uncertainty-aware optimization strategy for the CLIP vision encoder. By leveraging domain-generalized CLIP predictions and taskspecific model outputs, we selectively assign high-quality pseudo labels. Additionally, consistency training refines feature representations for samples awaiting pseudo labeling, enhancing robustness. Our uncertainty-based optimization approach balances exploration and exploitation, allowing the CLIP vision encoder to self-regulate its learning intensity. Empirical results demonstrate that our method outperforms prior approaches, particularly under challenging domain shifts with significant domain gaps. While our method requires optimizing the CLIP vision encoder, this introduces nontrivial computational overhead. Although we limit this process to fewer than eight update steps in our method, the added cost may still be a concern in resource-constrained settings, suggesting the need for lightweight adaptation strategies or parameter-efficient tuning methods in future work.

References

D. Chen, D. Wang, T. Darrell, and S. Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 295-305, 2022.

W. Dong-DongChen and Z. WeiGao. Tri-net for semi-supervised deep learning. In Proceedings of twenty-seventh international joint conference on artificial intelligence , pages 2014-2020, 2018.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research , 17(59):1-35, 2016.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence , 45(1):87-110, 2022.

D. Hu, R. Luo, J. Liang, and C. S. Foo. Towards reliable model selection for unsupervised domain adaptation: An empirical study and a certified baseline. Advances in Neural Information Processing Systems , 37:135883-135903, 2024.

X. Ji, J. F. Henriques, and A. Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9865-9874, 2019.

M. Jing, X. Zhen, J. Li, and C. Snoek. Variational model perturbation for source-free domain adaptation. Advances in Neural Information Processing Systems , 35:17173-17187, 2022.

T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:12104-12114, 2020.

V. K. Kurmi, V. K. Subramanian, and V. P. Namboodiri. Domain impression: A source data free domain adaptation method. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 615-625, 2021.

J. Lee, D. Jung, J. Yim, and S. Yoon. Confidence score for source-free unsupervised domain adaptation. In International conference on machine learning , pages 12365-12377. PMLR, 2022.

J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning , pages 6028-6039. PMLR, 2020.

M. Litrico, A. Del Bue, and P. Morerio. Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7640-7650, 2023.

M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. Advances in neural information processing systems , 29, 2016.

M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems , 31, 2018.

S. Mascarenhas and M. Agarwal. A comparison between vgg16, vgg19 and resnet50 architecture frameworks for image classification. In 2021 International conference on disruptive technologies for multidisciplinary research and applications (CENTCON) , volume 1, pages 96-99. IEEE, 2021.

J. Moon, J. Kim, Y. Shin, and S. Hwang. Confidence-aware learning for deep neural networks. In international conference on machine learning , pages 7034-7044. PMLR, 2020.

K.-B. Nguyen and J.-S. Yang. Boosting semi-supervised learning by bridging high and low-confidence predictions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1028-1038, 2023.

J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran. Measuring calibration in deep learning. In CVPR workshops , volume 2, 2019.

J. H. Park, G. Lee, S. Park, and S. I. Cho. Not all classes stand on same embeddings: Calibrating a semantic distance with metric tensor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17722-17731, 2024.

X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems , 35:14274-14289, 2022.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

B. Sun, J. Feng, and K. Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications , pages 153-171, 2017.

S. Tang, A. Chang, F. Zhang, X. Zhu, M. Ye, and C. Zhang. Source-free domain adaptation via target prediction distribution searching. International journal of computer vision , 132(3):654-672, 2024.

S. Tang, Y. Shi, Z. Ma, J. Li, J. Lyu, Q. Li, and J. Zhang. Model adaptation through hypothesis transfer with gradual knowledge distillation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5679-5685. IEEE, 2021.

S. Tang, W. Su, M. Ye, and X. Zhu. Source-free domain adaptation with frozen multimodal foundation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23711-23720, 2024.

S. Tang, Y. Zou, Z. Song, J. Lyu, L. Chen, M. Ye, S. Zhong, and J. Zhang. Semantic consistency learning on manifold for source data-free unsupervised domain adaptation. Neural Networks , 152:467-478, 2022.

A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

J. Tian, J. Zhang, W. Li, and D. Xu. Vdm-da: Virtual domain modeling for source data-free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology , 32(6):3749-3760, 2021.

C. Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics , 52:479-487, 1988.

H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5018-5027, 2017.

Y. Wang, J. Chauhan, W. Wang, and C.-J. Hsieh. Universality and limitations of prompt tuning. Advances in Neural Information Processing Systems , 36:75623-75643, 2023.

Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems , 33:6256-6268, 2020.

Z. Xu, K. Sun, and J. Mao. Research on resnet101 network chemical reagent label image classification based on transfer learning. In 2020 IEEE 2nd International Conference on Civil Aviation Safety and Information Technology (ICCASIT , pages 354-358. IEEE, 2020.

S. Yang, J. Van de Weijer, L. Herranz, S. Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Advances in neural information processing systems , 34:29393-29405, 2021.

H.-W. Yeh, B. Yang, P. C. Yuen, and T. Harada. Sofa: Source-data-free feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 474-483, 2021.

B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in neural information processing systems , 34:18408-18419, 2021.

Z. Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16816-16825, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline our main contributions. These contributions are consistently developed and validated throughout the paper in both methodology and experimental sections (Section 3 and 4).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 5 (Conclusion and limitations), we briefly discuss the limitations of our approach, acknowledging potential directions for future improvement and broader applicability.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Appendix D.1 and D.2 provide detailed information on dataset setup, model architecture, training configuration to ensure reproducibility of the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the full source code and instructions as a zip file in the supplementary material to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full training and test details, including hyparparameters, optimizer settings, and training strategies are provided in Section 4.2, 4.3, Appendix D.1 and D.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Hyperparameter tuning and multiple experimental trials were conducted to ensure the reliability of the results. These details are discussed in Section 4.2 and Section 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources used in our experiments, including GPU specifications, are detailed in Appendix D.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We use publicly available datasets with proper licensing, avoid any identifiable personal data, and our method poses no foreseeable harm or misuse.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not observe any potential negative societal impacts from this work. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not provide new pretrained models or datasets that pose a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All external assets, including CLIP models and benchmark datasets are properly cited in the main paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduce new code for the proposed framework and loss functions, and provide appropriate documentation in the supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve any crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve any experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were not used as part of the method, model, or experimental pipeline. Any LLM usage was limited to grammar correction or writing support.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Table of Contents

A Implementation detail of toy example

Noise Level:0.980,MI:-0.0084,TMI:0.0468

0.60

Probability

0.45

GT

GT

GT

Pred.

Pred.

Pred.

0.30

0.15

0.00

01234567

7891011

01234567891011

01234567891011

Noise Level: 0.694,MI:-0.0089,TMI:0.0157

0.60

Probability

0.45

GT

GT

GT

Pred.

Pred.

Pred.

0.30

0.15

0.00

01234567891011

01234567891011

01234567891011

Noise Level: 0.388, MI: -0.0076,TMI: -0.0051

0.60

Probability

0.45

GT

GT

GT

Pred.

Pred.

Pred.

0.30

0.15

0.00

01234567891011

01234567891011

01234567891011

Noise Level: 0.083, MI: -0.0255, TMI: -0.0321

0.60

Probability

0.45

GT

GT

GT

Pred.

Pred.

Pred.

0.30

0.15

0.00

01

34567891011

01234567891011

01234567891011

Noise Level: 0.001, MI:-0.0504,TMI: -0.0586

0.60

Probability

0.45

GT

GT

GT

Pred.

Pred.

Pred.

0.30

0.15

0.00

01234567891011

01234567891011

01234567891011

We further investigate the limitations of using standard mutual information (MI) as an optimization objective by designing a toy example to simulate the MI-based training process under decreasing uncertainty. As shown in Fig. 6, we generate soft pseudo labels by randomly sampling class probability vectors from a uniform distribution and then normalizing them via the softmax function to ensure they represent valid probability vectors. Controlled Gaussian noise is subsequently added to these vectors to simulate noisy predictions. For each noise level, three random samples are visualized to illustrate the distributional shift. Each subplot displays a pair of class probabilities: one representing the ground truth (pseudo label) and the other a noisy version (prediction) of it. We use the soft label as the ground truth to reflect the actual training setup, where mutual information (MI) is computed based on the soft pseudo label. As the noise level gradually decreases (from top to bottom), the noisy predictions become more concentrated and closely aligned with the pseudo label. However, as discussed in Fig. 2(c), despite the noticeable distributional improvements, the MI value remains relatively insensitive to these changes. This demonstrates that MI lacks the flexibility to dynamically capture information changes across varying prediction quality levels, particularly in uncertain earlystage adaptation scenarios. This observation supports the need for an alternative information measure like TMI, which dynamically modulates sensitivity through the proposed adaptation index ψ . By

doing so, TMI provides more meaningful optimization signals during the both high-uncertainty and stable training phases, leading to more robust adaptation performance.

B Effect of entropy index q on Tsallis entropy

0.8

Entropy

0.6

0.4

q=0.5

q=0.8

q=1.0

0.2

q=1.2

q=1.5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

Probability

To better understand how the entropy index q influences the behavior of Tsallis entropy[33], we visualize the entropy values over a range of probabilities in Fig. 7. The shape of the entropy function varies significantly with entropy index q . When q < 1 , the entropy curve becomes sharper, increasing sensitivity to high-confidence predictions and encouraging the model to focus on confident samples (exploitation). Conversely, when q > 1 , the curve becomes flatter, reducing the sensitivity to confidence and allowing the model to explore uncertain predictions more freely (exploration). This property allows Tsallis entropy to flexibly control the trade-off between exploration and exploitation during training by simply adjusting the entropy index q , which is especially beneficial in domain adaptation scenarios with high uncertainty.

C Calculation of joint probabilities

To compute the Tsallis mutual information (TMI) between CLIP's prediction ( p clip ) and the soft pseudo label ( ˆ p ), we first construct a joint probability matrix P following the standard formulation adopted in prior works[7]. This matrix P kk ′ captures how often each class prediction from CLIP co-occurs with each class prediction from the soft pseudo label, across all samples in a mini-batch. Specifically, for each sample in the batch, we calculate the outer product of the two probability vectors ( p clip , ˆ p ) and then take the average across all samples. This gives us a matrix P kk ′ ∈ R K × K , where K and N are the number of classes and samples, respectively. The calculation of matrix P is followed:

The marginal probabilities P k and P k ′ are then obtained by summing this joint matrix P kk ′ across the rows and columns. The row sums represent the class probabilities from CLIP's prediction ( p clip ), and the column sums represent those from the soft pseudo labels ( ˆ p ). This is formulated as:

The joint probabilities and marginal probabilities are then jointly used in the computation of TMI, as formulated in Eq. 7.

D Experimental details

D.1 Benchmarks

To evaluate the source-free domain adaptation performance of our proposed method, we used three benchmark datasets: Office-Home[34], VisDA-C[22], and DomainNet-126[21]. Office-Home comprises 15,500 images across four domains (Art, Clipart, Product, and Real-world), each with 65 categories. VisDA-C is a large-scale simulation-to-real dataset with over 152k images spanning 12 categories, where training images are synthetically generated, and validation images are real-world samples from MS-COCO. DomainNet-126 is a large-scale dataset derived from DomainNet. As a refined subset, DomainNet-126 contains 145k images from 126 classes, samples from four domains (Clipart, Painting, Real, and Sketch).

D.2 Implementation details

Weemployed ResNet50[16] as the feature encoder for both the pre-trained source model and the target model on the Office-Home and DomainNet-126 datasets, while ResNet101[37] was used for VisDAC. The target model was trained using stochastic gradient descent (SGD) as the optimizer with a momentum of 0.9 and a weight decay of 0.001. The learning rate was set to 0.01 for Office-Home and 0.001 for both VisDA-C and DomainNet-126. In contrast, CLIP's vision encoder (ViT/B-32[5]) was optimized using the Adam optimizer with a fixed learning rate of 1 × 10 -7 . For consistency training, weak augmentation involves random cropping and horizontal flipping, while strong augmentation included randomly sized cropping, Gaussian blurring, and horizontal flipping to introduce greater variability. In terms of training schedule, the cycle was set to 4 for Office-Home and DomainNet-126, and 8 for VisDA-C, with each cycle comprising 4 iterations. The entropy index q and momentum µ were adjusted slightly for each dataset to stabilize training. The weights α , β , and γ of loss functions were set to 0.4, 0.2, and 0.4 for all datasets, respectively. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU.

D.3 Ablation details

D.3.1 Comparison between Tsallis mutual information (TMI) and mutual information (MI)

To examine the effectiveness and robustness of the proposed Tsallis mutual information (TMI), we conducted a comparison between mutual information (MI) and TMI across different random seeds, and additionally evaluated performance when replacing the backbone of the CLIP vision encoder. As shown in Tab. 5, the performance advantage of TMI over MI consistently appeared. Notably, the performance gap was more pronounced when using lightweight backbones such as ResNet[16, 37] variants, compared to the ViT-B/32[5] used in the final model.

Moreover, CLIP in our framework contributed in two major ways: knowledge distillation and pseudo label refinement. To assess TMI's contribution in these areas, we measured Expected Calibration Error (ECE) of CLIP, CLIP accuracy, and pseudo label accuracy on tasks where the performance improvement was most pronounced. As shown in Tab. 6, the results suggested that CLIP trained with TMI achieved a more stable and well-calibrated learning state than with MI, which positively affected the quality of the pseudo labels. These findings indicate that TMI-based training can be especially useful in various tasks.

D.3.2 Analysis on pseudo label

We conducted a quantitative analysis of pseudo label accuracy and sampling ratio per cycle on the Office-Home dataset [34]. As shown in Tab. 7, % values indicate pseudo label accuracy, and values in parentheses show the sampling ratio. Pseudo label accuracy declined across cycles (92.7%, 89.8%, 88.4%, 87.5%) as more pseudo labels introduced inevitable noise. However, prior work[18, 40] shows that such noise can improve generalization capability. This trade-off between pseudo label accuracy and sampling ratio is analyzed in detail in the following section.

D.3.3 Trade-off between accuracy and the number of pseudo labels

The drop in pseudo-label accuracy during later cycles is attributed to the increasing number of assigned pseudo labels, which highlights a natural trade-off between label quality and coverage. To further analyze this trade-off, we conducted additional experiments on the Office-Home dataset using confidence thresholding (CT) to assign pseudo labels, controlling the balance between pseudo label accuracy and quantity. As shown in Tab. 8, CT-based methods produced high-quality pseudo labels but labeled fewer samples than ours. In contrast, our approach achieved better overall performance and more stable training shown in Tab. 9. This aligns with the training strategy[2] of gradually incorporating lower confidence labels. This enables broader learning and mitigates overfitting, suggesting that our method ensures practical and stable label quality without complex thresholds or strategies.

D.4 Effectiveness of CLIP for target model training

The primary objective of the proposed framework is to train the target model, not to directly use CLIP as the final model. The purpose of CLIP fine-tuning is therefore not to replace the target model, but rather to help the target model training with pseudo label refinement and knowledge distillation.

As shown in Tab. 10, the final target model (Proposed) outperformed both the frozen (CLIP w/o tuning) and fine-tuned CLIP models (CLIP w/ tuning), confirming that CLIP is best used as an auxiliary module rather than the main predictor.

D.5 Expanded calibration analysis

As shown in Fig. 8, PLMatch consistently achieved a lower expected calibration error (ECE)[19] compared to DIFO[29] across various domain adaptation scenarios, indicating better alignment between confidence and accuracy.

1.00

ECE = 0.45

1.00

ECE = 0.24

1.00

ECE = 0.22

1.00

ECE = 0.10

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

8.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(a) DIFO (A→C)

(b) DIFO (C→A)

(c) DIFO (P→A)

(d) DIFO (R→A)

1.00

ECE = 0.39

1.00

ECE = 0.39

1.00

ECE = 0.44

1.00

ECE = 0.21

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

8.00

0.25

0.50

0.75

1.00

0.00

00:0

0.25

0.50

0.75

1.00

0.00

00:0

0.25

0.50

0.75

1.00

0.00

000

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(e)DIFO (A→P)

(f) DIFO (C→P)

(g) DIFO (P→C)

(h) DIFO (R→C)

1.00

ECE = 0.42

1.00

ECE = 0.42

1.00

ECE = 0.39

1.00

ECE = 0.06

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(i) DIFO (A→R)

(k) DIFO (P→R)

(1) DIFO (R→P)

1.00

ECE = 0.11

1.00

ECE = 0.11

1.00

ECE = 0.14

1.00

ECE = 0.02

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(m) Ours (A→C)

(n) Ours (C→A)

(0) Ours (P→A)

(p) Ours (R→A)

1.00

ECE= 0.07

1.00

ECE=0.08

1.00

ECE=0.10

1.00

ECE = 0.10

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

000

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

°8.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(q) Ours (A→P)

(r) Ours (C→P)

(s) Ours (P→C)

(t) Ours (R→C)

1.00

ECE=0.08

1.00

ECE = 0.07

1.00

ECE=0.08

1.00

ECE =0.03

Gap

Gap

Gap

Gap

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

Confidence

Confidence

(u) Ours (A→R)

(v) Ours (C→R)

(w) Ours (P→R)

(x) Ours (R→P)

D.6 Feature distribution analysis

Figure 9 represents the t-SNE visualizations of the target domain features produced by various adaptation methods. Compared to existing approaches such as SHOT[12], GKD[28], NRC[38], and DIFO[29], our method forms more compact and well-separated clusters with clearer boundaries.

(a) SHOT

(b) GKD

(c) NRC

(d) COWA

(e) PLUE

(f) TPDS

(g) DIFO

(h) Ours

Confusion Matrix

Confusion Matrix

Confusion Matrix

6

463200152136267814

602600912166532415893

5000

-10

1101245137259721

8055

0011100675320143106

5000

0

5113462293361011

1021766236206382

6000

n - 9 12674343 15 90 10 175 5 104156328

- 26 109 80 5768 82 1396137 754 99 945 111 894

4000

-914

676448 17 8510161 4101165312

4000

-1025

6947118 21 82 15 132 16 29 191 104

5000

-21700 4385123186116213

-318004414117176114713

-12101450382446131910

4000

3000

1

0011954010139102

3000

10319980795201

o - 9 534 0 12 27 1834552119 22299 9 30

o - 12691 0 19 46 117468264 22 117 8 18

3000

- 2 38 20 59 479 456244 13 117 0

-2220252402964119533

1

2000

- 341 2 058 47350 6244 17 110 02

-3

2804622121074116

504

8

2000

- 463 43824325162412986 05

·4

330

49179579

4213

153

5

2000

-790041381654204913

-20 19 192 30 7 76 1 131 4 10235

560

94

1000

9.9

-17 17 168 27 8 60 1 127 5 1023602102

11003128264620531

4

1000

-3 21 00101053461020810

-14 22 147 21 9 65 5 104 14 43

8733

59

1000

6241

-35 27 619722 63 183 52 319 35 193 853

6215

-43 41 821890 90 16675 269 64 1021102877

01234567891011

0

012345678910 11

0

012345678910 11

-0

Predicted

Predicted

Predicted

(a) SHOT

(b) GKD

(c) NRC

Confusion Matrix

Confusion Matrix

Confusion Matrix

n - 17 126910143 12 142 19 136 11 9 121 158

-23 2925 4912 166 182 65 24 52 8 5

○$480 2151674258152810

7000

6000

- 4234

-23

5609

2977

44354615963587

9669

1977498981118781215

44 15 38 13 77 13 77 152189

5000

○ 356745100931431777

- 8 3206 4 5 23 17 114 33 13 42 2 8

-1213

42481181681748 18 127 74

S000

7000

- 79 68 86 7155 73 1394182659 94 116 125 370

5000

-283 110166544490576151617178559822145

4000

n - 38 158 2228713116 185 207394 78 100 57 133

6000

-2245125

436076 7 59 6132 210

-2171

4

456519745121720

4000

01

091945 234145512

3000

5000

-28139 147 152575141571867917

- 17 29 1 452 407 75334118 54 1 1

3000

-74 55 12 16 63 230 67B19228 235 16 12

-2245473

30 13 73466470 154271 7 31

2000

-19366270392651513

-5 62 6 30 88 213 76339636 821

331060317

5

4000

3000

oo -10 15 0 4 11202 61284154

125

2

2000

0o -47 454 2863101424035 13 9

0-340091141939

4375

123

-14 6 2 35154 3 448 2038 3

1

-13 13 3 013 322 4500 121399 1

1

1000

0-4 13 3 5638829521671

2

2000

-19 11 202 32 8 133 3 106 16 166

6619 71

1000

-30 21 136 3 1155 23115 836

6810

39

-19 22218 33 518 5 50 9 22377065

1000

6343

-40 2511185273 13 73 131 27 30 512052

012345678910 11

0

01234567891011

0

-0

Predicted

Predicted

Predicted

(d) COWA

(e) PLUE

(f) TPDS

Confusion Matrix

Confusion Matrix

356220100104202121113

90329141010

8000

- 7 3048 1 7 11 29 202 95 19 41 2 13

7000

- 6 3160 5

911131824982327

7000

n- 9 1540475316 313 116 2 7 113296

6000

2.9

9

4206

58140135911104216

m - 21 73 43 7663 83 103 240101642 75 53 989

m -24 75 65609971 43 251696 45 40 48 944

6000

-2402455616867102213

5000

4-0

902

46091053411 542

5000

- 8123 0521538540370 18 45 3 21

4000

-61261611195

10

2

2 20332 562211

5503271217023

4000

- 4 33 3 21 55 159 82342612 201

3000

- 7 45 9 19 68 14693360 12233 3 5

3000

0o-31001055972454180

0-2200043430

5218825

0176

7

2000

∞ · 411

9.44

1

20197173222013

7

623 3 1004368

10106

2000

-11 11 8916 213 27289

3953

50

1000

0-1

892142313155

4042

32

1000

=- 19 16 140589 54 17 75 294 15 29 45

4255

=-26 13 281697 56 4 68 163 14 15 64

4147

012345678910 11

-0

012345678910 11

-0

Predicted

Predicted

(g) DIFO

(h) Ours

D.7 Confusion matrix comparison

Figure 10 presents the confusion matrices of various SFDA methods, including SHOT[12], GKD[28], NRC[38], COWA[11], PLUE[13], TPDS[27], DIFO[29], and our proposed method on the VisDAC[22] dataset. Each matrix illustrates the classification performance across all 12 classes, where diagonal elements indicate correctly predicted samples. Our method shows a clearly dominant diagonal pattern with significantly fewer off-diagonal elements compared to other approaches, indicating stronger class-wise discrimination and fewer misclassifications. This indicates the superior feature alignment and inter-class discrimination.

D.8 Computational overhead

To provide a clearer analysis of computational limitations, we report training time per batch (TTB), inference time (IT) per sample and GPU memory usage for the Art → Clipart task from the OfficeHome dataset[34]. Although our method requires higher GPU memory due to using both CLIP and the target model, its hierarchical structure-with separate cycle- and iteration-level processes-enables faster training than DIFO[29], which lacks such decomposition. Inference time per sample is identical across methods, as it depends only on the target model.

Peer reviews:
comments: SUMMARY:
The paper tackles source-free domain adaptation (SFDA) by jointly (i) calibrating pseudo labels from a task-specific classifier and CLIP (“dual-facet” hard labels) and (ii) fine-tuning the CLIP vision encoder with a Tsallis-mutual-information (TMI) loss modulated by an uncertainty-based adaptation index ψ. The combination is trained in repeated cycles of Calibrated Pseudo-label Generation (CPG), Dynamic Vision Optimization (DVO) and Pseudo-Label Matching (PLMatch) (Fig. 3). DUET achieves state-of-the-art accuracy on Office-Home, VisDA-C and DomainNet-126, improving the best prior method DIFO by 1.4 – 2.9 pp on average.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Well-motivated dual supervision: Requiring agreement between CLIP and the task model before assigning a hard label reduces noise and yields better-calibrated confidence.
2. Novel uncertainty-aware TMI objective: The ψ-controlled entropy index smoothly anneals from exploration (ψ > 1) to exploitation (ψ < 1), stabilising training and showing clearer loss dynamics than standard MI.
3. Thorough empirical study: Main results on three benchmarks, plus ablations on loss components/weights (Tab. 3) and analyses of calibration, t-SNE, and confusion matrices.

Weaknesses:
1. Limited statistical rigour – Results are single-run averages; no variance or significance tests are reported.
2. Accuracy improves but requires up to 8 cycles × 4 iterations of target-model updates and several vision-encoder updates.
3. CLIP dependence – Method assumes access to a large ViL foundation model.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new method for source-free domain adaptation, consisting of several components as follows. To improve the quality of pseudo-labels, the proposed dual-facet pseudo-labeling strategy assigns labels only to target samples on which both the target model and CLIP agree. The vision encoder of CLIP is trained with a novel loss based on Tsallis Mutual Information (TMI), which has desirable theoretical properties. Leveraging TMI enables the method to gradually shift from exploration to exploitation. Finally, pseudo-labeled target samples are used to supervise the training of the target model; unlabeled samples are utilized for representation learning via consistency training; and a KL-divergence-based loss is applied to distill knowledge from CLIP to the target model. Experimental results demonstrate that the proposed method achieves competitive performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe main contribution of this work is the introduction of Tsallis Entropy (TE) and Tsallis Mutual Information (TMI) into the area of source-free domain adaptation, which provides a novel and potentially valuable perspective.\n\nWeaknesses:\n1. Apart from TE and TMI, other components of the proposed method are relatively conventional. Dividing target samples into easy and hard ones is intuitive and has been explored in prior works [1, 2]. Utilizing unlabeled samples through consistency training is a standard technique in self-supervised learning, particularly in contrastive learning frameworks [3]. Knowledge distillation is also widely adopted when leveraging pre-trained models such as CLIP [4]. Therefore, the novelty of these components—except the TMI-based loss—is limited. Moreover, the introduction and motivation for incorporating TE and TMI are not sufficiently detailed or justified.\n2. The writing quality of the manuscript is subpar, which significantly affects readability and comprehension. There are multiple grammatical errors, even in the Abstract, which should be carefully avoided. For instance, the abstract sentence: “In a self-supervised manner, relying on pseudo labels on target domain samples facilitates the domain adaptation performance providing strong supervision.”\nis grammatically incorrect and semantically ambiguous. I suggest rephrasing it for clarity and conciseness. One possible revision could be: “In a self-supervised setting, pseudo labels on target domain samples provide strong supervision, thereby improving domain adaptation performance.” Overall, the language quality is not adequate for publication and requires substantial improvement to enhance clarity and readability.\n\nReferences:\n\n[1] Yang, Jianfei, Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. \"Divide to Adapt: Mitigating Confirmation Bias for Domain Adaptation of Black-Box Predictors.\" *The Eleventh International Conference on Learning Representations.*\n\n[2] M. Xia et al., “A Separation and Alignment Framework for Black-Box Domain Adaptation,” *Proceedings of the AAAI Conference on Artificial Intelligence*, Mar. 2024, pp. 16005–16013. doi: 10.1609/aaai.v38i14.29532.\n\n[3] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR 2020\n\n[4] Liang, Jian, Dapeng Hu, Jiashi Feng, and Ran He. \"Dine: Domain adaptation from single and multiple black-box predictors.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8003-8013. 2022.\n\nQUESTIONS:\n1. I suggest that the authors provide a more detailed introduction to the principles and essential elements of Tsallis Entropy (TE) and Tsallis Mutual Information (TMI), as these concepts are not widely familiar to most readers. Additionally, the motivation for incorporating TE and TMI into the source-free domain adaptation (SFDA) framework should be more clearly justified. After all, this is the core contribution of the work.\n\n2. The authors are encouraged to improve the clarity and readability of the manuscript. Careful language polishing will help ensure the proposed ideas are effectively conveyed.\n\n3. A specific question: What exactly do the terms “cycle-level process” and “iteration-level process” mean in the context of this paper? This phrasing is somewhat unclear and may require clarification or rewording for better comprehension.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a two-stage, uncertainty-aware source-free domain adaptation (SFDA) method by leveraging an external, pre-trained vision-language model, CLIP. The authors alternate between fine-tuning the CLIP vision encoder and adapting the source model, incorporating Tsallis mutual information (TMI) to guide optimization. The proposed framework demonstrates improved SFDA performance and enhanced model calibration.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- **Novelty**  \n  The use of Tsallis mutual information for uncertainty-aware training captures model dynamics during adaptation. This approach appears novel and may offer valuable insights to the community.\n\n- **Empirical Effectiveness**  \n  The proposed method improves performance on SFDA benchmarks. The experimental results further demonstrate that the adapted model is better calibrated under the uncertainty-aware training paradigm.\n\n### **Weaknesses**\n\n- **Insufficient Experimental Analysis**  \n  - The computational efficiency and scalability of the method are not sufficiently discussed. This raises concerns about the practical value of the proposed approach. Please refer to **Question 3** for more details.  \n  - There is no direct comparison with some recent state-of-the-art methods that also leverage pre-trained foundation models, such as [1].  \n  - I also have concerns regarding the overall completeness of the experimental section; please see **Questions 1 and 2** for specific points.\n\n- **Methodological Design and Motivation**  \n  - The proposed method incorporates a pre-trained vision-language model (CLIP) as a complementary module within the SFDA framework while continuing to train the vision encoder of CLIP. This design choice introduces several questions: Why not use CLIP directly as the final target model? Could the CLIP vision encoder be better integrated into the target model? These concerns are further elaborated in **Question 1**.\n\n- **Presentation and Clarity**  \n  - The overall presentation and mathematical formulations could be improved for better clarity and readability. Please refer to **Question 4** for specific suggestions.\n\n\n**Reference**  \n[1] Proxy Denoising for Source-Free Domain Adaptation. Song Tang, et al. *ICLR 2025*.\n\nQUESTIONS:\n1. Since the TMI loss is applied exclusively to the CLIP vision encoder, I am curious whether and how the improvements in CLIP representations contribute to the final adaptation performance of the target model. Could the authors provide further analysis on this aspect? Besides, is there any performance comparison between the fine-tuned CLIP encoder and the final adapted target model on the target domain?\n    \n2. What is the quality of the filtered hard pseudo labels? Would incorporating the mixed soft prediction $\\hat{p}$ into the PLMatch framework further improve adaptation performance?\n    \n3. The authors mentioned the computational limitations of the proposed method. Are there any runtime or resource usage statistics to support its practical applicability?\n    \n4. Some notations and terminology can be clarified.\n    \n    - For instance, in Line 210, what does \"EMA\" stand for?\n        \n    - Also, the formulation of the TMI loss in Eq. (8) appears incorrect. The \"min\" operator may be confusing or misplaced, as the negative TMI is already used as the optimization objective.\n\nLIMITATIONS:\nThe authors have mentioned their limitations in the Conclusion and limitations Section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem that prior methods in Source-Free Domain Adaptation (SFDA) rely on inaccurate pseudo-labels in the early stages of training, and the limited task-specific adaptability of CLIP-based prompt tuning due to its frozen vision encoder, which results in low-quality image representations. To solve this problem, the authors propose assigning pseudo labels only to samples where the target model and the CLIP model agree, while utilizing consistency learning for the remaining samples to enhance feature representation. Additionally, they propose an uncertainty-aware optimization strategy based on Tsallis mutual information (TMI) that dynamically aligns the CLIP vision encoder to the target by balancing exploration and exploitation. The proposed method outperforms existing baselines on standard SFDA benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper clearly highlights the limitations of prior methods and motivates the proposed approach through insightful experiments, particularly by visualizing the weaknesses of DIFO and the frozen CLIP encoder in Figures 1 and 2.\n\nWeaknesses\n1. The descriptions in the Introduction and Method sections are not well aligned, which hinders comprehension. While the Introduction briefly outlines only Dual-facet pseudo labeling and TMI-based vision optimization, the Method section introduces several new components—such as the Cycle-level process / Iteration-level process framework, CPG, DVO, and PLMatch—without prior context. As a result, it is unclear whether these components represent core contributions or simply implementation details.\n\n2. As shown in Table 3, the performance gain of TMI, which is one of the main contributions of the paper, is marginal compared to the standard MI. Considering that TMI introduces additional computational cost (e.g., the calculation of ψ), the benefit of using TMI may not be significant in practice.\n\n3. Both TMI in the Cycle-level process and CLIP-guided knowledge distillation in the Iteration-level process appear to align the CLIP model with the target model. However, only TMI incorporates adaptation uncertainty (ψ), while CL does not. The paper lacks a clear explanation as to why uncertainty modeling is applied to one but not the other, and further clarification on the design intention behind this asymmetry would be helpful.\n\nQUESTIONS:\n1. It would be helpful if the Related Work section included a dedicated paragraph on SFDA\n2. In Figure 1, why does the pseudo-label accuracy of the proposed method drop in later iterations? Is it because the number of assigned pseudo labels increases? If so, could the trade-off between accuracy and the number of pseudo labels assigned be controlled? And under what conditions would this be most efficient?\n3. In Figure 3, specifically in the CPG part, what is the reason for the varying bar widths?\n4. Why was the averaging factor set to 2 in the prediction mixture? Have you experimented with other weighting schemes?\n5. The paper states that consistency training is applied only to the remaining samples, which are not assigned pseudo labels. However, Equation (10) computes the loss over all N samples. Shouldn't it be restricted to the unlabeled subset? This discrepancy needs clarification.\n\nLIMITATIONS:\nThe limitations of the proposed method are described in Section 5 (Conclusion and Limitations).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}