Proxy Denoising for Source-Free Domain Adaptation
Authors: Song Tang
Wenxin Su
Yan Gan
Mao Ye
Jianwei Dr. Zhang
Xiatian Zhu
Venue: ICLR 2025 Oral (2025)
Keywords: Domain adaptation
source-free
multimodal proxy space
proxy confidence theory
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to an unlabeled target domain with no access to the source data. Inspired by the success of large Vision-Language (ViL) models in many applications, the latest research has validated ViL's benefit for SFDA by using their predictions as pseudo supervision. However, we observe that ViL's supervision could be noisy and inaccurate at an unknown rate, potentially introducing additional negative effects during adaption. To address this thus-far ignored challenge, we introduce a novel Proxy Denoising (__ProDe__) approach. The key idea is to leverage the ViL model as a proxy to facilitate the adaptation process towards the latent domain-invariant space. Concretely, we design a proxy denoising mechanism to correct ViL's predictions. This is grounded on a proxy confidence theory that models the dynamic effect of proxy's divergence against the domain-invariant space during adaptation. To capitalize the corrected proxy, we further derive a mutual knowledge distilling regularization. Extensive experiments show that ProDe significantly outperforms the current state-of-the-art alternatives under both conventional closed-set setting and the more challenging open-set, partial-set, generalized SFDA, multi-target, multi-source, and test-time settings. Our code and data are available at https://github.com/tntek/source-free-domain-adaptation.

Published as a conference paper at ICLR 2025
PROXY DENOISING FOR SOURCE-FREE DOMAIN ADAPTATION
Song Tang Wemxin Su', Yan Gan', Mao Yea Jianweii Zhang² & Xiatian Zhu 'University of Shanghai for Science and Technology. UUiieeritää Hamburg, 'ComOriginMat Inc. "Chongqing University, UUniversity Electronic Science and Technology of China, UUniversity Surrey t tangs@usst. edu. en, :swwenxin43, evlab. ueste? @gmail il. con, xiat ian. zhu@surrey, uk
ABSTRACT
Source-Free Domain Adaptation (SFDA) aims adapt pre-trained source model to an unlabeled target domain with no access to the source data. Inspired by the success of large Vision-Language (ViL) models in many applications, the latest research has validated ViL's benefit for SFDA by using their predictions pseudo supervision. However, we observe that ViL's supervision could be noisy and inaccurate at an unknown rate, introducing additional negative effects during adaption. To address this thus-far ignored challenge, we introduce a novel Proxy Denoising (ProDe) approach. The key idea is to leverage the Vil. model as a proxy to facilitate the adaptation process towards the latent domain-invariann space. We design proxy denoising mechanism to correct ViL's predictions, grounded on a proxy confidence theory that models the dynamic effect of proxy's s divergence against the domain-invarian space during adaptation. To capitalize on the corrected proxy. we derive a mutual knowledge distilling regularization. Extensive experiments show that ProDe significantly outperforms current state-of- the-art alternatives under the conventional closed set setting and more challenging open set, partial set, generalized SFDA, multi-target, multi-source, and test-time settings. Our code and data are available at https: / /ttp//ggttttt//tt com/ com/tntek tntek/ source- free- doma in- adaptation.
INTRODUCTION
Unsupervised Domain Adaptation (UDA) uses well-annotated source data and unannotated target data concurrently achieve cross-domain transfer. However, this data access requirement raises increasing concerns about safety and privacy. Thus. there is call for restricted access to source domain training data, leading to more practical but challenging transfer learning setting, Source-Free Domain Adaptation (SFDA) (Li et al., 2020a; Xia et al., 2021; Roy et al., 2022).
In the absence from the source domain, cross- domainn distribution matching approaches are no longer applicable (Ganin & Lempitsky, 2015; Kang et al., 2019). Self-supervised learning then comes into play by generating and mining auxiliary information enable unsupervised adaptation in two main routes. The first makes SFDA as special case of UDA by explicitly creating pseudo source domain, enabling UDA methods such as adversarial learning (Xia et al., 2021; Kurmi et al., 2021) or minimizing domain shift (Tian et al., 2022; Kundu al., 2022). The second further refines generated supervision from the source model (Lao et al., 2021; Wang et al., 2022a; Huang et al., 2021) or target data (Yang et al., 2022; Tang al., 2022), as the constructed pseudo source domain may be noisy. These methods all perform alignment without any guidance from the target feature space to the unknown domain-invarian feature space.
There has been growing interest in leveraging pre-trained Vision-Language (ViL) models (e.g.. CLIP (Radford al., 2021)) for transfer learning challenges. This is because ViL models were trained with a massive amount of diverse vasion-language data, encompassing rich knowledge potentially useful for many downstream tasks. For instance, Ge et al. (2022); Lai et al. (2023); Singha et al. (2023) disentangle domain and category information in the visual features of the ViL model
"Corresponding author

---

Published as a conference paper at ICLR 2025
by learning domain-specific textual or visual prompts. ViL models have also been used address the SFDA problem (Tang et al., 2024e; Xiao et al., 2024). They treat the ViL model's 's predictions as ground truth which would be noisy in many unknown cases, finally harming their performance.
To address the limitation mentioned above, in this pa- per, we propose new Proxy Denoising (ProDe) ap- proach for SFDA. In contrast to (Tang et al., 2024c; Xiao al., 2024), we consider the ViL model/space as noisy proxy of the latent domain-invariant space', with a need be denoised. At the absence of any good reference models for measuring the noisy degree with the already strong ViL. model's pre- dictions, we exploit the dynamics of domain adap- tation process, starting at the source model space and terminating presumably in the latent domain- invariant space. In particular, this takes into account the proxy's 's divergence against the comminn-nvariian space (Fig. 1). Specifically, we model approximately the effect of Vil. model's prediction error domain adaption by formulating a proxy confidence theory, in relation to the discrepancy between the source do- main and the current under-adaptation model. This leads to novel proxy denoising mechanism for ViL prediction correction. To capitalize on the corrected ViL predictions more effectively, mutual knowledge distilling regularization is further designed.
Source domain
Proxy ViL Space
Proxy error
Latent domain- invariant space
In training model
Adapting
+ Adapting direction -D Proxy correction
D Aligning to proxy space
-D Desired adapting direction
Figure 1: Conceptual illustration of ProDe. We align the adapting direction with the de- sired trajectory by leveraging proxy space that approximates the latent comain-invarian space. This process incorporates direction adjustments based on proxy error correction, implementing proxy denoising, and finally achieves enhanced model adaptation.
Our contributions are summarized as follows: (1) We for the first time investigate the inaccurate predictions of Vil. models the context of SFDA. (2) We formulate novel ProDe method that reliably corrects the Vil. model's predictions under the guidance of proxy confidence theory. A mutual knowledge distilling regularization is introduced for better capitalizing on refined proxy predictions. (3) Extensive experiments open benchmarks show that our ProDe significantly outperforms previous alternatives in closed-set settings, as well as the more challenging partial-set, open-set, and generalized SFDA, multi multi-target, multi-source and test-time settings.
2 RELATED WORK
Source-Free Domain Adaptation One main challenge with SFDA is lack of supervision during model adaptation. To overcome this, current methods are broadly divided into three categories. The firsteategory involves converting SFDA to conventional UDA by introducing a pseudo-source domain. This can be achieved by building the pseudo-source domain through generative models (Tian et al., 2022; Li et al., 2020b) or by extracting a subset similar to the distribution of sources from the target domain (Du et al., 2023). The second category involves mining auxiliary information from the pre-trained source model to assist in aligning the feature distribution from the target domain to the source domain. Commonly used auxiliary factors include multi-hypothesis (Lao et al., 2021), prototypes (Zhou et al., 2024), source distribution estimation (Ding et al., 2022), or hard samples (Li et al., 2021). The last category focuses on the target domain and creates additional constraints correct the semantic noise in model transferring. In practice, domain-aware gradient control (Yang et al., 2021b), data geometry such as the intrinsic neighborhood structure (Tang al., 2021) and target data manifold (Tang et al., 2022; Tang et al., 2024a), have been exploited to generate high-quality pseudo-labele (Liang et al., 2020; Chen et al., 2022b) or inject assistance in an unsupervised fashion (Yang et al., 202 These methods refine auxiliary information from domain-specific knowledge, such as the source model and unlabeled target data, without resorting to external knowledge sources, such as pre-trained multimodal foundation models.
'The issue of noisy predictions is evidenced by the inferior sero-shot performance of the ViL model, e.g.. CLIP, the target domains (see Tab. 4). Here, "domain invariant space" refers to ideal latent embedding space where the mapped features from different domains align with the same probability distribution.
2

---

Published as a conference paper at ICLR 2025
D D
[a]
02 to
(d)
d
(
Clock
mudel
d'sd"
Adapting
Toxx))
Source R
denoising
Mutual know ledge
Proxy space
distllig
Domaan--nvriaan space
d cv
d, In-training model at any time off
Inita tuing
X
Inpus instance
(x)x
Target modal
Loght level
-
Probability lovel
Ni0 (x (.4)
Confisience variation prediction
14)
confidence
eerrable prompt contectt
updatiigg
Soltmax oparot
se0 Codresimiaitty
Figure 2: Left: Dynamics effect of ViL model's prediction error (or proxy error) during alignment. (a) In the initial adaptation phase, it is acceptable overlook the proxy errors. However, as the in-training model approaches the proxy space, these errors grow to be more noticeable, leading to continuous decline in the reliability of ViL predictions as shown in (b) and (c). Right: Our ProDe capitalizes on the corrected proxy. involving a mutual knowledge distilling regularization and proxy denoising mechanism imposing refinement on the ViL logits.
Vision-Language Models ViL models, such as CLIP (Radford et al., 2021) and GLIP (Li et al., 2022), have shown promise in various tasks (Liang et al., 2023; Wang et al., 2022c) due to their ability to capture modality invariant features. There are two main lines of research. The first line aims to improve their performance. For instance, text-prompt learning (Zhou et al., 2022; Ge et al., 2022) and visual-prompt learning (Wang et al., 2023; Jia et al., 2022) were adopted, using learnable prompts related to application scenarios. Data efficiency of these models be improved by re- purposing (Andonian et al., 2022) or removing noisy data (Wang et al., 2021b). The second line focuses using ViL models external knowledge to boost downstream tasks. Three strategies involved: Plain fusion (Liu et al., 2024), knowledge distillation (Pei et al., 2023) and information entropy regulating (Cha et al., 2022). Beyond latest ViL based SFDA models (Tang et al., 2024c; Xiao et al., 2024), we uniquely tackle the challenge of mitigating the noise of ViL's supervision.
METHODOLOGY
3.1 PROBLEM FORMULATION
We start with a labeled source domain and an unlabeled target domain, handling the same categories. Let X's and Y's be the source samples and labels. The target samples and truth target labels denoted as Xy = {æ: } and ) = (yr Y respectively, where n is the sample number. SFDA aims to learn target model 0, X' + V' given (1) the pre-trained source model 0, X's + V's. (2) the unlabeled target data Nr. In addition, we leverage a ViL model 6g that produces noise supervision.
To address noisy ViL supervision, exploit the dynamics of domain adaptation process. As shown in Fig. 2 (a), we deal with three spaces: source domain Ds (i.e., source image embedding space), comain-invarian space Dz, and ViL space Dy (the best possible proxy w.r.t Dz). this context, Dr typically refers to an ideal, unknown latent embedding space that is domain generalized. We want to align the in-training model Di from Ds to Dr as E ~ 0
Without access to Dz, we propose to perform proxy alignment by aligning DJ towards Dy. We denote the discrepancy between Dr and Dy as proxy error eyz, reflecting ViL's prediction errors. We then transform the task of minimizing the errors of ViL. predictions to control the proxy error by establishing a proxy confidence theory.
3.2 PROXY CONFIDENCE THEORY
Understanding the impact of proxy emorr domain adaptation is critical. To account for the dynamics of domain adaptation. as demonstrated in Fig. (a), we consider two typical situations of the proxy alignment process. We denote the distance of Dr Dy and Dr d' and diz
3

---

Published as a conference paper at ICLR 2025
respectively, and note that the distinction between Dy and Dz. i.e., the proxy eyz, is space-to-she sp-ce distance in the vector form. To ease understanding, we note two cases:
Casel: When Dj is way far from Dy, e.g., at the beginning of adaptation (1 = 0), it is held that dy = di > eyz. This implies that aligning to Dr or Dy is equivalent. Consequently, the proxy emrors eyz be ignored, that is, the ViL prediction can be deemed trustworthy.
Case2: When Df approaches Dy, e.g., the later phase in the adaptation (1 = > 0), tackling the proxy errors becomes increasingly crucial; Also, the distance relationship evolves to the equation that di = dy eyz (according to the vector geometric property that u. u, and u + v form a triangle, where u and are two sides). At this moment, ViL predictions become less reliable.
The proxy errors dynamically affect the proxy alignment, as reflected in the relative relationship between di and dzz defined as:
d TJ d
d' + evzl
|dyl + levzl = 1 |db]
levz d
(1)
|db
where 7/2 quantifies the error impact degree. |.1 means the absolute value (length) of a distance vector. During proxy alignment, the quantity / in Eq. (1) gradually increases from a very small value (e.g.. Case 1) to bigger ones (e.g., Case 2), leading to increase in impact degree t from 1. With this dynamics, as shown in Fig. 2 (b), the variance of ViL prediction gradually increases, implying progressive decrease in the reliability of ViL prediction.
At any time f, we treat the ViL predictions that approximate Gaussian distribution NN (0. (zz) ,6)
with the mean 0. (2:) and the prediction variance & X 1)p (Fig. (c)). This is because, we consider the ViL predictions to be influenced by various sources of noise and uncertainty, which justifies the Gaussian approximation according to the Central Limit Theorem (Chow & Teicher, 1988).
Given that eyz is unknown, we cannot formulate these dynamics explicitly. We thus approximate this problem by quantifying the prediction variance with the varying confidence of ViL. predictions. This conversion can be expressed in the form of probability distribution with proxy confidence as:
N (0& (z:), 5,) = P (GpvV) True, PVV),
where P(V) is the probability distribution of the proxy space Dy; GPP) stands for random event that the sampling result (i.c., a ViL prediction) from P(V) is confident; P (Grgy) = True, t) is denoted as the proxy confidence, indicating the probability of the event GP(y) being true time t. This confidence will decreases progressively, as the Vil. prediction reliability reduces relatively against the ability of the in in-training model.
By framing the ViL prediction as a probabilistic event, we can leverage the concept of proxy confidence, P (Gp(V) True, 1), quantify the reliability of ViL predictions at any point during adaptation. This facilitates the measurement about the impact of proxy errors. Specifically, we formulate the proxy confidence theory as in Theorem 1 (see proof in Appendix A.).
Theorem We note that the source domain ( (Ds), the Homain-invariaae space (D)), the proxy space (Dy) and the in-training model (Dz, follow the probability distributions P(S), P(I), P(V) and )(T)), respectively, where S, I, V and Tt are corresponding random variables. With our proxy alignment idea (see Sec. 3. 2), the proxy confidence can be expressed as:
P(T))
P PGP(V) = =Truc,t) x P(S)
This theorem tells that the effect of ViL. prediction errors on domain adaption can be approximately estimated by contrasting the distributions of the source model and the current in-training model.
3.3 CAPITALIZING ON THE CORRECTED PROXY
Overview To better leverage the corrected proxy. we propose a novel ProDe method featured with two designs: (1) A proxy denoising mechanism, refining the original ViL predictions at the logit level, and (2) mutual knowledge distilling regularization, encouraging extraction of useful knowledge from the ViL. model 0, to the in-training target model 0,, shown in Fig. 2 (d).
4

---

Published as a conference paper at ICLR 2025
Proxy denoising This module aims to denoise the ViL predictions. By Theorem (Eq. (3)),
further convert the ViL space's 's probability distribution with proxy confidence (i.e., Eq. (2)) into
P(T')
P()) log P(V) |log P(S) - log P(T))] P(S)
log
(4)
where the latter two items form an adjustment used to correct for the first item (i.e., ViL prediction). Under this formula, we realize our denoising mechanism as:
p = softmax (0,, (æ,, v) w|O, (xx) 0g (zx))),
(5)
where 0,/0 apply the ViL/source/target model get the corresponding logits, and the hyper- parameter w specifies the correction strength. The output p is a denoised ViL prediction.
Mutual knowledge distilling This component aims distill useful knowledge of the ViL. model to our target model. This is achieved by designing two loss terms:
Laet
(6)
LProDe
min a -Ex,, MI (pi.p.) + + qo log ge - Exe, ex Σ ft..
c wi log
c=1
The first term Lppt adapts both the target model and the learnable prompt of ViL. model by maximizing the unbiased mutual information MI(...) .) (Ji et al., 2019) between the denoised ViL prediction pl and the target prediction noisy) training data used, the ViL model (e.g.. CLIP) don't always outperform speical expert model such as the supervised source model. There are three reasons: (1) ViL models are generalists, while source domain models are specialized. (2) ViL models may include irrelevant data, whereas source domain models use curated, relevant data. (3) ViL. models might overlook task-specift features that are captured by source domain models. To avoid solution collapse (Ghasedi Dizaji et al., 2017), we use a common category balance constraint (Yang et al., 2021a) where ge = A - Pi.c is the average likelihood of class c over n training samples by the target model, across a total of C' categories.
softmax (z::). This design is motivated by that despite massive (often
The second term LeaF refers to a typical pseudo labeling strategy where classification objective is applied, with the pseudo label 1/1 obtained by the denoised ViL predictions and 1 [c = y/1 denotes an indicator function. Note that the training proceeds, the ViL predictions become less reliable and useful whilst the negative effect eyz would grow in a relative sense. That means our proposed denoising could get more important across adaptation.
We provide the model training procedure in Appendix B.
4 EXPERIMENTS
Datasets We evaluate four widely used domain adaptation benchmarks. Among them, Office- 31 (Saenko et al., 2010) and Office-Home (Venkateswara et al., 2017) small-scaled and medium- scale datasets, respectively, whilst VisDA (Peng et al., 2017) and DomainNet-126 (Saito et al., 2019) both challenging large-scale datasets. Their details are provided in Appendix C.
Settings We consider a variety of SFDA settings: (1) closed-set, (2) partial-set (initialized in
SHOT (Liang al., 2020)), (3) open-set (initialized in SHOT (Liang et al., 2020)), (4) generalized SFDA (Yang et al., 2021b), (5) multi-target (SF-MTDA, detailed in (Kumar et al., 2023)), (6) multi- source MSF-MSDA, detailed in (Ahmed et al., 2021)), and (7) test-time adaptation (TTA) (Wang et al., 2021a). More details are given in Appendix D.
4.1 COMPETITORS
To evaluate ProDe, we select 30 related comparisons divided into four groups (1) The first includes 2 base models involved in the SFDA problem: The source model (termed Source) and CLIP zero-shot (termed CLIP) (Radford et al., 2021). (2) The second includes 7 current state-of-ther art domain adaptation methods with ViL model (adopting CLIP in practice), covering UDA and SFDA settings: DAPL-R (Ge et al., 2022), PADCLIP-P (Lai et al., 2023), ADCLIP-R (Singha et al., 2023), PDA- (Bai al., 2024), DAMP-R (Du al., 2024), DIFO-R (Tang et al., 2024c) and DIFO-V (Tang
5

---

Published as a conference paper at ICLR 2025
et al., 2024c). Among them, DIFO-R and DIFO-V V are the SFDA methods, while others are UDA methods. The suffix of "-R" and "-V" means that the image-encoder in CLIP uses the backbone of ResNet and ViT, respectively. Specifically, DIFO-V employs the backbone of ViT-B/32 across all datasets, whilst the rest methods with "-R" use ResNer101 on VisDA and ResNet50 on the other three datasets. (3) The third comprises 16 state-of-the-art SFDA models without using ViL. model: SHOT (Liang et al., 2020), NRC (Yang et al., 202 GKD (Tang et al., 2021), HCL (Huang et al., 2021), AaD (Yang al., 2022), AdaCon (Chen al., 2022a). CoWA (Lee et al., 2022), ELR (Yi et al., 2023), PLUE (Litrico et al., 2023), CRS (Zhang et al., 2023), CPD (Zhou et al., 2024), TPDS (Tang et al., 2024a), GDA (Yang et al., 2021b), PSAT-ViT (Tang et al., 2024b) CoNMix (Kumar al., 2023) and DECISION (Ahmed et al., 2021). Among them, GDA and PSAT-ViT are specific for the generalized SFDA setting. while CoNMix and DECISION are SF-MTDA and SF-MSDA methods, respectively. (4) The fourth comprises 5 hate-of-theeart TTA models: Tent (Wang et al., 2021a), T3A (Iwasawa & Matsuo, 2021), CoTTA (Wang et al., 2022b), EATA (Niu et al., 2022) and SAR (Niu et al., 2023). Additionally, for a fair comparison with DIFO, the previous best SFDA method with ViL model, we have initiated ProDe into the same versions mentioned above: A strong version ProDe-V and a weak version ProDe-R.
4.2 COMPARATIVE EVALUATIONS
Closed-set SFDA. Tab. I~3 lists the quan- titative comparisons on the four evaluation datasets. Both ProDe-R and ProDe-V beat all non-multimodal SFDA methods by a large margin. Compared with the second- best method CPD (Office-31), TPDS (Office-Home), PLUE (VisDA) and GKD (DomainNet-1226 126), P improves by 1.8%. 11.0% 2.7% and 16.3% in average accuracy, respectively. As for those methods with CLIP, ProDe also beat them in the same backbone setting. In par- ticular, compared with the multimodal SFDA method DIFO, ProDe improves by 4.8% and 5.0% (DomainNet- 126) at most using ResNet and ViT-B/32, respectively. Actually, the weak version of our method, ProDe-R, competitive with the strong version of DIFO, DIFO-V. All of these results indicate that ProDe can signifi- cantly boost the cross domain adaptation under the SFDA setting.
Table I: Closed-set SFDA results (%) on Office-31. SF means source-free.
Methed
Venue
SFJA-DA A D - W - Avg.
Source
|-
79.1
76.6
59.9
95.5
61.4
98.8
78.6
SHOT
ICML2U
/
93.7
91.1
74.2
98.2
74.6
100.
88.6
NRC
NIPS21
96.0
75.3
99.0
75.0
89.4
GKD
IROS21
/
94.6
91.6
75.1
98.7
75.1
100.
89.2
HCL
NIP521
94.7
92.5
75.9
98.2
77.7
100.
89,8
AaD
NIPS22
/ 96.4
92.1
75.0
99.1
76.5
100,
89.9
AdaCon
CVPR22
87.7
83.1
73.7
91.3
77.6
72.8
81.0
CoWA
ICML22
94.4
76.2
98.5
77.6
90.3
BLR
ICLR23
93.8
93.3
76.2
98.0
76.9
100.
89.6
89.2
72.8
97.1
69.6
85.8
CPD
PR24
96.6
94.2
77.3
98.2
78.3
100.
90.8
TPDS
IJC224
97.1
94.5
75.7
98.7
75.5
99.8
90.2
DIFO-R CVPR24
93.6
92.1
78.5
95.7
78.8
97.0
89.3
DIFO.V CVPR24
97.2
83.0
97.2
83.2
92.5
ProDe-R
94.4
92.
79.8
95.6
79.0
98.6
89.9
ProDe-V
96.8
96.4
83.1
97.0
82.5
99.8
92.6
Comparison to the ViL model. We conducted quantitative comparison between our model and CLIP's s zero-shot performance. The results of our model are reported with average accuracy. As reported in Tab. 4, ProDe-R and ProDe-V improve least by 5.0 % (on VisDA) and 8.1% (on VisDA), respectively, compared with CLIP's results on the four datasets. This result shows that the multimodal CLIP space only approximates the domain- domain-invarian invariant space, suggesting the need for denoising that this paper focuses on.
Partial-set and open-set SFDA. For a complete evaluation, we also evaluate ProDe on two variation scenarios: Partial-sel set and open-set settings. As reported in Tab. 5, ProDe-V achieves gain of 0.1 % (partial-set) and 6.7% (open-set) compared with the best competitor DIFO-V.
Generalized SFDA. The generalized SFDA is an extended problem of closed-set SFDA, high- lighting the norreforgetting ability on the seen source domain. The same as (Yang et al., 2021b), we adopt the harmonic mean accuracy as evaluation protocol, which is computed by H (2 Acc, * Accr)/(Acc,, + Acc.) where Ace, and Acc, are the accuracies of the adapted target model on the source domain and the target domain, respectively. Note that the Acc, is computed based on the cource-testing set. The same to (Yang et al., 202 Tang et al., 2024b), on the source domain, the ratio of training and testing sets is 9:1. To evaluate effectiveness, two generalized SFDA methods, GDA and PSAT- ViT, are chosen as additional comparisons. Based on Tab. 6, is seen that VroDe-V outperforms all comparisons in terms of H-accuracy besides PSAT-ViT VIT with forgettnggt

---

Published as a conference paper at ICLR 2025
Table Closed-set SFDA results (%) Office-Home and VisDA. SF means source-free. The full results on VisDA are provided in Appendix E. 1.
Office-oome
Mothod
Venue
Ar-CCIAr PrAA--RRe Rw ArCI CCC-PCII--------P -PrCI-PRw PRR.rr-PPArrr AAP---Arrattee -CI----IIP---PP- ArRR +CIR PPrAg (yy RR
Source
43.7
67.0
73.9
49.9
60.1
62.5
51.7
40,9
72.6
64.2
46.3
78.1
59.2]
49.2
SHOT
ICML20
56,7
77.9
80.6
68.0
78.0
79.4
67.9
54.5
82.3
74.2
58.6
84.5
71.9
82.7
NRC
NIPS21
57.7
80.3
82.0
68.1
79.8
78.6
65.3
56.4
83.0
71.0
58.6
85.6
72.2
85.9
GKD
IROS21
78.2
79.1
82.6
AaD
NIPS22
59,3
79.3
82.1
68.9
79.8
79.5
67.2
57.4
83.1
72.1
58.5
85,4
72.7
88.0
AdaCon
CVPR22
47.2
75.1
75.5
60.7
73.3
73.2
60.2
45.2
76.6
65.6
48.3
79.1
65.0
86.8
CoWA
ICML222
56.9
78.4
69.1
79.9
57.2
82.4
72.8
60.5
84.5
ELR
ICLR25
58.4
78.7
81.5
69.2
79.5
79.3
66.3
58.0
82.6
73.4
59.8
85.1
72.6
85.8
PLUE
CVPR23
49.1
73.5
78.2
62.9
73.5
74.5
62.2
48.3
78.6
68.6
51.8
81.5
66.9
88.3
CPD
PR24
59.1
79.0
82.4
68.5
79.7
79.5
67.9
57.9
82.8
73.8
61.2
84.6
73.0
85.8
TPDS
UCV24
80.3
80.9
82.1
74.5
85.3
DAPL-R
TNNLS23
54.1
84.3
84.8
74.4
83.7
85.0
74.5
54.6
84.8
75.2
54.7
83.8
74.5
86.9
PADCLIP.RICCV23
RICCV33
78.1
86.7
ADCLIP-R
JCCVW23
55.4
85.2
85.6
76.1
85.8
86.2
76.7
56.1
85.4
76.8
56.1
85.5
75.9
87.7
PDA-R
AAAI24
55.4
85.1
85.8
75.2
85.2
85.2
74.2
55.2
85.8
74.7
55.8
86.3
75.3
86.4
DAMP-R
CVPR24
59.7
88.5
86.8
76.6
88.9
87.0
76.3
59.6
87.1
77.0
61.0
89.9
78.2
88.4
DIFO-R
CVPR24
62.6
87.5
87.1
79.5
87.9
87.4
78.3
63.4
88.1
80,0
63.3
87.7
79.4
88.6
DIFO
CVPR24
70.6
90.6
88.8 8
82.5
90.6
88.8
80.9
70.1
88.9
83.4
70.5
91.2
83.1
90.3
ProDe-R
64.0
90.0
88.3
81.1
90.1
88.6
79.8
65.4
89.0
80.9
65.5
90.2
81.1
88.7
ProDe-V
Table Closed-set SFDA results (%) DomainNet-126. SF means source-free.
Method
Venue
SF
C-+P
C-R
C-88
P--C
P-RR
P-+8
R-C
R-+P
R-+S
S-aC
8-+P
S-R
Avg.
Source
44.6
59.8
47.5
53.3
75.3
46.2
55.3
62.7
46.4
55.1
50.7
59.5
54.7
SHOT
ICML20
63.5
78.2
59.5
67.9
81.3
61.7
67.7
67.6
57.8
70.2
64.0
78.0
68.1
77.4
81.4
NRC
NIPS21
/
62.6
77.1
58.3
62.9
81.3
60.77
64.7
69.4
58.7
69.4
65.8
78.7
67.5
AdaCon
CVPR22
V
60.8
74.8
55.9
62.2
78.3
58.2
63.1
68. 1
55.6
67.1
66.0
75.4
65.4
CoWA
ICML22
80.6
60.6
79.8
60.0
69.0
65.8
79.9
PLUE
CVPR23
V
59.8
74,0
56.0
61.6
78.5
57.9
61.6
65.9
53.8
67.5
64.3
76.0
64.7
TPDS
ICC224
V
62.9
77.1
59.8
65.6
79.0
61.5
66.4
67.0
58.2
68.6
64.3
75.3
67.1
DAPL-R
TNNLS22
X
72.4
87.6
65.9
72.7
87.6
65.6
73.2
72.4
66.2
73.8
72.9
87.8
74.8
ADCLIP-R
ICCVW23
X
71.7
88.1
66.0
73.2
86.9
65.2
73.6
73.0
68.4
72.3
74.2
89.3
75.2
DAMP-R
CVPR24
X
76.7
88.5
71.7
74,2
88.7
70.8
74.4
75.7
70.5
74.9
76.1
88.2
77.5
69.4
88.7
74.8
74.7
DIFO-V
CVPR24
76.6
87.2
74.9
80.0
87.4
75.6
80.8
77.3
75.5
80.5
76.7
87.3
80.0
ProDe-R
79.3
91.0
75.3
80.0
90.9
75.6
80.4
78.9
75.4
80.4
79.2
91.0
81.5
ProDe-V
83.2
92.4
79.0
92.3
79.3
85.5
79.1
85.5
83.4
92.4
85.0
design (by a tiny gap of 0.2%). Mcanwhile, both ProDe-R and ProDe-V deliver balanced results across the source and target domains. This is due the correction in the proxy denoising, which incorporates information from the source model, thereby mitigating forgetting of the source domain.
SF-MTDA, SF-MSDA and TTA. This part evaluates ProDe in broader SF-MTDA. SF-MSDA and TTA settings. For SF-MTDA, we treat multiple target domains as single integrated domain and adapt the source model accordingly. For SF-MSDA, we follow the ensembling approach from (Ahmed et al., 2021), passing the target data through each adapted source model and averaging the soft predictions to derive the test labels. The results, as shown in the left side Tab. 7, demonstrate that ProDe substantially outperforms state-of-the-a alternatives in both settings.
The right side of Tab. reports results on the online SFDA setting of TTA, where all comparison methods maintain a fixed batch size of 64, similar to ours. It is seen that ProDe demonstrates advantages over previous state-of- ootte--f-the-rt methods.
4.3 MODEL ANALYSIS
Feature distribution visualization. Based on the task CI-AA in the Office-Home dataset, we conducted toy experiment to visualize the feature distribution of ProDe using the t-SNE tool.

---

Published as a conference paper at ICLR 2025
Meanwhile, five comparisons considered, including CLIP-V, V, SHOT, TPDS, DIFO-V and Oracle. Among them, CLIP-V is the zero-shot result, and Oracle is trained on target domain Ar with the ground truth. For clear view, all results are presented in 3D density charts. As shown in Fig. 3, from CLIP-V to Oracle, category clustering becomes increasingly apparent. The distribution shape of DIFO-V V and ProDe-V is closer the expert model than that of non-multimodal methods, SHOT and TPDS, Furthermore, although DIFO-V and ProDe-V have a similar pattern, ProDe-V's shape is more detailed with Oracle.
Ablation studies. This part isolates the effect of (1) the objective components in Eq. (6) and (2) proxy denoising (PD). Tab. presents the ablation study results, with the baseline being the results of the source model (1 row). When Apt or Let is used alone (2, row), their performances show similar average accuracy. However, when they work together, the best results are achieved (4 row). This comparison indicates that the proposed two losses jointly contribute to the final performance. Additionally, we further evaluate the mutual information item MI(..) in Apt with a variant of ProDe, denoted ProDe w KL, where MI(-.) is replaced by the KL divergence loss. A significant average gap 7.2% (compared with the results in 4 row) confirms the advantage of the mutual information optimization (5 row).
Furthermore, removing proxy denoising from the model (ProDe-V w/o PD in row) leads to a decrease in average accuracy by 1.1%, which confirms its effectiveness. To evaluate the effect of components the proxy denoising design, we respectively remove the source and target models' logits (see Eq. (5)) to obtain w/o PD-target. As listed in 7 and rows, using either adjustment alone led to a significant decrease in performance. Also, perform the correction at the probability level, instead of the logit level, in another comparison ProDe-V w/o PD-logits. The average 3.1 % decrease (compared with ProDe-V's results in row) confirms the rationality correction based on logits (9 row).
ProDe variation methods, ProDe-V w/o PD-source and ProDe-V V
Table 4: Comparison results with CLIP (%). Appendix E. presents the full results.
Table 6:
Generalized SFDA results (%) on Office-Home. S, T are the results of the adapted target model on the source and target domains, i.e., Acc,, Ace, respectively; WAD means With Anti-forgetting Design. Appendix E. presents the full results.
Method
Office-31
Office-Home
DomainNet-126
CLIP-R
71.4
72.1
83.7
72.7
ProDe-R
89.9
81.1
88.7
CLIP-V
79.8
76.1
82.9
76.3
ProDe.V V
92.6
84.5
91.0
85.0
Avg.
Method
Venue
WAD
(98.1-S)
Table 5: Partial-set and open-set results (%) on
Source
x 98.1
59.2
73.1
Home Appendix E presents the full results.
SHOT
ICML20
X
84.2 (13.9)
71.8
77.5
X
(11.3)
NRC
NTP521
X
91.3 (6.8)
72.3
80.7
Partial-set
Avg.
Open-set
Avg.
AdaCon
CVPR22
X
88.2 (9.9)
65.0
74.8
CaWA
ICML22
X
91.8 (6.3)
72.4
81.0
Source
62.8
Source
46.6
PLUE
CVPR23
X 96.3 (1.8)
66.9
79.0
SHOT
ICML20
SHOT
ICML20
72.8
TPDS
DCV24
X 83.8 (14.3)
73.5
78.3
HCL
NTPS21
79.6
HCL
NIPS21
72.6
GDA
(18.1)
CoWA
JCML22
83.2
CoWA
ICML.22
73.2
PSAT-WT
TMM24
(6.4
83.6
85.0
NIP522
NIPS22
DIFO-R
CVPR24
X
(19.8)
79.4
78.8
CRS
CVPR23
80.6
CRS
CVPR23
73.2
DIFO-V
CVPR24
X
78.0 (20.1)
83.1
80.5
DIFO-V
CVPR24
84.1
DIFO-V
CVPR24
75.9
ProDe-R
X
84.9 (13.2)
81.1
82.9
ProDe-V
ProDe-V
X
85.1 (13.0)
84.5
84.8
Table 7: SF-MTDA, SF-MSDA and TTA results (%) on Office-Home. The full results of TTA are provided in Appendix E.
Model
Venue
Ar-
Cl++
Pr-+
Rw-+
Avg.
Method
Veruee
Avg.
SF-MIDA
CoNMix
WACV23
75.6
71.4
73.4
75.4
Tent
ICLR20
61.7
ProDe-V
83.3
89.2
80.9
83.6
T3A
NcurIP21
63.8
Method
-tPr
-+C1
Avg.
TTA
CoTTA
CVPR22
SHOT-Ens
JCML20
82.9
82.8
59.3
72.2
74.3
EATA
ICML22
60.7
SF-MSDA
DECISION
CVPR21
83.6
84.4
59.4
74.5
75.5
SAR
ICLR23
60.3
ProDe- V-EEs
91.1
92.5
73.4
83.0
85.0
ProDe-V
76.5
8

---

Published as a conference paper at ICLR 2025
(a) CLIP-V
(b) SHOT
(c) TPDS
(d) DIFO-V
(e) ProDe-V
(f) Oracle
Figure 3: Feature visualization comparison in 3D density charts.
Table 8: Ablation study results (%) on Office-
Table 9: Comparison results (%) on Office-31, Office-Home and VisDA as image encoder back- bone in CLIP adopts architecture ViT-B/16. SI
31, Office-Home and VisDA.
LAps
Lite
Office-31 Office-Home VisDA Avg.
means source- free.
78.6
59.2
49.2
62.3
77.5
90.7
86.5
Method
Venue
SF
Office-31 Office-Home
VisDA
87.3
80.5
87.3
85.0
CLIP-VI6
ICML21
X
77.6
80.1
85.6
ProDe-V KL
83.7
72,9
89.8
82.1
INNLS23
ADCLIP-VI6
ICCVW23
X
86.1
90.7
ProDe-V V w/o PD
91.6
82.3
90.6
88.2
PAD-V16
AAAI24
91.2
85.7
89.7
ProDe-V
90.9
DAMP-VI6
CVPR24
ProDe wdo PD larget
91.0
82.3
89.9
87.7
DIFO-VI6
CVPR24
92.2
85.5
91.0
ProDe-V PD-logits
81.5
86.2
ProDe-V16
92.2
86.9
91.7
Impact image encoder backbone in CLIP. addition to the ResNet and ViT-B/32 architectures aforementioned, we also implement ProDe using another well-known architecture, ViT-B/16, which we refer to as ProDe- V16. Furthermore, we compare the performance of CIIP-VI6, DAPL-V16 ADCLIP-VII 16. PAD-V16, DAMP-VI6 and DIFO-V16, which also use ViT-B/16 as their image encoder. As listed Tab. 9, ProDe-V16 still surpasses all comparisons. Combining with the ResNet and ViT-B/32 results reported in Tab. 1~Tabb 2, it is concluded that the advantage of ProDe is robust to the selection of the image-encoder backbone.
Comparison with SFDA methods with ViT backbone. To achieve a comprehensive eval- uation, in this part, present comparisons with typical SFDA methods using ViT back- bones (cited from DPC (Zhan et al., 2024)), employing ViT-B/16. Specifically, the compar- ison methods include SHOT-ViT (Liang al., 2020), DIPE-ViT VIT (Wang et al., 2022a), DSIT- ViT (Sanyal et al., 2023), AaD-ViT (Yang et al., 2022) and DPC. The results in Tab. 10 show that ProDe-Ve V16 consistently outperforms DPC in most cases. An exception is that ProDe-V16 is only 1.1% behind on Office-31, which may be attributed to potential overfitting on this relatively small dataset. Notably, DPC, which utilizes a VIT. Generally, using a ViT for such a small training dataset unnecessary due to the tendency for overfitting.
Table 10: Comparison with SFDA methods with VIT backbone on closed-set SFDA setting (%).
Method
Office-310ffice VisDA DomainNet-
SHOT-ViT ICML20
91.4
DIPE-NI VIT
CVPR22
90.5
78.2
ICCV23
93.0
AaD-VIT
NeurIPS22
72.7
DPC
JICA124
93.3
85.4
ProDe-VI6
92.2
86.9
91.7
88.1
with ResNet backbone for the target model, ProDe-V V16 still surpasses
4.4 QUANTITATIVE ANALYSIS OF PROXY DENOISING IN PROXY ALIGNMENT VIEW
In this part, we make a feature space shift analysis using the measure of MMD (Maximum Mean Discrepancy) distance to verify whether our ProDe method ensures the proxy alignment. In this experiment, we initially train domain-invariant Oracle model over Office-Home data with real labels, and use the logits to express the Comainn-nvariian space O. Sequentially, we perform transfer experiment Ar-CI. During this adaptation, there are K (epoch number) intermediate adapting target models. We feedforward the target data through each intermediate model and take the logits as a space. Thus, we obtain K intermediate target feature spaces { These intermediate spaces can lead to three different kinds of distances corresponding to these frozen spaces, termed d'g (to the source domain), d (to the Oracle space) and d', (to the proxy CLIP space). In practice, the CLIP image encoder's backbone set to ViT-B/32.

---

Published as a conference paper at ICLR 2025
73-
0.6
0.6
d5
MMD diet.
0.4
MMD dist
0.5
Aucuracy (50)
68
MMD dist.
0.15
0.3-
PRO
8.10
CLIP
0.2
0.2
0.05
58
(43.7)
12
15
0.0
15
30
60
10
12
14
0.00
15
45 60
Training epoch
iteration
spoch
Training deration
(a)
(b)
(c)
(d)
Figure 4: Analysis for proxy denoising on the AI-CI task in Office-Home. (a) The MMD-distance varying curves (epoch view) between the intermediate spaces to the source, oracle and proxy CLIP spaces, respectively, i.e, d's. do and di (b) The details of di (iteration view) during the first epoch. (c) The accuracy curves of typical signals during the adaptation. (d) The MMD-distance varying curves of ProDe (dh)), ProDe-KL (d ProDe-CLIP (dt) during the first epoch (iteration view).
Fig. 4 (a) displays the varying curves (epoch view) of d's, d' and df. As expected, d's increases, along with decreasing on d' Meanwhile, d', exhibits a V-shaped trend. For a clear view, we zoom into the first epoch and observe its variation details, as shown in Fig. 4 (b). In particular, there is smooth transition from decrease to increase the curve of d'. This phenomenon indicates that the in-training model indeed approaches the proxy space and then moves away from it to close the domain-invariin space our proxy error control gradually comes into play.
Correspondingly, we also provide the accuracy varying curves of two typical signals in Fig. 4 (c). including the target prediction (termed ProDe) and the denoised CLIP prediction (termed PRO). In this experiment, CLIP zero-shot result (termed CLIP) is the baseline. It is seen that PRO is better than ProDe the early phase (0~4 epoch) and surpassed by ProDe in the rest epochs. The results indicate that the guidance of reliable ViL predictions can boost the adaptation performance. Meanwhile, the PRO and ProDe curves closely resemble each other. It understandable that the current prediction of the in-training target model, (a:)), is utilized to adjust the raw Vil. prediction (see Eq. (5)).
To better understand the impact of proxy denoising, we also conduct comparison using two variations of ProDe. In Pro the loss Apt is changed to conventional KL-Divergence, whilst ProDe-CLIP, the training based on the raw ViL prediction without proxy denoising. Employing the same MMD-distance quantification method mentioned above, we can plot two distance curves to the proxy space, termed d'. d In Fig. 4 (d), it is evident that ProDe moves away from the proxy space more quickly than the other two comparisons. This result suggests that ProDe is more match desired adapting direction.
responsive proxy errors, resulting in agile error correction Additionally, the three curves at the early iterations are similar, indicating the impact of denoising eyz is negligible during this stage. This observation provides empirical evidence supporting Casel in our assumption.
5 CONCLUSION
The success of multimodal foundation models has sparked interest in transferring general multimodal knowledge to assist with comain-specifie tasks, particularly in the field of transfer learning. However, for label-free scene scenarios such as SFDA discussed in this paper, the issue of filtering out noise from multimodal foundation models has been largely overlooked. To address this fundamental issue, this paper introduces a new ProDe approach. We first introduce a new approach called proxy denoising, which corrects the raw ViL predictions and provides reliable ViL guidance. This approach is based on novel proxy confidence theory that we developed by modeling the impact of the proxy error between the proxy ViL space and the latent coomain-invarian space, using the adaptation dynamics in the proxy alignment. Additionally, we propose mutual distilling method to make use of the reliable proxy. Extensive experiment results indicate that our ProDe can achieve state-of-the. results with significant improvements on four challenging datasets, confirming its effectiveness.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (62476169, 62206168, 62276048) and the Postdoctoral Fellowship Program of CPSF (GZC20233323),
REFERENCES
Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy-Chowdhury. Unsupervised multi-source domain adaptation without access source data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10103-10111 2021.
Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16430,1644,, 2022.
Shuanghao Bai, Min Zhang, Wanqi Zhou, Siteng Huang. Zhirong Luan, Donglin Wang, and Badong Chen. Prompt-based distribution aligmment for unsupervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 729-737, 2024.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In European Conference on Computer Vision, pp. 440-457, Springer, 2022.
Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 295-305, 2022a.
Weijie Chen, Luojun Lin, Shicai Yang, Di Xie, Shiliang Pu, and Yueting Zhuang. Supe-supe noisy label learning for source-free unsupervised domain adaptation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 10185-10192. IEEE, 2022b.
M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, et al. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Yuan Shih Chow and Henry Teicher. Central Limit Theorems, pp. 295-335, Springer US, New URL https:
York, NY, 1988. ISBN 978-1-4684-0504-0, doi: //doi. 10. 1007 000007/778--4444-0554-00000
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7212-7222, 2022.
Yuntao Du, Haiyang Yang, Mingcai Chen, Hongtao Luo, Juan Jiang, Yi Xin, and Chongjun Wang. Generation, augmentation, and alignment: pseudo-souree source domain based method for source-free domain adaptation. Machine Learning, pp. 1-21, 2023.
Zhekai Du, Xinyao Li, Fengling Li, Ke Lu, Lei Zhu, and Jingjing Li. Domain-agnostic mutual prompting for unsupervised domain adaptation. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2024.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pp. 1180-1189 189,, 2015.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv:2202.06687 2022.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 5736-5745, 2017.
11

---

Published as a conference paper at ICLR 2025
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data, Advances in Neural Information Processing Systems, 34:3635-3649 2021.
Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 2427-2440, 2021.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9865-9874, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 4893-4902, 2019.
Vikash Kumar, Rohit Lal, Himanshu Patil, and Anirban Chakraborty. Conmix for source-free single and multi-target domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4178-4188, 4188, 2023.
Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri. Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. In International Conference on Machine Learn- ing. pp. 11710-11728. PMLR, 2022.
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 115-625,22 2021.
Zhengfeng Lai. Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pscudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Qicheng Lao, Xiang Jiang, and Mohammad Havaei. Hypothesis disparity regularized mutual information maximization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8243-8251, 2021.
Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free unsupervised domain adaptation. In International Conference on Machine Learning, pp. 12365- 12377. PMLR, 2022.
Jingjing Li, Zhekai Du, Lei Zhu, Zhengming Ding, Ke Lu, and Heng Tao Shen. Divergence- unsupervised domain adaptation by adversarial attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11 11:11)88100 196-8211, 2021.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jeng Hwang, al. Grounded language-image pre-training. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10965-10975. 2022.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without ource data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9638-9647, 2020a.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641-9650, 2020b.
12

---

Published as a conference paper at ICLR 2025
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocablee vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the International Conference on Machine Learning, pp. 6028-6039, 2020.
Mattia Litrico, Alessio Del Buc, and Pietro Morerio. Guiding pseudo-labels with uncertainty
estimation for source-free unsupervised domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 7640-7650, 2023.
Jie Liu, Jinzong Cui, Mao Ye, Xiatian Zhu, and Song Tang. Shooting condition insensitive unmanned aerial vehicle object detection. Expert Systems with Applications, 246: 2024.
R. Müller, S. Kornblith, and G. E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
S. Niu, J. Wu, Y. Zhang. Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, pp. 16888-16905, PMLR, 2022.
S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023.
Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songeen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with student base for video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18983-189922 2023.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffmaan, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv: 170::0:200000000 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-141, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp. 8748-8708 8763. PMLR, 2021.
Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Amo Solin. Uncertainty- source-free domain adaptation. In Proceedings of the European Conference on Computer Vision, pp. 537-555. Springer, 2022.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the European Conference Computer Vision, pp. 213-226, Springer, 2010.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervise domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8050-8058, 2019.
S. Sanyal, A. R. Asokan, S. Bhambri, A. Kulkarni, J. N. Kundu, and R. V. Babu. Domain ssecifispeeiifccty inducing transformers for source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18928-18937, 2023.
Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee. AD-CLIP: Adapting domains in prompt space using CLIP. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pp. 4355 4364, 2023.
13

---

Published as a conference paper at ICLR 2025
S Tang, Yuji Shi, Zhiyuan Ma, Jian Li, Jianzhi Lyu. Qingdu Li, and Jianwei Zhang. Model adaptation through hypothesis transfer with gradual knowledge distillation. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5679-5685, IEEE, 2021.
Song Tang, Yan Yang, Zhiyuan Ma, Norman Hendrich, Fanyu Zeng, Shuzhi Sam Ge, Changshui Zhang, and Jianwei Zhang. Nearest neighborhood based deep clustering for source data-absent unsupervised domain adaptation. arXiv:2107. 120.222777..2..... 22585, 2021.
Song Tang, Yan Zou, Zihao Song, Jianzhi Lyu, Lijuan Chen, Mao Ye, Shouming Zhong, and Jianwei Zhang. Semantic consistency learning manifold for source data-free unsupervised domain adaptation. Neural Networks, 152:467-478, 2022.
Song Tang, An Chang, Fabian Zhang, Xiatian Zhu, Mao Ye, and Changshui Zhang. Source-free domain adaptation via target prediction distribution searching. International Journal of Computer Vision, 132(3)6654 672, 2024a.
Song Tang, Yuji Shi, Zihao Song, Mao Ye, Changshui Zhang, and Jianwei Zhang, Progressive source-aware transformer for generalized source-free free domain adaptation. IEEE Transactions on Multimedia, 26:4138 4152, 2024b. doi: 10.1 1..100997M.222233321.4444
Song Tang, Wenxin Su, Mao Ye, and Xiatian Zhu. Source-free domain adaptation with frozen multimodal foundation model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024c.
Jiayi Tian, Jing Zhang, Wen Li, and Dong Xu. VDM-DA: virtual domain modeling for source data-free domain adaptation. IEEE Transactions Circuits and Systems for Video Technology, 32 (6):3749-3760, 2022.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5385 5394,, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International conference on learning representations, 2021a.
Fan Wang, Zhongyi Han, Yongshun Gong, and Yilong Yin. Exploring domain-invariant parameters for source free domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7151-7160, 2022a.
Jue Wang, Haofan Wang, Jincan Deng, Weijia Wu, and Debing Zhang. Efficientclip: Efficient cross-modal pre-training by ensemble confident learning and language modeling. arXiv preprint arXiv:2109.04699 2021b
Q Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 11, 2022b.
Yan Wang, Jian Cheng, Yixin Chen, Shuai Shao, Lanyun Zhu, Zhenzhou Wu, Tao Liu, and Haogang Zhu. Fvp: Fourier visual prompting for source-fres free unsupervised domain adaptation of medical image segmentation. IEEE Transactions on Medical Imaging, 2023.
Zhaoqing Wang, Yu Lu, Qiang Li, Xungiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. CRIS: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition, pp. 11686-11695, 2022c.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9010-90199 2021.
Siying Xiao, Mao Ye, Qichen He, Shuaifeng Li, Song Tang, and Xiatian Zhu. Adversarial experts model for block-box domain adaptation. In ACM Multimedia, 2024.
14

---

Published as a conference paper at ICLR 2025
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1426-1435 2019.
Shiqi Yang. Joost de Weijer, Luis Herranz, Shangling Jui, al. Exploiting the intrinsic neighbor-
hood structure for source-free domain adaptation. In Advances in Neural Information Processing Systems, volume 34. pp. 11393-29400, 202
Shiqi Yang. Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of IEEE International Conference on Computer Vision, pp.
2021b.
Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In Advances in Neural Information Processing Systems, 2022.
Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling, A Jan McLeod, and Boyu Wang. When source-free domain adaptation meets learning with noisy labels. International Conference on Learning Representations, 2023.
M. Zhan, Z. Wu, R. Hu, P. Hu, H. T. Shen, and X. Zhu. Towards dynamic-prompiing prompting collaboration for source-free domain adaptation. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 182 188, 2024.
Yixin Zhang, Zilei Wang, and Weinan He. Class relationship embedded learning for source-free unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7619-7629, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
Lihua Zhou, Nianxin Li, Mao Ye, Xiatian Zhu, and Song Tang. Source-free domain adaptation with class prototype discovery. Pattern recognition, 145:109974, 2024.
15

---

Published as a conference paper at ICLR 2025
A PROOF OF THEOREM
Restatement of Theorem 1 Given a proxy alignment formulated in Sec. The source domain (Ds), the invariant space (Dz), the proxy space Dy) and the in-training model (Di-) satisfy the probability distributions P(S), P(I), P(V) and P(T"), respectively, where S, I, V and T are corresponding random variables. The factor describing the credibility of P(V) has relation below.
P(T))
P (GGPV) = True,t) P(S)
Proof 1 We use the spatial distance relation to represent the variation in confidence of ViL prediction, which is causally linked to the variation distance to DI, as demonstrated in Fig. 2 (a). At any given time the correction factor can be expressed as
P P(PPV)= (CGPO)=TTO t) x Distance (D4, DI)l d
Distance (Ds, Dz)l
dss
where di and ds refers the distance from D'= and Ds Dr, respectively. Easily finding, Eq. (7) satisfies the reliability feature of gradually decreasing from to 0 as D4 evolves from Ds to Dr-
To account for the fact that spaces defined by probability distributions, instantiate the space distance using the widely used measurement of KL-divergence. This gives us:
d
KL (P(T)))))( ST+ P(T)) log (L(P(S)(P((E)
PIT dT
P(
ds
[g P(S) log P(Z)
P(
ds
ST. P(T*) log P(T")aT' f P(T") P(T)d7
(8)
-Is P(S) log P(S)dS fs P(S) d(I)dS
H(T*) log P(Z)
H(S) log P(I)
where H)) stands for the information entropy. Since Dr domain-invarian space, P(T) always outputs for the category of interesting, such that log P(T) = 0. Eq. (8) can be further converted to
H(T*) log P(I)
H(T*)
P(T*)
(9)
H(S) + log P(Z)
H(S)
P(S)
B PSEUDO TRAINING CODE OF PRODE
Based on the proposed objective presented in Eq. (6), we achieve the model training iteration-wise. The training process are summarized as Alg. 1.
Algorithm Training of ProDe
Input: Source model 0.. ViL model 0... target dataset Xt, C prompts with context #iteration M.
Procedure:
1: Initialisation: Set target model 0., prompt context ="a photo of a".
for m = 1:M do
3: Sample batch x from Nr.
4: Forward updated prompts and 2 through 6..
5: Forward 24 through 0z-
6: Conduct proxy demoising for the ViI. predictions of X (Fq. (5)).
7: Update model 9 and prompt context v by optimizing objective LrroDa (Eq. (6)).
8: end for
9: return Adapted target model O..
C EVALUATION DATASETS
In this paper, the ProDe method is evaluated on four widely used benchmarks for domain adaptation problems follows.
16

---

Published as a conference paper at ICLR 2025
Office-31 (Saenko et al., 2010) is small-scaled dataset including three domains, i..., Amazon (A), Webcam (W), and DsIr (D), all which are taken of weal-world objects in various office environments. The dataset has 4,652 images of 31 categories in total.
Office-Home (Venkateswara et al., 2017) is a medium-scale dataset that is mainly used for domain adaptation, all of which contains 15k images belonging to 65 categories from working or family environments. The dataset has four distinct domains, i.., Artistic images (Ar), Clip Art (CI), Product images (Pr), and Real-worr images (Rw).
VisDA (Peng et al., 2017) is large-scale dataset with 12 types of synthetic to real transfer recognition tasks. The source domain contains 152k synthetic images (Sy), whilst the target domain has 55k real object images (Re) from the famous Microsoft COCO dataset.
DomainNet- (Saito et al., 2019) is another challenging large-scale dataset. It has been created by removing severe noisy labels from the original DomainNet dataset (Peng et al., 2019) containing 600k images of 345 classes from domains of varying image styles. The dataset is further divided into four domains: Clipart (C), Painting (P), Real (R), and Sketch (S), and contains 145k images from 126 classes.
D IMPLEMENTATION DETAILS
Souce model pre-training. For all transfer tasks on the four evaluation datasets, we train the source model 0g the source domain in a supervised manner using the following objective of the classic cross-entropy loss with smooth label, totally the same as other methods (Liang et al., 2020; Yang al., 2021a; Tang et al., 2022).
La (X.. ExjER' Σ ic-
log log pi...
where pf, is the c-th element of P (0,()))) that is the category probability vector of input instance at after 0g conversion with ending softmax operation Φ; [c = yf = (1 -1 a) [c - vi + a/C' is the smooth label (Müller al., 2019), in which 1 [c y is one-hot encoding of hard label y/ and o = 0.1. The source dataset is divided into the training set and testing set in a 0.9:0. ratio.
Network setting. The ProDe framework involves two networks, namely the target model and the ViL model. In practice, the target model comprises a deep architecture- based feature extractor and a classifier that consists of fully connected layer and weight normalization layer. As seen in previous work (Xu et al., 2019; Liang et al., 2020; Roy et al., 2022), the deep architecture is transferred from the deep models pre-trained on ImageNet. Specifically, ResNet-50 is used on Office-31 and Office-Home, whilst ResNet- 101 is employed on VisDA and Nomain-Net, As for the ViL model, we choose CLIP instantiate it where the text encoder adopts Transformer structure and the image encoder takes ResNet or ViT-B/32 according to the specific implementations, which are marked by suffix of " RR" or "-V".
Sxpeer-parameter setting. The ProDe model involves four parameters: The correction strength factor Eq. (5) and two trade-off parameters a, 3 and y in objective LProDe (Eq. (6)). On all four datasets, we set (w, a, (3) (1, 0.4). Parameter is sensitive to the dataset scale, also noted in the TPDS method (Tang et al., 2024a). In practice, the setting of y = 1.0/1.0/0.1 10..5 /0.5 is employed on Office-31, Office-Home, VisDA and DomainNet- respectively.
Training setting. We chose a batch size of 64 and utilized the SGD optimizer with a momentum of 0.9 and 15 training epochs on all datasets. The learnable prompt context is initiated by the template of 'a photo of [CLASS].', as suggested by (Radford et al., 2021), where the [CLASS] term is replaced with the name of the class being trained. All experiments conducted with Py Torch a single GPU of NVIDIA RTX. Each transfer task is repeated five times, and the final result is calculated as the average of the five attempts.
17

---

Published as a conference paper at ICLR 2025
Table 11: Full results (%) of closed-set SFDA on VisDA. SF means source-free.
Method
Venue
SI
plane
beyel
bus
car
horse
knile
meyyel
person
plant
skibrd
train
truck
Perclass
Source
60.7
21.7
50.8
68.5
71.8
5.4
86.4
20.2
67.1
43.3
83.3
10.6
49.2
SHOT
93.9
79.4
80.4
89.8
85.8
57.5
82.7
NRC
NIPS21
96.8
91.3
82.4
62.4
96.2
95.9
86.1
90.7
94.8
94.1
90.4
59.7
85.9
TROS21
95.3
81.7
58.1
93.9
94.0
80.0
80.0
.2
91.0
86.9
83.0
AaD
NIPS22
V
97.4
90.5
80.8
76.2
97.3
96.1
89.8
82.9
95.5
93.0
92.0
64.7
88.0
AdaCon
CVPR22
/
97.0
84.7
84.0
77.3
96.7
93.8
91.9
84.8
94.3
93.1
94.1
49.7
86.8
CoWA
ICML22
96.2
89.7
83.9
73.8
96.4
97.4
89.3
86.8
94.6
92.1
88.7
53.8
86,9
ELR
ICLR25
97.1
89.7
82.7
62.0
96.2
97.0
87.6
81.2
93.7
94,1
90.2
58.6
85.8
PLUE
CVPR23
89.0
96.6
CPD
PR24
V
96.7
88.5
79.6
69.0
95.9
96.3
87.3
83.3
94.4
92.9
87.0
58.7
85.5
TPDS
IJCV24
97.6
91.5
89.7
83.4
97.5
96.3
92.2
82.4
96.0
94.1
90.9
40.4
87.6
DAPL-R
TNNLS23
X
97.8
83.1
88.8
77.9
97.4
91.5
94.2
79.7
88.6
89.3
92.5
62.0
86.9
PADCLIP-R
97.1
93.0
91.8
ADCLIP-R
ICCVW23
X
98.1
83.6
91.2
76.6
98.1
93.4
96.0
81.4
86.4
91.5
92.1
64.2
87.7
PDA-R
AAAI24
X
97.2
89.4
76.0
97.4
87.5
95.8
79.6
87.2
89.0
93.3
62.1
86.4
DAMP-R
CVPR24
X
97.3
91.6
89.1
76.4
97.5
94.0
92.3
84.5
91.2
88.1
91.2
67.0
88.4
DIFO-R
CVPR24
97.6
88.7
83.7
80.8
95.9
95.3
91.9
85.0
89.4
93.2
93.2
69.0
88.6
DIFO-V
CVPR24
90.8
83.5
95.2
96.8
93.7
65.9
90.3
ProDe-R
96.6
90.3
83.9
80.2
96.1
96.9
90.3
86.4
90.8
94.0
91.3
67.0
88.7
ProDe-V
Table 12: Full results (%) partial-set SFDA and open-set SFDA on Office-
Partial-set Venue
Ar-ACI Ar-PPr Rr Cr- Cl- Pr Cl-RRw Pr-+Ar Pr-+Cl Pr-+Rw Rw-+AAr Ar Rw-CI Rw- Pr Avg.
Souree
45.2
70.4
81.0
56.2
60.8
66.2
60.9
76.2
77.3
62.8
SHOT
ICML20
64.8
85.2
92.7
76.3
77.6
88.8
79.7
64.3
89.5
80.6
66.4
85.8
79.3
ICL
NIPS21
65.6
85.2
92.7
77.3
76.2
87.2
78.2
66.0
89.1
81.5
68.4
87.3
79.6
CoWA
ICM1.22
69.6
93.2
92.3
78.9
81.3
92.1
79.8
71.7
90.0
83.8
72.2
93.7
83.2
AaD
NIP522
67.0
83.5
93.1
80.5
76.0
87.6
78.1
90.2
83.5
87.3
79.7
CRS
CVPR23
68.6
85.1
90.9
80.1
79.4
86.3
79.2
66.1
90.5
82.2
69.5
89.3
80.6
CVPR24
89.1
ProDe-V
70.2
89.7
90.4
84.1
90.7
91.4
85.5
69.9
92.9
87.8
68.5
89.7
84.2
Open-ser set
Ar-+CI
Ar- CI-+Ar CI->Pr CI-+Rw Pr-+Ar
Pr-+Rw Rw-+Ar Rw-
Avg.
Source
36.3
54.8
69.1
33.8
44.4
49.2
36.8
29.2
56.8
$1.4
35.1
62.3
46.6
SHOT
ICML20
64.5
80.4
84.7
63.1
75.4
81.2
65.3
59.3
83.3
69.6
64.6
82.3
72.8
HCL
NIPS21
64.0
78.6
82.4
64.5
73.1
80.1
64.8
59.8
75.3
78.1
69.3
81.5
72.6
CoWA
ICML222
63.3
79.2
85.4
67.6
83.6
66.9
56.9
81.1
68.5
$7.9
85.9
73.2
AaD
NIP522
63.7
77.3
80.4
66.0
72.6
77.6
69.1
62.5
79.8
71.8
62.3
78.6
71.8
CRS
CVPR23
65.2
76.6
80.2
75.3
77.8
70.4
78.3
73.2
DIFO-V
CVPR24
64.5
86.2
87.9
68.2
79.3
86.1
67.2
62.1
88.3
71.9
65.3
84.4
75.9
ProDe
Table 13: Results (%) of CLIP on the four evaluation datasets. The backbone of CLIP image-encoder in CLIP-R and CLIP-V V are the same ProDe-R R and ProDe respectively.
Office-31
Office-Home
VisDA Sy-tRo
DomainNet-126
Method
Venue
-W
Avg.
-+Ar
+CII
-rr
Rw
Avg
-R
Avg
CLIP-R
67.0
72.5
83.7
72.7
ProDe-R
79.4
96.5
93.9
89.9
80.6
65.0
90.1
88.6
81.1
88.7
80.3
79.2
91.0
75.4
81.5
CLIP-V ICML21
76.0
80.6
79.8
74.6
84.3
83,5
76.1
82.9
73.5
85.7
71.2
76.3
ProDe-V
82.8
98.3
96.7
92.6
82.7
72.6
92.0
90.7
84.5
91.0
85.3
83.2
92.4
79.1
85.0
E SUPPLEMENTAL EXPERIMENTS
E. SUPPLEMENTATION OF FULL EXPERIMENT RESULTS
Full results on VisDA. Tab. 1l is the supplement of average results the VisDA dataet (reported in Tab. 2), displaying the full classification results over the 12 categories. Specifically, the ProDe-R and ProDe-V V totally obtain best results 7/12 categories, leading to the advantage on average accuracy. On some cases, such as beycl, car and truck, ProDe has presents significant advantages over the previous methods.
18

---

Published as a conference paper at ICLR 2025
Table 14: Generalized SFDA results (%) on Office-Home. are the results of the adapted target domain on the source and target domains, respectively; H means the harmonic mean accuracy; WAD short for With Anti-forgg Design.
Mechod
Venue
WAD
Ar-+CI
Ar-PP
Ar- Rw
Cl+ T
Cl++Pt
C7 Cl-RR
T
H
T
H
T
H
Source
x
97.9
43.7
60.4
97.9
67.0
79.5
97.9
73.9
84.2
97.1
40.9
66.9
97.1
60.1
74.2
97.1
62.5
76.0
SHOT
ICML22
X
78.6
55.0
64.7
83.8
78.7
81.2
98.6
81.3
84.8
78.0
69.1
13.2
76.5
78.9
71.7
71.1
79.1
73.1
OKD
IROS21
56.5
66.9
87.0
78.3
82.4
82.2
56.6
80.3
80.9
50.4
30.6
81.4
78.7
NIPS21
57.2
69.3
92.9
81.3
87.7
81.7
54.6
88.8
AdaCon
CVPR22
X
75.2
47.2
37.9
91.0
75.1
82.5
93.9
15.5
83.7
19.4
08.7
08.8
88.2
73.3
50.0
83.4
73.2
78.0
CoWA
ICML22
X
89.0
57.3
69.7
93.0
79.3
85.5
94.6
81.0
87.3
86.5
69.3
77.0
86.3
77.9
81.9
83.4
79.6
$1.5
PLUE
CVPR23
29.1
63.9
86.6
TPLS
UCV24
39.3
67.4
83.6
82.1
35.0
15.4
70.6
77.3
79.4
73.3
76.2
GDA
JCCV21
68.3
56.7
60.9
72.0
75.5
73.8
74.5
78.5
76.8
77.2
66.5
71.5
79.7
74.0
76.7
78.5
78.4
78.4
PSAENIT
TMM24
81.6
73.1
77.1
57.0
88.1
87.6
88.1
89.2
88.7
82.7
82.1
82.6
82.7
89.8
85.7
83.5
88.9
86.1
DIFO-R
CVPR24
67.8
76.3
87.1
83.2
81.9
74.6
DIPLLY
CVPR24
70.6
72.2
75.0
90.6
M0.7
88.8
84.6
70.4
75.9
64.3
90.5
75.2
65.9
75.7
ProDe-K
79.4
64.0
70.9
K4.1
9000
87.0
87.7
88.3
89.0
79.5
81.1
80.3
76.2
90.1
82.5
73.7
88.6
20.4
PmoDe V
81.4
72.7
76.8
$4.3
92.2
88.1
88.1
90.5
89.2
76.5
82.5
79.4
77.8
91.5
84.1
74.0
90.7
$1.3
Mathod
Veme
WAD
Pr-+Ar T
Pr
Pr- Rw
T
Rw-
Rw-+CI
T
H
Rw
Avg.
T
H
Source
1992
51.7
68.0
99.2
40.9
57.99
99..2
72.6
83.8
98.1
64.2
77.6
98.1
46.5
62.9
98.1
79.1
17.00
98.1
59.2
73.1
SHOT
ICML220
88.2
68:2
76.9
N0.7
53.6
64.4
90.1
81.6
85.6
91:77
73.5
81.6
8A.8
59.4
09.8
92.2
83.5
57.6
94.2
71.8
77.5
GKD
IROS21
89.4
67.4
76.8
84.1
55.4
66.8
92.0
82.5
87.0
93.7
74.3
82.9
86.2
60.3
70.9
93.5
84.2
58.6
86.8
72.5
79.0
NIPS21
89.1
66.6
72.3
CVPRE2
93.4
60.2
73.2
59.8
94.3
76.6
65.6
77.0
48.3
86.1
68.0
CoWA
ICML22
94.6
68.1
79.2
93.2
56.4
70.3
95.0
82.5
88.3
96.3
72.9
83.0
93.7
61.3
74.1
95.5
83.7
59.3
91.8
72.4
$1.0
PLUE
CVPR25
98.7 62.2
76.3
98.5
48.3
64.8
98.9
78.5
87.6
98.1
68.6
80.7
95.1
51.8
67.1
97.8
81.5
85.9
96.3
66.9
79.0
JPDS
UCY24
87.7
83.8
ICCV21
878 65.1
74.8
86.3
66.1
90.3
81.6
85.7
77.2
78.3
60.2
83.4
83.1
80.0
70.2
74.4
PSAE-VT
TMM24
89.6 85.0
86.2
87.4
72.0
79.0
92.5
89.5
91.0
87.4
83.5
85.3
84.2
73.7
78.6
89.6
91.3
90.5
86.4
83.6
$5.0
DEO-R
CVPR24
85.6 78.3
81.8
76.6
63.4
69.4
86.0
88.1
87.0
89.4
80.0
84.4
80.7
63.3
70.9
87.2
87.7
87.4
78.3
79.4
78.8
DIPOYV
CVPR34
84.3
ProDe-R
89.5 79.8
92.1
8933
65.5
91.1
ProDe
88.9
82.5
85.5
85.0
72.4
78.2
92.0
90.8
91.4
92.7
83.0
87.6
87.5
72.6
79.3
93.1
92.2
92.6
85.1
84.5
54.8
Table 15: Full results (%) of the TTA setting on Office-Home.
Method
Venue
Ar-+Cl Ar-Pr Ar-Rw Cl++Ar CI-+Pr Cl-RRw Pr- Pr-+CI Pr-Rw Rw-+A Aw Rw +CI Rw-+P Avg.
Teat
ICLR20
47.6
63.2
72.3
$7.1
63.7
65.9
55.9
46.6
72.7
67.7
51.8
77.1
61.7
T3A
NeurIPS21
49.7
73.2
55.5
67.7
68.5
46.1
67.0
49.6
78.0
CaTTA
CVPR22
44.5
62.5
72.3
55.4
63.0
65.3
54.9
46.0
76.7
66.0
49.5
76.7
60.5
EATA
JCML22
46.4
62.5
72.2
55.3
65.8
65.8
53.8
43.4
76.4
66.5
50.5
76.4
60.7
SAR
ICLR23
45.3
61.9
71.9
55.4
66.4
65.7
53.7
42.7
72.5
66.4
49.3
76.2
60.3
ProDe-V
62.3
83.2
83.5
74.9
83.8
83.3
73.8
63.7
84.5
76.6
63.3
85.1
76.5
Table 16: Reliance analysis results (%) Office-31 in the Closed-set SFDA setting.
Method
Venue
A-+D
A-
D-*A
D--W
W-sA
W-+D
Avg.
DIPO w/ CLIP
CVPR24
97.2
95.5
83.0
97.2
83.2
98.8
92.5
ProDe CLIP
96.8
96.4
83.1
97.0
82.5
99.8
92.6
DIFO OpenCLIP
CVPR24
96.8
98.1
98.7
82.7
100.
ProDe w/ OpenC1IIP
96.1
96.7
86.5
97.4
86.8
98.2
93.6
Full results of partial-set and open-set SFDA. Tab. 12 is the supplementation of these average accuracy in Tab. 5, reporting the full classification accuracy over 12 transfer tasks in Office-Home. In the partial-set setting (the top the table), ProDe-V V beats other methods on 4/12 tasks, whilst DIFO- V. CoWA, and AaD dominate the rest of the tasks. As taking the open-set setting (the bottom in the table), ProDe-V gets the top results on 9/12 tasks. Moreover, besides the Ar-Rw, CI-Rw, Rw-PP task, the rest of the best eight tasks have 7.5% increase at least, compared with the best-second methods. So, the ProDe gains substantial improvement in average performance.
Full results of the comparison to CLIP's zero-shot. As the supplement of average results in the comparison to CLIP (reported in Tab. 4), Tab. 13 presents the full quantitative results categorized by the target domain name. For instance, for domain A in Office-31, we averaged the adapting accuracy of other domains to A, such as D-A, W A, notated by AAA As reported in Tab. 13, both ProDe-R and ProDe-V V obtain the best results across all groups, compared to the respective CLIP version.
19

---

Published as a conference paper at ICLR 2025
Table 17: Reliance analysis results (%) on Office-Home in the Closed-set SFDA setting.
Method
Venue
Ar-PPP
(0--AAr(--PP----- Pt-+Ar Pr---t-rP-P-CI Pt->Rw Rw RWW-CIR Rwr-Rr--Prrr Avg.
+1+++11
DIFO w/ CLIP
CVPR24
70.6
90.6
88.8
82.5
90.6
88.8
80.9
70.1
88.9
83.4
70.5
91.2
83.1
ProDe w/ CLIP
72.7
92.3
90.5
82.5
91.5
90.7
82.5
72.5
90.8
83.0
72.6
92.2
84.5
DIPO w/ OpenCLIP CVPR24
80.2
94.2
91.7
85.4
93.7
91.6
82.7
79.2
91.7
85.3
80.4
94.8
87.6
ProDe OpenCLIP
93.5
Table 18: Reliance analysis results (%) on VisDA in the Closed-set SFDA setting.
Method
Venue
plane
beyel
horse
knife
meyel
person
plant
skibrd
truck
Perclass
DIFO CLIP
CVPR24
97.5
89.0
83.5
97.8
97.3
93.2
83.5
93.7
ProDe w/ CLIP
98.3
92.4
86.6
80.5
98.1
98.0
92.3
84.3
94.7
97.0
94,1
75.6
91.0
OpenCLIP
CVPR24
ProDe w/ OpenCLIP
97.9
90.5
86.7
88.5
97.8
96.4
94.2
88.4
95.7
96.2
93.3
71.5
91.4
Table 19: Reliance analysis results (%) on DemainNet-126 in the Closed-set SFDA setting.
Method
Venue
C->P
C-RR
C-4S
P-4C
P-+R
P-+S
R-+C
R-+P
R-+S
S S-C
S->P
S-+R
Avg.
DIFO w/ CLIP
CVPR24
76.6
87.2
74.9
80.0
87.4
75.6
80.8
77.3
75.5
80.5
76.7
87.3
80.0
DIFO w/ OpenCLIP
CVPR24
91.2
91.5
79.4
85.2
91.2
79.7
85.7
82.7
80.5
85.9
81.3
91.4
84.6
OpenCLIP
88.6
Table 20: Results (%) of OpenCLIP on the four evaluation datasets.
Office-31
Office-Home -Pt
VisDA Sy->Re
DomainNet-126
Mothod
Venue
++A
+D +W AAy
+AA
+CC
RRw
AAyg
+C
+PP
-R
-5S
-+Av.
OpenCLIP
CVPR23
85.7
91.2 91.8
89.6
83.8
76.1
93.5
92.3
86.4
86.7
86.4
82.0
92.3
80.8
85.4
Op
Full results of generalized SFDA. As supplement to the average results of the generalized SFDA results (reported in Tab. 6). Tab. 14 presents the full results on 12 transfer tasks, including S-, and H-accuracy. In terms of H-accuracy, ProDe-V achieves the best results on half tasks. These results are not only due to significant improvements in the target domain (see T-accuracy) but also derive from balanced drop in the source domain (see S-accuracy).
Full results of TTA. As a supplement to the average results of the TTA results (reported in Tab. 7), Tab. 15 presents the full results the Office-Home dataset. On all 12 transfer tasks, ProDe-V achieves substantial increase, leading to 12.7% gains on top of the second-best method T3A.
E.2 EXPANDED MODEL ANALYSIS
Reliance analysis ViL models. As illustrated the right of Fig. our proxy denoising is executed at the logit level, which means that the proposed method does not depend on specific ViL model, such as CLIP, since it does utilize the internal structure of these models. To validate this claim, we conduct an extensive test with OpenCLIP (Cherti et al., 2023). Meanwhile, we selected DIFO, the previous best ViL-based method, for comparison. Tab. 16~19 present comparison results across all four datasets. Regardless of whether we use CL.IP or OpenCI.IP as the Vil. model, ProDe beats DIFO in average accuracy. Furthermore, the relative gains are consistent. comparison to DIFO, ProDe improves approximately by 0.3%, 2.0%. 1.0% and 4.5% Office-31, Office-Home VisDA and DomainNet-Il 126, respectively. This trend suggests that our method is generic with the ViL model, and can readily benefit from the advancement in ViL models.
In addition, Tab, 20 displays a comparison of the zero-shot results from OpenCLIP. In all tasks (which detailed in the "Full results of the comparison to CLIP's zero-shot" section of Sec E.1), ProDe w/ OpenCLIP surpasses OpenCLIP. This suggests that the task-specific target model effectively incorporates generic knowledge in Vil. models.
20

---

Published as a conference paper at ICLR 2025
Table 21: Ablation results (%) of prompt Iearn- ing on Office-31, Office-Home and VisDA.
Table 22: Comparison of training resource de- mands (per iter.) on Ar->CI in Office-Homee
Method
Office-11
Office-Home
VisDA
Avg.
Item Method
SHOT
AaD
ProDe
ProDe-V V w/o prompt
91.7
81.9
88.0
87.2
GPU memory consumption.) (G)
7.868
9.622
9.851
V
92.6
84.5
91.0
89.3
Training times]
0.407
0.547
0.491
Effect of prompt learning. In ProDe, prompt learning contributes knowledge synchronization. To isolate its effectiveness, we propose a variation method ProDe-V w/o prompt that removes prompt learning. As shown in Tab. 21, the absence of prompt learning results in 2.1% decrease in average accuracy. These results indicate that this prompt learning might reduce the proxy error by tuning space Dy close to the nomain-invarian space D,, meeting our expectations.
Training resource demands. To evaluate the training resource demands, we select two typical methods without using ViL. model, SHOT and AaD, as comparisons. We conducted the test using the transfer task Ar - CI from Office-Home, under the same testing conditions, including mini-batch size. The results, as shown Tab. 22, indicate that despite using a large ViL model, our approach does not incur significant additional training costs and requires a similar amount of computational resources. This is because: (1) The ViL model is frozen in our method, making its use efficient, and (2) Our ProDe approach does not require feature bank with periodic updates for deep clustering like SHOT, nor does it involve identifying neighborhoods as in AaD.
Sensitivity of prompt initialization. In the proposed approach, we employ the initialization template of "a photo of a <els>" for each class because it is the most used template to initiate the learnable prompt. The effect of prompt learning with this initiation is
Table 23: Ablation study results (%) for typical prompt tem- plates Office-31, Office-Home and VisDA.
Initialization template
Office-31
Office-Home
VisDA
'X (+X=4)
91.9
84.2
90.5
'X (XX-1))
91.9
82.9
90.5
evaluated as reported in Tab. 21.
"There is a (CLS),
93.0
83.1
90.6
"This is photo of (CLS).
92.4
83.2
90.7
For further analysis, we conduct an ab- lation study on nine typical initializa- tion templates. As shown in Tab. 23, there are no evident performance vari- ations crossing the Office-31, Office- Home, and VisDA datasets, indicating that our method is insensitive to the selection of templates. Furthermore, the semantic templates outperform those that use 'X' (see rows 1 and 2). These results align with our expectations.
'This maybe a photo of (CLS)'
92.6
91.0
'This almost a photo of [CLSS]
92.4
84.2
90.8
"This definitely photo of (CLSS):
92.5
84.2
90.7
picture [CLSS]
92.2
90.7
photo of (CLS]]
92.6
84,5
91.0
Parameter sensitivity. In this part, we discuss the parameter sensitivity of parameters a, 3 in LrroDe (see Eq. (6) and correction strength parameter in proxy denoising (see Eq. (5)). All experiments are conducted based on the transfer tasks Ar-CC in the office-Homa dataset. The varying range set to 0.5 a S 1.4, 0.1
& 1.0 and 0.5 < w 1.4 in 0.1 step size. Fig. 5 (a) depicts the results as a-3 vary. When the two parameters changes, there are no evident drops in the accuracy variation curves, except for two boundary situations: (1) a 0.5 and (2) 3 1.0. The results indicate that ProDe is insensitive to parameters and 3. Mcanwhile, when we select parameters, o's value should be lager than 3. Besides, in Fig. (b) and (c), we display the results when a x w and /3 x vary, respectively. Thus, we present the relation between the correction strength and regularization elements in LProDe. From the two sub-figures, it is seen that the performance has a significant drop as we adopt 1.4. This show that the correction strength in the proxy denoising block should not be too strong.
F LIMITATION AND FUTURE WORK
ProDe has shown impressive performance in multi-SFDA settings, highlighting its efficacy. However, it is important note that it is specifically designed for a white-box scenario, which may not be applicable in certain real-world contexts. For the kind of black-box application, such as models in the cloud, our proxy denoising may not work well since all details of the model, including the
21

---

Published as a conference paper at ICLR 2025
& Kocrncyy
(%) Kaemmoo
72.0
(%)
0.
1.0
1.4.0
0.5
B
70
ovl a.
1.0
0.5
11.5
0.5
1.0
0.5
(a)
a-3
(h)
am
(c) 8-w
Figure 5: Sensitivity analysis of
a, B and w.
required logits features, are transparent to In the future, finding ways to extend our method to this challenging scenario will be an interesting direction.
22


Reviews:
[{"review_id": "20zyddpWVh", "reviewer": "Reviewer_vhPY", "summary": "The paper addresses Source-Free Domain Adaptation (SFDA) in terms of utilizing Vision-Language Models (VLMs) for supervision. Specifically, the authors argue that prior works that utilize VLMs for SFDA treat their predictions as the ground truth without considering the potential noise in their predictions. To alleviate this issue, the authors propose Proxy Denoising (ProDe)—an SFDA framework that corrects the supervisory VLM’s predictions before target adaptation. Extensive experiments on various Domain Adaptation benchmarks demonstrate the effectiveness of the approach compared to prior works in multiple domain adaptation settings. Moreover, analysis experiments substantiate the intuition behind the proposed method.", "questions": "- How does the proposed method ProDe perform in a multi-source or multi-target domain adaptation setting? Can the authors present these results on OfficeHome or DomainNet?\n- Why have the authors used DomainNet-126 rather than the full DomainNet dataset? Given that DomainNet is the most challenging domain adaptation benchmark among the chosen datasets, it is an important result.\n- Following the discussion on fair comparisons in the previous section, can the authors present results in a fair setting? Eg. the authors can present results of prior SFDA works by using the CLIP vision encoder as the initialization rather than the ImageNet pre-trained backbones. Another possibility could be a baseline that provides VLM supervision to prior SFDA works.\n- Additionally, if the above is not possible, could the authors present results with ViTs rather than ResNet-50 / ResNet-101 for a more fair comparison? Given that the results with VLMs use ViT-B, the prior works should use a backbone with a similar capacity.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Presentation-** The paper is written well overall and conveys the central ideas of the work quite effectively. The paper is easy to follow and understand. Additionally, the authors have presented experiments on a wide range of benchmarks and settings.\n\n- **Novelty-** The authors claim that the paper is the first to analyze the inaccurate predictions of the teacher VLM in the context of SFDA and propose a method to alleviate the same.\n\n- **Results-** The authors present extensive experiments across several domain adaptation benchmarks and comparisons with several prior works that do and do not use VLMs for training (although whether some of the comparisons are fair is a question, see Weaknesses for details).", "weaknesses": "### (a) Concerns with Proxy Confidence Theory\n- Theorem 1 provides a relation between the confidence of the VLM’s predictions and the confidence of the source model and the current training model. This is based on the approximation of the VLM’s predictions to a Gaussian distribution and further expressing this in terms of the confidence of the VLM’s predictions.\n- However, the intuition behind how a Gaussian distribution is considered for the VLM’s predictions is not entirely clear. Moreover, the conversion in Eq. 2 also seems unclear, in terms of how the conversion is possible. Essentially, it would be better if the authors could explain L185-188 in more detail.\n\n### (b) Fairness of comparisons\n- The authors present comparisons of their proposed method ProDe with prior SFDA works and with works utilizing VLMs. Based on the implementation details provided in the supplementary, it appears that the prior SFDA works that do not utilize VLMs make use of ResNet-50 or ResNet-101 depending on the difficulty of the dataset.\n- Can these comparisons of ProDe with prior SFDA works be considered fair? ProDe uses supervision from a VLM that has been pre-trained on WiT-400M while the SFDA works consider an ImageNet pre-trained ResNet-50 or ResNet-101. There is a massive difference in the models being used for adaptation. Although the student model is a vision-only backbone in ProDe, it is supervised by a VLM during target adaptation.\n- The authors need to discuss these differences in the experiment settings to provide a more complete picture of the results. Additionally, the authors should present the comparisons in a fair setting, i.e., similar supervisory signals or backbones should be used in both the proposed method and the prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQJ9Qn82EX", "reviewer": "Reviewer_hEmB", "summary": "The previous methods using ViL for pseudo-supervision can generate noise, which introduces negative effects that have been overlooked. This paper proposes a ProDe method that first introduces a proxy confidence theory, which vividly analyzes and explains the sources of noise in ViL predictions, and specifically designs a denoising mechanism to correct ViL's predictions. Additionally, it introduces a mutual information extraction method to achieve knowledge synchronization between the ViL model and the target model.", "questions": "I would like to know if using ViL for pseudo-supervision is suboptimal when the distance between the source domain space D_S (or training dataset D_Tt) and D_I is closer than the distance between D_V and D_I. Additionally, the formulas (5) and (6) in the paper contain many hyperparameters; is tuning these hyperparameters a challenge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is based on proxy confidence theory and designs a reliable denoising algorithm to reduce the prediction noise of ViL, addressing an important issue that has been neglected in the use of ViL for pseudo-supervision. This may facilitate subsequent related work.", "weaknesses": "Although the ViL model is obtained based on a large dataset, for specific source and target domains, the ViL model can approximate the domain-invariant space. The validity of this assumption requires further theoretical support.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iObdbKVTBq", "reviewer": "Reviewer_SdQs", "summary": "The authors tackle Source-Free Domain Adaptation (SFDA), where a pre-trained model adapts to an unlabeled target domain without access to source data. While Vision-Language (ViL) models show potential for SFDA, they often generate noisy predictions, an issue the authors investigate in this context. To address it, they propose Proxy Denoising (ProDe), a novel method that leverages proxy confidence theory to correct the ViL model’s predictions and introduces mutual knowledge distillation to make better use of these refined predictions. Extensive experiments on standard benchmarks demonstrate that ProDe outperforms prior methods across conventional closed-set, as well as partial-set, open-set, and generalized SFDA settings. The authors intend to release their code.", "questions": "It would be great if the authors could clarify the doubts above especially those related to the theory and domain invariance space.\n\n# UPDATE (After Discussion Period):\nThe discussion addresses all my concerns, and I appreciate the detailed responses and clarifications provided. Consequently, **I am increasing my ratings**. The responses demonstrate a strong understanding of the core issues, effectively addressing ambiguity, unsupported assumptions, and the generalization of claims. The inclusion of additional experiments, such as evaluations with OpenCLIP and broader SFDA settings, further solidifies the robustness and adaptability of the proposed method. The effort to provide comprehensive results, insightful explanations, and necessary corrections is commendable. Thank you for your diligence and thoroughness in addressing these points.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper effectively addresses the important problem of source-free domain adaptation (SFDA), where models must adapt to new target domains without access to labeled source data—an increasingly relevant setup in practical scenarios where source data may be proprietary or sensitive. Demonstrating state-of-the-art performance on standard SFDA benchmarks, the proposed method showcases its robustness and potential impact in the field. The authors employ mutual knowledge distillation to synchronize knowledge, reducing noise and enabling reliable knowledge transfer, and they incorporate category balance regularization to prevent “category collapse,” ensuring balanced treatment of each class. Through extensive analysis, including feature distribution visualizations and thorough ablation studies, the paper provides deep insights into model behavior and validates the effectiveness of each component. Furthermore, the paper is well-documented, with a clear presentation of the experimental setup, benchmark datasets, and evaluation metrics, enhancing reproducibility and accessibility.", "weaknesses": "### Ambiguity and Misleading Terminology in Domain Invariance Claims:\nThe authors claim that their method moves toward a “domain invariant space”  $D_v$ starting from  $D_t$ , but this terminology is misleading and theoretically problematic. If a domain-invariant space were achievable, there would be no need for adaptation across other domains, as all target domains would align seamlessly with this invariant space. However, the results suggest that domain shifts still impact the model and hence there is a need for adaptation for every domain, which contradicts the premise of invariance. For example, if the model were genuinely invariant after training on a source domain (e.g., Ar), it should perform equally well across all other domains (e.g., Cl, Pr, Rw) without further adaptation. This contradiction suggests that  $D_v$  is not genuinely invariant, but rather biased towards the target domain.\n\nIt would be beneficial if the authors could redefine the term “domain invariant space.” They might consider an alternative term, such as “target-aligned space,” which more accurately reflects the observed need for per-domain adaptations.\n\n###\tUnsupported Assumption Regarding  $e_{VI}$  and Invariant Space Error:\nOn line 164, the authors assert that  $e_{VI}$  could be ignored, implying that the error between the vision-language model’s space and the purported domain-invariant space is negligible. This assumption is dubious without further justification. The vision-language model’s embedding space may indeed diverge from the so-called invariant space, leading to substantial misalignment and error. Ignoring  $e_{VI}$  risks undermining the model’s robustness in handling domain shifts.\n\nI suggest the authors provide empirical evidence for dismissing  $e_{VI}$ and validation of $d_I^0 \\approx d_V^0 \\gg e_{VI}$ by conducting experiments across a range of scenarios like different domain adaptation settings.\n\n###\tOracle Configuration:\nThere is a flaw in the Oracle experiment setup, particularly in the Cl-to-Ar scenario (Line 410). The Oracle is incorrectly trained on the source domain (Cl) rather than the target domain (Ar). If this was a typo, I recommend correcting it. Otherwise, if this is intentional, it would be helpful to clarify the rationale for training on the source domain and claiming it to be an oracle.\n\n###\tOver-Reliance on a Single Vision-Language Model (CLIP):\nThe authors rely solely on CLIP for their experiments, neglecting to evaluate the method’s effectiveness with other vision-language models (e.g., LLaVA, Llama). This limitation raises concerns about the generalizability of the approach. Vision-language models have varied architectures and domain alignment properties, and the performance may vary significantly across models. Without results on other models, it is unclear if the proposed method is tailored specifically to CLIP or if it can generalize to other ViL models.\n\nI suggest that the authors expand their experiments to include other vision-language models, such as LLaVA and Llama. Reporting these results would provide valuable insights into the generalizability of ViL models. \n\n###\tGeneralization Claims to SFDA Settings:\nThe authors claim that their approach can generalize to broader SFDA settings, yet they do not provide any insights, or experimental results for critical scenarios like source-free multi-target domain adaptation (SF-MTDA [1])  and source-free multi-source domain adaptation (SF-MSDA [2]). \n\nTo enhance the rigor of their claims, I recommend that the authors either include experimental results for SF-MTDA and SF-MSDA scenarios or provide a detailed discussion on potential limitations or adaptations necessary for these settings. This addition would clarify the scope and limitations of the method’s applicability.\n\n\n### References\n\n[1] Kumar, Vikash, et al. \"Conmix for source-free single and multi-target domain adaptation.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[2] Ahmed,Miraj, et al. \"Unsupervised Multi-Source Domain Adaptation Without Access to Source Data.\"  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJ6Q1X7VXE", "reviewer": "Reviewer_X5vf", "summary": "This paper introduces Proxy Denoising (ProDe) to improve Source-Free Domain Adaptation (SFDA) by addressing noisy predictions from Vision-Language (ViL) models. The authors propose a proxy denoising mechanism based on proxy confidence theory to correct these noisy predictions and guide adaptation toward a domain-invariant space. They further enhance this process with mutual knowledge distillation regularization. Experiments demonstrate that ProDe outperforms existing methods across various SFDA settings, including closed-set, open-set, partial-set, and generalized scenarios.", "questions": "*\tCould the authors provide more detailed descriptions for Fig. 1? Additional explanations would help readers better understand the key ideas of the paper.\n---\nI would consider increasing my score if the authors address the concerns raised by me and other reviewers.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*\tThe paper is well-written, well-organization, and easy to follow.\n*\tThe insight on rectifies the inaccurate predictions of ViL models significantly contributes to SFDA settings.\n*\tThe paper introduces a novel ProDe method, which effectively corrects ViL model predictions through the use of a proxy confidence theory, offering a reliable approach to prediction refinement.\n*\tThe mutual knowledge distillation regularization is a strong addition, enabling the model to capitalize on refined proxy predictions with improved efficiency.\n*\tThe authors evaluate the proposed method through extensive experiments, including challenging partial-set, open-set, and generalized SFDA settings, demonstrating the versatility of the method.", "weaknesses": "*\tOverall, this paper is of high quality, with clear motivation and novel insights. The proposed ProDe method also demonstrates strong performance in the SFDA scenario. It would be interesting to see additional results in a similar scenario, such as test-time domain adaptation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)