Vicinity-Guided Discriminative Latent Diffusion for Privacy-Preserving Domain Adaptation
Authors: Jing Wang
Wonho Bae
Jiahong Chen
Wenxu Wang
Junhyug Noh
Venue: NeurIPS 2025 poster ()
Keywords: Latent Diffusion Model
$k$-Nearest Neighbor
Domain Adaptation
Privacy Preserving
Primary area: general_machine_learning
TL;DR: The paper introduces a novel latent diffusion model that uses local vicinity structures to achieve state-of-the-art domain adaptation, preserving privacy without needing source data during adaptation.

Abstract: Recent work on latent diffusion models (LDMs) has focused almost exclusively on generative tasks, leaving their potential for discriminative transfer largely unexplored. We introduce Discriminative Vicinity Diffusion (DVD), a novel LDM-based framework for a more practical variant of source-free domain adaptation (SFDA): the source provider may share not only a pre-trained classifier but also an auxiliary latent diffusion module, trained once on the source data and never exposing raw source samples. DVD encodes each source feature’s label information into its latent vicinity by fitting a Gaussian prior over its k-nearest neighbors and training the diffusion network to drift noisy samples back to label-consistent representations. During adaptation, we sample from each target feature’s latent vicinity, apply the frozen diffusion module to generate source-like cues, and use a simple InfoNCE loss to align the target encoder to these cues, explicitly transferring decision boundaries without source access. Across standard SFDA benchmarks, DVD outperforms state-of-the-art methods. We further show that the same latent diffusion module enhances the source classifier’s accuracy on in-domain data and boosts performance in supervised classification and domain generalization experiments. DVD thus reinterprets LDMs as practical, privacy-preserving bridges for explicit knowledge transfer, addressing a core challenge in source-free domain adaptation that prior methods have yet to solve. Code is available on our Github: https://github.com/JingWang18/DVD-SFDA.

Vicinity-Guided Diseriminative Latent Diffusion for Privacy-Preserving Domain Adaptation
Jing Wang'
Wonho Bae'
Jiahong Chen Wenxu Wang* Junhyug Noh
University of British Columbia
BBwha Womans University
3 Ammaoon Inc.*
*Ocean University of China
wang94@alumni ubc. junhyugGewha. ac.kr
Abstract
Recent work on latent diffusion models (LDMs) has focused almost exclusively generative tasks, leaving their potential for discriminative transfer largely unex- plored. We introduce Discriminative Vicinity Diffusion (DVD), novel LDM-based framework for more practical variant of source-free domain adaptation (SFDA): the source provider may share not only a pre-trained classifier but also an auxiliary latent diffusion module, trained once on the source data and never exposing source samples. DVD encodes each source feature's 's label information into its latent vicinity by fitting Gaussian prior over its k-nearest neighbors and training the diffusion network to "drift" noisy samples back to label-consistent represen- tations. During adaptation, apply the frozen diffusion module to generate source- cues, and use a sim- ple InfoNCE loss align the target encoder these cues, explicitly transferring decision boundaries without source access. Across standard SFDA benchmarks, DVD outperforms state-of-the-art methods. We further show that the same latent diffusion module enhances the source classifier's s accuracy on in-domain data and boosts performance in supervised classification and domain generalization experi- ments. DVD thus reinterprets LDMs as practical, privacy-rreservin bridges for explicit knowledge transfer, addressing core challenge in source-free domain adaptation that prior methods have yet to solve. Code is available on our Github: httpp:://githbb www cow./inggiagg1//DDD--ST5.
sample from each target feature's latent vicinity,
Introduction
Latent Diffusion Models (LDMs) have recently gained prominence artificial intelligence for executing the diffusion process in a lower-dimensional latent space [Rombach et al. 2022). Compared to pixel-level diffusion models [Ho al., 2020 Song et al. 2021]. LDMs offer notable improvements in efficiency by removing the need for computationally expensive gradient updates at every diffusion step [Dhariwal and Nichol, 2021]. Their flexibility incorporating multi-modal signals, such as text prompts for image generation [Avrahami et al. 2023), motivates to investigate whether LDMs can be repurposed for a discriminative goal, specifically for source-free domain adaptation (SFDA), where core challenge remains unsolved: enabling explicil knowledge transfer from a source model without access to source data.
In SFDA [Liang et al.) 2020), classifier trained on a labeled source domain must adapt to an unlabeled target domain without directly accessing the source data. This setting arises in scenarios where original training data cannot be shared due to privacy or proprietary restrictions (e.g., medical images from different hospitals). Even if we can release the source model parameters, we typically
Work conducted outside Amazon.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cannot provide the raw source data to the practitioners, Existing SFDA methods struggle to deliver explicit knowledge transfer under such constraints. This motivates our reformulation of SFDA as privacy-preserving domain adaptation, where the source provider may release lightweight auxiliary module, without exposing raw data, to assist downstream adaptation. Our central question is: Can LDDMs enable explicit discriminative knowledge transfer under these constraints? We address this by proposing Discriminative Vicinity Diffusion (DVD).
DVD builds on the smoothness assumption from semi-supervised learning [[scen et al. 2019]. positing that nearby points in feature space tend to share the same labels. Concretely, DVD encodes each source data's 's label information into its latent vicinity (the k-nearest neighbors in latent space) by defining a Gaussian prior around this vicinity in the diffusion process. Intuitively. the learned drift-only diffusion then acts like reverse diffusion, pulling noisy latent samples back onto the source data manifold defined by that vicinity and restoring label-consistent representations. Leveraging the property of latent diffusion models, which iteratively guide samples toward more probable regions in latent space, DVD can generate source-like latent samples for any new feature whose encoded representation lies near learned vicinity. Thus, during target adaptation, DVD uses the target data's 's own local neighbors to generate source-like features, explicitly transferring source decision boundaries without exposing raw source data.
While typical SFDA setups allow the source provider share only a backbone classifier (e.g.. ResNet [He et al. 2016)), we extend this by permitting the release of an additional DVD module, without raw data, thus framing the problem as privacy-preserving domain adaptation. Although this introduces minimal overhead, it enables the target user not only to adapt under strict privacy constraints but also to better understand the adaptation process. The source provider also benefits: DVD augments latent features during training, yielding more robust decision boundaries and improved source-domain accuracy. Beyond SFDA, we show that DVD further enhances domain generalization, improving performance unseen domains without adaptation. Our main contributions are:
We introduce Discriminative Vicinity Diffusion (DVD), an LDM- framework for explicit cross croms-n knowledge transfer without sharing source data.
We propose latent vicinity guidance mechanism that leverages latent k-nearest neighbors (k-NNs) to parameterize Gaussian priors more effectively than simple noise injection in discriminative tasks.
We demonstrate that DVD achieves state-of-the results on multiple SFDA benchmarks by surpassing existing methods.
We further show that DVD improves source-domain accuracy through latent augmentation and exhibits strong domain generalization on
unseen domains.
2 Related Work
Contrastive Source-Free Domain Adaptation. The motivation behind SFDA is to enable domain adaptation when sharing source data is restricted due privacy, security, storage constraints. In SFDA, we adapt a source-trained model to an unlabeled target domain without access to raw source data (Liang al. 20200. Contrastive methods apply InfoNCE-style loss on model outputs to cluster target embeddings by class predictions [Chen et al. 2020 Yang et al.) 2022 Zhang et al.) 2022], offering strong performance with minimal complexity. We adopt this simple contrastive objective to align target features with source-derived cues.
Diffusion and Latent Diffusion Models. Diffusion generative models iteratively denoise samples via stochastic differential equations (SDEs), often incurring high computational cost [Ho et al. 2020 Song et al., (02TT. ). Latent diffusion models (LDMs) alleviate this by operating in low-dimensional
latent space. significantly reducing inference time and resource usage (Schramowski et al.) 2023
Avrahami et al. 2023). This efficiency underpins our choice to repurpose LDMs for discriminative
and source-free transfer rather than generation alone.
Diffusion for Domain Adaptation. Prior work has leveraged diffusion models for data augmenta- tion to enhance domain adaptation performance, e.g., text-to-inn tomiaegee diffusion to synthesize target-style samples with source labels [Benigmim et al., 2023) or 3D generative diffusion for shape transfer (Kim) and Chun 2023 These methods operate in pixel space and focus on expanding training data, whereas DVD applies latent diffusion to directly transfer discriminative cues via local Gaussian priors over k-nearest neighbors, enabling explicit source-free knowledge transfer.

---

Source Training
Diseriminative Vicinity Diffasion (DVD)
Ganssian Sampler
D
F
ce
Ns (25)
(b) Target Adaptation
My (x.)
P
Gr
N (z)
F
cets
Gauselan Sampler
D
Aggrogator (SILGA)
Zyus
Ppos
Zg
zi
Auxillaay Module Training (DVD SILGA)
Figure 1: Framework overview. In source pre-training, DVD aligns source features for consistent predictions within their vicinities. In target adaptation, the target latent vicinity guides DVD to generate features from nearby source latent vicinities based on their similarities. The parameterization of diffusion priors enables this guidance across domains. After these training phases, inference requires only the classifier F Gt, without invoking D.
Generative SFDA Methods. Many SFDA approaches generate source-like features or prototypes without access to data: CPGA builds class prototypes via contrastive learning [Qiu et al. 2021]. 3C-GAN uses GAN to produce pseudo-labeled target-style images [Li et al. 2020J, and SFADA synthesizes proxy samples (He et al. 2024 In contrast, DVD employs deterministic and latent- space diffusion process guided by label smoothness in local vicinities, sidestepping GAN training and complex data density modeling in the target domain.
3 Method
We introduce Discriminative Vicinity Diffusion (DVD), latent-space diffusion framework for Privacy- Preserving Domain Adaptation (a variant source-free domain adaptation). DVD operates with drift-only diffusion model [Heitz et al. 2023) in the encoder's s latent space, enabling efficient encoding and retrieval of source-domain discriminative knowledge without exposing raw source data.
3.1 Preliminaries
Notation and Setup. Let S = (x**"""""'} be the labeled source domain and T = (x* i M the unlabeled target domain. We denote the classifier by f F= G, where G (e.g., ResNet [He et al.) 2016)) maps inputs into latent space and F is small fully-connected head.
li=l
A Variant of SFDA: In typical SFDA, only pre-trained classifier (G., F) is shared. We generalize this to a privacy-preservng domain adaptation setting, where the source provider can also release an auxiliary module without ever exposing raw data. Specifically, introduce latent diffusion module D trained once on S. By encoding source discriminative cues into a compact and privacy-preserving representation, D enables more flexible and robust adaptation to T under strict privacy constraints.
Latent Vicinities. Both at source pre-training and at target adaptation time, we rely on k-nearest neighbors (k-NNs) in latent space capture local geometry. Specifically, for the source side, maintain a feature bank:
Bg = (xo(xtt) x E ES.. j=1...,,
(1)
3

---

Similar to [Huang et al. 2019]. given query's 's feature vector zdd (x(xx')) we use the cosine
similarity to determine its latent k-NNs within Eg:
d(z (Bg) 2 B
(2)
[B
B'S denotes the j-th element of By for input data x Distinct notations are used for the query's encoded features z and features in the bank B/s, which reflects potential differences before the bank updated with the current encoder parameters. Thus, the latent vicinity comprising latent k-NNs of the query zz, is defined as:
Na(z²*) == arg maX) Ed(z").Bu
(3)
JEK
where K. a set that includes indices of all k-NN features.
An analogous bank ET and procedures apply to target-domain features at adaptation time.
Diffusion Background. Conventional diffusion generative models define forward noising process and learn to reverse it by using a score function that estimates the gradient of the log-likelihood the transition probability distribution in the SDE. While effective, pixel-level diffusion typically requires (i) large computational resources per step and (ii) explicit modeling a noise schedule (-) In contrast, leverage a deterministic (drift-only) method in the latent feature space to implement DVD, eliminating the need for stochastic noise estimation and repeated encoder backpropagation.
3.2 Deterministic Latent Diffusion
Blending and Drift. Suppose we want to transform a latent vector from one distribution Po into another 1 ~ P1. We define a blending operation with a scalar a E [0, 1]:
Zo (1 - a) Zo + a Z1.
(4)
drift function D(za. a), parameterized by neural network, then refines the blended state by predicting the vector difference (z1 - -zo). Discretizing into T steps,
ag = 0, a1 }. oT 1,
we iteratively update
(5) starting with Zan Zg- After T steps, we obtain Z,, = 2 Following [Heitz et al. 2023). our drift network (z 04t)) is trained to predict the global difference (2) 2o. for each pair, rather than a local increment at each step. This formulation ensures that the drift field always points from the start to the end 21, while the actual local increment is scaled by (at+1 a²). As a result, this deterministic update path avoids error accumulation and guarantees transport along the straight line between Zo and z The detailed rationale for this formulation is provided in Appendix A.I
Zez41 Zo. (at+1 are) Dizec at))
Although this update proceeds in T discrete steps, it does not constitute iterative optimization loop, instead, it is single, deterministic drift path. Empirically, we find that small T (e.g., 8) suffices to recover lobss-consistent features, thanks the strong regularization provided by the latent vicinity prior and the pre- trained classifier. Unlike residual regression or loss- mess-minimizing iterations, these updates follow pre-defined transport path that is explicitly regularized by the Gaussian prior of the local latent vicinity. Each intermediate state therefore remains within a label-consistent region, preventing arbitrary drift or error accumulation. Moreover, Appendices [B.5 and B.6 demonstrate that DVD is robust across vicinity radii and drift steps, confirming the stability of the drift mechanism.
Training the Drift Function. To train the drift function D, we sample pairs (Zo, 21) (po pi). randomly pick a E [0. 1]. and form Z,, via (4). Our training objective is to minimize:
Las Ea...., D[ Zo., an) Ex =x) ..)) [2 - zu]11a]
(6)
This encourages D to reproduce the average difference between the original zo and z zf corre- sponds to labeled source feature, we additionally impose cross-entropy term to ensure the final state Zor (i.e,, 21) is classified correctly by F, where (-)) denotes the softmax function:
Lco Σ log a (F(21)) y...
(7)
4

---

Algorithms 1 and 2 detail the training and sampling procedures, Notably, no stochastic noise is added each step, and the forward pass does not require computing encoder gradients, making computationally lighter than pixel-level diffusion and well-suited for discriminative purpose.
3.3 Storing Source Knowledge via DVD
We begin by training a source classification model f. P o G. on the labeled source domain S using cross-entropy. Once Sa converges, we freeze G. and F and focus on training the drift function D. The goal to map each local latent prior surrounding a source embedding back to that embedding itself, thus "storing" discriminative knowledge in D without the need to revisit source data.
Concretely, for each source pair (x** f.), we first compute its latent representation z G2 (x*)). We then identify the latent vicinity N& (z for each source data using k-NNs in latent space and compute its mean and variance:
14
Σ
o
1 K
Σ
(2(22
-mp) -m6)²
(8)
These define a Gaussian prior P0 N( );²); ) sample zū Pū as the "start" state and set zi = z²d as the "target" state. Finally, we apply Algorithm
| to train D so that drifting from zi to zi under D yields representation that F correctly classifies as y
This teaches D "pull" noisy latent samples onto the true source manifold, effectively encoding all source discriminative cues within D and eliminating any need to access S during adaptation. Moreover, these latent-space augmentations also improve the source-domain classifier's s robustness, further benefiting the source provider.
Algorithm 1: Latent Diffusson Training
Input:
Algorithm 2: Latent Diffusson Sampling
(Zo, z1) ~ (po x P1) a Uniform [0, 1]
Input:
(Zo, 21) (po PI) Discrete at = f: steps
D's parameters Φi steps T
for f = 0 do
Initialize: Zon f Zo
Za. t (1-aa) az)zz + 0
for t 0 to T 1 do
Update using Lai (Eq. 6
Zaril f- Zor + (at+1 - 04) D(Zay o1);
Sample 2 via Algorithm
If zi has label, minimize L.e (Eq. 7
end
end
Output: 21 Zar 23 Zi
Output: Updated parameters /
3.4
Target Adaptation
Given only unlabeled target data T and pre-trained (G,, F, D), we initialize a new encoder G from G, and fine-tune only G., keeping F and D frozen. During adaptation, the diffusion module D generates source-aligned cues that guide G, to shift its latent outputs onto the source manifold, ensuring they lie within the frozen classifier F's decision boundaries. At inference, D is longer used with the predictions made by F o G.-
Local Prior Sampling for Target Data. Given target sample x encode as n (x("")) We then identify its k-NNs in the target feature bank lrT and compute their mean and variance. This defines Gaussian prior ph N(ng, a Sampling z pó and applying Algorithm 2 with the frozen drift model 1 produces z which carries source-aligned, discriminative cues tailored z Importantly, DVD does not collapse the entire target distribution onto the source; rather, it guides each target feature toward its most relevant source-informed latent cluster, ensuring semantic alignment with the frozen source classifier's s decision boundaries.
Cource-Informel Latent Geometry Aggregation (SiLGA).. Although z! carries source cues, relying on it alone can lead to label mismatches if the domain shift is large. To mitigate this, we blend
5

---

zi with local target neighbors:
1.
ExeNelatitt
pos
where z (z are the top-k neighbors drawn from the original target feature bank BT. recalcu- lated at each update rather than taken from the generated features. This ensures that the aggregation consistently reflects the current latent geometry of the target distribution. After sampling from the nearest nource-informed Gaussian prior, SiL.GA blends this feature with the target's local k-NN centroid, which is especially beneficial under large domain gaps (e.g. VisDA-C) where target features are far from source centroids.
We then adopt an InfoNCE-style contrastive objective (Oord et al. Chen et al. 2020 Wang et al. 2024] cluster unlabeled targets. Concretely, let
Contrastive Clustering.
2018
p' = oF(((())
Ples = a o (F)po=))
(10)
where (.) denotes softmax function and F is frozen. For mini-batch of size we minimize
m
explp' Alpp--pps- /T) Ejz exp(p' log exp'eppp'' popppp/11) p²/T)
Lole
Σ
(11)
where is temperature hyperparameter. To maintain focus exploring the role of DVD in knowledge transfer without source data, we avoid incorporating additional techniques, such as momentum updates [He al. 2020]. Minimizing Loie pulls each target feature z toward its SiLGA- based positive xoo.. effectively aligning G. with the source-discriminative cues learned in D. After sufficient epochs, G, converges to representation that classifies T accurately without seeing any source data, thus satisfying SFDA constraints. At inference time, only Ge and F are used, where D is not invoked, there is additional computational overhead beyond standard forward pass.
4 Experiments
We evaluate DVD on three tasks: source-free domain adaptation (Section 4. supervised single- domain classification (Section 4.2), and domain generalization (Section
4.3 We then analyze
hyperparameter sensitivity (Section 4.4 and runtime efficiency (Section 4.5 Additional results and
ablation studies provided in the Appendix.
4.1 Source-Free Domain Adaptation
In this section, we evaluate DVD's effectiveness in SFDA, focusing on its role discriminative knowledge transfer without source data during target adaptation. We extend our discussion on DVD's relaxed SFDA protocol, where an auxiliary module is trained once offline while preserving strict source-free privacy during adaptation, and provide additional comparisons with conventional domain adaptation methods in Appendix C.T
Datasets. We conduct experiments on three widely recognized SFDA benchmarks:
Office-31 [Saenko al. 2010 4,652 images across 31 classes collected from three domains
Amazon (A), Webcam (W), ), and DSLR SLR (D).
Office-Home (Venkateswara al. 2017]: 15,500 images in 65 classes from four domains Artistic
(Ar), Clipart (CI), Product (Pr), and Real-World (Rw).
VisDA-C 2017 (Peng al. 2017]: ; 280,000 images spanning classes, where the source domain
is rendered via 3D models, and the target domain consists of real images captured by RGB cameras.
Experimental Setup. To simplify the implementation of DVD, we adopt consistent framework for all benchmarks. ResNet-50 serves as the encoder G for Office-31 and Office-Home, while ResNet-101 used for VisDA-C 2017 to align with existing SFDA baselines. A two-layer linear head F performs classification. We use conditional UNet [Ho et al. 2020 for D, with 16 diffusion steps in both training and inference. SGD optimizer (learning rate 10-3, momentum 0.9, batch size 128) used for parameter updates. We employ the InfoNCE objective from SimCLR (Chen et al. 2020

---

Table 1: Comparison of the SFDA methods VisDA-C (ResNet-101).
Method
Add.
plane
beyel
bus
car
horse
knifc
meyel
person
plant sktbrrd train truck
Avg. + s.d.
ResNet-101 He et al. 2016
53.3
61.9
59.1
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52.4
SHOT
Liang ct a 2020
94.3
88.5
80.1
57.3
93.1
94.9
80.7
80.3
91.5
89.1
86.3
58.2
82.9
HCI
Huang ei al
2021
93.3
85.4
80.7
68.5
91.0
88.1
86.0
78.6
86.6
88.8
80.0
74.7
83,5
DaC
Zhang al.
2022
96.6
86.8
86.4
78.4
96.4
96.2
93.6
83.8
96.8
95.1
89.6
50.0
87.3
NRC++ Yang
2023b
91.9
82.8
97.1
96.2
90.0
95.2
91.1
49.6
87.8
SFADA
He ct al.
2024
94.2
79.6
79.8
65.7
92.6
94.1
87.3
80.8
88.1
91.4
83.3
55.0
82.7
CPD Zhou et al. 2024
96.7
88.5
79.6
69.0
95.9
96.3
87.3
83.3
94.4
92.9
87.0
58.7
85.8
3C-GAN
Li 2020
95.4
75.8
70.5
73.9
92.4
94.3
89.4
82.5
91.7
90.1
85.2
50.3
82.6
STADA
He et
2024
94.9
80.4
80.6
68.2
94.3
94.0
86.5
82.1
91.4
92.2
85.0
51.4
83.4
DVD (Ours)
98.4
92.1
83.9
83.6
98.1
96.5
92.1
82.9
97.0
95.2
92.6
54.6
Table 2: Comparison of the SFDA methods on Office-Home (ResNet-50).
Ar
CI
Pr
Rw
Method
Add.
Avg. 1 s.d.
Pr
Rw
Ar
Pr
Rw
Ar
CI
Rw
Ar
Cl
Pr
ResNet-50
He ct al
2016
34.9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
DaC 7hang et al. 2022
59.5
79.5
81.2
69.3
78.9
79.2
67.4
56.4
82.4
74.0
61.4
84.4
72.8
NRC+ Yang el al 2023b
57.8
80.4
81.6
69.0
80.3
79.5
65.6
57.0
83.2
72.3
59.6
85.7
72.5
SFADA He ct al. 2024
56.1
78.0
81.6
68.5
79.5
78.5
67.8
56.0
82.3
73.6
57.8
83.0
71.9
CPD Zhou ct al. 2024
59.1
79.0
82.4
68.5
79.7
79.5
67.9
57.9
82.8
73.8
61.2
84.6
73.0
3C-GAN Li
2020
<
57.4
79.3
81.8
69.1
79.8
80.0
66.5
56.2
83.9
72.4
60.1
85.4
72.7
SFADA
He et
2024
57.8
78.5
82.3
68.2
79.6
79.2
66.9
57.4
83.8
73.4
58.0
84,1
72.4
DVD (Ours)
60.1
79.6
82.5
69.1
80.8
80.6
67.9
58.5
84.3
73.5
59.3
87.6
73.7 + 0.7
with temperature 7 = 0.13. We define three parameters: kdlff and k (number of neighbors used to parameterize the priors for DVD), and k, (number of neighbors for preserving the local target feature structure). Unless otherwise stated, we (kefs, kdif k.) = (15, 1).
Results. Tables T 2 and present adaptation results on VisDA-C 2017, office-Home, and Office- 31, respectively. We measure performance by classification accuracy (%) on the target domain. "ResNet" denotes the baseline where a pre-trained source model directly applied to the target domain without adaptation, and "Add." indicates whether method uses additional modules beyond the backbone during training. As shown, DVD achieves satte-of-theaat results on benchmarks, demonstrating its ability to transfer discriminative knowledge without requiring source data during target adaptation. Recent SFDA studies have also explored ViT-based architectures [Xu et al.) 20222] Yang et al. 2203aa Sanyal et al. 2024 classification backbones. For a fair comparison, we further evaluate DVD with VIT-B/16 backbone, where it continues to outperform all existing methods (see
Appendix C.2
t-SNE Visualization. Figure illustrates the impact of our Discriminative Vicinity Dif- fusion (DVD) the VisDA-C 2017 target features. Each point is colored by its ground- truth class label. On the left (before adap- tation), domain shift causes heavy overlap across classes, resulting in scattered features and blurred decision boundaries. On the right (after adaptation with DVD), our method first samples from local Gaussian prior defined by each feature's 's k-nearest neighbors, then applies drift-only diffusion step to "pull" these samples onto the source manifold. This process propagates label consistency from the source into the target latent space, aligning features with the frozen classifier's 's decision boundaries. As re-
(b) With DVD Figure 2: t-SNE of target features on VisDA-C 2017.
(a) Without DVD
7

---

Table 3: Comparison of SFDA methods on Office-31 (ResNet-50).
Method
Add.
A-D
A-W
D-W
D-A
W-D
W-A
Avg. 1:
ResNct-50
He
2016
68.9
68.4
96.7
62.5
99.3
60.7
76.1
SHOT
Liang
2020
94.0
90.1
98.4
74.7
99.9
74.3
88.6
AaD
Yang
2022
95.5
92.1
98.5
74.0
99.4
75.8
89.2
NRC+
Yang 2023b
95.9
91.2
99.1
75.5
100.0
75.0
89.5
STADA
94.8
92.0
97.6
76.5
99.8
75.7
CPD [Zhou et 2024
96.6
94.2
98.2
77.3
100.0
78.3
90.8
3C-GAN Lt
20200
93.4
92.9
97.5
76,5
99.8
77.3
SFADA HH al.
2024
95.2
91.4
98.2
77.8
100.0
76.3
89.8
DVD (Ours)
96.7
95.2
98.6
79.2
100.0
77.4
91.2 1: 0.5
sult, clusters become tight and well-separated, visually confirming that DVD transfers discriminative knowledge without exposing any source data.
4.2 Supervised Single-Doman Domman Classification
Next, we examine how DVD act latent augmentation tool enhance supervised classification, benefiting source providers. Rather than relying solely on encoder outputs, we generate features via DVD and feed them the classifier.
Experimental Setup. We follow standard training protocols for supervised classification bench- marks. Specifically, we use SGD with momentum (0.9), a weight decay of 5 x 10-1, mini-batch size of 128, and train for 200 epochs. The learning rate starts at () and follows a cosine annealing schedule [Loshchilov and Hutter [017]. DVD is trained with the same hyperparameters used in the SFDA experiments. At test time, for each data, we identify its latent k-NNs, generate augmented feature via DVD, and pass it into the classifier.
Datasets. We evaluate DVD>-based latent augmentation three standard benchmarks (additional results on domain adaptation datasets are in the Appendix C.3] demonstrating improved source- domain classification performance):
CIFAR-10 and CIFAR-100 Krizhevsky et al. 2009]: Each dataset has 60,000 images (50,000 for training, 10,000 for testing). CIFAR- 10 contains 10 classes, and CIFAR-100 has 100 classes. ImageNet [Russakovsky et al., 2015]: The well-known ILSVRC 2012 benchmark with 1.2
million images in 1,000 object classes.
Results. Table presents the top-1 error (%). Although small network architectures are used for illustration (not to set new state-of-te ttate-of-the-ane art), DVD- WWD-ggnerated features consistently yield higher accuracy than using the encoder outputs alone. By leveraging k-NNs in latent space, DVD effectively enlarges the model's representational power while preserving label consistency. Hence, even with classification parameters held fixed, DVD acts
Table 4: Top-1 test error (%).
Method
CIFAR-10
CIFAR-100
ImageNet
ResNet-18
7.07
22.74
31.46
DVD
6.52
22.01 +
31.06 0.2
ResNet-50
6.35
22.23
24.68
DVD
5.92 0.2
21.95 = 0.3
24.30 0.3
VGG-16
7.36
28.82
25.82
DVD
6.42 0.1
26.58 1: 0.3 25.02 0.3
a latent augmentation module that improves generalization on in-distribution test data.
4.3 Domain Generalization
We next demonstrate how DVD enhances domain generalization, where models perform well on unseen target domains without any adaptation. Specifically, we consider the source-free domain generalization (SFDG) setting [Cho al.) 2023]. which prohibits fine-tuning on target data and focuses on transforming the test data or model outputs.
Experimental Setup. We follow (Cho et al. 2023 and adopt ResNet-50, pre-trained with CLIP [Radford al. 2021), backbone. During target inference, obtain the DVD-generated

---

feature 2 Li and then apply SiLGA leveraging target samples' latent k-NN features. For simplicity, we fix ls 5 for both the DVD prior parameterization and the SiLGA latent geometry searching.
Datasets,
We evaluate DVD DVD-generated features on four standard domain generalization benchmarks:
PACS [Li al. 2017 9,991 images across 7 classes, with each domain contributing roughly
2,000 images.
VLCS [Fang al. 2013 10,729 images spanning 5 categories, each domain providing about 2,000 to 3,000 images.
The same 65-class dataset used in SFDA, consisting of DomainNet eLi al., 2017 A large-scale benchmark with 586,575 images across domains and 345 object categories.
Offece-Home Venkateswara al. 2017: four distinct domains.
Results. Table 5 shows the classification accuracy (%) for the target domain. DVD consistently yields strong performance, improving upon ResNet-50 by at least 9% on every dataset. Moreover, outperforms existing methods without using additional Aomain-generalliztion techniques such as momentum ensembling or batch-statisiics averaging. This suggests that DVD is complementary other methods [Lim al. 2023 Cho et al. 2023] and can be further combined with advanced approaches to achieve even stronger generalization.
Table 5: Domain generalization accuracy (%) using ResNet-50.
Method
PACS
VI.CS
Office-H
DomainNet
ResNet-50 He al.
2016
84.5
72.8
61.3
40.2
ZS-CLIP(() Radford cl gl
2021
90.6
76.0
68.6
45.6
CAD Dubois et al.
2021
90.0
81.2
70.5
45.5
ZS ZS-CLIP(PC) Radford el al.
2021
90.7
80.1
72.0
46.3
PromptStyler Cho
93.2
82.3
73.6
49.5
DVD (Ours)
93.8.L 0.4
81.9 1 1.2
74.5
1
4.4 Sensitivity to Hyperparameters
DVD has three key hyperparameters: (1) the number of diffuusin steps T, (2) the source-side neighbor count k:K and (3) the target-side neighbor count I. To evaluate each in isolation, we vary one
while holding the others at their defaults (T = 16, k:lh = 15, x 15). Figure demonstrates that DVD's performance remains strong and stable across wide range of settings, confirming that tuning for robust cross-domain classification straightforward.
10.20
89.23
DVD to
29.00
89.00
83.75
$8.25
88 25
18.00
83.00
17.75
87.75
16
(a) Number of Neighbors (k)
(b) Numher of Diffusion Steps (T)
Figure 3: (Best viewed color.) Hyperparameter sensitivity analysis. The results demonstrate the robustness of our DVD across range of different hyperparameter settings.

---

4.5 Runtime Analysis
This section quantifies DVD's computational cost during training and at deployment, All timings were collected on an Nvidia V100 GPU and are reported as mean over 5 runs using identical conditions. We report:
Epoch (s) Convergence (s): training time per epoch and total time to converge under the shared schedule.
Inference Time (ms): forward-only time from device-resident input tensor to logits (no host I/O). Latency (ms): end-to-end wall-clock time, including minimal hosté>device transfers and pre/post- processing.
FPS: throughput under the same setting; approximately PPS x 1000/Infeeee Inference Time (ms).
Results. Table O shows that DVD's per-epoch time (516.3s) and overall convergence time (5,163.1 s) are on par with NRC+ and SHOT and much faster than DaC. Although DVD trains an auxiliary latent diffusion module, we use only small number of drift steps, so training overhead remains modest. At deployment, DVD does not invoke diffusion; prediction is single forward pass through F..., yielding 26. 1 ms inference time (38.2 FPS) and 51.1 ms end-to-end latency, effectively matching SHOT and NRC++.
NRC++, SHOT, and DVD all perform single fixehweiigt forward pass test time. DaC (Zhang et al. 2022] is slower because it maintains a momentun-updattd memory bank and computes prototype similarities inference, adding extra memory access and compute. Since DVD's diffusion is latent-space and training-only. it introduces no runtime penalty at deployment.
Table 6:
Training cost and deployment efficiency on VisDA-C (ResNet-101, Nvidia are reported as standart deviation over runs.
V100). Nvidia All values
Method
Epoch (s)
Convergence (s)
Inference (ms)
FPS
Lateney (ms)
DaC Zhang et al. 2022
632.813.1
12.656.3155.2
50.6110.8
19.810.55
86.211
NRC+
Yang et al. 2023b
469.2.12.7
1.6415
26.0.10.6
38.4100.4
51.011.7
SHOT
Liang et al. 2020
439.3+22.4
+.589
25.9+
38.6+0.3
50.8+0.6
DVD
(Ours)
516.3+2.9
5,163.1
26.1
38.2+0.2
51.1=0.8
5
Conclusion
In this paper, we introduced a novel use of LDMs for explicit discriminative knowledge transfer, an important yet underexplored capability in settings where raw source data cannot be shared. Motivated by the increasing demand for privacy-preserving and transparent AI, particularly in sensitive domains such as healthcare and finance, we proposed DVD, a method that leverages latent feature geometry to facilitate explicit target adaptation without requiring access to source data. Through k-nearest neighbor guidance. DVD enforces label-consistent transformations within the latent space, offering a structured and well-understood mechanism for adaptation under strict privacy constraints. Despite minimal computational overhead, DVD brings value to both sides: target users benefit from reliable and traceable adaptation behavior, while source providers can improve generalization via latent-space augmentation during training. This dual advantage highlights DVD's ss versatility as a robust and pwipppaa-pvieeereeeaee module for explicit cross-domain knowledge transfer.
domain adaptation by leveraging LDMs for explicit knowledge transfer, addressing key limitation in existing source-free methods, which often lack clear mechanisms for aligning source and target domains. Its core strength lies in balancing data sensitivity and decision accountability, making it especially suited for high-stakes fields such as healthcare, finance, and scientific research, where both confidentiality and reliable model behavior are significant. By explicitly transferring knowledge without exposing raw source data, DVD reduces the risk of privacy concerns. However, responsible use is important, care must be taken to avoid adapting models to biased or unrepresentative target domains. This highlight the importance of transparent validation and ethical deployment.
Broader Impacts. DVD enables
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (No. RS- (5-20225-10070597)
References
O. Avrahami, O. Fried, and D. Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Y. Benigmim, S. Roy, S. Essid, V. Kalogeiton, and S. Lathuilière. One-shot unsupervised domain adaptation with personalized diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 698-708, 2023.
J. C. Bezdek, S. K. Chuah, and D. Leep. Generalized k-nearest neighbor rules. Fuzzy Sets and Systems, 18(3):337-22 1986.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning (ICML), pages 1597-1607. PMLR, 2020.
J. Cho, G. Nam, S. Kim, H. Yang, and S. Kwak. Promptstyler: Prompt-driven style generation for nource-free domain generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15702-15712, 2023.
S. Chopra, S. Kothawade, H. Aynaou, and A. Chadha. Source-free domain adaptation with diffusion- guided source data generation. arXiv preprint arXiv:2402.04929, 2024.
Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34:8780-8794, 2021.
Y. Dubois, Y. Ruan, and C. J. Maddison. Optimal representations for covariate shifts. In NeurIPS 2021 workshop on distribution shifts: connecting methods and applications, 2021.
C. Fang, Y. Xu, and D. N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1657-1664, 2013.
Y. Grandvalet and Y. Bengio. Semi supervised learning by entropy minimization. Advances in Neural Information Processing Systems (NeurIPS). 17, 2004.
J. He, L. Wu, C. Tao, and F. Lv. Source-free domain adaptation with unrestricted source hypothesis. Pattern Recognition, 149:1 10246, 2024.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770-778, 2016.
K. He. H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
E. Heitz, L. Belcour, and T. Chambon. Iterative a-(de) blending: A minimalist deterministic diffusion model. ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY, USA, 2023. Association for Computing Machinery. doi: 10.1 14/33588433.35915400
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems (NeurIPS), 33:6840-6851, 2020.
Huang, Q. Dong, Gong, and X. Zhu. Unsupervised deep learning by neighbourhood discovery. In International Conference on Machine Learning (ICML), pages 2849-2855 PMLR, 2019.
J. Huang, D. Guan, A. Xiao, and S. Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances Neural Information Processing Systems (NeurIPS), 34:3635-3649 -6499, 2021.

---

A. Iscen, G. Tolias, Y. Avrithis, and O. Chum. Label propagation for deep semi-supervised learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
C. Jahan and A. Savakis. Unknown sample discovery for source free open set domain adaptation. In Proceedings of the pages 1067-1076, 2024.
the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
G. Kim and S. Y. Chun. Datid-3d: Diversity-preserved domain adaptation using text-to-image diffusion for 3d generative model. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 14203-14213, 2023.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5542- 5550, 2017.
R. Li, Q. Jiao, W. Cao, H.-S. Wong, and S. Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9641-9650. 2020.
J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference Machine Learning (ICML), pages 6028-6034 PMLR, 2020.
J. Liang, D. Hu, J. Feng, and R. He. Umad: Universal model adaptation under domain and category shift. arXiv preprint 1rXiv:2112.0883 2021.
H. Lim, B. Kim, J. Choo, and S. Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. International Conference on Learning Representations (ICLR), 2023.
M. Long, Z. Cao, J. Wang, and M. Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference Learning Representations (ICLR), 2017.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.037444 2018.
N. Papernot and P. McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: / /803.04765 2018.
X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint, page arXiv:1710.0699 2017.
V. H. Phan, T. L. Tran, Q. Tran, and T. Le. Enhancing domain adaptation through prompt gradient alignment. Advances in Neural Information Processing Systems (NeurIPS), 111455188-55551 2024.
Z. Qiu, Y. Zhang, Lin, S. Niu, Y. Liu, Q. Du, and M. Tan. Source-free domain adaptation via avatar prototype generation and adaptation. In International Joint Conference on Artificial Intelligence (ICCAI) 2021.
S. Qu, T. Zou, F. Röhrbein, C. Lu, G. Chen, D. Tao, and C. Jiang. Upeycling models under domain and category shift. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20019-20028, 2023.
S. Qu, T. Zou, L. He, F. Röhrbein, A. Knoll, G. Chen, and C. Jiang. Lead: Learning decomposition for source-fres free universal domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23334-23343, 2024.
12

---

A. Radford, J. W. Kim. C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML., pages 8748 1763. PMIR, 2021.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10684-10695, 2022.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 15(())211-255 2015.
K. Saenko, B. Kulis, M. Fritz, and TT Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), pages 213-226, 2010.
S. Sanyal, A. R. Asokan, S. Bhambri, YM, A. Kulkarni, N. Kundu, and R. V. Babu. Aligning non-causal factors for transformer-based source-free domain adaptation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1904-1913, 2024.
P. Schramowski, M. Brack, B. Deiseroth, and K. Kersting. Safe latent diffusion: Mitigating inappro- priate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 22522-22531, June 2023.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based gener- ative modeling through stochastic differential equations. International Conference on Learning Representation (ICLR), 2021.
A. Soni and T. Dutta. Toward improving robustness and accuracy in unsupervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 6987- 6995, 2025.
H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5385-5394, 2017.
J. Wang, W. Bae, J. Chen, K. Zhang. L. Sigal, and C. W. de Silva. What has been overlooked in contrastive source-free domain adaptation: Leveraging source-informed latent augmentation within neighborhood context. Transactions Machine Learning Research, 2024. ISSN 2835-8856.
T. Xu, W. Chen, P. WANG, F. Wang, H. Li, and R. Jin, CDTrans: Cross-domain transformer for unsupervised domain adaptation. International Conference on Learning Representations (ICLR), 2022.
J. Yang, J. Liu, N. Xu, and J. Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (WACV), pages 520-530, 2023a.
S. Yang. Y. Wang. K. Wang, Jui, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Yang, Y. Wang, J. van de Weijer, L. Herranz, S. Jui, and J. Yang. Trust your good friends: Source- free domain adaptation by reciprocal neighborhood clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Z. Zhang, W. Chen, H. Cheng, Z. Li, S. Li, L. Lin, and G. Li. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
L. Zhou, N. Li, M. Ye, X. Zhu, and S. Tang. Source-free domain adaptation with class prototype discovery. Pattern recognition, 145:109974, 2024.
Z. Zhuang, Y. Zhang, and Y. Wei. Gradual domain adaptation via gradient flow. In International Conference on Learning Representations (ICLR), 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made
in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
The paper has clearly indicated all assumptions made, the computational efficiency, and potential limitations.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The full set of assumptions and complete proof are given for all theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
14

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper includes all the implementation details. We have also uploaded the code for the implementation in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code has been uploaded to the supplemental material with sufficent instructions reproduce the main experimental results. All the datasets used are public, which can be uploaded directly online. Code is available on our Github:
com/ JingWang SVDD
Guidelines:
The answer NA means that paper does not include experiments requiring code.
15

---

Please see the NeurIPS code and data submission guidelines (https://nips.com http///dddde//dddsummmsssssi...o
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mtiips.c/ppblice
Saardddee/ooSSuummisite Poolicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper has included all the training and test details including data splits, hyperparameters, how they were chosen, and type of optimizer.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix,
supplemental
material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the experimental results reported in the paper include the mean and standard deviation of the repeated experiments with different random seeds. The error bars are also included for the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report I I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
16

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The paper has a specific section discussing all the details of the computational analysis which includes the type of compute workers, memory and time of execution to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics httblssstttt:///t http::///eerrpppeeee htw.or/pp...ccpooll///ccaceeseea Answer:
[Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration
due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper has included Broader Impacts section the end of the main manuscript.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
deepfakes for disinformation.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
17

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paperswi wwwwwcodeeee..oooo com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub-
missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
18

---

The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components. LLMs are only used for language clarity.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
for Please refer to LLM policy htttas////errrpscccc//ooffreeee/ what should or should not be described.
19

---

Appendix
This appendix provides supplementary materials that expand on the main paper:
In-Depth Motivation and Analysis for DVD (Sec. A Deterministic diffusion rationale, graphi- cal clssiictiasi example. latent-vicinity label propagation, and a probabilistic graphical- model formulation showing how latent vicinities drive cross-domain label transfer.
A Ablation Studies (Sec. B Analyses of the latent-vicinity prior, SiLGA feature aggregation, diffusion vs. k-NN mean pooling, representational power generative replay, robustness to vicinity radius k and synthetic domain shifts, effect of diffusion step size T, sensitivity to vicinity hyperparameters (Meiif k:gli ki.), and hyperparameter study (batch size, learning rate, temperature). Additional Experiments (Sec. C Fairness of comparisons across SFDA/PDDA/UDA protocols, results with a ViT-B/16 encoder, source-domain feature augmentation. supervised-clasifienn ablations, applicability to OSDA and PDA (DVD-CT), and robustness to initialization seeds.
A In-Depth Motivation and Analysis for DVD
We now explain how Discriminative Vicinity Diffusion (DVD) transfers knowledge to the target domain without requiring any source data. Our approach extends the classic smoothness assumption from semi-supervised learning [Grandvalet and Bengio 2004) into the latent spaces of both source and target domains, enabling DVD to effectively bridge domain shifts. This section is structured as follows:
Rationale for Deterministic Formulation: An explanation of why we adopt deterministic latent updates instead of a stochastic variant.
Graphical Motivation: An illustrative binary classification example showing how DVD realigns target features with source decision boundaries by exploiting latent vicinities.
Latent Vicinity Label Propagation: An explanation of how label smoothness within these local neighborhoods supports cross-domain label transfer, aided by a Gaussian prior.
Probabilistic Graphical Model Formulation: A formal treatment of DVD through a latent diffusion framework, highlighting how smoothness propagates across domains via local vicinities.
A.1 Deterministic Diffusion Rationale
In our DVD, the diffusion module is not used to generate diverse samples, but to reliably transport target features into semantically consistent regions aligned with the frozen source classifier. For this purpose, injecting random noise, which is common in generative diffusion, would be counterpro- ductive. Instead, design the latent drift to be fully deterministic, so that every generated feature follows a stable and reproducible path. This deterministic choice serves three key aspects of our objective:
Privacy and Reliability: By predicting only the global transport vector (z) 2o). the module avoids storing replaying raw source samples. Each update is reproducible and does not depend on random seeds, ensuring the same target feature is always mapped to the same semantically aligned region.
In contrast to generative tasks where diversity is valuable, SFDA requires sharpening class clusters under domain shift. Noise injection can blur decision boundaries and misalign features. A deterministic drift keeps adaptation focused on class-consistent manifolds, preserving discriminability.
Discriminative Adaptation:
Efficiency: Deterministie transport converges in a small number of steps, avoiding error accu- mulation and unnecessary computation. Stochastic variants require more steps and still degrade accuracy.
Table 7: Ablation on deterministic vs. stochastic drift on VisDA-2017.
Method
Target Ace, (%)
Runtime (second/iter)
Stochastic Drift (+ Noise)
87.2
2.5
Deterministic Drift (Ours)
88.9
1.2
20

---

To validate this design choice, we compare deterministic drift against a stochastic variant that injects Gaussian noise at each step, using the large-scale VisDA-2017 benchmark. As shown in Table the stochastic version reduces accuracy (-1.7%) and increases runtime significantly. These results confirm that noise injection not only undermines discriminative adaptation but also makes the procedure slower. By contrast, the deterministic scheme consistently yields sharper clusters, stronger alignment, and better efficiency, making it better suited for
way, without domain adaptation.
Overall, the deterministic formulation is not only simpler but also more principled match to the goal of DVD: to adapt target features efficiently, reproducibly, and in a compromising semantic alignment.
A.2 Graphical Motivation
To illustrate the main ideas behind DVD, Figure shows a binary classification example demonstrating how DVD transfers knowledge without accessing source data. This process unfolds in two key phases:
We define prior density by sampling from each source sample's s k--N neighbor- hood in latent space, thereby diffusing its ground-truth lahel throughout that vicinity. A pre-trained source classifier enforces consistency, ensuring that DVDDggnerraed features align with source labels.
DVD Training:
DVD Sampling for Target Adaptation: Once the DVD and source classifier are fixed, only the target encoder is adapted. For each target sample, we form a prior based on its k-NN neighbors to generate source-like features. We then apply contrastive learning with SiLGA, blending these generated features with local target neighbors. This encourages label consistency by realigning target representations to the source decision boundary, effectively bridging domain shifts without the raw source data.
By leveraging local neighborhoods in latent space, DVD explicitly propagates labels from the source domain to the target domain, thus laying the theoretical foundation for cross-domain knowledge transfer.
source fcature
target feature
influence of diffused ground truth
contrastive force on target features
effective region of diffusion
source decision boundary
DVD Training
DVD Sampling for Target Adaptation
C=1 1
C=1
After Applying Contrastive Forces
4
C=0
0
C=1
0
C=0 0
4
Figure 4: Graphical Motivation. A binary classification illustration of DVD- knowledge transfer without souree data (blue: class 0, green: class 1). (1) DVD Training: prior density is defined using the k-NN latent vicinity of each source sample, and a pre-trained source classifier "diffuses" ground-truth labels within that neighborhood. (2) DVD Sampling for Target Adaptation: With DVD and the classifier frozen, only the target encoder is updated via contrastive learning. We apply Source-Informed Latent Geometry Aggregation (SiLGA) to blend DVD- DyD-generated features with local target neighbors, thereby aligning target samples to source decision boundaries.
21

---

A.3 Latent Vicinity Label Propagation
DVD builds upon the latent smoothness assumption, which states that the nearvy latent feature points are likely to share the same labels [Papernot and McDaniel 2018). This idea extends the classical smoothness assumption, which initially formulated in the input data space [Bezdek al., 1986), to the latent space of a well-optimized encoder trained with a classification objective. However, both assumptions are confined to scenarios where the data are drawn from the same distribution.
The proposed DVD provides a practical means to extend the latent smoothness assumption to cross- domain scenarios, which addresses the challenges posed by domain shifts. This extension is made possible by parameterizing a Gaussian prior using latent vicinities, which are exploited in both domains. Specifically, DVD leverages the similarity of latent vicinities between the source and target domains to propagate labels effectively across domains. The process comprises two key stages:
Noise-Adding Training:
During training, the Gaussian prior is parameterized using the mean and variance the source query's s latent vicinity. A neural network is trained to estimate the score function, which enables the diffusion of the source query's s label throughout its latent vicinity. The pre-trained source classifier further ensures alignment by matching the DVD-generated feature predictions with the source labels.
Removal Sampling: the adaptation phase, the Gaussian prior is parameterized using the latent vicinity of the target query data. This step generates source-like features that share similar latent vicinity to the target query, which effectively transfers discriminative knowledge to the target domain without requiring access source data.
Through this mechanism, DVD acts as a bridge between the source and target domains, transforming target features into source-like representations by using the underlying latent vicinities of both domains as the guiding keys. For DVD parameterized by the latent vicinities of two data domains, the following principle applies: "The source label can be encoded into its latent vicinity by latent diffusion model. Consequently, nearby target features sharing a similar latent vicinity can be transformed by the latent diffusion model into representations likely to share the same label."
A.4 Probabilistic Graphical Model
To formalize the DVD process, we represent it as probabilistic graphical model (Figure 5), which captures the stochastic dynamics of latent vicinity diffusion:
x
z
y
B
Z1
N (M
Figure 5: The directed graphical model illustrates how DVD enables explicit knowledge transfer by leveraging latent vicinity similarities between the two domains. Solid lines denote the direct causal relations between variables, which include the encoder, the latent diffusion, and the classifier. The dashed lines denote the stochastic sampling from a prior parameterized by the latent vicinity.
22

---

Encoding and Classification.
Encoder: The encoder po (2|x) maps input data latent features
Classifier: The classifier P= (ylz) predicts labels y based on the latent features z.
Latent Vicinity Sampling.
During training, the prior qu (zo [B) initializes the diffusion process using the k-NN features of the source latent vicinity.
During sampling. the prior qu (zo [BB is defined by the target latent vicinity.
The latent diffusion model qo qs (zo) propagates discriminative knowledge across latent vicinities, guided by Gaussian priors.
Diffusion Process.
Training Objective. During training, DVD optimizes the parameters & by minimizing the diver- gence between the source prior and the target posterior:
z (N of), z ~ (z(z)zz))
(12)
This process diffuses discriminative knowledge within the source latent vicinity.
Target Adaptation. For target adaptation, DVD leverages the latent vicinity similarity between domains:
zi ~ Niri, of), zi (z(z)
(13)
This process retrieves discriminative knowledge from the target latent vicinity and generates source- like representations. By progressively aligning target features with source-like representations using contrastive learning, DVD facilitates discriminative knowledge transfer across domains by defining its priors using the latent vicinities from both domains.
Markoy Chain Representation.
The stochastic sampling process can be expressed as:
z Zap .... Zar + zi.
(14)
This progression ensures that target features converge to source-like representations supported by label consistency within similar latent vicinities, which enables effective classification in the unlabeled target domain.
By integrating latent vicinity information through Gaussian priors into diffusson processes, as shown in the probabilistic graphical model, DVD offers theoretically and empirically robust solution to domain shifts during target adaptation without source data.
B
Ablation Studies
This section dissects the components of DVD through targeted ablation experiments:
Latent Vicinity Prior:
Compare our k-NN-based Gaussian prior against baseline and noise-based alternatives to validate its impact.
Feature Aggregation (SiLGA): Remove the local local-neighbor blending step to assess its role in contrastive alignment.
Diffusson vs. k-NN Mean Pooling: Replace the diffusion module with simple neighbor averaging evaluate whether DVD's gains stem from generative transport or from local aggregation alone.
Representational Power & Replay: Contrast DVD) on a smaller backbone with larger vanilla model and disable generative replay isolate the effect of our diffusion mechanism.
Vicinity Radius& & Shift Robustness: Vary A and inject synthetic noise to test the resilience of latent vicinity constructions across domain perturbations.
Effeet of Diffusion Step Size: Study how DVD's performance depends on the number of diffusion steps T. We vary T at both training and inference to evaluate whether the transport dynamics generalize across mismatched schedules, and we also examine very large T, which can cause over-smoothing and degrade discriminative sharpness.
k-NN Sensitivity: Vary vicinity hyperparameters (kaif k dif k.)) across datasets to evaluate the robustness neighborhood size.
Hyperparameter Sensitivity: Examine the effect of batch size, learning rate, and temperature, confirming DVD's robustness across a wide range of values.
23

---

B.1 Impact of the Latent Vicinity Prior
In DVD, we construct the initial diffusion state Zo by sampling from Gaussian prior parameterized by the k-NNs of each query in latent space. To evaluate the importance of this latent vicinity prior, we compare against four alternatives:
Baseline: Directly use the encoder output 2 as Zn-
Input White Noise: Inject Gaussian noise into the input image, then encode it to form 2o-
Latent White Noise: Inject white noise directly into in the latent space and use as
Centroid Feature Distribution: Use the mean vector of the latent k--NNs (16) Zo-
Table 8 shows the resulting target-domain classification accuracy (%) on VisDA-C 2017. We observe that simply adding noise at either the input or latent level actually degrades performance compared
to the baseline. In contrast, using the latent vicinity to formulate the prior substantially improves accuracy, and our full latent vicinity prior method outperforms all competing strategies. The results highlight the value of leveraging latent vicinities for diffusion.
Table 8: Ablation experiments to evaluate individual DVD module's contributions on VisDA-C 2017.
Method
plane
beyel
bus
car
horse
knife
meycl person
plant
sktbrd
train
truck
Avg.
Baseline
95.8
89.3
87.4
76.8
95.1
94.2
85.6
84.7
93.2
92.5
91.3
49.8
86.3
Input White Noise
94.5
88.5
84.2
75.3
93.6
92.4
82.7
81.8
91.5
91.9
90.2
50.8
84.8
Latent White Noise
80.1
84.2
82.5
70.5
90.4
88.6
78.9
82.2
88.0
90.2
91.8
45.5
81.1
Centroid Feature Distribution
97.4
92.4
89.6
78.2
97.7
95.8
89.8
85.4
94.9
93.2
90.4
49.6
87.9
Without SiLGA
97.5
92.7
89.2
78.7
97.1
95.2
86.6
85,4
93.8
92.7
92.3
50.9
87.7
DVD (Full)
98.4
92.1
83.9
83.6
98.1
96.5
92.1
82.9
97.0
95.2
92.6
54.6
88.9
B.2
Feature Aggregation for Positive Keys
To mitigate label mismatches between source and target, our method aggregates target k-NN features with those from DVD, forming more informative positive key for contrastive learning. In Table entry "Without SiLGA" removes this step. The resultant drop in performance confirms that incorporating local target neighbors helps align features more accurately.
B.3 Diffusion Mean Pooling: Isolating the Contribution of DVD
To isolate the contribution our latent diffusion module, we compare DVD against two simplified variants across three tasks: SFDA (VisDA-2017), supervised classification (CIFAR-10), and domain generalization (DomainNet).
Baseline (InfoNCE only): Standard contrastive learning, where each anchor is paired only with its
own data augmentation.
k-NN Mean Pooling (No diffusion): For each anchor 2,, the k = 6 nearest neighbors are identified and their mean vector is used as the positive key.
DVD (Ours): Full framework with the latent diffusion module generating positive samples from source-informed priors.
All methods were trained under identical conditions with the same backbone for each dataset.
Table Ablation study of Diffusion vs. A-NN Mean Pooling across three tasks. For VisDA-2017 (SFDA) and DomainNet (DG), higher accuracy indicates better performance (f), while for CIFAR-10 (supervised classification), lower top-1 error better (L).
Method
VisDA-2017 (SFDA 1)
CIFAR-I0 (SC
DomainNel (DG ?)
Baseline (InfoNCE only)
82.2
6.35
40.2
k--N Mean Pooling (No diffusion)
86.8
6.13
46.5
DVD (Ours)
88.9
5.92
50.8
Results.
The results, summarized in Table demonstrate consistent pattern across all settings:

---

On VisDA-2017 (SFDA), nearest-neighbor pooling already provides a significant enhancement (+4.6%) by smoothing noisy target representations. However, DVD adds an additional +2. 1 % by transporting features toward source-informed clusters, not just averaging within the target domain. This demonstrates the distinet role of our latent diffusion module in cross-domain alignment.
On CIFAR-10 supervised classification, the dataset has no domain shift, yet DVD still reduces error from 6.35% to 5.92%. The gain here smaller but significant, demonstrating that diffusion sharpens decision boundaries even when the training and testing distributions coincide, confirming its general utility beyond adaptation.
On DomainNet (domain generalization), the gap is most significant: mean pooling improves accuracy by +6.3%, but DVD delivers a further +4.3% increase. This suggests that under severe distribution shifts, local averaging alone is insufficient, diffusion's 's generative transport explicitly bridges disparate domains by aligning features to semantically consistent priors.
These results confirm that while neighbor aggregation captures local geometry, it cannot substitute for the latent diffusion. Our DVD consistently outperforms mean pooling by non-trivial margin, across closed-set SFDA, standard classification, and domain generalization. The consistency of this trend highlights that the latent diffusion module introduces a unique and transferable mechanism: enforces semantic alignment by pulling features toward class-consistent regions, rather than passively smoothing neighborhoods.
B.4
Representational Power and Generative Replay
To evaluate whether DVD's performance gains result from effective adaptation rather than increased model capacity, compare it to a larger source-only ResNet-152 model (60M parameters) on the Office-Home dataset. As shown in Table while larger models often generalize better from source data, DVD built on smaller ResNet-50 backbone (40M parameters in total), which achieves 73.7% accuracy-significnnty outperforms the ResNet-152 baseline (58.7%). This suggests that the performance gain comes from DVD's adaptation mechanism, not from model size. We also disable the generative replay. which is driven by our latent diffusion module. Without it, performance drops to 69.3%. confirming its importance. Since this replay operates entirely in the target latent space without accessing source data, DVD remains strictly source-free.
Table 10: Comparison of representational power and generative replay on Office-Home.
Method
Backbone
Accuracy (%)
Source-only
ResNet-152 (60M)
58.7
DVD (Ours)
ResNct-50 (40M)
73.7
DVD w/o replay
ResNet-50 (40M)
69.3
B.5 Vicinity Radius and Domain Shift Robustness
Our method relies on the assumption that the latent vicinities remain informative across domains, which is motivated by the smoothness principles used in semi-supervised learning, where nearby samples are assumed to share similar labels. In DVD, we extend this across domains using the k-nearest neighbors in the latent space to allow the model to synthesize diffusion trajectories guided by localized latent structure.
To evaluate how sensitive DVD is the choice of vicinity radius (k), we vary k from 25 on Office-Home (Ar-CI), As shown in Table [1] DVD maintains stable performance across all settings, with accuracy varying by less than 1%. This indicates that the latent vicinity construction is not brittle and does not require fine-tuning of k. We further evaluate robustness under synthetic domain shifts by adding Gaussian noise (with standard deviation σr) the latent features during adaptation. DVD consistently outperforms SHOT [Liang et al.. 2020] across all noise levels, which demonstrates that latent diffusion guided by local structure is more resilient to domain perturbations than purely discriminative alternatives. These results support the claim that latent vicinities generalize well across domains and that the vicinity-guide diffusion mechanism is robust to both hyperparameter changes and representation noise.
25

---

Table 11: Robustness of latent vicinity-guided diffusion Office-Home (Ar+CCI). Left: adaptation accuracy as the vicinity radius k varies; Right: accuracy under increasing synthetic domain noise o.
Vicinity Radius (k)
Domain Noise (a)
Method
15
20
25
0.1
0.3
0.5
1.0
1.5
SHOT [ Liang et al. 2020
57.1
55.4
52.8
51.9
50.3
DVD (Ours)
58.9
59.8
60.1
59.4
58.6
60.1
59.3
57.6
56.8
55.2
B.6 Effect of Diffusion Step Size
An important design choice our DVD is the number of diffusion steps T. During training, the drift network is optimized to transport features along discretization schedule of fixed length T, where each step corresponds to a specific fraction of the global transport vector. This schedule implicitly encodes how large each update should be and how the drift should behave across intermediate points.
natural question whether the model generalizes whether performance degrades when training and inference schedules are mismatched. This ablation is important because practice one might hope to shorten inference by using fewer steps, or conversely refine adaptation with more steps, without retraining the drift. However, using excessively large T can also detrimental: each step becomes very small, numerical errors accumulate, and features lose discriminative sharpness. Together, these analyses reveal how tightly the transport dynamics coupled to the training schedule and how sensitive performance is to step-size choices.
different number of steps at inference, or
Table 12: Adaptation accuracy on VisDA-2017 under matched and mismatched diffusion steps 7.
Train Steps
Inference Steps
Target Accuracy (%)
8
88.5
16
16
88.9
32
32
88.2
64
64
87.6
100
100
87.1
100
85.1
8
100
86.2
Table demonstrates two complementary effects:
Matched schedules. When 7 at inference matches the schedule used in training, DVD maintains strong accuracy across wide range of step sizes. The best result is achieved at T 16, while accuracy gradually declines as T grows very large (e.g., 87.1% % at T 100). This drop arises because the large step size accumulates numerical errors and cause over-smoothing, where features blur across class boundaries and lose discriminative sharpness.
Mismatched schedules. When differs between training and inference, performance drops substantially. For example, training with T 100 but testing with T = 8 reduces accuracy by 3.8%. This shows that the drift network does learn a scale-free update rule; instead, it tailors its dynamics to the specific discretization seen during training.
These findings emphasize that the step schedule an integral part of the transport mechanism learned by DVD. Unlike in generative diffusion where 7 can often be varied at inference, our discriminative setting requires consistent T to preserve semantic alignment. In practice, this means should be chosen once, balancing efficiency and accuracy, and kept fixed for both training and deployment.
B.7 Sensitivity Analysis of k-NN Vicinity Selection
To examine the robustness of DVD, we conduct a sensitivity analysis on the vicinity hyperparameters (kdif, 1.a kt), which determine how source and target neighborhoods are constructed in the latent space. This study spans three diverse benchmarks: VisDA-2017, Office-Home, and NemainNet. covering synthetic-to-real transfer, multi-domain adaptation, and large-scale heterogeneous shifts.
26

---

Table 13: Sensitivity analysis of vicinity parameters across VisDA-2017, Office Office-Home, and Domain-
Net.
(Rig ki kt))
VisDA-2017 (%)
Office-Home (%)
DomainNet (%)
(5, 3)
87.4
72.7
49.2
(10, 10,5)
88.5
73.4
49.7
(15,
88.9
73.8
50.8
(20, 10)
88.3
73.1
50.0
(25, 12)
88.0
72.8
49.6
(30,30, 15)
87.8
72.6
49.5
(40, 40, 20)
87.5
72.3
49.1
As shown in Table [3] DVD consistently maintains strong accuracy across different settings of the vicinity size on all three benchmarks. When the &c values are set too small, the neighborhood fails capture sufficient local structure, leading lack of context and a slight drop in accuracy. On the other hand, very large k values incorporate too many distant or less relevant features, which blur semantic distinctions and weaken class alignment. Between these extremes, moderate settings provide the best balance: they capture enough vicinal information to guide target adaptation, while still preserving the discriminative structure of the latent space.
B.8 Hyperparameter Study
We report the sensitivity DVD to three standard hyperparameters: batch size, learning rate, and temperature (r). Unless otherwise noted, experiments are conducted on VisDA-C 2017 with ResNet- 101 and fixed default configuration (batch size 128, learning rate 3 x 10-3 temperature 0.13). These defaults lie within typical ranges used in contrastive SFDA. We used minimal tuning.
Table 14: Ablation on RsDA-CC2177 varying one hyperparameter time while keeping the others fixed at the defaults (batch size 128, learning rate = 3 x 10 temperature 0.13). Reported values are target accuracy (%).
Batch Size
Learning Rate
Temperature
Value
Acc.
Value
Acc.
Value
Acc.
88.2
10-4
88.1
0.05
88.2
64
88.6
1 1 10
88.6
0.07
88.5
128
88.9
3 X 10
88.9
0.13
88.9
256
88.2
1 10-2
88.3
0.20
88.6
512
87.6
1 10-1
87.2
0.50
87.9
0.80
87.1
1.00
86.6
Results. As summarized in T4] DVD maintains high accuracy across wide ranges of batch size, learning rate, and temperature, indicating minimal sensitivity and little need for careful tuning. Unlike self-supervised frameworks such as SimCLR that typically rely on very large batches (e.g., 4,096) to obtain sufficient negatives, thereby favoring higher learning rates, DVD (and contrastive SFDA more broadly) pairs each target feature with a nource-informe keeps positives influential and supports fine- ane-gnrined alignment; correspondingly. smaller learning rates stabilize optimization. The temperature controls similarity sharpness: lower values (e.g.. 0.13) yield tighter and more discriminative clusters, which are beneficial for adaptation, whereas SSL with massive batches often adopts higher temperatures (0.5-1.0) to avoid collapse. Overall, the flat response across these hyperparameters suggests DVD does not depend on precise and method-specific tuning.
positive, so moderate batch (e.g., 128)
27

---

C
Additional Experiments
Beyond core SFDA evaluations, extend DVD's validation to diverse settings:
Fairness of Comparisons:
Benchmark DVD alongside UDA and PPDA methods under a unified protocol to ensure consistent evaluation.
Transformer-Basel Encoder: Evaluate DVD on ViT-B/16 demonstrate compatibility with modern architectures.
Source-Domain Augmentation: Test DVD-generated features at inference on source domains to measure in-domain gains.
Supervised Classification Ablation: Apply the latent vicinity prior to CIFARR/mageNet to confirm its benefits beyond domain adaptation.
ODA and PSDA Applicability: Evaluate DVD under open-set and partial-set SFDA settings to demonstrate adaptability beyond the closed-set SFDA protocol.
Initialization Robustness: Run multiple random seeds on Office-Home to compare variance against other SFDA methods.
C.1 Fairness of Comparisons: SFDA vs. UDA vs. PPDA
Our setting may appear different from other SFDA methods at first glance, but we emphasize that DVD remains strictly source-free during target adaptation (test time), fully adhering to the core requirement of the SFDA paradigm. During target adaptation, DVD does not access retain any source-domain data. From a privacy perspective, this means there no risk of source data leakage any stage of adaptation. In essence, DVD satisfies the most important requirement of SFDA: no source data is ever exposed during adaptation.
Nevertheless, setting represents relaxation of the strictest SFDA protocol: while most SFDA methods only transfer a source- peirre-peetraned classifier, DVD also trains an auxiliary latent diffusion module offline during the source vre-training. To avoid confusion, privacy-preserving domain adaptation (PPDA) method in the main paper. To ensure fairness, we extended our empirical evaluation cover all three settings:
explicitly define DVD
SFDA: No source data or auxiliary modules during adaptation; only the source-pretrained classifier is available.
PPDA: Offline training of auxiliary modules on source data, with zero access to raw source data during target adaptation.
UDA: Full access to labeled source data during adaptation.
Table 15: Comparison on VisDA-2017 with ResNet-101 backbone under different problem settings.
Method
Setting
Source Access Adaptation
Target Acc, (%)
CAM+SPLR (A-AT)
Soni and Dutta 2025
UDA
69.5
CDTrans Xu et al.
2022
UDA
86.9
TTN Lim
2025
UDA
85.3
GGF Zhuang al. 2024
UDA
77.6
SFADA [He 2024
PPDA
83.4
DM-SFDA [Chopra
2024
PPDA
87.5
DVD (Ours)
PPDA
88.9
Table 16: Comparison on Office-Home with ResNet-50 backbone under different problem settings.
Method
Setting
Source Access Adaptation
Target Acc, (%)
CDAN
Long al.
2018
UDA
70.0
TTN
Lim al. 2023
UDA
69.7
PGA
Phan et
20224
UDA
75.8
GGF
Zhuang
2024
UDA
73.6
SFADA [He el
PPDA
72.4
DM-SFDA CCoopa
2024
PPDA
72.6
DVD (Ours)
PPDA
73.7
28

---

While some existing works with auxiliary modules identify themselves as SFDA, we classify them as PPDA under our stricter protocol. In particular, SFADA [He et al. [2024] and DM-SFDA [Chopra et al. 2024 were originally labeled as SFDA, but both rely on auxiliary modules. We therefore reclassify them as PPDA more accurately reflect their setting.
As shown in Tables 15 and [6 while DM-SFDA pre-trains its diffusion module on large external datasets, our DVD relies solely on the provided source data yet still achieves higher accuracy (+1.4% on VisDA-2017). This demonstrates that the performance gains of DVD do not stem from additional pre-training or external resourees, but from the effectiveness of our vicinal diffusion mechanism. Furthermore, DVD consistently outperforms all PPDA baselines and even matches surpasses several recent UDA methods that have full access to labeled source data during adaptation. This is particularly significant: despite operating under stricter gttvac-preserving constraints, DVD narrows or even closes the gap to conventional UDA. Overall, these results indicate that the PPDA setting does not inherently entail weaker performance, but can in fact deliver competitive or superior outcomes when combined with principled generative modeling. Thus, DVD offers stronger balance between accuracy and privacy than both SFDA and UDA methods, pointing toward practical paradigm for real-world deployment where source data sharing is infeasible.
C.2 Transformer-Basse Encoder
To further demonstrate the effectiveness of our DVD across different backbone architectures, we conducted experiments using the ViT-B/16 vision transformer [Xu et al. 2022]. We evaluated DVD on the same SFDA benchmarks as ResNet backbone. The results, summarized in Tables [7 [8 and 19 show that our DVD consistently improves the performance of ViT-based models, which outperform existing SFDA methods built on vision transformers by large margin. We observe that DVD achieves greater improvements on more effective backbones (larger gains on VIT than on ResNet), which indicates that its knowledge storage and retrieval become more effective with better feature extractors. This is expected, as the quality of extracted features directly impacts the informativeness of the latent vicinity in our DVD.
Table 17: Comparison of SFDA methods on Office-3l using ViT-B/16.
Method
A++
A-
D-W
D-+A
W-D
W-+A
Avg.
ViT-B/16
90.8
90.4
76.8
98.2
76.4
100.0
88.8
CDTraas Xu al. 2022
97.0
96.7
81.1
99.0
81.9
100.0
92.6
TVT Yang 2023a
96.4
96.4
84,9
99.4
86.1
100.0
93.8
C-SFirans
Sanyal
2024
95.0
96.2
82.3
98.6
83.7
100.0
92.3
DVD (ViT-B716)
97.2 10.4
97.8 +0.3
86.5
1.2
0.1
87.2
:10.9
100.0
:00.0
94.7 ::.5
Table 18: Comparison of SFDA methods on Office- using ViT-B/16.
Method
+
+
Pr +
Rw +
Avg.
CI
Pr
Rw
Ar
Rw
Rw
Ar
CI
Pr
ViT-B/16
61.8
79.5
84.3
75.4
78.8
81.2
72.8
55.7
84.4
78.3
59.3
86.0
74,8
CDTraas Xu et al.
2022
68.8
85.0
86.9
81.5
87.1
87.3
79.6
63.3
88.2
82.0
66.0
90.6
80.5
TVT Yang el 10250
73.4
84.4
86.8
81.1
85.7
86.2
78.1
70.5
87.5
83.2
73.2
88.2
82.5
C-SF Faaras Sanyal el al. 2024
70.3
83.9
87.3
80.2
86.9
86.1
65.0
87.7
82.6
67.9
90.2
DVD (ViT-B/16)
70.4
89.0
87.4
84.5
73.9
87.8
85.3
79.5
90.5
004
+0.5
0..3
112
=0.5
+0.8
+0.5
00.6
+0.4
+0.8
112
+005
+066
Table 19: Comparison of SFDA methods on VisDA-2017 using VIT-B/16.
Method
Plane
Beyel
Bus
Car
Horse
Knife
Mcyle Person Plant Sktbrd Train Truck
Avg.
VIT-B/16
97.7
48.1
86.6
63.4
87.7
47.7
94.4
35.5
67.1
CDTrans Xu 2022
97.1
90.5
82.4
96.6
96.1
97.9
90.3
62.8
88.4
TVT Yang all 2023a
92.9
85.6
77.5
60.5
93.6
98.2
76.4
93.6
91.7
55.7
83.9
C-SFTrans (Sanyal ei al. 2024
96.2
90.1
85.0
74.2
95.0
95.7
93.1
86.9
96.8
87.3
88.7
61.5
88.3
DVD (VIT-B/16)
92.5
90.5
92.7
77.5
92.5
98.5
96.0
90.0
94.5
93.3
91.0
74.5
90.2
10.3
10.6
10.5
+0.8
40.4
10.5
1:0.8
1:04
10.2
1003
10.5
40.9
10.4
29

---

C.3 Source-Domain Feature Augmentation
In the main manuscript, we demonstrated how DVD benefits the souree provider by enhancing single-domain supervised classification through its role as latent augmentation module. Here, we conduct additional experiments to evaluate how DVD-generated features directly enhance testing performance on the source domain of domain adaptation datasets. This performance serves as an indicator of the effectiveness of using the latent vicinity and DVD as a repository for ground truth knowledge while addressing data privacy concerns.
Experimental Setup. We follow standard training protocols for pre-training on source domains in SFDA. Specifically, we use SGD with momentum (0.9), a weight decay of 5 x 10-4, -, a mini-batch size of 128, and train for 200 epochs. During testing, for each source data, we identify its latent k-NNs, generate an augmented feature using DVD, and pass it into the classifier to obtain the prediction score.
Results. Table 20 shows that utilizing features sampled from our DVD surprisingly outperforms the standard testing approach, which provides benefits source model providers. During training, parameterizing the Gaussian prior of DVD with the query's 's latent vicinity not only encodes its ground truth within that vicinity, but also enriches the feature space with discriminative knowledge. The results again demonstrate that DVD can directly benefit source model providers by acting as a latent augmentation tool.
Table 20: Classification accuracy (%) for source-domain testing using different feature sets. ResNet refers to the results obtained by using features encoded directly from ResNet-101.
Office-31
Office-Home
Home
Method
VisDA-C 2017
A
D
W
Ar
CI
Pr
Rw
ResNet
91.5
100.0
98.7
81.5
81.5
93.9
85.1
99.6
DVD
96.9
100.0
100.0
96.9
90.6
100.0
96.9
100.0
C.4 Latent Vicinity Prior Ablation for Supervised Classification
In this section, present additional ablation experiments single-domain supervised classification to evaluate the significance of the latent vicinity prior. The experiments aim to offer deeper insights and further guidance to the source model provider effectively using our DVD. Recall that compare against four alternatives:
Baseline: Directly use the encoder output as
Input White Noise: Inject Gaussian noise into the input image, then encode it to form Zo-
Latent White Noise: Inject white noise directly into in the latent space and use it as Zp.
Centroid Feature Distribution: Use the mean vector of the latent k-NNs (116) as 2g-
Table 21 presents the results of three supervised classification benchmark datasets, using three different backbone networks. Note that the classification errors are reported as percentages (with lower values indicating better performance) in this ablation. Similar to the ablation results in VisDA- C 2017, adding noise either the input or latent level slightly improves testing performance, but ultimately leads to worse performance compared to our baseline. In contrast, formulating the prior using the latent vicinity significantly reduces classification errors, with our full latent vicinity prior method outperforming all other strategies.
C.5 Applicability of DVD to Open-Set and Partial-Set SEDA
We also explore whether DVD can be extended beyond closed-set SFDA more challenging protocols such as Open-Set Domain Adaptation (OSDA) and Partial-Set Domain Adaptation (PDA).
Limitations of DVD. Our current implementation of DVD explicitly encodes source label consistency into the Gaussian prior for each source class. During adaptation, the drift module generates label- consistent positives, which is ideal for closed-set transfer but not directly suited for open- or partial-set tasks where:
The target domain may contain classes absent in the source (OSDA).
30

---

Table 21: Ablation experiments on supervised learning.
Method
CIFAR-I0
CIFAR-100
ImageNet
ResNet-18
7.07
22.74
31.46
Baseline
6.83
22.58
31.26
Input White Noise
6.92
22.70
31.29
Latent White Noise
6.94
22.54
31.34
Centroid Feature Distribution
6.68
22.27
31.18
DVD (Full)
6.52
22.01
31.06
ResNet-50
6.35
22.23
24.68
Baseline
6.20
22.47
24.55
Input White Noise
6.29
22.70
24.65
Latent White Noise
6.32
22.54
24.62
Centroid Feature Distribution
5.98
22.05
24.35
DVD (Full)
5.92
21.95
24.30
VGG-16
7.36
28.82
25.82
Baseline
7.04
27.37
25.63
Input White Noise
7.25
27.94
25.80
Latent White Noise
7.18
28.05
25.52
Centroid Feature Distribution)
6.71
26.96
25.16
DVD (Full)
6.42
26.58
25.02
The source domain may include classes not present in the target (PDA).
such cases, forcing all target samples to align with source classes risks negative transfer.
Saafiddnce-Trreshoolee DVD (DVD-CT), To extend DVD to OSDA and PDA, we propose simple yet effective variant: conffdence-turesholleed DVD (DVD-CT)) Specifically, after generating source-aligned features zDND with the latent diffusion module, we pass them through the frozen source classifier to obtain softmax probabilities p(y/z/vv). For each target sample, we compute the maximum softmax score maxy p <bvD)
Following Qu et al. 2024 filter samples using pre-defined confidence threshold
OSDA, samples with confidence below т are flagged as belonging to unknown classes, following prior practice in confidence-based rejection methods.
PDA, such low-confidence samples are excluded from adaptation to prevent negative transfer from target-private categories.
This mechanism leverages DVD's source-aligned transport to improve the reliability of softmax confidence discriminator between known and unknown target classes. Unlike clustering- based OSDA methods [Qu et al., 2023] or subspace-decomppsiiion approaches [Qu et 2024). DVD-CT requires extra modules and can be implemented as a lightweight post-processing step on top of our standard DVD.
Table 22: Comparison with source-free OSDA and PDA methods. For OSDA, results are reported as H-score (%); for PDA, results top-1 accuracy (%).
Method
OffficeHHome (OSDA) VisDA (OSDA) Office-Homa (PDA) VisDA (PDA)
USD [Jahan and Savakis 2024
61.6
57.8
UMAD [Lang of al
202
66.4
66.8
66.3
68.5
GLC Qu et al.
2023
69.8
72.5
72.5
76.2
LEAD Qu
2024
67.2
74.2
73.8
75.3
UMAD+LEAD
Qu etall
2024
67.6
75.0
78.1
GLC+LEAD [Qu al.
2024
70.0
73.1
73.2
75.5
DVD (Closed-set)
65.6
64.2
67.5
68.0
DVD-CT (Ours)
70.8 (+5.2)
73.4 (+9.2)
73.2 (+5.7)
75.9 (+7.9)
compares DVD and DVD-CT with recent OSDA/PDA baselines. While the original DVD struggles in open/partial scenarios due its closed-set assumption, DVD-CT achieves competitive improvements, e.g., enhancing VisDA (OSDA) H-score from 64.2% to 73,4%. These
Results. Table 22
31

---

results show that DVD, with minimal modifications, can be extended to OSDA and PDA. While not yet outperforming clustering- or subspace-based methods, DVD-CT demonstrates that source- consistent generative alignment can still contribute to the novel class discovery. In future work, we will explore explicit mechanisms such as adaptive thresholds, subspace decomposition, and unsupervised clustering to further narrow the performance gap.
C.6 Robustness to Initialization Seeds
We evaluated the robustness of our DVD against random initialization by conducting experiments on the Office-Home dataset with five different seeds, replicating the baseline methods with the same seeds for comparison. To illustrate the variance of results across different initializations. we present bar chart with error bars in Figure 6 The results show that DVD, benefiting from explicit guidance from source-like representations in contrastive clustering, more robust against random initialization than other SFDA methods.
80
60
Average Accuracy
ResNet-101
DaC
NRC+
DVD (Ours)
Ar Ar Pr Ar Rw Cr CI CI Rw Pr- Pr CI Pr Rw Rw
-r Rw
Rw - Pr Avg.
Figure 6: (Best viewed color.) Bar plots with error bars showing classification accuracy (%) across 5 initialization seeds on the Office-Ilome dataset.
32


Reviews:
[{"review_id": "yhz9pM1LmJ", "reviewer": "Reviewer_uKCa", "summary": "This paper proposes a new framework, Discriminative Vicinity Diffusion (DVD), for Privacy-Preserving Domain Adaptation. Unlike traditional methods, DVD allows knowledge transfer from a source-trained model to a target domain without exposing source data. It uses a latent diffusion module to align target domain features with source domain decision boundaries, leveraging Gaussian priors and drift-only diffusion in latent space. The proposed method shows superior performance on multiple SFDA benchmarks and improves domain generalization for previously unseen domains. Key contributions include the use of latent vicinity guidance for better feature alignment and the introduction of latent geometry aggregation (SiLGA), which further enhances the adaptation process.", "questions": "See the weakness.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n\t1. Compared to source-free domain adaptation, the paper explores a more realistic research objective, assisting domain adaptation tasks while preserving some level of data privacy.\n\n\t2. By introducing Latent Diffusion into domain adaptation, the paper enhances target domain performance and also benefits source domain performance.\n\nWeaknesses\n\n\t1. The training of the DVD part is entirely based on distributions generated from source domain data, but during adaptation, these are directly used to generate source-like samples from target domain samples. Can this really generate features close to the source domain distribution? Is there any direct evidence for this?\n\t\n\t2. Regarding the drift function, the paper claims that it is not an iterative optimization process, but in reality, it approximates the target through multiple updates, similar to residual-based regression. Why was this design chosen, and could its accumulated errors cause z_1^t and z_1 to differ significantly?\n\t\n\t3. Why does SiLGA guarantee that the method will still work effectively when the distribution difference between the target and source domains is very large? Intuitively, SiLGA averages the target domain sample with its neighboring samples.\n\t\n\t4. In Equation (9), is the neighbor set derived from the generated z_1^t recalculated from the target domain, or from the original target domain samples?  The paper only mentions that \"they are the top-k neighbors in B^T\".\n\t\n\t5. In the experimental setup, the learning rate is set to3×10^(-3), and the temperature is 0.13, whereas SimCLR uses a range of [0.1, 0.5, 1]. Was this particular hyperparameter setting specially tuned? Does temperature significantly affect the results？\n\t\n\t6. Why are there standard deviations in the results for DVD only, while the comparison methods do not have them?", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal, which has addressed most of my concerns. At this stage, I intend to maintain my original score and would be interested in seeing the feedback from the other reviewers.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback, as well as for recognizing the novelty and practical strengths of our work. Below, we explicitly address each concern and question.\n\n---\n## **W1: Generating source-like features from target domain samples**\n\nOur goal is **not to force the entire target feature distribution to mimic the source,** but rather **to adapt each target feature toward its most semantically relevant region in the latent space, ensuring alignment with the frozen source classifier’s decision boundaries**.\n\nDuring source training, supervised clustering groups features of the same class into compact latent clusters, which we model as Gaussians centered at each cluster mean and structured by local kNN relationships. This **vicinal prior** enables us to sample semantically meaningful variations.\n\nAlthough domain shift can cause the overall target feature space to become dispersed under a source encoder, **label-consistent samples in the target domain still remain relatively close in latent space**, a property supported by prior SFDA literature (e.g., [Iscen et al., 2019]; [Yang et al., 2023]) and visualized in **Figures 1** and **2** of our paper.\n\nOur adaptation procedure leverages each target sample’s kNNs to select the most relevant source-informed Gaussian prior, thereby generating features that preserve source-learned semantics and **relocate target samples toward the correct semantic clusters**, rather than simply matching global distributions.\n\n**Quantitative Evidence:** To make this effect explicit, we measured the **average Euclidean distance between target features and their nearest source cluster centroids on VisDA-2017**. After applying DVD, this distance decreased from 3.82 (before adaptation) to 1.55 (after adaptation), providing direct evidence that our method effectively aligns target features with source clusters. We will include the complete analysis and results in the camera-ready.\n\n---\n## **W2: Drift function and iterative approximation**\n\nThank you for raising this important question. Our drift mechanism combines theoretical grounding in SDE/diffusion models with geometric regularization by vicinal priors, ensuring stable and class-consistent adaptation without the need for iterative optimization.\n\n**Why use a drift function with sequential updates?**\nOur drift function applies a small, fixed number of deterministic updates (typically $T=8$), transporting features from a sampled latent prior toward a class-consistent source anchor. **Unlike iterative optimization or residual regression, these updates follow a pre-defined and regularized path, never a loss-minimizing loop**. We apply this update strategy directly from [Heitz et al., 2023], which is also standard in deep SDE and diffusion model literature, to ensure efficient and stable adaptation in latent space.\n\n**Regularization by vicinal priors:** Each drift step is explicitly regularized by the local geometry of the latent space, the Gaussian prior centered at a semantically consistent feature cluster. This regularization keeps every intermediate point within a region of label-consistent representations, preventing arbitrary drift or error accumulation.\n\n**Error accumulation and robustness:** Because $T$ is small and fixed, and each step is geometrically regularized, error accumulation between $z_1^t$ and $z_1$ is minimal. The transport path is controlled and always attracted back toward the correct semantic cluster.\n\n**Empirical evidence:** Our ablation study (**Appendix B.4**, **Table 9**) demonstrates that DVD remains robust across a range of drift steps and vicinity radii, with less than 1% accuracy variation, confirming that our update is stable and does not suffer from error drift.\n\n---\n## **W3: Effectiveness of SiLGA under large distribution shifts**\n\nThank you for raising this important question. When the domain gap is large, target features encoded by the source model become more dispersed and farther from source cluster centroids. In such cases, using only samples from the source-informed Gaussians as contrastive positive keys risks misalignment with the true target geometry.\n\n**SiLGA addresses this directly:** After sampling from the nearest source-informed Gaussian prior, **SiLGA blends this feature with the centroid of the target’s local k-NN, effectively relocating it closer to the actual target distribution**. This averaging aligns the transferred feature with the local structure of the target domain, reducing both semantic and geometric mismatch, an effect especially beneficial under large domain shifts.\n\n**Empirical evidence:** As shown in our ablation studies (**Appendix B.1/B.2**, **Table 7**), adding SiLGA yields consistent gains, particularly on challenging benchmarks with large sim-to-real gaps like VisDA-C. Performance drops when SiLGA is removed, confirming its role.\n\nWhile SiLGA is a practical solution, we recognize the potential for more principled methods, such as optimal mappings or optimal transport, and plan to explore these in future work. Thank you for highlighting this direction.\n\n---\n## **W4: Clarification on neighbors in Equation (9)**\n\nWe apologize for the confusion. To clarify, in Equation (9), the neighbor set is always recalculated using **the original target domain samples stored in the feature bank $B^T$**. Specifically, for each generated feature $z_1^t$, we search for its top-$k$ nearest neighbors **among the target features currently encoded by the target encoder, not among previously generated or adapted features**. This design is intentional: by anchoring the source-informed Gaussian prior to the up-to-date latent geometry of the target domain, we ensure that the aggregation step reflects the most relevant and accurate local structure. This keeps the adaptation well-grounded in the target distribution. We will make this explicit in the camera-ready. \n\n---\n## **W5: Hyperparameter for learning rate and temperature**\n\nOur hyperparameters, including learning rate (3e-3), batch size (128), and temperature (0.13), all fall within typical ranges reported in the existing contrastive SFDA literature (learning rate 1e-3 to 5e-3, batch size 64 to 256, temperature 0.1 to 0.2). We used only minimal tuning and did not specially optimize these values.\n\n**Why SSL like SimCLR needs larger batch size:**\nThe application of SimCLR in self-supervised learning requires very large batch sizes (e.g., 4096) to provide a diverse set of negative pairs for the contrastive objective. The large number of negatives helps the network learn general representations from scratch. This setting, in turn, requires a higher learning rate to ensure effective gradient updates and prevent slow convergence.\n\n**Contrast with SFDA:**\nIn contrast, our DVD (or general contrastive SFDA) focuses on adapting each target feature to its most relevant source-informed positive, rather than maximizing the number of negatives. Using a moderate batch size (128) ensures that positive keys remain influential, enabling fine-grained semantic alignment. Thus, a smaller learning rate is optimal for optimization stability in this setting.\n\n**Role and rationale of temperature:**\nThe temperature controls the sharpness of the similarity distribution in the contrastive loss. Lower temperatures (like our 0.13) make the model more sensitive to mismatches between query and positive, promoting tighter and more discriminative clusters, an effect that is particularly important for domain adaptation. In SSL with massive batches, higher temperatures (0.5 to 1.0) prevent the network from collapsing to trivial solutions.\n\n**Empirical robustness:**\nAs shown in Tables 1 to 3, DVD achieves consistently high accuracy across a broad range of batch sizes, learning rates, and temperatures. These results demonstrate that DVD is robust to hyperparameter choices and does not depend on careful or method-specific tuning. All selected values fall well within the standard ranges for contrastive SFDA, and were chosen with only minimal tuning.\n\n**Table 1. Ablation for batch size on VisDA-2017. The learning rate and temperature are fixed at $3 \\times 10^{-3}$ and $0.13$, respectively.**\n\n|Batch Size|Target Accuracy (%)|\n|:-:|:-:|\n|32|88.2|\n|64|88.6|\n|**128**|**88.9**|\n|256|88.2|\n|512|87.6|\n\n**Table 2. Ablation for learning rate on VisDA-2017. The batch size and temperature are fixed at $128$ and $0.13$, respectively.**\n\n|Learning Rate|Target Accuracy (%)|\n|:-:|:-:|\n|1×10-4|88.1|\n|1×10-3|88.6|\n|**3×10-3**|**88.9**|\n|1×10-2|88.3|\n|1×10-1|87.2|\n\n**Table 3. Ablation for temperature on VisDA-2017. The batch size and learning rate are fixed at $128$ and $3 \\times 10^{-3}$, respectively.**\n\n|Temperature|Target Accuracy (%)|\n|:-:|:-:|\n|0.05|88.2|\n|0.07|88.5|\n|**0.13**|**88.9**|\n|0.2|88.6|\n|0.5|87.9|\n|0.8|87.1|\n|1.0|86.6|\n\n---\n## **W6: Standard deviations reported only for DVD**\n\nWe appreciate the reviewer’s attention to experimental rigor. We report standard deviations for DVD by running experiments with multiple random seeds to explicitly show robustness. For baseline methods, we cited official results from their original papers, which often do not include standard deviations. For fairness, we plan to reproduce these baselines using our protocol and provide their standard deviations in the camera-ready. We hope this transparency is seen as a strength of our work, not a limitation, and kindly ask for the chance to include these results after the rebuttal due to time constraint.\n\n---\n## **Reference**\n**[Iscen et.al., 2019]** Iscen et.al., Label propagation for deep semi-supervised learning, CVPR 2019.\\\n**[Yang et.al., 2023]** Yang et.al., Trust your good friends: Source- free domain adaptation by reciprocal neighborhood clustering, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023.\\\n**[Heitz et.al., 2023]** Heitz et.al., Iterative α-(de) blending: A minimalist deterministic diffusion model, SIGGRAPH 2023."}, {"review_id": "EGllu6AKl9", "reviewer": "Reviewer_gTCo", "summary": "This paper proposes Discriminative Vicinity Diffusion (DVD), a novel latent diffusion model framework for privacy-preserving domain adaptation (SFDA). By designing an auxiliary drift-only latent diffusion module, the method transfers discriminative knowledge without exposing raw source data. Experiments on multiple benchmarks demonstrate that DVD achieves state-of-the-art results in SFDA, while also enhancing performance in supervised classification and domain generalization tasks.", "questions": "My main concern about this paper is the fairness of its setting and experiments.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper proposes a novel privacy-preserving domain adaptation framework, which involves a latent vicinity guidance for explicit cross-domain knowledge transfer. \n\n2.\tThe experiments are comprehensive, and the proposed method achieves SOTA results across multiple benchmarks.\n\n3.\tThe paper is well-structured, and the motivation is clear.\n\n\nWeaknesses:\n\n1.\tThe problem setting is unclear. The authors propose an LDM-based framework for privacy-preserving domain adaptation, which is similar to source-free domain adaptation, but requires access to source data to train an additional diffusion module. This deviates significantly from the “source-free” assumption and aligns more with traditional DA settings.\n\n2.\tThe experiments are not fair. The experiments primarily compare against SFDA methods, which is not a fair evaluation. Could the authors also compare their method with SOTA DA and Privacy-Preserving DA methods?\n\n3.\tThe analysis of vicinity selection is limited. Since the choice of k-NN parameters significantly affects performance, it is recommended to conduct a more thorough sensitivity analysis across diverse datasets.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal, and all of my concerns have been addressed. I encourage the authors to include the discussions in the final vision. Therefore, I decide to update my score to Accept.", "rebuttal": "We sincerely thank the reviewer for their valuable feedback, and for recognizing the novelty and clarity of our work. We greatly appreciate your positive assessment of our work’s comprehensive experiments, the well-structured presentation, and the motivation for explicit cross-domain knowledge transfe in the source-free setting.\n\nIn response to your thoughtful concerns, we have undertaken a thorough re-examination of our protocol and literature positioning. Below, we explicitly address each point. We hope these clarifications fully resolve your concerns. If you have further questions or suggestions during the rebuttal, we would be grateful for your feedback and happy to provide additional details.\n\n---\n## **W1: Clarifying the Source‐Free Assumption**\n\nWe agree that our setting may appear different from other SFDA methods at first glance. We would like to emphasize, however, that DVD remains strictly source-free during target adaptation (test time), fully adhering to the core requirement of the SFDA paradigm. **During target adaptation, DVD never accesses or retains any source-domain data**. From a privacy perspective, this means there is no risk of source data leakage at any stage of adaptation. In essence, DVD satisfies the most important requirement of SFDA: **no source data is ever exposed during adaptation**. \n\nOur DVD, offline training of an auxiliary module using source data, followed by strictly source-free deployment, is now explicitly accepted as standard practice in recent literature. For example, HCL [Li et al., 2022] includes an auxiliary module alongside the source model during offline training. Likewise, DM-SFDA [ICML’24] pre-trains a diffusion model on large-scale external data and then fine-tunes it on the source domain, while still claiming alignment with the SFDA setting. By contrast, our DVD trains the auxiliary diffusion module strictly offline on the provided source data, without relying on external pre-trained models. This means DVD actually aligns even more rigorously with the intended SFDA assumptions as recognized by recent literature. \n\nNevertheless, we fully acknowledge that our setting represents a relaxation of the strictest SFDA protocol: while most SFDA methods transfer only a source-pretrained classifier for target adaptation, our DVD uses source-domain data once, offline, to train an auxiliary latent diffusion module, which is then fixed and never updated or exposed to source data again during adaptation. To avoid any confusion, we will explicitly define DVD as a privacy-preserving domain adaptation (PPDA) and clarify that most SFDA methods do not use auxiliary modules, as the reviewer rightly points out, in the camera-ready.\n\n---\n## **W2: Fairness of Comparisons (SFDA vs. UDA vs. PPDA)**\n\nWe thank the reviewer for highlighting the importance of comprehensive benchmarking. To address this, we have extended our empirical evaluation to include:\n\n- **SFDA** (Source-Free DA): no source data or auxiliary modules during adaptation; only the source-pretrained classifier is available.\n- **PPDA** (Privacy-Preserving DA): offline training of auxiliary modules on source data, with zero access to raw source data during adaptation.\n- **UDA** (Unsupervised DA): full access to labeled source data during adaptation (upper-bound reference).\n\nAlthough some prior works self-identify as SFDA, we reclassify them under our protocol:\n\n|Method|Original Label|Reclassified as PPDA|\n|:-:|:-:|:-:|\n|SFADA [PR’24]|SFDA|✓|\n|DM-SFDA [ICML’24]|SFDA|✓|\n\n**Table 1: VisDA-2017 (ResNet-101).**\n\n|Method|Setting|Source Access at Adaptation|Target Acc. (%)|\n|:-:|:-:|:-:|:-:|\n|CAM+SPLR (A-AT) [AAAI’25]|UDA|✓|69.5|\n|CDTrans [ICLR’22]|UDA|✓|86.9|\n|TTN [ICLR’23]|UDA|✓|85.3|\n|GGF [ICLR’24]|UDA|✓|77.6|\n|SFADA [PR’24]|PPDA|✗|83.4|\n|DM-SFDA [ICML’24]|PPDA|✗|87.5|\n|**DVD (Ours)**|PPDA|✗|**88.9**|\n\n**Table 2: Office-Home (ResNet-50).**\n\n|Method|Setting|Source Access at Adaptation|Target Acc. (%)|\n|:-:|:-:|:-:|:-:|\n|CDAN [ICML’18]|UDA|✓|70.0|\n|TTN [ICLR’23]|UDA|✓|69.7|\n|PGA [NeurIPS’24]|UDA|✓|**75.8**|\n|GGF [ICLR’24]|UDA|✓|73.6|\n|SFADA [PR’24]|PPDA|✗|72.4|\n|DM-SFDA [ICML’24]|PPDA|✗|72.6|\n|**DVD (Ours)**|PPDA|✗|73.7|\n\nNotably, while DM-SFDA [ICML’24] marks itself as source-free, it actually uses a diffusion model pre-trained on massive external datasets before source fine-tuning, thus introducing extra data and compute advantages. In contrast, our DVD trains its auxiliary diffusion module solely on the provided source data, without leveraging any external resources, yet achieves higher accuracy (**+1.4% on VisDA-2017, 88.9% vs. 87.5%**) than DM-SFDA and outperforms all other PPDA baselines.\n\nImportantly, DVD also **matches or surpasses the performance of recent UDA methods** that have full access to source data during adaptation. This highlights that DVD’s privacy-preserving setting does not compromise competitive performance and, in many cases, closes the gap to supervised adaptation.\n\nOur updated evaluation directly addresses the concern about fair comparison by including recent SOTA DA and PPDA baselines, alongside SFDA comparison in the main paper.\n\n---\n## **W3: Sensitivity Analysis of k-NN Vicinity Selection**\n\nThank you for highlighting the importance of the ablation of the number of nearest neighbors. To directly address your concern, as suggested, we conducted additional sensitivity analyses on the kNN hyperparameters ($k_s^{dif}$, $k_t^{dif}$, $k_t$) across diverse SFDA benchmarks, including **VisDA-2017**, **Office-Home**, and **DomainNet**. \n\n**Table 3: Sensitivity analysis of k-NN vicinity selection.**\n\n|$(k_s^{dif},k_t^{dif},k_t)$|VisDA-2017(%)|Office-Home(%)|DomainNet(%)|\n|:-:|:-:|:-:|:-:|\n|(5,5,3)|87.4|72.7|49.2|\n|(10,10,5)|88.5|73.4|49.7|\n|**(15,15,6)**|**88.9**|**73.8**|**50.8**|\n|(20,20,10)|88.3|73.1|50.0|\n|(25,25,12)|88.0|72.8|49.6|\n|(30,30,15)|87.8|72.6|49.5|\n|(40,40,20)|87.5|72.3|49.1|\n\nThe results above demonstrate that the performance of our DVD is **robust to the choice of k-NN vicinity parameters** across the three benchmarks **VisDA-2017**, **Office-Home**, and **DomainNet**. As we gradually increase the number of neighbors used to define each vicinity, DVD maintains high accuracy on all datasets. This means the model’s adaptation performance **does not depend heavily on fine-tuning $k$**, making DVD practical for real-world scenarios where the optimal $k$ is not known a priori. The results also demonstrate that a moderate neighborhood size provides a good balance: it includes enough neighbors to capture vicinal structure, but avoids introducing too much noise from distant or unrelated samples. When the $k$ values are set very small, the model lacks sufficient vicinal context from the latent space, leading to performance drop. Conversely, very large $k$ may average out meaningful vicinal semantics, making the generated features less discriminative and causing a slight decline.\n\nWe will include these additional sensitivity analyses in the camera-ready to fully address your concern. Your feedback has been invaluable in improving the completeness of our study. If you feel that further ablation experiments could strengthen the work, we would greatly appreciate your further suggestions during the rebuttal, and we are eager to incorporate any additional analysis you recommend. \n\n---\n## **Reference**\n**[Li et al., 2022]** Li et al., Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data, CVPR 2022.\\\n**[ICML'24]** Chopra et.al., Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation, ICML 2024.\\\n**[AAAI'25]** Soni et.al., Toward Improving Robustness and Accuracy in Unsupervised Domain Ddaptation, AAAI 2025.\\\n**[ICLR'22]** Xu et.al., CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation, ICLR 2022.\\\n**[ICLR'23]** Lim et.al., TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation, ICLR 2023\\\n**[ICLR'24]** Zhuang et.al., Gradual Domain Adaptation via Gradient Flow, ICLR 2024\\\n**[NeurIPS'24]** Phan et.al., Enhancing Domain Adaptation through Prompt Gradient Alignment, NeurIPS 2024\\\n**[PR'24]** He at.al., Source-Free Domain Adaptation with Unrestricted Source Hypothesis, Pattern Recognition 2024."}, {"review_id": "zCb4eNg6dd", "reviewer": "Reviewer_xMQp", "summary": "This paper introduces Discriminative Vicinity Diffusion (DVD), a novel framework for source-free domain adaptation (SFDA). The core idea is to repurpose a latent diffusion model (LDM) not for generation, but for discriminative knowledge transfer in a privacy-respecting manner. This is achieved by a latent vicinity guidance, using latent k-nearest neighbors to parameterize Gaussian priors more effectively than simple noise addition. The authors demonstrate state-of-the-art performance on several standard SFDA benchmarks and further show the versatility of their method in improving performance on supervised classification and domain generalization tasks.", "questions": "1.\tThe method uses a specific deterministic drift-only diffusion model. Although this approach seems effective, a brief explanation of why it was chosen over other diffusion schemes would be helpful. Is the deterministic, non-stochastic design crucial for effectively storing and retrieving discriminative cues?\n\n2.\tI recommend moving the critical ablation study into the main paper. This would provide direct evidence for the design choices.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper proposes a more practical SFDA variant, termed privacy-preserving domain adaptation. It is a justified relaxation of the classical SFDA problem by adding an auxiliary module without exposing the source data. This well-motivated method is proved to be more effective for SFDA tasks.\n\n2.\tThis paper repurposes LDMs for the discriminative knowledge transfer task and proposes a well-executed and logically sound implementation. Leveraging diffusion models’ strength in modeling data distributions, the core contribution is using the diffusion process to explicitly store and transfer decision boundaries.\n\n3.\tExperiments on three SFDA benchmarks consistently demonstrates its SOTA performance over previous methods. This paper also provides the results on the supervised classification and domain generalization tasks, further enhancing its effectiveness. \n\n4.\tThis paper is well-written and easy to follow.\n\n\nWeaknesses:\n\n1.\tThe second term in Eq. (6) seems to be incorrect. The network is designed to predict the difference between the corresponding state at the timestep t and z_0, rather than consistently predicting the difference between z_1 and z_0.\n\n2.\tThe choice of the length of timestep T should be further discussed. With the ODE Settings used by the authors, would it impair the performance with larger timestep? For example, training the diffusion model with 100 steps and inferencing with 8 steps.\n\n3.\tThe proposed method operates under a setting that allows for sharing an auxiliary module in addition to the source classifier. This is a relaxation of the SFDA setting and should be clearly positioned as such. This makes direct comparisons with methods designed for the stricter setting potentially problematic, which should be acknowledged.", "was_revised": true, "final_justification": "I chose to maintain my rating because I really did not find any strong reason to change it.", "rebuttal": "We sincerely thank the reviewer for their careful and thoughtful evaluation. We greatly appreciate your recognition of our key contributions, including the practical privacy-preserving variant of SFDA, our novel repurposing of latent diffusion models for discriminative knowledge transfer, and the well-executed and logically sound implementation. \n\nYour constructive suggestions and questions are highly valued. In the detailed responses below, we address each weakness and question explicitly. We are committed to implementing all necessary revisions in the camera-ready.\n\n---\n## **W1: The second term in Eq. (6)**\n\nThank you so much for your careful examination and for highlighting this technical point. We appreciate the opportunity to clarify both the mathematical motivation and the implementation. Our formulation of Eq. (6) follows exactly the way in [Heitz et al., 2023]. If you have any further questions or require additional details, please feel free to follow up during the rebuttal, or refer directly to [Heitz et al., 2023].\n\nAfter careful review, we confirm that there is no mistake in Eq. (6) and that our formulation is fully consistent with both our DVD’s motivation and the reference in [Heitz et al., 2023]. \n\nTo be explicit, both our implementation and [Heitz et al., 2023] are indeed **designed to predict the global difference $z_1-z_0$ at every position along the transport path, not the local difference between $z_{\\alpha_t}$ and $z_0$ at each step**. This is by design, and it matches both our method’s motivation and the reference method.\n\n### **Clarification** \n\nDuring training, for every sampled pair $(z_0, z_1)$ and blend parameter $\\alpha_t$, we construct $z_{\\alpha_t} = (1-\\alpha_t)*z_{0} + \\alpha_{t}*z_{1}$. The drift network $D(z_{\\alpha_t}, \\alpha_t)$ is then trained to output the global difference $(z_1 - z_0)$, which always points from the starting point toward the semantically meaningful features (source or target features). When advancing from $z_{\\alpha_t}$ to $z_{\\alpha_{t+1}}$, the sample is updated as:\n\n$z_{\\alpha_{t+1}} = z_{\\alpha_t} + (\\alpha_{t+1} - \\alpha_t) \\cdot D(z_{\\alpha_t}, \\alpha_t),$\n\nwhich **guarantees that the model moves precisely along the straight line between $z_0$ and $z_1$**, with step size determined by the change in $\\alpha$. **In other words, the drift network learns a direction field that always points along the full transport vector**. The actual local increment at each step is scaled by the step size $(\\alpha_{t+1} - \\alpha_t)$.\n\n### **Why not Predict The Local Difference** \n\nPredicting the global difference is more stable and efficient because the direction from the starting point to the endpoint remains constant for each pair. **The local increment at each step can then be easily obtained by scaling this drift prediction according to the change in $\\alpha$, i.e., $(\\alpha_{t+1} - \\alpha_t) \\cdot D(z_{\\alpha_t}, \\alpha_t)$**. This formulation is the key novelty of [Heitz et al., 2023], is explicitly justified in their Algorithm 3, and is precisely what our Eq. (6) implements.\n\nWe will make this clarification explicit in the camera-ready, ensuring that the training target, update rule, and their relation to the global and local differences are all transparent to readers.\n\n---\n## **W2: Length of Timestep and Mismatched Steps for Train/Inference**\n\nWe thank the reviewer for this important question about the choice of diffusion step length $T$. We will highlight these findings and including the mismatched-step ablation in the camera-ready.\n\nOur sensitivity analysis (main paper **Sec. 4.4**, **Fig. 3b**, and **Table 4**) demonstrates that DVD achieves stable and high accuracy for moderate $T$ (8 to 32), with our default $T=8$ providing a strong balance of efficiency and effectiveness.\n\n### **Why Not use Large $T$**\n\nIn generative diffusion models, larger $T$ is often beneficial as it allows for better refinement and noise averaging. However, in our discriminative adaptation setting, increasing $T$ makes each update step smaller and introduces many more steps, where even tiny numerical errors can accumulate. This accumulation leads to **over-smoothing where features become less sharply clustered, class boundaries blur, and discriminative power is reduced**. This effect is similar to setting the contrastive temperature too high: semantic clusters are less distinct, and features can drift from their correct regions. Thus, unlike in generation, excessively large $T$ is actually detrimental for SFDA tasks, causing both accuracy loss and unnecessary computation. More ablation results shown in **Table 1** also confirm this.\n\n### **Mismatched step ablation**\n\nAs suggested by the reviewer, we also conducted ablations where training and inference use different $T$. Results below show clear drops in accuracy with mismatched steps, confirming that the transport dynamics learned by DVD are tuned to a specific step schedule. Matching $T$ between training and inference is very important for optimal adaptation.\n\n**Table 1: Adaptation accuracy with varying and mismatched diffusion steps $T$ on VisDA-2017.**\n\n| Train Diffusion Steps (T) | Test Steps (T) | Target Accuracy (%) |\n|:-:|:-:|:-:|\n| 8  | 8 | 88.5 |\n| **16**| **16**| **88.9**|\n| 32 | 32 | 88.2|\n| 64 | 64 | 87.6 |\n| 100 | 100 | 87.1|\n| 100 | 8| 85.1|\n| 8| 100 | 86.2|\n\n---\n## **W3: Positioning DVD and Fairness of Comparisons**\n\nWe agree that our DVD that allows for an auxiliary module in addition to the source classifier, represents a relaxation of the strictest SFDA setting. We fully acknowledge that this should be clearly stated and transparently discussed in our paper.\n\nWe wish to clarify that our DVD protocol, offline training of an auxiliary module on source data, followed by strictly source-free deployment, is now explicitly recognized as standard practice in recent SFDA literature. For instance, HCL [Li et al., 2022] includes an auxiliary module during offline training, and DM-SFDA [ICML’24] pre-trains a diffusion model on large-scale external data before fine-tuning on the source domain, yet both still claim SFDA alignment. In contrast, DVD’s auxiliary diffusion module is trained exclusively on the provided source data, with no external pre-trained models involved. In this regard, DVD actually adheres even more closely to the core SFDA principles. \n\n**Due to space constraints, we kindly refer the reviewer to our response to Reviewer gTCo W2, where we provide detailed positioning, comprehensive comparisons with UDA and PPDA methods, and further discussion. If additional clarification or specific results are needed, we would be happy to provide them during the rebuttal.**\n\n---\n## **Q1: Explanation of Why Deterministic Diffusion**\n\nThank you for raising this important point about our use of a deterministic and drift-only diffusion model. We specifically chose this diffusion model for our latent diffusion module to guarantee **efficiency**, **stability**, and **interpretability** when transferring discriminative cues. These qualities are important for our DVD, where preserving semantic consistency is important.\n\n### **Why Deterministic Diffusion**\n\n- **Stability and efficiency:** Unlike stochastic (noise-injected) diffusion, the deterministic drift-only design **avoids the risk of propagating random errors, feature drift, or instability in the latent space**. This is especially important when working with compact and semantically structured representations like our DVD, where even small errors can misalign class boundaries.\n- **Theoretical support:** As shown in [Heitz et al., 2023], a deterministic drift field is mathematically sufficient to move features along well-defined and label-consistent paths in latent space, enabling reliable transport of discriminative knowledge without unnecessary complexity.\n- **Empirical validation:** Our ablations (**Appendix Table 9**) show that adding stochasticity not only increases computational cost, but reduces accuracy in the discriminative adaptation tasks. The deterministic scheme consistently yields robust and class-consistent features and superior adaptation performance.\n\nThus, the non-stochastic design is both important and effective for **storing and retrieving discriminative cues in DVD**, providing a principled balance of accuracy, interpretability, and efficiency. We will highlight this design rationale and its supporting evidence more explicitly in the camera-ready.\n\n---\n## **Q2: Moving Critical Ablations into Main Paper**\n\nThank you for emphasizing the importance of our ablation studies. In response, we will move key ablation results from the appendix into the main paper to enhance transparency. As suggested, we will also add the new ablation on mismatched diffusion steps between training and inference, which highlights the need for consistent transport dynamics in achieving robust adaptation. Additional visual and quantitative evidence will be included in the camera-ready to further support our design choices.\n\n---\n## **Reference**\n**[Heitz et.al., 2023]** Heitz et.al., Iterative α-(de) blending: A minimalist deterministic diffusion model, SIGGRAPH 2023.\\\n**[Li et al., 2022]** Li et al., Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data, CVPR 2022.\\\n**[ICML'24]** Chopra et.al., Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation, ICML 2024."}, {"review_id": "a0Cr44uPg9", "reviewer": "Reviewer_tWhM", "summary": "This paper explores an interesting approach that leverages latent diffusion to enhance both model adaptation and generalization. Specifically, the authors propose the Discriminative Vicinity Diffusion (DVD) framework, which integrates an auxiliary latent diffusion module to guide noisy samples toward a label-consistent feature space. The method is empirically validated through extensive experiments across multiple settings.", "questions": "This paper presents a novel and interesting framework. However, the empirical evaluation lacks necessary ablations to convincingly isolate the effect of the auxiliary diffusion module. My initial rating is borderline accept. If the aforementioned concerns are thoroughly addressed, I would be willing to raise my score.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n-\tThis paper is well-organized and easy to follow.\n-\tThis DVD framework is somewhat novel to me, with the guidance of latent kNN to perform feature manipulation.\n-\tExtensive experiments verify the effectiveness of this framework.\n\n**Weakness:**\n- It is unclear whether the observed performance improvements primarily stem from the auxiliary diffusion module or from the combination of local kNN consistency and contrastive learning. Both kNN-based consistency and contrastive learning are well-established and effective techniques in source-free model adaptation. A more informative ablation study should be conducted in which the auxiliary diffusion module is removed, and positive samples are instead directly obtained through mean pooling over top-k neighbors.\n- The same concern about attribution extends to standard classification and single-domain generalization experiments. Since the DVD framework leverages top-k neighbor information, it would be more rigorous to apply the same kNN-based strategy to the baseline models. This would help isolate the contribution of the diffusion module from the benefits introduced by incorporating nearest-neighbor priors.\n- As shown in Figure 3, there is a performance decline as the number of diffusion steps increases. This trend warrants further analysis. The authors should clarify why more iterations of diffusion negatively affect performance, especially given that iterative refinement is generally assumed to be beneficial in diffusion-based frameworks.\n\n**Minor Weakness:**\n- The latency and inference time reported in Table 6 raise concerns. It is unclear why the values for NRC++, SHOT, and DVD differ significantly, given that all methods use the same network architecture at inference time. While differences in the adaptation phase are expected, inference-time efficiency should remain consistent unless additional computation is explicitly introduced.", "was_revised": true, "final_justification": "Thank you for your detailed and timely response. All of my concerns have been adequately addressed. I am pleased to see that the proposed DVD framework can also be extended to more challenging scenarios with appropriate modifications. Based on this clarification, I am updating my score to Accept.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and for recognizing our DVD framework as novel and interesting, as well as for noting the paper’s clarity and extensive experimental validation. Below, we explicitly address each concern as requested, and will include these improvements in the camera-ready.\n\n---\n## **W1: Ablation on Diffusion Module vs. Mean Pooling**\n\nThank you for suggesting this important ablation. To incorporate your suggestion, we conducted the ablation study as follows:\n\n### **Implementation Details**\n\n**DVD:** Our full framework employs the latent diffusion module to generate positive samples by sampling from the source-informed Gaussian prior. We then use the InfoNCE contrastive loss to align each target feature (anchor) with its corresponding diffusion-generated positive.\n\n**kNN Mean Pooling W/O Diffusion:** **The diffusion module is removed**. For each anchor $z_t$, the $k_t=6$ nearest neighbors (cosine similarity) are identified, and their mean vector is used as the positive key: $z_\\text{pos} = \\frac{1}{k_t} \\sum_{i=1}^{k_t} z_i$, where $z_i$ denotes the latent feature of each neighbor. InfoNCE loss is applied using $z_t$ as anchor, $z_\\text{pos}$ as positive, and all other batch features as negatives.\n\n**Baseline (InfoNCE only)**: Standard contrastive learning; the positive key for each anchor is its own data augmentation, without kNN or diffusion.\n\nAll methods were evaluated under identical training conditions, using a ResNet-101 backbone, a learning rate of 3e-3, and a batch size of $128$.\n\n### **Empirical Results**\n\nThe results below demonstrate that while kNN-based augmentation improves over the baseline, the addition of the diffusion module provides a significant further enhancement. This confirms that DVD’s auxiliary diffusion is not merely duplicating the benefits of kNN or contrastive learning alone.\n\n**Table 1. Ablation: Diffusion vs. Mean Pooling vs. Baseline on VisDA-2017.**\n\n|Method|Target Accuracy (%)|\n|:-:|:-:|\n|Baseline (InfoNCE only)|82.2|\n|kNN Mean Pooling|86.8|\n|**DVD (Ours)**|**88.9**|\n\n---\n## **W2: Extending W1 Ablation to Supervised Classification and Domain Generalization**\n\nAs suggested by the reviewer, we have extended the ablation experiments suggested in **Major W1** on the supervised classification and domain generalization:\n\n### **Implementation Details**\n\n**kNN Mean Pooling W/O Diffusion:**\nFollowing the protocol described in **Sections 4.2** and **4.3** of our paper, at test time, we identify the latent k-nearest neighbors for each sample and compute their mean feature vector. This mean-pooled vector is then passed to the classifier for prediction. No diffusion module or additional augmentation is used in this variant.\n\n### **Empirical Results**\n\nTables 2 and 3 demonstrate that kNN mean pooling improves both supervised classification and domain generalization performance, confirming the value of leveraging local neighbor information. Importantly, incorporating our latent diffusion module (full DVD) consistently yields further improvements, reducing CIFAR-10 top-1 error to 5.92% and increasing DomainNet accuracy to 50.8%. These results isolate and highlight the unique contribution of our latent diffusion module beyond nearest-neighbor augmentation, demonstrating robust and transferable improvements across settings.\n\n**Table 2. Supervised Classification Ablation on CIFAR-10 using ResNet-50.**\n\n|Method|Top-1 Error (%)|\n|:-|:-:|\n|ResNet-50|6.35|\n|+ kNN Mean Pooling (no diffusion)|6.13|\n|+ **DVD (ours)**|**5.92**|\n\n**Table 3. Domain Generalization Ablation on DomainNet using ResNet-50.**\n\n|Method|Target Accuracy (%)|\n|:-|:-:|\n|ResNet-50|40.2|\n|+ kNN Mean Pooling (no diffusion)|46.5|\n|+ **DVD (ours)**|**50.8**|\n\n---\n## **W3: Declining Performance with Increased Diffusion Steps**\n\nThank you for highlighting the performance decline as diffusion steps increase in Fig. 3. Our sensitivity analysis (**Appendix Fig. 3b**) demonstrates that DVD performs robustly when the number of steps T is in a **moderate range** (8 to 32), but as $T$ becomes too large, accuracy gradually decreases. \n\n###  **Why Performance Drop with Large $T$** \n\nIn diffusion models for generative modeling, increasing $T$ often improves sample quality because each step incrementally refines the reconstruction, and stochastic noise can be averaged out over many steps. However, our DVD is designed for discriminative feature adaptation, not image generation. \n\nIn our setting, increasing $T$ makes each update step smaller, which could theoretically offer finer control. However, unlike in generative diffusion models where larger $T$ often improves sample quality, too many small steps in discriminative adaptation introduce cumulative numerical errors and can **over-smooth the feature representations**. This over-smoothing effect is analogous to setting the contrastive loss temperature too high: clusters become less distinct, and features may drift away from their intended class-consistent regions. This leads to reduced performance in SFDA tasks, even though higher contrastive temperature is favored for self-supervised representation learning (see our response to **Reviewer uKCa W5**). As a result, the discriminative power of the model decreases and adaptation becomes less effective. Additionally, the extra computational cost of large $T$ brings no practical benefit in our context and may even undermine reliability. Our expanded ablation results shown in **Table 4** also confirm this interpretation.\n\n### **More Thorough Ablation on $T$**\n\nWe have conducted a more comprehensive ablation analysis on the diffusion step $T$, as presented in **Table 4**. In addition, inspired by your observation, we also evaluated the effect of using **mismatched $T$** values between training and inference. The results show that performance drops notably when the step counts are inconsistent, indicating that the learned transport dynamics are optimized for a consistent diffusion step schedule. This evidence further justifies our choice of using a matched and moderate $T$ during both training and adaptation. We will include these new analyses and clarify their motivation in the camera-ready.\n\n**Table 4: Adaptation accuracy with varying and mismatched diffusion steps $T$ on VisDA-2017.**\n\n| Train Diffusion Steps (T) | Test Steps (T) | Target Accuracy (%) |\n|:-:|:-:|:-:|\n| 8  | 8 | 88.5 |\n| **16**| **16**| **88.9**|\n| 32 | 32 | 88.2|\n| 64 | 64 | 87.6 |\n| 100 | 100 | 87.1|\n| 100 | 8| 85.1|\n| 8| 100 | 86.2|\n\n---\n## **Minor W1: Latency and Inference Time**\n\nThank you for raising the question regarding the differences in inference time across methods in Table 6. We acknowledge that all methods use the same ResNet-101 backbone, and at first glance, inference latency should be similar.\n\n**Why DaC has higher inference time:** DaC [Zhang et al., 2022] exhibits much higher inference time due to its memory bank and prototype-based computations at inference. Unlike NRC++, SHOT, and DVD, which require only a single forward pass with fixed weights, DaC maintains a momentum-updated memory bank and dynamically computes class centroids and prototypes for each input, even during inference. This results in frequent memory access and additional similarity computations, significantly increasing overhead, especially on large datasets like VisDA. In contrast, NRC++, SHOT, and DVD do not perform these extra operations, so their inference time is lower and more consistent.\n\n**Updated Inference-Time for NRC++, SHOT, and DVD:** We greatly appreciate the reviewer’s observation regarding small inference-time differences among NRC++, SHOT, and DVD. These minor discrepancies likely arose from measurement noise and limited repeats in our initial report. To ensure accuracy, we repeated inference-time measurements five times for each method under identical conditions. After averaging, NRC++, SHOT, and DVD exhibit virtually identical inference speeds, as shown in the **updated Table 6** (mean ± std over 5 runs):\n\n**Updated Table 6:**\n\n|Method|Inference (ms)|FPS|Latency (ms)|\n|:-:|:-:|:-:|:-:|\n|DaC| 86.2 ± 1.2| 19.8 ± 0.5| 50.6 ± 0.8|\n|NRC++| 51.0 ± 0.7| 38.4 ± 0.4| 26.0 ± 0.6|\n|SHOT| 50.8 ± 0.6| 38.6 ± 0.3| 25.9 ± 0.4|\n|DVD| 51.1 ± 0.8| 38.2 ± 0.2| 26.1 ± 0.5|\n\nWe thank the reviewer for catching this and prompting us to report more robust numbers. We will clarify both the technical differences and these updated results in the camera-ready version for full transparency.\n\n---\n## **Reference**\n\n**[Zhang et al., 2022]** Zhang et.al., Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning, NeurIPS 2022."}]

Author rebuttal:

Meta-review: This paper proposes Discriminative Vicinity Diffusion (DVD) for privacy-preserving domain adaptation. Reviewers found the method well motivated, technically interesting, and empirically strong, with the discussion converging toward acceptance.

In their final remarks, the authors committed to incorporating all suggested revisions: improving clarity (e.g., clearer evidence for feature alignment, ablations on timestep choices), broadening evaluation with additional baselines and sensitivity analyses, extending applicability through variants such as DVD-AND, and enhancing technical transparency (refined hyperparameter discussion, moving key ablations to the main paper, and reporting standard deviations). These commitments directly address the reviewers’ concerns and will further strengthen the paper.
Decision: Accept (poster)