Controlled Visual Hallucination via Thalamus-Driven Decoupling Network for Domain Adaptation of Black-Box Predictors
Authors: Yuwu Lu
Chunzhi Liu
Venue: NeurIPS 2025 poster ()
Keywords: Domain Adaptation of Black-Box Predictors
Thalamus-Driven Decoupling Network
Visual Hallucination
Primary area: neuroscience_and_cognitive_science
TL;DR: A method for domain adaptation of black-box predictors based on Thalamus-Driven Decoupling Network

Abstract: Domain Adaptation of Black-box Predictors (DABP) transfers knowledge from a labeled source domain to an unlabeled target domain, without requiring access to either source data or source model. Common practices of DABP leverage reliable samples to suppress negative information about unreliable samples. However, there are still some problems: i) Excessive attention to reliable sample aggregation leads to premature overfitting; ii) Valuable information in unreliable samples is often overlooked. To address them, we propose a novel spatial learning approach, called Controlled Visual Hallucination via Thalamus-driven Decoupling Network (CVH-TDN). Specifically, CVH-TDN is the first work that introduces the thalamus-driven decoupling network in the visual task, relying on its connection with hallucination to control the direction of sample generation in feature space. CVH-TDN is composed of Hallucination Generation (HG), Hallucination Alignment (HA), and Hallucination Calibration (HC), aiming to explore the spatial relationship information between samples and hallucinations. Extensive experiments confirm that CVH-TDN achieves SOTA performance on four standard benchmarks.

Controlled Visual Hallucination via Thalamus-Driven Decoupling Network for Domain Adaptation of Black-Box Predictors
Yuwu Lu School of Artificial Intelligence South China Normal University Foshan, Guangdong. China 1uyumu20080163.cm com
Chunzhi Liu School of Artificial Intelligence South China Normal University Foshan, Guangdong, China 20230243238m, schu. edu. cn
Abstract
Domain Adaptation of Black-box Predictors (DABP) transfers knowledge from a labeled source domain to unlabeled target domain, without requiring access either source data or source model. Common practices of DABP leverage reliable samples to suppress negative information about unreliable samples. However, there are still some problems: i) Excessive aftention to reliable sample aggregation leads premature overfitting: ii) Valuable information in unreliable samples is often overlooked. To address them, we propose novel spatial learning approach, called Controlled Visual Hallucination via Thalamux-drin TDN). Specifically, CVH-TDN is the first work that introduces the thalamus-driven decoupling network the visual task, relying on its connection with hallueina- tion to control the direction of sample generation in feature space. CVH-TDN is composed of Hallucination Generation (HG), Hallucination Aligmment (HA), and Hallucination Calibration (HC), aiming to explore the spatial relationship information between samples and ballucinations. Extensive experiments confirm that CVH-TDN achieves SOTA performance on four standard benchmarks.
Decoupling Network (CVH-
Introduction
Traditional unsupervised domain adaptation (UDA) adapts models trained on fully labeled source domain to unlabeled target domain, aiming to alleviate the constraints of data collection and annotation in training deep neural networks 155) However, the application of UDA techniques limited in some scenarios like personal medical records, where access to source data is restricted due to arivacy-preserving policies. To solve this problem, source-free domain adaptation (SFDA) methods [68] were introduced recently, which assume the availability of only unlabeled target domain data and a pre-trained source model during adaptation. Although SFDA methods reduce the possibility of privacy leaks by using the pre-trained source model instead of source data, certain generation techniques like 99 10 can potentially reconstruct the source data by learning from the source model. Compared to UDA and SFDA settings, domain adaptation of black-box predictors (DABP) provides better data privacy protection with more flexible portability, which adapts a model using only the unlabeled target data and black-box predictor trained the source domain, egg., API service in the cloud 11. As demonstrated in Appendix A, we present a detailed exposition of the respective processes and the differences among UDA, SFDA, and DABP.
Corresponding author.
Both authors contributed equally this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recently, some researchers have paid attention to DABP, and the proposed methods 8
1 15 have made
modest contributions. Early works [16418] employ self-trainin learning with the pseudo-labelling techniques to subdivide target noisy labels and se- lect reliable samples for model train- ing. These self-training methods ini- tially absorb useful target data and adapt well, but eventually forget tar- get knowledge due to the accumula- tion of pseudo-label noise, leading model collapse [13]. Recent proposed methods [1T [4] 15 distill the source knowledge and then fine-tune the dis- tilled model to further fit the target domain through two-step (distilla- tion fine-tumne tine-tuning) process. The core of the two-step process is to use high- reliability samples or the most reliable estimates to suppress negative infor- mation about low-reliability samples during training. However, this sup- pression leads some problems: Excessive focus aggregating reli- able samples or estimates leads to premature overfitting, thereby limiting the model's ability to generalize effectively to the target data; ii) For low-reliability samples estimates, only simple processing is applied, without any effective methods specifically to extract their useful information. Figure illustrates the decline in model generalization due to overfitting in the previous methods. In addition, the fine-tuning process involves additional computational costs compared to pscudo-labeling methods.
ResNer: 66.9%
DINE: 78.4%
DINE (FT): 77.9%
BETA: 78.2%
BETA (FT): 78.5%
CVH-TDN: 88.7%
Figure 1: The feature visualizations on the Office-Home (A- using t-SNE [2]. The points represent target samples and the different colors correspond to their ground-trut classes. Previous DABP methods struggle with handling few low-reliability samples effectively. Even if the fine- tuning (FT) process is used to force the aggregation of low- reliability samples, the model still struggles to judge, leading to confusion between similar classes and a risk of degraded or stagnant performance. CVH-TDN solves this problem and improves model generalization without the FT process.
In the fields of pathology, Thalamus-driven Decoupling Network (TDN) 199 observes that abnormal synchronous activity between the thalamus and cortex, when combined with memory confusion, causes functional thalamic hallucinations and cognitive impairments. Therefore, TDN analyzes human responses from lesion studies and summarizes an explanatory framework for synchronous hallucinatory attention. In neuroscience, [20] further interprets this hallucination framework from the perspective of human attention, dividing the process of TDN into three parts: hallucination manifestations, human responses, and lesion treatment. In DABP, the black- predictor resembles a cognitively impaired "person" who possesses only partial knowledge and is prone to errors. As result, the target model's learning process is inevitably influenced by this noisy guidance from the black-box predictor. To address this issue, we draw inspiration from neuroscience, where such conditions are categorized as hallucination disorders: situations in which useful and interfering information are blended, preventing the brain to make reliable decisions.
Inspired by TDN, we propose a novel spatial learning method, named Controlled Visual Hallucination via Thalamus-drive Decoupling Network (CVH-TDN), to address the existing DABP problems. CVH-TDN builds on TDN's exploration of cognition inaittionnhllluciaationn relationships drive the explo- ration of the relationship between samples and hallucinations in the feature space. As shown in Figure 2, drawing the connection between hallucination and brain cognition revealed by TDN, we divide CVH-TDN into three main modules: Hallucination Generation (HG), Hallucination Alignment (HA), and Hallucination Calibration (HC). In HG, considering that the generation of hallucinations is random process that deepens with cognition, we design a progressive feature masking mechanism to control the direction of the generation process in feature space. HG generates some controlled masking in the areas specified by our strategy. With the increase of the prediction reliability, the masking degree increases in the specific areas of images that the model focuses on. In HA, based on the connections between real cognition and hallucination, we use contrast learning to explore the spatial relationship between the target sample and the corresponding generated image. In HC, we design a hierarchical calibration method that aggregates low-reliability samples with similar features

---

toward the nearest highhreliabilltyy samples and performs hierarchical ex- tension based on the feature spatial similarity among samples. The cali- bration corresponds to the cognition- driven hallucination understanding, where random but controllable halluci- nations are guided by cognitively sig- nificant objects for calibration. Exper- imental results demonstrate that our approach significantly outperforms the previous state-of-the-art methods on four benchmark datasets.
What Doject
B Prodetors
And Whhtl Are Those Oieect? Cheee API Reruuot
Domain
Cen You To W'hradher The bekgreet la Rallesia)
Rei Retiatle Cata
Hellciraton
Aignenent
OWN
Detr
Con You Find the
Key
Troa""
Ferser
Our contributions can be summarized as follows: (1) We observe the weak- nesses of existing DABP methods and address them by proposing novel method, called CVH-TDN, that sig- nificantly enhances the reasoning abil- ity of model and discrimination capac- ity of classes. (2) Based on the re- lationship between hallucination and cognition, CVH-TDN contains three parts: Hallucination Generation, Hal- lucination Alignment, and Hallucina- tion Calibration, aiming to explore the spatial relationships between samples and hallucinations. (3) We perform extensive experiments to verify the effectiveness of CVH-TDS and the results show that it achieves SOTA performance on four benchmarks. Moreover, we demonstrate the effectiveness of each component of our model and discuss the relationship between them through large number of ablation experiments.
nallucaation
Coneaadnn
Aigment
Exploration
Figure 2: Conceptual figure of CVH-TDN. The black-box predictors resemble agents with prior knowledge but lack the ability perform targeted discrimination. HG controls mask formation by modeling the location where hallucinations are pathologically generated, driven by the key cognitive impair- ments observed in TDN. HA improves feature discrimination by simulating how humans deal with cognitive impairments. HC draws on neurotherapeutic principles to guide unreliable data through reasoning using reliable feature representations.
2 Related Works
Domain Adaptation of Black-box Predictors. With the development of generative adversarial networks (GAN), GAN-based white-box attacks becoming more and more mature recently, requiring source models from which original source data can be recovered [33 In early work, [18] proposed the DABP, which has almost no risk of privacy leakage. Recent method DINE [1] has found that a two-step (distillation & fine-tuning) process can encourage source-target class alignment, which distills knowledge and then fine-tunes the distilled model adapt to the target distribution. DINE implicitly leverages the confirmed reliable knowledge from the distillation stage to cluster unreliable samples towards reliable ones during the fine-tuning process. Based on the two-step process, [14] establishes a threshold to explicitly divide the high- and low-reliability subdomains and further aligns the distribution discrepancy between the two subdomains. Based on [T]] 14] [1] introduces neighborhood clustering to prevent the model from forgetting minority classes. Another line of work [13 mimics human memory to mitigate the "forgetting" problem. This work distinguishes useful information and irrelevant information, primarily focusing on learning useful samples while handling irrelevant ones on the fly. Although these methods have contributed, they use high-reliability samples to suppress the negative information from low-reliability samples, which leads to the effective information of low-reliability samples being ignored. Different from the above methods, CVH-TDN employs the sample spatial similarity to replace the targeted sample information suppression. More recent DA works and the advantages of CVH-TDN are presented in Appendix A.
Visual Hallucination and Thalamus-Driven Decoupling Network. In computer vision, some works 23 have found that deep features correspondence information can be discovered by jointly learning the input data and hallucinatory data with signature features. However, they are all unidirectional exploration techniques in which hallucinations are only used to reconstruct the original samples. In the absence of hallucinatory data, revealing connections for deep feature correspondences among samples unexplored. In the field of neuroscience, Thalamus-driven Decoupling Network (TDN) [19] reveals the process and principle of human hallucinations, and mimics the effects of
3

---

Fasture Shappning
Anoual
Italiccraatin
Hollicianion Aligimen
Enrgo
Hallucination
Rundens Comnections
Ceneration
Immadiate Cuntesi
Enstional Cenmel
(b) Hnlluctission Caliiratoo
Dynanic Disiskan
Targee Domenn
Samples
Profiction
OA
Sanpla
Extemeter
Clyastter
Fullrcination Aliaument
OA
Seeyles
Orienting
itallucttatton Cullbcation
Modalating
(0) Genecation Algoritim Differences
OFjject
hateype
Claa
Acaptive S. Stoige
00 Dacitzzaation
Theke specttic
Sminny
Poslive
Intiilice
Preducton
Prediction
Alpolliiss
Hollacination Cernevticn
Figure 3: Overview of the completed training period of CVH-TDN. The processing flow of each module is indicated by arrows of the same color. Before training, we upload target data and obtain hard predictions from the cloud API. Feature extractor controls masking direction by evaluating knowledge from sharpened images, and the difference between the hallucination generation and other methods shown in (c). the hallucination alignment, we reduce the difference between samples and corresponding hallucinations by bidirectional alignment, as shown in (a). As shown in (b), we adopt hierarchical learning with the dynamic division for different types of samples based on spatial information in the hallucination calibration.
hallucinations when losing consciousness by interrupting information flow. Based TDN, we fill the gap in uncovering the correspondence deep features between hallucination and reality without hallucinated images. Unlike generation- or augmentation-ban based methods 26), our method generates controlled hallucination masking in the model interest area of the image without additional networks, which avoids model degradation by controlling the moving direction of the cluster center on the feature space.
3 Proposed Method
The whole overview and the entire process of our method are illustrated in Figure 3. First, we assume that a target dataset D, {xx)), containing N, unlabeled samples, where x, E X and X is the image set. According the DABP setting, we upload the target dataset D, black-box predictors (i.e., cloud API services) and obtain the hard predictions P. of target samples from source model N, stored in the predictors, where the source model trained by source dataset D3 {(=}, , )) = consisting of N₂ labeled samples. After obtaining the hard predictions, we introduce target data x and initialize them into controlled generation version data x9, With the increase of model training, xº are transformed from the unmasked or slightly masked version images to the masked version images in specific areas that the model focuses on. Our model Ma is parameterized by O and composed of two components, namely, a feature extractor and a prediction classifier C. The feature extractor is denoted by F X Z E K where K is the dimension of the feature space, and Z is the K-dimensional transitional output from the bottleneck layer. The prediction classifier is denoted by C: Z + Y R.V, where N is the number of predicted classes and Y is the classification prediction output.
3.1 Hallucination Generation
As revealed by TDN, the process of hallucination generation is jointly determined by controllable factors (e.g., context, emotion, memory, etc.) and uncontrollable factors (random connections). To generate exploitable hallucinations, we utilize the local areas and their adjacent context information to generate some specific masking blocks by evaluating the global image and comparing it with each local area. The strategy introduces a semi-automatic generation way that combines targeted generation and random masking. Specifically, we first set the masking block to square of patch size

---

and the class location weights oz to connect the k-th dimensional feature with the n-th class. The class location weights oi can be defined as follows:
2 h X E j=1 dz
o
(1)
where h is the number obtained through dividing the image height H by the masking height i [1,.... ": W is the image width and divided by the masking width p to obtain the number w, J E [L, the index of the feature space dimension K, 2t is the k-th dimensional feature of the transitional outputs Z, [1,-.. K]; n is the class index and y represents the classification prediction for the n-th class, [l., N.. And SY the gradient information obtained by backpropagation of n-th 2
P ]; the patch size of masking block is set to max (min (H, W' /32, 8); k is
class on the k-th dimensional feature.
Then, we perform a weighted combination of forward activation maps. process is defined follows:
The attention computing
ΣΣΣ ).0). h W K
Att.,- = max (aanh ΣΣ of 1.g 7
where ag 2,, is the linear combination of maps; mar (Tanh(-)) is used to suppress negative pixels from other categories that the model is not interested in. Z is the K-dimensional transitional output from the bottleneck layer, which stores =f, as a matrix, where each =x, is a scalar the corresponding position Z. If there is no max (Tanh())), attention values sometimes highlight not only the desired class but also other elements, and thus perform worse in feature localization. We only randomly generate masking where the model is of interest, and the masking generated formula is as follows:
Mask sptsktt11-1-1pppp = [Atti > & Ug > r2). ip+1:(+1)pp
where r is hyperparameter that controls the arousal ratio between regions of model interest and non-interest; ús,j is a random number, W4 U (0, 1); 22 is the uncontrollable masking ratio and set to 0.5. Subsequently, we simply sharpen the features of data r and combine the sharpened data i with the generated masks, as follows:
x = Mask @ x
where O is the element-wise multiplication symbol.
In the initial training, due to the lack of prior model training, the model attention for the entire image distributed and unreliable. At this stage, according to Eq. (3), 2 represent the unmasked or slightly masked hallucination version of the image. With increased model discrimination, the generation of 2 controlled by the model attention generates controllable masking in the area of model interest.
3.2 Hallucination Alignment
In the hallucination state model, predictions of sensory-level data assume crucial roles. These predic- tions reflect prior information about what could occur in the upcoming hallucination environment: i.e, what is expected to be seen in that context. The focus of this work is how to efficiently explore sensory level data (i.e., the model's immediate predictions and importance assessment of the original samples). For this, we first introduce permanent memory storage M. which stores the global predicted probabilities and mimics a memory structure that stores and processes sensory-level data. With each iteration, M overwrites the probabilities for the current batch samples. The storage M is the key container of our method, which controls the movement direction of the samples on the feature space by acquiring the sample state immediately and can be expressed as follows:
M {[Hier(x)),
(offmarr(Y)))
{(M4)) MM. MMMMM(
(5)
where (k) corresponds to the k-th subset in M; norm(-) denotes the normalization function: norm((z Zz) represents the spatial features of the samples, which is used to calculate the similarity between the samples; Hier(.) is the hierarchical strategy to judge the current sample is reliable R or unreliable UR. The judgment process can be expressed as:
Dif(Y.) TopK (Y, 1) TopK (Y..2.
(6)
5

---

TI N Dif = No Σ Topk (M,
(7)
where TopK(:,) (::j) defines the j-th largest value in the corresponding tensor; the threshold of 7 is to filter samples that are at the boundary of similar classes in the feature space, and the threshold of 72 to filter samples that are judged unreliable the current batch. Therefore,
Hier(a.) RI, if (V) vi &Tpp (M/)), 1) T2
(8)
UR., otherwise
where À is a hyperparameter that controls the ratio between reliable and unreliable samples in mini-batch. The combination of Ti and 72 constitutes our hierarchical strategy.
Then, design bidirectional weight module w to assign different weights by pairing features with different similarities between the original image and the corresponding generated image. The module can be defined as follows:
== cxp((og(sim((M))
norm(2*)))
where 2 is denoted the transitional output of the generated image x"; sim(-) is the operation of calculating cosine similarity. The hallucination alignment deploys contrastive learning to explore the spatial relationship, its process of calculating the loss is as follows:
forward -w x log(sim( (M) (YY))))
(10)
Chack = - logo - sim(M (M) (YY)))) ,(²)))
(11)
CHA = f fforwrrd + HLA
(12)
where y9 is the prediction of the generated data x and (.) is the Softmax. LH is composed of
bidirectional losses: H C generated version in the feature space; LH calibrates coowardd and reduces the noise caused by HA
ffowwrdd encourages samples whose features are similar to be aligned with the
common features to prevent overfitting of the model.
3.3 Hallucination Calibration
Previous DABP methods (13155] maintained fixed momentum parameter when updating the adaptive label smoothing [II. which is considered to perform better than the original model output Y and be expressed as follows:
s= AduLS(pf(h)) beginning
.N
S(x,) N
(13)
(u((.,)) + (1 - Y)Y,, otherwise
where AdaLS(.) the calculation of the adaptive label smoothing: pf-(*) is the hard prediction for (n)-th class of the i-th sample used as input from the black-box predictors before the training; f is the momentum hyperparameter that holds a fixed value in these methods. In this study, we found that this operation has some problems: i) The initial hard predictions from the black-box predictors are more robust than the training model, but this operation pays more attention to the early training model, resulting in the accumulated noise in the early training and affecting the subsequent training: ii) During the mid-to-end training stages, the model discriminant ability and the output reliability are enhanced, but the update variable is fixed, which limits the efficiency of extracting feature knowledge. Different [.g. begining
from them, we replace y with dynamic variable ji {{
Iter x (ū Jü--M p)) /Iter ftta_total, other to update S(z,) that is similar to the brain cognitive process, where Iter is the number of iterations and T. to initialize information extraction. The process details are described in Appendix F.
Then, the hallucination calibration is divided into two parts: i) Hierarchical sample calibration (Hsc) and ii) Common feature exclusion (Cfe). The former is calibrate samples with similar signature features, and the latter is to separate ambiguous samples with similar common features. In Hsc, the high-reliability samples are distinguished by the previous hierarchical strategy to cluster the low-reliability samples that are close in spatial distance. The computation can be expressed as:
Ni
M J.j E [1,2, ...,).. Φ).
Hsc H
(MI(M [H[M 2 w)
(14)

---

Table 1: Accuracies (%) on the Office-Home using the ResNet-50 backbone. denotes the hard task
whose source-only accuracy below 65%. H.Mean denotes the average accuracy of hard tasks. The poppperforning DABP methods are highlighted in bold.
Method
DABP
A-C
A-+P
A-R
C-A
C-+
C-R
P-
P-C
P-R
R-+A
R-+C
R-+P
Mean
H.Mean
Sourcc-only
44.1
66.9
74.2
54.5
63.3
41.2
73.2
46.7
77.5
60.6
50.4
CDAN
X
52.0
68.6
76.1
58.0
70.3
70.2
58.6
50.2
77.6
72.2
59.3
81.9
66.3
58.1
81.4
77.1
56.4
83.0
HMA
60.6
79.1
82.9
68.9
77.5
79.3
69.1
55.9
83.5
74.6
62.3
84.4
73.2
65.7
LNL-KL
49.0
71.5
77.1
59.0
68.7
72.9
56.4
46,9
76.6
66.2
52.3
79.1
64.6
55.4
HD-SHOT
48.6
72.8
77.0
60.7
70.0
73.2
56.6
47.0
76.7
67.5
52.6
80.2
65.3
55.9
SHOT
72.9
79.4
58.1
DivideMix
51.7
74.7
78.5
61.8
72.4
73.3
59.8
48.0
82.9
68.0
$6.4
81.6
67.4
58.4
DINE
52.2
78.4
81.3
65.3
76.6
78.7
62.7
49.6
82.2
69.8
84.2
69.7
60.4
BiMem
54.5
78.8
81.4
66.7
78.7
79.6
65,9
53.6
82.3
73.6
57.8
84,9
71.5
62.9
BETA
57.2
78.5
82.1
68.0
78.6
79.7
67.5
56.0
83.0
71.9
58.9
84,2
72.1
64.4
RFC
57.4
82.8
67.0
80.6
57.8
82.8
72.8
72.9
65.1
SEAL
58.5
81.4
84.7
71.7
80.4
82.1
72.2
54.3
86.0
76.2
60.6
86.3
74.5
66.3
where D& L (-) is the Kullback-Leibler divergence; sim2(:,0) is the index set of the first-O similar samples by calculating cosine similarity.
In Cfe, we introduce regularization loss to enhance the difference in the representation of different samples in the feature space, and the generated hallucination image is forcibly aligned with the original image to guide the separation of different classes of samples. The regularization loss can be expressed as:
Nu NR Σ
Na Σ
CCfe
(V.log((Y))) + ((
M j=1.8
(15)
where Y, arg Y; is hard label predicted by the training model, NR is the number of high- reliability samples, and 7' is the transpose operation. This process requires the gradient to be updated separately, so that the class differences can be improved through common feature information without being affected by the information of individual samples. The Lис is denoted by:
LиC Lasc Cffe LHC
(16)
According to I1, the training model is affected by negative information from low-reliability samples in early training, which leads to the decline of model discrimination ability. In the hallucination calibration, we make low-reliability samples unaffected by CCfe initially, and use Has to explore the spatial similarity among samples. In the middle of training, driven by the hallucination alignment, all type-U / It samples will become samples of type I, where NR = N,. L discovers the spatial information with deep feature correspondence by jointly learning hallucinations and similar samples with signature features.
3.4 Overall Objective Function
Following previous methods MT the conditional self self-regularization is adopted as the task- specific loss to complete the DABP task. The task-specific loss be expressed as:
Leask E x ( (Y))))x))
(17)
Therefore, the overall objective loss of CVH-TDN can be expressed as:
Lrota! LH A LuG + Lrask-
(18)
To explain why CVH-TDN works effectively and why it contributes DABP, derive error bound through theoretical analysis in Appendix B. The whole training process is shown in Appendix C.
Experiments
Implementation Details. We evaluated our method on four standard benchmark datasets, including Office-31 (27). Office-Home (28), VisDA-17 (29), and DomainNet [30]. Our method is implemented based on the PyTorch and train the model on machine with NVIDIA GeForce RTX4090 GPU.
7

---

Table 2: Accuracies (%) on the VisDA-17 using the ResNet-101 backbone.
Method
DABP
plane
bike
car
knife
meycle
person
plant
skibrrd
train
truck
Mean
H.Mean
Source-only
64.3
24.6
47.9
75.3
69.6
8.5
79.0
31.6
64.4
31.0
81.4
9.2
48.9
35.2
MCC
x
88.7
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
76.8
IIMA
97.6
88.4
84.3
76.0
98.4
97.1
91.3
81.4
97.0
96.7
88.8
60.7
88.1
87.9
COT
X
98.2
89.4
87.6
82.3
98.0
97.2
96.4
86.2
98.3
92.6
92.2
58.1
89.7
88.5
LNL-KL
82.7
83.4
76.7
44.9
90.9
38.5
78.4
71.6
82.4
80.3
82.9
50.4
71.9
70.8
HD-SHOT
75.8
85.8
78.0
43.1
92.0
41.0
79.9
78.1
84.2
86.4
81.0
65.5
74.2
74.4
SD-SHOR
79.1
85.8
77.2
43.4
91.6
41.0
80.0
78.3
84.7
86.8
81.1
65.1
74.5
74.8
DINE
81.4
86.7
77.9
55.1
92.2
34.6
80.8
79.9
87.3
87.9
84.3
58.7
75.6
74.3
BETA
94.9
90.2
85.4
61.1
95.5
93.1
85.0
83.8
92.9
91.9
91.1
55.0
85.1
85.9
RFC
95.6
89.7
87.8
75.8
96.5
96.5
90.4
82.8
96.0
70.0
85.7
55.1
85.2
84.2
97.9
88.0
73.5
96.1
92.4
85.7
93.9
95.6
91.2
89.2
89.5
CVH-TDN
96.9
91.6
87.3
836
97.0
95.8
92.7
90.6
95.9
95.5
92.6
67.2
90.6
90.1
To make fair comparisons, we follow
Table 3: Accuracies (%) on the Office-31 using the ResNet- 50 backbone.
[1]3] select the ResNet [1 pre- trained on ImageNet [32 as the back-
bone in all experiments, where ResNet- 50 is used for -1fice-31, Office-Home and DomainNet, while ResNet-101 used for VisDA-17. In our experi- ments, we adopt SGD optimizer with the weight decay 1e-3, the momentum 0.9, the feature extractor learning rate le-4, and the classifier learning rate le-
Method DABP/A-D A-+
D-*A
D-#
W-+A
W- Mean H.Mean
Sourve- only
79.9
76.6
56.4
92.8
60.9
98.5
77.5
58.7
MCC
x
95.6
95.4
72.6
V8.6
73.9
100
89.4
73.3
HMA
x
95.8
95.1
79.3
99.3
77.6
91.2
78.5
LNL-KI.
89.4
86.8
65.
94.8
67.1
98.7
83.6
66.1
HD-SHOT
86.5
83.1
66.
95.1
68.9
98.1
83.0
67.5
SD-SHOT
89.2
83.7
67.9
95.3
71.1
97.1
84.1
69.5
DINE
86.8
72.2
96.2
73.3
86.4
72.8
BiMom
92.8
88.2
73.9
96.8
75.3
99.4
87.7
74.6
3. The bottleneck size is set to 256 and the batch size is set to 64. Fol- lowing previous SOTA [4], the num- ber of warm-ups is set to 3, and both accuracy (Mean) and on-hard task (whose source-only ac- curacies are below 65%) average ac- curacy (H.Mean) are reported. Spe- cific dataset details are presented in Appendix D.
BETA
93.6
88.3
76.1
y5.5
76.5
99.0
88.2
76.3
RFC
94.4
93.0
76.7
95.6
77.5
98.1
89.2
77.1
SEAL
95.1
88.3
77.6
96.0
76.7
99.3
88.8
77.2
CVH-TDN
92.8
75.6
98.9
81.0
90.7
78.3
Table 4: Results with different values of hyperparameter Office-31 31 with ResNet-50 backbone.
A-D
A-W
D
D-+W
W-+
W-+
Mean
92.8
91.3
73.2
98.5
76,8
99.6
88.7
10%
96.4
92.8
75.6
98.9
81.0
99.6
90.7
20%
95.8
93.2
73.8
98.9
77.8
99.6
89.8
30%
95.4
93.0
72.4
99.0
74.4
99.8
89.0
Comparison Methods. To evaluate our method, we choose several related UDA and DABP works for compari- son. For UDA. we compare CDAN 33
Table 5: Results of ablation study on the Office-31 and VisDA-17 datasets.
MCC [4), HMA [35), and CST
Loss CHA LNG
Office
VisDA
36
Meanwhile, we compare our
HG
A- A- D-
A
Mean
CVH-TDN with previous SOTA DABP
Source unly
79.9
76.6
56.4
92.8
60.9
98.5
77.5
48.9
methods, including LNL-KII. 16) HD-
89.3
86.4
73.6
96.5
74.4
99.0
86.5
75.7
SHOT 8 SD-SHOT 8. DivideMix
89.8
74.5
98.5
75.2
99.6
87.4
78.1
37..
DINE IT BiMem [3], BETA
94.4
90.4
74.7
98.9
80.1
99.8
89.7
85.1
93.8
92.1
74.5
98.6
77.1
99.9
89.3
83.5
14
RFC 11). and SEAL (3I. To
92.9
90.2
73.9
98.6
78.9
99.4
89.0
82.5
ensure fair comparisons, we follow the training protocol, learning strategy, and network architecture for the source do- main as specified in DINE. The results of all comparison methods are obtained from the original papers, their associated codehases, or follow-up work.
96.4
92.8
75.6
98.9
81.0
99.6
90.7
90.6
CHA CHC Luc
94.5
91.9
75.8
98.7
79.7
99.6
90.0
88.7
94.7
92.3
76.1
98.7
80.1
99.6
90.3
87.1
Results. In Tables 2, 3, and Appendix E, the results show that CVH-TDN significantly outperforms the previous SOTA methods on all four benchmarks. For average accuracy, CVH-TDN outperforms the previous best results by 4.1% 11. 1.6%, 1.5%, and 1.4% on Office-Home DomainNet, Office-31, and VisDA-17, respectively. For these hard tasks with lower than 65% source-only accuracy, CVH- TDN outperforms the baseline BETA by 8.8%, 1.6%, 2.0%, and 4.2% on OfficeHHome, DomainNet. O1f 31, and VisDA-17 17. These results demonstrate that the exploration of sample spatial relationships

---

Imput
Source-only
BETA
wio HG
w/o BA
wi Hac
wio
CVH--DDN
Dus
Car
Motor
cycle
Person
Skate
board
Truck
Figure 4: The visualization results of the heat map on the VisDA-17. Each result is reported when the best accuracy is achieved.
by controlled hallucination is more ef- fective than the suppression of spe- cific sample information in increasing class diseriminability.
90.5
30.55
90.46 90.4
50.07
75
89.3
89.0-
Parameter Analysis and Compari- son. As shown in Figure 5(a), we plot the trends of prediction accuracy un- der different arousal ratios r with all 60 epochs on the VisDA-17 dataset. When r is equal to 0.5, we observe that the best performance can be ob- tained. And when is equal to the hallucination generation fails and all generated images are uncontrollable. These highlight that controllable hal-
0.5
88.5
0.8
/80.23
spoch
30
50
Value of
(a)
(b)
Figure 5: The accuracy trends of predictions the VisDA- /7 dataset. (a) shows the accuracy evolution CVH-TDN under different arousal values of r. (b) shows the accuracy with different values of hyperparameter ó.
lucination generation essential for improving the model reasoning ability, while uncontrollable generation can cause CVH-TDN to fail. Figure 5(b) plots the accuracy under different values, where determines the calibration influence of each high-reliability sample. the value of á increases, high-reliabille samples tend cluster low-reliability samples with similar signature features more capably. However, if á is too large, common features may also be considered, leading to premature model overfitting. Table 4 analyzes the effect of hierarchical strategy under difffrrntt À values on the Office-31. When is equal to 0, the hierarchical strategy fails, which leads to the negative influence of the information from unreliable samples the early learning of the model. A higher value of reduces the proportion of reliable samples, leading to decreased efficiency in exploring sample spatial relationships. More parameter analysis and comparison shown in Appendix F.
Ablation Study. We report the ablation study on the Office-31 / and VisDA-17 in Table 5 and present the Grad-CAM 38 visualizations for the VisDA-17 dataset in Figure 4.
As shown in Table 5, experimental results demonstrate that every part of our algorithm is necessary and indispensable. LH is designed to prevent overfitting of the model by controlling the original and hallucination images for bidirectional alignment, and can effectively improve model generalization only when combined with Hallucination Generation (HG). LHC is an efficient clustering loss, but since the effect of task-specific loss is also sample clustering, only the combined effect of the two leads to model overfitting. For this, LHA helps LHC to alleviate the conflict with task-specific loss. When Lu and HG are combined, the class discrimination ability and model reasoning ability are further improved. As shown in Figure 4, both Hsc and Cfe components of LIC contribute to improving the reasoning ability of the model by highlighting the areas that the model focuses on for given category. When Hsc is removed, the values of the regions of interest in the model are averaged,

---

resulting in a lot of non-main features being captured. Both modules of LиА and Cfe are designed to eliminate overfitting: L is to eliminate overall sample overfitting, and Cfe is to eliminate reliable sample overfitting. When Cff is removed, the ability to grasp key features is weakened, resulting in some key features not being captured. Combining the two, the model can extract the features in the image accurately and efficiently. In summary, H A and HG jointly determine the approximate region of feature extraction from the model, and Le optimizes features for the region of interest of the model. The results show that their combination significantly improves the model reasoning ability, More visual studies are shown in the Appendix G.
Conclusion
In this work, observe some weaknesses in existing DABP methods and solve them by proposing a novel algorithm, CVH-TDN, Inspired by the Thalamus-driven Decoupling Network, CVH-TDN contains the Hallucination Generation (HG), Hallucination Alignment (HA), and Hallucination Calibration (HC) to achieve sample clustering and hallucination control by exploring the spatial relationships. HG utilizes the attention of the training model to generate exploitable hallucinations. HA explores the spatial relationship between the samples and the hallucinations generated by HG through bidirectional alignment to encourage clustering of samples with similar features. HC uses hierarchical way to aggregate samples with similar signature features and separates ambiguous sam- ples with common features. Experimental results demonstrate the effectiveness of spatial similarity exploration in enhancing the model reasoning and class discriminability. CVH-TDN significantly outperforms previous SOTA methods on all comparison datasets.
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China under Grant 62176162 and in part by Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515012875 and Grant 2022A151514000
References
[1] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.
[2] O. Pinheiro. "Unsupervised domain adaptation with similarity learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8004-8013.
[3] D. Hu, J. Liang, X. Wang, and C.-S. Foo, "Pseudo-calibration: Improving predictive uncer- tainty estimation in unsupervised domain adaptation,"" in Proceedings of the 41th International Conference Machine Learning (ICML)), 2024, pp. 19 304-19 326.
[4] N. Inoue, R. Furuta, T. Yamasaki, and K. Aizawa, "Cross-domain weakly-supervie object detection through progressive domain adaptation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 5001-5009.
[5] B. Pu, X. Lv, J. Yang, H. Guannan, X. Dong, Y. Lin, L. Shengli, T. Ying, L. Fei, M. Chen, Z. Jin, K. Li, and X. Li, "Unsupervised domain adaptation for anatomical structure detection in ultrasound images," " in Proceedings of the 41th International Conference Machine Learning (ICML), 2024, pp. 41 204 41 220.
[6] U. Hwang, J. Lee, J. Shin, and S. Yoon, "SF(DA)2: Source-free domain adaptation through the lens of data augmentation," in International Conference on Learning Representations (ICLR), 2024. [Online]. Available:
[7] N. Karim, N. C. Mithun, A. Rajvanshi, H.-p. Chiu, S. Samarasekera, and N. Rahnavard, "C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation," in Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 24 120-24 131.
10

---

[8] J. Liang, D. Hu, and J. Feng, "Do we really need to access the source data? Source hypoth- esis transfer for unsupervised domain adaptation." " in Proceedings of the 37th International Conference Machine Learning (ICML), 2020, pp. 6028-6039.
[9] I. Goodfellow, J. Pouget-Abadie Abadie,, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), 2014.
[10] Z. Fei, M. Fan, L. Zhu, J. Huang, X. Wei, and X. Wei, "Masked auto-encoders meet generative adversarial networks and beyond," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023, pp. 449924459.
[11] J. Liang, D. Hu, J. Feng, and R. He, "Dine: Domain adaptation from single and multiple black-box predictors," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022. pp. 7993-8003.
[12] L. van der Maaten and G. Hinton, "Visualizing data using t-sne," Journal of Machine Learning Research (JMLR), vol. 9, no. 86, pp. 2579-2605, 2008.
[13] J. Zhang, J. Huang, X. Jiang, and S. Lu, "Black-box unsupervised domain adaptation with bi-directional atkinson-shiffrin memory," " in Proceedings of the IEEE/CVE International Con- ference on Computer Vision (ICCV), 2023, pp. 11 17771-11788 782.
[14] J. Yang, X. Peng. K. Wang, Z. Zhu, J. Feng, L. Xie, and Y. You, "Divide to adapt: Mitigating confirmation bias for domain adaptation black-box predictors," in International Conference on Learning Representations (ICLR), 2023, [Online]. Available: htwpppppwwwweeeeeeee//oo/oddddda
[15] S. Zhang, C. Shen, S. Lü, and Z. Zhang, "Reviewing the forgotten classes for domain adaptation of black-box predictors," " in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 38, 2024, pp. 16 1108830-6600 837.
[16] H. Zhang, Y. Zhang, K. Jia, and L. Zhang, "Unsupervised domain adaptation of black-box source models," ArXiv, vol. abs/2101 101.0939,....... 2021.
[17] J. Liang D. Hu, Y. Wang, R. Hc, and J. Feng, "Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer," IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 44, pp. 8602-8617, 2020.
[18] Z. Lipton, Y.-.. Wang, and A. Smola, "Detecting and correcting for label shift with black box predictors," in Proceedings of the 35th International Conference on Machine Learning (ICML), 2018, pp. 3122-3130.
[19] M. Onoftj. A. Espay, L. Bonanni, S. Delli Pizzi, and S. Sensi, "Hallucinations. somatic- functional disorders of pd-dlb as expressions of thalamic dysfunction," Movement Disorders (Mov Disord.), vol. 34, no. pp. 1100-1111, 2019.
[20] D. Collerton, Barnes, N. J. Diederich, R. Dudley, D. ffytche, K. Friston, C. G. Goetz, J. G. Goldman, R. Jardri, J. Kulisevsky, S. J. Lewis, S. Nara, C. O'Callaghan, M. Onofrj, J. Pagonabarraga, T. Parr, J. M. Shine, G. Stebbins, J.-P. Taylor, I. Tsuda, and R. S. Weil, "Understanding visual hallucinations: A new synthesis," Neuroscience Biobehavioral Reviews (Neurosci Biobehav Rev) vol. 150, p. 105208, 2023.
[21] Y. Zhang, X. Yu, X. Lu, and P. Liu, "Pro-uigan: Progressive face hallucination from occluded thumbnails." IEEE Transactions on Image Processing (TIP), vol. 31, pp. 3236-3250, 2022.
[22] K. Zhang, D. Li, W. Luo, J. Liu, J. Deng, W. Liu, and S. Zafeiriou, "Edface-celeb-11 m: Benchmarking face hallucination with million-scale dataset," IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), vol. 45, no. 3, pp. 3968-3978, 2023.
[23] Y. Zhang, I. W. Tsang, Y. Luo, C. Hu, X. Lu, and X. Yu, "Recursive copy and paste gan: Face hallucination from shaded thumbnails." IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 44, no. 8, pp. 4321-4338, 2022.

---

[24] J. Hoffmann, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko. A. Efros, and Darrell, "CyCADA: Cycle-consistent adversarial domain adaptation." " in Proceedings of the 35th International Conference Machine Learning (ICML), 2018, pp. 1989-1998.
[25] J. Tian, J. Zhang, W. Li, and D. Xu, "Vdm-da: Virtual domain modeling for source data-free domain adaptation." /EEE Transactions on Circuits and Systems for Video Technology (TCSVT), vol. 32, no. 6, pp. 3749-3760, 2022.
[26] Z. Zhang, W. Chen, H. Cheng, Z. Li, S. Li, L. Lin, and G. Li, "Divide and contrast: Source- free domain adaptation via adaptive contrastive learning." in Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS), 2022, pp. 5137-5149.
[27] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting visual category models to new domains," in Proceedings of the 11th European Conference on Computer Vision (ECCV), 2010, pp. 213- 226.
[28] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, "Deep hashing network for unsupervised domain adaptation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2017, pp. 5018-5027.
[29] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko, "Visda: The visual domain adaptation challenge," ArXiv, vol. abs/1710.06924, 2017.
[30] X. Peng. Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, "Moment matching for multi- source domain adaptation." " in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1406-1415.
[31] M. Xia, J. Zhao, G. Lyu, Z. Huang, T. Hu, G. Chen, and H. Wang, "A separation and alignment framework for black-box domain adaptation." in Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), vol. 38, 2024, pp. 16 005-1603..
[32] J. Deng, W. Dong, R. Socher, L.-J Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248-255.
[33] M. Long, Z. Cao, J. Wang, and M. I. Jordan, "Conditional adversarial domain adaptation." in Proceedings of the 32th International Conference on Neural Information Processing Systems (NeurIPS), 2018, pp. 1647-1657.
[34] Y. Jin, X. Wang. M. Long, and J. Wang, "Minimum class confusion for versatile domain adaptation." " Proceedings of the 17th European Conference on Computer Vision (ECCV), 2020, pp. 464 480.
[35] L. Zhou, M. Ye, X. Zhu, S. Xiao, X.-Q. Fan, and F. Neri, "Homeomorphism alignment for unsupervised domain adaptation." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 18 199-18710.
[36] H. Liu, J. Wang, and M. Long, "Cycle self-training for domain adaptation," in Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), 2021, pp. 22 908-628899 981.
[37] J. Li, R. Socher, and S. C. Hoi, "Dividemix: Learning with noisy labels as semi-supervised learning," International Conference on Learning Representations (ICLR), 2020. [Online]. Available: http:://openeevie.eeeeffooo 2id Vtwr
[38] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp- 618-626.
[39] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and Lempitsky, aaveraiaadversarial training neural networks," Journal of Machine Learning Research (JMLR), vol. 17, pp. 2096-2030, 2016.
12

---

[40] S. Chen, M. Harandi, X. Jin, and X. Yang, "Domain adaptation by joint distribution invariant projections," IEEE Transactions on Image Processing (TIP), vol. 29, pp.
2020.
[41] H. Yan, Y. Ding, P. Li, Q. Wang, Y. Xu, and W. Zuo. "Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation." Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2272- 2281.
[42] D. Hu, J. Liang, Q. Hou, H. Yan, and Y. Chen, "Adversarial domain adaptation with prototype- based normalized output conditioner," IEEE Transactions on Image Processing (TIP), vol. 30, pp. 9359-93933 99779, 2021.
[43] I.. Hoyer, Dai, H. Wang, and I.. Van Gool, "Mic: Masked image consistency for context- enhanced domain adaptation," in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 11 711-11 732.
[44] I.. Hoyer, D. Dai, and L. Van Gool, "Hrda: Context-aware high-resolution domain-adaptive semantic segmentation," in Proceedings of the 17th European Conference on Computer Vision (ECCV), 2022, pp. 372-391.
[45] K. Saito, Ushiku, T. Harada, and Saenko, "Strong-weak distribution alignment for adaptive object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6956-6965.
[46] M. Khodabandeh, A. Vahdat, M. Ranjbar, and W. G. Macready, "A robust learning approach to domain adaptive object detection," " in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 480-490.
[47] Y. Chen, W. Li, C. Sakaridis, D. Dai, and Van Gool, "Domain adaptive faster r-cnn for object detection in the wild," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2018, pp. 3339-3348.
[48] Q. Liu and H. Xue, "Adversarial spectral kernel matching for unsupervised time series do- main adaptation," Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021, pp. 2744-2750.
[49] R. Cai, Chen, Z. Li, W. Chen, Zhang. J. Ye, Li, X. Yang, and Zhang. "Time series domain adaptation via sparse associative structure alignment," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 6859-6867.
[50] X. Jin, Y. Park, D. Maddix, H. Wang, and Y. Wang, "Domain adaptation for time series forecasting via attention sharing," " in Proceedings of the 39th International Conference on Machine Learning (ICML), 2022, pp. 280-10
[51] M. Litrico, A. Del Bue, and Morerio, "Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation." in Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023, pp. 7640-7650.
[52] S. Yang, Y. Wang, K. Wang, S. Jui, and J. van de Weijer, "Attracting and dispersing: simple approach for source-free domain adaptation." " in Proceedings of the 35th International Conference Neural Information Processing Systems (NeurIPS), 2022, pp. 5802-5815.
[53] D. Chen, D. Wang, T. Darrell, and Ebrahimi, "Contrastive test-time adaptation." in Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 295-305.
[54] M. Jing, J. Li, K. Lu, I. Zhu, and H. T. Shen, "Visually source-free domain adaptation via adversarial style matching," " IEEE Transactions on Image Processing (TIP), vol. 33, pp. 1032- 1044, 2024.
[55] S. Ben-David, J. Blitzer, K. Crammer, Kulesza, Pereira, and Vaughan, theory of learning from different domains." Machine Learning, vol. 79, pp. 151-175, 2010.
13

---

[56] B. Schölkopf, J. Platt, and T. Hofmann, "Learning from multiple sources," in Advances in Neural Information Processing Systems, 2007, pp. 321-328.
[57] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, E, Laviolette, M. Marchand, and V. Lempitsky, "Domain-adversarial training of neural networks," Journal of Machine Learning Research, vol. 17, no. 1, p. 2096-2030, 2016.
[58] A. Baddeley, "Working memory: looking back and looking forward," Nature reviews neuro- science (Nat. Rev. Neurosci.), vol. 4, no. 10, pp. 829- 839, 2003.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims of this paper be found in the Abstract and Introduction sections, which accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limiting analysis in Appendix H.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the theoretical analysis of the proposed method in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all necessary information for reproducibility. The implementation details is shown in the Experiments section. The algorithm details in Appendix C. The dataset details in Appendix D. The experimental code and the main code are available in the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The experimental code and the main code available in the Supplementary Materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
aublic/guudees/Codeetbbmsss
www.sssinoooiiillo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplclgguiiaeeeaeccuuea
Stteee///ooSSSeesssSiSi.a ssissssliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details are described, including hyperparameters, experiments, and datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computational cost comparison in Table 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp::///nurrige www.rpsu.p...// cc/puullic/thhicccuueeeel......
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impact in Appendix H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
In the Related Works and Introduction sections, we have explained that our aim is to provide better data privacy protection with more flexible portability to prevent the leakage of source data or the trained source model.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the code and dataset utilized in this work are publicly available and are only intended to compare the performances of different algorithms classification tasks.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This submission poses no such risks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This submission poses no such risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This submission poses no such risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This submission poses no such risks.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htt.pp///earripcccc///ooeeeerrre for what should or should not be described.
21

---

Appendix
A. More Related Works
Unsupervised Domain Adaptation. UDA aims to tackle the challenge of generaliz- ing a model trained on a large number of labeled samples from the source domain to the target domain. UDA has been ex- tensively explored for many practical ap- plications, including image classification 39 semantic segmentation 442
Model
Maytaiionn
UDA
Cross Enstrep Loss
Ralon
Taract Model
Adketation
SEDA
Chooodile
object detection 47 and time series forecasting [8 55). However, UDA meth- ods assume that both the labeled source domain and the unlabeled target domain are available at the same time, which is not available in some scenarios where the source domain data cannot be accessed dur- ing training due to preecc-preservingg poli- cies.
Source Domain Saceples
Cross Entogy Loos
000
Cloud
Pooliction
Source
Predicias
DABP
Adapation Line
Tanget Doman Samplex
Source-free Domain Adaptation (White- box Predictors). Compared to the UDA setting, SFDA has higher privacy protec- tion because it does not need to touch any source data. Many SFDA methods have been proposed recently 668 5T]. which only require access to unlabeled target data and trained source domain model during training. Most existing research SFDA tasks is mainly based self- training 6 [151] class prototypes g [22] contrastive learning 7 [3) and generative models 54 25 Although the SFDA task has contributed the mitigation of privacy protection issues to some extent, recent re- search [13] has found that exposing the details of the white-box predictive model training is quite dangerous due certain reverse generation techniques like ( [0].
Figure 6: Illustration of the differences among UDA, SFDA, and DABP. The dotted lines in the figure indi- cate the operations performed by the cloud API with the source model under different settings. SFDA re- quires the entire source model to be obtained from the cloud API before training. DABP outperforms SFDA in data privacy protection and portability, simply requiring uploading target data the cloud API and then down- loading predictions before training.
Previous Methods
Ours
Setting Comparison and Method Im- provement. As shown in Figure 6, the respective processes and the differences among UDA, SFDA, and DABP are pre- sented. Compared with SFDA, DABP pro- vides better data privacy protection with more flexible portability, which only needs to upload target data to the cloud API and then download predictions before training. Figure 7 illustrates the advantages of CVH- TDN and the differences from previous DABP methods. Different from the previ- ous DABP methods 13-55 31, CVH- TDN leverages sample spatial similarity instead of suppressing targeted sample formation, thereby improving model generalization and class discrimination.
Highsdahiiy
Tapei Surples
Immeriiiity
Surple
xx
Prcetpees
GrestalCas
Prokeyps
Cau Disimtaiis
Rumday
Figure 7: Comparison between existing DABP methods and CVH-TDN. Top: Existing DABP methods focus on learning high-reliability knowledge and force low- reliability sample aggregation. Bottom: CVH-TDN proposes hallucination alignment to investigate the spa- tial connection between samples and hallucinations and introduces hallucination calibration to explore relation- ships among spatially similar samples. Our algorithm enhances the capabilities of model generalization and class discrimination.
22

---

B. Theoretical Analysis
We provide theoretical justifications grounded in the generalization bound of reasoning to clarify the working mechanism of our algorithm.
Since our algorithm is trained in an unlabeled target domain and generates the controlled samples based on our hallucination generation, we denote 2 Dr as the real sample distribution of the target domain. Based on the existing theories [55]. the error of CVH-TDN can be formulated as a convex combination of the errors of the reliable subdomain In ~ Dn. the unreliable subdomain Zyn Dun. and the generated subdomain zG ~ DG that represents the generated sample distribution of the reliable subdomain. And denote yn. yUR R and ÿn. Dun as the true labels and the predicted labels of œ and ZUR R. respectively. Denote ya as the true label and KG as the predicted label of ag- yr and ir are the true labels and the predicted labels of target domain, yo yR. yT = yR yUR, and er = ØR- - DUR. Let H denote a hypothesis, which can be expressed as:
cR (H, Vn) S cG (H, (G) duann (DR, DG) + EG.
(19)
cp (H, (1) OCR (H, (R) + (1 a EUR (H. R) er (H, yr) = oCR HH yR) + (1 - auuu (H,Y
(20)
(21)
where duan (Dn, DG) = 2 2suPH, DD ((
# H'(nn)] Ez (H *
H*()))] eg (H, (a) is the expected error of the generated sample distribution; EG min(eR (H, ÿR) + EG (H, BG)): ER (H, ÿR) is the expected error of the reliable subdomain; is the trade-off parameter that is controlled by À in Eq. (7); EUR (H, Nu is the expected error of the unreliable subdomain; cn (H, yn) and CUn (H, yun) denote the oracle errors of reliable and unreliable samples, respectively.
For a (H,ğR) S EG DG) + dnan(DR, Da) + sg, we analyze each component in detail in this paragraph:
(1) eG (H, (G) is the expected error of the generated sample distribution, which can be minimized with cross-entropy loss in the former term of Eq. (15). With the guidance of the black-box predictors. in the initial stage of training, the distribution of the reliable samples selected by the target model in the feature space is relatively close to that of the samples in the source domain. We can obtain good training results for these samples with similar features through the black-box predictors. As the training progresses, the model gradually adapts to the distribution of the target domain. the target model continuously selects more reliable samples. Thus, DG can better learn the knowledge of these reliable samples through Eq. (15), and align the distribution of the target features through Eq. (12), so as to continuously adapt to the target domain. Therefore, eg (H,ūG) is small in the whole training.
(2) EG is the shared error of the ideal joint hypothesis which is considered to be sufficiently small constant to represent the complexity of the generated sample hypothesis space.
(3) du (DR, DG) depends the expected error of the disagreement between two hypothesis on the reliable subdomain and the generated sample distribution of the reliable subdomain. In the early stages of training, it is easy find two hypotheses that both H and H' correctly predict the reliable samples. As the training progresses, hallucination alignment explores the spatial relationship between the samples and the hallucinations generated by hallucination generation through bidirectional alignment to encourage clustering of samples with similar features. Hallucination alignment helps to maintain the discrimination and generalization ability of the model in the target domain. Therefore, ExerDe H (xa) * H'(a)) is always small during adaptation phase. Hallucination calibration uses hierarchical way to aggregate samples with similar signature features and separates ambiguous samples with common features. Hallucination calibration helps to improve model reasoning ability and prevent reliable sample overfitting. The joint learning of hallucination alignment and hallucination calibration is conducted on reliable samples. As a result, E EER~DR H (*R) & H'(zn) always maintaining a small value during adaptation phase.
Then, during the transition from the early stage to the middle stage of adaptation, we derive an upper bound of how the error e (H, IR) is close eT (H, yr) though hallucination calibration, which is the oracle error with the truth label of the target domain.
Theorem 1. Let H be hypothesis in class n, we have:
23

---

(e, (r) er(H, a (R) + (1 a) EUR (UR)
a (H,)) (1 a) (. yun)
S ( (H,))) - e ) len H, UR) EUR yu R)R)
+ +URR (H, DUR) - CUR yUR)
Sa (dnsn (Dn. DUn) + 8) + PUB:
(22)
where the ideal risk in this hypothesis H is the combinatorial emor of the ideal joint hy- pothesis E én (H*, yn) + cgn (H* RUR) with H arg min (en (H, yR) + cgn (H, yuR)); == ER ( +EUR (""90R) is the predicted risk; the distribution discrepancy between reliable and unreliable subdomains is d,a.n (Dn, Dun) 2supH, H'en Bxg=DD (H()) # H'(nn) the predicted label rate of YUR.
EAER~DOR (TUR) + H((xn))]]: and PUR Then, we set E1 |RR (R) EUR (H, DCR)), and
PUR EUR (9U R. YUR)
lRR (H, yR) EUR (H, YUR).. = |EUR (H, (UR) - EUR (H, R)R) making Op (H. - er yr) I a(11 + 62) + cz. By applying the triangle inequality for classification errors (56) as presented in Lemma
can prove the upper bound of éI, 62, and E3.
Lemma 1. For any hypotheses H1, H2, and H3 in class n,
(H1, H2) I (H1, Ha) + (H2, Ha)
(23)
Therefore, for we can prove that:
E1 = (RR(H, yR) (H, yUR)
< ee H, yR) (R,H, H*] + (Hu H²) - ww)) (H, - -En(H, SER(H .UUR) + EUR(H. R)R) + (RR(H, H*) - EU H+)l
1 dnann (D.. Dun) +E
(24)
2
Similarly, for 62,
c2 |ER(H, πR) - EUR TUR)
cen H. (R) CR(H, H (H)
||m (H(HHHH")
ER(H" .ūR) + EUR(H (UR) + (eR(H, H") - EUR(H,
1
d dnan (Dn. Dun) en(H (H (n) + cHn(H" ;un)
2
1 dnan (DR, Dun) + E.
(25)
2
For 63.
E3 = (EURR TUR) - EUR yUR) S EUR(MUR.UUR) = PUR.
(26)
By proving 61, 62, and E3, we can derive Theorem (eq 0r) (H, a(e1 + +22) + 6
s a dndan DUR) +E (DdnnnDDR DUR) +8) + PUR
{ (
=a (dd.s..DR Dun) + + + PUR:
(27)
when the reliable subdomain is mostly correct, yR and DR are extremely similar with the ideal risk that is negligibly small [57], in which case PR 0, É is bounded by the predicted label rate of unreliable subdomain PUR- Empirical results demonstrate that d, (D,, DUR) is usually small across the two subdomains, which plays a significant role in tightening the upper bound though hallucination calibration. Therefore, our method can theoretically reduce the expected error of the model on the target domain.
C. Algorithm Details
The whole training process shown Algorithm 1. In addition, the experimental code and the main code are available in the Supplementary Materials.
24

---

Algorithm CVH-TDN for DABP task.
Input: Target samples D. = )- black-bor box hard predictions P., and training model Me E {P, Ch: The model parameter 8 and the hyperparameters X, μs, Pa, Ø, and
Parameter:
1: Initialize: Ms simple tests on Dr to initialize 0 and memory storage M; initialize smooth adaptive storage S with P. and ji (determined by it, and μo;
2: while Adaptation do Get sample batch B using Ma from De;
Hallucination Generation:
5:
Control hallucination image =? generated by evaluating æ; B using Eqs. (1)-(4).
6:
Hallucination Alignment:
7:
Dynamic divide the samples according their reliability using Eqs. (6)-(8);
Update M using Eq. (5) to extract the latest information: Calculate bidirectional alignment weights w using Eq. (9).
9:
10:
Hallucination Calibration:
11:
Update S and using Eq. (13) mimic the brain cognitive processes;
12:
Retrieve M explore the relationship between features and space and calculate the spatial similarity.
Model Training:
14:
Optimize target model Ma by minimizing Eq. (18).
15:
end while
Table 6: Accuracies (%) on the DomainNet using the ResNet-50 backbone. The rows represent the source domain and the columns represent the adapted target domain.
ResNet
inf
pnt
skt
Mcan
DINE
clp
put
qdr
clp
36.0
52.8
41.8
31.4
clp
29.6
11.1
37.3
inf
32.1
32.0
47.4
26.4
28.1
inf
29.5
37.6
3.4
53.8
26.5
30.1
pnt
29.6
23.2
4,9
36.7
27.8
24.4
pnt
37.3
12.9
4.2
60.5
34.7
29.9
gdr
1.9
4.3
7.7
5.3
6.6
rel
48.2
19.6
47.9
4.3
35.6
31.1
rel
45.1
14.4
49.7
5,5
35.0
29.9
Mean
34.0
14.8
30.7
6.7
37.7
27.9
25.3
Mean
32.9
10.0
31.8
7.2
48.0
28.0
26.2
BETA
inf
rel
skt
Mean
CVH-TDN
clp
pnt
qdr
skt
Mean
clp
13.4
41.2
61.8
41.1
34.1
clp
41.8
11.3
42.7
inf
34.9
41.6
3.7
56.8
30.7
33.6
inf
37.8
39.9
2.9
53.5
32.1
33,2
pnt
47.3
18.4
62.5
41.9
34.7
pni
45.2
18.9
4.0
60.3
37.9
33.3
gdr
0.9
8.1
6.4
15.0
2.9
rel
46.5
15.8
50.9
5.6
37.7
31.3
rel
52.7
21.7
52.5
5.7
39,2
34.4
12.4
Mean
37.5
12.2
35.6
8.1
50.0
31.9
28.2
Mean
41.0
15.6
36.3
7.3
46.3
32.6
29.8
D. Specific Dataset Details
Four standard benchmark datasets are used for evaluating our method and comparison, including Office-31 (27), Office-Home (28), VisDA-17 29)], and DomainNet (30). Office-31 is small-scale benchmark dataset, which contains 4,1 10 images with 31 categories from domains, Amazon (A), Dsir (D), and Webcam (W). Office-Home is a widely used medium-scale benchmark that contains total of 15.5K images with 65 categories from 4 distinct domains, Real World (R), Clipart (C), Art (A), and Product (P). VisDA-17 a challenging large-scale benchmark with 12 categories that include 152k synthetic source domain images and 55k target images of real objects, presenting greater challenge due large synthetic-to-real domain gap. DomainNet is the largest domain adaptation benchmark dataset, which consists of about 600K with 345 categories across domains: Clipart (clp), Infograph (inf), Painting (pnt), Quickdraw (qdr), Real (rel), and Sketch (skt).
E. Additional Dataset DomainNet
For the DomainNet dataset, we compare our algorithm with previous SOTA methods [1 under the ResNet-50 backbone. As shown in Table 6, compared with other algorithms, CVH-TDN achieves the highest average accuracy of 29.8% on the DomainNet, which is full of on-hard tasks (whose source-only accuracies are below 65%). These results demonstrate that, in the on-hard tasks,
25

---

The effects of different and As
Source-only
0=0.6, ū.= 0.6
==0.5, 0,=
70.3 70.9 71.7
p=0.5, 0095.99
Accuracy
60.1 59.6
59.8 59.3
60
57.2
52.2
44.1
Source-only
DINE
BETA
SEAL
CVH-TDN
Figure 8: The effect display of different ye and ü. in the A-C subtask of the Office-Home. Each result reported when the best accuracy achieved.
spatial exploration method based on hallucination control is more effective than methods that rely on suppressing specific sample information.
F. More Parameter Analysis and Motivation Demonstration
As shown in Figure 8, we report the performance of different methods conditions of μ and ã. on the Office-Home. In the brain cognitive processes encounter something they have never seen before, their brain cogmition is primarily influenced by external information, with small portion derived from their own understanding. And as their under- standing deepens, they can balance the weight of external information and their own understanding, increasingly trusting their own understanding of the matter. In this work, we mimic the brain cognitive process to improve the learning and updating process of Eq. (13), where म. simulates the initial external information and μ simulates human understanding. Additionally, when j ش. 0.6, indicates the use of the previously common adaptive label smoothing update [II] instead of our update strategy. Experimental results show that this improvement significantly enhances the compared DABP methods.
14
under various when humans
58
Moreover, we conducted further analytical experiments demonstrate the advantage of our method for more effectively leveraging both reliable and unreliable samples compared to existing approaches. First, we elaborate on some concepts: the higher the model prediction accuracy of class, the higher the proportion of reliable samples in that class. Mcanwhile, the samples exhibiting features that clearly distinguish their class tend to have higher probability of being reliable. Because different methods have different strategies for discriminating reliability, we have selected sample instances with consistent initial reliability judgments between BETA [14] and SEAL BI. Specifically, in Figure 9, the classes corresponding to high-reliability samples are: Bus, Skateboard, and Person; and the classes corresponding to low-reliability samples are: Car, Motorcycle, and Truck.
As shown in Figure 9, the samples corresponding to the classes of Bus and Skateboard were consis- tently regarded as reliable samples during the training of BETA and SEAL. They focus on learning from these reliability samples, and Figure 9 demonstrates that they effectively concentrate and lock the feature regions of interest on the salient features. Meanwhile, some low-reliability samples (corresponding to the class Car) have also been correctly rectified from the wrong judgments of the black-bon predictors through high sample knowledge. However, for some low-reliability samples (corresponding to the class Motorcycle and Truck). the previous methods all failed: BETA didn't 't lock onto the effective features, and SEAL mistakenly locked onto the features of other classes. addition, SEAL even misjudged some samples that were judged as high-reliability samples (corresponding to the class Person) as belonging to other classes. These observations are sufficient to show the limitations of using high-reliability samples to constrain the attention-locker area. Moreover, the
26

---

Bus
Car
Motorcycle
Person
Skateboard
Truck
Input
Black-box
Predictor
BETA
SEAL
CVH-TDN
Figure 9: The visualization results of the heat map on the VisDA-17. The redder the area, the higher the model's 's level of attention.
Source-only: 60.9%
Only Lrask: 71.9%
w/o CHA: 78.9%
w/o Lho: 77.1%
w/o HG: 80.1%
CVH-TDN: 81.0%
Figure 10: The ablation visualizations of the feature distribution in the W-A subtask of the Office-31 using t-SNE [12. Herein, the points represent target samples and the different colors correspond to their true classifications.
feature visualization of Figure 1 can also serve evidence for the inadequacy of their clustering ability.
In our method, driven by the hallucination alignment, all type-U R samples will become samples of type R in the middle of training. We conceptually blurred the distinction between high-reliability and low-reliability samples during the learning process: samples are no longer constrained by high- or low-reliability ones. As shown in Figures and CVH-TDN enhances the model's reasoning ability by expanding the coverage area of the model's 's region of interest through hallucination generation and hallucination alignment, resulting in more effective use of all samples.
G. More Ablation Visualization and Experimental Comparison
In Figure 10, we present the t-SNE visualizations 12 of the ablation study on the Office-31 Experimental results show that each component of our method improves the discrimination capacity of classes. It is worth noting that, in the small-scale dataset Office-31, the impact of removing HG
27

---

Source-only: 48.9%
DINE: 75.6%
BETA: 85.1%
CVH-TDN: 90.6%
Figure 11: Classification visualization with Confusion Matrix compare different methods on the VisDA-17. (Zooming in for a clear view)
Table 7: Results of cost comparison on the VisDA-17 with the ResNet-101 backbone.
Method
Time (s/epoch)
Space (MiB)
Accuracy (%)
DINE
124s
9881MiB
75.6
BETA
1101s
20247MiB
85.1
SEAL
Over 24G
89.2
CVH--TDN
205s
10589MiB
90.6
or LHA not as significant as shown in the heatmap visualizations 38] of the large-scale dataset VisDA-/7 in Figure 4. This indicates that in datasets with small domain gaps, efficient clustering algorithm is more important than an algorithm designed to prevent model overfitting, while the opposite holds for datasets with large domain gaps. Furthermore, LH focuses enhancing the model reasoning ability, while Lue emphasizes improving the class discrimination ability.
For fair comparison, set the 4. etc.) in the compared works on a machine with an NVIDIA GeForce RTX4090 GPU. Figure 11 shows the Confusion Matrix visualization, in which discriminating target samples of each class on the VisDA-17. This demonstrates that exploring the spatial relationships among samples by controlling hallucinations is effective in improving class discrimination ability than suppressing specific sample information.
rumning conditions (e.g., batch size 64, num workers
method consistently outperforms in
In Table 7, record the average runtime cost, the maximum GPU space usage, and the best accuracy of each comparison method. When adapting VisDA-17, it is worth noting that BETA [4] is divided into two stages that are highly computationally intensive: the first stage is the initialization, which requires initialization the two models due to their mutually-distille stage is the two-step process, which requires distillation and fine-tuming for each epoch. SEAL [31] is highly resource- intensive, and its official code cannot complete the adaptation task on VisDA-17 under the same conditions with 24GB GPU memory. Compared to other methods, CVH-TDN calculates the model area interest and generates attention-specife masking blocks with minimal cost, which does computationally expensive generation network. Moreover, structure M uses 57.05 MiB of GPU space usage and our smooth adaptive storage S uses 2.54 MiB of GPU space usage. Since they do not require gradient storage updates during training, their GPU space usage is nearly negligible.
network structures; the second
not require fine-tuning
memory
To ensure robustness, we report the performances across multiple runs with different random seed initializations. As shown in Table 8, we maintained small average accuracy gap (0.4%) among different seeds, reflecting the stability and superiority of our method.
Table 8: Results under different random seeds on the Office- Home with the ResNet-50 backbone.
Seed
A-*C
A-+P
A-RR
C-A
C-PP
C-R
P-AA
P-C
P-R
+A
R
R-C
R-PP
Mean
2022
71.7
88.4
83.6
70.1
86.7
82.8
70.8
69.2
83.3
74.7
74.2
91.6
78.9
2023
72.2
83.7
69.9
87.7
82.9
68.7
83.3
74.9
73.8
79.0
2024
71.7
88.7
83.3
69.7
86.1
83.3
70.2
68.9
83.7
73.8
72.6
91.3
78.6
2025
72.1
88.6
83.5
70.0
87.9
82.4
70.1
67.2
83.7
75.4
72.1
91.4
78.7
28

---

H. Broader Impacts and Limitations
Our work CVH-TDN focuses on the problem of Domain Adaptation of Black-Box Predictors (DABP), which provides better data privacy protection with more flexible portability compared with other DA settings. Inspired by research in pathology and neuroscience, CVH-TDN is specifically designed for the DABP classification task. While its effectiveness has been demonstrated through extensive experiments and theoretical soundness established, its applicability to other tasks remains an open question. Therefore, we plan to further explore the practical utility of this algorithm in broader range of task scenarios.
29


Reviews:
[{"review_id": "CeT0QYfjcl", "reviewer": "Reviewer_juSi", "summary": "The paper aims to propose a better method that mitigates overfitting to reliable samples by generating hallucinated images and enhancing their similarity to the original images when facing the DABP problem.\n\nHowever, the writing of this paper is not very strong, as it includes many unnecessary descriptions, such as the TDN, which seems to serve more as a narrative device rather than a methodological component. As a result, many key ideas are not explained clearly and concisely.  \n\nFor example, among the three losses, the first two appear to incorporate adversarial concepts—striving for similarity while simultaneously preventing excessive similarity. Additionally, it is unclear how the unreliable information is actually utilized. From Equation 14, it seems that the method still primarily relies on reliable information. There are many other issues—please refer to the following comments.", "questions": "1. In Equation (9), what is the rationale for sorting the weights w? Why is this operation necessary in the context of the method?\n\n2. The loss formulation in Equation (12) appears adversarial in nature. The objectives of the forward and backward losses seem inconsistent. How is balance maintained during optimization, and how does the model effectively learn under such a setting?\n\n3. The calibration section is somewhat unclear. Equation (16) also seems to follow an adversarial formulation. Could the authors clarify the intended mechanism and its theoretical or empirical justification?\n\n4. Why are the losses from the three stages directly combined and used to jointly train a single model for UDA? Would it not be more natural to train the model in separate stages?\n\n5. I am also curious about the API used in this work. If the model is simply distilling from a powerful closed-source model, the performance gains may be primarily attributed to the strength of the source model rather than the proposed method itself.", "limitations": "Please see the questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction of the thalamus-driven concept is interesting and adds a biologically inspired perspective.\n\n2. The proposed method demonstrates a reasonable degree of novelty.\n\n3. The final results are relatively strong and show promising performance.\n\nWeaknesses:\n\n1. The connection between the thalamus-driven concept and the proposed method feels somewhat tenuous and would benefit from a clearer and more refined explanation.\n\n2. The methodology section is difficult to follow, and the current model diagram is not very intuitive. A more focused or simplified illustration highlighting the core idea would help improve clarity.", "was_revised": true, "final_justification": "According to the rebuttal, my concerns and other reviewers' concerns have been answered.", "rebuttal": "We sincerely appreciate the reviewer’s constructive suggestions and have answered them as follows with our highest respect.\n\nQ1: The connection between the thalamus-driven concept and the proposed method feels somewhat tenuous and would benefit from a clearer and more refined explanation.\n\nA1: To better understand the connection between the Thalamus-driven Decoupling Network (TDN) and CVH-TDN, we present a clearer and more detailed explanation as: In DABP, the black-box predictor resembles a cognitively impaired “person” who possesses only partial knowledge and is prone to errors. As a result, the target model’s learning process is inevitably influenced by this noisy guidance from the black-box predictor. To address this issue, we draw inspiration from neuroscience, where such conditions are categorized as hallucination disorders: situations in which useful and interfering information are blended, preventing the brain from making reliable decisions. The TDN offers a comprehensive analysis of the full cycle of hallucination disorders and how to address them. Based on this, prior work [1] divides the TDN process into three stages from the perspective of human attention: hallucination manifestations, human responses, and lesion treatment. Correspondingly, we divide CVH-TDN into three core modules that align with the three stages: Hallucination Generation (HG), Hallucination Alignment (HA), and Hallucination Calibration (HC). Guided by the key cognitive impairments, HG generates hallucination images by simulating the locations of hallucination pathology, thereby masking some noise from the black-box predictor and facilitating subsequent HA and HC to enhance the model’s learning ability using these images. HA enhances feature recognition by mimicking how humans manage cognitive impairments, while HC draws on principles of neurological therapy to guide unreliable data through reasoning based on reliable feature representations.\n\nQ2: The methodology section is difficult to follow ... highlighting the core idea would help improve clarity.\n\nA2: We appreciate the reviewer’s valuable suggestion. We have revised and simplified the existing model diagram to make it more intuitive. For example, by adjusting the overall proportion of the prediction module and adding more detailed descriptions for each component. We have also revised some representations to make it easy to follow. All the revisions will be included in the final version.\n\nQ3: In Equation (9), what is the rationale for sorting the weights w? Why is this operation necessary in the context of the method?\n\nA3: We thank the reviewer for the insightful feedback. in Eq. (9), the sorting operation is originally introduced to prevent potential information misalignment caused by changes in sample correspondence during training. However, since Eq. (5) inherently ensures one-to-one alignment, further experimental results show that the presence or absence of the sorting step has negligible impact on the final performance. Based on this observation, we have simplified Eq. (9) to a more concise form, denoted as $w=\\exp (\\log (sim({{M}^{(2)}},norm({{Z}^{g}}))))$.\n\nQ4: The loss formulation in Equation (12) appears adversarial ... how does the model effectively learn under such a setting?\n\nA4: Eq. (12) is composed of bidirectional losses: $L_{HA}^{forward}$ encourages samples whose features are similar to be aligned with the generated version in the feature space; $L_{HA}^{back}$ calibrates $L_{HA}^{forward}$ and reduces the noise caused by common features to prevent overfitting of the model. During the optimization process, the weight $w$ dynamically evaluates the importance of features to determine the influence of $L_{HA}^{forward}$, thereby maintaining a balance between $L_{HA}^{forward}$ and $L_{HA}^{back}$. These descriptions can be found on lines 211-223. To better understand their relationship and support effective model learning in this setting, we have added ablation experiments on Office-31 to validate their correlation:\n\n| Office-31 | A→D | A→W | D→A | D→W | W→A | W→D | Mean |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | \n| CVH-TDN w/o Eq. (12) | 92.9 | 90.2 | 73.9 | 98.6 | 78.9 | 99.4 | 89.0 |\n| CVH-TDN w/o $L_{HA}^{forward}$ | 94.1 | 90.9 | 74.1 | 98.9 | 78.6 | 99.4 | 89.3 |\n| CVH-TDN w/o $L_{HA}^{back}$ | 92.9 | 89.6 | 72.1 | 98.5 | 76.4 | 99.6 | 88.1 |\n| CVH-TDN | 96.4 | 92.8 | 75.6 | 98.9 | 81.0 | 99.6 | 90.7 |\n\nWe observe that when only $L_{HA}^{forward}$ is applied, the lack of constraint from $L_{HA}^{back}$ leads to rapid alignment between original and generated samples. This causes the model to be heavily influenced by common features during training, resulting in a potential performance drop. Since $L_{HA}^{back}$ is specifically designed to complement $L_{HA}^{forward}$ by mitigating such interference during alignment, using $L_{HA}^{back}$ alone does not effectively improve the training process, and thus brings little to no performance gain. Therefore, the joint effect of bidirectional losses is essential for enabling efficient learning in this setting.\n\nQ5: The calibration section is somewhat unclear ... its theoretical or empirical justification?\n\nA5: In the calibration section, Eq. (16) can be divided into two parts: Hsc and Cfe. The Hsc is to calibrate samples with similar signature features, while the Cfe is to separate ambiguous samples with similar common features. In Hsc, the high-reliability samples are distinguished by the previous hierarchical strategy to cluster the low-reliability samples that are close in spatial distance. In Cfe, a regularization loss is introduced to enhance the difference in the representation of different samples in the feature space, and the generated hallucination image is forcibly aligned with the original image to guide the separation of different classes of samples. More detailed descriptions of the intended mechanism can be found on lines 239-257. To verify these judgments, as shown in Figure 4, both Hsc and Cfe of $L_{HC}$ contribute to improving the reasoning ability of the model by highlighting the areas that the model focuses on for a given category. When Hsc is removed, the values of the regions of interest in the model are averaged, resulting in a lot of non-main features being captured. When Cfe is removed, the ability to grasp key features is weakened, resulting in some key features not being captured.\n\nQ6: Why are the losses from the three stages ... train the model in separate stages?\n\nA6: Please refer to Figure 6. Unlike UDA, under the DABP setting, we do not have access to the source model and can only obtain noisy target labels via API, which prevents us from performing training based on the source model. From a setup perspective, the DABP training consists of only one stage—adapting the model to the target domain. From a methodological standpoint, CVH-TDN can be divided into two sub-stages: HG corresponds to the hallucination manifestations stage in TDN, while the combination of HA and HG corresponds to the hallucination learning stage. The combined update of loss reflects the practical implementation of our optimization strategy. From both a causal and pathological perspective, the hallucination learning stage necessarily follows the hallucination manifestations stage, during which human responses and lesion treatment occur concurrently. This process closely mirrors how hallucinations are understood and handled in TDN. Training the model in separate stages incurs higher computational costs, and its effectiveness still requires further validation. We plan to explore this direction in future work.\n\nQ7: I am also curious about the API used ... rather than the proposed method itself.\n\nA7: Please refer to Tables 1, 2, 3 and 6, under fair comparison conditions and with the same closed-source model, all results demonstrate that CVH-TDN significantly outperforms previous SOTA methods, highlighting the superiority of our approach. To further investigate whether the performance gain is primarily due to the strength of the source model, we conducted additional experiments under a more powerful multi-source setting and further enhanced the source model using the domain generalization (DG). To ensure a fair comparison, we select the same ResNet-50 for the black-box predictor learning on Office-Home under multi-source DABP setting. We select the DG method Swad [2] mentioned to train the source model as a strong cloud API, while following the DINE [3] and SEAL [4] strategy to simply train a source model as a weak cloud API:\n\n| Office-Home | → Art | → Clipart | → Product | → Real_World | Mean |\n| --- | --- | --- | --- | --- | --- |\n| weak | 54.9 | 49.9 | 69.6 | 76.7 | 62.8 | \n| DINE (weak) | 74.9 | 62.6 | 84.6 | 84.7 | 76.7 |  \n| SEAL (weak) | 75.3 | 66.4 | 86.9 | 85.2 | 78.5 |  \n| CVH-TDN (weak) | 74.6 | 73.1 | 91.3 | 85.1 | 81.0 |\n| strong | 65.3 | 57.4 | 78.8 | 81.0 | 70.6 |\n| DINE (strong) | 75.4 | 64.2 | 85.1 | 84.9 | 77.4 |  \n| SEAL (strong) | 75.7 | 67.1 | 86.8 | 86.1 | 78.9 |  \n| CVH-TDN (strong) | 75.0 | 75.3 | 91.8 | 86.7 | 82.2 |\n\nAs shown in the above table and the main results throughout the paper, CVH-TDN consistently demonstrates superior performance across different models and settings. While the performance gains are partially influenced by the strength of the source model, the primary contributing factor remains the learning capability of CVH-TDN itself.\n\nWe appreciate the reviewer’s feedback and hope our response has addressed the reviewer’s concerns.\n\n[1] Understanding visual hallucinations: A new synthesis. Neur. Bio. Rev, 2023\n\n[2] Swad: Domain generalization by seeking flat minima. In NeurIPS, 2021\n\n[3] DINE: Domain Adaptation from Single and Multiple Black-box Predictors. In CVPR, 2022\n\n[4] A Separation and Alignment Framework for Black-Box Domain Adaptation. In AAAI, 2024"}, {"review_id": "gJSNdpS58E", "reviewer": "Reviewer_7sjS", "summary": "The paper identifies two limitations of the existing black-box source-free UDA methods: (1) they focus too much on obtaining highly reliable samples or estimates, which can lead to overfitting, (2) they typically ignore insightful information in samples that are not reliable, which affects the model's generalizability to the target data. To address these limitations, the paper proposed visual hallucination learning, called controlled visual hallucination via thalamus-driven decoupling network (CVH-TDN), to explore spatial relations between real samples and hallucinations. The CVH-TDN comprises a hallucination generator (for generating hallucination images), hallucination alignment (for aligning target samples and the corresponding hallucination-generated image), and hallucination calibration (for dynamically updating adaptive label smoothing). Four benchmark datasets (Office-31, Office-Home, VisDA-17, and DomainNet) are utilised to conduct experiments, demonstrating the effectiveness of CVH-TDN.", "questions": "What is the computational cost of CVH-TDN? Particularly relating to the two memory storage containers adopted by CVH-TDN.", "limitations": "The authors should address the computational cost of CVH-TDN as a limitation, as researchers with limited computational resources may not be able to adopt CVH-TDN for their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "****Strengths****\n\n(1)  Introducing the thalamus-driven decoupling network (TDN) into black-box source-free UDA is interesting. This is the first time TDN has been applied in a vision task, particularly in source-free domain adaptation, leveraging TDN's connection with hallucination to guide sample generation within a feature space. \n\n(2)  The paper is well-written, well-presented, easy to follow and comprehend. The proposed method, CVH-TDN, is detailed and well-explained.\n\n(3)  The authors conducted extensive experiments on four benchmark datasets to demonstrate the effectiveness of the proposed CVH-TDN. The results show that CVH-TDN achieves an impressive performance across all the datasets and outperforms several SoTA methods in both overall-average accuracy and on-hard task average accuracy. Ablation studies on two benchmark datasets further demonstrate the effectiveness of each component of CVH-TDN. \n\n****Weaknesses****\n\nIn addition to hallucination image generation,  CVH-TDN has two memory storages: memory structure ***M*** and smooth adaptive storage ***S***, which indicates that CVH-TDN is computationally expensive. The authors did not address the computational cost of CVH-TDN.", "was_revised": true, "final_justification": "I have read the authors' responses to the issues I raised and the ones raised by the other reviewers. The authors have done well to  address all concerns, as the other reviewers gave positive comments on the authors' responses.  Thus, I maintained my positive score of 'Accept'. Thank you", "rebuttal": "We sincerely appreciate the reviewer’s constructive suggestions and have answered them as follows with our highest respect.\n\nQ1: The authors should address the computational cost of CVH-TDN as a limitation, as researchers with limited computational resources may not be able to adopt CVH-TDN for their work.\n\nA1: Please refer to Table 7 in Appendix, we have recorded the average runtime cost, the maximum GPU space usage, and the best accuracy of each comparison method on the VisDA-17 dataset. The Results of cost comparison on the VisDA-17 with the ResNet-101 backbone as following:\n\n| Method | Time(s/epoch) | Space(MiB) | Accuracy(%) |\n| --- | --- | --- | --- | \n| DINE | 124s | 9881MiB | 75.6 |\n| BETA | 1101s | 20247MiB | 85.1 |\n| SEAL | - | Over 24G | 89.2 |\n| CVH-TDN | 205s | 10589MiB | 90.6 |\n\nFor a fair comparison, we set the same running conditions (e.g., batch size = 64, num workers = 4, etc.) in the compared works on a machine with an NVIDIA GeForce RTX4090 GPU. Compared with other methods, CVH-TDN improves performance through the optimized framework while reducing training runtime cost and GPU space usage. This enables researchers with limited computing resources to adopt CVH-TDN for their work.\n\nQ2: What is the computational cost of CVH-TDN? Particularly relating to the two memory storage containers adopted by CVH-TDN.\n\nA2: The computational cost of CVH-TDN refers to Q1. For our memory structure $M$ and smooth adaptive storage $S$, we supplemented additional experiments on the VisDA-17 using an NVIDIA GeForce RTX4090 GPU to verify their space costs: $M$ uses 57.05 MiB of GPU space usage; $S$ uses 2.54 MiB of GPU space usage. Since they do not require gradient storage or updates during training, their GPU space usage is nearly negligible. \n\nWe appreciate the reviewer’s feedback and hope our response has addressed the reviewer’s concerns."}, {"review_id": "AvvnWUqLdN", "reviewer": "Reviewer_SF5Z", "summary": "The paper presents a novel approach for Domain Adaptation of Black-box Predictors (DABP) through a framework called Controlled Visual Hallucination via Thalamus-Driven Decoupling Network (CVH-TDN). CVH-TDN introduces three main components: Hallucination Generation, Hallucination Alignment, and Hallucination Calibration, designed to enhance the model's ability to generalize by effectively utilizing both reliable and unreliable samples. Extensive experiments demonstrate that CVH-TDN achieves state-of-the-art performance across multiple benchmarks.", "questions": "- How does the thalamus-inspired design specifically address the limitations of prior DABP methods? Could the biological analogy be simplified for broader ML audiences?\n- How sensitive is the performance of CVH-TDN to the choice of hyperparameters? Are there recommended settings based on the experiments?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The introduction of a thalamus-driven decoupling network in the context of DABP is innovative and offers a unique perspective on handling domain adaptation challenges.\n- The three-module design (HG, HA, HC) is well-structured and addresses key limitations of prior work (e.g., overfitting on reliable samples and ignoring unreliable ones).\n- The authors conduct extensive experiments on four standard benchmarks, providing strong evidence of the effectiveness of their proposed method.\n\nWeaknesses:\n- The proposed method may be complex, which could hinder its adoption in practical applications. Simplifying the framework or providing clearer guidelines for implementation might help.\n- While the paper discusses computational efficiency, it would benefit from more detailed comparisons of runtime and resource usage against other methods.", "was_revised": true, "final_justification": "Most of my concerns have been addressed, so I am willing to maintain my original positive score.", "rebuttal": "We sincerely appreciate the reviewer’s constructive suggestions and believe that the additional analysis and explanations significantly improve the quality of our submission. \n\nQ1: The proposed method may be complex, which could hinder its adoption in practical applications. Simplifying the framework or providing clearer guidelines for implementation might help.\n\nA1: Although our method may appear complex in design, each component has been streamlined into a plug-and-play module for practical implementation. Specifically, HG is an integrated generation module that produces pseudo-images directly from raw inputs; HA is a data integration module that can be combined with HG and takes raw images and a target model as input to compute the optimized loss; HC is a DABP task design module that works in conjunction with HG and HA to accomplish the final objective. In our released code, all three modules are modularized and independently packaged to simplify real-world deployment. In addition, we have revised and simplified Figure 3 to make the method more intuitive. For example, by adjusting the layout of the prediction module and adding more detailed descriptions for each component. Upon acceptance of the paper, we will release the code with clear implementation guidelines to facilitate reproducibility and further research by the community.\n\nQ2: While the paper discusses computational efficiency, it would benefit from more detailed comparisons of runtime and resource usage against other methods.\n\nA2: Please refer to Table 7, we have recorded the average runtime cost, the maximum GPU space usage, and the best accuracy of each comparison method on the VisDA-17 dataset. As described in lines 747-754, when adapting VisDA-17, it is worth noting that BETA is divided into two stages that are highly computationally intensive: the first stage is the initialization, which requires initialization of the two models due to their mutually-distilled network structures; the second stage is the two-step process, which requires distillation and fine-tuning for each epoch. SEAL is highly resource-intensive, and its official code cannot complete the adaptation task on VisDA-17 under the same conditions with 24GB GPU memory. The Results of cost comparison on the VisDA-17 with the ResNet-101 backbone as following:\n\n| Method | Time(s/epoch) | Space(MiB) | Accuracy(%) |\n| --- | --- | --- | --- | \n| DINE | 124s | 9881MiB | 75.6 |\n| BETA | 1101s | 20247MiB | 85.1 |\n| SEAL | - | Over 24G | 89.2 |\n| CVH-TDN | 205s | 10589MiB | 90.6 |\n\nCompared to other methods, CVH-TDN calculates the model area of interest and generates attention-specific masking blocks with minimal cost, which does not require fine-tuning or a computationally expensive generation network. To further assess the resource usage of our memory module, we conducted additional experiments for verification: $M$ uses 57.05 MiB of GPU space usage; $S$ uses 2.54 MiB of GPU space usage. The results demonstrate that our CVH-TDN improves performance through an optimized framework, while also reducing training runtime cost and GPU space usage, highlighting the overall effectiveness and efficiency of our approach.\n\nQ3: How does the thalamus-inspired design specifically address the limitations of prior DABP methods? Could the biological analogy be simplified for broader ML audiences?\n\nA3: Our thalamus-inspired design addresses the limitations of previous DABP methods from two perspectives: the methodological framework and the task-specific level. For the methodological framework, CVH-TDN builds upon a thalamus-driven decoupled network that models the cognition–hallucination relationship, further advancing the exploration of interactions between real and hallucinated samples in the feature space. Different from the previous DABP methods, CVH-TDN leverages sample spatial similarity instead of suppressing targeted sample information, thereby improving model. Please refer to Figure 7, CVH-TDN improves the learning principle of the previous DABP method. For the task-specific level, as detail in lines 230-238, we found that currently commonly used DABP task-specific operations have some limitations and proposed new operations to replace them. As shown in 692-702 and Figure 8, experimental results show that this improvement significantly enhances the compared DABP methods.\n\nTo convey the simplified biological analogy to a broader ML audience, we have illustrated and explained “Conceptual figure of CVH-TDN. The black-box predictors resemble agents with prior knowledge but lack the ability to perform targeted discrimination. HG controls mask formation by modeling the location where hallucinations are pathologically generated, driven by the key cognitive impairments observed in TDN. HA improves feature discrimination by simulating how humans deal with cognitive impairments. HC draws on neurotherapeutic principles to guide unreliable data through reasoning using reliable feature representations” in Figure 2. As shown in lines 73-86, We provide a more targeted and accessible explanation tailored to the broader ML audience.   \n\nQ4: How sensitive is the performance of CVH-TDN to the choice of hyperparameters? Are there recommended settings based on the experiments?\n\nA4: Please refer to Figure 4, we have conducted a detailed parameter analysis on our hyperparameter $\\lambda$. For the hyperparameters $\\mu $ and $\\tilde{\\mu}_s$ of our task-specific improvements, our parameter sensitivity analysis is presented in Figure 8, our method achieves the best overall performance when $\\mu = 0.5$ and $\\tilde{\\mu}_s = 0.95$, while it shows the strongest improvement over other methods when $\\mu = 0.5$ and $\\tilde{\\mu}_s = 0.8$. Please refer to Figure 5, we have conducted parameter analysis on our hyperparameters $r$ and $\\phi$ on the VisDA-17. In addition, we have supplemented a more detailed parameter sensitivity analysis as follows:\n\n| CVH-TDN | $r$=0 | $r$=0.3 | $r$=0.5 | $r$=0.8 |\n| --- | --- | --- | --- | --- | \n| $\\phi$=3 | 81.21 | 88.31 | 88.23 | 86.42 |\n| $\\phi$=4 | 81.42 | 89.12 | 89.50 | 87.56 |\n| $\\phi$=5 | 82.11 | 89.99 | 90.07 | 89.62 |\n| $\\phi$=6 | 82.46 | 90.22 | 90.58 | 89.53 |\n| $\\phi$=7 | 82.51 | 90.51 | 90.46 | 88.52 |\n| $\\phi$=8 | 81.99 | 90.36 | 90.40 | 87.39 |\n\nTherefore, we recommend using $\\lambda$=10%, $r$=0.5, and $\\phi$=6. In all scenarios, we keep these parameter settings the same. For other common parameters, refer to the Implementation Details section.\n\nWe appreciate the reviewer’s feedback and hope our response has addressed the reviewer’s concerns."}, {"review_id": "ycFC7ma8p3", "reviewer": "Reviewer_ATW6", "summary": "This paper tackles the task of Domain Adaptation of Black-box Predictors (DABP), which is practical in applications. \nThe authors claim that previous DABP methods face two major challenges: premature overfitting from over-relying on reliable samples, and neglecting valuable information in unreliable ones. To counter this, the authors propose CVH-TDN, which introduces a thalamus-driven decoupling network to visual tasks, leveraging its connection to hallucination for controlled sample generation in feature space. Comprising Hallucination Generation, Alignment, and Calibration, CVH-TDN explores spatial relationships between samples and hallucinations. Experiments on several datasets show improved performance.", "questions": "- As the patch size is $p$, is $z_{i,j}^k$ a matrix  or a scalar?\n- In Eq. (2), how is the \"/\" performed, as $Z$ is a feature map?\n- Could you explain why the consistency between $x^g$ and $x$ in Eqs.(10-12) beniefit model adaptation?\n- What is the black box model in the experiments?", "limitations": "The authors have discussed the limitations in Appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n- Domain Adaptation of Black-box Predictors (DABP) is practical in real-world applications, as we often can only access the black box model provided in the source domain.\n- The concept of controlled visual hallucination and thalamus-driving is new to domain adaptation, and the connection of domain adaptation and neural science is interesting.\n- The paper is ok to read with a clear figure to illustrate the idea.\n- Experiments show improved results over methods previously compared.\n\n## Weakness\n- The motivation for introducing the hallucination and thalamus to the black box DA is not clear. I suggest that the authors include the tight connection between DABP and hallucination/thalamus in the paper (introduction and methods). Specifically, why hallucination/thalamus suite for DABP, and how the challenges of DABP could be tackled by TDN.\n- Another major weakness is that too many symbols are introduced, making the paper hard to read. I believe many symbols are not necessary. Meanwhile, there lack motivation for the equations. For example, Eqs. (6-12) are not mainly listed without the reasons for why to define such equations. \n- Too many hyperparameters are introduced. How to set these hyperparameters?", "was_revised": true, "final_justification": "After the rebuttal, the motivations and readability are improved. I believe the paper meets the NeurIPS standard.", "rebuttal": "We sincerely appreciate the reviewer’s constructive suggestions and have answered them as follows with our highest respect.\n \nQ1: The motivation for introducing the hallucination and thalamus to the black box DA is not clear. I suggest that the authors include the tight connection between DABP and hallucination/thalamus in the paper (introduction and methods). Specifically, why hallucination/thalamus suite for DABP, and how the challenges of DABP could be tackled by TDN.\n\nA1: Please refer to lines 66-72, we have illustrated and explained the motivation for introducing the hallucination and thalamus to the black-box DA. Meanwhile, the influence of the TDN framework on the implementation of our method and how to address the challenges of DABP are detailed in lines 74–85 and illustrated in Figure 2. To address the reviewer’s concerns and enhance the clarity of our contributions, we present a clearer and more detailed explanation as: In DABP, the black-box predictor resembles a cognitively impaired “person” who possesses only partial knowledge and is prone to errors. As a result, the target model’s learning process is inevitably influenced by this noisy guidance from the black-box predictor. To address this issue, we draw inspiration from neuroscience, where such conditions are categorized as hallucination disorders: situations in which useful and interfering information are blended, preventing the brain from making reliable decisions. The TDN offers a comprehensive analysis of the full cycle of hallucination disorders and how to address them. Based on this, prior work [1] divides the TDN process into three stages from the perspective of human attention: hallucination manifestations, human responses, and lesion treatment. Correspondingly, we divide CVH-TDN into three core modules that align with the three stages: Hallucination Generation (HG), Hallucination Alignment (HA), and Hallucination Calibration (HC). Guided by the key cognitive impairments, HG generates hallucination images by simulating the locations of hallucination pathology, thereby masking some noise from the black-box predictor and facilitating subsequent HA and HC to enhance the model’s learning ability using these images. HA enhances feature recognition by mimicking how humans manage cognitive impairments, while HC draws on principles of neurological therapy to guide unreliable data through reasoning based on reliable feature representations. \n\n\nQ2: Another major weakness is that too many symbols are introduced, making the paper hard to read. I believe many symbols are not necessary. Meanwhile, there lack motivation for the equations. For example, Eqs. (6)-(12) are not mainly listed without the reasons for why to define such equations.\n\nA2: We thank the reviewer for the insightful feedback. In the revised version, we have simplified symbols to make the paper easy understand as possible. For example, we have removed the sorting operation and modified the original Eq. (9) to $w=\\exp (\\log (sim({{M}^{(2)}},norm({{Z}^{g}}))))$. \n\nFor the motivation of the equations, we have provided detailed explanations for the motivation behind each equation. Eqs. (6)–(8) are designed to clarify the operations in Eq. (5), and their purpose directly supports the design of Eq. (5). The motivation of Eq. (5) is to control the movement direction of the samples on the feature space by acquiring the sample state immediately, which has been elaborated in detail in lines 196-200. Eq. (9) is to assign different weights by pairing features based on the similarity between the original image and its corresponding generated image, which has been explained in lines 211-212. Eq. (10) encourages samples with similar features to be aligned with their generated counterparts in the feature space, as detailed in lines 221-222. Eq. (11) calibrates Eq. (10) and reduces the noise caused by common features to prevent overfitting of the model, as detailed in lines 222-223. Finally, Eq. (12) integrates Eqs. (10) and (11), as detailed in lines 220. Therefore, the motivations behind all proposed equations have been clearly explained in this paper.\n\nQ3: Too many hyperparameters are introduced. How to set these hyperparameters?\n\nA3: Although our method involves several hyperparameters, most of them are fixed, and we have conducted comprehensive analyses for each to ensure their effectiveness. Please refer to Figure 4, we have conducted a detailed parameter analysis on our hyperparameter $\\lambda$. Please refer to Figure 5, we have conducted parameter analysis on our hyperparameters $r$ and $\\phi$ on the VisDA-17. Therefore, we recommend using $\\lambda$=10%, $r$=0.5, $\\phi$=6. In all scenarios, we keep these parameter settings the same. Please refer to Figure 8 and lines 692-702, the hyperparameters $\\mu $ and $\\tilde{\\mu}_s$ are task-specific to DABP, yet their transferability and generality have been validated through integration with existing methods, thereby advancing previous learning strategies in DABP. The parameter sensitivity analysis is presented in Figure 8, our method achieves the best overall performance when $\\mu = 0.5$ and $\\tilde{\\mu}_s = 0.95$, while it shows the strongest improvement over other methods when $\\mu = 0.5$ and $\\tilde{\\mu}_s = 0.8$. For other common parameters, refer to the Implementation Details section.\n\nQ4: As the patch size is $p$, is $z_{i,j}^{k}$ a matrix or a scalar?\n\nA4: $z^{k}$ is the $k$-th dimensional feature and $z_{i,j}^{k}$ is the feature scalar corresponding to the i-th row and j-th column.\n\nQ5: In Eq. (2), how is the \"/\" performed, as $Z$ is a feature map?\n\nA5: In Eq. (2), \"/\" is a division operation, $Z$ is the $K$-dimensional transitional output from the bottleneck layer. $Z$ stores ${z_{h,w}^{k}}$ as a matrix, where each $z_{h,w}^{k}$ is a scalar at the corresponding position in $Z$. The definitions of $h$, $w$, and $k$ are provided in lines 168-170. To conserve writing space in the main text, Eq. (2) is simplified and presented as: \n\n$Att_{i,j} = max\\left( Tanh\\left(  \\sum\\limits_n \\sum\\limits_k a_k^n z_{i,j}^k /Z \\right), 0 \\right)$.\n\nHowever, the simplified formulation is not entirely precise and may result in some confusion. The accurate formulation of Eq. (2) is as follows: \n\n$Att_{i,j} = max\\left( Tanh\\left(  \\sum\\limits_n \\sum\\limits_k a_k^n z_{i,j}^k /\\sum\\limits_{h}\\sum\\limits_{w}{\\sum\\limits_{k}{z_{_{h,w}}^{k}}}) \\right), 0 \\right)$.\n\nWe consider that the simplified formulation of Eq. (2) might have caused confusion for the reviewer. To clarify, we plan to add the detailed explanation of $Z$ and modify the original Eq. (2) to the accurate formulation in the final version.\n\nQ6: Could you explain why the consistency between $x$ and $x^{g}$ in Eqs. (10-12) beniefit model adaptation?\n\nA6: According to previous studies [1, 2], aligning different versions of generated images with the original ones can enhance model generalization. Therefore, how to generate informative images and efficiently align them with the original images has become a highly active research direction in recent years. Our HG is to generate informative images, and our HA is to efficiently align generated images with the original images. We use contrast learning in HA to explore the spatial relationship between the target sample $x$ and the corresponding generated image $x^{g}$. As shown in lines 221-223, in HA, Eq. (10) encourages samples $x$ whose features are similar to be aligned with the generated version $x^{g}$ in the feature space; Eq. (11) calibrates Eq. (10) and reduces the noise caused by common features to prevent overfitting of the model. Eq. (12) integrates Eq. (10) and Eq. (11) and serves as the target optimization loss of HA. As shown in Figure 4 and Table 5, HA can effectively improve model generalization only when combined with HG. As is well known, enhancing the generalization ability of a model can most directly improve its adaptability.\n\nQ7: What is the black box model in the experiments?\n\nA7: In our paper, we do not mention the term \"black box model\". We assume that the reviewer’s question might be aiming to clarify what is meant by a \"black-box predictor\" in this context. In DABP setting, it adapts a model using only the unlabeled target data and a black-box predictor trained on the source domain, e.g., an API service in the cloud. Please refer to Figure 6, the black-box predictor refers to a model trained solely on source domain data and deployed on a cloud server, where its internal architecture is inaccessible and predictions can only be obtained via an API interface.\n\nWe appreciate the reviewer’s feedback and hope our response has addressed the reviewer’s concerns.\n\n[1] CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, 2018.\n\n[2] Vdm-da: Virtual domain modeling for source data-free domain adaptation. TCSVT, 2022."}]

Author rebuttal:

Meta-review: This work considers domain adaptation with Black-box Predictors (DABP).  The paper points out that existing black-box source-free UDA methods risk overfitting by relying only on highly reliable samples and neglecting less reliable ones, which harms generalization. To overcome this, it proposes Controlled Visual Hallucination via Thalamus-Driven Decoupling Network (CVH-TDN) to exploit spatial relations between real and hallucinated samples.

Almost all reviewers believe this work is novel and performance is stronge, and can be accepted.

I recommend to accept this work as a poster.
Decision: Accept (poster)