Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation
Authors: Mingjia Li
Shuang Li
Tongrui Su
Longhui Yuan
Jian Liang
Wei Li
Venue: NeurIPS 2024 poster (2024)
Keywords: test-time adaptation
diffusion models
generative models
classification
segmentation
Primary area: other
TL;DR: We introduce DUSA, an effective method with improved efficiency for test-time adaptation that exploits structured semantic priors from pre-trained score-based diffusion models for enhancing pre-trained discriminative models.

Abstract: Capitalizing on the complementary advantages of generative and discriminative models has always been a compelling vision in machine learning, backed by a growing body of research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion score to facilitate the test-time adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowledge from a single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of our DUSA in adapting a wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted to dissect the pivotal elements in DUSA. Code is publicly available at https://github.com/BIT-DA/DUSA.

Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation
Mingjia Li
Shuang Li*
Tongrui Su Beijing Institute of Technology
Beijing Institute of Technology mingjiali@bit... edu.
Beihang University shuangli baigggiiaiae abuabaa..... edu.
molarsudbit. www.ibbttteuu...... edu.ca cn
Longhui Yuan Beijing Institute of Technology longhulyuan@bit. edu.. en
Jian Liang
Wei Li'
Kuaishou Technology
Inceptio Technology 1iweimce@gmal.com com
1iangj lannnngjnnO@kuassshhooo com
Abstract
Capitalizing on the complementary advantages of generative and discriminative models has always been compelling vision in machine learning, backed by growing body research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowl- edge from single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of DUSA in adapting wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted dissect the pivotal elements DUSA. Code publicly available /tttp:://gitnnn com - -AA/SITDAAADUSAA
to facilitate the test-time
Introduction
The combination of generative and discriminative modeling has always been appealing due their distinct nature in data comprehension [ 3
4) Discriminative models are adept at making accurate 10 11]. but can be fragile when confronted with unseen data 12). This vulnerability can be attributed to their tendency to learn spurious correlation as shorteut, hindering their transferability (I3). Generative models, however, are proficient in capturing the underlying structure of the data, giving them an edge in grasping the whole picture [14 15 16 and
predictions training data (5 7 8
enhancing robustness [7 18 Prior works have verified the effectiveness of generative objectives in yet the utilization of pre-trained generative models is under-explored.
discrininative learning
The recent surge of diffusion models [20
21 [22] has ignited interest in adopting them for applications 26 26]
In the context of test-time adaptation, pre-trained task model is updated the fly make accurate predictions
beyond image generation [23] 24
25
incoming target samples without access their labels. This presents challenges, as the target data distribution may differ from that encountered during pre-training. The literature reveals that
not only extract discriminative features from capacious diffusion models [23 24 26 27 28 but also convert these models into generative nut-of-distribution samples [25 29 30).
classifiers that demonstrate human-level generalization
Such properties render them viable choices for facilitating the test-time adaptation of discriminative models, which may underperform unseen data (T1. Diffusion-TTA (32 ranks among the first
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to employ diffusion models for test-time udaptation, where task outputs are used to modulate the conditioning of a diffusion model with the objective of likelihood maximization. While Diffusion- TTA is competitive, there's s still room to unleash the full potential of diffusion models. To achieve this, two key aspects warrant further exploration. First, the conditioning space is typically low-dimensional in diffusion models (20) 22], which restricts its ability to capture the intricacies of complex data and thus impedes its expressiveness as a discriminative prior. Further, the common image-level condition lacks a fine-grained connection with data, limiting its potential in guiding dense prediction tasks. Conversely, the high-dimensional latent space of diffusion models exhibits a surprisingly interpretable semantic structure [33
35 36 37,. making it a good fit for assisting discriminative tasks and easily extensible to dense prediction. Second, Diffusion- TTA is heavily reliant on the Monte Carlo method over many as 180 timesteps to estimate biased approximation of likelihood resulting in high computational complexity proportional to sampled timesteps.
34
32 38).
With this work, we aim to boost test-time adaptation performance by digging into the semantic structure of diffusion models in the latent space, while lifting reliance on the Monte Carlo sampling of timesteps. Although few works elucidate the semantic properties of the latent space [34, 37] they all take a generative viewpoint and are not tailored for discriminative tasks. We instead depart
from the perspective of score functions (39 40 41] the latent space, which is closely related to the
denoising diffusion formulation 38
42
43 Our method features exploring the structured semantic
priors underlying sionUsion models as Score estimators for test- time Adaptation (DUSA).
Concretely, we start by providing theoretical illustration of the semantic structure underneath the score functions Vx p(x y), where the conditional probability p(y x) is implicitly embedded. The theoretical findings not only unveil discriminative priors hidden within score-based diffusion models, but applies to every single timestep and avoids likelihood estimation. A test-time objective is then derived by substituting the pre-trained task model and difftusion model for the implicit priors and score functions, respectively. Intuitively, the precise score estimation by diffusion models forms well-structured semantic space, where the task model can learn implicit discriminative priors. Given their generative nature, the priors are further blessed with improved robustness. ultimately benefiting task prediction. Another key advantage of our approach lies in shifting computational complexity from timesteps the number of classes, which aligns closely with our focus discriminative tasks. Thereby, a more efficient adaptation scheme can be enabled through our practical designs.
Besides, the capacity of our DUSA is testified across a variety of task model families, test-time adapta- tion protocols, and task categories. Our DUSA consistently outperforms the competitive counterparts in adapting pre-trained classifiers with different backbones to out-of-distributio scenarios, whether in the mild protocol of data from a single domain (44) or the more challenging one with a continually changing datastream [45). We also showcase the versatility of our DUSA by applying it almost as-is to test-time semantic segmentation. All diffusion models employed are trained on the corresponding source domain of the task model. Extensive analyses of the components in method back the validity of DUSA and underline the benefits of borrowing knowledge from generative modeling.
Our main contributions can be summarized as:
A novel proposition is given from a theoretical perspective to extract discriminative priors from score-based diffusson models, which are timestep-basee and versatile enough to handle both classification and dense prediction tasks at test time.
Inspired by the proposition and enhanced by practical designs, our DUSA effectively leverages the structured semantic priors and rivals in test-time adaptation with improved efficiency.
DUSA outperforms the best existing methods by +5.1% and -7.3% in fully and continual test- time adaptation on ConvNeXI-L and +4.2% in test-time semantic segmentation on SegForrr-- validating the excellence of our method in extracting valuable priors from diffusion models.
2 Preliminaries
Test-time adaptation. A model well-trained on source data can face severe performance degrada- tion on -ntro--distribution (OOD) target samples. To tackle this, test-time adaptation (TTA) [44] is proposed to boost model performance at inference time. Formally, an off-the-shelf model sx) pre-trained on labeled source data Ds = {(x;, y:) y adopted as the task model, where the source data follows probability distribution x; ~ Ps (x) and is inaccessible during adaptation. TTA aims

---

Tash prediction
Logil
Eop Σ, (400)x))
Quary selected
Input
Task Model fo
Agoroggtted Eog.
classes
CBM: Candidate Selection Module
Condition (cy
Logil
Seledeed
Nomalloo
Sofmaa
D1p-2 select
Noiay
Diflusion eg
nulliomial solcet
Figure 1: Overview of DUSA. Our method adapts a discriminative task model fo with generative diffusion model 6g. Given image Xg at test-time, the task model outputs logits. To improve efficiency, devise CSM to select classes to adapt and return their probabilities (probs). The embeddings of the classes are then queried as diffusion model conditions, yielding conditional noise predictions from noisy image Xx. The aggregated noise Éa is then constructed from ensembling conditional noises with probs, which aligned with the added noise e following Eq. (TO). Both models are updated.
pushing the limits of model performance on unlabeled target data D7 { on the fly, where target data follows xj ~ (x)) and Px(x) + Px(x). With batched target data arriving online, we obtain predictions from the task model fx(x) and update it on live target samples without labels.
Diffusion models. Diffusion models excel at modeling data distribution p(x) by learning to restore the gradually destroyed data structure [20] 38 46). For diffusion models, forward and reverse process is defined. In the forward process, small Gaussian noise is iteratively applied to real data X
g gx, x;-1) = (xx;; v 1,
where {8, 10)/) is a variance schedule defining the noise added at each timestep t. With the reparameterizztion trick, the noisy version of Xg can be directly obtained in single step:
X Vāxxa + vI ãie, c (O,II),
where 0z 1 Br, a. Ogg, and the sampled noise is of the same dimensionality Xo- (x,, f, c) is trained to predict the noise added to x, with condition c, minimizing the simplifee lifed denoising objective:
In the reverse process, conditional diffusion model 20 21 22
Csimpte (Ø) = (x),,)), [||e t,
3 Structured Semantic Priors in Diffusion Score for Test-Time Adaptation
In this section, first review relevant method (Sec. 3.1 then provide the theoretical insight behind our DUSA (Sec. 3.2). At last, we advocate a few practical designs with efficiency in mind (Sec. (Sec.B.3) The framework of DUSA is illustrated in Fig. [ Given pre-trained task model, set of classes to optimize is selected by the Candidate Selection Module (CSM) based on task model prediction to improve adaptation efficiency. We focus the selected classes and aggregate the conditional noise estimations with CSM-modulated probabilities on these classes, upon which our DUSA objective is constructed. The structured semantic priors of the diffusion model are then propagated to the task model through our objective. For more details please refer to Alg. in Appendix
3.1 A Brief Review of Diffusion-TTA
Diffusion-TTI 332 takes the first step in exploring conditional diffusion models for test-time adapta- tion. In Diffusion-TTTT TTA, the task model prediction poly xo) is integrated with class embeddings

---

f J 1 to get a soft condition c = Ey Po(y Xo))cy, where N is the number of classes. The sample-
wise adaptation performed with the diffusion loss: L(0, Φ) = I [ cy (x,, t, c) Inspired by (25), the loss is averaged over hundreds of (1, E) pairs remedy the performance degradation caused by incorrect class prediction using a single timestep (2). at the cost of reduced efficiency.
3.2 Unlocking the Discriminative Power of Conditional Diffusion Models
Unlike previous attempts that rely heavily on massive number of timesteps to provide an appropriate
estimation of likelihood p(x c) [25] 29 32). we shed light on the semantic structure underneath the denoising capability of conditional diffusion models from the perspective of score functions [39] 40
41 42).
which will be shown to hold for every single timestep. Proofs can be found in Appendix
We first present our main theoretical contribution to reveal the semantic structure of score functions: Proposition 1. Let p(x) and {p(x) y) y E } be continuously differentiable probability densities, their score functions Vx log p(x) and { log p(x y) : y E V}, the following equation holds:
l(g Σ, plu x) Vx log p(x 11)
(4)
Remark. The equation holds under mild assumptions about the densities but applies any data with an entire set of conditions {y y E y). All score functions can be estimated by score matching or denoising diffusion. Note that the posteriors {ply x) y y) are not directly modeled, and thus can be seen as the implicit priors hidden in the construction of (conditional) score functions.
To link our Proposition I with a trained diffusion model, we revisit Tweedie's 's Formula (49). which serves key connection between score functions and the formulation of diffusion models (43
Lemma (Tweedic's ' Formula). Let z uz ~ (2; pz Ex), then the posterior expectation of μz given z can be estimated by:
Eip z] = z + Σ, V p(z)
(5)
As Eq. (2 indicates q(x: | Xo) = N(xii VoiXo, (1 (I). we have the following corollary:
Corollary 1. Let (O,, be a sampled noise in a forward process parameterized by (āt)T= to get noisy data xs, the connection between score function Vx, log p(x:) and noise e can be given by:
Vx, log:(x;)
(6)
Recall that our Proposition I makes no assumptions on the form of data x, thus it holds for the noisy data x; at any single timestep Applying Eq. (6) to Eq. (4) at timestep 4, we get the following:
e
Σ, p(y P(y Vx,
x, log p(xe y).
a
For diffusion models, the conditional score function Xx, p(xe y)) can be estimated as follows: c (x, 1, c.y)
xx log ((x y) Rz
(8)
vI āz
and therefore we can combine Eq. and Eq. 8 to give structured estimation in the latent space:
E ply xx (x,, C,)),
where all score functions are now replaced by noise predictors. Note that p(y xx)) is the implicit prior of conditional diffusion model at single timestep f, and thus can be learned by directly plugging the task model prediction on Xp. dubbed as po xn). into Eq. O.. The objective to minimize is:
COUSA (0, 6) = E. |le -,,, po(y Xx) c,) 12
(10)
Intuitively, the optimization of this objective encourages the task model to extract knowledge from the semantic structure of capacious diffusion model, promising better robustness for adaptation. Corollary 2. The objective Eq. (T0p is extensive to xo-prediction or v-prediction (50) in diffusion.
With a total of K classes and T timesteps and computational burden primarily borne by diffusion models, our DUSA shows task-relevant time complexity of O(K), while that of Diffusion-TTA is the diffusion-reeevant OTT). Enhanced by practical designs in Sec. 3.3 we empirically find that our DUSA establishes leading performance even with a small budget, leaving Diffusion- TTA behind.

---

Free lunch in modern diffusion models. The proposed objective in Eq. 10 requires the joint training of task model (x) and diffusion model eg (x, c,)) over all conditions {c,, y E } simutaneously. The training inefficiency largely stems from the excessive adaptation of the diffusion model, which may not always require knowledge from the task model.
Indeed, Eq. 1OO can be interpreted from two distinct perspectives. From one viewpoint, the task model extracts knowledge from the implicit priors of the diffusion model. From another, a weighted optimization is applied to conditional noise estimations, allowing the diffusion model to adapt to the incoming test-time data based on task model predictions. Altematively, the adaptation of diffusion models can be achieved by introducing unconditional noise estimations with null condition Ø:
R Ep(X,, t, 2).
(11)
Combining Eq. (9) and Eq. (TT reveals another semantic structure within the diffusion model:
(x(xx,1,,0) Σ, plv (x)) (,))
(12)
where E Ex(x,, i, 2) and {ex(x,, t, Cyy) : E } represent noise estimations from a specific diffusion model capable of handling both unconditional and conditional generation. Note that the implicit priors {ply xt) y E }} in Eq. (12) serve as a critical link between the unconditional and conditional noise estimations. Therefore, an unconditional adaptation of the diffusion model implicitly facilitates its conditional adaptation to the test-time scenario, without reliance on the task model.
Modern conditional diffusion models [20] [22] maintain their unconditional generation capability by employing an additional null condition that replaces the original class conditions with a certain probability (typically 10%). Leveraging this feature, we can adjust the objective to enhance efficiency:
Leond Ee Eeel =0)=EE(
E,, poly ( xx)) 4, cn))l8], Luncond = Ee Ee[lle= Ex(Xe, 1, 2)
L (0,, o) = (0) + Luncomd
(13)
where the diffusion model is now unconditionally adapted. An immediate concern is whether such modifications would impact adaptation performance. We empirically find it can significantly boost training efficiency with minor to no performance degradation, please refer Sec. 4.1 for more details.
Readily applicable to dense prediction tasks. It is worth noting that our method is not confined to classification tasks, but can be easily applied to a handful of dense prediction tasks as well. Taking semantic segmentation as an example, the task model (y Xx) is now a dense labeler assigning per-pixel class labels to the input, where y is the predicted segmentation map of shape H x W x K, H x W is the size of input image Xo. and K is the number of interested classes. Again, our proposition nowhere strict on the form of data, and therefore should be readily applicable to every single pixel in Xo. The new objective to minimize is then casily obtained by utilizing Eq. 44 in per- per-pixel fashion:
||ee -Ex K
LDUSA-SSg (0, () = Be
k= pa Xo)),,n, Ex L, Ck)h.
(14)
where (h, w) denotes the pixel location in an image sample of size H x W, and c₁ represents the class embedding of a class i in the segmentation task. We highlight that per-pixel noise can be efficiently acquired by extracting elements from the image-level noise estimation es (Xt, L, cs.), which takes the entire data sample x, and class-wise condition C₁ as inputs. As a vast majority of diffusion models are trained with image-level annotations, this design is advantageous it allows the use of off-the-shelf diffusion models without modifying their training schemes. In contrast, Diffusion- -TA (32] requires the integration of per-pixel conditions into diffusion models to accommodate dense prediction tasks.
3.3 Improving Adaptation Efficiency with Practical Designs
Identifying appropriate timestep. Since our DUSA intends to extract structured semantic priors from single timestep, a critical question emerges: which timestep should we utilize to maximize adaptation performance? Iterating over all T timesteps for a certain task model on a specific task is just not practical, and thus universal preference must be advocated. While the semantic structure uncovered Eq. (4) is valid for all timesteps in theory, the estimation of score functions by denoising diffusion models 20 22 can be unreliable. As pointed out by [5] 52 for diffusion models we have
5

---

a scheduled āg decreasing with t and ār + 1 when t + O which directly amplifies error in score estimation by Eq. (2) at smaller timesteps. A large timestep also not recommended, as denoising at higher noise levels is more challenging (38), posing greater challenge to score estimation. Based on the preceding conclusions, we select timestep = 100 and find it suits well for all our experiments.
Utilizing task model for candidate selection. Similar to Diffusion Classifiers 25
29
our DUSA
has a computational complexity that scales proportional to class number. To circumvent the slowdown by a large number of classes, we utilize task model prediction to significantly improve adaptation efficiency. With the observation that classifiers typically maintain a lop-k accuracy (53) and top-1 accuracy is our main concern in the test-time adaptation of discriminative models, we opt to apply our DUSA to the most promising classes. Specifically, we deem the posterior p(y x) of less likely class candidates to be zero and only optimize the semantic structure among the selected classes.
A mere decrease in class number can lead to optimization issues, as the task model can be biased towards certain classes, especially with a small batch size. This can be blamed on the underutilization of the semantic structure, further exacerbated by the erratic task model prediction on the pruned classes due a lack of constraints. To tackle this, we devise Candidate Selection Module (CSM). In the module, we first adopt LogitNorm [54] 55 to force constraints on the pruned classes to stabilize training, where the logits output of the task model are lg normalized before selection. Intuitively, we discourage the optimization in the magnitude of logits to mitigate overconfidence, especially on pruned classes. Then we handle the class bias problem by introducing randomness in selection. In detail, with selection budget b = k + m, we split it into two parts: k for top-k classes select, and m for a multinomial selection without replacement from the remaining classes, where the sampling probabilities are calculated from the logits before normalization. We only focus on the selected b normalized logits, and apply softmax to get their probabilities for our DUSA objective. After a series of practical designs, we succeed in reducing the time complexity of DUSA from O k)) to O(b), where a small b should be valid for a large number of classes, as will be shown in Sec. 4.4
4 Experiments
Datasets and models. Our experiments are conducted on three benchmarks: ImageNet-C C [31] for fully and continual test-time classification, ADE20K (56) with corruptions defined in [31] (dubbed
as ADE20K-CO -C) for test-time semantic segmentation. All image corruptions are at the highest
severity level 5. We use ResNet-50 (5). ViT-B/16 [57]. Co- (6) pre-trained on ImageNet for
ImageNet-C experiments, and SegFormer- B5 (58) pre-trained on ADE20K for ADE20K-C ones. We
follow (59] and use the GN variant of ResNet-50 for stability. More details are in Appendix FTT
Compared methods. We compare DUSA with Tent (44), CoTTA (45] EATA (60). SAR (59 ROTTA [6].. and ceffusiio-TTA (3) for image classification. For semantic segmentation. we compare with BN Adapt (62. 63
Tent and CoTTA 45) More details are in Appendix 12
Evaluation metrics. Top-1 accuracy (Acc) is reported on each corruption type for image classifi- cation. For semantic segmentation, the mean Intersection-over -eeeriiooooov---eeeeee Union (mloU) is reported. The main results of our DUSA all come with mean and standard deviation statistics over 3 independent runs.
Implementation details. The batch size is 64 for test-time classification tasks unless otherwise stated. Following (32,, we use Adam optimizer (64] with learning rate of 0.00001 for our DUSA and Diffusion-TTi TTA. For other baselines, SGD with momentum 0.9 or Adam optimizer is used in line
with the literature [44 59 60). As for test-time semantic segmentation, the batch size is and Adam with a learning rate of 0.00006/8 8 is used, following (45). We use ImageNet [65] trained DiT (2
and ADE20K trained ControlNet 48 as diffusion models. More details are in Appendix F3
4.1 Fully Teme-Time Adaptation of ImageNet Pre-trained Classifiers
Table [ shows our DUSA in comparison with relevant methods under the online setting of test- time adaptation to every single corruption domain in ImageNet-C, also known as fully TTA [44]. Generally, our DUSA and Diffusion- both achieve a substantial performance gain, thanks to the knowledge from capacious generative diffusion model. Sepecifically, our DUSA yields significant improvement of +21.9%, +3333% + 15.7% on ResNet-50, ViT-B/16, and Con over pre-trained classifiers. Besides, DUSA consistently outperforms the multi-timester enhanced

---

Table I: Fully test-time adaptation of ImageNet classifiers ImageNet-C. C. The best results are in bold and runner-ups are underlined. GN/LN is short for Group/LLyer normalization.
Nuise
Blur
Wenther
Digital
Method
Gunes.
Slat
Impul.
Defoc.
Glass
Mution
Zoom
Snow
Frost
Fog
Beit.
Contr.
Blastic
Pinel
JPEG
Avg.
ReuNer 30 (GN)
22.1
23.0
22.0
19.8
11.4
21.5
25.0
40.3
47.0
34.0
08.3
36.3
18.5
29.3
52.6
51.4
Tent
25.3
29.1
21.5
14.9
21.6
22.3
27.5
32.1
3.5
69.9
42.0
10.3
48.6
54.6
29.1
CaTTA
22.1
23.0
22.0
21.5
25.1
40.3
47.0
34.0
68.8
36.4
18.5
29.3
52.6
31.5
EATA
38.6
40.9
39.7
27.3
26.7
36.5
38.5
50.8
89.1
55.5
72.0
49.9
40.5
55.7
58.2
45.3
SAR
39.6
42.4
41.0
19.8
22.9
37.1
38.7
27.3
47.4
55.1
72.4
48.8
7.2
54.9
57.1
40.8
RoTTA
22.8
23.8
41.3
47.5
34.6
09.2
36.8
19.2
29,9
52.
31,9
Diffusion-TIIX
42.0
56.5
60.0
22.6
45.6
57.9
6L44
51,3
DUSA (Oun)
47.3.a0
46.3
37.3.p
37.6
48.4uaa
50.3.n3
59.1.n
S5.6..a
63,3
73.3.m0
56.3
63.2.n1
60,9
53.3
DUSA-U (Oun)
45.0
47.1
46.
56.8
37.7
47.9 47.9.a:
49.5 49.5.00
59.0uni
55/4411
63.0an:
73.1.nt
54.3
$6.4
62.9
60.5
$3.0
VIT-B/16 (I.N)
38.3
35.4
$8.1
29.5
24.2
$2.8
30.5
36.4
45.0
50.4
68.3
22.5
39.4
52.7
53.5
89.8
Tent
53.9
54.5
54.1
44.4
42..2
53.8
6.7
4.6
61.9
65.4
72.9
54.9
58.0
65.1
64.11
50.8
CoTTA
38.3
35.4
38.1
29.5
24.2
32.8
30.5
36.4
45.0
50.4
68.3
22.5
39.4
52.7
53.5
39,8
EATA
55.4
56.3
55.3
48.9
58.6
58.2
63.5
64.1
67.5
74.3
56.5
65.7
68.5
66.6
60.9
SAR
53.9
54.3
S4.1
46.0
47.8
54.2
39.4
28:2
61.4
64.3
72.8
54.3
59.2
64.8
63.5
55.2
RoTTA
42.6
39.9
+2.9
30,6
26.4
34.8
31.7
39.2
47.8
$2.4
68.8
23.3
42.0
55.0
$4.0
42.1
Diffusion-TIA
52.1
54.5
53.5
64.2
72.6
47.4
66.4
58.6
DUSA (Ouns)
56.6.11
57.9.a2
57.0aa
53..m
56.7.03
62.4
61.6..
(5.9
05.7a31
70.118
75.3.m.
60.2.03
67.9.b.
69.7.m.
65.8
63.1
DUSA-U (Ouns)
56.3.a.
57.5.a.
36.7
52.5.6.1
56.4.ns
61.9u3:
00/4..0
65.8un3
65/4433
70.0
75
15.3
58.7
67.8.p.)
69.4..n
64.3
62.6
Con-NeXi-L.
56.7
56.2
58.3
35.1
20:7
47.6
43.5
58.0
48.0
76.6
55.7
34.0
42.3
63.3
50.5
Tent
$7.4
$7.8
58.9
35.7
24.3
51.3
46.3
59.88
58.4
11.01
77.1
61.2
35.1
50.0
64.4
49.9
CSTTA
56.7
56.2
58.3
35.1
20.7
47.6
43,5
59.0
59.9
48.0
76.6
55.7
34.0
42.3
633.3
50.5
EATA
$1.5
38.7
51.6
47.0
49.3
77.2
61.3
40.2
50.3
64.5
53.4
SAR
57.0
56.7
53.8
37.4
50.9
46.3
60.1
57.6
12.4
77.0
61.9
37.1
51.4
64.1
50.4
RoTTA
57.0
56.7
58.7
35.1
21.3
48.0
44.0
59.5
60LD
48.9
76.6
56.3
34.6
43.1
63.4
50.9
Diffusion-TIA
58.7
29.6
58.3
50.3
48.8
57.6
54.8
63.3
64.8
68.5
77.4
60.9
62.0
65.6
05.5
61.1
DUSA (Ours)
64.2.n1
65.5.a)
65.6281
70.
66.6233
79.7-00
68.9.a3
66.1.63
70.7.n3
69.3us
DUSA-U (Cuus) 63.8.o.
65.2.am
65.2
54.0ama 53.3 63.3ga:
60.6...
66.211
72.5
79.6.a
68.1
65.
65.9.03 70.3.02 68.7 65.8
Table 2: Continual test-time adaptation of ImageNet pre-trained ConyNext-L on ImageNet-CC C. The best results are in bold and runner-ups are underlined. LN is short for Layer normalization.
Time
Method
Cuuss.
Shot
Impul.
Defoc.
Glass
Metion
Zoom
Frust
Fog
Contr:
Elastic
Pixel
JPEG
Avg.
ConoNeNi-L.
56.2
58.3
15.1
30.7
47.6
43.5
48.0
76.6
55.7
54.0
42.3
63.3
50.5
That
38.7
33.8
53.7
50.0
600.3
67.4
77.5
64.9
23.4
52.5
55.1
COTTA
56.7
56.2
58.3
35.1
20.7
47.6
43.5
59,9
59.9
48.1
76.6
55.8
34.1
42.5
63.3
50.5
SAR
57.0
59.6
62.6
40.9
32,5
55.1
51.1
61.1
61.2
68.3
78.0
65.4
78.4
52.1
65.2
55.9
EATA
61.0
63.5
42.5
35.2
55.3
52.4
68.6
66._1
46.2
56.7
66.9
58.3
ROTTA
57.0
58.2
60.9
34.2
24.5
47.9
45.3
60.9
62.5
51.7
74.9
49.8
30.3
42.6
62.5
51.5
Diffasco-TTA
58.1
63-2
63.2
54.1
56.6
61.8
62.5
65.2
65.5
68.1
75.3
58.9
37.3
$4.8
60.9
60.4
DUSA (Ours)
64.1ou
67.7
68.3.)
$48..0
56.2
64.6.00
65.6.m)
69.8.cn
69.9
74.5aa)
79.0.u)
70.3
68.5
71.9.31
70.7
67.7
Diffusion-TTA by +2.0%, +4.5%, +5.1% these classifiers, justifying the exploration of semantic priors underneath the diffusion score estimations and demonstrating a clear superiority of our DUSA.
Furthermore, thrilling finding is that DUSA is not reliant on the integration of task model prediction in objective Eq. (TO) to maintain the powerful semantic priors implicitly embedded. In Table 1. we provide the adaptation results from Eq. [3), namely DUSA-U. Despite the diffusion model being trained unconditionallyy in DUSA-U and thus having no chance to borrow knowledge from task models, the performance is still on par with DUSA. This reinforces our conviction that diffusion models inherently possess such semantic priors, even without explicit inclusion of task models. Although DUSA-U is more lightweight (diffusion model only trained on null condition), we stick to DUSA when benchmarking against Diffusion-TTA to ensure fairness in the training budget b.
4.2 Continual Test-Time Adaptation of ImageNet Pre-trained Classifiers
We also experiment under the online continual test-time adaptation protocol, where the task model should adapt to continually changing scenarios (45). The outcomes are reported in Table Our DUSA withstands long period of adaptation and outperforms Diffusion-TT -TA by a large margin of +7.3% on ConvNeXt-L. During adaptation, Diffusion-
witnesses a performance drop when the OOD datastream type shifts from Weather (Brit.) to Digital (Contr.), while our DUSA shows remarkable adaptation stability. This suggests that DUSA effectively learns from robust and transferable semantic priors from score-based generative modeling, allowing it shine over prolonged adaptation.
4.3 Fully Test-Time Adaptation of ADE20K Pre-trained Segmentors
The versatility of our DUSA to dense prediction tasks is evaluated by fully test-time semantic segmentation on the ADE20K dataset with corruptions. We experiment on SegFormer-Bi B5 and report
7

---

Table 3: Test-fime semantic segmentation of ADE20K pre-trained SegFormer-B5 ADE20K-C. The best results are in bold and runner-ups are underlined. LN/BN is short for Layer/Batch normalization.
Notse
Weather
Digial
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Prost
Brit.
Cantr.
Elastic
Pixel
JPEG
Segformer BS (LN+BN)
14.2
15.8
15.6
23.1
16.8
22.5
10.3
22.3
21.5
38.6
23.1
24.5
33.1
35.3
23.9
Adage
17.0
Tent
11.2
13.0
12.5
17.0
13.5
16.9
7.7
17.7
17.4
29.7
32.5
18.6
20.0
25.8
26.4
18.7
CoTTA
14.6
16.1
158
22.6
16.5
22.1
9.8
20.9
20.4
38.8
42.3
21.9
24.3
33.6
35.4
23.7
(Ours)
23.6.13
23
23.3.12
24.7
12.5
27.4
30.6.oa
Clean
Corrupted
Source
BN Adapt
Tent
CoTTA
DUSA (Ours)
Ground Truth
Glass
Brit
Pixel
Figure 2: Visualization of segmentation results on ADE20K-CO From left to right: clean and corrupted images, results of the source model, BN Adapt, Tent, CoTTA, our DUSA, and ground-truth labels.
the results in Table Notably, previous methods fail on most tasks, except for CoTTA which achieves modest improvements few tasks through combination of stochastic weight restoration and data augmentation. For all tasks involved, our DUSA takes the lead by exploiting the semantic priors from the nigh-dimensional latent space of pre-trained text-to-image diffusion model, justifying the extensiveness of our proposition to wider range of discriminative tasks beyond classification. Segmentation results for all the methods involved the illustration, showcases its ability to address errors by incorporating semantic priors from diffusion models, overcoming a limitation faced by other methods that depend solely on the task model's precision. We provide more visualizations over a wide range of scenarios in Appendix !]
visualized in Fig. [ Our DUSA, shown in
4.4 Ablation Study
Selection of timestep t. As discussed in Sec. 3.2 our DUSA significantly reduces the number of timesteps to a single timestep diffusion models. Fig. 3 illustratee influence of timestep selection on DUSA through adapting the ConyNeXt-L classifier corruptions from the four main categories. Consistent with our analysis in Sec. 3.3, the guidance from diffusion models is from perfect when the chosen timestep is either too small - 0) or too large (l T). We empirically find that / 100 shows a good performance here, and generalizes well to other backbones and tasks as well. The other timesteps, e.g., t 50, however also emerge strong contenders and outperform Diffusion-TT.. by a considerable margin. For simplicity, we adopt 100 in all our experiments.
70
Accuracy (%)
Iccurrcy (%6)
50
Diffusion ThA
40
40-
Res bo
20-
30
Gauss. Defoc
Snow Contr.
20
ResNet 50, bs-64 VIT-B/16, bam4 ResNet-50, bsm4
VIT-B/16, bs-64
10-
10
50.00
300
400
300
800
7:0
? 6o s aa
Timestep (0-indexed)
Budget Schemes (k:m)
Accuracy of Con-NeXt-L across different selections of timestep.
Figure 3:
Figure 4: Accuracy ViT-B/16 on JPEG and ResNet-50 Contrast, across different budgets for adaptation.

---

Effect of components in DUSA. To grasp deeper understanding of DUSA, we provide a detailed ablation of critical designs in Table where the number in parentheses means the budget 6 k + m allowed for diffusion model forward (D.F.) in Eq. (TOp. i.e., number of classes to adapt for each sample. The results are obtained on ResNet-50 and ConvNeXt-L over the corruptions within the Noise category. We also include Pixelate corruption results on ConvNeXt-L. to offer well-rounded understanding, upon which the transferability of diffusion models to OOD data is demonstrated.
Before adaptation, the source-only models serve as the baseline. In- troducing the objective in Eq. (IO) when freezing the diffusion model brings about a performance gain of +3.6% on ResNet-50 for Noise, but causes a degradation of 13% on ConvNeXt-L. This is largely due to the instability from discarding classes, as pointed out in Sec. [33 Applying LogitNorm instantly mit- igates the issue and brings about consistent gain of +21.6% and +3.8% against baselines. The im- provement is made without training diffusion models and therefore can be viewed exploiting the generative semantic priors formed in diffusion pre-training. However, the outcomes are still below the baseline for Pixelate. We conjecture that such corruption might be OOD even for diffusion model with strong robustness. The further adaptation of diffusion models removes this concern, pushing all results a competitive level. We attribute this finding the fast convergence of generative modeling [T] on unseen data, which is favorable to the online nature of test-time adaptation. Again, the inclusion of LogitNorm yields a significantly boosted accuracy at 46.4%, 65.0% and 70.4%.
Table 4: Ablation on critical components in DUSA. Compo- nents in colored rows are not carried over to subsequent rows.
ResNet-50
ConvNeXt-L
Variants
m
D.F.
D.B.
Noise
Noise
Pixel
Source only
0
0
0
22,4
57.1
42.3
score priors inspired loss (4)
4
0
4
0
26.0
44,1
9.2
LogiiNorm (4)
0
44.0
60.9
9.6
adapt diffusion (4)
41.2
57.8
LogitNorm (4)
0
4
4
46.4
65.0
70.4
LogitNorm (6)
6
0
6
6
46.5
65.1
70.7
uniform select (6)
2
46.2
65.1
70.7
multinomial select (DUSA)
46.3
65.1
70.8
null conditioning (6) (DUSA-U) 4 2
7
46.1
64.7
70.5
To provide a basis for the further ablation of budget schemes, the budget is raised from 4 to 6 with a slight increase in performance. A mild drop in accuracy is witnessed when handing the class bias problem in Sec. 3.3 with a budget m = used with uniform sampling, which is then improved by our multinomial selection in the penultimate line. We underline that such design is indispensable for small batch size, which is also practical [59]. For better consistency, h = 4, m = 2 are universally adopted for DUSA in classification, the ratio between them to be delved into below. Interestingly, with access to a diffusion model capable of unconditional generation, DUSA-U could achieve performance comparable DUSA using Eq. (33), while significantly reducing the computational cost associated with diffusion model backward ). We believe this observation can back our claim of the existence of structured semantic priors inherently embedded in diffusion models.
Specifying a budget scheme. As a justification for our design of the selection strategies in CSM, we take a thorough investigation into the effects of different budget schemes over varied classifiers (ResNet-50 & ViT-B/16) and batch size (4 & 64) in Fig. 4 At a smaller batch size (bs) of 4 (dashed lines), the budget scheme k m plays vital role in performance. Concretely, a large k is favorable to weaker task models (ResNets) for eager adaptation, while a proper m is a must to prevent more powerful ones (ViTs) from overfitting to a subset of classes. When the batch size increased to the standard 64 (solid lines), our DUSA becomes insensitive to budget schemes, and a consistent gain is observed for both classifiers. DUSA with b = 4 even exceeds Diffusion-TTA with b underscoring the advanced efficiency made possible by our proposition and practical designs. For a budget scheme, we find m competitive across varied b, and stick to k m for DUSA.
5 Related Work
474 focuses on improving source data pre-trained model performance on out- of--iitiiibttiinnn target data without label access during inference time. Early works lay more emphasis on adapting the activation statisties of batch normalization (BN) (62) 63. 66,66 67 Test-time training (68
Test-time adaptation. Test-time adaptation
69 70, 71) methods manage to adapt through devising a test-time self-supervised objective which is also injected into the pre-training stage, resulting complicated pipelines and increased computational cost. To lessen dependence source data and extra loss

---

injection, fully test-time adaptation [44] is advocated to achieve adaptation with only unlabeled target data. Concretely, previous works are majorly based on entropy minimization objectives: Tent [44] directly minimizes entropy on batched data predictions, MEMO (22 proposes marginal entropy minimization via data augmentation, EATA [60] pursues a sample efficient entropy minimization
and anti-forgetting regularization, while SAR (59) advocates sharpness-awan and reliable entropy minimization. Other works delve into extensive distributional shifts in TTA (73), e.g., continual
adaptation without forgetting [45 60), correlative data streams [61] and label shifts (59 74). Our
DUSA is much different as it is not reliant on error-prone entropy- objectives and rather extracts knowledge from semantic priors of generative models for better adaptation of the task model.
Generative models for discriminative tasks. The long- long-standing discussion on the connections
between generative and discriminative models [1, 75 76 has inspired a handful of attempts to integrate the two seemingly disparate paradigms 2 19 77 78 79 80). Specifically, a collection of works showcase the impressive power of generative pre-training followed by supervised fine-
84) Besides, few works utilize generative models as zero-shot Integrating generative modeling into the task of test-time adaptation is gaining traction. Prior works manage to boost task model performance with a variety of generative techniques, including GANs (86). MAAss [87]. energy-based (88) and flow-based [89]. The recent prevalence of diffusion models with extraordinary generation capability stimulates a range of works adapting them for discriminative tasks. As for TTA, two distinct research directions arise. Appreciating the generative power [90 11] of diffusion models, a series of works propose to adapt samples in the input space (92, 93 94 Another largely under-explored direction is to repurpose
tuning [1] 24 27
8T 82
83,
recognizers 25 29 85
the generative objective of diffusion models as a proxy of discriminative ability enhancement, with Diffusion-TTA (32) pioneering in incorporating task predictions into the class condition of denoising
objective in an inversion 7 25 29 95] style. Our DUSA belongs to the latter direction but is fundamentally different from [32]. in that we delve deeper into the semantic priors of diffusion models from the perspective of score functions and achieve better adaptability and versatility.
Timestep selection in diffusion models. The importance of timestep selection in diffusion models has been widely recognized in the literature, In image editing tasks, diffusion models are observed to exhibit natural coarse-to-fine pattern during the reverse process (9T 96 98 99 100). Conse- quently, the trade-off between realism and fidelity in editing largely stems from the chosen intervals of timesteps. In discriminative tasks, timestep selection is also crucial the quality of extracted features. An early study (83) clucidated the features within diffusion models, revealing that the most informative ones are derived from smaller timesteps. Subsequent works have reinforced this finding by utilizing a single small timestep across various tasks, including semantic segmentation, referring image segmentation, depth estimation [24). object detection [26). semantic correspondence [1OT]. and one-shot image segmentation [102). While our DUSA aligns with the existing literature on using a single timestep, it differs by extracting semantic priors rather than focusing on feature extraction.
Conclusion
In this paper, we introduce DUSA, competitive test-time adaptation method built the structured semantic priors underlying diffusion models, which serve as score estimators. A proposition is offered to unveil the semantic structure in these based models, upon which test-time objective is derived to fully exploit the implicit semantic priors. Our approach is also shown to generalize well to modern diffusion models and dense prediction tasks. Additionally, we enhance the adaptation efficiency through a few practical designs. The effectiveness of our DUSA is demonstrated across three challenging benchmarks, where consistently outperforms competing methods. We hope our method will pave the way for better utilization of generative modeling for discriminative tasks.
Acknowledgments and Disclosure of Funding
This paper was supported by the National Natural Science Foundation of China (No. 62376026), Beijing Nova Program (No. 20230484296) and KuaiShou.
10

---

References
[1] Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NeurlPS, pages 841 8488 2001.
[2] Rajat Raina, Yirong Shen, Andrew Y. Ng. and Andrew McCallum. Classification with hybrid genera- tive/discriminative models. In NewrlPS, pages 545-552 2003.
[3] Victor Garcia Satorras, Zeynep Akata, and Max Welling. Combining generative and discriminative models for hybrid inference. In NeurlPS, pages 13802-13811 2019.
[4] Will Grathwohl, Kuan-Chich Wang. Joem-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2020.
[5] Kaiming He, Xiangyu Zhang. Shanqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, pages 770-778, 2016.
[6] Zhuang Liu, Hanzi Mao, Chao-Yuan Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, pages 11976-1 1196, 2022.
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to- fo-image generation using textual inversion. In ICLR, 2023.
[8] Le Yang, Haojun Jiang, Ruojin Cai, Yulin Wang. Shiji Song, Gao Huang, and Qi Tian. Condensenet v2: Sparse feature reactivation for deep networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3569-3578, 2021.
[9] Le Yang, Yizeng Han, Chen, Shiji Song. Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 2369-2378, 2020.
[10] Ziwei Zheng, Yang. Yulin Wang, Miao Zhang, Lijun He, Gao Huang, and Fan Dynamic spatial focus for efficient compressed video action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[11] Le Yang. Ziwei Zheng. Jian Wang. Shiji Song, Gao Huang, and Fan Li. Adadet: An adaptive object detection system based on early-exit neural networks. IEEE Transactions on Cognitive and Developmental Systems, 16(1):332-345, 2023.
[12] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. ECCV, pages 456-473, 2018.
[13] Robert Geirhos, Jörn- Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(11):665- 673, 2020.
[14] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurlPS, pages 2172-21 2016.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[16] Sarthak Mittal, Korbinian Abstreiter. Stefan Bauer, Beruhard Schölkopf, and Arash Mehrjou. Diffusion based representation learning. In ICMI., pages 24963-24982, 2023.
[17] Xuwang Yin. Soheil Kolouri, and Gustavo K Rohde. Gat: Generative adversarial training for adversarial example detection and robust classification. In ICLR, 2020.
[18] Xuwang Yin. Generative robust classification. arXiv preprint arXiv:22/2.07288, 2022.
[19] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In CVPR, pages 2142-2152, 2023.
[20] Robin Romhach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[21] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, pages 36479-36494, 2022.
[22] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195 4205, 2023.
11

---

[23] Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Padmanabhan. Archana Swaminathan, Tianyi Zhou, and Abhinav Shrivastava. Do text-free diffusion models learn discriminative visual representations? arXiv preprint arXiv:2311.1992. 2023.
[24] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing texl-to-image diffusion models for visual perception. In ICCV, pages 5729 5739, 2023.
[25]
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secredly zero-shot classifier. In ICCV, pages 2206-22111 2023.
[26] Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy J Ma. Diffusionengine: Diffusion model is scalable data engine for object detection. arXiv preprint AxXiv:230900389. 2023.
[27] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pages 2955-2966, 2023.
[28] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurlPS, 36, 2024.
[29] Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. In NeurlPS, pages 58921-58937, 2023.
[30] Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. In ICLR, 2024.
[31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
[32] Mihir Prabhudesai, Tsung- Wei Ke, Alexander Cong Li, Deepak Pathak, and Katerina Fragkiadaki. Test-time adaptation of discriminative models via diftusion generative feedback. In NewrlPS, 2023,
[33] René Haas, Inbar Huberman- Huberman-SSieeelglann Spiegelglas. Rotem Mulayoff, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. arXiv preprint arXiv:2303. 2023.
[34] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. In ICLR, 2023.
[35] Yusuf Dalva and Pinar Yanardag. Noiseclr: contrastive learning approach for unsupervised discovery interpretable directions in diffusion models. arXiv preprint arXiv:2312.0533 2023.
[36] Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In ACM MM, pages 5321-5329, 2023.
[37] Myyg Park, Mingi Kwon, Jacwoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. In NeurIPS, 2023.
[38] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840-6851, 2020.
[39] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput.. 23(7):1661-1674 2011.
[40] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, pages 11895-11907, 2019.
[41] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NeurlPS, pages
2020.
[42] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[43] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.1197 2022.
[44] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[45] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, pages 7201-7211, 2022.
[46] Jascha lcarning using nonequilibrium thermodynamies. In ICML, pages 2256-2265,425 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised
[47] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with gext-g diffusion models. In ICML, pages
2022.
12

---

[48] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836-3847, 2023.
[49] Bradley Efron. Tweedic's formula and selection bias. JASA, 106(496):160 100111141,,,, 2011.
[50] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[51] Yang Song and Diederik Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 2021.
[52] Valentin De Bortoli, Michael Hutchinson, Peter Wirnsberger, and Amaud Doucet. Target score matching. arXiv preprint arXiv:2402.0866 2024.
[53] Mihir Prabhudesai, Tsung- Wei Ke, Alexander Cong Deepak Pathak, and Katerina Fragkiadaki. Test-time adaptation with diffusion models. In ICML Workshop, 2023.
[54] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML,, pages 23631 2022.
[55] Mingjia Li. Binhui Xie, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Vblc: Visibility boosting and logit-constraint learning for domain adaptive semantic segmentation under adverse conditions. In AAAI, pages 8605-8613, 2023.
[56] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k datasct. In CVPR, pages 033-641,220 2017.
[57] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[58] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, pages 12077-12090, 2021.
[59] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
[60] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, pages 16888-166 16905, 2022.
[61] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation dynamic scenarios. In CVPR, pages 15922-15932, 2023.
[62] Zachary Nado, Shreyas Padhy, Sculley, Alexander Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 1006:, 2020.
[63] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurlPS, pages 11539-11551, 2020.
[64] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[65] Jia Deng, Wei Dong, Richard Socher, Li-Jia Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, pages 248-255 2009.
[66] Yanghao Li, Naiyan Wang. Jianping Shi, Jiaying Liu, and Xiaodi Hou, Revisiting batch normalization for practical domain adaptation. ICLR Workshop, 2017.
[67] Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprint 12, 2021.
[68] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, pages 9229 9248, 2020.
[69] Yuejiang Liu, Parth Kothari, Bastien Germain van Delft, Baptiste Gurlet, Taylor Mordan, and Alexandre Alahi, TTT++: When does self-supervised test-time training fail or thrive? In NewrlPS, 2021.
[70] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros. Test-time training with masked autoencoders. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, NeurIPS, 2022.
[71] Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. M13: Meta test-time training for self sulf-supe test-time adaption. In AISTATS, pages 3080 -0900000000000000 2022.
[72] Marvin Mengxin Zhang. Sergey Levine, and Chelsea Finn. MEMO: Test time robustness via adaptation and augmentation. In. NeurIPS, pages 38629-36600 3869993842..022 2022.
13

---

[73] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In JCML, pages 42058-42080, 2023.
[74] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter- free online test-time adaptation. In CVPR, pages 8344-8353, 2022.
[75] Jing-Hao Xue and D Michael Titterington. Comment "on discriminative generative classifiers: A comparison of logistic regression and naive bayes". Neural Process. Lett., 28: 169-187, 2008
[76] Chenyu Zheng, Guoqiang Wu, Fan Bao, Yue Cao, Chongxuan Li, and Jun Zhu. Revisiting discriminative
vs. generative classifiers: Theory and implications. In ICML, pages 42420-42477, 2023.
[77] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In CVPR, pages 4467-4477, 2017.
[78] Xiulong Yang and Shihao Ji. Jem++: improved techniques for training jem. In ICCV, pages 6494-6503, 2021.
[79] Xiulong Yang, Sheng-Min Shih, Yinlin Fu, Xiaoting Zhao, and Shihao Ji. Your vit is secretly a hybrid diseriminative-geeeraivs diffusion model. arXiv preprint arXiv:2208.0779 2022.
[80] Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: self-supervised representation generation method. arXiv preprint arXiv:2312.03700 2023.
[81] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell, Adversarial feature learning. arXiv preprint arXiv: 1605.09782, 2016.
[82] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexci A Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2544, 2016.
[83] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In ICLR, 2022.
[84] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, ct al. The effectiveness of mac pve-pretraining for billion-scale pretraining. In ICCV, pages 5484-5494, 2023.
[85] Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael Ryoo. Peekaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint arXiv: 132244, 2022.
[86] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pages 9641-9650, 2020.
[87] Jiaming Liu, Ran Xu, Senqiao Yang. Renrui Zhang, Qizhe Zhang. Zehui Chen, Yandong Guo, and Shanghang Zhang. Adaptive distribution masked autoencoders for continual test-time adaptation. arXiv preprint ar 12/.22.22.. 12480, 2023.
[88] Yige Yuan, Bingbing Xu, Liang Hou, Fei Sun, Huawei Shen, and Xueqi Cheng. Tea: Test-time energy adaptation. arXiv preprint arXiv:2311.14402, 2023.
[89] David Osowiechi, Gustavo A Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ismail Ben Ayed, and Christian Desrosiers. Tuflow: Unsupervised test-time training with normalizing flow. In WACV, pages 2126-2134, 2023.
[90] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint ar arxiv:2007.12599 2022.
[91] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[92] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar, Diffusion models for adversarial purification. In ICML., pages 16805-16827, 2022.
[93] Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-drr driven adaptation to test-time corruption. In CVPR, pages 11786-11796, 2023.
[94] Kaiyu Song and Hanjiang Lai. Target to source: Guidance-hased diffusion model for test-time adaptation. arXiv preprint arXiv:2312.05211 2023.
[95] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, pages 6038 6047, 2023.
[96] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llv: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 2021.
[97] Giannis Daras and Alexandros G Dimakis. Multiresolution textual inversion. arXiv:2211.17115, 2022.
arXiv preprint
[98] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miia Aittala. Timo Aila, Samuli Laine, et al. ediff-i: Text-to soat-to-imag diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0133 1.1124, 2022.
14

---

[99] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exiite: Attention- based semantic guidance for text-lo-image diffusion models. TOG, 42(4):1-10, 2023.
[100] Andrey Voynov, Kfrr Aberman, and Daniel Cohen-Or. Sketch-guided texl-to-image diffusion models. In SIGGRAPH, pages 1 1-11, 2023.
[101] Eric Hedlin, Gopal Sharma, Shweta Mahajan. Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[102] Aliasghar Khani, Sacid Asgari, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamarnch, SLime: Segment like me. ICLR, 2024.
[103]
TrineTrain Contributors. Openmmlab's pre- pre-trainnng toolbox and benchmark. https://gittuu: com/
spen-malab/mmprerrain 2023.
[104]
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and
benchmark. https://gitthb. com/ ccom/open-mmaa///eeee tttaam/lammmmmmmmtttaiaiooo 2020.
[105]
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander
Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming, arXiv preprint arXiv: 190707444 2019.
[106] Ross Wightman. Pytorch image models.
lattps://githul: httpr//gtttaaaattochhhhiieeeeeee
models 2019.
15

---

A Broader Impacts and Limitations
Broader Impacts. In this work, an approach to incorporate generative diffusion models into the discriminative task of test-time adaptation is introduced. The leverage of knowledge from generative modeling teaches whole picture of the scenario to the task model, enabling better robustmess and adaptability and is thus favorable to some relevant scenarios like medical analysis and quality control of manufacturing varying environments. However, the adoption of generative models might come with risk of learning from a biased source of knowledge, leading to improper decisions from the adapted task model. possible remedy for this is to foster the research into more unbiased or balanced training of generative models.
Limitations. Our work presents an effective way of borrowing semantic priors from score-based diffusion models to benefit the discriminative task model that demands adaptation at test time without labels. We acknowledge that, as with any research endeavor, limitations exist in our work. Firstly, competitive method involves diffusion model pre-trained at least the same set of source data as the task model, which may not be easily accessible for certain scenarios. However, we believe such dilemma be mitigated with advances in generative diffusion models, which have demonstrated remarkable progress in both generality and robustness. Our approach achieves significant adaptation efficiency gains against prior catuusioonbbased TTA methods, but there still gap in the time compared to methods that only update the task model. Therefore, it may not meet extreme efficiency demands in scenarios like autonomous driving. Our method, however, yields superior performance and is thus appealing to another set of tasks where trading slight loss in efficiency for boosted accuracy tolerable, e.g., non-emergent medical diagnosis. With this said, we highlight that our theoretical findings are not confined to diffusion models but extensive to all score-based models, therefore substituting more lightweight technique in score estimation for the computationally expensive diffusion models can be promising avenue. We leave it for future work.
B Licenses for Existing Assets
All models and datasets used in our experiments are publicly available, and their licenses are listed below. (D) means data, (M) means model, (C) means code.
(D) ImageNet-C 31): CC-BY 4.0
(D) ADE20K 2016 [6): BSD 3-clause license
(M) ResNet-50 [): Apache-2.0 license
(M) ViT-Base/16 57
Apache-2.0 license
(M) ConvNeXi-L 6): MIT license
(M) DiT-XL/2 22
CC-BY-NC 4.0
(M) Stable Diffusion v1.5 20:
The CreativeML OpenRAIL M license
(M) ControlNet 48 The CreativeML OpenRAIL M license
(C) MMPreTrain
103
Apache-2.0 license
(C) MMSegmentation
104 Apache-2.0 license
(C) Diffusion-TTA
32 MIT license
C Proofs
Proposition 1 Let pxx) and {p(x) y) y E }} be continuously differentiable probability densities, their score functions Vx log ((x) and (Vx p(x y) y E Y}. the following equation holds:
Vx log = Σ ply x) logpog y)-
(15)
Proof. The proof starts from the law of total probability. With the assumption of continuously differentiable, we can take derivatives of the probability densities input data. Note that we always
16

---

assume the existence of x, implying that p(x) is non-zero. The complete proof is then as follows:
p(x) Σ plu)p(x yg)
(law of total probability)
Vxp(x)= Σ p(y) Vxp(x y)
(differentiate both sides on x)
Σ plu)p(x Vxlog y)
dog-derivative trick)
(16)
p(x))zlogp(x) = Σ p( x Vog lv)
(Bayes theorem)
V Vxlogp(x) = Σ plu Vog (v)
(eliminate p(x))
Corollary 1.
Let (0, be sampled noise in a forward process parameterized by { ) get noisy data X the connection between score function Xx, log p(x;) and noise e can be given by:
t=1
Xx log log
(17)
v1 Ct.
Proof. According to the definition in Eq. (), we have
(xx Xx) N Nxx @axo -1--a a.)]..
(18)
Tweedie's Formula in Lemma we have the estimation
1 states that, for observed gaussian variable lz N(zz pzz Ez).
Eiuz z] En V logp(z).
(19)
Applying Tweedic's Formula, we now have
Elpx, x1] X; Σx, V logp(x;)
VoxXo = x πt V x logp(x:)
(20)
VaxX0 (1 az V x, log l(:)(
Reuse the reparameterized version of Eq. with e (O,
x; Vo + v1 d+C
(21)
Vāxoa X, VI a,e.
Combining Eq. 20 and Eq. (21). we obtain
Vx, log = VT Cr.
(22)
Corollary The objective in Eq. (10) is extensive to Xn-prediction v-prediction (50) in diffusion.
Proof. We provide the proof for xo-prediction and v-prediction variants separately. As recall, the objective Eq. (10) is built on the estimation given by e-prediction in Eq. (
For the sake of clarity, here we re- presennn the formula but neglect the estimation error:
Erla xx) xx)) (.y).
(23)
y
For Xo xo-rridic,ion, we employ data predictor x (xe, t, c) to predict Xo.
As have the formula x, Varxi VI- 0,e, can be deduced that Ve E (c, : E (Ø).
(x,) + vI (xe t)
(24)
17

---

Thus we have
Xn X, VI āge
Vāt
X, VT -1
E Ply (x,
(Eq. (3
Vāt
Eyp(y xx) vI - (xe,
Σ plu xx) - 1)
(25)
Var
ply X1) (x,,
(Eq. (24)
vār = -Epo plu xx)xx(x,,,t,)) Cy))
Therefore objective Eq. 10) can be rewritten as
DusSA (0, Φ) Bee [1xx
Σ po (x) c,),)
(26)
For v-prediction, employ velocity predictor vo L, c) to predict constructed velocity objective
Vāie vI- xxXo.
(27)
Similar to the treatment above in Eq. have Vc fc, y JE
V Vo( V 4,
ii (x,,
(28)
Applying the conclusions in Eq. 23 Eq. and Eq. 28 we have
Vi VālE
āxXo Vār Σ ply (x)es(x(,tt vI as Σ p(y xx)xx((x,ttcc)
Σ plu xx) (x,, v1 (x ly k,))
(29)
Enly ply (x,,,( cy).
Therefore, objective in Eq. 10 can be rewritten
(A = E. [Ive - (x)
(30)
D On the Estimation Bias and Efficiency of Diffusion-TTA and DUSA
D.I
Diffusion-TTA
Prior works 25
29 30,
rely on the Monte Carlo method for a good estimation of the likelihood.
The core of diffusion model formulation the evidence lower bound (ELBO), also called variational lower bound (VLB), where the log-likelihood of data is bounded:
Pa(x c)
log c) 2 By log
(31)
g(x1.T xo)
where & is the forward (diffusion) process, and (-) is the reverse process modeled by diffusion models parameterized by ø.
18

---

The ELBO can be further deduced and simplified with diffusion loss, we have
B log (x c)
(x1T xx)
E Px x1,
Σ DEL Xx, xo) ll Po xx, x),
(32)
-KILg(XT xo) (x(x)))
T
--E,,
Σ
welle - Ex(,,, c) log x2,00 +C
1=2
~ -TE,, [||6 6x L, c)
+ C.
(logpo(Xo) x1, c) typically small, wy f 1)
In the equations above, a theoretical approximation is made by simplifying weights wy to and ignoring the log (x X1, c) term. This simplification introduces theoretical bias relative the true likelihood, as the approximation in Eq. 32) prevents the equality in Eq. (31) from holding.
Therefore, with uniform prior on label space, the objective of Diffusion-TIi
32
max p(y x) max p(x v)p(v)
(Bayes theorem)
y
9
max p(x y)
(uniform prior)
y
(33)
max exp ( TB,.c,llee c cy) C)
(Eq. 32
min E,,, [||e ex(xx, cy)l]]
The objective thus takes expectation over and f, where Diffusionn TTA finds f plays crucial rule in the final result and applies Monte Carlo method on 180 samples of t, for every single data sample.
D.2 DUSA
In contrast, our DUSA approach does not rely on the ELBO for likelihood maximization. Instead, we leverage the structured semantic priors introduced in Eq.
guide the task model in extracting knowledge from the diffusion model using a single timestep. Notably, the only estimations in our method are the noise predictions e s (x,, L Cy)) in Eq. S which are provably unbiased.
To elaborate, the diffusion models are typically optimized with the following simplified objective:
Eiimpte (Φ) Ex,le,t [llc cx(x,, 1.)
(34)
At its optimal point, the noise estimations should satisfy:
(x,, cyy)) @Laimpte (0) -x
= 0,
(35)
OE
which implies the following:
(x cyy] Ex.e,,[e]=
(36)
Therefore, the conditional noise estimation Eg an unbiased estimator of the true noise e:
Bias(E,(XX,, t, Cy), c)), c) = Ex,lecsles(xx, c,)] 0.
(37)
Similiarly, the unconditional noise estimation ~ E (xe, t, in Eq. [1 is also provably unbiased.
19

---

E
Algorithm of DUSA
Algorithm 1: DiffUsion Score for test-time Adaptation (DUSA)
1: Input: Test samples {x0]}=1> discriminative task model fa, generative diffusion model Es. timestep learning rate v7.
2: Output: Task predictions {=1
3: for j = 1-1o to M do
4: Predict logits z = fx(x)) with task model.
5: Make task prediction Øj = argmax, Zy.
6: Apply CSM to prune logits z to K classes and get 2.
7: Take softmax over to get probabilities {po(y } y=
8:
y Compute conditions {cy) for all classes in 2.
9:
Sample noise e and get noisy sample x² as in Eq.
10:
Get conditional noise predictions {x]( t, } K
11:
Calculate the objective L(O, Φ) in Eq. (T0) or Eq.
Update task model weights < " VoLC( Φ).
13:
Update diffusion model weights & t & 11 ())
end for
F
More Experimental Details
F1 More Details on Datasets
ImageNet-C [31]. ImageNet-C consists of corrupted images computed from applying algorithmic corruptions to the ImageNet (65) validation set, which has 50,000 images. To construct ImageNet-C. 15 corruptions that fall into 4 categories are applied separately to the whole validation set, including Gaussian noise (Gauss.), shot noise (Shot), impulse noise (Impl.), defocus blur (Defoc.), glass blur (Glass), motion blur (Motion), zoom blur (Zoom), snow (Snow), frost (Frost), fog (Fog), brightness (Brit.), contrast (Contr.), elastic transformation (Elastic), pixclation (Pixel) and JPEG compression (JPEG). Each corruption type has 5 severity levels and a higher severity level means a more severe distribution shift. We use the highest severity level 5 in all our experiments.
ADE20K (56). ADE20K is a semantic segmentation dataset containing more than 20k images annotated with pixel-level labels on instances and object parts, among which images are for validation. A total of 150 semantic classes are benchmarked for evaluation. We apply the conruptions defined in (3I) with tools provided by (105) to construct ADE20K-C which shares the corruption types with ImageNet-C, at the highest severity level 5. We use this corrupted benchmark for test-time semantic segmentation tasks.
F.2
More Details on Compared Methods
BN Adapt 662 63 66). BN Adapt has the straightforward idea that the statistics in Batch Normal- ization layers are data-dependent, and therefore can be adapted at test time for better generalization.
Tent Tent is a pioneer in addressing the fully test-time adaptation problem. In the work, test-time batch normalization is employed to recalibrate the statistics within Batch Normalization, where features are normalized based the current batch's statistics. Additionally, Tent utilizes entropy minimization to adjust the affine parameters of Batch Normalization layers.
CoTTA focuses on performing test-time adaptation for continually changing distribu- tions. Firstly, it generates more robust and reliable pseudo labels through multiple data augmentations and employs mean-teacher architecture to reduce error accumulation. Secondly, to prevent catas- trophic forgetting, stochastic restoration strategy is proposed, which randomly rolls back portion of the student model's parameters. Meanwhile, the mean-teacher architecture, all the parameters of the student model remain trainable.
COTTA 455.
20

---

EATA [60]. EATA is devoted to improving efficiency and preventing knowledge forgetting. Con- cretely, the unreliable and redundant test samples are filtered out by the value of prediction entropy and similarity to the mean prediction. Besides, an anti-forgetting regularization based on fisher importance proposed. Additionally, only affine parameters in batch normalization layers are adapted.
SAR (59]. SAR points out the unreliability of entropy minimization and the instability of Batch Normalization layers during test-time adaptation. Motivated by this, proposes sharpness-awan and reliable entropy minimization for Group/Layer Normalization-basee models. For time efficiency, only affine parameters in Group/Layer Normalization layers are optimized.
RoTTA is dedicated to conducting test-time adaptation on complex test streams characterized by continually changing data distributions and temporally correlated label distributions. To begin with, it develops a cnddiction-balance sampling strategy grounded in uncertainty and timeliness, ensuring the maintenance of robust snapshot of the test distribution for adaptation. Furthermore, robust batch normalization layer is devised to recalibrate normalization statistics by applying exponential moving averages to selected samples. Lastly, it introduces timeliness reweighting strategy to attain stable and robust adaptation. Only affine parameters in robust batch normalization layers undergo training.
RoTTA 61].
Diffusion-TTA (2). Diffusion-TTA is proposed to adapt pre-trained task models using feedback from pre-trained diffusion models. The integration of the task model into the diffusion model is achieved by modulating the conditioning of the diffusion model using the output of the task model. With the generative objective of diffusion models, the knowledge is backpropagated through conditioning to the task model. Hundreds of timesteps are sampled in diffusion model for single sample as an approximation of the likelihood estimation to improve adaptation performance.
F,3 More Details on Implementation
All pre-trained models involved in our paper are publicly available, including ResNet-50 (GN ViT- B/16 (3 ConyNext-L (LNFF DIT-XL//Dannd Ne based on Stable Diffusion v1.5 from t timm [106] or their official repository. The classifiers and DiT-XI/2 are pre-trained on ImageNet, while ADE20K is used to pre-train SegFormer-B5 and ControlNet.
As for code, we (re)implement all test-time adaptation methods for classification under a framework modified from MMPreTrain [103), except for Diffusion-TTA we adopt its official implementation. For test-time semantic segmentation tasks, we (re)implement all methods under a framework modified from MMSegmentation [104).
All experiments performed are with a batch size of 64, except for part of the analysis in Fig. Our DUSA is trained with a batch size of 8 and a gradient accumulation of 8 steps, to yield an effective batch size of 64. Limited by its implementations, Diffusion-TTI is run with a batch size of 1, and gradient accumulation of 64 steps is applied, also crafting an effective batch size of 64. We follow [32] and use Adam (64) optimizer with a learning rate of 0.00001 (1.0 x 10-5) and a weight decay of 0.0 for both our DUSA and Diffusion- which applies to all classifiers. As for other compared methods, we use Stochastic Gradient Descent (SGD) with momentum 0.9 and a learning rate of 0.00025 (2.5 4) for ResNet-50, while learning rate of 0.001 (1.0 x 10-3) is
used for ViT-B/16, in accordance with the literature [4 59 60 to obtain decent baseline results. For ConvNext-L, we use Adam with learning rate of 0.00001 (1.0 x 10 5) with weight decay of 0.0 for all methods involved. The diffusion model for classification tasks is a Diffusion Transformer
aug
1eiees__24___ mpz
pph
http:///saagaaee////////////ttta ckpt
21

---

DiT-XL/2 [2] trained on ImageNet [65] from scratch. For all classification tasks on ImageNet-C, we adopt the standard pipeline (31] and center crop images to 224 224 for task models. Our DUSA and Diffusion- TTAA both use D//-X1/2 with an input size of 256 x 256 as the diffusion model, therefore we resize the cropped 224 224 image to 256 x 256 before passing to DiT-XII/2 as input. During adaptation, we freeze the VAE encoder and condition embedder of DiT, while training the denoising transformer which functions in a latent space of 32 x 32 x 4. All parameters of the task models are adapted in DUSA, as is done in Diffusion-TTA and COTTA.
For hyperparameters in DUSA, we set 100, A: and m for all classification tasks. For the sake of fair comparison and practical considerations in compute resources, we give Diffusion- TTIA the same budget 6, i.e., 6 timesteps in diffusion models are randomly sampled for the training of each image sample, which applies to all results reported on Diffusion-TTie TTA, including those in Fig. B and Fig. [ For both our DUSA and Diffusion-TTA that involve diffusion models, the noise e added to input data is randomly sampled. As for other compared methods, we follow all hyperparameters in their original setup and please refer to their paper for more detailed hyperparameter settings.
For test-time semantic segmentation, we follow [45) and use a batch size of 1, Adam optimizer with a
learning rate of 0.00006/8 / (6.0 10 10-5/8) and weight decay of 0.0 for all methods. The diffusion model for this task is a ControlNet (48) finetuned from Stable Diffusion v1.5 [20] on ADE20K. Note that ControlNets come with extra conditioning capability, but we dismiss extra conditions beyond text so that it can be recognized as a typical text-to- text-to-image diffusion model. In detail, the conditioning of our used ConvNeXt accepts a colored segmentation map as input, and there is color 0g,00,0,0) for a background/undefinee category, which is not among the 150 ADE20K classes, so we find it suitable to use all-zero colormaps as the conditioning and regard the ControlNet as only receiving meaningful conditions from texts. While adapting, we again freeze the VAE encoders and text embedder, while training the denoising UNet, along with the ControlNet branch plugged in. For the task model SegFormer-BB, the input size is 512 512, so we resize the shorter side of input images to 512 while keeping the aspect ratio for all methods involved. As for our DUSA, the ControlNet requires an input size of 512 x 512. To get the most of semantic priors from diffusion models, we apply a sliding strategy to both ControlNet input (which is a non non-square input with a 512 short side) and SegFormer output, which aligns the task model forward with compared methods, and thus the sliding on one image is finished in two steps. Note that SegFormer-B B5 has a 4x downsample while ControlNet has 8x downsample, therefore the logits of SegForer-- can be larger than the latent space of ControlNet, so we further perform a 2x downsample on the logits to prepare for the aggregation of noise predictions in Eq. (14).
As we resort to the conventional text-to-image formulation of diffusion models, a whole noise estimation map is predicted each time a condition is given. This is much different from classification as the segmentation results are at the pixel level, making an image-level candidate class selection non-trivial. Note that LogitNorm is still present before selection, which is applied to the logits in the channel dimension. We make a little modification to the selection strategy here, instead of attempting to change the structure of diffusion models done in (32), to prove the versatility of DUSA. Specifically, we allow a budget of 20 classes for each input image, which is also split into a task model- top-1 selection budget and random budget, inheriting the spirits of DUSA for classification. We first gather the unique set of top-1 predicted classes from all pixels. If the number of gathered classes already exceeds the budget, we randomly suppress the redundant classes. Otherwise, if there is a surplus in the budget, we further perform a random selection from the remaining classes until the threshold is reached. The subsequent steps are the same as in classification.
F.4 More Details on Compute Resources
We use Nvidia A6000 GPU with 48GB memory for all our experiments. For faster training, we use Automatic Mixed Precision with autocasts fp16 for both classification and semantic segmentation. Additionally for semantic segmentation, gradient checkpointing is enabled for the task model. For test-time adaptation on classifiers with a batch size of 8, our DUSA takes around 43GB of memory and 1.5h training time for a single task (roughly 0.1 Is per image), while our DUSA-U takes around 28GB of memory and 1h training time for a single task (roughly 0.07s per image). For test-time semantic segmentation with a batch size of 1, we experiment on two Nvidia A6000 GPUs, with total of 44GB and 2.5h training time for a single task (roughly 4.5s per image). As perform three independent runs for each task, total of 14 GPU days required for results in
22

---

Table 1 GPU days for Table 2 and 5 GPU days for Table accumulating into around 22 A6000 GPU days for our main results. Preliminary experiments make the full research project require more compute than reported, but it's non-trivial for us to benchmark them all.
G Detailed Ablation Results on DUSA Components
We provide the detailed results in Table for ResNet-50 and ConvNeXt--L on the three variants of corrutiion in the Noise category in Table []] namely Gaussian noise, Shot noise and Impulse noise, along with Pixelate corruption the Digital category for ConvNexttL..
Table 5: Ablation on critical components in DUSA. Components colored rows are not carried over to subsequent rows. Task-level results are provided for the Noise category.
ResNet- 50 (GN)
ConvNeXt-LL (LNN
Variants
k
m
D.F.
D.B.
Gauss.
Shot
Impul.
Gauss.
Shot
Impul.
Pixel
Source-only
0
0
0
22.1
23.0
22.0
56.7
56.2
58,3
42.3
scorc priors inspired loss (4)
23.5
26.7
27.8
31.1
48.1
53.1
9.2
LLogitNorm
42.5
45.3
44.2
59.3
61.7
61.7
adapt diffusion (4)
4
0
4
4
40.5
42.4
40.8
59.0
59.1
55.4
49.3
LogitNorm (4)
4
0
4
45.4
47.4
46.3
64.0
65.5
65.5
70.4
LogitNorn (6)
0
6
45.4
47.6
46.5
64.0
65.6
65.6
70.7
uniform select (6)
45.0
47.3
46.1
64.2
65.4
70.7
multinomial seleet (DUSA)
45.2
47.3
46.4
64.2
65.5
70.8
null conditioning (6) (DUSA-U)
4
2
45.1
47.2
46.1
63.7
65.3
65.1
70.5
H Ensembling Timesteps in DUSA
In DUSA, we formulate our approach to extract knowledge from single timestep, thereby enhancing the efficiency of adaptation. However, it is intriguing to investigate whether an ensemble of multiple timesteps would further improve performance. We experiment on ConyNeXt-L and present our findings in Table 6 The results indicate that, while ensembling timesteps does provide benefits, the performance gains may not be substantial enough to justify the increased computational overhead.
Table Effects of ensembling timesteps our DUSA. Experiments were conducted across four
typical scenarios that fall into four main categories in ImageNet-C.
Timestep(s)
Gauss.
Defoc.
Snow
Contr.
(50)
64.0
50.8
69.5
69.3
(100)
64.2
54.7
70.1
68.9
(200)
63.4
55.1
69.8
100)
64.3
54.0
70.2
69.2
55,1100,000
64.3
55.4
70.2
69.1
Visualization of Test-time Semantic Segmentation Results
We visualize the test-time semantic segmentation results of our DUSA and compared methods in Fig. [) A model checkpoint is saved after test-time adaptation over a whole corrupted ADE20K validation The checkpoint then used to yield segmentation maps. We show results from four main categories of corruption, and segmentation maps are colorized with the ADE20K palette for better visual effects. Our DUSA, which exploits the structured semantic priors underneath the score- based diffusion model, shows superior capability in correcting erroneous predictions and providing fine-grained segmentation results.
23

---

Clean
Corrupted
Source
BN Adapt
Tent
CoTTA
DUSA (Ours)
Ground Truth
Impolse
Defocus
Glass
Snow
Brightess
Elastic
Pixclute
JPEG
Figure 5: Visualization of test-time semantic segmentation results on ADE2OK-C. From left to right: clean image from ADE20K, corrupted version of the image, results from source model, BN Adapt, Tent, CoTTA, our DUSA, and lastly the ground truth. DUSA results exhibit a favorable visual effect.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated the contributions made in the paper and the scope both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of our work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification: We state the assumptions made in all theoretical results and properly referenced the Lemmas involved. Formal proofs are provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided implementation details in Appendix F.3 the algorithm of our method in Appendix E and attached code in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The datasets and pre-trained models used in work are all from previous works and publicly available. We have provided dataset details in Appendix [.]]] model weights URLs in Appendix F.3 and reproducible instructions with code attached in the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
Sttpe//uuiiee//oeuuuoeeee sow.:ssuuuissioooooocco... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
becicc/guuieee gui /ooeeeS/ooodebmiiccccc wwt.SSummmsiiooooo.....oo ssiiccoiccy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification: A brief yet informative specification is provided in Sec. 4 For more details, specify dataset usage details in Appendix F.I models, optimizers and hyperparameters details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We report all our main results with mean & standard deviation, the results obtained by independent rumning over three random seeds for each task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resource are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have preserved anonymity in all submitted materials.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification:
We have discussed the potential societal impacts in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The datasets and models used in the paper are all publicly available and from previous works.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets, models and code used in the paper are from previous works and are publicly available. We have cited related papers, respect their license, and include their license in Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released in our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "iOvoAt0hd7", "reviewer": "Reviewer_V2Wk", "summary": "Tackling the limitations of prior research on test-time adaptation using pre-trained diffusion models, this study expands the diffusion prior for more practical scenarios. The authors extend the use of the diffusion prior to dense prediction tasks, enhancing inference speed with a refined diffusion loss function that eliminates the need for time averaging. This time-agnostic characteristic of the proposed diffusion loss undergoes rigorous validation through mathematical derivations and straightforward experimental validations.\n\nWhile the pre-trained diffusion prior proves effective in various tasks, this study also demonstrates its utility in test-time adaptation for both classification and semantic segmentation with remarkable performance gain. These advancements are particularly crucial for practitioners in the field, offering more efficient and robust tools for real-time applications.", "questions": "## Accuracy of Diffusion-TTA with a Single Time Step as in Figure 3\n\nThe reduction of necessary time steps from an average of 180 to a dramatically lower number is a significant advantage. However, the verification of this reduction and a clear comparison with standard diffusion-TTA remain unclear. Specifically, it is questioned whether averaging out the loss function $\\mathcal{L}_{DUST}(\\theta, phi; t)$ over T=180 under the same conditions as TTA-Diffusion would affect performance. Conversely, the performance of TTA-Diffusion with only a single timestep, as shown in Figure 3, is also in question.\n\n## Formal Proof of Biased Approximation\n\nDiffusion-TTA relies heavily on the Monte Carlo method across up to 180 timesteps, resulting in a biased approximation of likelihood and high computational complexity. This methodological reliance is confusing since Monte Carlo estimations generally converge towards more accurate values with more samples. The paper mentions a biased approximation but lacks concrete proof. It's questioned whether the proposed DUST method can demonstrate that it does not rely on biased approximations; if not, such claims might be considered exaggerated.\n\n## Advantages Over Diffusion-TTA in Dense Prediction Tasks\n\nThere are doubts about whether Diffusion-TTA is also applicable and effective for dense prediction tasks, which is missing in the manuscript.\n\n## Effect of Training Set on Diffusion Prior\n\nIt would be interesting to see the effects of using a diffusion model trained on ImageNet instead of one trained on a large dataset like Stable Diffusion for an apple-to-apple comparison. Given that Stable Diffusion benefits from a larger dataset, not comparing it to other models might be seen as a limitation. Despite the inevitable nature of research progression, the lack of analysis on this aspect in both the current paper and TTA-Diffusion is questioned. The potential for performance improvement remains, considering the model's training on noise through diffusion.", "limitations": "The limitations of the paper have been highlighted within the weaknesses section, with detailed questions addressed above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "## Test-Time Adaptation for Dense Prediction Tasks\n\nRecent advancements have seen the extensive use of large-scale pre-trained diffusion models as priors for various tasks. The application of diffusion priors in test-time adaptation for dense prediction tasks is a particularly intriguing discovery and is deemed to be a valuable approach moving forward.\n\n## Time-Step Efficiency of the Proposed Method\n\nTraditional diffusion models often require generating multiple samples due to the expectation across temporal steps in their loss functions. This research successfully removes the expectation over temporal steps, thereby enhancing efficiency. However, the formulation of the proposed method isn't fully understood, prompting additional questions.", "weaknesses": "## Misleading Title of the Paper\n\nThe algorithm described in the Appendix should ideally be integrated into the main body of the paper.\nAmbiguity in the Use of the Adjective 'Fresh'\n\n## the term \"fresh\" is used ambiguously in lines 66-68:\n> A fresh proposition is provided from a theoretical perspective to extract discriminative priors from score-based diffusion models, which are capable of handling both classification and dense prediction tasks at test time in a single timestep.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The algorithm in Appendix should be integrated into the main paper.**\n\n**A1:** Thanks. Following your advice, we will integrate the algorithm into the main body of the paper in the revision.\n\n**Q2: The term \"fresh\" is used ambiguously in lines 66-68.**\n\n**A2:** Good suggestion. We agree that \"a fresh proposition\" might be ambiguous, and we clarify by restating it as \"a novel proposition\" and will update it in the revision.\n\n**Q3: The verification of this reduction and a clear comparison with standard Diffusion-TTA remain unclear. Whether averaging out the loss function $\\mathcal{L}_{DUSA}(\\theta, \\phi; t)$ over T=180 under the same conditions as Diffusion-TTA would affect performance.**\n\n**A3:** Thanks. Diffusion-TTA is built on the Monte Carlo estimation of likelihood, and demands **randomly sampling up to 180 timesteps** to boost performance. By contrast, our DUSA utilizes semantic priors from a single timestep, and needs only a **single fixed timestep** to achieve superior performance.\n\nFor a clear comparison with the standard Diffusion-TTA, **we put our DUSA under the same conditions of Diffusion-TTA**: (a) we change the timestep selection **from fixed to random sampling**, (b) we increase the timesteps number **from 1 to 180**. The comparison results are as follows:\n|ConvNeXt-L|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|\n|Diffusion-TTA|59.3|50.3|64.7|50.5|\n|DUSA|**64.0 (+4.7)**|**54.7 (+4.4)**|**69.9 (+5.2)**|**68.0 (+17.5)**|\n\nIt can be seen that our DUSA significantly outperforms Diffusion-TTA even under its same conditions.\n\n**Q4: Comparision of Diffusion-TTA and DUSA with only a single timestep, as shown in Figure 3.**\n\n**A4:** Thanks. To compare under only a **single fixed** timestep, **modifications are made to Diffusion-TTA**: (a) we reduce the utilized number of timesteps **from 180 to 1**, (b) we change the timestep selection **from random sampling to a fixed timestep**. The comparison results for $t=100$ are as follows:\n|ConvNeXt-L|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|\n|Diffusion-TTA|59.8|48.1|64.3|61.5|\n|DUSA|**64.2 (+4.4)**|**54.7 (+6.6)**|**70.1 (+5.8)**|**68.9 (+7.4)**|\n\nOur DUSA surpasses Diffusion-TTA by a large margin, justifying the superiority of DUSA on only a single timestep. More results on varied timesteps can be found in **Figure R1** in uploaded PDF.\n\n**Q5: The paper mentions a biased approximation but lacks concrete proof. Whether the proposed DUSA method can demonstrate that it does not rely on biased approximations.**\n\n**A5:** Thanks. Indeed, our original claim \"... for a biased approximation of likelihood\" may cause confusion, and a better expression is \"... to estimate a biased approximation of likelihood\". Actually, the bias **comes from Diffusion-TTA's theoretical approximation of the likelihood** instead of the Monte Carlo method, and we provide a proof below.\n\nDiffusion-TTA is built on the evidence lower bound (ELBO) of log-likelihood in diffusion:\n$$\\log p_\\phi(x_0|c)\\geq \\mathbb{E}\\_q\\Big[\\log\\frac{p\\_\\phi(x\\_{0:T}|c)}{q(x\\_{1:T}|x_0)}\\Big],\\quad(1)$$\nwhere $q$ is the forward process and $p_\\phi$ is the backward process. We can further derive the ELBO in (1) with denoising network $\\epsilon_\\phi$:\n$$\\mathbb{E}_q\\Big[\\log\\frac{p\\_\\phi(x\\_{0:T}|c)}{q(x\\_{1:T}|x_0)}\\Big]=-\\mathbb{E}\\_\\epsilon\\Big[\\sum\\_{t=2}^Tw_t\\\\|\\epsilon-\\epsilon\\_\\phi(x_t,t,c)\\\\|_2^2-\\log p\\_\\phi(x_0|x_1,c)\\Big]+C\\approx-T\\mathbb{E}\\_{\\epsilon,t}[\\\\|\\epsilon-\\epsilon\\_\\phi(x_t,t,c)\\\\|_2^2]+C,\\quad(2)$$\nwhere the **theoretical approximation** is made by simplifying weights $w_t$ to 1 and ignoring the $\\log p\\_\\phi(x_0|x_1,c)$ term. Note that Diffusion-TTA actually estimates the simplified objective in (2), thus the result is **theoretically biased** from real likelihood because the approximation in (2) just prevents the equality sign in (1) from holding.\n\nIn contrast, our DUSA objective in Eq. 10 is built on the theory in Eq. 4 with **no theoretical approximation**. The task model thus directly utilizes semantic priors from the diffusion model. We will add the discussions in the revision.\n\n**Q6: Advantages over Diffusion-TTA in dense prediction tasks. Whether Diffusion-TTA is also applicable and effective for dense prediction tasks.**\n\n**A6:** Thanks. Diffusion-TTA is applicable to dense prediction tasks, as depicted in [26]. However, a pixel-level conditioning capability is assumed on the diffusion model for Diffusion-TTA to be effective, which requires re-training of diffusion models and reduces versatility. We re-train the diffusion model on ADE20K according to [26] to fit segmentation task, and comparison results are as follows:\n|SegFormer-B5|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|\n|Diffusion-TTA|15.3|23.5|23.7|22.9|\n|DUSA|**23.6 (+8.3)**|**24.7 (+1.2)**|**27.3 (+3.6)**|**27.1 (+4.2)**|\n\nIt can be seen that our DUSA consistently outperforms Diffusion-TTA on the dense prediction task.\n\n**Q7: The effects of using a diffusion model trained on ImageNet instead of one trained on a large dataset like Stable Diffusion. Given that Stable Diffusion benefits from a larger dataset, not comparing it to other models might be seen as a limitation.**\n\n**A7:** We appreciate the reviewer's insights in exploring our DUSA on diffusion models varying in training set. We would clarify that, for image classification, an ImageNet pre-trained Diffusion Transformer (DiT) is already used throughout our paper. We only use Stable Diffusion-based ControlNet for the segmentation task. As suggested, we further compare to other diffusion models and provide results for classification in **Table R2** in uploaded PDF and segmentation in **Table R3** in uploaded PDF.\n\nWe can find that a diffusion model trained on the same distribution as the task model is generally more beneficial for adaptation. Intuitively, the diffusion model's being aware of the pre-training data of the task model eliminates the disagreement between them, fostering their cooperation under a narrowed distribution gap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0SZJcPTTAV", "reviewer": "Reviewer_oxUu", "summary": "This paper proposes to perform Test-Time Adaptation (TTA) with the help of diffusion models, based on the theoretical observations that effective discriminative priors are hidden within conditional diffusion losses.\n\nThe method involves a joint adaption of the task discriminative model and the generative diffusion model, with its optimization objective given from a theoretical perspective. To make it more efficient, the paper shows that the objective can be decoupled to fit modern classifier-free guidance (CFG) based models, and can still work well when only using one single timestep and a few selected class candidates. Moreover, the method is shown to naturally fit dense prediction tasks.\n\nExperimental results on fully TTA and continual TTA tasks show the efficacy on both image classification and semantic segmentation, surpassing previous diffusion-based TTA methods.", "questions": "The reviewer has some questions about the proposed objective.\n- Can you give some insights or intuitions on why the modified objective (Eq. 11) works as well as the original objective (Eq. 10)?\n- Related to the above, why the unconditional part in Eq. 11 is still needed?\n  - Firstly, the unconditional version of CFG-based models is much weaker than the conditional version, and they usually do not agree well with each other. Do the unconditional training really help the conditional noise estimation  $\\epsilon_\\phi(x_t,t,c_y)$ ?\n  - Moreover, the conditional part in Eq. 11 does not involve tuning the diffusion weights $\\phi$, then why does the diffusion model (unconditional version) still need to be constrained by a regular CFG objective, given that it has already been well pre-trained?", "limitations": "The authors have adequately discussed most of the limitations, and there have no concerns about negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Novelty in research problem and method.** Utilizing the discriminative natures within generative diffusion models to enhance discriminative  task models is an interesting and underexplored topic, and could be valuable for the research community. Extracting the underlying semantic  knowledge from diffusion losses from a theoretical perspective is novel (Eq. 10). Empirical contributions to improve the efficiency and versatility are also valuable.\n\n**Significant results.** The evaluations on two discriminative tasks under two different TTA settings show the effectiveness of the proposed method, which outperforms previous baselines by large margins, including diffusion-based ones. These empirical results are significant.", "weaknesses": "**Clarity on the training objective in Eq. 11.** Though the objective proposed in Eq. 10 is well-supported from the theoretical perspective, the alternative and more efficient one in Eq. 11 is somewhat counter-intuitive and needs further elucidation. Please refer to the \"Questions\" section.\n\n**The role of diffusion models.** The proposed method (especially for image classification) relies on a conditional diffusion model that is pre-trained on the same image dataset and with the same set of class labels. Though the experiments on semantic segmentation are done with a large text-to-image model pre-trained on web images, the paper does not show whether it is better than a model pre-trained on the same distribution. Besides, how does the quality (e.g., sampling FID, model capacity, training duration, training dataset diversity) of diffusion models affect TTA performance?\n\nThe method also relies on the fine-tuning of diffusion models, rather than only evaluating off-the-shelf models. This makes the TTA process more costly, since diffusion models are usually much larger than discriminative task models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Though Eq. 10 is well-supported from theoretical perspective, the alternative Eq. 11 needs further elucidation. Insights or intuitions on why the modified objective (Eq. 11) works as well as the original objective (Eq. 10)?**\n\n**A1:** Thanks for the valuable question. Please note that we can view Eq. 10 from two perspectives: (a) the task model learns from the diffusion model, (b) weighted optimization is performed so that $\\epsilon_\\phi(x_t,t,c_y)$ with a larger weight $p_\\theta(y|x_0)$ is more responsible for estimating true noise. Therefore, Eq. 10 **encourages the conditional diffusion model to adapt to test-time data**, under **instruction of task model predictions**.\n\nSimilarly, two perspectives of Eq. 11 can be shown. The conditional part adapts the task model as in Eq. 10. For the unconditional part, we start from Eq. 9:\n$$\\epsilon\\approx\\sum_yp(y|x_t)\\epsilon_\\phi(x_t,t,c_y).\\quad(\\text{Eq. 9})$$\nWithout loss of generality, we ignore the estimation error below. In a CFG-based diffusion model, the true noise in Eq. 9 can be unconditionally estimated:\n$$\\epsilon_\\phi(x_t,t,\\varnothing)=\\sum_yp(y|x_t)\\epsilon_\\phi(x_t,t,c_y).\\quad(1)$$\nPlease note that all noise estimations are from **a same diffusion model**, and the weights $p(y|x_t)$ are the **implicit priors in the diffusion model**. Therefore, we can view (1) as an **internal constraint** of the diffusion model. Intuitively, the unconditional part **implicitly enforces conditional adaptation of the diffusion model to test-time data**, now under **instruction of the implicit priors from the diffusion model itself**, explaining why Eq. 11 works as well as Eq. 10. We will add the discussions in the revision.\n\n**Q2: The unconditional version of CFG-based models is much weaker than the conditional version, and they usually do not agree well with each other. Do the unconditional training really help the conditional noise estimation?**\n\n**A2:** Thanks. We agree that the unconditional version is weaker and there can be disagreement. However, this version is still decent for noise estimation, as it also undergoes extensive diffusion training. Besides, in A1 we show that **the unconditional training helps adapt conditional noise estimations to test-time data** in CFG-based models. Therefore, we prefer to view the two versions as **cooperative**, and their agreement is not forced.\n\nWe compare unconditional training with freezing the diffusion model in **Table R1** in uploaded PDF. We can see that the unconditional results are comparable to conditional training. This strengthens our belief that **the unconditional training does help the conditional noise estimations**, with reduced overhead.\n\n**Q3: The conditional part in Eq. 11 does not involve tuning the diffusion weights, why does the diffusion model (unconditional version) still need to be constrained by a regular CFG objective?**\n\n**A3:** Thanks. The regular CFG objective **explicitly enhances both conditional and unconditional noise estimations** for better generation. In contrast, Eq. 11 focuses on utilizing the diffusion model to **facilitate task model adaptation** (conditional part) and leveraging the implicit priors to **adapt the conditional noise estimations** (unconditional part), as shown in A1.\n\nWe constrain the diffusion model with unconditional training to handle a **potential distribution shift** between diffusion model's training set and test-time data. Results are shown in **Table R1** in uploaded PDF. If the diffusion model is generalizable to unseen data, unconditional training can be removed with limited sacrifice in performance (DUSA-Frozen for Gaussian). Otherwise, should the test-time data be out-of-distribution for the diffusion model, the absence of unconditional training might cause degradation (DUSA-Frozen for Defocus). Therefore, the constraints are preferred for a more consistent performance gain.\n\n**Q4: Whether a large text-to-image model pre-trained on web images is better than a model pre-trained on the same distribution for segmentation.**\n\n**A4:** Thanks. Actually, the diffusion model we adopt for semantic segmentation is a ControlNet (based on Stable Diffusion v1.5) fine-tuned on ADE20K, same distribution as the task model. Although not equal to training from scratch, we assume the fine-tuning makes the diffusion model aware of the task model's training data distribution.\n\nWe compare the results of adapting with ControlNet against Stable Diffusion models in **Table R3** in uploaded PDF. It can be seen that the diffusion model's training on the same distribution as the task model **generally merits more performance gain**, as it narrows the gap between the task model and the diffusion model for better cooperation.\n\n**Q5: How does the quality of diffusion models affect TTA performance?**\n\n**A5:** Thanks. We compare the quality of popular diffusion models in **Table R4** in uploaded PDF. We apply our DUSA to them and show classification in **Table R2** and segmentation in **Table R3** in uploaded PDF. It can be seen that while a diffusion model of better quality fosters TTA performance (SD series), one training on the same distribution as the task model is generally more favorable (blue lines).\n\n**Q6: The method also relies on the fine-tuning of diffusion models, rather than only evaluating off-the-shelf models. This makes the TTA process more costly, since diffusion models are usually much larger than discriminative task models.**\n\n**A6:** Thanks. We agree that the fine-tuning of diffusion models makes TTA more costly, but we highlight the substantial performance gain obtained by updating the diffusion model (DUSA) against freezing it (DUSA-Frozen), as shown in **Table R1** in uploaded PDF.\n\nBesides, our DUSA is **more than 100x faster** in comparison with Diffusion-TTA, and approaches the strong traditional TTA method EATA in speed. The results are in **Table R5** in uploaded PDF, where our DUSA shows leading performance with little extra cost.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2bGhfcKsq", "reviewer": "Reviewer_6Z3D", "summary": "This work extends the Diffusion-test time adaption framework of [26] where the diffusion loss is averaged over timesteps to a simpler and theoretically justified framework, where a single timestep of the diffusion model can extract the semantic priors from the generative model. This reduces the instability of the Monte Carlo-based likelihood estimation over the timesteps. The proposed approach is applied to  ImageNet classification on ConvNext and segmentation.", "questions": "1. How is the update performed for the diffusion model as in [26] updating the diffusion model is optional. Does it incur any additional overhead? \n2. Do different tasks, for example, segmentation and classification, have a single timestep where they perform well, or depending on the granularity of the task the choice of the timestep is different?", "limitations": "The limitations of the work are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is well-written and well-motivated with self-contained justification for using a single timestep for extracting the semantic prior from the diffusion model. \n2.  Extensive experiments on the full test-time adaptation and continual test-time adaptation of ImageNet show that the approach outperforms prior work by a large margin. \n3. Adequate ablations are performed to show the effect of  selection of timestep and different components of the proposed framework.", "weaknesses": "1. During optimization updates are performed on the diffusion weights as well. How the joint update for the two networks is performed is unclear. The timestep selection should impact the diffusion model's update schedule and the output logits. \n2. The computational overhead needed for the approach and the gain over the prior work are not discussed. \n3.  The work claims that the approach can be applied to any discriminative task. Class embedding as conditional is generally required for the diffusion model. This should limit the discriminative settings that can be considered for TTA with diffusion models.\n4. The paper does not discuss how are the conditionals computed for the segmentation task. The increase in computational overhead should also be discussed with the increased complexity of the task.  \n5. Single timestep for adapting diffusion models on vision tasks has been discussed in [a] and [b]. These works can be discussed in the related work.\n\n[a] Unsupervised semantic correspondence using stable diffusion. NeurIPS 2023.\n\n[b] Slime: Segment like me. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: How the joint update for the two networks is performed is unclear.**\n\n**A1:** Thanks. Before adaptation, **a single fixed timestep is selected** and other timesteps are dismissed. Hereafter, the diffusion model is **only updated on this single timestep**, i.e., we added a **same** level of noise to all images and denoise them at a **same** timestep. Note that the task model predicts on clean images, thus **not affected by timestep**. The diffusion noise estimations and the task model predictions are then integrated into Eq. 10 for a joint update.\n\n**Q2: The timestep selection should impact the diffusion model's update schedule and the output logits.**\n\n**A2:** Thanks. In our DUSA, the diffusion model is only updated on a **single fixed timestep**. We agree that timestep selection should impact the amount of noise in $x_t$ and the timestep condition $t$ for noise estimation, but we argue that the timestep in DUSA is **fixed throughout adaptation** and how we select it makes no difference to DUSA's pipeline. Besides, the task model predicts logits from clean images and is **not affected by timestep selection**.\n\n**Q3: The computational overhead needed for the approach and the gain over the prior work are not discussed.**\n\n**A3:** Thanks. Firstly, with $K$ classes and $T$ diffusion timesteps, the prior work Diffusion-TTA has a computational complexity of $\\mathcal{O}(T)$, indeed 180 timesteps for each image. By contrast, our DUSA has an $\\mathcal{O}(K)$, further reduced to $\\mathcal{O}(b)(b\\ll K)$ by practical designs (lines 136-139, 193). We list the computation time with gain as follows:\n|ConvNeXt-L|Time/Image (A6000)|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Diffusion-TTA|27.71s|59.3|50.3|64.7|50.5|\n|DUSA|**0.11s (-99.6%)**|**64.2 (+4.9)**|**54.7 (+4.4)**|**70.1 (+5.4)**|**68.9 (+18.4)**|\n\nIt can be seen that our DUSA shows significant gain over Diffusion-TTA with a tremendous reduction (-99.6%) in computational overhead.\n\n**Q4: Class embedding as conditional is generally required for the diffusion model. This should limit the discriminative settings that can be considered for TTA with diffusion models.**\n\n**A4:** Thanks. We agree that diffusion models generally require class embeddings. However, the specific form of class embeddings can vary. For classes with semantic meanings, we can construct a prompt containing class names and use a text model as embedder. For classes without explicit meanings, we can discretize them to IDs and map them to embedding vectors. In this way, we believe any discriminative task can be taken into consideration.\n\n**Q5: How are the conditionals computed for the segmentation task? The increase in computational overhead should also be discussed with the increased complexity of the task.**\n\n**A5:** Thanks. As we stick to text-to-image diffusion models, the conditionals for segmentation are computed **at class level**. Specifically, we construct a prompt \"a photo of a {class}\" and use the diffusion model's text encoder to obtain class embedding. Please note that **a single conditional noise estimation** in diffusion could also be viewed as **a noise estimation for every single pixel**. Therefore, the computation of conditionals for the segmentation task is **as simple as** in classification: given a class, predict the noise. The only difference is the way of ensembling noise estimations with predictions as in Eq. 12, which introduces little overhead.\n\nThe increase in computational overhead is first raised by a large network to adapt in segmentation. Besides, the overhead is partially due to using a larger diffusion model (ControlNet, 1.7B params) than that in classification (DiT, 675M params) and the sliding-window strategy for non-square images. Lastly, the overhead also comes from the unpolished use of our CSM module for this task. We find that our method is versatile and competitive across tasks, and it's majorly the task complexity that adds to overhead. We will add these discussions in the revision.\n\n**Q6: Single timestep for adapting diffusion models on vision tasks has been discussed in [a] and [b]. These works can be discussed in the related work.**\n\n**A6:** Good suggestion. [a] gets strong results in finding semantic correspondences by obtaining transferable prompt embeddings at a single timestep. [b] achieves significant gains in the one annotation-based image segmentation task at any granularity by learning reusable text embeddings at a single timestep. While also using a single timestep, our DUSA focuses on utilizing the diffusion model to adapt a discriminative model. We find the works relevant and will discuss them in the revision.\n\n[a] Unsupervised semantic correspondence using stable diffusion. NeurIPS 2023.\n\n[b] Slime: Segment like me. ICLR 2024.\n\n**Q7: How is the update performed for the diffusion model as in [26] updating the diffusion model is optional. Does it incur any additional overhead?**\n\n**A7:** Thanks. As described in A1, we update the diffusion model on a single fixed timestep. Like [26], updating the diffusion model is optional in DUSA, and we show the performance of freezing the diffusion model in **Table R6** in uploaded PDF. It can be seen that when not updating diffusion model, our method still outperforms [26]. Besides, with updating, our DUSA takes the lead by a large margin, with a vastly reduced overhead of **-99.6%** adaptation time compared to [26], as shown in A3.\n\n**Q8: Do different tasks, for example, segmentation and classification, have a single timestep where they perform well, or depending on the granularity of the task the choice of the timestep is different?**\n\n**A8:** Thanks. We vary the selected timestep for segmentation and report the results in **Figure R2** in uploaded PDF. As the timestep being too large or too small is not preferred (lines 169-172), we focus on a narrower range here. It can be seen that there is no global best for all scenarios, but our $t=100$ proves effective across tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W0oYYi5KUp", "reviewer": "Reviewer_Your", "summary": "The paper proposes DUSA, which uses diffusion models for test-time adaptation. Class codes are chosen for an image based on the results of a pretrained classifier and random sampling; these codes are used to generate noise candidates, which are combined based on the predicted probabilities for the associated classes. Error and weight updates are computed for both the classifier and the diffusion model based on the error between the aggregated and ground truth noises. This takes advantage of the semantics of the diffusion model to handle corrupted, challenging images not only for classification, but also for segmentation. The results include a variety of tasks and classifiers.", "questions": "What is the latency of the method? Limitations mentions that it would not work for real time (e.g. autonomous driving) applications, but how far off is it?\n\nWas any exploration of ensembling time steps attempted?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] Figure 1 is an outstanding, very well-designed portrayal of the method. \n\n[S2] The method is novel, with its use of single time steps and CSM.\n\n[S3] The results are outstanding, with large improvements across multiple tasks, reproducible on multiple models and with hyperparameter settings that do not seem too brittle.", "weaknesses": "[W1] Existing art, e.g. [1], show there may be some benefit to using a few time steps. It also seems that in the image classification regime, the increase in computational complexity is unacceptably large. However, given existing work in the TTA area already uses many steps, it would have been nice to see some exploration of the impact of using more than 1 step, insofar as it would likely make the approximation in Equation 9 more accurate.\n\n[W2] Considering the related work is positioned at the end, the introduction should do a little more to introduce and motivate the task. TTA is substantially more niche than other tasks, and the paper's narrative flow suffers from not making the motivation more explicit.\n\n\n\n\n\n[1] Mukhopadhyay et al., Do text-free diffusion models learn discriminative visual representations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Existing art, e.g. [1], show there may be some benefit to using a few time steps. It would have been nice to see some exploration of the impact of using more than 1 step, insofar as it would likely make the approximation in Equation 9 more accurate. Was any exploration of ensembling time steps attempted?**\n\n**A1:** Thanks for the valuable advice. The inspiring work [1] unveils the discriminativeness of U-Net features in diffusion models and achieves superior unsupervised unified representation learning performance across diverse tasks, where ensembling of timesteps is found beneficial. Our DUSA shares a similar spirit of exploring the discriminativeness in generative diffusion models, thus the ensembling of timesteps is definitely worth trying out.\n\nTo explore the impact of ensembling timesteps as suggested, we experiment by performing DUSA on more than one timestep simultaneously and averaging their losses, and the results are as follows:\n|Timestep(s)|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|\n|{50}|64.0|50.8|69.5|69.3|\n|{100}|64.2|54.7|70.1|68.9|\n|{200}|63.4|55.1|69.8|66.6|\n|{50,100}|**64.3**|54.0|**70.2**|69.2|\n|{50,100,200}|**64.3**|**55.4**|**70.2**|69.1|\n\nIt can be seen that, the ensembling of different timesteps **does bring a further improvement in performance**, thanks to the knowledge from multiple timesteps. Furthermore, our DUSA has its special advantage of draining knowledge from every single timestep, and can still get good results with reduced computational overhead. We appreciate the insightful advice and will add the discussions in the revision.\n\n[1] Mukhopadhyay et al., Do text-free diffusion models learn discriminative visual representations?\n\n**Q2: Considering the related work is positioned at the end, the introduction should do a little more to introduce and motivate the task.**\n\n**A2:** Thanks for the good suggestion. In test-time adaptation, a pre-trained model is updated on the fly to make accurate predictions on the incoming target samples without label access, which is challenging as the target data distribution might drift from that in pre-training. Being a competitive family of generative models, diffusion models exhibit great capability in modeling data distributions and even show discriminative potential in learned features [1,2,3,4], making them a strong candidate to provide guidance for discriminative tasks. In this work, we aim to extract discriminative priors from diffusion models to facilitate the challenging task of test-time adaptation.\n\nWe thank the reviewer's efforts in improving the readability of our manuscript and will further polish up the expression and integrate the above discussions into the introduction in the revision.\n\n[1] Mukhopadhyay et al., Do text-free diffusion models learn discriminative visual representations?\n\n[2] Zhang et al., Diffusionengine: Diffusion model is scalable data engine for object detection.\n\n[3] Zhao et al., Unleashing text-to-image diffusion models for visual perception.\n\n[4] Xu et al., Open-vocabulary panoptic segmentation with text-to-image diffusion models.\n\n**Q3: What is the latency of the method? Limitations mentions that it would not work for real time (e.g. autonomous driving) applications, but how far off is it?**\n\n**A3:** Thanks for the question. As shown by the results in the table below, our DUSA predicts at a rate of 0.11s/image for classification with Diffusion Transformer (DiT), which is **-99.6% in time** compared to Diffusion-TTA. For segmentation, the time is 4.5s/image for segmentation with ControlNet.\n|ConvNeXt-L|Time/Image (A6000)|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Diffusion-TTA|27.71s|59.3|50.3|64.7|50.5|\n|DUSA|**0.11s (-99.6%)**|**64.2 (+4.9)**|**54.7 (+4.4)**|**70.1 (+5.4)**|**68.9 (+18.4)**|\n\nThe latency is largely related to the selection of the diffusion model as well as the task complexity. With this said, we can freeze the diffusion model (DUSA-Frozen) and only update the task model in DUSA, which will still provide decent performance on a handful of tasks with **further reduced latency** and nearly real time, and the results are as follows:\n|ConvNeXt-L|Time/Image (A6000)|Gauss.|Defoc.|Snow|Contr.|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DUSA|0.11s|64.2|54.7|70.1|68.9|\n|DUSA-Frozen|**0.05s**|58.4|33.7|64.2|62.0|\n\nWe will continue researching how to reduce the latency for discriminative tasks further while utilizing the diffusion model's strong power of generative modeling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their efforts in reviewing our paper and the constructive feedbacks that are quite helpful in improving the quality of the paper. We are more than encouraged that reviewers find:

+ our research topic of utilizing discriminative semantic priors within diffusion models to enhance discriminative task models to be **novel**, **interesting** and **valuable** (*Reviewer oxUu*).
+ our method to be with **significant advantage in efficiency** and **intriguing discovery** (*Reviewer V2Wk*), **novel** (*Reviewer Your*, *Reviewer oxUu*) and **valuable** (*Reviewer oxUu*), **well-motivated** (*Reviewer 6Z3D*).
+ our experimental results to show **remarkable gain** (*Reviewer V2Wk*), **outperform by large margins** (*Reviewer 6Z3D*, *Reviewer oxUu*) and **significant** (*Reviewer oxUu*), **adequate ablations** (*Reviewer 6Z3D*), **outstanding** and **reproducible** (*Reviewer Your*).
+ our advancements to be **particularly crucial** (*Reviewer V2Wk*).
+ our paper to be **well-written** (*Reviewer 6Z3D*), with **outstanding** and **well-designed** illustration (*Reviewer Your*).

We highly value the concerns and suggestions from each reviewer and have done our utmost to address them with detailed responses. A one-page PDF is also uploaded to provide better demonstrations of our responses, where the figures and tables are referred to as **Figure R** and **Table R** for the sake of clarity.

We hope that our rebuttal has sufficiently addressed the concerns raised by the reviewers. Please reply if you have any further questions, and we will be more than happy to continue the discussion.

Meta-review: The reviewers acknowledge the soundness and the technical novelty of the paper. The rebuttal has a positive impact on the final ratings and all the reviewers scored weak accept or higher. The ACs followed the reviewers' recommendation.
Decision: Accept (poster)