Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback
Authors: Mihir Prabhudesai
Tsung-Wei Ke
Alexander Cong Li
Deepak Pathak
Katerina Fragkiadaki
Venue: NeurIPS 2023 poster (2023)
Keywords: test-time adaptation
diffusion models
generative models
classification
segmentation
depth prediction
TL;DR: We introduce Diffusion-TTA, an effective plug-and-play method for test time adaptation that uses generative feedback from a pre-trained diffusion model to adapt large-scale pre-trained discriminative models.

Abstract: The advancements in generative modeling, particularly the advent of diffusion models, have sparked a fundamental question: how can these models be effectively used for discriminative tasks? In this work, we find that generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters and depth predictors, to each unlabelled example in the test set using generative feedback from a diffusion model. We achieve this by modulating the conditioning of the diffusion model using the output of the discriminative model. We then maximize the image likelihood objective by backpropagating the gradients to discriminative model’s parameters. We show Diffusion-TTA significantly enhances the accuracy of various large-scale pre-trained discriminative models, such as, ImageNet classifiers, CLIP models, image pixel labellers and image depth predictors. Diffusion-TTA outperforms existing test-time adaptation methods, including TTT-MAE and TENT, and particularly shines in online adaptation setups, where the discriminative model is continually adapted to each example in the test set. We provide access to code, results, and visualizations on our website: diffusion-tta.github.io/

Diffusion-TTA: Test-time Adaptation of Diseriminative Models via Generative Feedback
Mihir Prabhudesai" Tsung- Wei Ke Alexander Ci Deepak Pathak Katerina Fragkiadaki (mprabhud, tsunguek, ac12, dpathak, katef) @cs. cmu edu Carnegie Mellon University
points
70
+22.8 points
CLIP: VIT-L/14
+11.9 points
Top- Accuracy
+1.8 points
26
+14.0 points
VIT-B/32
ConyNext-l.arg
points
22
26
CLIP: VIT-B/16
15
ResNet50
CLIP: VIT-B/32
(a) ImageNet-C (online)
(b) FGVC Aireraft (single-sample)
Figure 1: Diffusion-TTA improves state-of-the-art pre-trained image classifiers and CLIP models across various benchmarks. Our model adapts pre-trained image discriminative models using feedback from pre-trained image generative diffusion models. Left: Image classification performance of pre-trained image classifiers improves after online adaptation. The image classifiers pre-trained ImageNet and adapted on ImageNet-CI in an unsupervised manner using generative feedback. As can be seen, we get significant boost across various model architectures. Right: Accuracy of open-vocabulary CLIP classifiers improves after single-sample adaptation, where we adapt to each unlabelled sample in the test set independently. CLIP is trained millions of image-text pairs collected from the Internet (38], here we test on the FGVC dataset [30]
Abstract
The advancements in generative modeling, particularly the advent of diffusion mod- els, have sparked fundamental question: how these models be effectively used for discriminative tasks? In this work, we find that generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters and depth predictors, each unlabelled example in the test set using generative feedback from a diffusion model. We achieve this by modulating the conditioning of the diffusion model using the output of the discriminative model. We then maximize the image likelihood objective by backpropagating the gradients to discriminative model's 's parameters. We show Diffusion- TTA significantly enhances the aceuracy of vari- ous large-scale pre-trained discriminative models, such as, ImageNet classifiers, CLIP models, image pixel labellers and image depth predictors. Diffusion- TTA outperforms existing test-time adaptation methods, including TTT-MAE and TENT, and particularly shines online adaptation setups, where the discriminative model is continually adapted to each example in the test set. We provide access to code, results, and visualizations on our website: diffusion- tta. www.iiinntttiiggtu.....ooe
Equal Technical Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Currently, all state-of-the-ar predictive models in machine learning, that care to predict an output from an input æ, are discriminative in nature, that means, they are functions trained to map directly x to (a distribution of) y. For example, existing successful image understanding models, whether classifiers, object detectors, segmentors, or image captioners, are trained to encode an image into features relevant to the downstream task through end-to-end optimization of the end objective, and thus ignore irrelevant details. Though they shine within the training distribution, they often dramatically fail outside of it [14 25 mainly because they learn shortcut pathways to better fit p(u]z)) Generative models on the other hand are trained for a much harder task of modeling p(xly). At test-time, these models can be inverted using the Bayes rule to infer the hypothesis ÿ that maximizes the data likelihood of the example at hand, p(xll)) Unlike discriminative models that take an input and predict output in a feed-forwart manner, generative models work backwards, by searching over hypotheses y that fit the data æ. This iterative process, also known as maalysisby-synnthees inference. However, the ability to generate leads to richer and
[52], is slower than feed-forwer a more nuanced understanding of the data, and hence enhances its discriminative potential [22]. Recent prior works, such Diffusion Classifier (27), indeed find that inverting generative models generalizes better at classifying out- outrof-distribution images than popular discriminative models such as ResNet 17] and ViT [1] However, it is also found that pure discriminative methods still outperform inversion of generative methods across almost all popular benchmarks, where test sets are within the training distribution. This is not too surprising since generative models aim solve a much more difficult problem, and they were never directly trained using the end task objective.
In this paper, instead of considering generative and discriminative models as competitive alternatives, we argue that they should be coupled during inference in a way that leverages the benefits of both, namely, the iterative reasoning of generative inversion and the better fitting ability of discriminative models. A naive way of doing so would be ensemble them, that is Pely (p)((ll))) where and stand for discriminative and generative model parameters, respectively. However, empirically we find that this does not result much performance boost, as the fusion between the two models happens late, at the very end of their individual predictions. Instead, we propose to leverage generative models to adapt discriminative models at test time through iterative optimization, where both models are optimized for maximizing sample's 's likelihood.
We present Diffusion-based Test Time Adaptation (TTA) (Diffusion-TTA), a method that adapts discriminative models, such as image classifiers, segmenters and depth predictors, to individual unlabelled images by using their outputs to modulate the conditioning of an image diffusion model and maximize the image diffusion likelihood. This operates as an inversion of the generative model, to infer the discriminative weights that result in the discriminative hypothesis with the highest conditional image likelihood. Our model is reminiscent of an encoder-decoder architecture, where a pre-trained discriminative model encodes the image into hypothesis, such as an object category label, segmentation map, or depth map, which is used as conditioning to a pre-trained generative model to generate back the image. We show that Diffusion- TTA effectively adapts image classifiers for both in- and outto--distribution examples across established benchmarks, including ImageNet and its variants, as well as image segmenters and depth predictors in ADE20K and NYU Depth dataset.
Generative models have previously been used for test time adaptation of image classifiers and segmentors, by co-training the model under a joint discriminative task loss and a self-supervised image reconstruction loss, e.g., TTT-MAE [12] or Slot-TTA (37]. At test time, the discriminative model is finetuned using the image reconstruction loss. While these models show that adaptation boosts performance, this boost often stems from the subpar performance of their initial ford-forward discriminative model. This can be clearly seen in our TTT-MAE bascline [12), where the before- adaptation results are significantly lower than a pre-trained feed-forward classifier. In contrast, our approach refrains from any joint training, and instead directly adapts pre-trained discriminative models at test time using pre-trained generative diffusion models.
We test our approach on multiple tasks, datasets and model architectures. For classification, we test pre-trained ImageNet clasisifers on ImageNet (9, and its -iitt---iittiittion variants (C, R, A, v2, S). Further we test, large-scale open-vocabulee
CLIP-based classifiers on CIFAR 100, Food101, FGVC, Oxford Pets, and Flowers 102 datasets. For adapting ImageNet classifiers we use DIT 36 as our generative model, which is diffusion model trained on ImageNet from scratch. For adapting open-vocabulary based classifiers, we use Stable Diffusion [40] as our generative model. We

---

show consistent improvements over the initially employed classifier as shown in Figure ! We also test on the adaptation of semantic segmentation and depth estimation tasks, where segmenters of SegFormer [49] and depth predictors of DenseDepth []] performance are greatly improved on ADE20K and NYU Depth v2 dataset. For segmentation and depth prediction, we use conditional latent diffusion models 40) that are trained from scratch on their respective datasets. We show extensive ablations of different components of our Diffusion- how diffusion generative feedback enhances discriminative models. We hope our work to stimulate research on combining discriminative encoders and generative decoder models, to better handle images outside of the training distribution. Our code and trained models are publicly available in our project's 's website: diffusion- tta. gi github.... io/
method, and present analyses on
2 Related Work
Generative models for discriminative tasks Recently there have been many works exploring the application of generative models for discriminative tasks. We group them into the following three categories: (i) Inversion-Based Methods: Given a test input and a conditional generative model (r)c), these methods make a prediction by finding the conditional representation e that maximizes the estimated likelihood p=(z]c) of the test input. Depending on the architecture of the conditional generative model, this approach can infer the text caption [ 27), class label [27], or semantic map 7 of the input image. For instance, Li et al. 277 showed that finding the text prompt that maximizes the likelihood of an image under a text-to-image model like Stable Diffusion [4 is a strong zero-sho, classifier, and finding the class index that maximizes the likelihood of an image under a class-conditional diffusion model is a strong standard classifier. (ii) Generative Models for Data Augmentation: This category of methods involves using a generative model to enhance the training data distribution 45 [1].. These methods create synthetic data using the generative model and train discriminative classifier the augmented training dataset. For instance, Azizi et al. 2 showed that augmenting the real training data from ImageNet with synthetic data from Imagen [41] their large-scale diffusion model, improves classification accuracy. (iii) Generative Models as feature extractors: These techniques learn features with a generative model during pretraining and then 42 50). Generative pre-training is typically a strong initialization for downstream tasks. However, these methods require supervised data for model fine-tuning and cannot be directly applied in zero-shot manner.
fine-tune them on downstream discriminative tasks 3 TO,
18
35
Our work builds upon the concept of eversson-BBsse Methods (i). We propose that instead of inverting the conditioning representation, one should directly adapt pre-trained discriminative model using the likelihood loss. This strategy enables us to effectively combine the best aspects of both generative and diseriminative models. Our method is complementary to other methods of harnessing generative models, such as generative pre-training or generating additional synthetic labeled images
Test-time adaptation Test-time adaptation (TTA) also referred as unsupervised domain adaptation (UDA) in this context, is a technique that improves the accuracy of a model on the target domain by updating its parameters without using any labelled examples. Methods such pseudo labeling and entropy minimization (46) demonstrate that model accuracy can be enhanced by using the model's s own confident predictions a form of supervision. These methods require batches of examples from the test distribution for adaptation. In contrast, TTA approaches based on self celf-supervised learning (SSL) have demonstrated empirical data efficiency. SSL methods jointly train using both the task
and SSL loss, and solely use the SSL loss during test-time adaptation [5] 12 16 31
37 44. These
methods do not require batch of examples and adapt each example in the test set independently. Our contribution falls within the SSL. TTA framework, where we use diffusion generative loss for adaptation. We refrain from any form of joint training and directly employ pre-trained conditional diffusion model to perform test-time adaptation of a pre-trained discriminative model.
3 Test-Time Adaptation with Diffusion Models
The architecture of Diffusion-TTA method is shown in Figure 2 and its pseudocode in shown in Algorithm [T] We discuss relevant diffusion model preliminaries in Section 3.T and describe Diffusion-TI. TTA in detail in Section 3.2
3

---

Macseny LEAEEERAEE
Task Output
transform
Input
Pre-trained Discriminative Model fo
Condition c
Diffusion Loss
2
Noise
Noisy
Pre-triined Diffusion Model 6o
Predicted
Actual
Noise
Figure 2: Architecture of Diffusion-TTA. Our method consists of discriminative and generative modules. Given an image z, the discriminative model fo predicts task output y. The task output is transformed into condition c. Finally, we use the generative diffusion model 6a to measure the likelihood of the input image, conditioned on c. This consists of using the diffusion model 6 to predict the added noise from the noisy image Ię and condition c. We maximize the image likelihood using the diffusion loss by updating the discriminative and generative model weights via backpropagation. For task-specific version of this figure, please refer to Figure 7 in the Appendix.
3.1 Diffusion Models
diffusion model learns to model a probability distribution p(z) by inverting process that gradually adds noise to the image The diffusion process is associated with variance schedule {3, (0, 1)) which defines how much noise is added at each time step. The noisy version of sample time /. can then be written क Vaya + vT arş where
(0, 1), is sample from Gaussian distribution (with the same dimensionality x), a 1 3r, and ā₂ TT== ag One then learns a denoising neural network co (2); that takes as input the noisy image 2; and the noise level t and tries predict the noise component c. Diffusion models can be easily extended to draw samples from distribution p(z|c) ()) conditioned on condition where с can be an image category, image caption, semantic map, a depth map or other information
[8 40 54 Conditioning on can be done by adding e as an additional input of the network é Modern conditional image diffusion models are trained on large collections D {(x", N of N images paired with conditionings by minimizing the loss:
Cair (d:D)
Σ
Ies(Van + /I āie, c'e, ell2.
(1)
xx' .''e'ED
3.2 Test-time Adaptation with Diffusion Models
In Diffusion-TTA, the condition e of the diffusion model depends on the output of the discriminative model. Let fe denote the discriminative model parameterized by parameters 6, that takes as input an image y and outputs y = (2). We consider image classifiers, image pixel labellers or image depth predictors candidates for discriminative models to adapt.
For image classifiers, y represents probability distribution T. categories, y E 1) y 1, 1. Given the learnt text embeddings of a text-conditional diffusion model for the L categories & E Rd, j (I..L), write the diffusion condition
Σイ 1/j Ej.
For pixel labellers, y represents the set of per-pixel probability distributions over I. categories, (v² [0, 1)4.," T 1, 1, æ). The diffusion condition then is e -1E/1 Uy ly x). where E, is the embedding of category j.

---

For depth predictors represents a depth map R R**xA where h the width and height of the image, and c y,-
As is differentiable with respect to the discriminative model's weights 0, we can now update them via gradient descent by minimizing the following diffusion loss:
L(0, 6) E.,. E.,elllplVäre vI a.c, cll
We minimize this loss by sampling random timesteps coupled with random noise variables e. For online adaptation, we continually update the model weights across test images that the model encounters streaming manner. For single-sample adaptation, the model parameters are updated to each sample in the test set independently.
Algorithm 1 Diffusion-TTA
1: Input: Test image discriminative model weights 0, diffusion model weights Φ adaptation steps N, batch size B, learning rate 7).
2: for adaptation step (1,..., ......,) N) do
3: Compute current discriminative output y = fe(z)
4: Compute condition based on output y
5: Sample timesteps {t: { and noises {
6: Loss L(0, Φ) マ Ee ( + VI- ag. t,) - =
7: Update classifier weights f
8: Optional: update diffusion weights @ t V (6,
9: end for
10: return updated output y fa
Implementation Details. Our experiments reveal that minimizing the variance in the diffusion loss significantly enhances our test-time adaptation performance. We achieve this by crafting larger batch size through randomly sampling timesteps from a uniform distribution, coupled with randomly sampling noise vectors from standard Gaussian distribution.
We conduct our experiments on a single NVIDIA-A100 40GB VRAM GPU, with a batch size of approximately 180. Since our GPU fits only a batch size of 20, we utilize a gradient accumulation of nine steps to expand our batch size to 180. Consequently, this setup results in an adaptation time of roughly 55 seconds per example. We use Stochastic Gradient Descent (SGD) with learning rate of 0.005 or Adam optimizer [24 with a learning rate of 0.00001. We set momentum 0.9.
For all experiments, we adjust all the parameters of the discriminative model. We freeze the diffusion model parameters for the open-vocabulary experiments in Section 4.2 with CLIP and Stable Diffusion 2.0, and otherwise, update all diffusion model parameters. For CLIP, instead of adapting the whole network we only adapt the LoRA adapter weights (33 added the query and value projection layers of the CLIP model. For our experiments on adapting to ImageNet distribution shift in Section 4.1 we observe a significant performance boost when adapting both the discriminative and generative model. This finding suggests that our method effectively combines the distinct knowledge encoded by these two models. We present a more detailed analyses in Section 4.4
Experiments
We test Diffusion-TTA in adapting ImageNet classifiers 17 29 CLIP models
38. image pixel
labellers [49], and depth predictors [1] across multiple image classification, semantic segmentation, and depth estimation datasets, for both in-distribution and duist--iiittiuution test images. Our experiments aim to answer the following questions:
1. How well does Diffusion-TTI TTA test-time adapt ImageNet and CLIP classifiers in comparison to other TTA methods under online and single-sample adaptation settings?
2. How well does Diffusion-TTe -TA test-time adapt semantic segmentation and depth estimation models?
5

---

3. How does performance of our model vary for in- and out-of-distribution images?
4. How does performance of our model vary with varying hyperparameters, such as layers to adapt, batchsize to use, or diffusion timesteps to consider?
Evaluation metrics. We report the top-1 accuracy, mean pixel accuracy, and Structure Similar- ity index [4]] on classification, semantic segmentation, and depth estimation benchmark datasets respectively.
4.1 Test-time Adaptation of Pre-trained ImageNet Classifiers
We Diffusion-TTA to adapt multiple ImageNet classifiers with varying backbone architectures and sizes: ResNet-18 [17), ResNet-50, ViT-B/32 [I]] and ConwNext-Tinyll.an [29]. For fair comparisions, Diffusion Transformer (DiT-XI/2) [36] as class-conditional generative model, which is trained on ImageNet from scratch.
Datasets We consider the following datasets for TTA of ImageNet classifiers: ImageNet 9 and its out-of-distribution counterparts: ImageNet-C [19] (level-5 gaussian noise), ImageNet-A 21
ImageNet-R (20), ImageNetV2 (39. and Stylized ImageNet [3]
Baselines We compare model against the following state-of-the-art TTA approaches. We use their official codebases for comparision:
TTT-MAE [2 is a per-image test-time adaptation model trained under a joint classification and masked autoencoding loss, and test-time adapted using only the (self-supervised) masked autoencoding loss. For comparison, we use the numbers reported in the paper when possible, else we test the publicly available model.
TENT [46] a TTA method that adapts the batchnorm normalization parameters by minimizing the entropy (maximizing the confidence) of the classifier at test time per example.
COTTA 47 test-time adapts a (student) classifier using pseudo labels, that are the weighted- averaged classifications across multiple image augmentations predicted by teacher model.
ImageNet
ImageNet-A
ImageNet-R
ImageNet-C
ImageNet-V2
ImageNet-S
Customized /IT-1/16 classifier 12
17.5
+TTT-MAE (smplle-sample)
82.0
21.3 (+6.9)
39.2 (+6.2)
27.5 (+10.0)
72.3 (-0.2)
(+0.3)
ResNet18
2.6
TENT (online)
(-6.5)
(-0.8)
34.7 (+0.1)
(+9.5)
(-5.1)
(+2.1)
CoTTA (online)
(.55)
(-0.7)
34.7 (+0.1)
11.7 (+9.1)
(-5.0)
9.7 (+2)
Diffusion-TIT (single-sample)
77.2 (+7.7)
6.1 (44.7)
39,7 (+5.1)
4.5 (+1.9)
63.8 (46.7)
12.3 (+4.6)
ViT-B/32
45.2
39.5
TENT (online)
75.7 (0.0)
9.0 (0.0)
45.3 (+0.1)
38.9 (-0.6)
61.1 (+0.1)
10.4 (-5.4)
CoTTA (online)
75.8 (+0.1)
8.6 (-0.4)
45.0 (-0.2)
40.0
60.9 (-0.1)
1.1 (-14.7)
Diffusion-TII TTA (single-sample)
77.6 (+1.9)
11.2 (+2.2)
46.5 (+1.3)
41.4 (+1.9)
64.4 (+3.4)
21.3 (+5.5)
ConNNext-TH Tiny
81.9
47.8
16.4
70.9
20.2
TENT (online)
79.3 (-2.6)
10.6 (-12.1)
42.7
(-13.7)
19.9 (-0.3)
CoTTA (online)
80.5 (-1.4)
13.2
47.2 (-0.6)
13.7 (-2.7)
68.9 (-2.0)
19.3 (-0.9)
Diffision-TTA (sipgle-sample)
83.1 (+1.2)
25.8 (+3.1)
49.7 (+1.9)
21.0 (+4.6)
71.5 (+0.6)
22.6 (+2.4)
Table Singe-sample test-time adaptation of ImageNet-trainedd classifiers. We observe consistent and significant performance gain across all types of classifiers and distribution drifts.
(ImageNet-CC R, A, V2, and S) for single-sample adaptation in Table Further we present online adaptation results in Table We evaluate on ImageNet-C, which imposes different types of corruption on ImageNet. For both COTTA and TENT we always report anline-adaptation results as they are not applicable the single-sample setting.
We present classification results on in-distribution (ImageNet) and

---

ImageNet Corruption
Gaussian Noise
Fog
Pixelate
Snow
Contrast
Customized VIT-L/16 classifier [P]
17.1
38,7
47.1
35.6
6.9
+ TTT--MAE (online)
37.9 (+20.8)
51.1 (+12.4)
65.7 (+18.6)
56.5 (+20.9)
10.0 (+3.1)
ResNet50
6.3
23.2
26.5
16.7
3.6
TENT (onlinc)
12.3 (+6.0)
43.2 (+18.0)
41.8 (+15.3)
28.4 (+11.7
12 (48.4)
CoTTA (online)
12.2 (+5.9)
42.4 (+17.2)
41.7 (+15.2)
(+11.9)
11.9 (+8.3)
Diffusion-TTi (online)
19.0 (+12.7)
43.2 (+18.0)
50.2 (+23.7)
33.6 (+16.9)
2.7 (-0.9)
VIT-B/32
39.5
35,9
55.0
30,0
31.5
TENT (oaline)
38.9 (-0.6)
35.8 (-0.1)
55.5 (+0.5)
30.7 (+0.7)
32.1 (+0.6)
CoTTA (online)
40.0 (+0.5)
34,6 (-1.3)
54.5 (-0.5)
29.7 (-0.3)
32.0 (+0.5)
Diffusion-TIA (online)
46.3 (+7.0)
56.2 (+20.3)
64.7 (+9.7)
50.4 (420.4)
33.6 (+2.1)
ConyNext-Tinn Tiny
16.4
32.3
37.2
38.3
32,0
TENT (oaline)
2.7 (-13.7)
5.0 (-27.3)
43.9 (+6.7)
15.2 (-23.1)
40.7 (-18.7)
CoTTA (online)
(-2.7)
29.8 (-2.5)
37.3 (+0.1)
26.6 (-11.7)
32.6 (+0.6)
DDiffusion TTA (online)
(+31.0)
65.9 (+33.6)
69.0 (+31.8)
(+24.3)
46.2 (+14.2)
Con-Next-Large
34.4
44.5
39,8
TENT (online)
(-2.2)
53.5 (+19.1)
51.1 (+1.8)
44.6 (+0.1)
52.4 (+12.6)
CaTTA (online)
(+0.3)
(-18.7)
34.6 (-15.3)
(-36.8)
(:29.1)
Diffusion-TTA (online)
54,9 (421.9)
67.7 (433.3)
71.7 (422.4)
64.8 (+20.3)
55.7 (+15.9)
Table 2: Online test-time adaptation of ImageNet-trained classifiers on ImageNet-C. Our model achieves significant performance gain using online adaptation across all types of classifiers and distribution drifts.
Our conclusions are as follows:
Our method consistently improves classifiers on in-distribution (ImageNet) and OOD test images across different image classifier architectures. For all ResNet, ViT, and ConvNext-Ting Tiny, we observe significant performance gains.
Diflusion-TTA outperforms TENT and COTTA across various classifier architectures. (ii)
Our results consistent with the analysis in [55]: methods that primarily work under online settings (TENT or CoTTA) are not robust to different types of architectures and distribution
shifts (see Table 4 in (55)).
(iii) Diffusion-TTA outperforms TTT-MAE even with smaller-size classifier. ConvNext- (28.5M params) optimized by Diffusion-TII TTA (online) achieves better performance than a much bigger custom backbone of VIT-L+ VIT-B (392M params) optimized by TTT-MAE (online). (iv) Diflusion-TTA significantly improves ConyNext-Large. in online setting. ConyNext- Large is the largest available ConvNext model with 197M parameters and achieves state-of- of-thh--rtttttt performance on ImageNet.
4.2 Test-time Adaptation of veen-Vocabulary CLIP Classifiers
CLIP models are trained across millions of image-caption pairs using contrastive matching of the language and image feature embeddings. We test Diffusion-TTA for adapting three different CLIP models with different backbone sizes: ViT-B/32, ViT-B/16. and ViT-L/14. Experiments conducted under single-sample adaptation setup.
We convert CLIP into a classifier following 38 where we compute the latent embedding for each text category label the dataset using the CLIP text encoder, and then compute its similarity with the image latent to predict the classification logits across all class labels. Note that the CLIP models have not seen test images from these datasets during training.
Datasets We consider the following datasets for TTA of CLIP classifiers: ImageNet 9.. CIFAR- 100 26), Food101 (6, Flowers 102 [32], FGVC Aireraft (30), and Oxford-IIIT Pets 34 annotated with 1000,100,101, 102, 100, and 37 classes, respectively.
7

---

Food101
CIFAR-100
Aircraft
Oxford Pets
Flowers
ImageNet
CLIP-ViT-B32
82.6
61.2
17.8
82.2
66.7
57.5
+ Diffusion-TTA (Ours)
$6.2
(+3.6)
62.8 (+1.6)
21.0 (+3.2)
84.9 (+2.7)
67.7 (+1.0)
60.8 (43.3)
CLIP-VIT-B/16
88.1
69.4
22.8
85.5
69.3
62.3
Diffusion-TIA (Ours)
88.8 (+0.7)
69.0 (-0.4)
24.6 (+1.8)
86.1 (+0.6)
71.5 (+2.2)
63.8 (+1.5)
CLIP-ViT-L/14
93.1
79.6
32.0
91.9
78.8
70.0
-DiffusionTTAA (Ours)
93.1 (0.0)
80.6
(+1.0)
33.4 (+1.4)
92.3 (+0.4)
79.2 (+0.4)
(+1.2)
Table 3: Single-sample Test-time adaptation of CLIP classifiers. Our evaluation is performed across multiple model sizes and a variety test datasets.
We show test-time adaptation results in Table Our method improves CLIP classifiers of different sizes consistently over all datasets, including small-scale (CIFAR-100), large-scale (ImageNet), and fine-grained (Food101, Aircraft, Pets, and Flowers102) datasets.
Corruption:
Clean
Gaussian- Noise
Fog
Frost
Snow
Contrast
Shot
Segmentation: SegFormer
66.1
65.3
63.0
58.0
55.2
65.3
63.3
Diffusion-TT -TAA
66.1 (0.0)
66.4 (+1.1)
65.1 (+2.1)
58.9 (+0.9)
56.6 (+1.4)
66.4 (+1.1)
63.7 (+0.4)
Depth: DenseDepth
79.1
81.6
72.7
77.4
81.3
Diffusion-TTA
92.6 (+0.3)
82.1 (+3.0)
84.4
(+2.8)
73.0
(40.8)
74.1
(+1.)
77.4
82.1 (+0.8)
Table 4: Single-sample test-time adaptation for semantic segmentation on ADE20K and depth estimation NYU Depth v2. We observe consistent and significant performance gain across all types distribution drifts.
Image
ground-truit
Before-THe TTA
After-TTA
Figure 3: Semantic segmentation before and after TTA. We colorize ground-truth / predicted segmentation based on discrete class labels. Our method improves segmentation after adaptation.
4.3 Test-time Adaptation Semantic Segmentation and Depth Estimation
We use Diffusion-TTA adapt semantic segmentors of SegFormer 49 and depth predictors of DenseDepth [I] We use a latent diffusion model [40] which is pre-trained on ADE20K and NYU Depth v2 dataset for the respective task. The diffusion model concatenates the segmentattion/dephh map with the noisy image during conditioning.
Datasets We consider ADE20K and NYU Depth v2 test sets for evaluating semantic segmentation and depth estimation. Note that both discriminative and generative diffusion models trained on the same dataset. We evaluate with different types of image corruption under single-sample settings.
We show quantitative semantic segmentation and depth estimation results in Table 4 Further we show
some qualitative semantic segmentation results before and after TTA in Figure 3 Our Diffusion-TTA improves both tasks consistently across all types of distribution drifts.

---

4.4 Ablations
We present ablative analysis of various design choices. We study how Diffusion-TTi varies with hyperparameters such as diffusion timesteps, number of samples per timestep and batchsize. We also study the effect adapting different model parameters. Additionally we visualize before and after TTA adaptation results in Figure
Ablation of hyperparameters In T we ablate hyperpa- rameters our model on ImageNet-A dataset with ConvNext as pre-trained classifier and DiT our diffusion model. If we perform test-time adaptation using single randomly sampled timestep and noise latent (+diffusion ТTA), we find significant reduction in the classification accuracy (-2.9%). Increasing the batch size from 1 to 180 by sampling random timesteps from an uniform distribution (+ timestep aug BS=/80) gives significant boost in accuracy (+4.8%) As shown in Figure 4 correct class labels do not always have lower loss across all timesteps, and thus using only single diffusion step is bad approximation of the likelihood. Further sampling random noise latents per timestep (+ noise aug) gives an added boost of (+0.2%% Finally, adapting the diffusion weights of DiT (+ adapting diffusion weights) in Section 4.1 gives further (+1.1%) boost.
0.50
grouns-truth class
0.05
Diffupion
DXM
.005
...9
Diffusion
Timestaps
Figure 4: Diffusion loss of class labels at each diffusion timestep.
The ground-truth class does not al- ways have lower loss across timesteps, compared to incorrect classes. Thus using only single dif- fusion timestep is bad approxima- tion of the likelihood.
Ablation of parameters/layers to adapt In Table 6 we ablate different paramters to adapt. We conduct our ablations on on ImageNet and ImageNet-R while using ResNet18 and DiT our discriminative and generative models. We compare against following baselines: 1. we randomly initialize the classifier instead of using the pre-trained weights we do not use any pre- pre-trained classifier, instead we initialize the logits using zeros and optimize them per example using diffusion objective (adapr logits), we average the probabilities of the adapt logits baseline with the probabilities predicted by ResNet18 (adapt logits + ensemble),
we adapt only the batch normalization layers ResNet18 (adapt BN), we adapt only the last fully connected layer in ResNet18 (adapt last FC layer), 6. we adapt the whole ResNet18 (adapt classifier), and 7. we adapt the whole ResNet18 and DiT, which refers to our method Diffusion-TTA Adapting the whole classifier and generative model achieves the best performance, We randomly sample one image per category for the experiment.
ImageNet-A
ImageNet
ImageNet-RR
Next-TTxy 29
22.7 19.8 (-2.9)
ResNet18
68.4
37.0
diffusion Joss TTA
random init adapt logits adapt logits ensemble
D.7
4,0
timestep aug
24.5 (+4.8)
71.8
31.0
+ noise aug adapting diffusion weights
24.7 (+0.2)
72.6
36.5
25,8 (+1.1)
adapt
69.0
37.0
adapt FC layer
68.6
37.5
Table 5:
Ablative analysis of
adapt classifier
73.0
38.0
adapt classifier adapt diffusion (Ours)
42.6
Diffusion-TTA components for Con- vNext classifier and DiT diffusion model. We evaluate single-sample adptation for classification on ImageNet-A.
Table 6: Ablative analysis of model / layer adaptation. We evaluate single-sample adaptation for classifica- tion on ImageNet and ImageNet-BR R. Our Diffusion-TT: adapts both the classifier (ResNet18) and diffusion model (DiT), achieving best adaptation improvement.
4.5 Discussion Limitations
Diffusion- TTAsiioo-TTAA optimizes pre-trained discriminative and generative models using a generative dif- fusion loss. A trivial solution to this optimization could be the discriminative model fo overfitting

---

Ground buih RGB
Refon- TTA top-3 elaviification
Refore-TAA RGR Predicrion
Afte-TTAA classification
Afle-TTA RGB Prodiction
1.-
60
Figure 5: Visualizing Diffusion-TTA improvement across adaptation steps. From left right: We show input image, predicted class probabilities before adaptation (green bars indicate the ground truth category), predicted RGB image via DDIM inversion when the diffusion model is conditioned on predicted class probabilities, the predicted class probabilities after adaptation and predicted RGB image after adaptation. As can be seen, Classification ability of the classifier improves diffusion model's 's ability to reconstruct the input image improves, thus indicating that there is strong correlation between the diffusion loss and the classification accuracy.
to the generative loss and thus predicting precisely (clz). Empirically find that this does not happen: Diffusion- TTA consistently outperforms Diffusion Classifier (27). as shown in Table 7 We conjecture that the reason why the diseriminative model does not overfit to the generative loss is because of the weight initialization of the (pre-trained) discriminative model, which prevents from converging this trivial solution. For instance, if the predictions of the generative and discriminative model are too far from one another that is: distance(pa p, (p([)))) is too high, then it becomes very difficult to optimize 0 reach location such that it predicts (c(r)) To verify our conjecture, we run a few ablations in T i) We randomly initialize 0, ii) Instead of optimizing we directly optimize the conditioning variable e. In both cases, we find that results are significantly worse than when optimizing the pre-trained discriminative model.
Diffusion-TTA modulates the task conditioning of an image diffusion model with the output of the discriminative model. While this allows us to use readily available pre-trained models it also prevents us from having tighter integration between the generative and the discriminative parts. Exploring this integration is a direct avenue of future work. Currently the generative model learns p(x1z), where is the input image. Training different conditional generative model for different layers such as p([1ly) or p(2)) where 11 and are features at different layers, could provide diverse learning signals and it worth studying. Finally, Diffusion-TTA is significantly slow multiple timesteps to get good estimate of the diffusion loss. Using Consistency Models 43 to improve the test-time-adapee -dmettdaptttion speed is a direct avenue for future work.
have to sample
Conclusion
We introduced Diffusion-TTA, test time adaptation method that uses generative feedback from a pre-trained diffusion model to improve pre-trained discriminative models, such as classifiers, segmenters and depth predictors. Adaptation is carried out for each example in the test-set by backpropagating diffusion likelihood gradients to the discriminative model weights. We show that our model outperforms previous state-of-the-art TTA methods, and that it is effective across multiple discrininative and generative diffusion model variants. The hypothesis of visual perception as inversion of generative model has been pursued from early years in the field [33] 53) The formidable performance of today's generative models and the presence forward and top-down pathways in the visual cortex [15] urges to revisit this paradigm. We hope that our work stimulates further research efforts in this direction.
10

---

Acknowledgements We thank Shagun Uppal. Yulong Li and Shivam Duggal for helpful discussions. This work supported by DARPA Machine Common Sense, an NSF CAREER award, AFOSR Young Investigator Award, and ONR MURI N00014-22-1-2773. AL is supported by the NSF GRFP DGE1745016 and DGE2140739.
References
[1] I. Alhashim and P. Wonka. High quality monocular depth estimation via transfer learning. arXiv e-prints, abs/1812.1 ar11:arXiv:11122 1941, 2018. URI. httts://arxiv..t/ wwwgg/bb/////g
11941
[2] S. Azizi, S. Korbblith, C. Saharia, M. Norouzi, and J. Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprini arXiv:2304 08466, 2023.
[3] D. Baranchuk, Rubachev, A. Voynov, V. Khrulkov, and A. Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21 2021.
[4] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[5] A. Bartler, A. Bühler, F. Wiewel, M. Döbler, and B. Yang. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pages 3080-3090. PMLR, 2022.
[6] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[7] R. Burgert, K. Ranasinghe, X. Li, and M. S. Ryoo. Peckaboo: Text to image diffusion models zero-shot segmentors. arXiv preprint arXiv:2211.33222 2022.
[8] K. Clark and P. Jaini. Text-to-image diffusion models zero-shot classifiers. arXiv preprint arXiv:2303.152333 2023.
[9] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 148-255. Icee, 2009.
[10] Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782 2016.
[11] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1xXi:20000.11229 2020.
[12] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. Advances Neurul Information Processing Systems, 33:29374-29338 2022.
[13] R. Geirhos, P. Rubisch, Michaelis, M. Bethge, A. Wichmann, and W. Brendel. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and ro- bustness. In International Conference on Leurning Representations, 2019. URL https //openrevies net/foruumiid-Byyhh91
[14] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 1(:65-66677, 2020.
[15] C. D. Gilbert and M. Sigman. Brain states: Top-down influences in sensory processing. Neuron, 54(5):677-666, 2007. ISSN 0896-6273. doi: URI.httss: httph:ttt/:/sttt //w::////wwww... wwtp::/wwwwnnneeeeee..........ee com/ htt.a///w......eeee/.e.ecooooaaa
[16] J.-B. Grill, F. Strub, Altché, C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new ap- proach to self-supervised learning. Advances neural information processing systems, 33: 21271-21284, 2020.
11

---

[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-
778, 2016.
[18] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[19] D. Hendryeks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv: /2261, 2019.
[20] D. Hendryeks, S. Basart, Mu, S. Kadavath, Wang, F. Dorundo, Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[21] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[22] G. E. Hinton. To recognize shapes, first learn to generate images. Progress in brain research, 165:535-547, 2007.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint urXiv:2106.09685 2021.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6900 2014.
[25] W. Koh, Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. I.. Phillips, S. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and Liang. WILDS: benchmark wn-the-wild distribution shifts. CoRR, abs2201..07421, 2020.
Uttpsss/://aara org/ abs/2012 07421
[26] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[27] A. C. Li, M. Prabhudesai, S. Duggal, E. Brown, and D. Pathak. Your diffusion model is secretly a zero-shot classifier. arXiv preprint w.Xiv:2303,66233, 2023.
[28] Y. Li, H. Liu, Q Wu, F. Mu, J. Yang, J. Gao, C. Li, Li, and Y. J. Lee. Gligen: Open-set grounded 2023.
text-to-image generation. arXiv preprint arXiv:2301.0700
[29] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986 2022.
[30] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[31] S. Manli, N. Weili, De-An, Y. Zhiding, Tom, A. Anima, and X. Chaowei. Test-time prompt tuning for zero-shot generalization in lision-language models. In NeurlPS, 2022.
[32] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Gruphics & Image Processing, pages 722-729. IEEE, 2008
[33] B. A. Olshausen. Perception an inference problem. 2013. URL https://apt semanticscholar. waaeerorgopppooo org/CorpusII:::: 1194136
[34] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
[35] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
12

---

[36] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[37] M. Prabhudesai, A. Goyal, S. Paul, S. Van Steenkiste, M. S. Sajjadi, G. Aggarwal, T. Kipf, D. Pathak, and K. Fragkiadaki. Test-time adaptation with slot-centric models. In International Conference on Machine Learning, pages 28151-28160 PMLR, 2023.
[38] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8876. PMLR, 2021.
[39] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[40] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[41] C. Saharia, W. Chan, S. Saxena, I.. Li, Whang, L.. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35: 36479-36494, 2022.
[42] M. Singh, Q. Duval, K. V. Alwala, H. Fan, V. Aggarwal, A. Adcock, A. Joulin, Dollár, C. Feichtenhofer, R. Girshick, et al. The effectiveness of mae pre-pretraining for billion-scale pretraining. arXiv preprint arXiv:2303. 13496, 2023.
[43] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. 2023.
[44] Y. Sun, X. Wang. Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International conference machine learning, pages 9229-9248. PMLR, 2020.
[45] B. Trabucco, K. Doherty, M. Gurinas, and R. Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944 2023.
[46] D. Wang, F. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 2020.
[47] Q Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[48] Z. Wang. A. C. Bovik, H. R. Sheikh, and F. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
[49] E. Xie, W. Wang, Z. Yu, A. Anandkumar, M. Alvarez, and Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 14:17077-10000, 2021.
[50] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803. 2023.
[51] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint 2rXi::2302.115500 2023.
[52] A. Yuille and D. Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10:301-8, 08 2006. doi: 1..00166.ics.20000550000.
[53] A. Yuille and D. Kersten. Vision as Bayesian inference: analysis by synthesis? Trends in Cognitive Sciences, 10:301-308, 2006.
[54] L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprinl arXiv:2302.05554, 2023.
[55] H. Zhao, Y. Liu, A. Alahi, and Lin. On pitfalls of test-time adaptation. arXiv preprint 23X.:23066.3536.. 2023.
13

---

A Appendix
We present Diffusion-TTA, test-time adaptation approach that modulates the text conditioning of text conditional pre-trained image diffusion model to adapt pre-trained image classifiers, large-scale CLIP models, image pixel labellers, and image depth predictors to individual unlabelled images. We show improvements on multiple datasets including ImageNet and its out-of-distribution variants (C, V2, and S). CIFAR-I00, Food101, FGVC, Oxford Pets, and Flowers over the initially employed classifiers. We also show performance gain on ADE20K and NYU Depth v2 under various distribution drifts the pre-trained pixel labellers and depth predictors. In the Supplementary, include further details on our work:
1. We provide comparison Diffusion Classifier open- Vocabulary classification in Sec- tion ti. A.1
2. We provide additional results with DiT ObjectNet dataset in Section A.2
3. We present architecture diagrams for the tasks of image classification and semantic segmen- tation in Section
4. We provide a detailed analysis of the computational speed of Diffusion-TII. in Section A.4
5. We detail the hyperparameters and input pre- pre-proeessing in Section A.5
6. We provide details of the datasets used for our experiments in Section
A.6
Food101
CIFAR-
FGVC
Oxford Pets
Flowers102
ImageNet
Diffusion Classifier 27
77.9
42.7
24.3
85.7
56.8
58.4
CLIP-ViT-L/14
93.1
79.6
32.0
91.9
78.8
70.0
Diiffusion- TTA (Ours)
93.1 (0.0)
80.6 (+1.0)
33.4 (+1.4)
(+0.4)
79.2
(+0.4)
(+1.2)
Table 7: Comparison to Diffusion Classifier open-vocabulary classification. Our model outperforms Diffusion Classifier consistently over all benchmark datasets.
A.1 Comparison to Diffusion Classifier Open- Vocabulary Classification
We perform simgle-sample test-tiem adaptation of CLIP classifiers and compare to Diffusion Clas- sifier (27) open-vocatee ponn-vocabllary classification. We test Food101, CIFAR-100, FGVC, Oxford Pets, Flowers 102, and ImageNet datasets. As shown Table 7 Diffusion- TTA consistent improves CLIP-VIT-L/14 and outperforms Diffusion Classifier over all benchmark datasets.
ObjectNet
ImageNet
ObjectNet
Customized VIT-L/16 classifier 12
37.8
pose
(b) scene
(c) viewpoint
-TTT-MAE
38,9
ResNet18
23.3
-DiffusionTTI
26.0 (+2.7)
VIT-B/32
26.6
Diffusion-TTA
31.3 (+4.7)
ConvNext-Tine
41.0
Diffusion-TTA
43.7 (+2.7)
Table 8: Single-sample test-time adap- tation on ObjectNet. We use ImageNet pre-trained ResNet18, ViT-B/32, and Con- vNext as our classifiers. Our method im- proves pre-trained image classifiers on out- of-distribution images of ObjectNet.
Figure 6: Comparison of image samples from Ima- geNet and ObjectNet. ObjectNet images have more (a) different poses, (b) complex background scenes, and (c) camera viewpoints.
14

---

A.2 Test-time Adaptation of Pre-trained ImageNet Classifiers on ObjectNet
We extend Table in our paper to ObjectNet 4 dataset. ObjectNet is a variant of ImageNet, created by keeping ImageNet objects in uncommon configurations and poses as shown in Figure ObjectNet has 113 common classes with ImageNet, a single class in ObjectNet can correspond to multiple classes in ImageNet. If the predicted class belongs it's respective set ImageNet classes, then it's considered as 'correct' if not 'incorrect'. We report the results for ObjectNet in Table & following the setup of Section [.1 in the main paper. Our Diffusion-TTA. consistently improves pre-trained ImageNet classifiers with different backbone architectures.
u.1.
i
C
Separie
(a) Diffusion-TT TTA for image classification
(b) Diffusion- -TTA for semantic segmentation
Figure 7: Architecture diagrams for the tasks of (a) image classification and (b) semantic segmentation. For image classification, we generate a text prompt vector (condition e) by doing weighted average over the probability distribution predicted by the pretrained classifier. We then condition on a pre-trained text-conditioned diffusion model via adaptive layer normalization or cross attentions. For semantic segmentation, we generate per-pixel text prompt (condition c) using pre-trained segmentation model. We then condition e on a pre-trained segementation conditioned Diffusion models via channel-wise concatenation. In the Figure on @ denotes channel-wist concate- nation.
A.3 Architecture Diagrams for the Tasks of Image Classification and Semantic Segmentation
We show architectural details for the tasks of image classification and semantic segmentation in Figure 7 Both tasks share the same architectural design, but differ in the way to condition on For image classification, we generate a text prompt vector (condition e) by weighted averaging over the probability distribution predicted by the pre-trained classifier. We condition text-conditioned diffusion model via adaptive layer normalization or cross attentions. For semantic segmentation, generate per- per-pixel text prompt (condition e) using pre-trained image pixel labeller. We condition e on a pre-trained segementation conditioned Diffusion models via channel-wise concatenation.
pre-trained
A.4 Analysis of Computation Speed
We compare the computational latency between Diffusion-TTA and Diffusion Classifier 27 on image classification. Diffusion Classifier (27] a discrete label search method that requires a forward pass through the diffusion model for each category in the dataset. The computational time increases linearly with the number categories. In contrast, Diffusion-TTA is gradient- based optimization method that updates model weights to maximize the image likelihood. The computational speed depends on the number of adaptation steps and the model size, not the number of categories. As shown in Figure 8 out method is less efficient than Diffusson Classifier with fewer categories, but more efficient with more categories. Notably, discrete label search methods, like Diffusion Classifier, do not apply to online adaptation and would be infeasible for dense labeling tasks (searching over pixel labellings is exponential wrt the number pixels in the image).
Computation compared to Diffusion Classifier. Diffusion Classifier [27] inverts diffusion model by performing discrete optimization over categorical text prompts, instead we obtain continuous gradients to search much more effectively over pre-trained classifier's parameter space. This design choice makes significant trade-offs in-terms of computation costs. For instance, Diffusion Classifier requires to do forward pass for each category seperately and thus the computation increases lincarly
15

---

10000.0
Inference time per Image Seconds)
Diffusion Classifier (Li et al.)
Ours
Liner wit Number Categories
100.0
Constant Number of Categories
1.0
CFAR100 Cangera))
Pets 27 Calagonee
Food101 Calgoraa)
ImageNetA (20 Calagemu)
maageNet-1K
ImageNet 21K
(1000 Calgormu
(21,00 Calgores)
Figure 8: Comparison of computation speed between Diffusion-TTA and Diffusion Classifier on image classification. We plot the required computation time with increasing numbers of image categories. Diffusion Classifier [27 requires a forward pass through the diffusion model for each category in the dataset and the computation increases linearly with the number of categories. On the other hand, our method adapts pre-trained classifiers and thus the computation is instead dependent on the throughput of the classifier. Our method becomes more computationally efficient than Diffusion Classifier as the number of categories in the dataset increases.
with the number categories, however for us it's independent of the number of categories. We compare the per-example inference speed in Figure 3 across various datasets.
A.5
Hyper-parameters and Input Pre- Pre-Processing
For nil-time-addatationn of individual images, we randomly sample 180 different pairs of noise and timestep for each adaptation step, composing a mini-batch size 180. Timestep sampled over an uniform distribution from the range 1 to 1000 and epsilon is sampled from unit gaussian. We apply 5 test-time adaptation steps for each input image. We adopt Stochastic Gradient Descent (SGD) (or Adam optimizer (24]), and set learning rate, weight decay and momentum to 0.005 (or 0.00001), 0, and 0.9, respectively. To isolate the contribution the improvement due to the diffusion model we do not any form of data augmentation during test-time adaptation.
We use Stable Diffusion v2.0 40 to adapt CLIP models. For the CLIP classifier, we follow their standard preprocessing step where we resize and center-crop to a resolution of 224 x 224 pixels. For Stable Diffusion, we process the images by resizing them to a resolution 512 512 pixels which is the standard image size in Stable Diffusion models. For the CLIP text encoder in Stable Diffusion and the classifier we use the text prompt of "a photo of <class_name>" where sclass_names name of the class label as mentioned in the dataset.
For the adaptation of ImageNet classifiers, use pre- pre-trained Diffusion Transformers (DiT) specifically their XL/2 model of resolution 256 x 256, which is trained on ImageNetIK. For ImageNet classifiers we follow the standard data pe-processing pipeline where we first resize the image to 232 x 232 and then do center crop to resolution of 224 x 224 pixels. For DiT we resize the image 256 256, before passing input to their model.
A.6
Datasets
In the following section, we provide further details on the datasets used in our experiments. We adapt CLIP classifiers to the following datasets incuding ImageNet: 1. CIFAR-100 (2) is a compact, generic image classification dataset featuring 100 unique object classes, each with 100 corresponding
test images. The resolution of these images is relatively low, standing at size of 32 x 32 pixels.
Food101 ( is an image dataset for fine-grained food recognition. The dataset annotates 101 food categories and includes 250 test images for each category. 3. FGVC Airplane 30 is an image dataset for fine-grained categorization containing 100 different aircraft model variants, each with 33 or 34 test images. 4. Oxford Pets (34 includes 37 pet categories, each with roughly 100 images for
16

---

Food101 CIFAR-100
FGVC
Oxford Pets
Flowers 102
ImageNet
Figure 9: Image datasets for zero-shot classification. From left to right: we show an image example obtained from Food101, CIFAR-100, FGVC, Oxford Pets, Flowers102, and ImageNet dataset. CL.IP classifiers are not trained but tested on these datasets.
testing. 5. Flower102 32 hosts 6149 test images, annotated in 102 flower categories commonly found in the United Kingdom. For all of these datasets, we sample 5 images per category for testing.
ImageNet
ImageNet-C
ImageNet-A
ImageNet-R
ImageNetV2
ImageNet-S
Figure 10: Image samples from ImageNet and its out-of-distribution variants. From left to right: we show images the puffer category the ImageNet, ImageNet-C, ImageNet-A, ImageNet-R, ImageNetV2, and Stylized ImageNet dataset. ImageNet-C applies different corruptions to images. ImageNet-A consists of real-world image examples that are misclassified by existing classifiers. ImageNet-R renders images in artistic styles, e.g. cartoon, sketch, graphic, etc. ImageNetV2 attempts to collect images from the same distribution as ImageNet, but still suffers from minor distribution shift. ImageNet-S transforms ImageNet images into different art styles, while preserving the global contents of original images. Though these images all correspond to the same category, they are visually dissimilar and can easily confuse ImageNet-trained classifiers.
We consider ImageNet and its nut-of-distributioo variants for test-time adaptation. 1. ImageNet 9 large-scale generic object classification dataset, featuring 1000 object categories. We test on the validation set that consists of 50 images for each category. 2. ImageNet-C [19] applies various visual corruptions to the same validation set as ImageNet. We consider the shift due to gaussian noise (level-5) in our single-sample experiments. We consider 5 types of shift (gaussian noise, fog, pixelate, contrast, and snow) in our online adaptation experiments. 3. ImageNet-A (2I] consists of real-world image examples that misclassified by existing classifiers. The dataset convers 200 categories in ImageNet and 7450 images for testing. 4. ImageNet-R (20) is a synthetic dataset that renders 200 ImageNet classes in art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game styles. The dataset consists of 30K test images. 5. ImageNet collects images from the same distribution as ImageNet, composed of 1000 categories, each with 10 test images. 6. ImageNet-S transforms ImageNet images into different artistic styles, while preserving the global contents of original images. For ImageNet and its C, R, V2, and S variants except A, we sample 3 images per category, as these datasets contains a lot more classes than the other datasets. For ImageNet-A evaluate on it's whole test-set.
17


Reviews:
[{"review_id": "5xunRtyQIq", "reviewer": "Reviewer_UVNj", "summary": "This work proposed Diffusion-TTA, a test-time adaptation method that, given a test image, updates the classifier’s weights to maximize the image likelihood (i.e., the denoising score matching loss) using the pre-trained diffusion models. Specifically, the classifier uses the clean input image to predict a distribution over labels, which is then used for a weighted sum of learnt class embeddings into a prompt input of diffusion models. In experiments, Diffusion-TTA is tested for adapting both open-vocabulary CLIP models and pretrained image classifiers, where it improves the classification accuracy across different network architectures.", "questions": "1. Can the authors provide results of Synthetic SD Data and SD Features with other image classifiers, such as CLIP-ViT models, to make the comparison more sound? Similarly, can the authors provide results of COTTA, TENT and TTT-MAE across different network architectures (ResNet-18, ViT-B/32 and ConvNext-Tiny)?\n2. I suggest the authors add a comparison with TPT in both related work and experiments (adapting CLIP models). How does Diffusion-TTA compare with TPT regarding OOD accuracy and inference efficiency? \n3. Can the authors give insights of why the accuracy curve behaves like this (a spike at K=5 and does not change after K>=10) and justify the selection of K (K=5 vs K=1)?", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using pre-trained diffusion models as image priors to update classifiers at test time is novel. In particular, I like the idea of “using classifier’s output logits for a weighted sum of learnt class embeddings”, which is very intuitive to me. \n2. It shows the improvement of classification accuracy across different network architectures after using Diffusion-TTA.\n3. The ablation studies (in Table 3) are well conducted to show the importance of each component.", "weaknesses": "1. One of my major concerns is about the comparison with baselines in experiments. In Section 4.1 (“adapting open-vocabulary CLIP models”), only Diffusion-TTA works with CLIP, other baselines do not use CLIP. For example, Diffusion Classifier does not use any image classifier, and both Synthetic SD Data and SD Features use the ResNet-50 classifier. I think this setting causes an unfair comparison with baselines. In Section 4.2 (“adapting image classifiers”), I also don’t see what the pretrained image classifiers the baselines (COTTA, TENT and TTT-MAE) have used. It would be good to report their accuracies on the same set of network architectures (ResNet-18, ViT-B/32 and ConvNext-Tiny). In the current version, I’m not sure if the proposed method is better than the baselines under the same conditions. \n2. The claim “this is the first work that adapts pre-trained large-scale discriminative models to individual images” is wrong. From what I know, at least TPT [1] is a prior test-time adaptation method using only a single test image. I think TPT is more efficient than the proposed method in adapting CLIP models, since it only updates the input prompts instead of the CLIP weights, and it does not need to backpropagate through large diffusion models with a large batch size. \n3. In Figure 6, it looks weird to me that the accuracy curve has a spike at K=5 and does not change after K>=10. Also, a smaller K (maybe K=1?) achieves very similar accuracy with K=5, it seems that we may not need a distribution of output to do a weighted sum. Instead, we can just use the class embedding from the top-1 prediction as the prompt input.\n4. Some suggestions regarding the writing presentations: 1) In line 155 and line 156, the variable format is not consistent, such as $z_i$ and $i \\in T$. 2) Since the baselines are already discussed in the related work section, there is no need to give a full detailed description of them in experiment section again (line 234-250). They can be moved to Appendix if necessary. 3) The fonts in Figure 5 is too small.\n[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q2.1: Synthetic SD Data and SD features baseline use ResNet-50 classifier and not CLIP.**\n\nYou are correct, Synthetic SD Data, SD features, and Diffusion Classifier are not very relevant or fair baselines for Table 1 of the paper as they don’t use a pre-trained classifier. However, we thought showing their results help readers understand the TTA with discriminative models is critical for recognition tasks.  To avoid confusion, we will remove these baselines from Table 1 and list them in a separate Table in the supplementary.\n\nWe followed Diffusion Classifier’s codebase to run Synthetic SD-Data baseline using CLIP.  Notably, SD-Feature baseline is not applicable to CLIP classifier: CLIP takes RGB pixels (HxWx3)  as input, whereas SD-Feature takes high-dimensional latent features (HxWx64) from U-Net as input.\n\n |  | Food101 | FGVC | Pets |\n| :---      |   :----:   |  :----:   |  :----:   |\n| SD-Data | 12.6 | 9.4 | 31.3 |\n| **CLIP-ViT/B-32** | 78.4 | 18.8 | 77.8 |\n| $~~~~$+SD-Data | 76.5 (-1.9) | 19.4 (+0.6) | 71.2 (-6.6) |\n| $~~~~$+Diffusion-TTA | **80.2 (+1.8)** | **22.2 (+3.4)** | **81.1 (+3.3)** |\n\nWe find that SD-Data baseline fails to improve classification on Food101 and Pets dataset (-1.9 and -6.6), while the improvement on FGVC is marginal (+0.6).  Our conjecture is that images generated by Stable Diffusion have a huge distribution shift compared to real images in the test datasets.  Fine-tuning CLIP on synthetic images fails to generalize to real images.\n\n**Q2.2: Mention the backbones used for COTTA, TENT and TTT-MAE baselines.**\n\nPlease see Q1.1 and Q1.2 in the global comment.\n\n**Q2.3: Weird Behavior of K  in TopK**\n\nYes, we find that the peak at K=5 was indeed an artifact of the specific dataset and architecture that we were ablating.  Since submission, we have conducted a very detailed ablation of K across multiple datasets and architectures, we empirically find that not choosing topK, gives the best results on average. Currently all our reported results use K=5, therefore we expect our numbers to increase by a few decimals after incorporating this change. We subsample 1 image per category for the ablation study.\n\n| ConvNext-Tiny | ImageNet | ImageNet-R |\n| :---      |   :----:   |  :----:   |\n| K=2 | 82.6  | 47.5 |\n| K=5 | 82.9  | 45.5 |\n| K=50 | 82.6  | 49 |\n| No K | **83.0**  | **51** |\n\n| ResNet18 | ImageNet | ImageNet-R |\n| :---      |   :----:   |  :----:   |\n| K=2 | 73.5  | 37 |\n| K=5 | 77.3  | 41 |\n| K=50 | **78.2** | **43** |\n| No K | 78.2  | 42.5 |\n\n| CLIP-ViT/L-14 | FGVC | Pets |\n| :---      |   :----:   |  :----:   |\n| K=1 | 21  | 79.3 |\n| K=5 | 21  | 79.3 |\n| K=50 |  21  | 79.3 |\n| No K | 21  | 79.3 |\n\n**Q2.4: “this is the first work that adapts pre-trained large-scale discriminative models to individual images” is wrong, TPT also adapts large scale pre-trained classifiers. Add TTA baselines on the Open-Vocabulary CLIP experiment.**\n\nThank you for pointing out the TPT method. We were not aware of the method. We will remove the sentence and add TPT to the related work. \nWe present comparisons with TPT on zero-shot classification in the following table.  For TPT baseline, we tune the prompt and evaluate classification on each dataset.\n\n| | Food101 | CIFAR100 | FGVC | Pets | Flower102 | ImageNet | Average Improvement |\n| :---      |   :----:   |  :----:   |  :----:   |  :----:   |  :----:   |   :----:   |     :----:   |   \n| **CLIP-ViT-B/32** | 78.4 | 60.0 | 18.8 | 77.8 | 64.1 | 56.3 ||\n| $~~~~$+TPT | 79.8 (+1.4) | **65.2 (+5.2)** | 18.4 (-0.4) | 77.3 (-0.5) | 62.8 (-1.4) | 57.5 (+1.2) | +0.91 |\n| $~~~~$+Diffusion-TTA | **80.2 (+1.8)** | 61.8 (+1.8) | **22.2 (+3.4)** | **81.1 (+3.3)** | **64.3 (+0.2)** | **58.1 (+1.8)** | **+2.01** |\n| **CLIP-ViT-B/16** | 84.7 | 68.8 | 21.4 | 80.5 | 67.6 | 60.6 | |\n| $~~~~$+TPT | 85.2 (+0.5) | **68.4 (-0.4)** | 20.4 (-1.0) | 80.0 (-0.5) | **70.0 (+2.4**) | **61.6 (+1.0)** | +0.31 |\n| $~~~~$+Diffusion-TTA | **85.5 (+0.8)** | 67.6 (-0.8) | **22.6 (+1.2)** | **80.8 (+0.3)** | 69.2 (+1.6) | 61.5 (+0.9) | **+0.67** |\n| **CLIP-ViT-L/14** | 91.2 | 79.6 | 29.0 | 89.2 | 75.2 | 68.9 | |\n| $~~~~$+TPT | 90.6 (-0.6) | 80.6 (+1.0) | 30.2 (+1.2) | 87.6 (-1.6) | **76.9 (+1.6)** | **70.6 (+1.7)** | +0.55 |\n| $~~~~$+Diffusion-TTA | **91.2 (0.0)** | 80.6 (+1.0) | **30.6 (+1.6)** | **89.8 (+0.6)** | 76.1 (+0.9) | 69.9 (+1.0) | **+0.85** |\n\n\nWe conclude that:\n\n1. Our Diffusion-TTA consistently outperforms TPT  across all CLIP backbone architectures.\n2. TPT only adapts prompts and is thus restricted to vision-language classifiers like CLIP.  Our method instead can improve any pre-trained classifier, and even dense pixel labeller (e.g., semantic segmentation)\n3. Our method works for both single-example and online test-time adaptation, whereas TPT only works on single-example settings.\n\n\n**Q2.5: Writing Suggestions**\n\nThank you for your suggestions.  We have revised our paper based on your suggestions.\n"}, {"review_id": "JihW2fjH4h", "reviewer": "Reviewer_6xj6", "summary": "Authors aim to improve test-time adaptation of CLIP and ImageNet trained models on different distribution data. They leverage text conditioned image diffusion model to adapt image classifier’s parameters by maximizing image diffusion likelihood. In particular, image classifier’s output probabilities are used as weights for class conditioning the diffusion model and diffusion likelihood gradients are backpropagated to classifier through the probabilities. Adaptation of CLIP across different datasets and adaptation of ImageNet trained models across different ImageNet variants using the diffusion loss show improved top-1 accuracy.", "questions": "1.\tWhich diffusion model was used in Sec. 4.1?\n\n2.\tCLIP adaptation is shown across multiple datasets. Adapting CLIP with ImageNet variants would be interesting.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tPaper is well-written and easy to understand.\n\n2.\tProposed setup is a simple plug and play method that does not require any additional training or warm-up of the models.\n\n3.\tResults demonstrate that the proposed adaptation improve the results, particularly benefit smaller models.\n\n4.\tAppreciate the analysis that shown diffusion loss and cross-entropy loss are correlated, which support the gains observed in the results.", "weaknesses": "1.\tProposed setup can be seen as straight forward extension of Li et al. [22] which introduced similar guidance strategy to optimize over text prompts. Here, classifier parameters are updated instead of text prompts.\n\n2.\tIncreased time complexity of test time adaptation with the diffusion model, where each sample takes 55 seconds long for adaptation.\n\n3.\tIncreased model complexity by including diffusion model with large number of parameters in the adaptation process.\n\n4.\tThe adaptation is not as effective on larger models when compared to smaller models.\n\n5.\tPrior works comparisons are included in Table 2 but the network architectures are different across these comparisons and wouldn't be a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q3.1: Straight forward extension of Diffusion Classifier**\n\nWe believe this is not the case for the following reasons:  Diffusion Classifier searches over **discrete** class labels while Diffusion-TTA optimizes the parameter of the diffusion model and the classifier thru gradient descents.  Gradient-based optimization enables us to apply our method to online adaptation settings and extend it beyond classification tasks, such as in the task of semantic segmentation.  In contrast, discrete label search does not apply to online adaptation and would be infeasible for dense labeling tasks (searching over pixel labellings is exponential wrt the number of pixels in the image). \n\n**Q3.2: The proposed method is slow.**\n\nIn a single GPU, our method takes 55 seconds for adaptation mainly due to sequential gradient accumulation steps.  We can reduce the computation latency to 1.2 seconds per example using multiple gpus.  Also, we can further speed up our method by:\n\n1. Dynamically deciding when to adapt the model based on the entropy in the diffusion loss.\n2. Incorporating better timestep weighting mechanisms [1] and recent advances with Consistency Models [2] which allow one-step image generation can provide significant boosts in the speed.\n\n[1] Consistency Models. Song et al. ICML 2023.\n\n[2] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al. arXiv:2303.15233.\n\n\n**Q3.3: Adaptation improvements smaller on larger models**\n\nThat’s indeed true that larger models provide smaller improvements in single example settings. However we find that in online settings we get significant improvements even with larger models.  In the Table below, we show results adapting two of the largest ImageNet Classifiers ViT L/16 (307M parameters) and ConvNext-Large (197M parameters).  We test on the Gaussian-Noise split in ImageNet-C dataset.\n\n| | ConvNext-Large | ViT-L/16 |\n| :---      |   :----:   |  :----:   |\n| No TTA | 37.0 | 34.9|\n| Diffusion-TTA (single-example) | 37.0 | 33.3 |\n| Diffusion-TTA (online) | **54.9 (+17.9)** | **50.0 (+15.1)**|\n\nAs can be seen in the table above,  even though our model doesn’t improve in single-example setting, we get significant improvements in the online setting.\n\n**Q3.4: Which diffusion model was used in Sec. 4.1?**\n\nWe used Stable Diffusion v1.4 for this section.\n\n**Q3.5: Baselines in Table 2 have different neural architectures**\n\nPlease see Q1.1 and Q1.2 in the global comment.\n\n**Q3.6: Show CLIP adaptation on ImageNet dataset variants**\n\nGreat suggestion! Our Diffusion-TTA improves CLIP-ViT/B-32 on ImageNet variants (ImageNet, ImageNetV2, and ImageNet-R) consistently.\n\n|  | ImageNet | ImageNet-V2 | ImageNet-R |\n| :---      |   :----:   |  :----:   |  :----:   |\n| **CLIP-ViT/B-32** | 56.3  | 51.3 | 55.5 |\n| $~~~~$+Diffusion-TTA (single-example) | **58.1 (+1.8)** | **52.5 (+1.2)** | **58.2 (+2.7)** |\n"}, {"review_id": "kQ9EX4kLlB", "reviewer": "Reviewer_qddk", "summary": "This paper proposes a test-time adaptation (TTA) method that adjusts pretrained image classifiers at test time by leveraging the power of pretrained, text-conditioned generative diffusion models. Given an unseen image, the proposed method Diffusion-TTA minimizes the diffusion loss with respect to the weights of the pretrained classifier and/or diffusion model via gradient descent at test time. In other words, Diffusion-TTA provides guidance to the classifier to adapt its performance particularly under the out-of-distribution setting.\n\nFrom the empirical evaluation, Diffusion-TTA shows its effectiveness in encouraging consistent improvements over the initially employed classifier. Furthermore, the method is applicable to utilize different neural net architectures for the classifiers such as ResNet, ViT, and ConvNext-Tiny.\n\n----\nAfter reading the rebuttal, I've raised my score from 6: weak accept to 7: accept.", "questions": "How’s the performance of TTA-MAE or other existing TTA methods on the Open-Vocabulary CLIP experiment?\n\nHave the authors tried partial weight updating on the classifier, e.g., only adjusting weights of a few top layers, and observing the performance gain?\n\nIn L175, it’s mentioned that adapting the weights of both generator and classifier slightly boosts the performance of Diffusion-TTA for the ImageNet distribution shift case. By how much boost in comparison to keeping the weights of the generator fixed?", "limitations": "The limitation in terms of the generality of the method, i.e., only shown to be effective for image classification thus far, has been addressed. Furthermore, although the inference speed is perhaps not a focus in this study, I’d recommend also addressing it (~55 seconds per example on a single NVIDIA-A100 40GB) as another limitation.\n\nI’m hoping that the code implementation is made available soon.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Sufficient novelty:\nDiffusion-TTA follows a similar approach from Diffusion Classifier (Li et al. 2023) by utilizing the same objective, i.e., minimizing the diffusion loss, but executing a different optimization approach towards the objective. Diffusion-TTA employs pretrained classifiers and updates the weights via back-propagation of the diffusion loss, while Diffusion Classifier performs a discrete optimization over categorical text prompts to the generative model without using any pretrained classifier. This approach undertaken by Diffusion-TTA, which results in a positive empirical outcome in the context of TTA, is a new perspective to me.\n\nRigorous ablation study: \nThe impact of each implementation component defining Diffusion-TTA is empirically investigated and reported so that the readers are aware about the contribution from each component.\n", "weaknesses": "Empirical evaluation: \nI think that the added values of Diffusion-TTA over the existing SOTA methods remain a bit inconclusive. Specifically, it’s unclear to me that Diffusion-TTA is generally better than TTA-MAE given two occasions in ImageNet experiments (in-distribution and ImageNet-V2) where TTA-MAE seems to provide slightly or statistically insignificant negative outcomes. On the other hand, TTA-MAE produces larger improvements than Diffusion-TTA.\n\nI’d suggest having comparisons in which the other existing methods use the same backbone networks as Diffusion-TTA for the classifiers (ResNet18, ViT-B/32, or ConvNext-Tiny) whenever applicable.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q4.1:  Improvements over TTT-MAE unclear**\n\nIf we consider the **after-TTA absolute scores** our method significantly outperforms TTT-MAE on all except ImageNet-V2 dataset. On ImageNet, ImageNet-A, ImageNet-R, ImageNet-C, and ObjectNet, our method gets (+1.1, +4.5,  +10.5,  +13.9, +4.8)  boosts respectively over TTT-MAE. \n\nWe don’t think relative improvement is the right number to track, rather one should track **the absolute score after TTA**. The reason for this is the following: One can easily abuse the relative TTA improvement metric by getting very low before-TTA results, and then showing high gains in performance on top of it, while still having very low absolute after-TTA numbers! On a separate note, adding a different classifier in TTT-MAE  requires re-training the whole network on ImageNet-1K, while for us we can easily plug-in and improve with any pre-trained classifier.\n\nFor online TTA, Diffusion-TTA significantly outperforms TTT-MAE across different types of distribution shifts using a smaller classifier: ConvNext (28.5M parameters) vs. customized ViT-L+ViT-B (392M parameters).  Our method achieves 47.4, 65.9 69, 62.4, and 46.2 on Gaussian-Noise, Fog, Pixelate, Snow, and Contrast split in ImageNet-C, compared to 37.9, 51.1, 65.7, 56.5, and 10 obtained by TTT-MAE.  See the online adaptation results in the global comment of Q1.2 for more details.\n\n**Q4.2: Mention the backbones used for COTTA, TENT and TTT-MAE baselines.**\n\nPlease see Q1.1 and Q1.2 in the global comment.\n\n**Q4.3: Add TTA baselines on the Open-Vocabulary CLIP experiment.**\n\nWe compare our method against TPT [1] on the task of open-vocabulary classification in the following table. TPT is a test-time adaptation method for large-scale text-to-image models such as CLIP.  TPT optimizes the text prompts to encourage consistent predictions across augmented views of the same test image by minimizing the marginal entropy. Our method on the other hand optimizes the network parameters using generative feedback.\n\n[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022.\n\n\n| | Food101 | CIFAR100 | FGVC | Pets | Flower102 | ImageNet | Average Improvement |\n| :---      |   :----:   |  :----:   |  :----:   |  :----:   |  :----:   |   :----:   |     :----:   |   \n| **CLIP-ViT-B/32** | 78.4 | 60.0 | 18.8 | 77.8 | 64.1 | 56.3 ||\n| $~~~~$+TPT | 79.8 (+1.4) | **65.2 (+5.2)** | 18.4 (-0.4) | 77.3 (-0.5) | 62.8 (-1.4) | 57.5 (+1.2) | +0.91 |\n| $~~~~$+Diffusion-TTA | **80.2 (+1.8)** | 61.8 (+1.8) | **22.2 (+3.4)** | **81.1 (+3.3)** | **64.3 (+0.2)** | **58.1 (+1.8)** | **+2.01** |\n| **CLIP-ViT-B/16** | 84.7 | 68.8 | 21.4 | 80.5 | 67.6 | 60.6 | |\n| $~~~~$+TPT | 85.2 (+0.5) | **68.4 (-0.4)** | 20.4 (-1.0) | 80.0 (-0.5) | **70.0 (+2.4**) | **61.6 (+1.0)** | +0.31 |\n| $~~~~$+Diffusion-TTA | **85.5 (+0.8)** | 67.6 (-0.8) | **22.6 (+1.2)** | **80.8 (+0.3)** | 69.2 (+1.6) | 61.5 (+0.9) | **+0.67** |\n| **CLIP-ViT-L/14** | 91.2 | 79.6 | 29.0 | 89.2 | 75.2 | 68.9 | |\n| $~~~~$+TPT | 90.6 (-0.6) | 80.6 (+1.0) | 30.2 (+1.2) | 87.6 (-1.6) | **76.9 (+1.6)** | **70.6 (+1.7)** | +0.55 |\n| $~~~~$+Diffusion-TTA | **91.2 (0.0)** | 80.6 (+1.0) | **30.6 (+1.6)** | **89.8 (+0.6)** | 76.1 (+0.9) | 69.9 (+1.0) | **+0.85** |\n\n\nWe conclude that:\n\n1. Our Diffusion-TTA consistently outperforms TPT  across all CLIP backbone architectures.\n2. TPT only adapts prompts and is thus restricted to vision-language classifiers like CLIP.  Our method instead can improve any pre-trained classifier, and even dense pixel labeller (e.g., semantic segmentation)\n3. Our method works for both single-example and online test-time adaptation, whereas TPT only works on single-example settings.\n\n**Q4.4: Adjusting certain parts of the weights of the classifier?**\n\nWe present the ablation study of adapting different layers of the classifier:\n\n1. Batch Normalization layers only\n2. The last FC layer only\n3. The whole classifier\n\nWe employ ConvNext-Tiny and subsample 1 image per category for the ablation study. We find that adapting the whole classifier results in the best performance. We will include these ablations in our paper.\n\n| Adapt Classifier: | BN | Last FC layer | Whole model|\n| :---      |   :----:   |  :----:   |  :----:   |\n| ImageNet | 69  |  68.6 | **78.2** |\n| ImageNet-R | 37 |  37.5 | **42.5** |\n\n**Q4.5: Adding an ablation for freezing or not the weights of the image diffusion model**\n\nWe present the ablation study of freezing the diffusion model or not.  We employ ConvNext-Tiny and subsample 1 image per category for the ablation study\n\n| Adapt  | Diffusion + Classifier | Classifier only |\n| :---      |   :----:   |  :----:   |\n| ImageNet | **78.2** | 72.6 |\n| ImageNet-R | **42.5** | 37 |\n\nWe show that adapting the diffusion model results in significant performance gain. See the same observation in Table 3 in our paper. We will include these ablations in our paper.\n\n**Q4.6: Method is limited to classification.**\n\nPlease see Q1.3 in the global comment.\n\n**Q4.7: The proposed method is slow**\n\nIn a single GPU, our method takes 55 seconds for adaptation mainly due to sequential gradient accumulation steps.  We can reduce the computation latency to 1.2 seconds per example using multiple gpus.  Also, we can further speed up our method by:\n\n1. Dynamically deciding when to adapt the model based on the entropy in the diffusion loss.\n2. Incorporating better timestep weighting mechanisms [1] and recent advances with Consistency Models [2] which allow one-step image generation can provide significant boosts in the speed.\n\n[1] Consistency Models. Song et al. ICML 2023.\n\n[2] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al. arXiv:2303.15233.\n\n**Q4.8: Code Availability?**\n\nYes, we will open source our code upon paper acceptance. \n"}, {"review_id": "tTEc49V4HE", "reviewer": "Reviewer_VyeN", "summary": "The paper discusses a method that uses generative models as test-time adapters for discriminative models. The authors propose a technique to adapt pre-trained classifiers and CLIP models to individual unlabeled images. They achieve this by modifying the text conditioning of a text-conditional pretrained image diffusion model and maximizing the image likelihood through backpropagation. The proposed approach improves classification accuracy on various datasets, including ImageNet. The authors compare their method with previous test-time adaptation techniques and demonstrate its superior performance. They highlight that this is the first work to adapt large-scale pre-trained discriminative models to individual images without requiring joint discriminative and self-supervised training objectives.", "questions": "* How are the learned class or text embeddings obtained, specifically referring to line 156 or Fig.1? Are the embeddings the same as those employed by the classifier or CLIP models? If so, does this imply that the class/text embeddings change during test time adaptation?\n\n* What is the time required to adapt the model to a single image? Can this approach be scaled up for practical adaptation applications?\n\n* In Figure 6, have you investigated the effects of increasing the value of K for a CLIP model?", "limitations": "The obtained results are interesting; however, it is worth noting that the observed improvements, particularly in relation to the CLIP models, appear relatively small. It is interesting that the base classifiers, in particular the CLIP models, already exhibit good performance even without any fine-tuning or test-time adaptation.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The paper addresses a fundamental challenge in deep learning models, which is test-time out-of-distribution generalization. This problem is known to be difficult to solve. \n\n* The authors propose an interesting approach that combines generative and discriminative models to tackle this issue. Particularly intriguing is their idea of using a diffusion model as a guide for the classifier, by modulating the text conditioning of the model. \n\n* The experimental evaluation conducted in the paper is good, demonstrating the effectiveness of the proposed method. The authors thoroughly evaluate their approach on various datasets, including conducting ablations to analyze the discriminative and generative components. \n\nOverall, the paper provides a valuable contribution to the field with its innovative approach and robust experimental evaluation.", "weaknesses": "* While I agree with your premises regarding the class conditional model, I noticed that the improvements achieved are marginal on certain datasets, in particular for open-set problems. It would be beneficial to have a more in-depth analysis specifically focusing on ImageNet for open-set evaluation.\n\n* Additionally, it would be valuable to incorporate confidence intervals in the evaluation to provide a better understanding of the statistical significance of the results. Furthermore, I believe it would be advantageous to include additional metrics such as top-k accuracy and f-score (where possible), rather than solely relying on accuracy. These metrics would provide a more comprehensive evaluation of the model's performance in various scenarios. \n\n* It is worth mentioning that CLIP, even without any adaptation, seems to perform well, and the extent of improvement achieved by the proposed method is not decisively evident in most cases, especially without the inclusion of confidence intervals.\n\n* Finally, the clarity of the introduction can be improved. I think a better discussion of discriminative vs generative models would be a valuable addition to this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q5.1: Marginal Improvements in Open-Set Datasets**\n\nYou are correct that the improvements on open-set problems are not as high on the closed-set problems. We think this is mainly because we use Stable Diffusion as our diffusion model.  Stable Diffusion conditions the image generation on the text embeddings from CLIP . However, the language model in CLIP is much weaker than open ended large-language models like T5-XXL which is used in Imagen. Clark etal [1] validates this claim, where they find Imagen results in a significantly better classifier than Stable Diffusion. We don’t have this issue for DiT as it learns its own text embedding for each class.\n\nHere, we compare our method against a strong baseline–TPT [2]. TPT is a test-time adaptation method for large-scale text-to-image models such as CLIP. TPT optimizes the text prompts to encourage consistent predictions across augmented views of the same test image by minimizing the marginal entropy. Our method on the other hand optimizes the network parameters using generative feedback.\n\nPlease see our response to Q2.4 for the comparison among our method against a strong baseline–TPT.  We find that Diff-TTA consistently outperforms TPT across all CLIP backbone architectures.\n\n[1] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al. arXiv:2303.15233.\n\n[2] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022.\n\n**Q5.2: ImageNet Variant on Open vocab setting**\n\nGreat suggestion! Our Diffusion-TTA improves CLIP-ViT/B-32 on ImageNet variants (ImageNet, ImageNetV2, and ImageNet-R) consistently.\n\n|  | ImageNet | ImageNet-V2 | ImageNet-R |\n| :---      |   :----:   |  :----:   |  :----:   |\n| **CLIP-ViT/B-32** | 56.3  | 51.3 | 55.5 |\n| $~~~~$+Diffusion-TTA (single-example) | **58.1 (+1.8)** | **52.5 (+1.2)** | **58.2 (+2.7)** |\n\n**Q5.3: Confidence intervals/ Top-k to get better understanding of results**\n\nWe report confidence intervals of top-1 and top-3 accuracy over 5 random seeds on the ImageNet-R dataset.  We conduct experiments using the ResNet50 classifier and DiT diffusion model. Our performance gains are consistent across different random seeds.  We will include more detailed analysis in our final submission.\n\n|   | Before-TTA | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 | Mean |\n| :---      |   :----:   |  :----:   |   :----:   |  :----:   |   :----:   |  :----:   |  :----:   |\n| Top-1 Acc.  | 40.0 | 42.5 | 42.2 | 42.7 | 41.8 | 42.8 |  42.4 ($\\pm$ 0.41) |\n| Top-3 Acc.  | 51.7 | 52.0 | 51.8 | 52.2 | 51.8 | 51.8 |  51.9 ($\\pm$ 0.13) |\n\n\n**Q5.4: Better discussion of the  differences between generative/discriminative models in the introduction**\n\nThank you for your suggestion. We will include the following sentences in our intro.\nSo far discriminative models have mainly been used for discriminative tasks such as image classification. Recently,  Diffusion Classifier[1] and Clark etal l2], explored the use of recent generative models (diffusion models) for the task of classification. They find that while discriminative models are better at learning unary concepts, generative models offer distinct benefits. Notably, Clark etal in Table 2 of their paper find that generative models such as Imagen exhibit reduced texture bias than state-of-the-art discriminative models such as CLIP-L14 or ViT-22B.  Further Diffusion Classifier[1] in their Table 2 find that generative models such as Stable Diffusion outperform state-of-the-art discriminative models such as CLIP-L14 and OpenCLIP-H14  in modeling object relations on Winoground daetaset. Motivated by the desire to leverage the strengths of discriminative models and generative models, we propose to combine them using test-time adaptation.\n\nThe exact form of their combination is  motivated by neuroscience findings that suggest that the brain performs a form of analysis by synthesis by rendering  inferred concepts in an iterative feedback loops of encoding and decoding processes [3]. Our model is a form of analysis by synthesis using conditional diffusion models as the renderer (decoder) and classifiers or segmentors as the encoder. \n\n[1] Your diffusion model is secretly a zero-shot classifier. Li et al. arXiv 2023.\n\n[2] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al. arXiv:2303.15233.\n\n[3] Generative Feedback Explains Distinct Brain Activity Codes for Seen and Mental Images. Breedlove et al.  Current Biology 2020.\n\n**Q5.5: How are text embeddings obtained/updated**\n\nThe class or text embeddings are obtained from the pre-trained diffusion model. \n\n- For stable diffusion, the text-embeddings are obtained by encoding each class name in the dataset using  the CLIP text encoder.  \n- DiT, on the other hand learns a set of class embeddings for ImageNet classes, we simply use that in our experiments.  \n- In our experiments, as we update the diffusion model we do update the text embeddings during TTA.\n\n**Q5.6: Time to Adapt to a Single image?**\n\nIn a single GPU, our method takes 55 seconds for adaptation mainly due to sequential gradient accumulation steps.  We can reduce the computation latency to 1.2 seconds per example using multiple gpus.  Also, we can further speed up our method by:\n\n1. Dynamically deciding when to adapt the model based on the entropy in the diffusion loss.\n2. Incorporating better timestep weighting mechanisms [1] and recent advances with Consistency Models [2] which allow one-step image generation can provide significant boosts in the speed.\n\n[1] Consistency Models. Song et al. ICML 2023.\n\n[2] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al. arXiv:2303.15233.\n\n**Q5.7: Increasing the value of K for CLIP**\n\nPlease see our response to Q2.3\n"}, {"review_id": "SBkEYGWOY2", "reviewer": "Reviewer_4gPG", "summary": "The authors propose combining a diffusion model with a discriminative model for test-time adaptation: They propose adapt a pretrained classifier using a diffusion model with its likelihood loss. They show results on several OOD robustness benchmarks, commonly used in TTA.", "questions": "Please respond / fix the points I outlined above.\n\nPlease fix Table 2 to make baselines comparable to your results. Please add a RN50 baseline since it is common to report TTA results on this architecture.\n\nI would advise the authors to do a round of proof-reading of the manuscript, as there are some grammar errors and typos, e.g. line 234, line 123: adaptation\n", "limitations": "The authors have not included a Broader Impact section and did not discuss the potential negative societal impact of their work. I think they could discuss how the chosen diffusion model with affect the adapted classifier. Practitioners should note that biases present in the diffusion model will likely also manifest in the adapted classifier. The classifier itself may or may not have biases of its own which in turn may or may not be corrected by the diffusion model.\n\nOne limitation of the method is that it is very compute-intense and thus will likely be of limited use to practitioners.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the idea of combining a generative model with a discriminative model to try to get the benefits of both. The proposed method is elegant, but unfortunately heavy on the compute side.\n\nThe paper is well-written and easy to follow. The approach is clear. \n\nIn general, I think the method is interesting and the idea looks novel to me. But I do think there are several major issues with the paper which need to be resolved. I am happy to engage in a discussion with the authors and will raise my rating if my points are addressed.", "weaknesses": "Weaknesses:\nMy biggest issue is with the experimental results. I find them impossible to judge (see below) and thus, do not understand whether this method works similarly or better than other methods. Given that the proposed method is very heavy on the compute side, I think it would need to make up for it in much stronger results. If e.g. this method performs similarly to TENT / COTTA, then people would still use TENT / COTTA because those methods are very cheap to use. Given that this is an empirical and not a theory paper, I also think more results / ablations should be presented.\n\n### Issues with the results section\nTable 2: 1) It is not clear which baseline models were used for TENT / COTTA / TTT here. The final numbers heavily depend on the chosen architecture and the results are only comparable if the same architecture was chosen. Please indicate the architecture for the baseline methods both in the text and in the Figure captions. 2) It is not clear whether accuracy or error are displayed in Table2. Please indicate this, and also please add \\uparrow and \\downarrow to indicate whether a higher or a lower number is better. An accuracy (if accuracy is displayed?) of 11% on ImageNet-C looks quite bad and I am wondering where this number comes from? The TENT authors report an accuracy of 44% for a ResNet50 in their paper. I think it would be necessary for the authors to add results for a ResNet50 such that the numbers are comparable.\n\nRight now, the results look strange. If we assume that the architecture is the same, then it seems TENT strongly degrades performance. E.g. on ImageNet-R, the TENT number is 24.3 for TENT, 34.6 for the baseline RN18 and 39.7 for Diffusion-TTA. Does this mean that there is an optimization issue because TENT should not degrade performance or is it a different architecture? In that case, the numbers are not comparable. Please rework Table 2 as following: Show the baseline performance without adaptation in the first row, then report numbers from the literature for this architecture, then report your results on the same architecture.\n\n\n### General comments\nLine 36: “In this paper, we take an alternative perspective. Instead of considering generative and discriminative models as competitive, we argue that they should be coupled in a way that leverages the best of both worlds: discriminative models are good at building powerful conditional density models but overfit to training distribution, and generative models generalize better but struggle to learn discriminative features.” -> minor: to the training distribution. The statement that “generative models generalize better” seems unsupported to me. As far as I see, [18] does not provide evidence for better OOD generalization for generative models. Further Carlini et al. [A] show that diffusion models actually memorize their training data and emit the memorized data at test time. Could the authors comment on how they think the findings presented in [A] square with the results in this paper? If diffusion models memorize their training data, I find it unintuitive that they would learn features that would allow them to generalize to OOD data. Put simply, if the task can be solved with memorization, there is no need to learn generalizable features.\n\n[A] Carlini et al. “Extracting Training Data from Diffusion Models”, https://arxiv.org/pdf/2301.13188.pdf\n\nIn my opinion, the paper lacks intuition and needs a better explanation why the method works. As written above, it is not evident to me that generative models should have better generalizable features compared to discriminative models. I am wondering whether the approach works because of an effect similar to model ensembling where we know that combining predictions of multiple models leads to better results than using a single model. I wonder if using a diffusion model has a similar effect, because it regularizes the weights to be compatible with both the discriminative and the generative model. Could the authors comment on this thought / better explain why their method works?\n\nDo the authors think one could train a discriminative network from scratch using their approach? If not, why do they think it (only?) helps in the test time adaptation stage? Is it maybe that one finds the “correct” loss basin with regular training and then finds a better loss value within that basin with test time adaptation? I am thinking of the paper by Wortsman et al. [B] where the authors showed that one can interpolate weights of different models because they are in the same low error basin. Could the authors maybe compare the training and test losses (regular cross-entropy) of the baseline and test-adapted models when interpolating the weights from the regular to the adapted model? I am wondering whether the loss goes down monotonically.\n\n [B] Wortsman et al. “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time”, https://arxiv.org/abs/2203.05482\n\n\nThe text in Figure 5 is too small below the bars and largely unreadable.\n\nFigure 6: It would be helpful to repeat this analysis on some other dataset. The solo peak at K=5 somewhat looks like an artifact and I wonder whether one would observe similar behavior in other datasets. How do the authors interpret this strange “peakiness” of the curve at K=5?\n\nMore discussion on the ablation analysis would be nice. Why is it that “using a single randomly sampled timestep and noise latent (+diffusion TTA), we find a significant reduction in the classification accuracy (−2.9%)”?\n\t\nI think the authors should perform a complexity analysis since their method involves making a forward and a backward pass through a large diffusion model to make use of its loss. This is a large overhead compared to much simpler TTA methods such as TENT. While there is a complexity analysis in the Appendix, Fig.2, the overhead is not compared to e.g. TENT which should be much faster.\n\nIn a similar vein, I would argue it will be hard to actually find an application for this method since the computational requirements are so massive.  The authors write that “We conduct our experiments on a single NVIDIA-A100 40GB VRAM GPU (…). Since our GPU fits only a batch size of 20, (…)”. TTA methods are usually designed to be light-weight such that they can be applied on the fly, e.g. if an autonomous car needs to adapt to changing weather conditions. This method does not fulfil the requirements usually considered for TTA, and thus, will be of rather limited use for practitioners.\n\nLine 173: “For all experiments, we adjust all the parameters of the classifier.” I am curious about this choice. Several papers show that at least for ResNets, it is better to do TTA with affine BN layers [C, D]. Does this method not work when adapting affine BN layers or does it work better if all layers are adapted?\n\n[C] Wang et al. “Tent: Fully Test-time Adaptation by Entropy Minimization”\n\n[D] Rusak et al. “If your data distribution shifts, use self-learning” \n\nThe appendix should be referenced in the main text.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q6.1: Backbone architecture unclear for baselines.**\n\nPlease see Q1.1 and Q1.2 in the global comment.\n\n**Q6.2: Add RN50 backbone to baselines.**\n\nWe have included ResNet50 in our online TTA results in Q1.2.  Here, we report TTA results using ResNet50 under single-example settings.\n\n||ImageNet|ImageNet-R|ImageNet-V2|\n|:---|:----:|:----:|:----:|\n|**ResNet50**|76.0|40.0|63.5|\n|$~~~~$+CoTTA|70.4 (-5.6)|39.2 (-0.8)|57.6 (-5.9)|\n|$~~~~$+TENT|70.4 (-5.6)|39.3 (-0.7)|57.7 (-5.8)|\n|$~~~~$+Dif-TTA (single-example)|**78.6 (+2.6)**|**42.5 (+2.5)**|**66.9 (+3.4)**|\n\n**Q6.3: TENT accuracy very low, Accuracy or Error not mentioned.**\n\nWe report the accuracy in all our Tables. In our online adaptation results (see Q1.2), we show that TENT and CoTTA only improve on certain distribution shifts (ImageNet-C shifts) and classifiers (ResNet). They both fail to improve classification on other ImageNet variants (see Q1.1).  Our conjecture is that TENT and CoTTA are online adaptation methods, and they are not robust when input examples share very little distribution shifts.\n\nA similar conclusion can be found in a recent paper  “On Pitfalls of Test-Time Adaptation”[1]. The authors show that methods that primarily work under online settings (TENT or CoTTA) are not robust to different types of distribution shifts (as reported in Table 4 in their paper).  In fact, these methods turn out to be very sensitive to model architectures and optimization hyper-parameters.\n\nOur method instead exploits a strong generative prior and thus is more robust. \n\n**Q6.4: Unsupported claim of better generalization of generative models. Comment on diffusion models memorize train data.**\n\nThanks for the suggestion, We will remove the sentence regarding better generalization of generative models  and revise the justification of our method as follows:\n\nDiffusion Classifier[1] and Clark et al [2] explored how to use generative models (diffusion models) for the task of classification. They find that while discriminative models are better at learning unary concepts, generative models offer distinct benefits. In particular, in Table 2 of [2], the authors show that generative models (Imagen) exhibit reduced texture bias than state-of-the-art discriminative models (CLIP-ViT/L-14 or ViT-22B).  In Table 2 of [1], generative models (Stable Diffusion) outperform state-of-the-art discriminative models (CLIP-ViT/L-14 and OpenCLIP-ViT/H-14)  in modeling object relations on Winoground dataset. Motivated by the desire to leverage the strengths of discriminative models and generative models, we propose to combine them using test-time adaptation.\n\n[1] Your diffusion model is secretly a zero-shot classifier. Li et al.\n\n[2] Text-to-image diffusion models are zero-shot classifiers. Clark, K et al.\n\n**Q6.5: Paper lacks intuition and needs a better explanation why it works?**\n\nWe conjecture that Diff-TTA works, as both discriminative and generative models capture distinctive aspects of the data, as discussed in previous question.\n\nOur method integrates both these models. Specifically \n- It optimizes the classifier to achieve high image likelihood under the loss landscape of a pre-trained diffusion model \n- It optimizes the diffusion model to have image likelihood given the output of a pre-trained classifier. \n\nWe think in satisfying both these constraints, it optimizes for a form of consensus between the two models.  \n\nTo better understand what’s happening, we compare Diff-TTA against the following baselines. \n\n- Logit-Adapt: In this method we do not use any pre-trained classifier, instead we initialize the logits using zeros and optimize them per example using diffusion objective.\n\n- Logit-Adapt-Ensemble: We average the probabilities of the above Logit-Adapt baseline with the probabilities predicted by the pre-trained classifier.\n\n- Classifier-Adapt: In this method we adapt the pre-trained classifier weights while freezing the diffusion model.\n\n- Diff-Adapt + Classifier-Adapt (Ours): In this method we adapt both the pre-trained classifier weights and the diffusion weights. This refers to our method Diff-TTA.\n\n||ResNet18|Logit-Adapt|Logit-Adapt-Ensemble|Classifier-Adapt|Diffusion-Adapt + Classifier-Adapt (Ours)|\n|:---|:----:|:----:|:----:|:----:|:----:|\n|ImageNet|68.4|71.8|72.6|73.0|**78.2**|\n|ImageNet-R|37.0|37.0|36.5|38.0|**42.6**|\n\n**Q6.6: Can a discriminative model be trained from scratch?**\n\nWe find that training a classifier scratch from does not work. Our conjecture is that good initialization of the classifier makes the loss landscape more convex and therefore easier to optimize.\n\n||Adapt classifier from scratch |Adapt  pre-trained classifier|\n|:---|:----:|:----:|\n|ImageNet|0.7 |**78.2**|\n|ImageNet-R|4|**42.5**|\n\n**Q6.7: Visualize loss curves vs interpolating weight constant, akin to Model Soup?**\n\nThanks for this suggestion.  Please refer to Figure 2 in the attached pdf, where we make this plot. We indeed find that, in instances where Diff-TTA correctly classifies the image, the loss monotonically reduces as we interpolate between the model weights. \n\n**Q6.7: K evaluation on more datasets**\n\nPlease see our response to Q2.3. \n\n**Q6.8: Why single sample timestep doesn't work**\n\nOur method performs TTA by minimizing the variational lower bound. Using a single timestep instead of an expectation over multiple timesteps results in a bad approximation of the lower bound, thus hurting performance. In Figure 3 of the attached pdf, we visualize this phenomenon using a test sample.\n\n**Q6.9: Heavy compute cost.**\n\nPlease see our response to Q3.2. \n\n**Q6.10: Complexity analysis compared to TENT**\n\nUsing a single or multiple gpu, Diff-TTA takes 55 or 1.2 seconds to adapt on one image, though TENT takes 0.03 sec per image. \n\n**Q6.11: Adapting just the BN Layers vs all layers**\n\nPlease see our response to Q4.4. \n\n**Q6.12: Discuss Negative Impact**\n\nThanks for the bringing this up. Due to lack of space we will include this in our final paper.\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
Thank you for your constructive feedback and for spending the time to read our paper in detail.

Below we address the common concern of all the reviewers: 

**Q1.1: Use same backbones when comparing against TENT and CoTTA baselines.**

We re-evaluate COTTA and TENT using the same backbone as ours. 

Note that, TTT-MAE builds atop customized classifiers.  Therefore we cannot evaluate TTT-MAE with the same classifiers without re-training it on the ImageNet dataset. 

We evaluate Diff-TTA / TTT-MAE under single-example settings, and CoTTA / TENT under online settings.  COTTA and TENT are not applicable for single-example settings: they explicitly require pseudo-labels (entropy minimization) to improve the classification accuracy.

||ImageNet|ImageNet-A|ImageNet-R|ImageNet-C|ImageNet-V2|
|:---|:----:|:----:|:----:|:----:|:----:|
|**Customized ViT-L/16 backbone**|82.1|14.4|33.0|17.5|72.5|
|$~~~~$+TTT-MAE (single-sample)|82.0 (-0.1)|21.3 (+6.9)|39.2 (+6.2)|27.5 (+10.0)|72.3 (-0.2)|
|**ResNet18**|69.5|1.4|34.6|2.6|57.1|
|$~~~~$+TENT (online)|63.0 (-6.5)|0.6 (-0.8)|34.7 (+0.1)|**12.1 (+9.5)**|52.0 (-5.1)|
|$~~~~$+CoTTA (online)|63.0 (-6.5)|0.7 (-0.7)|34.7 (+0.1)|11.7 (+9.1)|52.1 (-5.0)|
|$~~~~$+Diffusion-TTA (single-example)|**77.2 (+1.7)**|**6.1 (+4.7)**|**39.7 (+5.1)**|4.5 (+1.9)|**63.8 (+6.7)**|
|**ViT-B/32**|75.7|9.0|45.2|39.5|61.0|
|$~~~~$+TENT (online)|75.7 (0.0)|9.0 (0.0)|45.3 (+0.1)|38.9 (-0.6)|61.1 (+0.1)|
|$~~~~$+CoTTA (online)|75.8 (+0.1)|8.6 (-0.4)|45.0 (-0.2)|40.0 (+0.5)|60.9 (-0.1)|
|$~~~~$+Diffusion-TTA (single-example)|**77.6 (+1.9)**|**11.2 (+2.2)**|**46.5 (+1.3)**|**41.4 (+1.9)**|**64.4 (+3.4)**|
|**ConvNext-Tiny**|81.9|22.7|47.8|16.4|70.9|
|$~~~~$+TENT (online)|79.3 (-2.6)|10.6 (-12.1)|42.7 (-5.1)|2.7 (-13.7)|69.0 (-1.9)|
|$~~~~$+CoTTA (online)|80.5 (-1.4)|13.2 (-9.5)|47.2 (-0.6)|13.7 (-2.7)|68.9 (-2.0)|
|$~~~~$+Diffusion-TTA (single-sample)|**83.1 (+1.2)**|**25.8 (+3.1)**|**49.7 (+1.9)**|**21.0 (+4.6)**|**71.5 (+0.6)**|

We conclude:

- Diff-TTA outperforms TENT and CoTTA across various architecture backbones and distribution shifts. 

- Diff-TTA outperforms TTT-MAE even with a smaller classifier.  ConvNext (28.5M parameters) optimized by Diffusion-TTA (online) achieves better performance than a much bigger custom backbone of ViT-L + ViT-B (392M parameters) optimized by TTT-MAE (online)

---
Since submission we have extended the proposed method in two ways:

**Q1.2: Online Test-Time Adaptation:**

We have extended our method to an online TTA setting:  We adapt model weights to a set of streaming examples without resetting  the model weights at each input example. We compare our method to TTT-MAE, CoTTA, and TENT for online adaptation.

|ImageNet Corruption:|Gaussian-Noise|$~~~$Fog|Pixelate|Snow|Contrast|
|:---  |  :----:| :----:| :----:| :----:| :----:|  
|**Customized ViT-L/16 classifier**|17.1|38.7|47.1|35.6|6.9|   
|$~~~~$+TTT-MAE (single-sample)|27.9 (+10.8)|45.1 (+6.4)|61.4 (+14.3)|43.2 (+7.6)|9.3 (+2.4)|
|$~~~~$+TTT-MAE (online)|37.9 (+20.8)|51.1 (+12.4)|65.7 (+18.6)|56.5 (+20.9)|10 (+3.1)|
|**ResNet50**|6.3|25.2|26.5|16.7|3.6|
|$~~~~$+TENT (online)|12.3 (+6.0)|43.2 (+18.0)|41.8 (+15.3)|28.4 (+11.7)|**12 (+8.4)**|
|$~~~~$+CoTTA (online)|12.2 (+5.9)|42.4 (+17.2)|41.7 (+15.2)|28.6 (+11.9)|11.9 (+8.3)| 
|$~~~~$+Diffusion-TTA (single-sample)|12.7 (+6.4)|33.0 (+7.8)|30.4 (+3.9)|28.7 (+12.0)|7.0 (+3.4)| 
|$~~~~$+Diffusion-TTA (online)|**19 (+12.7)**|**43.2 (+18.0)**|**50.2 (+23.7)**|**33.6 (+16.9)**|2.7 (-0.9)| 
|**ViT-B/32**|39.5|35.9|55|30|31.5| 
|$~~~~$+TENT (online)|38.9 (-0.6)|35.8 (-0.1)|55.5 (+0.5)|30.7 (+0.7)|32.1 (+0.6)| 
|$~~~~$+CoTTA (online)|40.0 (+0.5)|34.6 (-1.3)|54.5 (-0.5)|29.7 (-0.3)|32 (+0.5)|
|$~~~~$+Diffusion-TTA (single-sample)|41.4 (+1.9)|47.3 (+11.4)|43.5 (-11.5)|36.1 (+6.1)|20.8 (-10.7)| 
|$~~~~$+Diffusion-TTA (online)|**46.5 (+7.0)**|**56.2 (+10.3)**|**64.7 (+9.7)**|**50.4 (+20.4)**|**33.6 (+2.1)**|
|**ConvNext-Tiny**|16.4|32.3|37.2|38.3|32| 
|$~~~~$+TENT (online)| 2.7 (-13.7)|5 (-27.3)|43.9 (+6.7)|15.2 (-23.1)|40.7 (-18.7)|
|$~~~~$+CoTTA (online)|13.7 (-2.7)|29.8 (-2.5)|37.3 (+0.1)|26.6 (-11.7)|32.6 (+0.6)|
|$~~~~$+Diffusion-TTA (single-sample)|21.0 (+4.6)|46.9 (+14.6)|41 (+3.8)|45.6 (+7.3)|25 (-7.0)| 
|$~~~~$+Diffusion-TTA (online)|**47.4 (+31.0)**|**65.9 (+33.6)**|**69 (+31.8)**|**62.6 (+24.3)**|**46.2 (+14.2)**| 

From the table we see that:
- Online adaptation of Diff-TTA significantly outperforms the single-example setting of Diff-TTA
- Diff-TTA outperforms TENT and CoTTA across various backbones. CoTTA and TENT fail to improve classification on certain corruptions and classifiers.  Our results are consistent with the analysis in [1]: the authors find that methods that primarily work under online settings (TENT or CoTTA) are not robust to different types of architectures and distribution shifts (as reported in Table 4 of [1]). 
- Our method outperforms TTT-MAE even with a smaller-size classifier.  ConvNext (28.5M params) optimized by Diffusion-TTA (online) achieves better performance than a much bigger custom backbone of ViT-L + ViT-B (392M params) optimized by TTT-MAE (online)

[1] On the Pitfalls of test-time adaptation. Zhao et al

**Q1.3: Diff-TTA for Semantic Segmentation:**

We have extended our method to semantic segmentation tasks by replacing the image classifier with a pre-trained segmentor.  See Figure 1 and 4 in the attached pdf for the architecture diagram and qualitative results.

Both the segmentor and the diffusion model are trained on ADE20K dataset. Under single-example settings, we find Diff-TTA consistently improves pre-trained SegFormer.


| ADE20K Corruption: | Clean | Gaussian-Noise | $~~$Fog | Frost | Snow | Contrast | Shot
| :---      | :----:   | :------:        | :----:   | :----:   |   :----:   |  :----: |  :----: |
| **SegFormer** | 66.1 | 65.3 | 63.0 | 58.0 | 55.2 | 65.3 | 63.3
| $~~~~$+Diffusion-TTA | 66.1 | **66.4 (+1.1)** | **65.1 (+2.1)** | **58.9 (+0.9)** | **56.6 (+1.4)** | **66.4 (+1.1)** | **63.7 (+0.4)**



Meta-review: This work harnesses generative diffusion models as a loss for test-time adaptation to update image classifiers and improve robustness to shift. The subject of this submission, test-time adaptation, is now a popular topic for robustness, and this submission contributes its own approach that is relevant to the current era of large-scale generative modeling: the proposed Diff-TTA method. All five expert reviewers with backgrounds on diffusion and adaptation agree on acceptance (4gPG : 7, qddk: 7, UVNj: 6, 6xj6: 6, VyeN: 6). The authors provide a rebuttal, and then reviewers maintained or raised their scores (UVNj, 6xj6, qddk), because of the experimental results provided. More specifically, the rebuttal included more comprehensive and controlled experiments (with matched backbones, more ablations, and more settings + tasks like open vocabulary and semantic segmentation). The experiments cover a breadth of benchmarks and baselines to show reliable improvement (at the cost of more computation time). The AC sides with acceptance: all reviewers agree, the rebuttal only improved the submission, and this work connects the active area of test-time adaptation with the rapidly-advancing topic of diffusion modeling.

The negatives emphasized by reviewers are novelty (6xj6: Li et al. [22], UVNj: TPT) and computation time (4gPG, 6xj6, qddk). For novelty, the AC acknowledges Li et al. [22], but considers the update in this work to be distinct, and finds it justified by experiments (including the ablation of the choice of parameters in the rebuttal) For test-time adaptation to single images, TPT and the original TTT by Sun et al. ICML'20 are indeed relevant. For computation time, the time needed is clearly mentioned in the implementation details, and the increased time follows from the use of large discriminative + generative models that are the focus of this work. The computation needed distances this work from the common emphasis for test-time adaptation on efficient updates, and may limit applicability, but this work nevertheless provides an informative "proof of principle" (4gPG, in post-rebuttal discussion). As a further plus, Diff-TTA improves in the online setting over the episodic setting (as shown in the rebuttal), as many TTA methods do (and should, one can argue), so perhaps future work could amortize some of the computation across samples.

The AC advises the authors to incorporate the reviewers' feedback, and in particular to include the experiments on the suggested backbones (both for the controlled comparisons and on ResNet-50 as a common model for benchmarking). To more comprehensively connect to research on test-time adaptation, the AC would also like to point out the use of diffusion models for adaptation of inputs (Diffusion for Adversarial Purification. Nie et al. ICML'22; Diffusion-Driven Adaptation. Gao et al. CVPR'23), although it is of course up to the authors to decide if these are sufficiently related to their work.
Decision: Accept (poster)