Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
Authors: Yanghao Wang
Long Chen
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Classifier
Few-shot Classification
Noise Optimization
Prompt Optimization
Primary area: deep_learning

Abstract: Although today's pretrained discriminative vision-language models (e.g., CLIP) have demonstrated strong perception abilities, such as zero-shot image classification, they also suffer from the bag-of-words problem and spurious bias. To mitigate these problems, some pioneering studies leverage powerful generative models (e.g., pretrained diffusion models) to realize generalizable image classification, dubbed Diffusion Classifier (DC). Specifically, by randomly sampling a Gaussian noise, DC utilizes the differences of denoising effects with different category conditions to classify categories. Unfortunately, an inherent and notorious weakness of existing DCs is noise instability: different random sampled noises lead to significant performance changes. To achieve stable classification performance, existing DCs always ensemble the results of hundreds of sampled noises, which significantly reduces the classification speed. To this end, we firstly explore the role of noise in DC, and conclude that: there are some ``good noises'' that can relieve the instability. Meanwhile, we argue that these good noises should meet two principles: 1) Frequency Matching: noise should destroy the specific frequency signals; 2) Spatial Matching: noise should destroy the specific spatial areas. Regarding both principles, we propose a novel Noise Optimization method to learn matching (i.e., good) noise for DCs: NoOp. For frequency matching, NoOp first optimizes a dataset-specific noise: Given a dataset and a timestep $t$, optimize one randomly initialized parameterized noise. For Spatial Matching, NoOp trains a Meta-Network that adopts an image as input and outputs image-specific noise offset. The sum of optimized noise and noise offset will be used in DC to replace random noise. Extensive ablations on various datasets demonstrated the effectiveness of NoOp. It is worth noting that our noise optimization is orthogonal to existing optimization methods (e.g., prompt tuning), our NoOP can even benefit from these methods to further boost performance.

Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
Yanghao Wang, Long Chen"
The Hong Kong University of Science and Technology ywangtg&connect. uut.ustt hk, longchen@ust. hkk
Abstract
Although today's 's pretrained discriminative vision-language models (e.g., CLIP) have demonstrated strong perception abilities, such zero-shot image classifica- tion, they also suffer from the bag-of- -orrddfworrs problem and spurious bias. To mitigate these problems, some pioneering studies leverage powerful generative models (e.g., pretrained diffusion models) to realize generalizable image classification, dubbed Diffusion Classifier (DC). Specifically, by randomly sampling a Gaussian noise, DC utilizes the differences of denoising effects with different category conditions classify categories. Unfortunately, an inherent and notorious weakness existing DCs is noise instability: different random sampled noises lead to significant perfor- mance changes. To achieve stable classification performance, existing DCs always ensemble the results of hundreds of sampled noises, which significantly reduces the classification speed. To this end, we firstly explore the role of noise in DC, and conclude that: there while, we argue that these good noises should meet two principles: 1) Frequency Matching: noise should destroy the specific frequency signals; Spatial Matching: noise should destroy the specific spatial areas. Regarding both principles, we propose a novel Noise Optimization method to learn matching (i.e., good) noise for DCs: NoOp. For frequency matching, NoOp first optimizes a dataset-speciftt noise: Given dataset and a timestep t, optimize one randomly initialized parame- terized noise. For Spatial Matching, NoOp trains a Meta-Network that adopts an image as input and outputs ipage-specific noise offset. The sum of optimized noise and noise offset will be used in DC to replace random noise. Extensive ablations on various datasets demonstrated the effectiveness of NoOp. It is worth noting that our noise optimization is orthogonal to existing optimization methods (e.g., prompt tuning), our NoOP can even benefit from these methods to further boost performance. Code is available at http:://tthbbboom////KUUStoonono
"good noises" that relieve the instability. Mean-
Introduction
Pretrained visual-language models (VLMs) learn the alignment of text and image from the text-image pairs. Thanks to the large-scale and wild training data, these discriminative VLMs gain some generalization capability and can even achieve zero-shot visual classification, e.g., CLIP [T. When
encountering some rare or customized categories, some works like prompt optimization 2 3 can also adapt CLIP with a few-shot training set efficiently. However, the CLIP models are criticized for an inherent bag-of-we bag-of-words problem 4 []. This problem leads to spurious bias, harming compositional inference and counterfactual reasoning. Thus, recent works 6 pretrained diffusion models for classification.
10 try leverage the
"Long Chen the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Diffusion Classifier
photo of (c:ed
Classification Basis
4 WV
Input Image Xg
argmin (xp, ct,t) E112
4t
Noisy Image
Diffusion Model
Predicted Noises Ê
Select the & that is closest to Prediction: the jeh category
Sampled Noise e
(b) Noise Instability
Frequency Matching
(d) Spatial Matching
Diflusion Classifier
Samoyed
Image
Optimized Noisc
STL-10
Low Frequency
XD
Beagle le
Image
Optimized
C2
DID
High Frequency
Noise
Figure 1: (a) Diffusion Classifier pipeline. (b) Noise Instability phenomenon: Different noises will lead different predictions. (c) Frequency Matching: Good noise should destroy the specific frequency signals. (d) Frequency Matching: Good noise should destroy the specific spatial areas.
The typical implementation framework is Diffusion Classifier (DC) 6 7]. As shown in Figure Ta), for a given clean image Tn and a specific timestep I,, DC randomly samples noise and follows the forward process of diffusion [1 to get the noisy image 24. Then input the aı into the diffusion denoising network 6ø conditioned on K different categories {c respectively and get K noise predictions {2o (It, Cs, } The category whose corresponding noise prediction has the smallest distance to the ground truth noise is the final predicted category, i.e.,
è = argmin (1ē0 f) cll2. i 1,2,..., K.
64
Due to the randomness of sampled noise in DC, we can sample different noises (e.g., 61, 62) for the same image Zg- Unfortunately, an inherent and notorious weakness of existing DC is: Noise Instability. As shown in Figure Ib), different sampled noises may lead to varied performance. To mitigate this instability, all prior works 66 [0] try to sample multiple noises and calculate the expectation of the distance between them and the predicted noises for classification, i.e.,
= argmin E. [[[ãa(I)) c,,t) - i= 1,2,,.... K.
Although this ensembling strategy somewhat relieve the instability issue, the computation overhead increases linearly with the number noise samples, making the inference very slow (e.g., one Pets image takes 18 seconds an RTX 3090 GPU). Thus, there tradeoff between the unstable single-sample sampling and expensive multiple-sample ensembling for the current DC framework.
In this paper, we first try to answer natural question: whether there are some good noises for diffusion classifier? By "Good noise", we mean that the DC's classification results are relatively stable to different randomly sampled noises. To the extreme case, if there is good noise, may only need to use this noise to avoid multiple samplings, and achieve good classification results. To answer this question, sampled noise destroys some parts the image, and DC tries to find the category that can best guide the diffusion model to reconstruct the destroyed parts. Thus, the "good noise" should destroy the parts that can best reflect the difference in reconstruction effect under different categories' guidance. Therefore, argue that good noise should meet the following two principles:
first explore and analyse the role of sampled noise in DC. Intuitively, the
1) Frequency Matching. According to the Frequency Bias Theory [12 13, given a dataset, the nategory-relatee signals mainly in specific frequency range. For example (c.f., Figure (c)).

---

contains categories "car" and "airplane". These categories can be distinguished by their overall shapes or structures, which mainly belong to low-frequency signals. Similarly, Describable Textures (DTD) [5] has categories like "banded" and "dotted". They are mainly distinguished by high-frequency signals (e.g., mutated texture). This intuitively leads to the first principle: a good noise should destroy the dataset-specific frequency signals that are related to the categories.
STL-10 14
2) Spatial Matching. Given one image, the category- reeette signals are mainly in specific spatial areas. For example, the signals in background areas are not related to the foreground category. As shown in Figure d)), for a specific image, a good noise to classify it should show some similar spatial patterns to this image (e.g., more noticeable damage to foreground and edges). For different images, the good noises should show different spatial layouts. This leads to the second principle: good noise should destroy the image-specific spatial areas that are related to the categories.
In this paper, we propose the first noise optimization method by learning matching noises for DC, dubbed Noise Optimization (NoOp). NoOp can effectively mitigate the noise instability issue by considering both frequency matching and spatial matching: 1) For frequency matching: Given specific dataset with few-shot training samples, we randomly sample one noise as initialization. Then, directly optimize this parameterized noise based on classífication loss. 2) For spatial matching: We design a Meta-Network that adopts the image (i.e., To) as input and outputs an image-specific noise offset. Similarly, we optimize this Nett-Neework based on classification loss. Finally, we replace the random noise of DC with the sum of the optimized noise and the noise offset.
We evaluated the effectiveness of method over eight few-shot classification datasets. Extensive ablation results showed the stability of NoOp. Besides, we conduct empirical experiments to support the two proposed principles. Furthermore, NoOp is orthogonal to existing optimization (e.g.. prompt- optimization based DC), thus our NoOp can even gain extra performance gains by incorporating existing techniques. Conclusively, our contributions are summarized as follows:
Although the noise instability of diffusion models has been widely discussed in the image generation and editing area, to the best of our knowledge, we are the first ones to study the noise instability in the discrimination task, i.e., diffusion classifier.
We propose two principles about good noise for DC: Frequency Matching and Spatial Matching, and conduct two empirical experiments to verify them.
Regarding the Frequency Matching and Spatial Matching, we design an effective noise optimization method (NoOp). It optimizes a dataset-specific parameterized noise and a Meta-Network that can output the image-specific noise offset. By using the sum result of optimized noise and noise offset, the DC can gain stable and significant improvements across datasets.
Extensive experiments show the effectiveness and stability of NoOp. Meanwhile, our NoOp has the orthogonal capability with other optimization methods like prompt optimization for DC. This verified that NoOp is new few-shot learner with a unique effect. We look forward to these observations opening the door for robust generative classifiers
2 Related work
Diffusion Models (DM) for Image Classification. As state-of-the-an generative model, DM [1] 16] shows remarkable vinual-language modeling capacity. In that case, recent studies try to unleash its discrimination potential for image classificatio.. Studies 66 7 9 g try to leverage the vision- language knowledge of pretrained text-to-image diffusion models for zero-shot classification, while Yue el.al. [10] adopts the prompt optimization techniques into DM for few-shot classification. In this paper, we dive into the classification task for a new few-shot learning method of diffusion classifier.
DM for Perception Tasks. Beyond image classification, DM can also be used in more challenging perception tasks. From a framework perspective, Chen et.al. [7 leverage the DM to generate the detection bounding boxes in a refinement manner. From the pretrained knowledge perspective, studies
apply the knowledge of DM in image segmentation [8 19 and depth estimation [20].
Noise Instability in DM. In the image generation field, there is a widely discussed phenomenon, iee, the start point of the denoising process matters a lot to the final generation quality. To relieve this problem, recent studies can mainly be divided into three types. 1) Finding better sampling distribution instead of the Gaussian distribution by being supervised by a third-party model [1 22].
2) Leveraging prior knowledge [33 24 25
26
27
28
29
30 to directly refine the noise or the
3

---

photo of
V
V
hird
photo of [dog, bird]
Input Image X
Input Image Xa
Noisy Imuge
Noisy Image
Sampled Noise
Learable Nosse
classification
(a) Prompt Optimization
(b) Noise Optimization
Comparisons between existing prompt optimization and our noise optimization. (a) Prompt optimization learns few text tokens with reconstruction loss. (b) Noise optimization optimizes parameterized noise with classification loss.
Figure 2:
denoising path. 3) Directly optimizing the image-noise matching mechanism during the training process [2]. However, we find that the noise instability also exists in the image classification task, i.., different noise leads to different classification performance. In this paper, we analyze the role the noise and propose an optimized method to mitigate this problem.
Method
Few-Shot Classification Formulation. For the K-way-N-shot image classification task, typically there is a training set D with K categories {ci,..,, C.) For each category, there are labeled training samples. The few-shot learning aims at improving the model based on the training set to perform better classification on the full test set. For the diffusion model, it has multiple discrete timesteps, and each timestep controls the ratio of the original image To and noise e. The timestep input as a condition into the denoising network ég, which means corresponds to (-specific model co(:,-, l). Thus, for each timestep we can consider model with specific noise level plus specific denoising network co(::,, t). In this paper, we focus on noise optimization, thus simplifying the problem by fixing the timestep at t (e.g., 500).
3.1 Preliminaries
Diffusion Classifier (DC). Diffusion model (DM) trains denoising network Eş that can reconstruct the noisy image into the original image. To be specific, given an original image To. its corresponding category text c, and timestep t, DM first samples noise from the Gaussian distribution, then adds it to To to get the noisy image by the following forward process.
(I) where the is a (-related predefined parameter to control the ratio of xo and Then DM inputs It, e and into the denoising network eg to predict objective is minimizing the MSE loss between éØ (x () and ground truth e:
Et v Va + vI TiC.
added noise, iee, Eg(a,, C,L, The training
min E.,..c. [|le èo (ze, c, t)
Based on this training objective, the diffusion classifier (DC) can leverage the effect difference of different to perform image classification. Specificly, given test image zo and t, DC first adds noise to get the noisy image He (c.f., Eq. (T). Then use K categories {CI,..., CK as guidance to denoise 2; respectively and get K noise prediction {ēo G, " Thus, based Eq. (2) DC selects the category that can best denoise Ie as the final category prediction:
a = argmin E. E (xt, c,,t) - el12 i = 1,2, K..,... K.
Prompt-Optimization of DC. The pertaining dataset of diffusion models may have distribution gap with the downstream classification datasets. Thus, it is hard to use few natural words to accurately describe a category directly. To fill this gap, existing few-shot DCs IT] are mainly based on prompt optimization 22 3. As shown in Figure a), for each category, they set n learnable text tokens [[V]] [V2] ... [V,]) and add them into the text prompt to supplement the details of the category. Then they fix the whole diffusion model and only tune tokens the few-shot training set based Eq. 2

---

Spatiol Matching
Meta-Network
Trainable
Eifirette
Denoising Network ća
Frozen
No Grodeers
Noise Offset e
a photo of a (q:0)
Z-Score Normalization
E'
+
P2
Xt
Ez
Training Set
cls CE(c, {z1:2k3)
PR
7x
Training Objective
εK é = é C₁ t)
Logits
Normed
Learnable Noise
Logits
Figure 3: Pipeline of NoOp. Given a diffusion classifier and few training samples, we optimize parameterized noise (6) and a Meta-Netwe Network based on the classification loss. The Meta-Network is used predict an image-specific noise offset (E'). Besides, we use Z-score normalization for more stable training. In inference, we use the sum of optimized noise and the offsee as the final noise (e*).
3.2 Noise Optimization
From new perspective, our method focuses on finding a better noise that can be used in DC. As shown in Figure (b)). we try to optimize the noise DC for better performance on image classification. To find the guideline of good noise, we first consider the physical role of noise in DC: noise used for destroying some signals the original image xp- the lost signals are category- related, the denoising effect differences can emerge under different category guidance. Thus, good noise can target the category-relatee signals. Specifically, the noise should meet two principles:
1) Frequency Matching: Since the sample features within one dataset usually satisfy specific distribution, the category-related signals of the samples the dataset will fall into specitic frequency range. For different datasets, the category-related signals are distributed in different frequencies. Thus, the noise should destroy the specific frequency signals of the target dataset. Motivated by this, we randomly initialize noise e for the whole dataset and make it learnable. This noise is expected to be optimized for destroying the category- related frequency signals.
2) Spatial Matching: Different images within one dataset have different spatial layouts. The category relevance of different spatial parts varies lot, i.e. some spatial parts of the image category-related and some are not. Thus, the noise should have different degrees of destruction regarding different parts of the image. Motivated by this, set a Meta-Network Up that adopts the original image as input and outputs an image-specific noise offiset e'. This offset indicates the spatial adjustment with respect e, which can make the noise better destroy the category-related spatial parts.
Training. As shown in Figure 3 firstly, we have a training set D, a pretrained denoising network Eg, the learnable and from a Gaussian distribution. The Meta-Network a light U-Net architecture containing multiple convolutional layers of up-sampling and down-sampling For each (mm, e) 'D first input the To into the Un to get the noise offset é' = (oo) Then, we get a refined noise c* = e + é'_ After that, we conduct Eq. (T) to get the noisy image Ie (the noise used for the forward process is e"). Then we input xt, and all categories {CI,..., cK) into denoising network to get K noise predictions (x, c,, } j= Based on the distance between noise predictions and ground truth noise e., can calculate the classification logit for each category:
Ug. The parameterized is initialized by randomly sampling
= -Ile" - Eo(2,, c, t)l
(4)
where the minus operator "_"is is to convert the distance into the classification logit.
5

---

After getting the classification logits, we found that all logit values are very close. This is because they are calculated based on the distance between predicted noises and the ground truth noise. And all the distances are very close due to the diffusion model's property. Thus, if we directly optimize the and Up hased on these logits, the optimization gradient is close to zero, making training difficult. To mitigate this problem, we use Z-score normalization [33] on the category dimension to effectively amplity the signal difference between the logit. Specifically, we first calculate the mean μ and biased variance a of P2:,P2.... .... PK} respectively:
Σ j=1
Pa
0
(ps-
(5)
j=1
Then, based on the mean and variance, calculate each normalized logit:
2, Pi
1, ..... K.
(6)
After getting the normalized logits {z1, F2. ..., =K we can optimize our e and Ug by minimizing the cross-entropy loss on the training set:
E, U9
argmin c.Ug
Σ
log
exp (2) Σ
KN
exp (2:)
(FO.0)ED
where = is the normalized logit of the ground truth category. After training, we can get an optimized datasct-specific and a Meta-Network that can produce an image-specific noise offset e'. We use e" = 6 + e' as the final optimized noise that can meet frequency matching and spatial matching well. We use e* to replace the randomly sampled noise in DC methods for inference.
4 Experiments
4.1 Few-Shot Learning
Settings. To evaluate how NoOp can benefit the DC, we conducted few-shot learning experiments.
Specifically, we followed the few-shot evaluation protocol of CLIP M. using 1, 2, 4, 8, and 16
shots for training, respectively, and deploying models in the full test sets. We evaluated three diffusion models, i.e., Stable Diffusion-vl..4 Stable Diffusion-vl.5 [34], Stable Diffusion-v2.0 (35)
across eight datasets: CIFAR-I0 10 [36). CIFAR- 100 36 Flowers102
37 DTD [15]. OxfordPets (38]
EuroSAT [39. STL-10 [14 and FGVCAirerafi 40 We compared our NoOp with the ensembling
methods 7 6 sampling 5 noises for each image and consequently taking times the computation for inference). For fair comparisons, we fixed the timestep = 500. We used the Adam optimizer [41 with le and 1 learning rates for the learnable noise the Meta-Network respectively. After training 20 epochs, we reported the top-1 accuracies. Results are averaged on three random seeds.
Results. From the results in Figure we have two observations: 1) Though our NoOp only refines the noise (a very scall-scaled part of the diffusion model), the performance improves with a relatively large scale. This demonstrated how severe the noise instability problem of DC is. 2) Across different datasets and DC versions, our NoOp can gain consistent improvements and outperform the 5-times expensive ensembling methods (e.g., according to the average over eight datasets, with only two shots of NoOp can beat ensembling). This indicates that NoOp is an efficient few-shot learner.
4.2 Compare with Prompt-Optimization DC
Settings. To explore the difference between noise optimization and prompt optimization. We compared the performance of a prompt-optimization based DC method i.e. TiF Learner [10), our NoOp, and the combination of both on FGVCAircraft [40] and ISIC-2019 [2]. For fairness, we used Stable Diffusion-v2.0
35 as the classifier. All other settings are the same as Sec.
Results. Shown in Table I the second and third rows are the few-shot performance TiF Learner and NoOp, respectively. We can see that both of them can improve the classification independently. And generally, the performance can be further improved by combining them. This indicates that NoOp is a new few-shot learner, which has different effect from the current prompt optimization.
'The prompt-based optimization DC introduced in Sec. 3.1

---

Average over decasets
DTD
CIFAR-10
65
88
20
Accurac (k)
R2D0 w-33
50
0 1
1
3
16
Number of lobeled training esamples per dlss
Number of labeled training examples per class
Number lobates training esamples per class
CIPAR-10C
EuroSAT
Flowers102
40
Accurscy
Accurscy
20
20
5:00
+2.
15
Redp Medp vi.
N.mber of leblle training esamples dass
Number of labeled training exampies los class
Number of lobates training examptes per class
STL10
OxfordPets
FGVCAircraft
Accursiy 180)
70
Accuracy
Holp +--4
1600 INCp 2-1.
5400 V-3.0
5640 V1-.A
560p v-2
Number labeeedd
labeled
Number labeled
Figure 4:
Main results of few-shot learning on the eight datasets. Overall, NoOp effectively turns DC into strong few-shot learner, achieving significant improvements over zero-shot DC (shot=0) and generally outperforms (with only 2 shots) the expensive ensembling methods.
Table 1: Comparison of Tif Learner, NoOp, and the combination of them. Overall, both prompt optimization (TiF) and noise optimization are effective few-shot learners. Moreover, their effects are complementary, which means that they can further improve performance when used combination.
ISIC-2019
FGVCAireratt
Method
2
16
4
Zero-shot DC
13.25
13.25
13.25
13.25
13.25
11.31
11.31
11.31
11.31
11.31
Learner
17.25
13.89
19.76
19.91
17.53
15.60
17.04
16.98
19.47
21.03
NoOp
18.41
20.80
23.72
18.41
20.82
11.82
12.51
12.81
13.95
14.61
NoOp TiF
18.32
19.23
21.59
15.99
18.54
19.59
22.44
25.74
4.3 Cross-Dataset Transfer
Settings. To demonstrate NoOp's generalization ability open-set recognition, we evaluated our method the cross-dataset transfer experiments. We optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested it the source dataset and eight target datasets. We compared our method with the ensembling methods (5 times computation), and A denotes our method's gain over ensembling.
Results. As shown in Table the noise and Meta-Network learned on ImageNet can also be used in most of the target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average. This demonstrated that the noise optimization can learn some generalized knowledge that is beneficial to the classification, i.e., how to destroy the target part the image and better utilize the reconstruction capacity to achieve classification. To this end, this result indicates that our method can be further leveraged in open-worke scenarios.

---

Table 2: Comparison of the Ensemble method in the cross-dataset transfer setting. Noises optimized on the source datasets (4-shot ImageNet) and applied to the eight target datasets. NoOp
demonstrates good transferability across datasets. A denotes NoOp's gain over Ensemble
Source
Target
ImageNat
DTD
CIFAR-10
CIFAR 100
EuroSA
Flowers 102
STL-10
OxfortPets
POVCAierat
Average
Ensemble (Time x5)
25.94
17.34
69.91
17.96
28.37
17.47
75.44
42.41
9.42
34.79
NoOp
26.34
21.70
63.26
29.36
29.48
71.66
45.90
10.56
37.65
+0.40
+4.36
-6.65
+11.40
+12.01
-3.78
+3.49
+1.14
+2.86
6.5
Only F F-Matching
6.0
Only S-Matching
F- F-Matching Matching
45
5.5
Accurac
44
5.0
43
4.0
Only F-Matching
Only Matching
3.0
0
5
10
15
20
25
0
10
14
18
Training epochs
Training epochs
(a) Test Accuracy
(b) Training Loss
Figure 5: Ablation Study. The test accuracy and training loss curve across epochs of only frequency matching ("F-Matching" in blue line, i.e., only optimize a dataset-specific noise), only spatial matching "SS-Matching" in green line, iee, only optimize a Meta-Network for image-specific noise offset) and combining of them (red line).
4.4 Ablation Study
Settings. To evaluate the effect of parameterized noise optimization (for frequency matching) and Meta-Network optimization (for spatial matching). We ablated the effectiveness of each component on 16-shot OxfordPets [38]. We used Stable Diffusion-v2.0 335 as the classifier and reported the top-1 accuracy and training loss across training epochs. All other settings are the same as Sec. 4.1
Results. As shown Figure we have two observations: 1) Both frequency matching (F-Matching) optimization and spatial matching (S- (--Mtching) optimization can improve the classification during training. This indicates that the two components are designed appropriately. 2) Further accuracy improvement and faster convergence are gained by combining them, demonstrating that F-Matching and S-Matching optimize the noise two different perspectives.
4.5 Extend Flow-based Diffusion Models
Settings. To evaluate the generalization and robustness of NoOp, we transferred it to flow-based diffusion models [43 32). We demonstrated the few-shot results of Stable Diffusion-vI.5 [34
(SD-v1.5) and the Rectified Flow [32] (ReFlow, fine-tuned from SD-v1.5) on CIFAR- 10 [36 and OxfordPets 38) All other settings are the same Sec. 4.1
Results. As shown in Table we have two observations: 1) The performance of ReFlow better than SD-v1.5, although they have the same training set and architecture. This suggests that the flow-based diffusion model is also superior in the discriminative task, not just limited the generative domain. 2) Our NoOp can not only improve the DDPM-based [II] SD-v1.5 5 but also the flow-based [32 43 ReFlow. This indicates that our analysis of the noise role and optimization of noise is a general framework for the diffusion family.

---

Table 3: The fhw-shot results of NoOp with SD-Table 4: The computational time comparisons v1.3 and the ReFlow. Our NoOp is compatible with between ensemble and NoOp. The unit of time general diffusion models.
cost is one hour with 32 V V100 GPUs.
CIFAR-I0
OxfordPets Reflow
Setting Ensemble
Training
Inference
Total
Acc (%)
Shot Numher
v-15
Reflow
SD
0
153.0
153.0
25.94
0
47.30
70.15
39.90
56.23
Shot = 0
0
30.6
30.6
17.34
shot
60.72
73.64
42.00
58.79
Shot
0.4
30.6
31.0
24.62
shot
54.37
43.31
60.63
Shot
31.4
25.31
shot =
60.11
74.51
41.97
60.19
Shot = 4
1.6
30.6
32.2
26.34
shot 8
63.03
76.34
44.67
62.74
Shot =
3.2
30.6
33.8
27.59
shol 16
64.89
79.73
44.15
63.89
Shol = 16
6.4
30.6
37.0
29.13
4.6 Computational Overhead Analysis
Settings. As the original Diffusion Classifier suffers from high computational cost and slow inference speed, to verify whether our method can improve the efficiency, We used ImageNet as an example to demonstrate how our method can improve the efficiency of the original Diffusion Classifier. The unit of time cost is one hour with 32 NVIDIA V100 GPUs. For the ensemble method, we randomly sample five different noises.
Results. As shown in Table 4 our training time is quite smaller than the inference time. Even for the 16-shot training, the total time only around 24% while the accuracy gains 3.19% compared with the 5-times ensembling method. This demonstrates that our method can significantly improve the overall efficiency. Additional formal cost analysis the Appendix B
4.7 Role of Noise Validation
Settings. this paper, we argue that the good noise in DC trying to destroy the category-related
signals. To validate this assumption, we compared two sets of noisy images. 1) Noisy images
by adding random noise. 2) Noisy images by adding optimized noise (from NoOp). For fairness,
we fixed 500 to get the noisy images. We tested on 4 datasets: OxfordPets [3 DTD [I5).
EuroSAT [39 and Flowers 102 (37), and then reported the averaged CLIP Score [44. The CLIP
Score can reflect how much category-related signals are destroyed. A lower CLIP Score indicates that
the noise destroys more category-relater signals. we used Stable Difsusion-v2.0 (4 as the classifier
and VTT B/3- CLIP for CLIP Score calculation. All other settings are the same Sec. 4.1
Results. As shown in Figure Gl)), the CLIP Scores are decreased on all datasets. This indicates that our optimized noise can better destroy the category-relates signals of images, which can support our analysis of the role of noise in DC.
4.8 Frequency Matching Validation
Settings. To validate our proposed frequency matching principle, we selected two representative datasets for noise optimization: 1) CIFAR-10 10 [36], coarse-grained dataset where the categories are mainly distinguished by low-frequency signals (object shape and structure). 2) Describable Textures (DTD) [1]] a texture dataset where the categories are mainly distinguished by high-frequency signals (mutated texture). We randomly sampled one noise and then directly optimized it (w/o optimizing the Meta-Network) these two different datasets, respectively. During the training process, recorded the frequency change this noise by calculating the tigh-frequency signal ratio of the noise. The method to get the high-frreuency signal ratio is based on the 2D Fourier transform (c.f., Appendix B higher fregefrey signal ratio means this noise a relatively fighfrrequency noise.
Results. As shown in Figure when training on the low-frequency dataset, i.e., CIFAR- 10, the frequency of the noise will decrease. Otherwise, when training on the high-frequency dataset, i.e., DTD, the frequency of the noise will increase. This indicates that, the frequency of noise will move towards matching the frequency of the dataset, which verifies our analysis of noise role and the frequency matching principle.

---

83.00
21AA
Optimized Noise
ratio
82.75
215
82.50
36.5.
82.25
02.00
81.75
10
01.50
High
81.25
Describeble Texturen (OTD)
Training epochs
EUroSAT
DTD
OxfordPets
Flowers102
(a) Noise Role
(b) Frequency Matching
Figure 6: Empirical Validations. The average CLIP Scores comparison of the noisy images to validate the role of good noise. (b) The high-frequency signals ratio comparison of DTD and CIFAR-10 to validate the frequency matching principle.
80
28.4
70
50
60
Accuracy (%)
40
occuracy
50
30
20
10
10
t=500
NoOp Timesteps
(b) Timesteps Ensemble
Figure 7: Ablation study and ensemble results. (a) The performance of NoOp across different
timesteps. (b) The performance of the timesteps ensemble (5 different timesteps) of NoOp.
4.9 Time steps Ablation and Ensemble
Settings. In the main paper, we fixed the timestep at 500 to focus on the noise optimization. To evaluate whether our NoOp is consistently effective on other timesteps. We conducted timestep ablation study. We tested / = 300, 400, 500, 600, 700 with Stable Diffusion-v2.0 0 [35 as the pretrained diffusion model on 4-shot CIFAR-10 36). Moreover, we also test the ensemble with different
timesteps (i.e., t = 300, 400, 500, 600, 700) after using NoOp. The pretrained diffusion model is Stable Diffusioo- v2.0 (35 and dataset 4-shot CIFAR-10 10 [6]. We also showed the results of zero-shot w/o ensemble, zero-shot w/ensemble, and NoOp w/o ensemble for comparison.
Results. In Figure a). NoOp can gain consistent improvements indicates that NoOp is robust and effective across timesteps. Besides, as shown in Vigure[]b) we can that compared with the one-timestep (i.e., 500) NoOp. the timesteps ensemble can improve around 12% top-1 accuracy and outperforms the zero-shot ensemble 5.4%. This demonstrates that the ensemble strategy can also be applied for our NoOp if there are enough computational resources.
different timesteps. This
5 Conclusion
In this paper, we revealed the noise instability in the diffusion classifier and analyzed the role of noise. Then we proposed two principles guidelines for designing the noise optimizing framework. i.e., NoOp, to refine the random noise into stable matching noise. Extensive experiments show that our NoOp is an effective few-shot learner. can not only lear generalized knowledge for cross-dataset recognition and be compatible with the flow matching models. Moreover, we conducted two empirical experiments to support our proposed principles. As highlight, we find that our NoOp is orthogonal to existing optimization methods like prompt tuning. This shows that noise optimization has a unique physical meaning and effect. the future, we are going to extend the noise optimization into flow matching models and unified models (e.g., auto-regressive model plus diffusion model).
10

---

6 Acknowledgements
This work was supported by the National Natural Science Foundation of China Young Scholar Fund (62402408) and the Hong Kong SAR RGC Early Career Scheme (26208924).
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748- 8748-8763. PmLR, 2021.
[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022..
[3] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[4] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- languuge models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01953 2022.
[5] Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. Clip behaves like a bag-of-words model cross-modally but not uni-modally. arXiv preprint arXiv:2502.03560 2025.
[6] Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Ad- vances in Neural Information Processing Systems, 36:58921-58937, 2023.
[7] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[8] Zipeng Qi, Buhua Liu, Shiyan Zhang, Bao Li, Zhiqiang Xu, Haoyi Xiong, and Zeke Xie. A simple and efficient baseline for zero-shot generative classification. arXiv preprint 12X5522412.12594, 2024.
[9] Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu, Robust classification via a single diffusion model. arXiv preprint arXiv:2305. 15241,, 2023.
[10] Zhongqi Yue, Pan Zhou, Richang Hong, Hanwang Zhang, and Qianru Sun. Few-shot learner parameterization by diffusion time-steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23263 22222,,, 2024.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[12] Zhiyu Lin, Yifei Gao, and Jitao Sang. Investigating and explaining the frequency bias in image classification. arXiv preprint arXiv:2205.03154 2022.
[13] Shunxin Wang, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio. What do neural networks learn in image classification? frequency shorteut perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1433-1442, 2023.
[14] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[15] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.

---

[16] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
[17] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVE international conference on computer vision, pages 19830-19843, 2023.
[18] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open- vocabbaayyy segmentation. arXiv e-prints, pages arXiv-2306, 2023.
[19] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEFSCVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[20] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
pages
[21] Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:24/1.09550 2024.
[22] Changgu Chen, Libing Yang, Xiaoyan Yang. Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Find: Fine-tuning initial noise distribution with policy optimization for diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6734, 2024.
[23] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4695-4703, 2024.
[24] Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Semantic image inversion and editing using rectified stochastic differential equa- tions. arXiv preprint arXiv:2410. 10792, 2024.
[25] Shuangqi Li, Hieu Le, Jingyi Xu, and Mathieu Salzmann. Enhancing compositional text-to- image generation with reliable random seeds. arXiv preprint arXiv:2411.78810 2024.
[26] Ruoyu Wang, Huayang Huang, Ye Zhu, Olga Russakovsky, and Yu Wu. The silent prompt: Initial noise as implicit guidance for goal-driven image generation. arXiv preprint arXiv:2412.05101, 2024.
[27] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting eet-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9380-9389, 2024.
[28] Lichen Bai, Shitong Shao, Zikai Zhou, Zipeng Qi, Zhiqiang Xu, Haoyi Xiong, and Zeke Xie. Zigzag diffusion sampling: Diffusion models can self-improve via self-reflection. In The Thirteenth International Conference on Learning Representations, volume 2, 2024.
[29] Zipeng Qi, Lichen Bai, Haoyi Xiong, and Zeke Xie. Not all noises are created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407. 14044,, 2024.
[30] Lichen Bai, Masashi Sugiyama, and Zeke Xie. Weak-to-strong difffusion with reflection. arXiv preprint arXiv:2502.000 00473, 2025.
[31] Xu Shifeng, Yanzhu Liu, and Adams Wai-Kin Kong. Easing training process of rectified flow models via lengthening inter-patt distance. In The Thirteenth International Conference on Learning Representations.
[32] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint
2022.
12

---

[33] Chris Cheadle, Marquis P Vawter, William Freed, and Kevin G Becker. Analysis of microarray data using z score transformation. The Journal of molecular diagnostics, 5(2):73-81, 2003.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3, 2022.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[37] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphies & image processing, pages 729-729. IEEE, 2008
[38] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[39] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[40] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 130.51130..515,, 2013.
2013.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6980, 2014.
[42] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[43] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:22 2022.
[44] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A 2021.
reference-free evaluation metric for image captioning, arXiv preprint
[45] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-mmage diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pages 3836-384 2023.
[46] Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, and Adams Wii-Kin Kong. Robust water- marking using generative priors against image editing: From benchmarking to advances. arXiv preprint arXiv:2410. 18775, 2024.
[47] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion- training-free cross-domain image composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2294-2305 2023.
[48] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[49] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
[50] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in neural information processing systems, 34:980-999 9933, 2021.
13

---

[51] Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy. Deyu Meng, Chongyi Li, et al. Deep fourier up-sampling. Advances Neural Information Processing Systems, 35:22995- 230088 2022.
[52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- asssiste intervention, pages
Springer, 2015.
[53] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint
18X.::1803.08375: 2018.
[54] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
[55] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208.0100 01618, 2022.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We give the claims and summarize our contributions of our work in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
We discuss the limitation of our method in appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We have some relevant proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We give the specific model version, hyperparameters, and code (in supplemen- tary) for reproducibility.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The supplementary materials include the detailed code we provided, and the data used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee///deuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper give the training and inference details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All our quantitative results are from three different and random runs for statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We give the computation details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpss:/neurrpps. http:///wworrrp.cccoooaaaaaccaaa
Answer: [Yes]
Justification:
Our research complies with all relevant requirements.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our research complies with all relevant license requirements.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thhors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We give the code with documented instructions for execution in the supple-
mentary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification:
have not used any LLMs for this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy bettps://neurigee ce. /c////oaneeeeecce/222////111 for what should or should be described.
21

---

Appendix
This appendix is organized follows:
introduces the backgrounds of diffusion classifiers (DC), including the basic theory, mlplmmmntatoon,, and advantages compared with Vision-language Models (VLM).
Section A
Section gives the theory of the 2-D Fourier transform, i.e., the frequency analysis method for images in our
related experiments.
Section gives the justification of the NoOp's 's mechanism.
Section provides more experimental results. Firstly, we give a few additional few-shot learning results compared with the prompt-optimization DC in Section D.I] Secondly, we provide the ablation results of Meta-Network with different architectures in Section D.2 Finally, give the qualitative results of our NoOp's stability in Section D.3
Section D provides the implementation details of our NoOp and the reproduction details. Besides, we also give the computation cost.
Section analyzes the limitations of noise optimization DC and its societal impacts.
A Background of Diffusion Classifiers (DC)
The recent surge of visual generation benefits from Diffusion models 11 16.
and high-quality images and videos generated by sampling from Gaussian noise. Meanwhile, the downstream tasks include editing [45 [6), composing [47], and erasing 48 are also frequently researched.
The Theory of DC. DC leverages the vision-language alignment knowledge of pre-trained diffusion models, which are trained to progressively denoise noisy images through an iterative forward and reverse Markov process. Specifically, the diffusion process is defined by the forward Markov chain, progressively adding Gaussian noise to a clean image zo to produce the corresponding noisy image Tzi
(< 1) = (ax; v Vā (1 T.)
where ai determines the amount of noise added at each step. Conversely, the reverse process attempts reconstruct the original image from the noisy version using learned denoising functions μi and Eg:
pa( NV (2 (e t). [x f)).
A diffusion classifier uses the denoising capability of the diffusion model to approximate the clas- sification probabilities of each category. Specifically, given clean image zo, class labels cs the classifier selects the class minimizing the weighted reconstruction error as follows:
è argmin Ee, [wel]x To (2,, ci, t)
(10)
where 10, Uniform (0, T)), and w, is time-dependent weighting function. To estimate the expectation efficiently, Monte Carlo sampling employed:
[lf(x) R N f() 200 ~ q(Ie(zo),
(11)
where each noisy sample (1)) is generated independently.
The Implementations of DC. In practice, DC implementations often utilize pretrained text-to- image diffusion models such as Stable Diffusion B4 B5]. The timestep weights can be even or varied, such as based on the signal-to -oisee ratio (SNR). Another typical implementation involves computing a re- weighted reconstruction error across diffusion time-steps, selecting the class whose conditioning prompt results in minimal reconstruction eтror at an early time-step [10]. More efficient implementations reduce computational overhead by applying variance reduction strategies, including shared-noise sampling and candidate class pruning. Alternatively, few-shot DC learn class-specific concepts on top of pretrained diffusion models to enhance the classification of nuanced attributes, improving robustness against spurious visual correlations [10].
22

---

The advantages of DC. Compared with langon- model (VLM) classifiers such as CLIP, DC exhibits several significant advantages. Firstly, DC demonstrates superior multimodal compositional reasoning and attribute binding capabilities, outperforming VLMM classifiers on tasks requiring nu- anced understanding of visual features and their textual descriptions G 7 Secondly, DC inherently possesses robustness to texture-shape biases and spurious correlations, a common issue in discrimina- tive few-shot training with VLMs, by utilizing a principled generative modeling approach that isolates semantic attributes effectively through diffusion time-steps [10]. Lastly, the generative modeling nature DC allows seamless adaptation to few-shot and zero-shot learning settings without extensive retraining, making them versatile and efficient for various downstream tasks 66 10
B Theory of 2-D Fourier Transform
The wwo-dimensional (2-D) Fourier transform a fundamental tool in frequency domain analysis of images. Given spatial domain image fx, y). the continuous 2-D Fourier transform F F(u, v) is mathematically defined as:
F(u,V)= { f(x y)e (ole--2((urt))) drdy.
(12)
where is the imaginary unit, u and represent the spatial frequencies in the horizontal and vertical directions, respectively. Conversely, the inverse 2-D Fourier transform recovers the original spatial image from its frequency representation:
[(z,y)=
F(u, 2 dudv.
(13)
In digital image processing, discrete counterparts of these transforms are commonly utilized due to practical constraints. Given discrete image (z, v) of size M N, the Discrete Fourier Transform (DFT) is defined as:
M-1N N-1
F(u, v)
Σ
Σ f(,
52 ur M +)
(14)
y=0
and the inverse Discrete Fourier Transform (IDFT) is:
M M-11 1 N
f(x,8) = M N Σ v=0
Σ
F(u,o) (M7 ((+
(15)
The Fourier transform decomposes an image into its constituent frequency components, effectively distinguishing low-frequency (smooth or slowly varying) and high-frequency (sharp edges and fine details) content [49 50]. In our frequency matching validation (c.f., Sec. leveraging this decomposition provides valuable insights into how diffusion processes interact with different frequency components, informing analysis and manipulation strategies various frequency scales [5I] After the above frequency decomposing technique, for each image or noise latent, we can represent its frequency by its high-frequency signals ratio. Specifically, we cutoff threshold (e.g., 0.3 was used for Figure 7(b)) to divide all the signals into two parts: high-frequency and low-frequency. The ratio of high-frequency signals to low-frequency signals is used to represent its frequency.
C Justification of the mechanism of noise optimization
To justify the mechanism of noise optimization and why works for the Diffusion Classifier, we first recall the mechanism of the Diffusion Classifier (DC).
Given an inference image, the DC firstly adds some noise to destroy some of the signals that related to the category. Then, using different categories as the prompts to reconstruct the noisy image. The final prediction the category that can reconstruct it the best.
In this pipeline, a very crucial part is adding noise because it decides which signals are removed by adding noise. The best situation is trying remove the -elaged signals and maintain other
23

---

cetteoor-indepennn signals. In that case, the reconstruction difference of diffusion models under different prompts can be fully unleashed.
Thus, our method can learn an optimized noise to target the category-related signals. This kind of optimized noise is usually in the low-probability areas of the standard Gaussian distribution, and thus is very difficult to sample. In comparison, the randomly sampled noises are usually in the high-pre high-probability areas of the standard Gaussian distribution but lack the target-destroying capacity.
We compared the random noise and the optimized noise and reported their mean, variance, and standard Gaussian distribution probability density (pdf). For the randomly sampled noise, the mean, variance, and pdf (log) are -0. -0.0032. 1.0021, and 232655, respectively. In contrast, for the optimized noise, they are -0.0043, 1.0282, and 234799.
We can see that, compared with the random sampled noise, the optimized noise's mean is farther away from 1, variance is farther away from 0, and the probability density is lower. This demonstrated that the optimized noise is usually in the lowproobaillity areas of the standard Gaussian distribution to target the category-relacee signals, And this kind of noise is difficult to sample. Thus, we obtain it by optimization.
D Additional Results
D.1 Additional Few-shot Learning Comparisons
Settings. To further evaluate the effectiveness of our NoOp, we conducted additional few-shot ($hot=1,,,448) learning experiments. Specifically, we compared our NoOp with two baselines, i.e., ensemble (sample five different noises for one image) zero-shot DC 17 6 and prompt learning DC T0 2 (c.f., Section 3.1) across eights datasets: CIFAR-10 (36), CIFAR-1OO [36,, Flowers102 37),
DTD 11,. OxfordPets [38] EuroSAT [3), S0 14 and FGVCAireraft [40]. We used Stable Diffusion-v2.0 335 as the pretrained diffusion model. Results are averaged on three random seeds.
Results. As shown in Figure 8 we can have two observations: 1) As two few-shot learners, both prompt learning and noise optimization can outperform expensive ensemble methods with less than 4 shots. 2) Prompt learning improves significantly on EuroSAT; Flowers /02 and FGVCAircraft while NoOp performs better on other five datasets. However, we can see that NoOp is generally more stable few-shot learner. Because for some datasets and shot numbers, the prompt learning may decrease the performance (e.g., when the shot number is 1, the performance of prompt learning is lower than zero-shot on six datasets). In contrast, our NoOp shows more stable and consistent improvements across datasets and shot numbers.
D.2 Meta-Network Ablation
Settings. To ablate the Meta-Netwrok, we compared different Meta-Network architectures (CNN and ViT) and reported the parameter count, FLOPs, and classification accuracy. For the CNN-based Meta- Network, we used our original Meta-Network in the paper since consists of convolutional layers. The ViT-based Meta-Network consists of six ViTBlocks with layernorm and residual connections. We conducted the experiments on the 16-shot OxfordPet with Stable Diffusion-v2.0.
Results. As show in Table 5 both CNN-based and ViT-based Meta-Networks can gain remarkable performance (6-7% accuracy improvement compared with random noise) with only 6-8 M parameters and less than 3 G FLOPs. This indicates that a light Meta-Network with common architectures can accurately learn the noise offset, demonstrating our method is robust and efficient.
D.3 The Stability of NoOp
Settings. To evaluate the stability of NoOp, we randomly sampled three noises (three different seeds)
from the Gaussian distribution as the initial noises. Then we conducted the zero-shot classification
with them on CIFAR-/0 10 (36] test set, respectively. Then we used NoOp to optimize them on the
8-shot training set and test. The pretrained diffusion model we used is Stable Diffussio-v2.0 (35) We
reported the 1 accuracies before and after training. All other settings the same as Sec. 4.8
24

---

Average over darasets
24
22
65
Accurscy Uha
35
Accurec 0%)
60
Accuracy
55
18
50
16
Fronun Learsing
14
Loeming
Promppt Leanning
2
2
2
Number of training examples per class
Number of abeted training exampres per dass
Number labeled training examples per class
CIFAR-100
EuroSAT
Fowers102
Eneorbie
Prieeet Learsing
35
Accuraiy 1N0
25
30
Acturacy
40
20
25
Aroestia riine Holp
20
kreandia (Tina xai Hello
20
5
15
Number labeled training esamolas per class
Number abeled training esampres class
Number labeled training acamples class
OxfordPets
45
70
40
Accurscy 11
Accuracc 18)
35
Accurscy
23
30
20
25
Encamble (lina Pruvsal Learshe
kreanble (Tima
Leareing
20
training esamoless per
abeled training esamptes pert
training acamples per class
Figure 8: Additional results of few-shot learning the eight datasets. The prompt learning has two problems: (1) It generally harms the performance when the shot number is small (e.g., when the shot number is 1, the prompt learning performs
than zero-shot DC on six datasets). (2) The variance of prompt learning performance is large (e.g., quite high accuracy improvement for EuroSAT and Flowers100 102, while quite low for OxfordPets). contrast, our NoOp can gain more stable and consistent improvements across datasets and shot numbers.
Table 5: The ablation study of the Meta-Network with various architectures.
Figure 9: The stability of NoOp.
Architecture
Params (M)
FL.PSs (G)
Accuracy (9)
random noise
40.07
Seed 2
CNN
7.70
2.61
47.18
62.5
Seed
6.59
2.46
46.28
60.0
57.5
55.0
52.5
50.0
Betore Training
After Training
Results. As shown in Figure 9 the variance of performances before training is obviously larger than the performances after training. This verified the noise instability problem of DC and the stability of our NoOp, i.e., our method is robust and stable across different random initial noises.
E Implementation Details
In this section, we give all the implementation details of our NoOp, the reproduction details, and the computation overhead.
25

---

We used the Discrete Euler as the timestep scheduler, max length padding for the text tokenizer across all the experiments. The imitial noise is randomly sampled from the standard Gaussian distribution, while the Meta-Network is a U-Net 52 architecture with 3 up-sampling layers and down-sampling layers. Each sampling layer consists of 2-D) convolutional layers with Rel uu activation [53] and BatchNorm [54]. During training, we used fixed learning rates (w/o warm up and scheduler). The training batch size is 32.
Details of NoOp.
Other Reproduction Details. For prompt optimization DC (c.f., Sec. 3.1 implementation in Sec. 4.2 and Sec. D.I we followed the implementations of TiF Learner 10 respectively.
and textual inversion 55
Hardware. All experiments conducted on 32 NVIDIA V100 GPUs.
Computation Overhead. Use 16-shot CIFAR-10 as an example, the training time of one epoch and the inference time is around 11 seconds and 220 seconds, respectively. This indicates that the noise optimization quite efficient compared with the high-cost inference.
F Limitations and Societal Impacts
Linitations. Since NoOp is still in the framework of the diffusion classifier, there is inherited limitation: given one image and some category candidates, the inference of NoOp needs to input the image with different categories into the denoising network, respectively. This brings multiple forward propagation (the number of forward propagation equals the number category candidates) with high computation cost compared some VLM classifiers (e.g., CLIP only needs forward time in its visual encoder for each image).
Societal Impacts. On the positive side, few-shot learning technique, NoOp reduce models' reliance on large-scale data, accelerate innovation and technology inclusion, promote applications in various fields, and improve privacy protection and resource efficiency. However, it should be noted that there are potential risks exacerbating the digital divide and impacting the job market.
26


Reviews:
[{"review_id": "goI4NpSoNs", "reviewer": "Reviewer_Drnp", "summary": "This paper introduces a method that enhances diffusion-based open-set classification by learning image- or class-specific noise offsets from a support set via a Meta-Network. The approach modifies the original diffusion classifier framework by incorporating a training phase based on support examples. The method is evaluated on few-shot classification tasks, with comparisons to baseline models. Several aspects such as training cost, scalability with respect to the number of classes, and reliance on support data are identified as important factors for further analysis.", "questions": "1. The method involves learning noise offsets from a support set and training a Meta-Network. In scenarios with a large number of classes, is this approach still scalable? Specifically, can the method effectively learn discriminative noise signals when the class space is large, and does the training cost grow significantly with the number of classes? A discussion or experiment on scalability would be informative.\n2. Since the method learns noise from a small support set, is there a risk of distribution shift between the support and test data during inference, especially in open-world settings? It would be helpful if the authors could evaluate the method on open-set recognition or under domain shifts to better understand its generalization.\n3. The original Diffusion Classifier suffers from high computational cost and slow inference speed. Given that the proposed method introduces additional training (e.g., Meta-Network optimization), how does the overall training and inference cost compare with the original DC and other few-shot or zero-shot methods? Does the method improve or worsen the efficiency bottleneck of DC?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of optimizing noise for diffusion classifiers is novel and intuitive.  \n2. The method is simple and efficient, requiring only a lightweight Meta-Network and a noise offset, and compatible with existing diffusion backbones.  \n3. The experiments are comprehensive, covering multiple datasets and model variants.\n\nWeaknesses: \n1. Diffusion classifiers were originally designed for zero-shot classification, but the proposed method relies on few-shot training to optimize the noise. This compromises the zero-shot capability of DC and may limit its applicability in fully open-world or zero-data scenarios.\n2. Although the experiments are extensive, a cross-dataset evaluation would better demonstrate the generalization ability of the proposed method.\n3. There may be some format problems. Captions show below the figure, etc.", "was_revised": true, "final_justification": "My concerns are well addressed and I will keep the score.", "rebuttal": "Thanks for your valuable feedback on our work. We sincerely appreciate your recognition of our method as both novel and effective, as well as the comprehensive evaluations.\n\nWe hope to address all your concerns below.\n\n---\n\n## **Concern 1: Evaluation for open-world generalization**\n\nThank you for your valuable suggestions.\n\nIn response to your feedback, we evaluated our method in the cross-dataset transfer setting to demonstrate its generalization ability on open-set recognition. We optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested it on the source dataset and eight target datasets. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.\n\n|             |  &nbsp;&nbsp;Source |  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   |      |           |             |             |   &nbsp;&nbsp;&nbsp;Target       |       |         |         |         |         |         |\n| :------- |   :----:   | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n|            | ImageNet | |DTD | CIFAR10 | CIFAR100 | EuroSAT | Flowers102 | STL10 | OxfordPets | Aircraft | Average |\n| Ensemble (Time ×5) | 25.94 | | 17.34 | **69.91** | 17.96 | 28.37 | 17.47 | **75.44** | 42.41   | 9.42 | 34.79 | \n| Ours     | 26.34   | |**21.70**  | 63.26 | **29.36**   | **29.31**  | **29.48**    | 71.66 | **45.90** | **10.56**  | **37.65** | \n| Δ          | *+0.40* | | *+4.36*  | *-6.65*   |  *+11.4*    |  *+0.94*      | *+12.01* | *-3.78* | *+3.49* | *+1.14* | *+2.86* |\n\nAs we can see, the noise and Meta-Network learned on ImageNet can also be used in most of the target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average. \n\n**This demonstrated that the noise optimization can learn some generalized knowledge that is beneficial to the classification, i.e., how to destroy the target part of the image and better utilize the reconstruction capacity to achieve classification. To this end, this result indicates that our method can be further leveraged in open-world scenarios.**\n\n---\n\n## **Concern 2: Scalability to scenarios with a large number of classes**\n\nThank you for your insightful question. \n\nIn the original paper, we conducted experiments on eight datasets. However, the dataset with the largest class number is Flowers102 (102 classes). The reviewer suggested we verify whether our method can be scaled to scenarios with a large number of classes. Thus, we experimented on ImageNet (1000 classes, nearly 10 times more than Flowers102). We used the same setting as Section. 4.1 with Stable Diffusion-v2.0 and only replace the dataset with ImageNet.\n\n\n|       | Ensemble (Time ×5) | shot=0 | shot=1 | shot=2 | shot=4 | shot=8| shot=16 | \n| :---        |    :----:   |          :---: |         :---: |         :---: |         :---: |         :---: |         :---: |\n| Accuracy (%)   |   25.94    |   17.34    | 24.62 | 25.31 | 26.34 | 27.59 | 29.13 |\n\n**This result indicated that our method can be applied to scenarios with a large number of classes, even with only an optimized noise and a small Meta-Network.** Our method can outperform the ensemble baseline with only four shots, and the accuracy improves as the number of shots increases.\n\n---\n\n## **Concern 3: Computational cost analysis and comparisons**\n\nAs the reviewer mentioned, the original Diffusion Classifier suffers from high computational cost and slow inference speed, and whether our method can improve or worsen the efficiency. It is worth mentioning that **our method can significantly improve the overall efficiency**. For this concern, we split into two parts (formal cost analysis and experimental comparisons) to respond.\n\n### **Formal cost analysis** \nGiven a dataset $\\mathcal{D}$ with $K$ categories. We assume that the number of training samples for each category (i.e., the shot number) is $N$. The size of the test set is $M$ (i.e., $M$ test samples). \n\nIn that case,\n\nTraining Time $ \\propto KN$,\n\nTest Time $ \\propto KM$.\n\nUsually, $N=1,2,4,8,16$ for few-shot learning is significantly smaller than $M$. Thus, the training time of our method can be ignored compared with the test time. However, after this low-cost training, the sampling times of noises can be significantly decreased to improve the efficiency of the original Diffusion Classifier.\n\n### **Experimental comparisons**\n\nWe used ImageNet as an example to demonstrate how our method can improve the efficiency of the original Diffusion Classifier. The unit of time cost is one hour with 32 NVIDIA V100 GPUs. For the ensemble method, we randomly sample five different noises.\n\n| Method      | Training Time | Inference Time | Total Time | Accuracy (%) |\n| :---        |    :----:   |          :---: |          :---: |          :---: |\n| Ensemble (Time ×5)  | 0 | 153.0 | 153.0 | 25.94 | \n| Shot = 0   | 0 | 30.6 | 30.6 | 17.34 |  \n| Shot = 1   | 0.4 | 30.6 | 31.0 | 24.62 | \n| Shot = 2   | 0.8 | 30.6 | 31.4 | 25.31 |  \n| Shot = 4   | 1.6 | 30.6 | 32.2 | 26.34 |  \n| Shot = 8   | 3.2 | 30.6 | 33.8 | 27.59 |  \n| Shot = 16 | 6.4 | 30.6 | 37.0 | 29.13 |  \n\nAs we can see, our training time is quite smaller than the inference time. Even for the 16-shot training, the total time is only around 24% while the accuracy gains 3.19% compared with the 5-times ensembling method."}, {"review_id": "jvqxA8n79C", "reviewer": "Reviewer_CARn", "summary": "In order to circumvent the tradeoff between the unstable single-sample sampling and expensive multiple-sample ensembling for current Diffusion Classifier (DC) frameworks, this paper proposes a new noise optimization method named NoOp to predict the 'good noises' for classification. Such framework consists of two parts: (1) a dataset-specific learnable noise for frequency matching; and (2) a Meta-Network designed for predicting the image-specific noise offset for spatial matching. Given a pre-trained diffusion model and a K-way-N-shot image classification task, the learnable noise and the Meta-Network is jointly optimized using the cross-entropy loss with Z-score normalization on the training samples and then evaluated on the full test set.\nExtensive experiments conducted on 3 different diffusion models and 8 different classification datasets validate the effectiveness of this method.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The method is simple.\n2) The experiments are extensive.\n3) The presentation is coherent and the writing is easy to follow.\n\nWeaknesses:\n\n\n1) The detailed architecture design of the Meta-Network seems missing. Why using a U-Net architecture rather than using a simple CNN or ViT for such a 'light' Network? What is the parameter count and FLOPs?\n\n 2) The motivation behind the frequency matching noise for each dataset is a little confusing. From my perspective, it learns the joint noise bias for the whole dataset, which means it can be directly absorbed into the Meta Network as a final learnable element-wise bias parameter. Why separate it out and make the whole pipeline more complicated?\n\n3) I am also wondering how to apply such a framework to open-set classification problem. Or in other words, how to design the frequency matching term if I want to train one single classifier for all of the datasets included in this paper?", "was_revised": true, "final_justification": "Since most of my concerns are addressed by the authors, I decided to keep my score as 4 and am inclined to accept this paper.", "rebuttal": "We sincerely appreciate your constructive feedback, and thanks for finding that our method is simple and the paper is well-written.\n\nWe hope to address all your concerns below.\n\n---\n\n\n\n## **Concern 1: The detailed architecture design of the Meta-Network and the ablation study of the Meta-Network with various architectures**\n\nWe apologize for any confusion.\n\n### **The details of Meta-Network**\n\nThe Meta-Network is a U-Net architecture with 3 up-sampling layers and 3 down-sampling layers. Each sampling layer consists of 2-D convolutional layers with ReLu activation and BatchNorm (c.f., more details can be found in Lines 108-110 of the appendix and the source code). As the basic module of the U-Net is the convolutional layer, our Meta-Network can be considered as a simple CNN network.\n\nThe reason why we use a U-Net architecture is that the Meta-Network's input (image) and output (noise offset) share the same shape. For this kind of scenario, U-Net is a common architecture to maintain the shape.\n\n### **Additional Meta-Network Ablation**\n\n\nTo respond to your question about the ablation of various architectures, parameter count, and FLOPs. We compared different Meta-Network architectures (CNN and ViT) and reported the parameter count, FLOPs, and classification accuracy. For the CNN-based Meta-Network, we used our original Meta-Network in the paper since it consists of convolutional layers. The ViT-based Meta-Network consists of six ViTBlocks with layernorm and residual connection. We conducted the experiments on the 16-shot OxfordPet with Stable Diffusion-v2.0. Other settings are the same as Section. 4.1.\n\n\n| Architecture      | Params (M) | FLOPs (G)| Accuracy (%) |\n| :---        |    :----:   |          :---: |          :---: |\n| random noise    | - | - | 40.07 |\n| CNN    | 7.70 | 2.61 | 47.18 |\n| ViT   | 6.59 | 2.46 | 46.28 |  \n\nAs we can see, both CNN-based and ViT-based Meta-Networks can gain remarkable performance (6-7% accuracy improvement compared with random noise) with only 6-8 M parameters and less than 3 G FLOPs. This indicates that a light Meta-Network with common architectures can accurately learn the noise offset, demonstrating our method is robust and efficient.\n\n---\n\n## **Concern 2: Motivation behind the Frequency Matching noise for each dataset**\n\nWe apologize for any confusion and clarify our motivation for Frequency Matching noise below.\n\n### **Leveraging the prior that images within one dataset share a common frequency bias.**\n\nWe agree with the reviewer's perspective: \"Noise bias can be directly absorbed into the Meta Network as a final learnable element-wise bias parameter.\" However, we already have a prior that category-related signals of images within one dataset usually share a frequency range. Based on this prior, we can derive that there is a common noise bias for all images within this dataset. Then, we parametrized this common noise bias as the learnable noise to reduce the learning difficulty of the Meta-Network. In that case, the training can be more stable and converge to a better point compared with only training the Meta Network to learn element-wise bias. \n\n### **A high-level perspective**\n\nIn theory, the Meta-Network can satisfy both Frequency Matching and Spatial Matching. But it's more difficult to learn the dataset-specific bias and image-specific bias at the same time. If we add another component to learn the dataset-specific bias, the task will be easier for the Meta-Network (i.e., only focus on learning image-specific bias), making the training more stable.\n\n### **The experimental perspective**\n \nWe conducted the ablation study in Figure 6. We can see that jointly training a learnable noise (Frequency Matching) and a Meta-Network (Spatial Matching) is more stable and can achieve a better performance compared with only training a Meta-Network. This result can support the significance of optimizing a shared noise for the Frequency Matching.\n\n---\n\n## **Concern 3: Extending to open-set scenarios**\n\nThanks for your insightful question.\n\nTo respond to the reviewer's question about how to apply our method to open-set classification, we tested our method in the cross-dataset transfer setting. To be specific, we optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested on eight target datasets. This can be considered as an open-set problem since a lot of categories in the target datasets did not appear in the source dataset. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.\n\n|             |  &nbsp;&nbsp;Source |  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   |      |           |             |             |   &nbsp;&nbsp;&nbsp;Target       |       |         |         |         |         |         |\n| :------- |   :----:   | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n|            | ImageNet | |DTD | CIFAR10 | CIFAR100 | EuroSAT | Flowers102 | STL10 | OxfordPets | Aircraft | Average |\n| Ensemble (Time ×5) | 25.94 | | 17.34 | **69.91** | 17.96 | 28.37 | 17.47 | **75.44** | 42.41   | 9.42 | 34.79 | \n| Ours     | 26.34   | |**21.70**  | 63.26 | **29.36**   | **29.31**  | **29.48**    | 71.66 | **45.90** | **10.56**  | **37.65** | \n| Δ          | *+0.40* | | *+4.36*  | *-6.65*   |  *+11.4*    |  *+0.94*      | *+12.01* | *-3.78* | *+3.49* | *+1.14* | *+2.86* |\n\n\nFrom the results, the noise and Meta-Network learned on ImageNet can also be used in the eight target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average.\n\n**This demonstrated that the noise optimization can learn some generalized bias that is beneficial to the classification on one general source dataset and then generalize to different target datasets. To this end, this result indicates that our method is robust and can be applied in open-set generalization scenarios.**"}, {"review_id": "01EHisXQO7", "reviewer": "Reviewer_a2bg", "summary": "This paper tackles the noise instability problem in diffusion classifiers (DCs), where the randomness of Gaussian noise injected during the forward process leads to high variance in classification outcomes. To address this, the authors propose a noise optimization method (NoOp) that learns to selectively destroy category-relevant frequency components (frequency matching) and spatial regions (spatial matching). They introduce a learnable dataset-specific noise and a meta-network that predicts image-specific noise offsets. The sum of these two replaces the standard random noise in DCs. Through extensive experiments, the authors demonstrate that this optimized noise both stabilizes and improves classification performance, eliminating the need for costly multi-sample ensembling.", "questions": "Please see weaknesses", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written and easy to follow.\n- The authors provide comprehensive experiments on eight datasets and SD/ReFlow models. They also provide various ablation studies and compatibility with the prompt optimization method.\n\n**Weaknesses**\n- By using NoOp to produce a deterministic, optimized noise at a fixed timestep (e.g., $t=500$), the method intuitively resembles the effect of ensembling over multiple random noises. It would be better if the authors could provide a theoretical analysis to guarantee that this deterministic approach approximates the expectation of classification under noise sampling.\n\n- The reported performance in this paper appears significantly different from baselines [1], which raises concerns about the consistency and comparability of the experiments. Moreover, because NoOp is trained explicitly on a particular dataset and at a single timestep, its generalization ability is questionable, especially when compared to vision-language model (VLM) based classifiers that benefit from richer semantic priors and typically achieve stronger cross-dataset performance. By relying on a fixed, optimized noise, this approach also sacrifices the inherent generative and stochastic diversity of diffusion models, which further limits its broader appeal.\n\n[1] Clark et al., Text-to-Image Diffusion Models are Zero-Shot Classifiers, NeurIPS'23.", "was_revised": true, "final_justification": "The authors have attempted to address my concerns, but the theoretical justification remains insufficient, and the performance gains on VBench metrics are not adequately substantiated. Therefore, I will maintain my score as borderline reject.", "rebuttal": "We thank the reviewer for the constructive feedback! We hope to address concerns below.\n\n\n## **Concern 1: \"Theoretical analysis to guarantee that this deterministic approach approximates the expectation of classification under noise sampling\"**\n\nWe apologize for any confusion. \n\nHere we clarify that the Ensembling (expectation of classification under noise sampling) and our Noise Optimization improve the classification performance with **different mechanisms**. \n\n### **Why does Ensembling work?**\n\nLet the $m$ base predictors be\n\n$\\hat{h}_i(x)=f(x)+\\varepsilon_i(x), \\qquad i=1,\\dots,m$, \n\nwhere $f(x)$ is the true function and the noises $\\varepsilon_i(x)$ are i.i.d. such that\n\n$\\mathbb{E}[\\varepsilon_i(x)] = 0, \\qquad \\operatorname{Var}[\\varepsilon_i(x)] = \\sigma^{2}.$\n\nWe define their expectation (simple average)\n\n\n$\\bar{h}(x)=\\frac{1}{m}\\sum_{i=1}^{m}\\hat{h}_i(x).$\n\nThen we compare the mean squared error (MSE) of a single model with the ensemble model.\n\nFor bias,\n\n$\\mathbb{E}[\\hat{h}_i(x)] = f(x)\n\\quad\\text{and}\\quad\n\\mathbb{E}[\\bar{h}(x)] = f(x).$\n\nBoth the single model and the ensemble are unbiased.\n\nFor variance, \n\n$\n\\operatorname{Var}\\\\bigl[\\bar{h}(x)\\bigr]\n= \\operatorname{Var}\\\\left(\n      \\frac{1}{m}\\sum_{i=1}^{m}\\varepsilon_i(x)\n  \\right)\n= \\frac{1}{m^{2}}\\sum_{i=1}^{m}\\operatorname{Var}[\\varepsilon_i(x)]\n= \\frac{\\sigma^{2}}{m}.\n$\n\nWith squared‑error loss, $\\operatorname{MSE}=\\text{Bias}^2+\\text{Variance}$.\nBecause the biases are identical, the ensemble’s smaller variance yields\n\n\n$\n\\operatorname{MSE}\\bigl[\\bar h(x)\\bigr]\n  = \\frac{\\sigma^{2}}{m}\n  <  \\sigma^{2}\n  = \\operatorname{MSE}\\bigl[\\hat h_i(x)\\bigr], m \\ge 2.\n$\n\nHence, averaging on multiple sampled noises strictly decreases expected risk.\n\n### **Why does Noise Optimization work?**\n\nTo answer this question, we recall the mechanism of the Diffusion Classifier (DC). \n\nGiven an inference image, the DC firstly adds some noise to destroy some of the signals that are related to the category. Then, using different categories as the prompts to reconstruct the noisy image. The final prediction is the category that can reconstruct it the best. \n\nIn this pipeline, a very crucial part is adding noise because it decides which signals are removed by adding noise. The best situation is trying to remove the category-related signals and maintain other category-independent signals. In that case, the reconstruction difference of diffusion models under different prompts can be fully unleashed.\n\nThus, our method can **learn an optimized noise to target the category-related signals**. This kind of optimized noise is usually in the low-probability areas of the standard Gaussian distribution, and thus is very difficult to sample. In comparison, the randomly sampled noises are usually in the high-probability areas of the standard Gaussian distribution but lack the target-destroying capacity. \n\n\n### **Additional evaluation to justify why Noise Optimization works**\n\nHere we compared the random noise and the optimized noise and reported their mean, variance, and standard Gaussian distribution probability density (pdf).\n\n|       | mean | varience  | pdf (log) |\n| :---        |    :----:   |          :---: |         :---: |\n| random noise      | -0.0032       | 1.0021   | -23265 |\n| optimized noise  | -0.0043       | 1.0282      | -23479 |\n\n\nWe can see that, compared with the random sampled noise, the optimized noise's mean is farther away from $0$, variance is farther away from $1$, and the probability density is lower. This demonstrated that the **optimized noise is usually in the low-probability areas of the standard Gaussian distribution to target the category-related signals**. And this kind of noise is difficult to sample. Thus, we obtain it by optimization.\n\n**Thus, the optimized noise does not try to approximate the expectation of multiple randomly sampled noises. Instead, it improves the classification performance through a different mechanism. That is, it targets and destroys the category-related signals. **\n\n---\n\n## **Concern 2: Performance difference from baseline [1]**\n\nWe apologize for any confusion. \n\nHere, we explain why our reported performance is different from baseline [1].\n\nWe take CIFAR-10 with Stable Diffusion-v1.4 as an example. In our paper, we reported that the accuracy is 67.59% (c.r.f.,  the blue dashed line in Figure 4, CIFAR-10). However, in Table 1 of [1], the CIFAR-10 accuracy is 72.1%. \n\n**The reason for this inconsistency is that we only ensemble with five different noises while [1] uses more ensemble noises (such as 20+).** The specific number of ensemble scale for different datasets is dynamically decided by its pruning operation (c.r.f., Algorithm 1 in [1]). More noises are used for ensemble, resulting in higher inference cost and better performance. \n\n\nFor our paper, we focus on noise optimization. Thus, for the expensive ensembling method, we only use five noises for ensembling.\n\n---\n\n## **Concern 3: Evaluation for cross-dataset generalization**\n\nWe thank the reviewer for valuable questions. \n\nIn response to the question of whether our method can achieve similar cross-dataset generalization to vision-language models (VLMs), we evaluated our method for cross-dataset generalization settings.\n\nSpecifically, we optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested it on the source dataset and eight target datasets. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.\n\n|             |  &nbsp;&nbsp;Source |  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   |      |           |             |             |   &nbsp;&nbsp;&nbsp;Target       |       |         |         |         |         |         |\n| :------- |   :----:   | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n|            | ImageNet | |DTD | CIFAR10 | CIFAR100 | EuroSAT | Flowers102 | STL10 | OxfordPets | Aircraft | Average |\n| Ensemble (Time ×5) | 25.94 | | 17.34 | **69.91** | 17.96 | 28.37 | 17.47 | **75.44** | 42.41   | 9.42 | 34.79 | \n| Ours     | 26.34   | |**21.70**  | 63.26 | **29.36**   | **29.31**  | **29.48**    | 71.66 | **45.90** | **10.56**  | **37.65** | \n| Δ          | *+0.40* | | *+4.36*  | *-6.65*   |  *+11.4*    |  *+0.94*      | *+12.01* | *-3.78* | *+3.49* | *+1.14* | *+2.86* |\n\nAs we can see, the noise and Meta-Network learned on ImageNet can also be used in the eight target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average. \n\nThis demonstrated that the noise optimization can learn some generalized knowledge that is beneficial to the classification, i.e., how to destroy the target part of the image and better utilize the reconstruction capacity to achieve classification. **To this end, this result indicates that although NoOp was trained on a particular dataset and at a single timestep, it can also be generalized to other datasets, showing remarkable generalization ability.**\n\n---\n\n## **Concern 4: Sacrificing the stochastic diversity of diffusion models**\n\nWe agree with the reviewer's perspective, that is, the fixed and optimized noise will sacrifice the stochastic diversity of diffusion models. \n\nHowever, we clarify that diversity is not always a positive property across tasks.\n\n### **Diversity in generation task**\nFor the generation task, sampling different noises as the initial point can lead to diverse generation results. This satisfied the requirements of humans regarding to generation application.\n\n### **Diversity in classification task**\nFor the classification task, we use the Diffusion Classifier as an example. The stochastic diversity of different noise will lead to the Noise Instability problem (c.r.f., Figure 1 (b) in our paper). In that case, the stochastic diversity is negative for classifying an image.\n\n### **Ensembling the optimized noises across timesteps**\n\nAlthough our method optimizes to a fixed noise for a given timestep, it can still leverage the diverse knowledge across timesteps. To be specific, for different timesteps, we can optimize different noises. This enables our method to leverage knowledge of the diffusion model conditioned on different timesteps. \n\nWe also conducted experiments for ensembling our NoOp across different timesteps in the Section. A.3.3 of the appendix. The results indicated that NoOp can optimize different noises for different timesteps and ensemble the prediction to gain a further performance improvement.\n\n\n---\n\n### References:\n\n[1] Clark, Kevin, and Priyank Jaini. \"Text-to-image diffusion models are zero shot classifiers.\" Advances in Neural Information Processing Systems 36 (2023): 58921-58937."}, {"review_id": "mt7Jg9Adzv", "reviewer": "Reviewer_rqku", "summary": "This paper addresses the instability issue in Diffusion Classifiers, where classification outcomes are sensitive to random noise. The authors propose NoOp, a noise optimization strategy that replaces random sampling with learned “good noise” guided by two principles: frequency matching and spatial matching. A dataset-specific noise and an image-specific noise offset are jointly optimized to stabilize classification. Extensive experiments on multiple datasets show NoOp significantly improves few-shot performance and outperforms costly ensembling baselines. Additionally, NoOp is orthogonal to prompt tuning, allowing further gains. The approach is well-motivated, empirically validated, and shows promise for robust generative classification.", "questions": "- The manuscript introduces the principles of “frequency matching” and “spatial matching” as criteria for good noise in diffusion classifiers (DCs), but these are currently heuristically defined and empirically justified. There is no formal theory or precise quantification that explains why these principles are optimal for classification stability or generalization.\n- The manuscript compares NoOp mainly with random ensembling and prompt-tuning methods, but omits comparison with other known noise-control or augmentation strategies in generative models (e.g., golden noise, prior-guided sampling, rectified flows with learned priors).\n- Typo: line 175 cross-entropy.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The experimental section is thorough, covering multiple datasets, diffusion backbones, and ablation studies. The inclusion of both qualitative and quantitative validations of the proposed principles (frequency and spatial matching) is commendable.\n- The writing is clear and easy to follow.\n\n### Weaknesses\n- While the idea of optimizing noise is interesting, the method itself is relatively simplistic: it boils down to optimizing a learnable noise vector and a small Meta-Network. This lacks technical novelty or deeper theoretical insight. The two proposed principles (frequency and spatial matching) are intuitive but underdeveloped in formulation and justification.\n- The idea of optimizing noise in diffusion models has been explored in generative contexts[1]. The application to classification is somewhat novel, but not fundamentally different in methodology. The contribution feels incremental and may not be strong enough for acceptance at a top-tier venue like NeurIPS.\n\n\n[1] InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, CVPR 2024.", "was_revised": true, "final_justification": "Thanks for the authors' rebuttal. The authors have addressed my concerns and I decide to raise my score to 4.", "rebuttal": "Thanks for your thoughtful feedback. We sincerely appreciate your recognition that our experiment is thorough and the paper is well-written.\n\nWe hope to address all your concerns below.\n\n## **Concern 1: \"Technical novelty or deeper theoretical insight of simple method\"**\n\n**We agree that our method is simple. However, technical novelty is not the contribution that our paper intends to produce, and simplicity $\\neq$ no insight**.\n\nFirstly, we clarify that the main contribution of our paper is that we analyze the role of noise in the current diffusion classifier (DC). That is, the noise destroys some parts of the image, and DC tries to find the category that can best guide the diffusion model to reconstruct the destroyed parts. And this analysis of the noise role can exactly explain why there is a Noise Instability problem in DC. Then, based on the role analysis, we proposed the two principles (Frequency Matching and Spatial Matching) to guide the noise to destroy the image. **The above analysis of the noise role, the noise instability problem, and guidance for noise optimization constitutes the insights of our paper and derives the design of our method.**\n\nSecondly, although the framework of our method is simple, the results show its effectiveness with consistent improvements. **Simple but effective**, it exactly indicates that we catch the key research gap that exists in the Diffusion Classifier.\n\n\nThirdly, there are a lot of **advantages of the simple method**. For example, simple and efficient algorithms are easier to implement in large-scale industries and for subsequent research expansion.\n\n\nSimilarly, [1, 2, 3, 4, 5] are some related and representative works with simple and effective methods.\n\n\n---\n\n## **Concern 2: Formulation and justification for the two proposed principles (Frequency and Spatial Matching)**\n\nThanks for the reviewer's insightful feedback.\n\nFirstly, by analyzing the role of noise in the Diffusion Classifier, i.e., destroying the category-related signals, it is natural to target the specific range of frequency space and spatial space. For the frequency space, there are some works [6, 7] that have discussed the category-related signals are located in the dataset-specific range. **The Frequency Bias Theory [6, 7] can be a justification of our proposed Frequency-Matching principle (we will supplement it in the future version).** For the Spatial Matching, it's intuitive that even for images within one dataset, the category-related signals should be located in different areas. Thus, we proposed another Spatial Matching principle. These two principles guide us to design the optimization framework with a dataset-specific learnable noise and a Meta-Network to produce the image-specific noise bias.\n\nSecondly, as the reviewer mentioned that \"both qualitative and quantitative validations of the proposed principles (frequency and spatial matching) is commendable.\" These results can verify that **the optimization can actually align with these two principles** and provide some empirical justifications.\n\nFinally, we agree with the reviewer that a stricter formulation or justifications are required to prove that these two principles are optimal for classification stability and generalization. However, we do not overclaim that our proposed principles are optimal because the optimal principles for classification are a general and open question in this field. Instead, we claim that **they are two typical and intuitive principles that can guide an effective optimization framework design**.\n\n---\n\n## **Concern 3: Difference and contribution of our method compared with the current optimizing noise in the generation field**\n\nWe apologize for any confusion.\n\nAs we discussed in Lines 99-107 of our paper, the noise optimization and selection have been widely explored in the generation scope [4, 5, 8, 9, 10, 11, 12, 13]. However, in the classification domain, the motivation and optimization guidelines are significantly different from the generation domain. We clarify the differences and contributions of our work below.\n\n### **Different motivations** ###\n\nTo the best of our knowledge, we are the first paper to focus on the noise optimization for the discriminative task. The noise instability is a very severe problem that decreases the inference stability and efficiency in Diffusion Classifier and even more general diffusion-based perception scenarios. Our motivation is to reveal the research gap, and one intuitive and effective method to mitigate this problem is noise optimization.\n\n### **Different optimization guidelines** ###\n\nFor the generation domain, the guideline or the final goal of noise optimization is to control the generation process and get the desired synthetic images. Based on this guideline, the statistics-based, attention-based, and VLM-based criteria were proposed to guide the optimization and find a good starting point for the desired generation. In contrast, the guideline for our method is derived from the role of noise played in the Diffusion Classifier, i.e., destroying category-related signals of the image. We argue that a good noise for the Diffusion Classifier can destroy the target part of the images and then fully unleash the diffusion model's reconstruction difference under different category conditions. Based on this guideline, we propose two principles (Frequency Matching and Spatial Matching) to guide the corresponding noise optimization framework design.\n\n### **Technical gaps** ###\n\nThere are technical gaps for noise optimization between generation and classification. For example, leveraging Z-normalization to scale the logits difference and maintain the gradient during training (c.r.f., Lines 166-173).\n\n### **Our contribution** ###\n\nOur contribution is analysing the role of noise in the Diffusion Classifier and revealing the Noise Instability problem. To mitigate this problem, we proposed two principles to guide the noise optimization and derive the design an effective method.\n\n---\n\n## **Concern 4: Comparison with methods like noise-control or augmentation strategies in generative models**\n\nThe noise-control or augmentation strategies like golden noise [5] and rectified flows with learned priors [14] mainly focus on the optimization objective for generation. However, there is an **objective gap between generation and classification**. \n\nFor the generation task, the objective can be maximizing the preference score or minimizing the RFDS (Rectified Flow Distillation Sampling) loss [14]. However, the optimization objective of classification is to maximize the reconstruction difference under different category conditions. \n\nWith different optimization objectives, they can not be comparable. Moreover, apart from the optimization objectives, the noise optimization frameworks of [5, 14] are too simple (directly optimizing an NPNet or parameterized noise) to be compared as baselines because their main significant contributions are to find effective optimization objectives instead of optimization method.\n\n\n\n---\n\n### References:\n\n\n[1] Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models.\" International Journal of Computer Vision 130.9 (2022): 2337-2348.\n\n[2] Zhou, Kaiyang, et al. \"Conditional prompt learning for vision-language models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Ma, Nanye, et al. \"Inference-time scaling for diffusion models beyond scaling denoising steps.\" arXiv preprint arXiv:2501.09732 (2025).\n\n[4] Li, Shuangqi, et al. \"Enhancing compositional text-to-image generation with reliable random seeds.\" The Thirteenth International Conference on Learning Representations. 2024.\n\n[5] Zhou, Zikai, et al. \"Golden noise for diffusion models: A learning framework.\" arXiv preprint arXiv:2411.09502 (2024).\n\n[6] Lin, Zhiyu, Yifei Gao, and Jitao Sang. \"Investigating and explaining the frequency bias in image classification.\" arXiv preprint arXiv:2205.03154 (2022).\n\n[7] Wang, Shunxin, et al. \"What do neural networks learn in image classification? a frequency shortcut perspective.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[8] Guo, Xiefan, et al. \"Initno: Boosting text-to-image diffusion models via initial noise optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[9] Wang, Ruoyu, et al. \"The silent prompt: Initial noise as implicit guidance for goal-driven image generation.\" arXiv e-prints (2024): arXiv-2412.\n\n[10] Chen, Changgu, et al. \"Find: Fine-tuning initial noise distribution with policy optimization for diffusion models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[11] Miao, Boming, et al. \"Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[12] Qi, Zipeng, et al. \"Not all noises are created equally: Diffusion noise selection and optimization.\" arXiv preprint arXiv:2407.14041 (2024).\n\n[13] Samuel, Dvir, et al. \"Generating images of rare concepts using pre-trained diffusion models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.\n\n[14] Yang, Xiaofeng, et al. \"Text-to-image rectified flow as plug-and-play priors.\" arXiv preprint arXiv:2406.03293 (2024)."}]

Author rebuttal:

Meta-review: This paper presents a simple yet effective method to optimize the noise used to destroy the image in diffusion classifier, which tends to  be unstable and requires ensembling. The reviewers find this paper well-motivated [rqku], intuitive [Drnp], and easy to follow [CARn, a2bg, rqku]. They also appreciate the extensive [Drnp, CARn, a2bg, rqku] experiments to validate the proposed insights empirically. Reviewer a2bg expressed concerns on the theoretical justification and the complexity of the method. After considering all the reviews and rebuttal, AC decides that the merits / insights of this paper outweigh the concerns of reviewer a2bg and recommends acceptance.
Decision: Accept (poster)