Title: Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Classifier

Few-shot Classification

Noise Optimization

Prompt Optimization
Abstract: Although today's pretrained discriminative vision-language models (e.g., CLIP) have demonstrated strong perception abilities, such as zero-shot image classification, they also suffer from the bag-of-words problem and spurious bias. To mitigate these problems, some pioneering studies leverage powerful generative models (e.g., pretrained diffusion models) to realize generalizable image classification, dubbed Diffusion Classifier (DC). Specifically, by randomly sampling a Gaussian noise, DC utilizes the differences of denoising effects with different category conditions to classify categories. Unfortunately, an inherent and notorious weakness of existing DCs is noise instability: different random sampled noises lead to significant performance changes. To achieve stable classification performance, existing DCs always ensemble the results of hundreds of sampled noises, which significantly reduces the classification speed. To this end, we firstly explore the role of noise in DC, and conclude that: there are some ``good noises'' that can relieve the instability. Meanwhile, we argue that these good noises should meet two principles: 1) Frequency Matching: noise should destroy the specific frequency signals; 2) Spatial Matching: noise should destroy the specific spatial areas. Regarding both principles, we propose a novel Noise Optimization method to learn matching (i.e., good) noise for DCs: NoOp. For frequency matching, NoOp first optimizes a dataset-specific noise: Given a dataset and a timestep $t$, optimize one randomly initialized parameterized noise. For Spatial Matching, NoOp trains a Meta-Network that adopts an image as input and outputs image-specific noise offset. The sum of optimized noise and noise offset will be used in DC to replace random noise. Extensive ablations on various datasets demonstrated the effectiveness of NoOp. It is worth noting that our noise optimization is orthogonal to existing optimization methods (e.g., prompt tuning), our NoOP can even benefit from these methods to further boost performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the instability issue in Diffusion Classifiers, where classification outcomes are sensitive to random noise. The authors propose NoOp, a noise optimization strategy that replaces random sampling with learned “good noise” guided by two principles: frequency matching and spatial matching. A dataset-specific noise and an image-specific noise offset are jointly optimized to stabilize classification. Extensive experiments on multiple datasets show NoOp significantly improves few-shot performance and outperforms costly ensembling baselines. Additionally, NoOp is orthogonal to prompt tuning, allowing further gains. The approach is well-motivated, empirically validated, and shows promise for robust generative classification.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The experimental section is thorough, covering multiple datasets, diffusion backbones, and ablation studies. The inclusion of both qualitative and quantitative validations of the proposed principles (frequency and spatial matching) is commendable.\n- The writing is clear and easy to follow.\n\n### Weaknesses\n- While the idea of optimizing noise is interesting, the method itself is relatively simplistic: it boils down to optimizing a learnable noise vector and a small Meta-Network. This lacks technical novelty or deeper theoretical insight. The two proposed principles (frequency and spatial matching) are intuitive but underdeveloped in formulation and justification.\n- The idea of optimizing noise in diffusion models has been explored in generative contexts[1]. The application to classification is somewhat novel, but not fundamentally different in methodology. The contribution feels incremental and may not be strong enough for acceptance at a top-tier venue like NeurIPS.\n\n\n[1] InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, CVPR 2024.\n\nQUESTIONS:\n- The manuscript introduces the principles of “frequency matching” and “spatial matching” as criteria for good noise in diffusion classifiers (DCs), but these are currently heuristically defined and empirically justified. There is no formal theory or precise quantification that explains why these principles are optimal for classification stability or generalization.\n- The manuscript compares NoOp mainly with random ensembling and prompt-tuning methods, but omits comparison with other known noise-control or augmentation strategies in generative models (e.g., golden noise, prior-guided sampling, rectified flows with learned priors).\n- Typo: line 175 cross-entropy.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the noise instability problem in diffusion classifiers (DCs), where the randomness of Gaussian noise injected during the forward process leads to high variance in classification outcomes. To address this, the authors propose a noise optimization method (NoOp) that learns to selectively destroy category-relevant frequency components (frequency matching) and spatial regions (spatial matching). They introduce a learnable dataset-specific noise and a meta-network that predicts image-specific noise offsets. The sum of these two replaces the standard random noise in DCs. Through extensive experiments, the authors demonstrate that this optimized noise both stabilizes and improves classification performance, eliminating the need for costly multi-sample ensembling.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well written and easy to follow.\n- The authors provide comprehensive experiments on eight datasets and SD/ReFlow models. They also provide various ablation studies and compatibility with the prompt optimization method.\n\n**Weaknesses**\n- By using NoOp to produce a deterministic, optimized noise at a fixed timestep (e.g., $t=500$), the method intuitively resembles the effect of ensembling over multiple random noises. It would be better if the authors could provide a theoretical analysis to guarantee that this deterministic approach approximates the expectation of classification under noise sampling.\n\n- The reported performance in this paper appears significantly different from baselines [1], which raises concerns about the consistency and comparability of the experiments. Moreover, because NoOp is trained explicitly on a particular dataset and at a single timestep, its generalization ability is questionable, especially when compared to vision-language model (VLM) based classifiers that benefit from richer semantic priors and typically achieve stronger cross-dataset performance. By relying on a fixed, optimized noise, this approach also sacrifices the inherent generative and stochastic diversity of diffusion models, which further limits its broader appeal.\n\n[1] Clark et al., Text-to-Image Diffusion Models are Zero-Shot Classifiers, NeurIPS'23.\n\nQUESTIONS:\nPlease see weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to circumvent the tradeoff between the unstable single-sample sampling and expensive multiple-sample ensembling for current Diffusion Classifier (DC) frameworks, this paper proposes a new noise optimization method named NoOp to predict the 'good noises' for classification. Such framework consists of two parts: (1) a dataset-specific learnable noise for frequency matching; and (2) a Meta-Network designed for predicting the image-specific noise offset for spatial matching. Given a pre-trained diffusion model and a K-way-N-shot image classification task, the learnable noise and the Meta-Network is jointly optimized using the cross-entropy loss with Z-score normalization on the training samples and then evaluated on the full test set.\nExtensive experiments conducted on 3 different diffusion models and 8 different classification datasets validate the effectiveness of this method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The method is simple.\n2) The experiments are extensive.\n3) The presentation is coherent and the writing is easy to follow.\n\nWeaknesses:\n\n\n1) The detailed architecture design of the Meta-Network seems missing. Why using a U-Net architecture rather than using a simple CNN or ViT for such a 'light' Network? What is the parameter count and FLOPs?\n\n 2) The motivation behind the frequency matching noise for each dataset is a little confusing. From my perspective, it learns the joint noise bias for the whole dataset, which means it can be directly absorbed into the Meta Network as a final learnable element-wise bias parameter. Why separate it out and make the whole pipeline more complicated?\n\n3) I am also wondering how to apply such a framework to open-set classification problem. Or in other words, how to design the frequency matching term if I want to train one single classifier for all of the datasets included in this paper?\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method that enhances diffusion-based open-set classification by learning image- or class-specific noise offsets from a support set via a Meta-Network. The approach modifies the original diffusion classifier framework by incorporating a training phase based on support examples. The method is evaluated on few-shot classification tasks, with comparisons to baseline models. Several aspects such as training cost, scalability with respect to the number of classes, and reliance on support data are identified as important factors for further analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of optimizing noise for diffusion classifiers is novel and intuitive.  \n2. The method is simple and efficient, requiring only a lightweight Meta-Network and a noise offset, and compatible with existing diffusion backbones.  \n3. The experiments are comprehensive, covering multiple datasets and model variants.\n\nWeaknesses: \n1. Diffusion classifiers were originally designed for zero-shot classification, but the proposed method relies on few-shot training to optimize the noise. This compromises the zero-shot capability of DC and may limit its applicability in fully open-world or zero-data scenarios.\n2. Although the experiments are extensive, a cross-dataset evaluation would better demonstrate the generalization ability of the proposed method.\n3. There may be some format problems. Captions show below the figure, etc.\n\nQUESTIONS:\n1. The method involves learning noise offsets from a support set and training a Meta-Network. In scenarios with a large number of classes, is this approach still scalable? Specifically, can the method effectively learn discriminative noise signals when the class space is large, and does the training cost grow significantly with the number of classes? A discussion or experiment on scalability would be informative.\n2. Since the method learns noise from a small support set, is there a risk of distribution shift between the support and test data during inference, especially in open-world settings? It would be helpful if the authors could evaluate the method on open-set recognition or under domain shifts to better understand its generalization.\n3. The original Diffusion Classifier suffers from high computational cost and slow inference speed. Given that the proposed method introduces additional training (e.g., Meta-Network optimization), how does the overall training and inference cost compare with the original DC and other few-shot or zero-shot methods? Does the method improve or worsen the efficiency bottleneck of DC?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}