Generative Classifiers Avoid Shortcut Solutions
Authors: Alexander Cong Li
Ananya Kumar
Deepak Pathak
Venue: ICLR 2025 Poster (2025)
Keywords: distribution shift
shortcut
generative models
robustness
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that generative classifiers are more robust to realistic distribution shifts than discriminative classifiers.

Abstract: Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail even under minor distribution shift. This failure mode stems from an overreliance on features that are spuriously correlated with the label. We show that generative classifiers, which use class-conditional generative models, can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones. These generative classifiers are simple to train, avoiding the need for specialized augmentations, strong regularization, extra hyperparameters, or knowledge of the specific spurious correlations to avoid. We find that diffusion-based and autoregressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks and reduce the impact of spurious correlations in realistic applications, such as medical or satellite datasets. Finally, we carefully analyze a Gaussian toy setting to understand the inductive biases of generative classifiers, as well as the data properties that determine when generative classifiers outperform discriminative ones.

Published as a conference paper at ICLR 2025
GENERATIVE CLASSIFIERS AVOID SHORTCUT SOLUTIONS
Alexander C. Li
Ananya Kumar Stanford University ananyal @stanford. edu
Deepak Pathak
Camegie Mellon University al ellexaddrrl r 1.ccm.cc... edu
Carnegie Mellon University
braahak&ccs cmu edu
ABSTRACT
Discriminative approaches classification often learn shortcuts that hold in- distribution but fail even under minor distribution shift. This failure mode stems from overreliance on features that are spuriously correlated with the label. We show that generative classifiers, which use class-conditional generative models, can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones. These generative classifiers are simple train, avoiding the need for specialized augmentations, strong regularization, extra hyperparam- eters, knowledge of the specific spurious correlations to avoid. We find that diffusion-baate based and autoregressive generative classifiers achieve state-of-the-ae performance on five standard image and text distribution shift benchmarks and re- duce the impact of spurious correlations in realistic applications, such as medical or satellite datasets. Finally, we carefully analyze a Gaussian toy setting to under- stand the inductive biases of generative classifiers, as well the data properties that determine when generative classifiers outperform discriminative ones.
INTRODUCTION
Ever since AlexNet (Krizhevsky et al. 2012). classification with neural networks has mainly been tackled with discriminative methods, which train models to learn (y x). This approach has scaled well for in in-distribution performance (He et al., 2016; Dosovitskiy et al. 2020). but these methods are susceptible to shortcut learning (Geirhos et al., 2020), where they output solutions that work well on the training distribution but may not hold even under minor distribution shift. The brittleness of
these models has been dell-documented (Recht et al. 2019 Taori et al.. 2020), but beyond scaling up the diversity of the training data (Radford et al. 2021) so that everything becomes in-distribution, no approaches so far have made significant progress in addressing this problem.
In this paper. we examine whether this issue can be solved with an alternative approach, called generative classifiers (Ng & Jordan [001] Yuille & Kersten 2006] Zheng et al. 2023). This method trains class-conditionss
generative model to learn po y) and it uses Bayes' rule inference time to compute pe (y/ x) for classification. We hypothesize that generative classifiers may be better at avoiding shorteut solutions because their objective forces them to model the input in its entirety. This means that they cannot just learn spurious correlations the way that discriminative models tend to do; they must eventually model the core features as well. Furthermore, we hypothesize that generative classifiers may have an inductive bias towards using features that are consistently predictive, i.e., features that agree with the true label as often as possible. These are exactly the core features that models should learn in order to do well under distribution shift.
Generative classifiers date back at least as far back as Fischer discriminant analysis (Fisher 1936). Generative classifiers like Naive Bayes had well-documented learning advantages (gg & Jordan 2001) but were ultimately limited by the lack of good generative modeling techniques at the time.
Today, however, we have extremely powerful generative models (Rombach et al., [002]] Brown et al. 2020), and some work is beginning to revisit generative classifiers with these new models (Li et al. 2023 Clark & Jaini 2023). Li al. (2023) in particular find that ImageNet-trained diffusion models exhibit the first "effective robustness" TTaori et al.] 2020) without using extra data, which suggests that generative classifiers are have fundamentally different (and perhaps better) inductive biases. However, their analysis is limited to ImageNet distribution shifts and does not provide any understanding. Our paper focuses on carefully comparing deep generative classifiers against today's

---

Published as a conference paper at ICLR 2025
Diffusion-based
Autoreeressive
Out Estribution Accuracy
Diseriminatvve
Inbel
Men M)
plan (ava))
60
Transformer
40-
Diffusion Model
6
Hello
world
log pe zu (r.elllen(21, =
log Polar
log
In-distribution Accurccy
Figure I: Generative classifiers. We repurpose today's best generative modeling algorithms for classification. Generative classifiers predict arg max, (s v)p(y). We use diffusion- gener- ative classifiers image tasks and autoregressive generative classifiers text tasks, and find that they scale better -iiitrriiitiitttionn than discriminative approaches.
discriminative methods on a comprehensive set of distribution shift benchmarks. We additionally conduct thorough analysis of the reasons and settings where they work. We list our contributions:
Show significant advantages of generative classifiers realistic distribution shifts. Gen- crative classifiers are simple and effective compared to previous distribution shift mitigations. training
They utilize existing generative modeling pipelines, avoid additional hyperparameters stages, and do not require knowledge of the spurious correlations to avoid. We run experiments on standard distribution shift benchmarks across image and text domains and find that generative classifiers consistently do better under distribution shift than discriminative approaches. Most notably, they are the first algorithmic approach to demonstrate "effective robustness" (Taori al. 2020), where they do better -uittf-distributtin than expected based on their in-distribution performance (see Figure [] right). We also surprisingly find better in-distribution accuracy on most datasets, indicating that generative classifiers are also less susceptible to overfitting.
Understand why generative classifiers work. We carefully test several hypotheses for why generative classifiers do better. We conclude that the generative objective p(æ: y) provides more consistent learning signal by foreing the model to learn all features of x..
Provide insights from Gaussian data. We compare generative (linear discriminant analysis) simplified setting. We find the exis-
and discriminative (logistic) classification methods tence of "generalization phases" that show which kind of approach does better, depending on the strength of the spurious correlations and noisy features in the data. These phases shed light the inductive bias of generative classifiers towards low-variance features.
RELATED WORK
Learning in the presence of spurious features It is been well-known that deep networks trained with empirical risk minimization (ERM) have tendency rely on spurious correlations predict the label, such as the background in an image or the presence of certain words (Beery et al. 2018
Ribeiro et al. 2016 Geirhos et al., 2020) McCoy et al. 2019). Notably, overfitting to these shortcuts performance under distribution shift, since these spurious correlations may no longer be predictive (Hendrycks & Dietterich 2019 Rosenfeld et al. 2018 Taori et al. 2020). The performance ct al. 2017 Sagawa et al. 2019). and this imbalance is aggravated in highly overparametrized mod- els (Sagawa et al. 2020). Theoretical works attribute this problem to the inductive bias of classifiers trained with cross-entropy loss; these classifiers prefer to find max mar-gn solutions, and thus fit spu- rious features even when they are not fully predictive like the core feature (Nagarajan et al. 2020 Puli al. 2023). To address these failures discriminative models, people objectives that try to Lee et al.] 2023)
causes degradation
rare ("minority") groups particular tends sutter (Dixon et al. [2018] Zhao
balance learning across different groups (Sagawa et al., 2019; Setlur et al.] 2023
or add data augmentation to smooth out the spurious feature (Shen et al. 2022) However, these
methods still tend fail to capture the core feature and often lead to degradations in in-distribution performance. Some approaches focus on identifying the specific spurious features, annotating which examples contain them. and using that to rebalance the data (Wu et al.) 2023) Ghosh et al. [203] Kirichenko et al. (022). Unfortunately, these approaches require significant manual effort, are not scalable, and may not work for problems where humans do not understand the learned features.
2

---

Published as a conference paper at ICLR 2025
Ideally, we find an approach with the right inductive bias to generalize well under distribution shift without requiring extra supervision.
Classification with Generative Models Few deep learning approaches have trained class- conditional generative models and used them directly for classification, perhaps due to the difficult task of modeling p( y) with weaker generative models. However, recent generative models have significantly improved, especially with better techniques in diffusion probabilistic models (Sohl-) Dickstein et al. 2015; Ho et al., 2020). and deep generative classification methods have recently been proposed Li et al. 2023: Clark & Jaini 2023). Li et al. (2023) showed that ImageNet-trainees class-conditional diffusion models are competitive with discriminative classifiers and achieve the first nontrivial "effective robustness" (Taori et al., 2020) on ImageNet-A (Hendrycks et al.) 2021) without using extra data. Prabhudesai et al. (2023) show that a hybrid generatie-diieriiinne
clas- sifier can test-time adaptation improve performance on several synthetic corruptions. Other work (Clark & Jaini [023] Jaini et al.) 2023) has shown that large pretrained generative models are more biased towards shape features and more robust to synthetic corruptions, but this may be due to effect of pretraining on extra data, or the fact that diffusion specifically confers resilience to input perturbations. Other works have found that generative classifiers can improve adversarial robustness (Grathwohl et al. 2020 Zimmermann al., 2021; Chen et al. 2023 2024a). However, adversarial robustness has been shown to not translate to robustness to distribution shift (Santurkar et al.) 2020 Taori et al. 2020). Overall, it still remains unclear whether generative classifiers are more robust to the spurious correlations seen in realistic distribution shifts or why they might be better.
3 PRELIMINARIES
3.1 TYPES OF DISTRIBUTION SHIFT
We consider classification under two types of distribution shift. In subpopulation shift, there are high-level spurious features that are correlated with the label. For example, on CelebA (Liu et al.) 2015), where the task to predict whether a person's hair is blond or not blond, the spuriously correlated feature is the gender. This occurs because there are very few blond men in the dataset, so models typically learn to the "man" feature. The spurious feature determines groups: the majority group contains examples where the spurious feature is correct, and the minority group contains examples where the spurious feature is incorrect. We also consider domain shift, where the test domain's data distribution is similar the training domain's 's distribution. For example, training images in CamelyonI 1 (Koh et al. (221)) come from 3 hospitals, whereas the test images come from a disjoint 4th hospital. Spurious features that worked on the training distribution, egg., artifacts of the way slide staining or sample collection was done, may hurt accuracy under distribution shift (Veta et al. 2016; Komura & Ishikawal 2018 Tellez et al., 2019). We examine 5 common distribution shift benchmarks in total: besides CelebA and Camelyon, we use Waterbirds (Sagawa et al.] (2019); subpopulation shift), FMoW (Koh et al. (2021) 1); both subpopulation and domain shift), and CivilComments (Koh et al. (202I) subpopulation shift).
3.2 SHORTCOMINGS OF DISCRIMINATIVE CLASSIFIERS
Discriminative classifiers, which seek maximize pa x), can overly rely on the spurious features and fall victim to shortcut solutions (Geirhos et al. 2020). This is because they can use the spuriously correlated features to correctly and confidently fit the majority group examples. After this happens, the loss on these examples flattens out, and there is less gradient signal available to encourage the model to use core features (Li et al. 2019 Pezeshki et al. 2021). The model then overfits to the remaining minority examples where the spurious correlation does not help (Sagawa Nagarajan et al. 2020). These shortcut solutions often work in-distribution but can fail, sometimes catastrophically. under even minor distribution shift. Significant effort has been put into preventing this, mainly by rebalancing the data so that the spurious correlation no longer holds (Sagawa et al,, 2019 Kirichenko et al. 2022 Liu et al., 2021 (eetlur et al.) 2023). However, these methods all add additional hy hyperparameters and complexity the training process, and often require knowledge of the exact distribution shift to counteract, which is impractical for realistic problems where there may be many spurious correlations.
al., 2020
3

---

Published as a conference paper at ICLR 2025
4 GENERATIVE CLASSIFIERS
We now present generative classifiers, a simple paradigm for classification with class-conditional generative models. To classify input z, generative classifiers first compute pe (aly) with class- conditional generative model and then utilize Bayes' theorem to obtain po (o(y/r). This paradigm had been popular in machine learning with methods like linear discriminant analysis and Naive Bayes (Ng & Jordan 2001), but has fallen out of favor in the modern era of deep learning. We revisit this paradigm with deep learning architectures and show its advantages for robustness to distribution Algorithm [ gives overview of the generative classification procedure.
shift in Section 5
4.1
INTUITION
Why could generative classifiers do better on these distribution shifts? In contrast to discriminative classifiers, which can minimize their training objective using just few spurious features, generative classifiers need to model the entire input æ. This means that they cannot stop at just the spurious features; their training objective requires them to learn both core and spurious features. This should translate to better training signal throughout the course of the training. We confirm this in Sec- tion 5.3 Note that learning both types of features does not mean that it uses them equally when classifying an input. The generative classifier should learn which type of features are more consis- tently correlated with the label and weight them accordingly. Section 6.3 and 6.4 demonstrates this inductive bias in simple setting with Gaussian data.
4.2 DIFFUSION-BASED GENERATIVE CLASSIFIER
For image classification, we use diffusion models (Sohl-Dickstein et al. 2015 Ho et al.) 2020).
which are currently the slate-of-the-aa approach for image modeling. Diffusion models are trained to iteratively denoise an image and do not have an exact likelihood that can be computed a single forward pass. They are typically trained with a reweighted variational lower bound of log pa (zly). To use them in generative classification framework, we use that value approximate log po y):
logpo(s) v) z E.,,, (1l v) - <[13]
Training the class- diffusion models is done normal we use off thfelf training pipelines to train diffusion models from scratch, without modifying any hyperparameters. At in- ference time, we follow the Diffusion Classifier algorithm from (Li et al., (023), which samples multiple noises e, adds them to the image to obtain noised x, = Va.z + vI Ctgt, and does multi- ple forward passes through the network to obtain a Monte Carlo estimate of Eq. [ This is done for each class, and the class with the highest conditional likelihood log pa(x) y). which corresponds to the lowest denoising error, is chosen.
4.3 AUTOREGRESSIVE GENERATIVE CLASSIFIER
For text classification, we introduce generative classifiers built on autoregressive Transformer mod- els, as they are the dominant architecture for text modeling. Since we need to now learn po (z y), where x is sequence of text tokens and y is label, we make small modification to the training procedure. Instead of starting each sequence of text tokens with a "beginning of sequence" (BOS) token, we allocate C' special class tokens in our vocabulary, one per class, and replace BOS with the desired class token. Obtaining logpo(a y) can be done in a single forward pass:
n poss mec.))) =
n
log v) = log
Σ
log po Teis v)
(2)
We train our Transformer as usual using cross-entropy loss over the entire sequence, with the ground truth label y² at the beginning. To classify a text sequence at inference time, we do C' forward passes, one with each possible class token. We then choose the class token with the lowest cross-entropy loss over the entire sequence as our prediction. Figure (middle) shows a diagram of this method.
Overall, generative classifiers can be easily trained using existing generative modeling pipelines and do not require any specialized architectures, extra hyperparameters, data augmentation. multi-stage training. or knowledge of the specific shortcuts to avoid. Training is also cheap: all of our generative models can be trained on single GPU in 2-3 days.
4

---

Published as a conference paper at ICLR 2025
Method
Waterbirds
ColobA
Camelyon
FMoW
CivilComments
ID
WG
ID
WG
ID
OOD
ID
OOD WG
ID
WG
ERM
88.8
32.2
92.4
50.5
95.2
78.3
51.1
27.5
90.6
53.3
LIF (aa al. 2020
86.4
28.9
90.8
34.0
90.5
66.3
49.6
31.0
87.9
49.4
JTT (Liu et 2021
88.1
32.9
91.9
42.1
88.1
65.8
52.1
31.8
89.2
55.6
RWY/DFR
90.8
31.6
94.1
68.9
95.2
78.3
39.3
26.1
90.1
58.1
Generative (ours)
96.8
79.4
91.2
69.4
98.3
90.8
62.8
35.8
79.8
61.4
Table I: Accuracy on distribution shift benchmarks. We show in-distribution (ID) and either gorst-group (WG) ditoof-distribution (OOD) accuracy, depending on the type of shift in each dataset. Our generative approach performs the best on all five distribution shifts and ID datasets.
EXPERIMENTS
We compare our generative classification approach to discriminative methods that com- monly used today. We aim to answer the following questions this section. First, generative classifiers have better robustness to distribution shift? If so, why are they more robust than discrim- inative methods? We multiple hypotheses determine which explanation correct.
5.1 SETUP
Benchmarks We use five standard benchmarks for distribution shift. Camelyon undergoes domain shift, so we report its OOD accuracy the test data. Waterbirds, CelebA, and CivilComments undergo subpopulation shift, so we report worst group accuracy. FMoW has both subpopulation shift over regions and a domain shift across time, so we report OOD worst group accuracy. The first four image benchmarks, while CivilComments text classification. Waterbirds and CelebA are natural images, whereas Camelyon contains whole-slide images of cells and FMoW contains satellite images. In total, these benchmarks cover multiple shift types, modalities, and styles.
Model Selection We believe that it is unrealistic or impractical to know the exact distribution shift that will happen on the test set. Thus, we do not use knowledge of the spurious correlation or distribution shift when training or performing model selection, and instead tune hyperparameters and perform early stopping the in-distribution validation accuracy, not the worst-group accuracy. This is the most valuable setting, as it matches how models are often deployed in practice, and thus is popular experimental setting for evaluating methods (Koh et al. 202 Yang et al. 2023 Liu al. [202T] Setlur et al., 2023). We use class-bale balanced accuracy for model selection as it uniformly improves performance on each dataset for all methods (Idrissi et al. 2022).
Baselines We compare generative classifiers against several discriminative baselines. ERM min- imizes the average cross-entropy loss of the training and is the standard method for training classifiers. We additionally evaluate several methods designed to combat spurious features. Learn- ing from Failure (L.FF) (Nam et al., 2020) simultaneously trains one network to be biased and uses identify samples that a second network should focus on. Just Train Twice (JTT) (Liu et al., 2021) is similar two-stage method that first trains a standard ERM model for several epochs, and then heuristically identifies worst-group examples as training points with high loss under the first model. JTT then upsamples these points and trains a second classifier. DFR (Kirichenko et al. 2022) fine-tunes model on training set that has been carefully balanced to make the spurious feature unpredictive. As in prior work (Yang et al. 2023), DFR samples data from each class equally when there are no spurious feature annotations. This is equivalent to RWY (Idrissi et al., 2022) and can help if there is class imbalance related the spurious correlation. For fairness, we train all models, generative and discriminative, from scratch to eliminate the effect of differing pre-training datasets.
Models For image-based tasks, all discriminative baselines use ResNer-50, ResNet-101, and ResNet- 152, whereas our generative classifier approach trains a class-conditional U-Net-based la- tent diffusion model (Rombach al.) (2022). For text-based tasks, all discriminative baselines use an encoder-only Transformer, whereas our generative classifier approach trains a Llama-style au- toregressive language model (Touvron et al. 2023) from scratch. See Appendix B for more details.
5

---

Published as a conference paper at ICLR 2025
Waterbirds
CelebA
Camelyon17
Discriminative Generative
90
OOD Test WG Accuracyy
OOD Test Accuracy
60
0OD Test WG
50
40
30
85.0
90.0
95.0
82.5
85.0
90.0
92.5
85
95
100
ID Class balanced Test Accuracy
ID Class balanced Test Accuracy
Val
FMoW
CivilComments
OOD Test WG Accuracy
30
55
WG
10:
45
20
40
60
70
80
90
Accuracy
Accuracy
Figure 2: In-distribution out-of-distribution accuracy for each dataset. Each point corre- sponds to a separate training run, other than the diffusion-based generative classifier results, which are checkpoints of run with default training hyperparameters. We observe better OOD scaling trends (i.e., effective robustness) for generative classifiers on CelebA, CivilComments, and poten- tially Camelyon17, although results are noisy for this dataset (the red line in Camelyon 17 denotes linear fit for the relationship between ID and OOD accuracy for discriminative models). On the remaining two datasets, they follow the same trend and do better both ID and OOD.
5.2 RESULTS ON DISTRIBUTION SHIFT BENCHMARKS
Main Results Table compares generative classifiers against discriminative baselines the dis- tribution shift benchmarks. Compared to the discriminative baselines, generative classifiers have better worst-group OOD accuracy all five datasets. Surprisingly, generative classifiers also achieve significantly better in-distribution accuracy on three of the five datasets, which indicates less overfitting. These results suggest that generative classifiers may have an advantage in both (a) learn- ing core features that generalize across distribution shifts, and (b) learning features that generalize from the training set to the ID test set.
Accuracy above the line Comparing the best generative classifier against the best discriminative classifier provides a one-dimensional understanding of each approach. To provide a better sense of which method may scale better in the future, Figure 2 plots the in-distribution and out-of out-of- distribution accuracies of each family of methods. We can classify the benchmarks into two sets:
1. Generative classifiers better both ID and OOD, and lay the same trend line discrimina- tive models. This includes Waterbirds and FMoW.
2. Generative classifiers have significantly better OOD performance trend, but the worse in-distribution. This includes CelebA, CivilComments, and potentially Camelyon.
The second where generative classifiers have better OOD accuracy than discriminative clas- sifiers at any ID accuracy, demonstrates "effective robustness" (Taori al.) 2020). This suggests fundamentally better
behavior for generative classifiers in some scenarios and indicates that they may be the right approach to classification after further scaling. Our results cor- roborate findings from Li et al. (2023). which found some signs of effective robustness on ImageNet. Section examines toy setting and provides insights into the cause of this "effective robustness."

---

Published as a conference paper at ICLR 2025
Normalzzd Gradicat Magnitude
Disc, maj
OOD Test WG Aecuracy
30
ResNer-50
Train Objective
ID
WG
Disc, min
-3
ResNett-101
Gen, mas
ply x)
91.4
35.7
Gen, min
Res
p(z) and ply 2)) p(a y) (ours)
91.7
35.4
79.8
61.4
10
Table 2: Alternative training objectives for autoregres- sive model on CivilComments. ply x) is standard discrim- inative approach with cross- entropy loss, and "p(y x) and p(æ))" tests if adding an unconditional generative mod- eling improves performance.
75
80
85
90
Epoch
ID Test Accuracy
Figure 3:
Gradient norms.
Figure Scaling disc. model size does not improve accuracy Waterbirds. This shows that model size is not confounder in our experiments.
Gradient for disc. model rapidly decays 0, its learning sig- nal is reduced, while it does not
decay for generative classifier.
5.3
WHY Do GENERATIVE CLASSIFIERS Do BETTER?
We test several hypotheses for how generative classifiers outperform the discriminative baselines.
Learning More from Majority Examples Our intuition that the generative objective log Pv (2 v) provides more consistent learning signal across epochs. In contrast, discriminative models may use spurious features to make confident and correct predictions on the training set and lose the gradi- ent signal necessary to use the core features. We test this by measuring the gradient norm on majority and minority examples across epochs. We compute the per-example gradient norm || [o y:) and average it over the majority and minority groups. We normalize this by the average majority group gradient norm at epoch 5 in order to fairly compare different architectures that have different loss landscapes. Figure 3 shows these metrics on CivilComments with toxic comments about the Black demographic the minority group. For the discriminative model, the majority group gradient quickly vanishes, and the minority group gradient starts high but eventually decays. The generative classifier, however, has very similar gradient norm across the majority and minority groups, and the gradient norm actually slightly increases over training. These results support our intuition that the generative objective helps the model learn more from examples with and without the spurious features. Note: the per-example gradient norm for generative classifiers does not go to O, since there is always a way to increase the likelihood of a single data point.
Does an unconditional objective p(x) improve discriminative performance? One hypothesis is that the generative classification objective pa y) teaches the model better features in general, similar to how generative pre-training methods (Devlin et a 20188] He al., (022) learn features that are useful for fine-tuning, We test this on CivilComments, as the architecture makes it simple add an unconditional generative objective p(x). Instead of placing the class-specific token the beginning of the sequence, we place at the end. Predicting the text tokens of x now corresponds to predicting p(x), and predicting the class-specific token at the end corresponds to p(u x). Table 2 shows that adding the objective p(x)) does not affect performance, so we reject this hypothesis.
Model Size In our image classification experiments, our generative classifier used a standard 395M parameter UNet (Rombach et al. 2022), which is far more than the 26M parameters in its ResNet-50 (He et al. 2016) discriminative counterpart. Could the greater parameter count could be responsible for the difference in performance and OOD behavior? We first note that the discrimi- native classifier used for CivilComments in Table I contains 67M parameters, which is more than the 42M parameters we use in our autoregressive generative classifier. Furthermore, the architec- tures and parameter counts of the discriminative p(y I x)) and generative p(x y) classifiers exactly matched in Table 2 On image tasks, we test whether parameter count matters by scaling from ResNet-50 up to Res.ett-15 (He et al. (016).. We perform sweep over model size and train- ing hyperparameterss (learning rate and weight decay). Figure [4 and Figure g show that increasing discriminative model size does not improve performance. This aligns with previous work: Sagawa et al. (2020) found that increasing discriminative model size can actually hurt OOD performance.
7

---

Published as a conference paper at ICLR 2025
6 ILLUSTRATIVE SETTING
We explore a simplified Gaussian data setting and find that linear generative classifiers can also display better robustness distribution shift compared to their discriminative counterparts. We rigorously explore this behavior in order to understand the inductive bias of generative classifiers. Finally, we connect findings back to practice and explain the varying empirical behavior for generative vs discriminative classifiers.
6.1 DATA
Consider binary classification with label y E (-1,+1). The fea- tures (Zcores Papu: Unoise
Rd where:
Spurious feature
Zcore y = Ny, E
(3)
y= +1
sspu y yB w.p. P. else
- yB E R
(4)
y y= y=-1
Tpoise
y = NNO, miise) Rd-2
(5)
We set the spurious correlation ratio p = 0.9 and core feature stan- dard deviation ø 0.15, which is small enough that the data can be perfectly classified by using only the core feature Zeore and ig- noring the remaining features. Figure[3]shove a visualization of the and spurious features. The majority groups consist of samples where the spurious and core features agree (top right and bottom left of Fig. 5). and the minority groups consist of samples where the spurious core features disagree (top left and bottom right).
Core feature
Figure 5: Visualization fea- tures (noise dims not shown).
This synthetic dataset has previously been used to understand the failure modes of discriminative classifiers in previous work (Sagawa et al.) 2020 [drissi et al., [0222; Setlur et al., 2023 and is natural simplified setting for us to study the advantages of generative classifiers.
6.2 ALGORITHMS
Discriminative We analyze unregularized logistic regression, done previous work (Sagawa et al., 2020; Nagarajan et al. 2020). Since the data is linearly separable, logistic regression learns the max-margin solution when trained via gradient descent (Soudry al., 2018).
Generative We use linear discriminant analysis (LDA), a classic generative classification method that models each class a multivariate Gaussian. It fits separate class means μ111 and p441 but leamns shared covariance matrix Σ for both classes. Assuming balanced classes, LDA predicts:
p p( p( p(r y
p(r
sign log N( M41,
arg max p(u v) = sign log
(6)
N(x 1-1
This corresponds to a linear decision boundary with coefficients WLDA = E-1 1-1). We chose LDA because it has the least inductive bias among common linear generative classifiers (e.g.. is equivariant to rotations). For this reason, we rejected methods like Naive Bayes, which learns an axis-aligned generative model, even though theoretical analysis would have been easier.
6.3 THE INDUCTIVE BIAS OF LDA
We carefully examine setting where the generative approach has the same In-distribution perfor- mance, but it outperforms the discriminative approach the worst group (OOD). Figure the behavior of LDA and logistic regression on toy data with data dimension d = 1026 and noise variance o2 nogise 0.36. We find that both methods have similar in-distribution accuracies. but LDA does significantly better on the minority group. fact, Figure 6 (middle) shows that LDA has essentially no performance gap between the majority and minority groups, which indicates that does not use the spurious feature at all. In contrast, logistic regression has a large performance gap between the groups. This can be explained by looking at the linear coefficients learned by both methods. Figure 6 (right) shows the ratio [Wspu /eore! between the weights the shorteut and
8

---

Published as a conference paper at ICLR 2025
Acc Gap (majority -minorit))
1.0
Normalized Weight Norm
[Waul
0.3
4
Disc
Test Accuracy
Disc Iwooselk
0.8
0.2
Gen
Gen ID>
Disc
2
Gen [Weell
Werre
Gen WG
0.1
Gen [Wowsoll
0.6
Disc ID
[Word
Disc WG
0.0
16
64 256
1024 4096
16
64 256
1024 4096
16
64 256 1024 4096
Number of training points
Number of training points
Number of training points
Figure 6: Illustrative setting for shorteut learning, Left: in in-istribution accuracies are roughly the same between generative (LDA) and discriminative (logistic regression) methods, but LDA achieves much higher minority group accuracy. Middle: the difference between the majority and minority test accuracies as function of the number of training examples. The generative method displays better robustness to the spurious correlation. Right: spurious feature weights |appal and noise feature weights |[nnoise 22, normalized by the magnitude of the core feature weight Wzore. LDA puts much less weight on the spurious feature, even with very little training data. Logistic regression puts more weight on the noise, and only achieves good in-distribution accuracy by using the spurious feature. Shading denotes +1 standard deviation over 25 seeds.
core features. Ideally, this ratio goes to 0 as fast as possible as the model sees more data. Logis- tie regression, however, places significant weight on the spurious feature until it gets thousands of training examples. LDA is far more data-efficient and places almost weight the spurious fea- ture with as few as 16 training examples. Interestingly, logistic regression puts more weight on the noisy features than LDA does. It is only by putting significant weight on the spurious feature that it achieves good in-distribution performance, though this hurts worst group accuracy.
These differences in behavior indicate that LDA has a significantly different inductive bias. We suspect that the most important factor is the core feature variance o². We increased from 0.15 to 0.6 and reran the same analysis. Figure 14 shows that LDA now consistently underperforms logistic regression, both in-distribution and out-of-distributiin. Why is this? Intuitively, when the learned probability (x;;; y²") of feature x; is low (i.e., the feature is not consistently correlated with the label) compared to other features, the generative classifier downweights this feature in its prediction. This helps improve robustness to distribution shift, since, by definition, we believe that the core feature Ecore should be the most consistently predictive.
6.4 GENERALIZATION PHASE DIAGRAMS
The feature dimension d, spurious feature scale B, noisy feature variance a Jiisse and core feature variance x² influence the solutions that models prefer to learn. Intuitively, the spurious feature scale controls the saliency of the shortcut feature, and larger makes it easier for the model to learn this shortcut. The noisy feature variance oo controls how easy it is for a model to overfit to training examples (Nagarajan et al. 2020). The core feature variance o² controls how consistently the core feature predicts the label. Varying these properties of the data creates family of datasets, and use them to understand when generative classifiers outperform their discriminative counterparts.
Each plot in Figure 7 varies B and dnoise for given number of training examples n 32 and = 1024. Each successive subplot corresponds to increasingly larger variance o² in the core feature, and each plot is divided into regions depending on which method does better ID or OOD for the given (B, (oose) at that location. We call this generalization phase diagram, since resembles phase diagram which shows the impact of pressure and temperature on the physical state of substance. our case, there are four possible generalization phases:
1. The generative classifier is better both ID and OOD. This typically happens at high a2 noise> since the discriminative model overfits using the noise features.
2. The discriminative classifier is better both ID and OOD. This happens at low o2 moise-

---

Published as a conference paper at ICLR 2025
n 32, 0 0.05
n = 32, 0 = 0.15
n= 32, 0= 0.3
0.64
0.64
0.64
Noisy Noisy feature variance
Noivy featuree variance
Noisy feature variance
and OOD
0.35
0.35
0.35
LDA better and OOD
Disc better ID,
Dise better ID, Gon better OOD
0.15
0.15
Gen hetter 001>
0.15
0.04
0.04
0.04
SVM better ID and OOD
Disc better ID and OOD
Disc better ID and OOD
0.00
0.00
0.00
0.00
0.73
1.47 2.20
3.00
0.00
0.73
1.47 2.20
3.00
0.00
0.73
1.47 2.20 3.00
Spurious feuture scale
Spurious feature
Spurious feature scale
Figure 7: Generalization phase diagrams. We vary the scale B of the spurious feature and the variance moise of the noise features and evaluate their effect on the ID and OOD test accuracy of generative classifiers (LDA) vs discriminative classifiers (logistic regression). Each plot corresponds to different variance o² of the feature, and the color of each pixel denotes which classifier does better for a particular combination of B and o² mnise We observe three main phases of generalization: (1) discriminative has better ID and OOD accuracy, (2) generative has better ID and OOD accuracy, and (3) discriminative does better ID and generative does better OOD. Each plot corresponds different standard deviation of the core feature. As increases, the core feature becomes less reliable, and the generative classifier uses the spurious and noise features more. This shows the inductive bias generative classifiers: they prefer consistently predictive features.
3. The discriminative classifier is better ID, but the generative classifier is better OOD. This phase occurs sweet spot of B and o noise- Moderate noise allows some overfitting. but the spurious feature strong enough for the discriminative model to achieve good ID ac- curacy. However, its heavy reliance on the spurious feature reduces its OOD performance.
4. The generative classifier is better ID, but the discriminative classifier is better OOD. This is exceedingly rare (see the dark, unlabeled regions in Figure 7
Notably, there no free lunch. Even in this setting, neither generative nor discriminative classifiers are uniformly better than the other. However, we do note that B and o noise are unbounded above, and generative classifiers should do comparatively better as the strength of shorteuts noise increases.
Finally, while it is hard to map and o directly onto a realistic image or text dataset, they do offer insights on important properties of the data that determine which method is suitable for a given task. Indeed, can categorize the distribution shift benchmarks into these phases based on their generative vs discriminative behavior. Waterbirds and FMoW fall in phase (generative better ID and OOD), CelebA and CivilComments fall in phase (diseriminative better ID and generative better OOD), and Camelyon lies on the transition boundary between phase and 3, since the generative classifier achieves better OOD and similar ID accuracy compared to discriminative basclines.
7 CONCLUSION
Discriminative approaches classification have dominated the field since AlexNet catalyzed the widespread adoption of deep learning. Despite their prevalence, these methods face increasing lim- itations, including vulnerability to distribution shift and escalating data requirements. In this paper, we present a simple alternative. We revisit the paradigm of generative classifiers and show that they realistic
have significant advantages in both mn-disrriution and out-of-distribution performance distribution shift benchmarks. We carefully analyze their behavior, and finally show insights from a simplified, illustrative setting into when generative classifiers can be expected to do better.
As deep generative classifiers have not been well-explored, there is significant room for future work. The inference cost of these generative classifiers, especially diffusion-based ones, is currently im- practically high. It is also unclear how incorporate complex augmentations, such Mixup, into generative classifiers. Finally, ideas from this work may be useful other contexts, such as lan- guage modeling. Tasks like sentiment analysis, code completion, and reasoning are currently being done discriminative approach: given a context x, predict the correct answer y by sampling from Pe(y | I). Improving the performance and out-of- of-distribution robustness of these models by doing generative approach p(r y) would be particularly exciting direction.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 456-473, 2018.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 33:1877-1901, 20200 2020.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. arXiv preprint arXiv:2305. 15241, 2023.
Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your diffusion model is secretly a certifiably robust classifier. arXiv preprint arXiv:2402.02316, 2024a.
Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for supervisd learning. arXiv preprint arXiv:2401.14444 2024b.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2023.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. preprint arXiv: 1810.0408, 2018.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Sociery, pp. 67-73, 2018.
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov. Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition scale. preprint arXiv:2010. 19299, 2020.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, (2):179-188, 1936.
Robert Geirhos, Henn-K Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
Shantanu Ghosh, Ke Yu, Forough Arabshahi, and Kayhan Batmanghelich. Route, interpret, re- peat: Blurring the line between post hoc explainability and interpretable models. arXiv preprint N3KV::2307.05350. 2023.
Will Grathwohl, Kuan-Chi Chich Wang, Joern-Henric Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint a903: 12261, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 152771, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 66511 2020.

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimat large language models. arXiv preprint arXiv:2203.15556 2022.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracyy In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. arXiv preprint 2tt.:::2209.16779999 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2rXi::2001108361,, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness spurious correlations. arXiv preprint arXiv:2204.02937, 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning. pp. 5637-5664. PMLR, 2021.
Daisuke Komura and Shumpei Ishikawa. Machine learning methods for histopathological image analysis. Computational and structural biotechnology journal, 16:34-42, 2018.
Alex Krizhevsky, lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from under- specified data, 2023.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffu- sion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303. 162.2 2023.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in neural information processing systems, 32, 2019.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05001 2017.
arXiv preprint
Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv: 2019.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15775. 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De- Di-bin classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20684 2020.
Andrew Y. Ng and Michael Jordan. On discriminative generative classifiers: A comparison of logistic regression and naive bayes. MIT Press, 2001.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guil- laume Lajoic. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1 156--2222 2021.
12

---

Published as a conference paper at ICLR 2025
Mihir Prabhudesai, Tsung-War Wei Ke, Alexander Cong Li, Deepak Pathak, and Katerina Fragkiadaki. Diffusion-tta: Test-time adaptation of discriminative models via generative feedback. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Aahlad Manas Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't blame dataset shift! shortcut learning due to gradients and cross entropy. Advances in Neural Information Processing Systems, 36:7 36:71874-71910, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning. 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv: 03305, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: /9/ 1.08731, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 2019.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arxiv:2008.04488 2020.
Amrith Setlur, Don Dennis, Benjamin Eysenbach, Aditi Raghunathan, Chelsea Finn, Virginia Smith, and Sergey Levine. Bitrate-constrained dro: Beyond worst case robustness unknown group shifts. arXiv preprint arXiv:2302.0293/ 2023.
Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation feature manipulation. In International conference on machine learning, pp. 19773-19808, PMLR, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025: 02502, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70); 1-57, 2018
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. 2020. URL https: wwt.:oogg//ooo.ooooooo.ooooooaa
00644
David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen Van Der Laak. Quantifying the effects of data augmentation and stain color normal- ization in convolutional neural networks for computational pathology. Medical image analysis, 101504,,, 2019.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint m1Xi:2332.13377, 2023.
Mitko Veta, Paul J Van Diest, Mehdi Jiwa, Shaimaa Al-Janabi, and Josien PW Pluim. Mitosis counting in breast cancer: Object-level interobserver agreement and comparison an automatic method. PloS one, 11(8))=0161286 2016.
Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware mitigation of spurious correlation. In International Conference Machine Learming, pp. 37765- 37786. PMLR, 2023.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change hard: A closer look at subpopulation shift. arXiv preprini arXiv:2302. /2254, 2023.
Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006.
Jieyu Zhao, Tianlu Wang, Mark Yatskar. Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 1707.094777 2017.
Chenyu Zheng, Guoqiang Wu, Fan Bao, Yue Cao, Chongxuan Li, and Jun Zhu. Revisiting dis- criminative vs. generative classifiers: Theory and implications. In International Conference on Muchine Learning, pp. 42420-42477. PMLR, 2023.
Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin A Dunn, and David A Klindt. Score- based generative classifiers. arXiv preprint arXiv:21 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL ANALYSIS
A.I ADDITIONAL RESULTS ON THE EFFECT OF DISCRIMINATIVE MODEL SIZE
OOD Test WG Accuracy
80
0OD Test Accuracy
28-
70
26-
ReaNet-50
60
ReaNet-50
ResNet-101
ResNet-1D1
24
ReaNet-152
50
ReaNet-152
92.5
95.0
Test Accuracy
Val Accuracy
(a) FMoW
(b) Camelyon17
Figure 8: Scaling up discriminative model size does improve performance. Each point with the same color is a model trained with different hyperparameters (learning rate and weight decay). Results Waterbirds are shown in Figure
We add additional results to our investigation into the role of discriminative model size. Previously, our analysis of CivilComments in Section 5.3 showed that matching the parameter count between the discriminative and generative classifiers did not account for the qualitative differences in their generalization behavior. Furthermore, Figure4showed showed that increasing model size on Waterbirds did not improve performance. Figure shows additional results. On FMoW, scaling only helps when going from ResNet-50 ResNet-101; further scaling did not help. On Camelyon 17, increasing model size had no effect on performance. Overall, we confidently conclude that model size is responsible for generative classifiers' improved robustness distribution shift.
A.2 SCALING CAN IMPROVE GENERATIVE CLASSIFIERS
FMoW
CivilComments
Camelyon17
000 Test WG Accuracy
36
Model Size
60.0
Model Size
69M
000 Test Accuracy
34-
57.5
32
Test WG
55.0
$2.5
30
Model Size
28
47.5
26
ID Vall Accuracy
Test Acc
Accuracy
Figure 9: Effect of scaling up generative classifiers. Increasing the number parameters helps sig- nificantly on FMoW and CivilComments, but sometimes hurt: OOD accuracy drops Came- Iyon17 with larger generative classifier.
Scaling model size has proved extremely effective for generative models in other settings (Brown al. 2020 Kaplan al. 2020 Hoffmann al.) 2022). This has typically been done the "almost infinite data" regime, where only a few epochs are used, and overfitting is not an issue. Does scaling similarly help here for our generative classifiers?
We tried different model scales on three of our distribution shift benchmarks: FMoW, CivilCom- ments, and Camelyon17. Figure shows the results of our investigation. On FMoW and Civil- Comments, scaling model size significantly improves performance both in- and out-of- disttiibibtniinn. However, on Camelyon17 17, a smaller model actually does significantly better out-of-diste -ifiiitribttion than model that is 5.5 times as large. This indicates that overfitting become issue in this setting.
15

---

Published as a conference paper at ICLR 2025
where we have limited training data and must be careful about overfitting. Overall, excited by the fact that scaling generative classifiers can be beneficial in some settings, unlike discrimimi- native classifiers, which consistently show poor use of extra model capacity (see Figure 4 Table Figure 8
A.3 RESULTS ON ADDITIONAL DATASETS
Entity-30
Living-1 -1
55
Discriminative
OOD Test Aest.Aruur Accuracy
OOD Accuracy
Generative
50
70
45
40-
50
80
90
80
85
90
95
ID Test Accuracy
ID Test Accuracy
Figure 10:
In-distribution vs out-of-dist accuracy for additional subpopulation shift datasets (Santurkar et al. 2020). We again observe OOD scaling trends for generative classifiers. Each point for discriminative model corresponds separate model with different architecture, augmentation, or adversarial training method. Accuracies for the discriminative models are taken from Santurkar et al. (2020)
We additionally run experiments two highly-used subpopulation shift benchmarks from BREEDS (Santurkar et al., 2020): Living-17 (with 17 animal classes) and Entity-30 (with 30 classes). As usual, train ing set and evaluate them on the in-distribution and out-of-distribute test sets. We compare against discriminative baselines reported in the original BREEDS paper, which includes interventions such as stronger augmentations or adversarial training. Figure IO displays the same trends here as our main results. Both datasets display effective robustness (for given ID accuracy, the OOD accuracy of the generative classifier is higher), though the effect much stronger Entity-30.
iiffusion-based generative classifiers from scratch each train-
A.4 CORRELATION BETWEEN GENERATIVE AND DISCRIMINATIVE PERFORMANCE
We take a careful look at how well generative capabilities like validation likelihood and sample quality correlate with classification performance. Figure II shows how these three metrics evolve over the course of training for diffusson-based generative classifier on CelebA.
We first find that the model does not need to generate good samples in order to have high classifi- cation accuracy. The first generation in Figure I1 has significant visual artifacts, yet the generative classifier already achieves 90% class-balanced accuracy. This makes sense: for ground-truth class y², so pe y²) can
y", the classifier only needs (0 y²) > pe(x y) for all other classes be low as long as (1 y * y even lower. fact, given a generative classifier Pe(x y), one can construct another generative classifier p(r y) = Apo(o y) + (1 P) (z). which has the same accuracy as pe but generates samples that look increasingly like Pather as O+.
However, even though sample quality is not necessary for high accuracy, we do find that validation diffusion loss correlates well with class-balanced aceuracy. As the loss decreases, class- balanced accuracy correspondingly increases. Figure 12 shows how an increase in validation diffusion loss due to overfitting translates to corresponding decrease in classification accuracy on Waterbirds.
Finally, Figure TT shows how can check the samples audit how the generative classifier models the spurious vs core features. The samples are generated deterministically with DDIM (Song et al.) 2020) from a fixed starting noise, so the sample from the last checkpoint shows that the model is increasing the probability of blond men (the minority group CelebA). This means that the model is modeling less correlation between the hair color (causal for the blond not blond label) and the gender (the shortcut feature). This is one additional advantage of generative classifiers: generating samples is built-in interpretability method (Li et al.) 2023). Again, as we note above, generation of specific feature is sufficient but not necessary show that it being used for classification.
16

---

Published as a conference paper at ICLR 2025
Diffusion Validation Losss
Cls-balanced Accuracy
Loss
92
0.15
Ace
ww hww.h///wwwmmmmm www w www.
91
0.14
0.13
0
50
100
150
200
Epoch
Figure 11: Correlation between accuracy and generative performance. Top: class-conditional DDIM samples generated from the same noise using intermediate checkpoints. Bottom: diffusion validation loss and class-balanced accuracy on CelebA by training epoch. Main findings: First, high classification accuracy be achieved Second, generative validation loss is highly correlated with classification accuracy. Third, as training progresses, the minority group (blond men) becomes more likely, indicating that the generative classifier correctly models less correlation between hair color (causal) and gender (shortcut).
without good sample quality (see the first generation).
Diffusion Validation Loss
Cls-balanced Accuracy
0.185
Wmm mm
Loss
-9555
Acc
0.180
95.0
0.175
94.5
0.170
-4..0
0.165
100
200
300
400
500
Epoch
Figure 12: Overfitting in diffusion loss Waterbirds directly translates to overfitting in classifica- tion accuracy. smooth loss for better visual clarity.
A.5 EFFECT OF IMAGE EMBEDDING MODEL.
For our image results in the main paper, we trained latent diffusion models from scratch for each dataset. In order be consistent with the generative modeling literature and keep the diffusion model training pipeline completely wmmodified, we trained the diffusion models on the latent space of pre-trained VAE (Rombach et al. 2022). This VAE compresses 256 x 256 x images into 32 x 32
Embedding model
Waterbirds
CelebA
Camelyon
ID
WG
ID
WG
ID
OOD
Pre-trained VAE (Rombach et al. 2022
96.8
79.4
91.2
69.4
98.3
90.8
PCA patch embeddings (Chen et 2024b)
93.8
61.7
91.3
71.1
98.7
93.8
Table 3: Effeet of image embedding model. We compare different image encoders, which map the pre-trained deep
the image from 256 x 256 x 3 to 32 32 x 4. For main results, VAE released in the original LDM paper (Rombach et al. 2022). We compare it to a PCA-based patch embedding that tokenizes each 8 8 x 3 patch independently and is trained separately on each dataset. We find that the pre- pra-tre VAE not consistently better, only does better on of the 3 datasets that we tested the PCA encoder on.
17

---

Published as a conference paper at ICLR 2025
4 latent embeddings, which are cheaper model. Perhaps our generative classifier is benefiting from an encoder that makes use of extra pre-training data? We test this hypothesis by trying to remove as much of the pre-trained encoding as possible. Following previous analysis work on diffusion models (Chen et al. 2024b). we replace the VAE network with simple PCA-based encoding of each image patch. Specifically, we turn each image into 32 x 32 total 8 x 8 x pixel patches, and use PCA to find the top 4 principal components of the patches. When encoding, the corresponding singular values to ensure that the PCA embeddings have approximately the same variance in each dimension. Overall, we perform this process separately on each training dataset, which completely removes the effect of pre-training, and train a diffusion model for each dataset within the PCA latent space. [Table 3 compares this embedding model to the VAE and finds that it actually performs better 2 of the datasets. We conclude that the pre-trained encoder does not have a significant directional effect on our generative classifier results.
normalize by
A.6 COMPARISON WITH PRE-TRAINED DISCRIMINATIVE MODELS
CelchA
OOD Tast WG Accuracy
60-
40-
Disc (pretrained)
20-
Generative (scratch)
ID Accuracy
Finetuning pretrained discriminative model improves performance, but it still does not achieve the same "effective robustness" our generative classifier.
Figure 13:
All of our experiments so far train the classifier (whether discriminative or generative) from scratch. This is done to ensure fair, apples-to-apples comparison between methods. What happens if we use pretrained discriminative model? In preliminary comparisons, we use a ResNet-50 pretrained with supervised learning on ImageNet-Ik (Krizhevsky et al. (2012) and finetune it on CelebA. Fig- ure [3 shows the results of this unfair comparison between pretrained discriminative model versus our generative classifier trained from scratch. We find that pretraining helps, but it does not signifi- cantly close the gap with the generative classifier. This is in spite of the fact that the discriminative model has seen an extra 1.2 million labeled training images, those labels have more bits (since there are 1000 classes instead of just two), and the pretraining classification task has minimal spurious correlations that are relevant to the downstream task.
18

---

Published as a conference paper at ICLR 2025
A.7
ADDITIONAL PLOTS FOR GENERALIZATION PHASE DIAGRAMS
Gen
Acc Gap (majority minority
0.4
0.9
Gen
Gem
0.3
Ratío |Wapl//|
Test Accuracy
Disc ID
0.8
Dise WG
0.2
1.5
0.7
0.1
1.0
0.6
0.0
0.5
16
64
256
1024
4096
16
64
256
1024 4096
16
64
256
1024 4096
Number of training points
Number training points
Number training points
Figure 14: Comparing logistic regression and LDA when the core feature variance has been in- creased from o 0.15 0.6. The generative approach's accuracy drops much more in this setting.
0 0.05
0= 0.15
0.3
Spurious feature
0.5
y= +1
0.0
y
0..5
-1.0
Causal feature
Causal feature
Causal feature
Figure 15: Effect of varying the standard deviation a of the core feature. d - 2 noise dimensions not
shown. These correspond the shown in Figure
n==22
n-
n=256
0.64
0.64
0.64
Noisy feauree variance
Gen better ID and OOD
Noisy feeture variance
Gen better LD and OOD
Noisy festure variance
Gen better ID and ODD
0.36
0.36
0.36
Dise better ID,
Gan betta DOD
Dive frctes ID.
0.16
Cizn better 0OD
0.16
0.16
0.04
0.04
0.04
0.00
0.00
0.00
0.75
1.51
2.26
3.00
0.00
0.75
1.51
2.26
3.00
0.00
0.75
1.51
2.26
3.00
Spurious feature scale
Spurious feature scale
Spurious feature scale
Figure 16: Each plot corresponds different number of training examples.
19

---

Published as a conference paper at ICLR 2025
1.0
Acc Gap (majority minority)
hoise
0.01
Normalized Weight Norm
Gen
[Waol
Disc
Disc
[Werel
Test
0.9
0.05
20-
Disc
[Weoow 1
[Woom
0.8
Gen ID
Gen
Wasul
10
Weome
Gen WG
0.00
0.7
Dise
Gen [Waml
|Wosselz
Disc WG
0.6
16
64
256 1024 4096
16
64
256
1024 4096
256 1024 4096
Number of training points
Number of training points
Number of training points
Nce Gap (majority minority)
nosse 1.00
1.0
0.3
Normalized Weight Norm
d
Dise [Pesel
[Waul
Test Accuracy
6
0.2
Disc [Mard
0.8
Gen Disc
Gen ID
4
Gon [Wecsl
Gen WG
0.1
Werr
0.6
Disc
2
[Wowsel
TWcarl
Disc WG
0.0
16
64 256 1024 4096
16
64
256
1024 4096
16
64 256 1024 4096
Number of training points
Number of training points
Number of training points
Figure 17: Effect of noise the generalization of SVM vs LDA. Larger nnoise makes it casier for SVM to overfit, since uses the high-norm noise features to increase margin. Lower Goise makes it harder to overfit, since the noise features are too small to significantly increase the margin.
B EXPERIMENTAL DETAILS
Algorithm Generative Classifier
1: Input: Training D (1xi, w:)]
2: Training model (e(x)y) y):
3: Minimize generative loss ((ay)=o log (x(
4: Classification of test input æ:
5: for class y; E y do
6: Compute pe p=(x))))
7: end for
8: Return arg max,, Po (2r (x)(:))(())
B.1 IMAGE-RASED EXPERIMENTS
B.1.1 DIFFUSION-BASED GENERATIVE CLASSIFIER
We train diffusion models from scratch in liwer-dimensional latent space (Rombach al. 2022). We use the default 395M parameter class-conditional UNet architecture and train it from scratch with AdamW W (Loshchilov & Hutter 2017) with constant base learning rate of 1e-6 and weight decay or dropout. We did not tune diffusion model hyperparameters and simply used the default settings for conditional image generation. Again, we emphasize: under distribution shift, using the default hyperparameters from image generation.
achieved SOTA accuracies
Each diffusion model requires about A6000 days to train. For inference on Waterbirds, CclebA, and Camelyon, sample 100 noises and use them with each of the classes. For FMoW,
20

---

Published as a conference paper at ICLR 2025
the adaptive strategy from Diffusion Classifier (Li et al. 2023 that uses 100 samples per class, then does an additional 400 samples for the top remaining classes.
B.1.2 DISCRIMINATIVE BASELINES
We use the official training codebase released by Koh et al. (2021) train discriminative base- lines. For image-based benchmarks, we train 3 model scales (ResNet-50, ResNet-101, and ResNet- 152) and sweep over 4 learning rates and 4 weight decay parameters. We use standard augmenta- tions: normalization, random horizontal flip, and RandomResizedCCrpp
B.2 AUTOREGRESSIVE GENERATIVE CLASSIFIER
For training, we pad shorter sequences to a length of 512 and only compute loss for non-padded tokens. We Llama-style architecture (Tovron models from scratch. We train for up to 200k iterations, which can take A6000 days. We use repository without default hyperparameters, so sweep over learning rate, weight decay, and dropout based on their effect on the data log-likelihood. The resulting family of models then shown in Figure 2
2023) and train 15M and 42M parameter
B.2.1 DISCRIMINATIVE BASELINES
For CivilComments, we use randomly initialized encoder-only transformer with the same architec- ture as DistilBert (Sanh et al. 2019). We train for 100 epochs and sweep over dropout rate, learning rate, and weight decay.
21


Reviews:
[{"review_id": "mQuZHjjqrr", "reviewer": "Reviewer_itDA", "summary": "The paper explores the performance of generative classifiers as compared to discriminative classifiers under distribution shift. It first documents the performance of both types of models on 5 standard benchmarks for distribution shifts concluding that generative classifiers indeed perform better under the distribution shift but surprisingly also often in in-distribution classification. It argues that this advantage stems from the generative classifiers having to learn all instead of just the short-cut features for max-margin classification induced by the cross-entropy loss. It further explores the behavior of the two classifier types on a synthetic toy problem controlling for the ratios and variance of spurious, core and noise features and provides useful insights into the effects of these data properties through outlining \"generatlization phase diagrams\". This investigation leads to no free lunch conclusion stating that the advantage of generative vs discriminative classifier is a function of the data properties which is in line with the results of the benchmark experimentation.", "questions": "1. The superior ID performance of generative models in Table 1 suggests to me that perhaps the discriminative models are too weak. For example, the original Sagawa paper reported the ID accuracy of discriminative classifiers for the Waterbird dataset between 94-97% (depending on regularization) and close to 95% for the CelebA. If this were true, to comparison would be unfair and not very useful. This feeling of \"unfair / skewed comparison\" is further reinforced by the comment in page 7 - discriminative classifier for CivilComments is larger and it indeed performs better on ID. Can you comment and clarify, how you ensure the fairness of the comparisons? If the comparison is indeed not fair, what does this mean for the presented results and the conclusions we can take from it?\n2. In page 5 you speak about \"class-balanced accuracy\". I am not sure how this is defined. Can you please clarify?\n3. Could  not the evolution of the gradient norm in Figure 3 simply suggest that the discriminative classifier learns faster?\n4. Section 6.3 Figure 6 - you fix $\\sigma^2_{noise} = 0.36$. Why particularly this value? Is this somehow representative of the behavior? Would the graphs look similarly at different noise levels?\n5. Section 6.3 Figure 6 uses the feature scale B=1? Or some other value?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The empirical investigation in the paper corroborates some rather well-accepted intuitions of potential benefits of generative classifiers in distribution-shift scenarios and thus provides useful support for future work in this direction. The systematic toy-problem exploration helps to understand the trade-offs and uses some tools that could be adapted for exploration of more complex scenarios in the future. Some further useful results are presented in the appendix (perhaps worth at least referencing in the main text?). The paper is well balanced in its conclusions, contributing to general understanding of the investigated phenomenons. The paper is well written and easy to follow.", "weaknesses": "It is unclear to me, how the conclusions documented on the rather simplistic and well controlled toy-problem could be translated to more complex practical problems under realistic scenarios. This is, however, acknowledged by the authors themselves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sY6vblGcfR", "reviewer": "Reviewer_RwsJ", "summary": "The paper explored the ability of generative classifiers to handle domain shifts. They use diffusion models and AR models for images and text data and show significant improvement in OOD classification with their classifier compared to other baselines.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper handles the important topic of OOD generalization in machine learning\n- The method shows good performance on several benchmarks\n- The paper is well written", "weaknesses": "- The main issue is limited novelty. Using generative models for classification isn't a new idea, and previous works already explored their robustness to OOD generalization and to adversarial attacks (and also the use of diffusion models as classifiers. For example Zimmermann et al \"Score-Based Generative Classifiers\", Grathwohl et al \"YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED\nMODEL AND YOU SHOULD TREAT IT LIKE ONE\", Chen et al \"Robust Classification via a Single Diffusion Model\" and Fetaya et al \"Understanding the Limitations of Conditional Generative Models\". The paper should also add them as related works.\n\n- While I like the use of simple toy examples, I don't think there was any insight or useful observation from the whole section. For example, the observation that \"As σ increases, the core feature becomes less reliable, and the generative classifier uses the spurious and noise features more\" is quite expected and doesn't give any added value. This section also is only relevant to generalization so it mirrors the results in Ng&Jordan 2001. \n\n- Also regarding the Illustrative setting, the feature you name $x_{spu}$ is not a spurious feature as it has a direct causal relationship by the label (as you designed it so). It just isn't as useful for discrimination as the \"core\" feature. \n\n- Regarding model size in the experiments, table 2 is also on CivilComments so that only covers one dataset. Moreover, on this dataset you performed worse than the discriminative models on the ID while you performed better than the other models on 3/4 of image datasets. This shows that the CivilComments behavior might not transfer. Also, parameter count isn't an exact comparison.\n\n- In two of the datasets, waterbirds and FMoW you can see in Fig. 2 that the ID and OOD seem to be very highly correlated when looking over all models. Looking at these plots, especially FMoW, it is not clear at all that the better OOD performance isn't just because your base model had better ID performance and not because of some innate property of generative models. \n\n\nSmall remarks:\n- L28 you write \"Ever since AlexNet (Krizhevsky et al., 2012), classification has mainly been tackled with discriminative methods\" but this isn't accurate as discriminative models have been the standard approach even before AlexNet, just mainly SVMs/decision trees/boosting etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97LQcBIWVs", "reviewer": "Reviewer_5wra", "summary": "The paper shows that generative classifiers always perform better in the realistic OOD setting, which can be implemented easily based on existing class-conditional generative models. They conduct simulations to show that generative classifiers can model both core and spurious, instead of mainly spurious ones. They also present the data properties that affect when generative classifiers outperform discriminative ones.", "questions": "1. In line 322, why do you choose epoch 5? Can this affect the final results?\n2. In line 341, the increase of gradient norms makes me confused. It should decrease during the training because the gradients will converge to 0.\n3. In line 364, the paper claims that parameter count does not seem to be responsible for the performance of the generative classifier. Does this mean, when considering downstream discriminative tasks, we can not enjoy the scaling law of generative foundation models?\n\nIf you can address the weaknesses and questions sufficiently, I will improve the rating.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is novel and well-written.\n2. The paper shows that generative classifiers have advantages compared to discriminative classifiers, both in ID and OOD settings. I think this is a fundamental challenge to existing discriminative learning, which is significant.\n3. The experiments are sufficient to validate the proposed hypothesis.", "weaknesses": "1. I think some recent papers about the superiority of generative classifiers should be mentioned in this paper, including [a,b,c].\n\n[a] Diffusion Models are Certifiably Robust Classifiers, NeurIPS, 2024\n\n[b] Robust Classification via a Single Diffusion Model, ICML, 2024\n\n[c] Revisiting Discriminative vs. Generative Classifiers: Theory and Implications, ICML, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7cUlZuY85b", "reviewer": "Reviewer_vNha", "summary": "This paper revisits generative classifiers to address the problem of spurious features that affect discriminative classifiers trained with cross-entropy loss on out-of-distribution data. Leveraging class-conditional generative models, based on diffusion for image related problems and autoregressive models for text related problems, the authors show that generative classifiers outperform discriminative classifier both in-distribution and out-of-distribution on various benchmarks. The authors also propose some analysis on a toy task.", "questions": "See above.\n\n===Post rebuttal===\n\nI thank the authors for the rebuttal; I recommend acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* While the concept of generative classifiers is not new, applying recent generative models to the challenging out-of-distribution generalization problem is timely and interesting. The paper is well written and has a strong overall message.\n\n* The paper provides evaluation on several datasets covering both image and text-related problems. The improvement due to generative classifiers over discriminative baselines seems important in both out-of-distribution and even in-distribution scenarios.\n\n* The authors provided intuitions why the generative classifiers performs better; in particular I found the analysis on the gradient norm and the toy evaluation well lead and insightful.", "weaknesses": "* Comparison to classifier with pretraining and data augmentation: although pretraining does not seem to improve the proposed model, it does have a big importance for standard classifiers; in particular, could the authors show (e.g. on one chosen image-based dataset) that finetuning a pretrained ResNet-50 classifiers (on imagenet) does not reduce significantly the gap with the considered approach ? Moreover, applying data augmentation is also a standard trick for image-based classification. Although its application to the generative classifiers is still unclear (as mentioned in the conclusion), was it included in the discriminative models reported in the paper ?\n* Multi-class classification: the authors seem to have mostly included binary classification tasks; does the proposed model scale to multi-class classification problems ? If yes, including some evaluation on a multi-class problem could add some value to the experimental evaluation (e.g. DomainBed - https://arxiv.org/abs/2007.01434 - provides some multi-class image-based benchmarks where comparison can be made with several invariant based approaches).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)