CARD: Classification and Regression Diffusion Models
Authors: Xizewen Han
Huangjie Zheng
Mingyuan Zhou
Venue: NeurIPS 2022 Accept (2022)

Abstract: Learning the distribution of a continuous or categorical response variable y given its covariates x is a fundamental problem in statistics and machine learning. Deep neural network-based supervised learning algorithms have made great progress in predicting the mean of y given x, but they are often criticized for their ability to accurately capture the uncertainty of their predictions. In this paper, we introduce classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of y given x.  We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets, the experimental results on which show that CARD, in general, outperforms state-of-the-art methods, including Bayesian neural network-based one, designed for uncertainty estimation, especially when the conditional distribution of y given x is multi-modal. In addition, we utilize the stochastic nature of the generative model outputs to obtain a finer granularity in model confidence assessment at the instance level for classification tasks.

CARD: Classification and Regression Diffusion Models
Xizewen Han' Huangjie Zheng'
Mingyuan Zhou
Department of Statistics and Data Sciences The University of Texas at Austin Austin, TX 78712
McCombs School of Business
The University of Texas at Austin Austin, TX 78712
(xizewen. han, han, huangjie zheng) cchenn)@uteeeee @utexas. edu
mingyuan.zhoueuccoom utexas, edu
Abstract
Learning the distribution of continuous categorical response variable y given its covariates 2 a fundamental problem in statistics and machine learning. Deep neural network-based supervised learning algorithms have made great progress predicting the mean of given az, but they are often criticized for their ability to accurately capture the uncertainty of their predictions. In this paper, we introduce classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and pre-trained conditional mean estimator, to accurately predict the distribution of given æ. We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets, the experimental results on which show that CARD in general outperforms state-of-the-a methods, including Bayesian neural network based ones that are designed for uncertainty estimation, especially when the conditional distribution of y given æ multi-modal. In addition, we utilize the stochastic nature of the generative model outputs to obtain finer granularity in model confidence assessment at the instance level for classification tasks. Our implementation is publicly available at htttp:://gthubb.comm hwt.ggiihuuucooo.iee how..////////////AAA
Introduction
A fundamental problem in statistics and machine learning is to predict the response variable given of covariates æ. Generally speaking, y is continuous variable for regression analysis and categorical variable for classification. Denote fæx) E RC as deterministic function that transforms into C' dimensional output. Denote fz( as the c-th dimension of fz)) Existing methods
typically assume an additive noise model: for regression analysis with E RC one often assumes
y f(x) E, (0, E), while for classification with y E {l...., C}, one often assumes y arg max (Ji (ar) + E1,-.. fi,-.-,fc(()) fc(())
+c),, where e, iid EV, (0, 1), standard type-1 extreme value distribution. Thus have the expected value of y given
E æi] f(x) in regression ex)(f.(x))
and (y x) Fly m] = poftmax.(J(am)
classification.
ΣS_, explf. (=))
These additive-noise models are primarily focusing on accurately estimating the conditional mean E æ), while paying less attention to whether the noise distribution can accurately capture the uncertainty of y given For this reason, they may not work well if the distribution of y given x clearly deviates from the additive-noise assumption. For example, if p(y x) mudti-modal, which commonly happens when there are missing categorical covariates in æ. then Ely ax] may not be close to any possible true values of given that specific æ. More specifically, consider a person whose weight, height, blood pressure, and age known but gender is unknown, then the testosterone or estrogen level of this person is likely to follow a bi-modal distribution and the chance
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

of developing breast cancer is also likely to follow a bi-modal distribution. Therefore, these widely used additive-noise models, which use deterministic function f(x) to characterize the conditional mean of y. are inherently restrictive in their ability for uncertainty estimation.
In this paper, our goal is accurately recover the full distribution of y conditioning æ: given set of N training data points, denoted as D = {( y 1, To realize this goal, we consider the diffusion-based (a. score-based) generative models (Sohl- Sohh-Dickstein et al. 2015 Song and Ermon 2019 Ho et al. 2020: Song and Ermon 2020 Song et al. 2021c) and inject covariate-dependee
into both the forward and reverse diffusion chains. Our method can model the conditional distribution of both continuous and categorical variables, and the algorithms developed under this method will
be collectively referred to Classification And Regression Diffusion (CARD) models.
Diffusion-based generative models have received significant recent attention due to not only their ability to generate high-dimensional data, such as high-resolution photo-realistic images, but also their training stability. They can be understood from the perspective of score matching (Hyvärinen and Dayan 2005; Vincent 2011] Kadkhodaie and Simoncelli (021) and Langevin dynamics (Neal 2011 Welling and Teh [201] I). as pioneered by Song and Ermon (2019). They can also be understood
from the perspective of diffusion probabilistic models (Sohi-Dickstein et al., [2015] Ho et al. 2020). which first define a forward diffusion to transform the data into noise and then a reverse diffusion to regenerate the data from noise.
These previous methods mainly focus on unconditional generative modeling. While there exist guided-diffusion models (Song and Ermon 2019 Song al., 2021c] Dhariwal and Nichol 202T Nichol et al., 2022 Ramesh et al. 2022) that target on generating high-resolution photo-realistic images that match the semantic meanings content of the label, text, or corrupted-images, focus on studying diffusion-basee conditional generative modeling at a more fundamental level. In particular, our goal to thoroughly investigate whether CARD can help accurately recover p(y æ, D), the predictive distribution of given a after observing data D = {(,,, y: Ji)). In other words, our focus is on regression analysis of continuous or categorical response variables given their corresponding covariates.
We summarize main contributions as follows: 1) We show CARD, which injects covariate- dependence and a pre-trained conditional mean estimator into both the forward and reverse diffusion chains to construct denoising diffusion probabilistic model, provides accurate estimation of p(yl æ. D). 2) We provide a new metric to better evaluate how well a regression model captures the full distribution p(y æ. D). 3) Experiments that CARD achieves state-of-the-art resul(s, using both existing metrics and the new one. 4) For classification tasks, we push the assessment of model confidence in its predictions towards the level of individual instances, a finer granularity than the previous methods.
standard benchmarks for regression analysis show
2 Methods and Algorithms for CARD
Given the ground-truth response variable yo and its covariates x, and assuming sequence of intermediate predictions yı.r made by the diffusion model, the goal of supervised learning is to learn model such that the ligeelooo is maximized by optimizing the following ELBO:
Po P3 (x)
log po (3o x) log
Po(o x)dy):r
log
(1)
Yo. x)
where q yo. is called the forward process or diffusion process in the concept of diffusion models (Sohl-Dickstein et al. 2015 Ho et al., 2020). Denote (q || p) as the Kullback-Leibler
(KL) divergence from distribution p to distribution q. The above objective can be rewritten as
T
CELBO x) = Lo(yo. x) +
Σ
C x) + LT (vo.
(2)
Lo(yo: x²) = F, log y x)),
(3)
C, 1y- x) = F, [KKI. ( ye: yo. po (3: ye,
1 x),
(4)
Cy x) = E,, (DKa((((0)) yoo x) || p(yr x))]
(5)
Here we follow the convention to assume Lr does not depend on any parameter and will close to zero by carefully diffusing the observed response variable yo towards a pre-assumed distribution

---

P(yr z). The remaining terms will make the model pe (e( yr, .) approximate the corresponding tractable ground-trun truth denoising transition step 3(3/-11 yt. Mo. M) for all timesteps. Different from the vanilla diffusion models, we assume the endpoint of our diffusion process to be
p yr ( - (x),,,
(6)
where (z) is the prior knowledge of the relation between a and yo. e.g. a network pre-trained with D to approximate F Fly æ²), 0 we assume the relation is unknown. With diffusion schedule [3: 1-1r E (0. 1)² we specify the forward process conditional distributions similar fashion as
Pandey al. 2022). but for all timesteps including i 1:
(v: 11-1. = .V (v: v Bey--1 + (1 (1-v1- -1 1 (
which admits a closed-form sampling distribution with an arbitrary timestep t:
(v: yo. fo (x)) = / NN vāryo
(1 -+)
where a; := - 3r and ã; == II. a,, Note that the mean term in Eq. 7 can viewed as an interpolation between data yo and the predicted conditional expectation (ar), which gradually changes from the former to the latter throughout the forward process.
Such formulation corresponds to tractable forward process posterior:
(3 Vos yo. x) = (3 y/s yo. fo(x)) NV (y--1 yo. fo(x)),B.1),
(9)
where
Biv B
(1 at-1)var
Yt + 1+ (Vār l)(va vās-1)
yo
fa(w)
0t
Cke
70
91
8.
04t-
3r.
We provide the derivation in Appendix A.1
The labels under the terms used in Algorithm
2.1 CARD for Regression
For regression problems, the goal of the reverse diffusion process is to gradually recover the distribu- tion of the noise term, the aleatoric or local uncertainty inherent in the observations (Kendall and
Gal
2017 Wang and Zhou 2020), enabling us to generate samples that match the true conditional x).
p
Following the reparameterization introduced by denoising diffusion probabilistic models (DDPM) (Ho et al. (020), we construct eø (x,, Vr. (ar)),,), t), which is function approximator parameterized by a deep neural network that predicts the forward diffusion noise e sampled for ye. The training and inference procedure can be carried out in standard DDPM manner.
Algorithm 1 Training (Regression)
1: Pre-train (x( that predicts (Ey x)) with MSE
2: repeat
3: Draw Yo (
4: Draw to Uniform (1..orm((1...T)) (T)
5: Draw e ~ Ao
6: Compute noise estimation loss
L. = |e - c (z, / a.y0 + VI die (1 (a))
7: Take numerical optimization step on:
VeLe
8: until Convergence
3

---

Algorithm 2 Inference (Regression)
1: yr (a)(((
2: for t = 7 to do
3: Draw 2 ~ (0,
4:
Calculate reparameterized ÿo Vaa Ur -1- (o.. -VI (1, (ts
5: Let yt-1 v000 + 719I 72fx(2)) + Brrz if > 1, else set y/-1 50
6: end for
7:
return yo
2.2 CARD for Classification
We formulate the classification tasks in a similar fashion as in Section 2.1 where we:
Replace the continuous response variable with a one-hot encoded label vector for yo:
2. Replace the mean estimator with a pre-trained classifier that outputs softmax probabilities of the class labels for (a)
This construction no longer assumes yo to be drawn from a categorical distribution, but instead treats each one-hot label class prototype, i.e. we assume continuous data and state space, which enables us to keep the Gaussian diffusion model framework. The sampling procedure would output reconstructed Yo in the range of real numbers for each dimension, instead of vector in the probability simplex. Denoting C' as the number of classes and lc as C-dimensional vector of 1s, we convert such output probability vector in softmax form of temperature- (Brier 1950), which computes the squared error between the prediction and 1c. Mathematically, the probability of predicting the 1hth class and the final point prediction can be expressed as
Brier score
Pr(y = k:) exp( yyo 1c)l/T)
E exp( (y0 1/) 9 = arg max ( (- (- (yo-111) 1c)))
(10)
where > 0 is the temperature parameter, and (yo 1c) indicates the t dimension of the vector of element wise square error between yo and Ic, i.e., (yo Ic) ll - 111.. Intuitively, this
construction would assign the class whose raw output the sampled yo is closest to the true class, encoded by the value of in the one-hot label, with the highest probability.
Conditional on the same covariates I, the stochasticity of the generative model would give different class prototype reconstruction after each reverse process sampling, which enables us to construct predicted probability intervals for all class labels. Such stochastic reconstruction in similar fashion as DALL-E 2 (Ramesh et al. (2222) that applies diffusion prior to reconstruct the image embedding by conditioning on the text embedding during the reverse diffusion process, which is a key step in the diversity of generated images.
3 Related Work
Under the supervised learning settings, to model the conditional distribution p(y (>) besides just the conditional mean Ely through deep neural networks, existing works have been focusing on quantifying predictive uncertainty, and several lines of work have been proposed. Bayesian neural networks (BNNs) model such uncertainty by assuming distributions over network parameters. capturing the plausibility of the model given the data (Blundell et al. 2015 (eeeaaendn---bbaaee and
Adams 2013 Gal and Ghahramani 2016 Kingma et al.) 2013 Tomezak et 20.1)) Kendall and
Gal (2017) also model the uncertainties in the model outputs besides model parameters, by including
the additive noise term as part of the neural network output. Meanwhile, ensemble- based methods
[Lakshminarayanan et al. 2017: Liu et al.) 2022) have been proposed to model predictive uncertainty by combining multiple neural networks with stochastic outputs. Furthermore, the Neural Processes Family (Garnelo et al. 2018b Kim et al. 2019 Gordon et al.) 2020) has introduced series of models that capture predictive uncertainty in an out-of-distributtin fashion, particularly designed for few-shot learning settings.

---

These above mentioned models have all assumed parametric form in p(y æ),, namely Gaussian distribution, or mixture of Gaussians, and optimize the network parameters based on a Gaussian negative log-likeling on-likelihood objective function. Deep generative models, on the other hand, have been known for modeling implicit distributions without parametric distributional assumptions, but very few works have been proposed to utilize such feature to tackle regression tasks. GAN-based models are and
introduced by Zhou et al. (2021) and Liu et al. (2021) for conditional density estimation and predictive
uncertainty quantification. For classification tasks, on the other hand, generative classifiers (Revow ct al. 1996; Fetaya et al. 2020; Ardizzone et al. 2020 Mackowiak et al., (02T) a class of models that also perform classification with generative models; among them, Zimmermann al. (2021) propose score- -baeed generative classifiers to tackle classification tasks with score-based generative models (Song et al. 2021b c). They model p(æ y) and predict the label with the largest conditional likelihood of æ, while CARD models p(y az)) instead.
In recent years, the class of diffusion-based (or score-based) deep generative models has demonstrated its outstanding performance in modeling nigh-dimensional mudti-modal distributions (Ho et al.) 2020)
Song et al. 2021a Kawar et al.) 2022 Xiao et al., 2022 Dhariwal and Nichol, 2021: Song and Ermon 2019 2020 with most work focusing on Gaussian diffusion processes operating in continuous state spaces. Hoogeboom et al. (202T) introduce extensions of diffusion models for categorical data,
and Austin et al. (2021) have proposed diffusion models for discrete data as a generalization of the multinomial diffusion models, which could provide alternative way of performing classification with diffusion- daffusion-bbsed models.
Experiments
For the hyperparameters of CARD in both regression and classification tasks, we set the number of timesteps as T 1000, a linear noise schedule with 31 10-4 and 31 0.02, same as Ho al. (2020). We provide a more detailed walk-through of the experimental setup, including training and network architecture, in Appendix A.8
4.1 Regression
Putting aside its statistical interpretation, the word regress indicates a direction opposite to progress, suggesting less developed state. Such semantics in fact translates well into the statistical domain, in the sense that traditional regression analysis methods often only focus on estimating E(y x), while leaving out all remaining details about p(y) x). In recent years, Bayesian neural networks (BNNs) have emerged as a class of models that aims at estimating the uncertainty (Hermández- Adams 2015 Gal and Ghahramani 2016; Lakshminarayann
and et al. 2017 Tomezak al.) (021)) I).
providing a more complete picture of p(y x). The metric that they use to quantify uncertainty estimation, negative log-likelihood (NL.).. is computed with Gaussian density, implying their assumption such that the conditional distributions p(y x x)) for x are Gaussian. However, this assumption is very difficult to verify for real-world datasets: the covariates can be arbitrarily high-dimensional, making the feature space increasingly sparse with respect to the number of collected observations.
To accommodate the need for uncertainty estimation without imposing such restriction for the parametric form of p x). apply the following two metrics, both of which are designed to empirically evaluate the level of similarity between the learned and the true conditional distributions:
1. Prediction Interval Coverage Probability (PICP);
2. Quantile Interval Coverage Error (QICE).
PICP has been described in Yao al. (019). whereas QICE is a new metric proposed by us. We describe both of them in what follows.
4.1.1 PICP and QICE
The PICP is computed
1 PICP =
pmm 1 giih
(11)
N
5

---

where "/ and hhigh represent the low and high percentiles, respectively, of our choice for the predicted y outputs given the same æ input. This metric measures the proportion of true observations that fall in the percentile range of the generated y samples given each æ input. Intuitively, when the learned distribution represents the true distribution well, this measurement should be close to the difference between the selected low and high percentiles. In this paper, we choose the 2.5th and 97.5th percentile, thus an ideal PICP value for the learned model should be 95%.
Meanwhile, there is a caveat for this metric: for example, imagine a situation where the 2.5th to 97.5th percentile of the learned distribution happens cover the data between the 1st and 96th percentiles from the true distribution. Given enough samples, we shall still obtain PICP value close to 95%, but clearly there is a mismatch between the learned distribution and the true one.
Based on such reasoning, we propose a new empirical metric QICE, which by design can be viewed as PICP with finer granularity, and without uncovered quantile ranges. To compute QICE, first generate enough y samples given each æ; and divide them into M bins with roughly equal sizes. We would obtain the corresponding quantile values at each boundary. In this paper, we set M 10, and obtain the following 10 quantile intervals (QIs) of the generated y samples: below the 10th percentile, between the 10th and 20th percentiles, .... between the 80th and 90th percentiles, and above the 90th percentile. Optimally, when the learned conditional distribution is identical to the true one, given enough samples from both learned and true distribution we shall observe about 10% of true data falling into each of these 10 QIs.
We define QICE be the mean absolute error between the proportion of true data contained by each QI and the optimal proportion, which is 1/M for all intervals:
1 M Fm
1 M
1 where T'm N
QICE
Mr. 20 form
1
(12)
Intuitively, under optimal scenario with enough samples, we shall obtain QICE value of O. Note that each I'm is indeed the PICP for the corresponding QI with boundaries at g glown and yhe hhihh.... Since the true y for each æ is guaranteed to fall into one of these Qls, we are thus able to overcome the mismatch issue described in the above example for PICP: fewer true instances falling into one QI would result in more instances captured by another QI, thus increasing the absolute error for both QIs.
QICE is similar to NLL in the sense that it also utilizes the summary statistics of the samples from the learned distribution conditional on each new æ to empirically evaluate how well the model fits the true data. Meanwhile, it does not assume any parametric form on the conditional distribution, making it much more generalizable metric to measure the level of distributional match between the learned and the underlying true conditional distributions, especially when the true conditional distribution is known to be multi-modal. We will demonstrate this point through the regression toy examples.
4.1.2 Toy Examples
To demonstrate the effectiveness of CARD in regression tasks for not only learning the conditional mean E(y| æ), but also recreating the ground truth data generating mechanism, we first apply CARD on toy examples, whose data generating functions are designed to possess different statistical characteristics: some have uni-modal symmetric distribution for their error term (linear regression, quadratic regression, sinusoidal regression), others have heteroscedasticity (log-log linear regression, log-log cubic regression) or multi-modality (inverse sinusoidal regression, 8 Gaussians, full circle). We show that the trained CARD models can generate samples that are visually indistinguishable from the true response variables of the new covariates, as well as quantitatively match the true distribution in terms of some summary statistics. We present the scatter plots of both true and generated data for all 8 tasks in Figure [ For tasks with uni-modal conditional distribution, we fill the region between the 2.5th and 97.5th percentile of the generated y s. We observe that within each task, the generated samples blend remarkably well with the true test instances, suggesting the capability of reconstructing the underlying data generation mechanism by CARD. A more detailed description of the toy examples, including more quantitative analyses, is presented in Appendix A.13
4.1.3 UCT Regression Tasks
We continue to investigate our model through experiments on real-world datasets. We adopt the same set of 10 UCI regression benchmark datasets (Dua and Grafi 2017) well as the experimental

---

10
42
403
302
-5
1.-
14.
10.
04-
0.-
0.1
-3
0.-
03
-02
24
0.00 5.26
3.00
Figure Regression toy example scatter plots. (Top) left to right: linear regression, quadratic regression, log-log linear regression, log-log cubic regression; (Bottom) left to right: sinusoidal regression, inverse sinusoidal regression, Gaussians, full circle.
protocol proposed by Hernández-Looato and Adams (2015) and followed by Gal and Ghahramani (2016) and Lakshminarayanan et al. (2017). The dataset information is provided in Table 14
We apply multiple train-test splits with 90% 110% ratio in the same way as Hernández-Lobato and Adams (015) (20 folds for all datasets except 5 for Protein and for Year), and report the metrics by their mean and standard deviation across all splits. We compare our method all aforementioned
BNN frameworks: PBP, MC Dropout, and Deep Ensembles, as well as another deep generative model that estimates conditional distribution sampler, GCDS (Zhou al.] 202T). We note that GCDS is related to a concurrent work of Yang et al. (2022). who share comparable idea but use it in a different application. Following the same paradigm of BNN model assessment, we evaluate the accuracy and predictive uncertainty estimation of CARD by reporting RMSE and NLL. Furthermore, we also report QICE for all methods to evaluate distributional matching. Since this new metric was not applied in previous methods, re-ran the experiments for all BNNs and obtained comparable or slightly better results in terms other commonly used metrics reported in their literature. Further details about the experimental setup for these models can be found in Appendix A.9 The experiment
results with corresponding metrics are shown in Tables 2 and 3 with the number of times that each model achieves the best corresponding metric reported the last row.
We observe that CARD outperforms existing methods, often by considerable margin (especially on larger datasets), in all metrics for most of the datasets, and is competitive with the best method for the remaining ones: we obtain state-of -ortte---tt--tt--- results 10 for NLL, and 5 out of 10 for QICE. It is worth noting that although we do not explicitly optimize our model by MSE or by NLL, we still obtain better results than models trained with these objectives.
out of 10 datasets terms of RMSE, 8 out of
4.2 Classification
Similar to Lakshminarayanan et al. (017). our motivation for classification is not to achieve state-of- the-art performance in terms of mean accuracy on the benchmark datasets, which strongly related to network architecture design. Our goal is two-fold:
1. We aim solve classification problems via generative model, emphasizing its capability to improve the performance of a base classifier with deterministic outputs in terms of accuracy;
2. We intend to provide an alternative sense of uncertainty, by introducing the idea of model confidence at the instance level, i.e., how sure the model is about each of its predictions, through the stochasticity of outputs from a generative model.
As another type of supervised learning problems, classification is different from regression mainly for the response variable being discrete class labels instead of continuous values. The conventional operation is to cast the classifier output as point estimate, with a value between 0 and 1. Such
7

---

Table 1: RMSE of UCI regression tasks. For both 'Kin8nm and 2aval dataset, we multiply the response variable by 100 to match the scale of others.
Dataset
RMSE Deep Ensembles
PRP
MC Dropour
GCDS
CARD (ours)
Boston
2.89 = 0.77
3.06 + 0.96
3.17 + 1.05
2.75 0.38
2.61 + 0.63
Concrete
5.55 1: 0.46
5.09
1 0.60
4.91 1 0.47
5.39
1 0.55
4.77
1 0.46
Energy
0.32
0.64
Kin8em!
9.42 1: 0.29
7.10 1 0.26
8.65 0.47
8.88 1 0.42
6.32 1 0.18
Naval"
0.08
0.14
Power
4.10 = 0.15
1.04 0.14
1.02 0.15
4.11 0.16
3.93 + 0.17
Protein
4.65 + 0.02
4.16 + 0.12
4.45 + 0.02
4.50
3.73 + 0.01
Wine
0.64 0.04
0.62 0.04
0.63 0.04
0.66 0.04
0.63 1 0.04
Yacht
0.88 + 0.22
0.84 0.27
1.19 0.49
0.79 + 0.26
0.65 + 0.25
8.864
8.77:
8.794
9.204
best
0
9
Table 2: NLL of UCI regression tasks.
Dataset
NLL 1 Deep Ensembles
PBP
MC Dropout
GCDS
CARD (ours)
0.16
18.06
2.35
Cancrete
3.19-0.00 1.00-0.000 0.06
3.21 0.18
2.93 1.1.2
13.64
+.88 6.88
2.56 0.02)
Energy
2.05 0.05
1.50 0.11
1.40 0.27
1.46 0.722
1.04 0.06
Kinšuun
Naval
-3.97
4..45 0.38
-5.94 0.10
55.06 0.48
-7.54 0.05
Power
2.92 0.02
2,90 : 0.03
2:89 0.02
2.83 0.06
2.82 : 0.02
Protein
3.05= 0.00
2.80 0.08
2.89 0.02
2.81 = 0.00
2.49 0.03
Wine
Yacht
1.58 09 0.08
1.73 2:3:2000 0.22
1111
0.61 0.34
0.50 0.08
Year
3.00 NA
3.424 NA
3.41 NA
3.433 NA
3.341 NA
best
Table 3: QICE (in %) of UCI regression tasks.
Dataset
QICE L Deep Ensembles
PBP
MC Dropout
GCDS
CARD (ours)
Boston
3.50 + 0.88
3.52 = 0.82
3.37
0.00
11.73 1 1.05
3:45 0.83
0.60
Energy
6.54 + 0.90
5.22 1.02
3.62
0.58
7.41 + 2.19
4.91
0.94
KinSom
1.31 + 0.25
1.50 0.32
1.17 0.22
7.73
0.92 = 0.25
Power
0.82 + 0.19
1.32 0.37
1.09
= 0.26
1.77 0.33
0.92 = 0.21
Protein
1.69 1: 0.09
2.52 = 0.41
2.17 0.16
2.33 1: 0.18
0.71 0.11
Wine
2.79
3.13
3.30
Yoeht
6.93 # 1.74
10.33 1.34
7.22 = 1.41
5.01 + 1.02
8:03 1.17
Year
2.961 NA
2.431 NA
2.563
1.611 NA
0,53+ NA
best
2
design is intended for prediction interpretability; since humans already have a cognitive intuition for probabilities (Cosmides and Tooby, 1996), the output from classification model is intended to convey a sense of likelihood for a particular class label. other words, the predicted probability should reflect its confidence, i.e., level of certainty, in predicting such a label. Guo al. (2017) provide the following example of a good classifier, whose output aligns with human intuition for probabilities: if the model outputs a probability prediction of 0.8, we hope indicates that the model 80% sure that its prediction is correct; given 100 predictions of 0.8, one shall expect roughly 80 of them correct.
In that sense, a good classification algorithm not only can predict the correct label, but also can reflect the true correctness likelihood through its probability predictions, i.e., providing calibrated confidence (Guo et al., 2017). To evaluate the level of miscalibration by a model, metrics like Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) (Nacini et al. 2015) have been adopted in recent literature (Kristiadi et al. 2022 Rudner et al. 2021 for image classification tasks, and calibration methods like Platt scaling and isotonic regression have been developed to improve such alignment (Guo et al. (017).
Note that these methods are all based on point estimate predictions by the classifier. Furthermore, these alignment metrics can only be computed at subgroup level in practice, instead of at the instance level. In other words, one may not be able make the claim with the existing classification

---

framework such that given a particular test instance, how confident the classifier is in its prediction be correct. We discuss our analysis in ECE with more details in Appendix A..66 which may help justify our motivation in introducing an alternative way to measure model confidence at the level of individual test instances in Section 4.2.1
4.2.1 Predict with Instance Level Model Confidence via Generative Models
We propose the following framework to assess model confidence for its predictions at the instance level: for each test instance, we first sample N class prototype reconstructions by CARD through the classification version of Algorithm 2 and then perform the following computations:
1. We directly calculate the prediction interval width (PIW) between the 2.5th and 97.5 percentiles of the N reconstructed values for classes, i.e., with C different classes total, we would obtain C PIWs for each instance;
2. We then convert the samples into probability space with Eq. (10). and apply paired two- sample (-test as an uncertainty estimation method proposed Fan et al. (202T): we obtain the most and second most predicted classes for each instance, and test whether the difference in their mean predicted probability is statistically significant.
This framework would require the classifier not produce the exact same output each time, since the goal is to construct prediction intervals for each of the class labels. Therefore, the class of generative models is a preferable modeling choice due its ability produce stochastic outputs, instead of just a point estimate by traditional classifiers.
In practice, we view each one-hot label as a class prototype in real continuous space (introduced in Section 2.2). and we use a generative model to reconstruct this prototype in a stochastic fashion. The intuition is that if the classifier is sure about the class that particular instance belongs to, would precisely reconstruct the original prototype vector without much uncertainty; otherwise, different class prototype reconstructions of the same test instance tend to have more variations: under the context of denoising diffusion models, given different samples from the prior distribution at timestep T, the label reconstructions would appear rather different from each other.
4.2.2 Classification with Model Confidence on CIFAR-10 Dataset
We demonstrate our experimental results on the CIFAR-10 dataset. We first contextualize the performance of CARD in conventional metries including accuracy and NLL with other BNNs in R8 architecture in Table 4 The metrics of other methods were reported in Tomczaki et al. (202T). a recent work BNNs that proposes tighter ELBOs to improve variational inference performance and prior hyperparameter optimization. Following the recipe in Section 2.2, we first pre-train a deterministic classifier with the same ResNet- 18 architecture, and achieve a test accuracy of 90.39%. with which we proceed to train CARD. We then obtain our instance prediction through majority vote, i.e., the most predicted class label among its N samples for each image input, and achieve improved test accuracy with mean of 90.93% across 10 runs, showing its ability to improve test accuracy from the base classifier. Our NLL result competitive among the best ones, even though the model is not optimized with cross-entropy objective function, as assume the class labels to be in the real continuous space.
Table 4: Comparison of accuracy (in %) and NLL for CIFAR-10 classification with other BNNs.
Model
CMV-MF-VI
CM-MF-VI
CV-MF-VI
MF-VI
MC Dropout
MAP
CARD
Accuracy
86.25 + 0.06
86.66
79.78
0.30
77.08 1.14
83.64 0.28
84.69
0.35
90.93 + 0.02
NLL
0.41 0.00
0.39 + 0.00
0.59
0.00
0.68
0.02
0.49 + 0.00
0.93 + 0.02
0.46 0.00
We now present the results from one model run with the proposed framework in Section 4.2.1 for evaluating the instance-level prediction confidence. After obtaining the PIW and paired two-sample (-test (a = 0.05) result from each test instance, we first split the test instances into two groups by the correctness of majority-vote predictions, then we obtain only the PIW corresponding to the true class for each instance, and compute the mean PIW of the true class within each group. In addition, we split the test instances by t-test rejection status, and compute the mean accuracy in each group. We

---

report the results from these two grouping procedures in Table 5 where the metrics are computed across all test instances and at the level of each true class label.
Table 5: PIW (multiplied by 100) and t-test results for CIFAR-10 classification task.
Class
Accuracy
Accuracy by (-test Status
Correct
Incorrect
Rejected
Not-Rejected (Count)
All
90.95%
2.37
21.52
91.25%
42.86% (63)
91.00%
3.28
18.83
91.51%
45.45% (11)
2
96. 00%
U.55
29.27
96.19%
33.33% (3)
87.30%
2.65
24.40
57.53%
25.00% (4)
31.90%
5.48
21.45
82.10%
63.64% (11)
5
93.30%
2.41
30.02
93.67%
20.00% (5)
6
70%
4.16
19.57
85.21%
46.15% (13)
94.20%%
1.84
26.01
94.38%
33.33% (3)
80%
1.96
19.35
93.07%
25.00% (4)
95.30%
0.56
15.75
95.49%
33.33% (3)
93.00%
1.50
14.04
93.26%
50.00% (6)
We observe from Table 5 that under the scope of the entire test set, the mean PIW of the true class label among the correct predictions is narrower than that of the incorrect predictions by an order of magnitude, indicating that when CARD is making correct predictions, its class label reconstructions have much smaller variations. We may interpret such results as that CARD can reveal what it does not know through the relativity in reconstruction variations. Furthermore, when comparing the mean PIWs across different classes, we observe that the class with a higher prediction accuracy tends to have sharper contrast in true label PIW between correct and incorrect predictions; additionally, the PIW values of both correct and incorrect predictions tend to be larger in a less accurate class. Mcanwhile, worth noting that if we predict the class label by the one with the narrowest PIW for each instance, we can already obtain test accuracy of 87.84%. suggesting strong correlation between the prediction correctness and instance-level model confidence (in terms of label reconstruction variability). Moreover, we observe that the accuracy of test instances rejected by the f-test is much higher than that of the not-rejected ones, both across the entire test set and within each class.
We point out that these metrics can reflect how sure CARD is about the correctness of its predictions, important indicator of whether the model prediction of each instance
and thus be used can be trusted or not. Therefore, it has the potential to be further applied in the human- machiee collaboration domain (Madras al.) 2018 Raghu al. 2019 Wilder et al. 2020 Gao et al. [021] such that one can apply such uncertainty measurement to decide if we can directly accept the model prediction, or we need allocate the instance humans for further evaluation.
5 Conclusion
In this paper. we propose Classification And Regression Diffusion (CARD) models, class of conditional generative models that approaches supervised learning problems from a conditional generation perspective. Without training with objectives directly related to the evaluation metrics, we achieve state-of-the-art results on benchmark regression tasks. Furthermore, CARD exhibits strong ability to represent the conditional distribution with multiple density modes. We also propose a new metric Quantile Interval Coverage Error (QICE), which can be viewed generalized version of negative log- ligelikll in evaluating how well the model fits the data. Lastly, we introduce framework to evaluate prediction uncertainty at the instance level for classification tasks.
Acknowledgments
The authors acknowledge the support of NSF IIS 1812699 and 2212418, and the Texas Advanced Computing Center (TACC) for providing HPC resources that have contributed to the research results reported within this paper.
10

---

References
Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training normalizing flows with the information bottleneck for competitive generative classification. In Proceedings of the 34th Conference on Neural Information Processing Systems, 2020.
Jacob Austin, Daniel Johnson, Jonathan Ho, Danny Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Christopher Bishop. Mixture density networks. In Aston University Neural Computing Research Group Report, 1994.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, 2015.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. In Monthly Weather Review, volume 8, page 1, 1950.
Leda Cosmides and John Tooby. Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature judgment under uncertainty. In cognition, volume 58(1), I), pages 1-73, 1996.
David R. Cox. Prediction by exponentially weighted moving averages and related methods. Journal of the Royal Statistical Society: Series B (Methodological), 23(2):414-422, 1961.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL http:///aacr ics. ucs. edu/ml
Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou. Contextual dropout: An efficient sample-dependent dropout module. In Proceedings of the 9th International Conference on Learning Representations, 2021.
William Feller. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the 1st Berkeley Symposium on Mathematical Statistics and Probability, pages 403-432. University of California Press, 1949.
Ethan Fetaya, Joern-Henrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limitations of conditional generative models. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty deep learning. In Proceedings of the 33rd International Conference on Machine Learning. PMLR, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
Ruijiang Gao, Maytal Tsechask,,,,,,, Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease. Human-AI collaboration with bandit feedback. In Proceedings of the 30th International Joint Conferences on Artificial Intelligence, 2021.
Marta Garnelo, Dan Rosenbaum, Chris Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Tch, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning, 2018a.
Marta Gamelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S.M. Ali Eslami, and Yee Whye Teh. Neural processes. ICMI 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018b.

---

Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In Proceedings of the 8th International Conference Learning Representations, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729- 9738, 2020.
José Miguel Hernández-Looee Lobato and Ryan P. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th Conference on Neural Information Processing Systems, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Per Joachims, nnuncert: Uncertainty quantification with BNNs, 2021, URL httpt::tttttttttiieee com/
Imuncert/nmuncert
Zahra Kadkhodaie and Eero P. Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR 2022 Workshop on Deep Generative Models for Highly Structured Data, 2022.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Durk P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Proceedings of the 29th Conference on Neural Information Processing Systems, 2015.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bit frequentist improves Bayesian neural networks. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pages 2278-2324, 1998. doi: 10.1 109//9997.
Yuanzhi Liang, Linchao Zhu, Xiaohan Wang, and Yi Yang. A simple episodic linear probe improves visual recognition in the wild. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
12

---

Shiao Liu, Xingyu Zhou, Yuling Jiao, and Jian Huang. Wasserstein generative learning of conditional distribution. arXiv preprint arXiv:2112.1000 2021.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with overhead for either training or testing: The all-round blessings of dynamic sparsity. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Proceedings of the 5th International Conference Learning Representations, 2017.
Radek Mackowiak, Lynton Ardizzone, Ullrich Köthe, and Carsten Rother. Generative classifiers as a basis for trustworthy image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In Proceedings of the 32nd Conference on Neural Information Processing Systems, 2018.
Jishnu Mukhoti and Yarin Gal. Evaluating Bayesian deep learning methods for semantic segmentation. arXiv preprint arXiv: 1811.12709 2018
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015.
Radford M Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo. page 113, 2011.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. DiffuseVAE. Efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Learning Research, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd Conference Neural Information Processing Systems, 2019.
Hieu Pham and Quoe V. Le. Autodropout: Learning dropout patterns to regularize deep networks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mul- lainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv: 190.. 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint 2rXhv:2204.06125, 2022.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In Proceedings of the 6th International Conference on Learning Representations, 2018.
Hongyu Ren, Shengjia Zhao, and Stefano Ermon. Adaptive antithetic sampling for variance reduction. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019.
Michael Revow, Christopher KI Williams, and Geoffrey E Hinton. Using generative models for handwritten digit recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 18(6):592-606 1996.
13

---

Tim G. J. Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal. Tractable function- space variational inference Bayesian neural networks. In ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning, 2021.
Jascha Sahl-Dicksteen, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Conference on Machine Leurning. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceed- ings of the 9th International Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- generative models, In Proceedings of the 34th Conference on Neural Information Processing Systems, 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of bosre-based diffusion models. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021b.
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the 9th International Conference on Learning Representations, 2021c
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1955) 2014.
Marcin B. Tomezak, Siddharth Swaroop, Andrew Y. K. Foong, and Richard E. Turner. Collapsed variational bounds for Bayesian neural networks. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- lion, 23(7):1661-1677 2011.
Zhendong Wang and Mingyuan Zhou. Thompson sampling via local uncertainty. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning. PMLR, 2011.
Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Proceedings of the 29th International Joint Conferences on Artificial Intelligence, 2020.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou, A regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09675. 2022.
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Moshi-Velez. Quality of uncertainty quantification for Bayesian neural network inference. ICML. 2019 Workshop on Uncertainty and Robustness in Deep Learning, 2019.
Huangjie Zheng, Xu Chen, Jiangchao Yao, Hongxia Yang, Chunyuan Li, Ya Zhang, Hao Zhang, Ivor Tsang, Jingren Zhou, and Mingyuan Zhou. Contrastive attraction and contrastive repulsion for 2021.
representation learning. arXiv preprini arXiv:2105.033:
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusson probabilistic models and diffusion-based adversarial auto-encoders. arXiv preprint arXiv:2202.0967) 2022.
Xingyu Zhou, Yuling Jiao, Jin Liu, and Jian Huang. A deep generative approach to conditional sampling. Journal of the American Statistical Association, pages 1-28, 2021.
14

---

Roland S. Zimmermann, Lukas Schott, Yang Song, Benjamin A. Dunn, and David A. Klindt. Score-based generative classifiers. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix B
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Appendix and URL Abstract
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "nb6t7kxRpit", "reviewer": "Reviewer_fUR1", "summary": "This paper proposes diffusion-based approach on classification and regression tasks through conditional distribution modeling of y given x.\nThe authors also introduce a new metric for the regression task, called QICE based on PICP, which evaluates distribution similarity in terms of the percentiles.\nThe experiments are conducted on toy dataset, as well as UCI and Fashion MNIST, which show the effectiveness of proposed CARD compared to the baselines.", "questions": "- The major drawbacks of diffusion models are the model and time complexity. How different is the running time compared to the baselines? (both for the training and inference) Also, how different are the model complexities? The current experiment settings are relatively simple, hence, there might not be huge difference. However, when it comes to more complex situation, it can be a huge gap so that the performance is outweighed by its drawbacks.\n- How does the behavior change as the denoising step increases, $t$ from $0$ to $T$? For example, in the case of Figure 1, and histogram or categorical probability change in the classification task.\n- Could you compare CARD against neural processes? Also, I wonder about the quality of extrapolation in the regression task.", "limitations": "The authors have discussed the limitations and potential negative societal impact in the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper discuss fundamental questions in the research community, which are classification and regression. Especially, the authors establish the conditional diffusion-based classification and regression.\n- To overcome the drawback of PICP that it cannot represent the matching of the distribution centrality, the authors proposed a new metric, and The proposed QICE can be viewed as discretized version of NLL.\n- The current version of experiments are relatively simple. While the advantage of diffusion models is that they can generate high-resolution images, it would be informative if the authors conduct experiments on such datasets.\n- The description on the baselines are missing, CMV-MF-VI for example, in Table 5.\n- The error bars are missing in the classification experiment, where the result seems that there is no huge difference among several models. \n- The authors did not provide the code neither in the supplementary material nor as external link.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "hiC7hS0Z3l", "reviewer": "Reviewer_RrLR", "summary": "Diffusion models provide state of the art results in image and video synthesis and this paper extends them to uncertainty quantification for regression and classification problems. The proposed approach extends denoising diffusion probabilistic models to take advantage of a pretrained conditional mean model to form a conditional generative model. Through empirical experiments, the paper shows superior performance of the proposed approach compared to state of the art uncertainty estimation techniques.", "questions": "- The classification extension generates y_0 as real numbers and then a combination of softmax and Brier score converts these generations to probabilities. What are the implications of this modeling assumption? I would also think approaches like discrete diffusion (Austin et al. 2021) can help systematically address the probability simplex concern raised here and I would love to hear authors' thoughts on this.\n- Besides accuracy, classification experiments can benefit from reporting commonly used metrics in uncertainty quantification literature like ECE and NLL, which captures the whole distribution.\n- The experiments are conducted using toy and small scale datasets and models. It would be good to see an extension of this approach to ImageNet style large scale settings.\n- Referring to lines 141-144, the authors mention that PICP cannot capture if the learned quantiles systematically over/under estimate the true distribution. PICP does not have to utilize two sided quantiles. Isn’t this caveat mitigated if one studies the confidence intervals considering one sided quantiles?\n- PICP is a common metric and it would be good to include it for UCI regression tasks. That will also help put the new proposed QICE metric in perspective.\n- Algorithm 2 line 5. You might need to take the square root of \\tilde{\\beta_t}.\n- Table 1 is not very informative and can easily go to an appendix. I would personally prefer to see more analysis on classification with possible extensions to other datasets. At least consider reporting the toy experiment results instead of this table. On a related note, toy experiment results are indicated to be recorded in Table 8 in Appendix A.3 but they are actually in Table 14.\n- Line 239. Typo in “alignment”\n \nAustin, J., Johnson, D.D., Ho, J., Tarlow, D. and van den Berg, R., 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34, pp.17981-17993.\n", "limitations": "The proposed approach makes assumptions on its extensions to classification settings and I believe the manuscript would benefit from a discussion on the implications of these assumptions. Furthermore, experimental results on classification could be improved as discussed in the above section. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The work extends denoising diffusion probabilistic models for regression and classification tasks enabling uncertainty quantification. To my knowledge, this is the first study that attempts to take advantage of diffusion-based generative models for such tasks.\n\nQuality: The paper is technically strong for regression but I have some questions about classification, which I list below. Also I think experimental results would be more compelling if the authors report commonly used metrics like NLL and ECE.\n\nClarity: The paper is well written and organized.\n\nSignificance: Uncertainty quantification is important for safe deployment of machine learning models and the paper advances state of the art by taking advantage of recently rising denoising diffusion models. I believe this is an important contribution to the field.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "awsXskw-DrB", "reviewer": "Reviewer_VuVw", "summary": "The paper presents a new perspective on the supervised learning of predicting a continuous or categorical response variable y given its covariates x.\nSpecifically, it proposes classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of y given x.  The paper compares CARD with Bayesian neural network-based approaches on both toy examples and real-world datasets.", "questions": "The assumption behind\np(y_T ) = N (fφ (x), I ) (6)\nneed clarification. Please explain whether this applies to real-world datasets, e.g. ImageNet.  \"where fφ(x) is pre-knowledge of the relation between x and y0, e.g., pre-trained with D to approximate E[y|x], or 0 if we assume the relation is unknown.\". Does this mean fφ(x) can be computed by a DNN, e.g. ResNet50 for classification?\n\nHow does CARD compare with methods in \"What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?\" NIPS 2017, ( Eqn. 2 for regression uncertainty estimation)?\n\nPlease discuss related work on generative classifiers, e.g. Score-Based Generative Classifiers (https://arxiv.org/abs/2110.00473). \n\nIf the issues can be adequately addressed in the revision, I would be happy to increase my rating from 4 to 5. \n\n=== 8/7/2022 after rebuttal\n\nI appreciate the authors' effort to address my comments. It clarified a few questions. I am happy to increase my rating to 5. ", "limitations": "The limitations are not adequately stated. It only mentions that computation efficiency of CARD may also be further investigated if the dataset size becomes larger. While the preliminary empirical results look promising, in-depth experiments on typical real-world datasets, e.g. ImageNet needs to be studied. The authors need to connect with the literature of generative classifiers. Furthermore, there is a lack of insights or theoretical justification on the performance of CARD over discriminative models. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\nThe paper reformulates the classical P(y | x): given the ground-truth response variable y0 and its covariates x, and assuming a sequence of intermediate uncertain prediction y1:T made by the diffusion model. It constructs εθ (x, yt , fφ (x), t), which is a function approximator parameterized by a deep neural network that predicts the forward diffusion noise ε sampled for yt. The training and inference procedure can be carried out in a standard DDPM manner.\n\nThe paper presents derivation for forward process posteriors and derivation for forward process sampling distribution with arbitrary time steps.\n\nCARD is compared with MC Dropout, Gaussian Dropout, Concrete Dropout, Bayes by Backprop, and variants of Contextual Dropout on toy datasets, UCI regression tasks and Fashion MNIST classification tasks,\n\nWeaknesses\n\nThe paper does not provide insights or theoretic justification on the reason that CARD can help accurately recovery p(y | x, D), the predictive distribution of y given x after observing data D. \n\nThe paper also does not offer insights on why CARD outperforms other Bayesian NN approaches. \n\nThe datasets used are very limited. The authors should consider real-world datasets such as ImageNet.\n\nThe paper should put their work in the context of related work better. This would require a related work section which is missing. For example, generative classifiers are not discussed. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ApbRu4wybcv", "reviewer": "Reviewer_8bC3", "summary": "This work applies diffusion models to discriminative task. The diffusion model is used to model the uncertainty of $y$. The author designs a special forward process to incorporate a pretrained discriminative model, which is the main distinction from prior diffusion models. Experiments valid that the CARD can model the uncertainty of the prediction.", "questions": " ", "limitations": " ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. It is an interesting idea to apply diffusion models to discriminative tasks, due to its strong capacity to model the uncertainty.\n\n2. The incorporation of a pretrained discriminative model seems a nice design.\n\nWeakness:\n1. The main difference between prior conditional diffusion models is that the transition in the forward process is related to the condition, i.e., use $q(y_t|y_{t-1}, x)$ instead of $q(y_t|y_{t-1})$. What will happen if we use $q(y_t|y_{t-1})$? Perhaps the author can consider $q(y_t|y_{t-1})$ as a baseline and make some comparison.\n\n2. In Section 3.2.2, I didn't find a comparison of different methods on the classification uncertainty. While the author presents some uncertainty result of CARD, it is important to compare it with other methods. For example, a tranditional classifer can also capture the uncertainty, since it outputs the probability values of each class. These probabilities can be used for uncertainty evaluation. What is the uncertainty evaluation performance of CARD when comparing with the traditional classifier? I suggest the author run experiments in Table 6 for some other methods, and make a comparison.\n\n3. The datasets used in this experiment are simple. How will CARD perform when scaling up to high resolution inputs? For example, CIFAR10 and ImageNet.\n\n\nTypos:\n\n1. In Eq.(1), $p_\\theta(x, y_0)$ should be $p_\\theta(y_0|x)$, and $q(y_{1:T}|x)$ should be $q(y_{1:T}|y_0, x)$.\n\n2. The ELBO in Eq.(2) is also related to $y_0$, thus should be denoted as $L_{ELBO}(x, y_0)$ instead of $L_{ELBO}(x)$.\n\n3. Eq.(6) is related to $x$, thus should be denoted as $p(y_T|x)$.\n\n4. Eq.(9) is related to $y_0$, thus should be denoted as $q(y_{t-1}|y_t, y_0, f_\\phi(x))$.\n\n5. In L3 in Algorithm 1, $q(y_0)$ should be $q(y_0|x)$.\n\n6. The $\\epsilon_\\theta$ between L6 and L7 in Algorithm 1 missed $f_\\phi(x)$ as its input.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept