Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes diffusion-based approach on classification and regression tasks through conditional distribution modeling of y given x.
The authors also introduce a new metric for the regression task, called QICE based on PICP, which evaluates distribution similarity in terms of the percentiles.
The experiments are conducted on toy dataset, as well as UCI and Fashion MNIST, which show the effectiveness of proposed CARD compared to the baselines.
strengths_and_weaknesses: - The paper discuss fundamental questions in the research community, which are classification and regression. Especially, the authors establish the conditional diffusion-based classification and regression.
- To overcome the drawback of PICP that it cannot represent the matching of the distribution centrality, the authors proposed a new metric, and The proposed QICE can be viewed as discretized version of NLL.
- The current version of experiments are relatively simple. While the advantage of diffusion models is that they can generate high-resolution images, it would be informative if the authors conduct experiments on such datasets.
- The description on the baselines are missing, CMV-MF-VI for example, in Table 5.
- The error bars are missing in the classification experiment, where the result seems that there is no huge difference among several models. 
- The authors did not provide the code neither in the supplementary material nor as external link.
questions: - The major drawbacks of diffusion models are the model and time complexity. How different is the running time compared to the baselines? (both for the training and inference) Also, how different are the model complexities? The current experiment settings are relatively simple, hence, there might not be huge difference. However, when it comes to more complex situation, it can be a huge gap so that the performance is outweighed by its drawbacks.
- How does the behavior change as the denoising step increases, $t$ from $0$ to $T$? For example, in the case of Figure 1, and histogram or categorical probability change in the classification task.
- Could you compare CARD against neural processes? Also, I wonder about the quality of extrapolation in the regression task.
limitations: The authors have discussed the limitations and potential negative societal impact in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Diffusion models provide state of the art results in image and video synthesis and this paper extends them to uncertainty quantification for regression and classification problems. The proposed approach extends denoising diffusion probabilistic models to take advantage of a pretrained conditional mean model to form a conditional generative model. Through empirical experiments, the paper shows superior performance of the proposed approach compared to state of the art uncertainty estimation techniques.
strengths_and_weaknesses: Originality: The work extends denoising diffusion probabilistic models for regression and classification tasks enabling uncertainty quantification. To my knowledge, this is the first study that attempts to take advantage of diffusion-based generative models for such tasks.

Quality: The paper is technically strong for regression but I have some questions about classification, which I list below. Also I think experimental results would be more compelling if the authors report commonly used metrics like NLL and ECE.

Clarity: The paper is well written and organized.

Significance: Uncertainty quantification is important for safe deployment of machine learning models and the paper advances state of the art by taking advantage of recently rising denoising diffusion models. I believe this is an important contribution to the field.

questions: - The classification extension generates y_0 as real numbers and then a combination of softmax and Brier score converts these generations to probabilities. What are the implications of this modeling assumption? I would also think approaches like discrete diffusion (Austin et al. 2021) can help systematically address the probability simplex concern raised here and I would love to hear authors' thoughts on this.
- Besides accuracy, classification experiments can benefit from reporting commonly used metrics in uncertainty quantification literature like ECE and NLL, which captures the whole distribution.
- The experiments are conducted using toy and small scale datasets and models. It would be good to see an extension of this approach to ImageNet style large scale settings.
- Referring to lines 141-144, the authors mention that PICP cannot capture if the learned quantiles systematically over/under estimate the true distribution. PICP does not have to utilize two sided quantiles. Isn’t this caveat mitigated if one studies the confidence intervals considering one sided quantiles?
- PICP is a common metric and it would be good to include it for UCI regression tasks. That will also help put the new proposed QICE metric in perspective.
- Algorithm 2 line 5. You might need to take the square root of \tilde{\beta_t}.
- Table 1 is not very informative and can easily go to an appendix. I would personally prefer to see more analysis on classification with possible extensions to other datasets. At least consider reporting the toy experiment results instead of this table. On a related note, toy experiment results are indicated to be recorded in Table 8 in Appendix A.3 but they are actually in Table 14.
- Line 239. Typo in “alignment”
 
Austin, J., Johnson, D.D., Ho, J., Tarlow, D. and van den Berg, R., 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34, pp.17981-17993.

limitations: The proposed approach makes assumptions on its extensions to classification settings and I believe the manuscript would benefit from a discussion on the implications of these assumptions. Furthermore, experimental results on classification could be improved as discussed in the above section.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a new perspective on the supervised learning of predicting a continuous or categorical response variable y given its covariates x.
Specifically, it proposes classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of y given x.  The paper compares CARD with Bayesian neural network-based approaches on both toy examples and real-world datasets.
strengths_and_weaknesses: Strengths 

The paper reformulates the classical P(y | x): given the ground-truth response variable y0 and its covariates x, and assuming a sequence of intermediate uncertain prediction y1:T made by the diffusion model. It constructs εθ (x, yt , fφ (x), t), which is a function approximator parameterized by a deep neural network that predicts the forward diffusion noise ε sampled for yt. The training and inference procedure can be carried out in a standard DDPM manner.

The paper presents derivation for forward process posteriors and derivation for forward process sampling distribution with arbitrary time steps.

CARD is compared with MC Dropout, Gaussian Dropout, Concrete Dropout, Bayes by Backprop, and variants of Contextual Dropout on toy datasets, UCI regression tasks and Fashion MNIST classification tasks,

Weaknesses

The paper does not provide insights or theoretic justification on the reason that CARD can help accurately recovery p(y | x, D), the predictive distribution of y given x after observing data D. 

The paper also does not offer insights on why CARD outperforms other Bayesian NN approaches. 

The datasets used are very limited. The authors should consider real-world datasets such as ImageNet.

The paper should put their work in the context of related work better. This would require a related work section which is missing. For example, generative classifiers are not discussed. 

questions: The assumption behind
p(y_T ) = N (fφ (x), I ) (6)
need clarification. Please explain whether this applies to real-world datasets, e.g. ImageNet.  "where fφ(x) is pre-knowledge of the relation between x and y0, e.g., pre-trained with D to approximate E[y|x], or 0 if we assume the relation is unknown.". Does this mean fφ(x) can be computed by a DNN, e.g. ResNet50 for classification?

How does CARD compare with methods in "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?" NIPS 2017, ( Eqn. 2 for regression uncertainty estimation)?

Please discuss related work on generative classifiers, e.g. Score-Based Generative Classifiers (https://arxiv.org/abs/2110.00473). 

If the issues can be adequately addressed in the revision, I would be happy to increase my rating from 4 to 5. 

=== 8/7/2022 after rebuttal

I appreciate the authors' effort to address my comments. It clarified a few questions. I am happy to increase my rating to 5. 
limitations: The limitations are not adequately stated. It only mentions that computation efficiency of CARD may also be further investigated if the dataset size becomes larger. While the preliminary empirical results look promising, in-depth experiments on typical real-world datasets, e.g. ImageNet needs to be studied. The authors need to connect with the literature of generative classifiers. Furthermore, there is a lack of insights or theoretical justification on the performance of CARD over discriminative models.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work applies diffusion models to discriminative task. The diffusion model is used to model the uncertainty of $y$. The author designs a special forward process to incorporate a pretrained discriminative model, which is the main distinction from prior diffusion models. Experiments valid that the CARD can model the uncertainty of the prediction.
strengths_and_weaknesses: Strength:
1. It is an interesting idea to apply diffusion models to discriminative tasks, due to its strong capacity to model the uncertainty.

2. The incorporation of a pretrained discriminative model seems a nice design.

Weakness:
1. The main difference between prior conditional diffusion models is that the transition in the forward process is related to the condition, i.e., use $q(y_t|y_{t-1}, x)$ instead of $q(y_t|y_{t-1})$. What will happen if we use $q(y_t|y_{t-1})$? Perhaps the author can consider $q(y_t|y_{t-1})$ as a baseline and make some comparison.

2. In Section 3.2.2, I didn't find a comparison of different methods on the classification uncertainty. While the author presents some uncertainty result of CARD, it is important to compare it with other methods. For example, a tranditional classifer can also capture the uncertainty, since it outputs the probability values of each class. These probabilities can be used for uncertainty evaluation. What is the uncertainty evaluation performance of CARD when comparing with the traditional classifier? I suggest the author run experiments in Table 6 for some other methods, and make a comparison.

3. The datasets used in this experiment are simple. How will CARD perform when scaling up to high resolution inputs? For example, CIFAR10 and ImageNet.


Typos:

1. In Eq.(1), $p_\theta(x, y_0)$ should be $p_\theta(y_0|x)$, and $q(y_{1:T}|x)$ should be $q(y_{1:T}|y_0, x)$.

2. The ELBO in Eq.(2) is also related to $y_0$, thus should be denoted as $L_{ELBO}(x, y_0)$ instead of $L_{ELBO}(x)$.

3. Eq.(6) is related to $x$, thus should be denoted as $p(y_T|x)$.

4. Eq.(9) is related to $y_0$, thus should be denoted as $q(y_{t-1}|y_t, y_0, f_\phi(x))$.

5. In L3 in Algorithm 1, $q(y_0)$ should be $q(y_0|x)$.

6. The $\epsilon_\theta$ between L6 and L7 in Algorithm 1 missed $f_\phi(x)$ as its input.
questions:  
limitations:
review_title: null

Meta-review: metareview: This paper proposes a different approach to uncertainty quantification in regression and classification problems. It extends denoising diffusion probabilistic models in combination with a pre-trained conditional mean model to provide a conditional generative model. The proposed approach is evaluated on UCI regression tasks and CIFAR10 classification. Additional results on CIFAR100 and ImageNet were presented during the rebuttal period and additional analyses. 

The paper received some (relatively weak) support among reviewers. All the reviewers agree the paper proposes an interesting and novel formulation of uncertainty-quantified regression and classification models. The main issues raised by the reviewers were that of (i) conducting experiments on larger datasets, (ii) more analysis in the classification case, and (iii) the effect of conditioning on x in the diffusion processes. I believe the authors have provided additional significant empirical evidence of the benefits of their approach and some of the reviewers updated their scores accordingly. Given this, the potential impact of the paper, the novelty nature of the contribution, and despite the seemingly weak scores, I believe there are no significant concerns remaining about this paper and it is worth presenting in its current form to the NeurIPS community. 

confidence: Less certain
Decision: Accept