Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to improve diffusion probabilistic models by considering multivariate diffusions. The motivation inherits previous probabilistic modeling methods that use auxiliary variables. By augmenting the diffusion space, it enables mixing between data dimensions and auxiliary dimensions, leading to better-aligned generative and inference processes. Based on this the paper further demonstrates how to parameterize the inference process, e.g., MALDA. Experimental results are promising in terms of test likelihood estimation.
strength_and_weaknesses: Strength:

- The paper is well-written and introduces an interesting novel approach for establishing diffusion probabilistic models in auxiliary space.
- The motivation of the work is explained very well in Section 1 and Section 4, which is helpful for understanding the paper.
- Technical contributions are solid (to the best of my knowledge).
- Experimental results achieve state-of-the-art likelihood (in terms of bits-per-dim).

Questions:
- How is the auxiliary variable initialized in time zero? Is it sampled from $q_\phi(\mathbf{y}^v_0|x)$ during training? It seems that $q_\phi(\mathbf{y}^v_0|x)$ can be any distribution in theory. In the paper, an example of $q_\phi(\mathbf{y}^v_0|x)=\mathcal{N}(\mathbf{0},\mathbf{I})$ is given, but this could be quite different from the data distribution, i.e., $q_\phi(\mathbf{y}^z_0|x)$. Since in the practical implementation the score for each dimension is output from different channels from the same model, I wonder whether a better choice of $q_\phi(\mathbf{y}^v_0|x)$ could alleviate the training burden and perhaps results in a better likelihood or faster training?

Minors:
- In Eq.(18) no text explanation under the underbrace.
- On page 6 there is something wrong with the typography.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the technical part is of high quality. The novelty is good. The experiments seem reproducible.
summary_of_the_review: Overall, I think this paper is of good quality, in that it presents a novel and interesting perspective of understanding and a concrete approach for diffusion models, achieving good results on standard benchmarks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces Multivariate Diffusion Models which extend standard diffusion models to include auxiliary variables. Towards this end, the authors provide a new learnable inference and generative process that generalizes many familiar diffusion SDEs (e.g. VP) and provides a modified ELBO required for training. The authors also introduce two linear diffusion models in ALDA and MALDA and show that these lead to state-of-the-art BPD and NLL on MNIST and CIFAR10.
strength_and_weaknesses: First, the paper is exceptionally well written, and the main ideas and motivation are clear and convincing. I really found the idea to include auxiliary variables as a compelling step forward given the numerous examples previously in machine learning (e.g. Augmented normalizing flows). I was initially concerned that having $K$ variables per dimension might incur a high computational cost but table 1 suggests otherwise. 

From a theory standpoint, I must admit I did not have time to go through the details in the appendix but the items presented in the main paper appeared sound. The main insights of using linear diffusion and computing the transition kernels (eq 15-16) felt a bit opaque to me in the main text but I understand this is fleshed out more in Appendix C. It would be nice if the authors could move some of this material to the main text as well to help readability. The fixed inference parameterization and its extension to a learnable parameterization sounds reasonable, but as a question it appears that we get this for free? So are all MDM models have learnable inference processes by construction? It would be nice to see an ablation on this as table 2 is not very clear on this point.

I did not find many clear weaknesses in this paper. But I will outline some areas of improvement. First tables 2 and 3 can really be combined into one table, the results are identical for MDMs. Secondly, the experiments are a bit limited as they only include MNIST and CIFAR10. Most diffusion papers also include a larger dataset in Imagenet 32 X 32 or Imagenet 64 X 64. It would be good to see if MDMs are equally performant in this setting. Also, it would be nice to get a convergence vs. timestep analysis on ELBO for MDMs vs other diffusion models. Do MDMs converge faster due to higher modeling flexibility? Finally, it appears ALDA is significantly worse than MALDA? The intuition for this is not very clear to me, can the authors provide more details here?


clarity,_quality,_novelty_and_reproducibility: As I mentioned above the paper is exceptionally well-written and polished. The quality of the presented theory is very high and appears to be thorough based on a quick skim of the appendix. With regards to novelty, the paper is certainly novel but leans on similar themes in past ML research in using auxiliary variables to improve modeling flexibility. 

Minor:
- minor grammatical errors throughout (e.g. "by compute objectives" in the hybrid score matching section)
summary_of_the_review: Overall this paper is of high quality and has a nice theoretical component. The experiments are limited to small datasets but there does not appear to be a significant bottleneck in scaling them. Based on this I recommend accepting this paper with a score 8.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper provides a general derivation of Evidence Lower Bounds in Multivariate Diffusion Models and some parametrization schemes. The paper also introduces MALDA diffusion model under its framework and evaluates it on MNIST and CIFAR10.
strength_and_weaknesses: ## Strength:

The derivation seems to be unfied and generic. The experimental results seem good.

## Weakness:

### Novelty:

The novelty of the paper is marginal: 

(1) For the theoretical derivations, all the results are readily known from previous papers (e.g., Huang et al. (2021); Durkan & Song (2021), Dockhorn et al., (2021)). The paper is just summarization of the previous derivations using a generic framework. 


(2) The empirical results are not novel and lack motivation: ALDA is introduced in Mou et al. (2019). The modified version (MALDA) is not well-motivated (at least not shown in the main text) and only marginally different from the ALDA method.


### Clarity:

The paper is a ill-structured and unclear. 

(1) Most of the details are deferred to the Appendix, making the paper hard to follow. 

(2) The theoretical part of the paper is not structured. It is written as derivations of formula without clear goals. Assumptions are added to the derivations in the middle, e.g. Section 3.3 assumptions on $f(A,s)$; Section 3.4. linear case. Instead, the best way to present theoretical results is through a list of assumptions and theorems. This makes follow-up papers easy to verify the assumptions and use the results without knowing the details of the derivations. (Same philosophy as abstracting codes through APIs, if the authors are more familiar with coding and engineering.)

(3) The experimental part lacks motivations and exposition of the proposed method. E.g., what is the difference between ALDA and MALDA? what is the motivation for MALDA? Do you claim ALDA as your own contribution, as in the abstract and introduction section; or do you give full credit to  Mou et al. (2019)?




clarity,_quality,_novelty_and_reproducibility: Please see above.
summary_of_the_review: The paper is not novel and ill-structured.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Ratings: 8/8/3.
Confidences: 3/3/4.

In this work, the authors study the (fully?) general case of diffusion models, where the forward diffusion process and its reverse are parameterized by non-diagional covariances for the diffusion term; i.e. multivariate diffusion models (MDMs). In particular, the authors look at the case where auxiliary dimensions are included. Their contributions are (1) deriving ELBOs for training multivariate diffusion models with auxiliary variables, (2) Showing that the diffusion transition covariance does not need to be manually derived for each new diffusion, (3) construct a family of diffusion models whose q(z_T) is a spherical Gaussian, (4) combining the above into an algorithm called Automatic Multivariate Diffusion Training (AMDT) that enables estimation without diffusion-specific derivations. In experiments, the authors show that these innovations with multivariate diffusions provide some performance gain.

This appears to be a theoretically strong paper. Neither me nor the reviewers could find major problems with the main results. I think the main results could have significant impacts on the community. 

However, some reviewers were unhappy about the presentation. The empirical section also has some weaknesses. The experiments are relatively small-scale, and the gains in likelihood are quite small. In addition, I would have liked to see stronger baselines. For example, the authors use a diffusion model baseline that gets ~3.1 BPD on CIFAR-10, which is far from SOTA; a more logical choice would have been Variational Diffusion Models [Kingma et al, 2021] which gets 2.65 BPD on the same dataset (whose code is available). Why do the authors choose such a sub-par baseline? Most importantly, it's currently not completely clear if the proposed methods can help at all getting better likelihoods in such a regime.

The authors provided a rebuttal to all reviews, but only bfsX engaged, which also gave the only reject rating in the end. However, we feel that the authors did a good job with his rebuttal. Ultimately, we think this outweighed by the positive reviews, and the positive reading by myself.

justification_for_why_not_higher_score: The empirical section is somewhat weak.
justification_for_why_not_lower_score: The strong theoretical section makes up for weaknesses in the empirical section.
Decision: Accept: poster