A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models

Authors: Shivam Kumar, Yun Yang, Lizhen Lin
Primary area: learning theory
Keywords: Deep Generative Model, Conditional Distribution, Smoothness Disparity

Abstract:
In this work, we explore the theoretical properties of conditional deep generative models under the statistical framework of distribution regression where the response variable lies in a high-dimensional ambient space but concentrates around a potentially lower-dimensional manifold. More specifically, we study the large-sample properties of a likelihood-based approach for estimating these models. Our results lead to the convergence rate of a sieve maximum likelihood estimator (MLE) for estimating the conditional distribution (and its devolved counterpart) of the response given predictors in the Hellinger (Wasserstein) metric. Particularly, our rate depends solely on the intrinsic dimension and smoothness of the true conditional distribution. These findings provide an explanation of why conditional deep generative models can circumvent the curse of dimensionality from the perspective of statistical foundations and demonstrate that they can learn a broader class of nearly singular conditional distributions. Our analysis also emphasizes the importance of introducing a small noise perturbation to the data when they are supported sufficiently close to a manifold. Finally, in our numerical studies, we demonstrate the effective implementation of the proposed approach using both synthetic and real-world datasets, which also provide complementary validation to our theoretical findings.

Decision: reject
Reviewer scores: 6, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729125873134

thread_type:
official_review

<thread_object>
creator: reviewer_vl1u

<content>
summary:
This paper aims at providing statistical bounds for conditional generative models under various smoothness assumptions. The overall goal is to model conditional distributions $Y|X$ via conditional generative models, i.e., $y = G(x)_{*}P_Z$ of some latent distribution $P_Z$. First empirical convergence rates of so-called sieve MLE estimators are provided (Theorem 1), showing that as paired data tends to infinity the approximation of the measure tends to zero in probability.  Then also the Wasserstein distance between the pushforward distribution of the MLE estimate and the true generated distribution can be controlled via a pertubation strategy that depends on the data dimension and smoothness. Then with some ansatz function class spaces for the true generating process, a convergence theorem for sieve MLE with generating functions in some sparse NN class is presented. 

Finally, the sieve MLE is validated on some toy examples which is learned via a VAE and on the MNIST dataset.

strengths:
The paper tackles a very interesting topic. There has not been much theory of conditional generative models, in particular not on their statistical rates with respect to the true smoothness and intrinsic dimension. 

Theorem 1 presents a very nice approach on how to tackle these problems, which allows for splitting these problems up into the covering number and the accuracy of the approximation. This gives a nice guideline: Pick a function class that has a good covering scaling and good approximation capabilities. In particular the noise injection strategy seems very neat in corollary 2, if the data distribution is close to a singular one. Generally, the writing and the maths is very rigoruous, with clearly defined objects up to a few exceptions (see weaknesses). It is also nice that the qualitiative behaviour of the theorems is verified in understandable toy examples.

weaknesses:
First, let me say, that I come from a background in conditional generative models, not so much in statistical theory. Therefore I had difficulties following some claims in the paper. 

1) First, when I think of where conditional generative models are actually used, I am confused about the whole modelling assumption in section 2.1. Usually the latent space Z should take care of the uncertainty in Y, so that T(x,Z) models this perfectly. of course for trained models there will still be a mismatch, but treating this as an independent Gaussian seems very unrealistic. Can you elaborate this assumption maybe taking the example of inverse problems?

Can you maybe find for lets say a normalizing flow always a similar regular normalizing flow with a small gaussian noise added to it such that it becomes the "real" generative model? This would increase my confidence in this assumption.

2) It would be good to have a running example, something super simple like taking the pushforward of a Gaussian under a linear operator (with non trivial rank) to explain the rates and corollaries. 

3) I am worried about the scaling in corollary 1 for the $\sigma^{*}$, basically this means that if the noise level tends to zero, that the approximation accuracy has to increase by a lot. This might be due to the choice of the Hellinger distance. Why not try to control the Wasserstein distance in equation 7 in the first space? Since $W_1(p, p \star \mathcal{N}(0,\sigma))$ are close for small $\sigma$ this might alleviate this problem.

4) I feel like section 2.4 is not rigorous in the sense that it is very unclear if the article looks at joint measures of $(X,Y)$ or conditional measures $Y|X=x$. Looking at the proofs I think it holds for the condiitonals and arbitrary $X=x$? ´

5) For the experiments part, it would be very enlightening to consider manifolds with a known dimension (and maybe make them conditional) to validate how the intrinsic dimension comes into play [1]. Further, for the MNIST example, one does not need to take the MNIST dataset as the ground truth, but rather one can take a pretrained generator as the ground truth and then validate how smoothness and approximation error impact the Wasserstein/FID/... between the pretrained generator and a learned model. This would be much more in spirit with validating the bounds, since the point of this paper is not to come up with a practical algorithm I presume. 

6) Can the in probability bounds be strengethened to expectation bounds over x between the conditional measures under stronger assumptions, see also [4].Conditional Wasserstein Distances with Applications in Bayesian OT Flow Matching

7) Also, one algorithm where the composition approach might come into play is flow matching/diffusion with odes [2]. Here, if one takes a simple Euler scheme to sample one indeed has such a simple compositional structure. I would find it very interesting to briefly state the relation to those, as they are bit more SOTA and practical interest than VAEs at the moment. 

8) Can you make a practical example where the smoothness disparity is obvious?

9) This paper is missing some citations on generative models and manifolds [1,3], and the relation between conditional measures and joint measures [4] and also in what sense conditional generative models optimize the MLE estimator (somewhat clear for NFs and VAEs, less clear for diffusion/...). Here I am thinking along the lines of [5]

Overall, I am mainly missing practical takeaways and a running example where I can see that the assumptions made are indeed practical. Indeed, also the modelling assumptions remain unclear, and would greatly benefit from more explanations. I feel like the whole paper can be a bit streamlined to make it more digestible for the learning community as opposed to the statistics community, but overall I appreciate the deep theoretical analysis if my points can be addressed. 

[1] Diffusion Models Encode the Intrinsic Dimension of Data Manifolds, ICML 

[2] Flow Matching for Generative Modeling, ICLR 

[3] Score-Based Generative Models Detect Manifolds, NeurIPS

[4] Conditional Wasserstein Distances with Applications in Bayesian OT Flow Matching, arXiv

[5] Maximum Likelihood Training of Score-Based Diffusion Models, NeurIPS

questions:
see the weaknesses, they are a bit intertwined.

POST REBUTTAL: Concerns have been mostly addressed. raising score from 6 to 8.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594572080

thread_type:
official_review

<thread_object>
creator: reviewer_qxt2

<content>
summary:
The authors describe a method for estimating high dimensional conditional distributions that concentrate around a manifold. Such distributions appear in common settings such as conditional generative modeling. The authors consider the convergence of an estimator that improves based on the number of data points to the true distribution in the Hellinger distance sense. The authors then consider applying this estimator to some synthetic and real datasets to understand its empirical performance. The analysis considers two types of neural networks: a sparse neural network and fully connected one that parameterize the push forward of the latent variable to the conditional outcome. The analysis is heavily dependent on empirical process theory to establish different properties of the architectures.

strengths:
The proposed analysis is helpful in understanding why generative models work as they do. Additionally the proposed error analysis is useful for anticipating the number of data points needed to achieve a specific targeted error with respect to the Wasserstein or Hellinger distances. 

The empirical results support the main claims of the work to some extent.

weaknesses:
Some of the theoretical results seem hard to verify in practice. It would be interesting for some contrived examples if the authors can show the convergence of the distances as a function of the number of points. 

The theorems contain some constants that do not have a clear way of being estimated making some of the errors hard to use in practice. 

Note that the paper does not appear to be in the right ICLR format since the margins are considerably smaller.

questions:
How easy are some of the conditions to verify for the convergence theorems to hold?

How do you interpret the constants in the various theoretical statements? For example, some are dependent on the domain or other parameters, in what sense do they grow for perturbations of their dependencies? Can these be estimated in practice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700324801

thread_type:
official_review

<thread_object>
creator: reviewer_qywh

<content>
summary:
This paper studies using neural network generative models for distribution regression. Statistical rates are established, and a significant effort is devoted to resolving the curse of dimensionality issue. Numerical results are provided to support the theory.

strengths:
The theoretical results appear to be correct and sound. Moreover, there are detailed analyses for different setups and matching lower bound.

There are numerical results to support the performance of neural network generative models for distribution regression.

weaknesses:
The paper is not very easy to follow in places, mostly due to the technical presentations and pure statistical discussions. I would suggest authors to prepare a preliminary section before mixing background introduction with main results in current Section 2.

It is relatively difficult for me to judge the technical novelty of the paper. On the one hand, the distribution regression problem and low-dimensional generative neural networks seem to pose technical difficulties. On the other hand, the results as well as most of the assumptions are replication of many existing works, such as Schmidt-Hieber, 2020 and Tang and Yang, 2023.

Practical impact of this work may be limited, due to many technical assumptions.

Numerical results loosely connect to the theoretical results.

The paper is not prepared in the standard ICLR style --- the left and right margins are altered significantly. Under the standard style template, this submission will go beyond the page limit.

questions:
Do numerical results support the rates proved in theory? Maybe we can take logarithm on the sample size and inspect the slope of the error.

Assumption 3 excludes interior points. Does this assumption hold (approximately) in practical applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721421207

thread_type:
official_review

<thread_object>
creator: reviewer_uhnp

<content>
summary:
A statistical analysis of conditional deep generative models using a likelihood approach (specifically a sieve MLE) is proposed and analyzed. A convergence rate for the estimator is established using neural network approximation theory for the (conditional) generator. Numerical examples are provided showing the efficacy of the method.

strengths:
1. Very well written paper
2. Method is clearly explained and proof are followable

weaknesses:
1. The method seems to be extending the techniques of previous work and filling in some details with approximation.
2. Question on the applicability to multiple chart manifolds.
3. Numerical examples are very simple.

questions:
1. In regards to the discussion in 2.4.3 and Appendix C with the multiple manifold case: If a partition of unity is used to define local transport maps, we require that the local distributions on each chart need to have density lower bounded. However, a partition of unity will have its weights vanish to 0 on the boundary, so the induced local distribution would have density that vanishes to 0 on the boundary. How can you apply optimal transport regularity theory in that case, when your target has density that is not lower bounded?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732307562865

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to sincerely thank all four reviewers for their time and thoughtful feedback. Their insightful comments have significantly improved the quality of our revised manuscript. We also appreciate their recognition of the strength and clarity of our theoretical results, as well as the accessibility of the accompanying proofs.

First and foremost, we apologize for the formatting issue in the original submission. This oversight stemmed from the use of `usepackage\{fullpage}`. To address this while adhering to the page limit restrictions, we have made the following updates to the manuscript: 

- Moved the MNIST dataset numerical results to the appendix.
- Adjusted formatting for readability and compactness, particularly in lines `456–460` and `489–493`.
- Verified that the final version of the manuscript, including the author names, is well within the 10-page limit.
- Corrected the proof in the multiple-chart manifold case (lines `893-917`) to ensure the transported density remains lower bounded.

Additionally, we conducted further numerical studies to address some reviewers' comments, specifically analyzing how the $W_1$ distance varies with sample size, while keeping the noise level fixed at $\sigma_* = 0.01$. The experimental setup is consistent with the manifold case described in the manuscript. Below is a summary of the empirical results:

| Sample Size | Two Moon (with $\sigma_* = 0.01$) | Ellipse (with $\sigma_* = 0.01$) |
|-------------|----------------------------------|--------------------------------|
| 4000        | 0.251                            | 0.295                          |
| 6000        | 0.232                            | 0.285                          |
| 7000        | 0.216                            | 0.271                          |
| 8000        | 0.214                            | 0.253                          |
| 9000        | 0.212                            | 0.259                          |
| 10000       | 0.196                            | 0.251                          |

Although exact rates are difficult to extract, these results validate the large-sample properties for manifolds, aligning well with our theoretical expectations. This further supports the consistency and convergence trends of our framework.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a theoretical analysis of conditional deep generative models, focusing on estimating high-dimensional conditional distributions that concentrate around manifolds. The authors provide statistical guarantees and convergence rates for different network architectures and present empirical results to support their theoretical claims.

The main strength of the paper lies in its attempt to provide a theoretical foundation for conditional generative models, offering convergence rates and error bounds for different network architectures. However, the paper suffers from several weaknesses. The practical implications and immediate impact on deep learning applications are limited. Many of the assumptions and constants in the theoretical framework are difficult to verify or estimate in practice. Moreover, the theoretical contribution appears to primarily combine existing work without offering significant new insights.

The decision to reject is primarily based on the limited novelty and practical impact of the work. While the paper provides a theoretical analysis, it does not offer substantial new insights or immediate practical applications in deep learning. The theoretical results largely align with expected outcomes and do not demonstrate significant deviations from existing work. Additionally, the initial submission's violation of formatting guidelines raises concerns about adherence to conference standards.