On Statistical Rates of Conditional Diffusion Transformers: Approximation,  Estimation and Minimax Optimality

Authors: Jerry Yao-Chieh Hu, Weimin Wu, Yi-Chen Lee, Yu-Chao Huang, Minshuo Chen, Han Liu
Primary area: generative models
Keywords: Conditional Diffusion Transformer, Statistical Rates, Approximation, Estimation

Abstract:
We investigate the approximation and estimation rates of conditional diffusion transformers (DiTs) with classifier-free guidance. We present a comprehensive analysis for “in-context” conditional DiTs under various common assumptions: generic and strong Hölder, linear latent (subspace), and Lipschitz score function assumptions. Importantly, we establish minimax optimality of DiTs by leveraging score function regularity. Specifically, we discretize the input domains into infinitesimal grids and then perform term-by-term Taylor expansions on the conditional diffusion score function under the Hölder smooth data assumption. This enables fine-grained use of transformers’ universal approximation through a more detailed piecewise constant approximation, and hence obtains tighter bounds. Additionally, we extend our analysis to latent settings. Our findings establish statistical limits for DiTs and offer practical guidance toward more efficient and accurate designs.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730589025820

thread_type:
official_review

<thread_object>
creator: reviewer_1naw

<content>
summary:
This paper mathematically investigates the approximation and estimation properties of conditional diffusion Transformers (DiTs) with classifier-free guidance. Theoretical connections between conditional DiTs and their unconditional latent variants are established for data distributions with certain smoothness.

strengths:
1. This paper seems mathematically sound. 
2. The settings studied in this work are abundant and quite complex. 
3. The roadmaps of proofs are clear.

weaknesses:
1. It seems that some estimates scale poorly in the asymptotic sense, particularly in high-dimensional and long sequences settings (see details in the "Questions" section). 
2. Without verifications by numerical simulations, one cannot guarantee that the derived approximation and estimation estimates can be achieved in practice, i.e. whether these theoretical constructions can be found through concrete training dynamics. Is it possible to numerically verify (partial of) the dependence in these mathematically derived upper bounds?

questions:
1. For the score approximation: 
- $\sigma_t^2 \to 0$ when $t \to 0$, leading to infinitely large upper bounds (Table 1).
- $\epsilon^{1/d} \to 1$ when $d \to \infty$, leading to $O(1)$ upper bounds (Table 1) that are vacuous in high-dimensional settings. 
- The upper bound is *exponentially* large given high-dimensional inputs, due to the multiplicative term $(\log N)^{d_x}$ (Theorem 3.1). 
- The parameter bounds of Transformers are also *exponentially* large given long input sequences (with fixed dimensions), since $\||W_{\{Q,K\}}\||=O(N^{L/d})$ (Theorem 3.1). 

2. For the score estimation: 
- In Theorem 3.3, why does the upper bound deriving from Assumption 3.1 (weaker) have a much better dependence of $\log n$ ($(\log n)^{-dL}$ v.s. $(\log n)^{\Omega(1)}$) over the upper bound deriving from Assumption 3.2 (stronger)? Also, the former error term $(\log n)^{-dL}$ decreases exponentially fast as the data dimension or sequence length increases, which seems unreasonable. 
- In Theorem 3.3, for the upper bound deriving from Assumption 3.2, $\nu_3=O(L/d)$, which gives $N=n^{\Omega(d/L)}$, $t_0 \sim O(n^{-d/L})$ and $\log (1/t_0) = \Omega(d/L\cdot\log n)$. This term can be merged into the subsequent polynomial dependences in data dimensions and sequence lengths (i.e. $d^{14}L^4$), and hence introduces no singularity. However, the power of $1/n$ is approximately $1/(\nu_3\cdot d)=\Omega(1/L)$, leading to the vacuous sample complexity estimate given long input sequences. 

3. For the distribution estimation: 
- In Theorem 3.4, there are the same issues raised in the second point of the score estimation, since the upper bounds in Theorem 3.4 and Theorem 3.3 are in similar forms. 

4. It seems that there is no characterization of the model capacity dependence. That is, all the provided estimates do not contain (at least, explicitly) error terms related to $h, s, r$. 

5. Minor issues:
- The notation $s$ denotes both the head size and Hölder smoothness (Definition 3.1). 
- In the caption of Table 1, "... where $L$, $\tilde{L}$ are sequence length of transformer inputs", length ->lengths.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615690375

thread_type:
official_review

<thread_object>
creator: reviewer_gbgr

<content>
summary:
This paper provide a theoretical analysis for statistical rates of conditional diffusion transformer, mainly focusing on the score approximation, score estimation and distribution estimation.

strengths:
1. This paper is well-written, which provides detailed theoretical analysis and clear explanations. Additionally, the main theorems are presented concisely and effectively, and the discussions in remarks are particularly helpful for interpreting the results.
2. Comparing with the previous work ([1]), the theoretical analysis alleviates the double exponential factor and achieves minimax optimal statistical rates for DiTs under Hölder smoothness data assumption.

[1]. Hu J Y C, Wu W, Li Z, et al. On statistical rates and provably efficient criteria of latent diffusion transformers (dits)[J]. arXiv preprint arXiv:2407.01079, 2024.

weaknesses:
1. As a follow-up work of [1], maybe the contribution mainly focuses on polishing and extending the previous results, which may lead to a lack of novelty. Could you provide some new insight points about it?
2. With the good rates theoretically,  could this work provide some implementations in practice?

[1]. Hu J Y C, Wu W, Li Z, et al. On statistical rates and provably efficient criteria of latent diffusion transformers (dits)[J]. arXiv preprint arXiv:2407.01079, 2024.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685073331

thread_type:
official_review

<thread_object>
creator: reviewer_msoc

<content>
summary:
The paper studies the statistical understanding of the conditional diffusion transformers (DiTs) with classifier-free guidance. The authors focus on understanding the approximation and estimation rates of these models under various data conditions, particularly in "in-context" settings. They show that both conditional DiTs and their latent variants can achieve minimax optimality, similar to unconditional DiTs, in certain scenarios. The analysis is based on discretizing input domains into small grids and applying Taylor expansions to the conditional diffusion score function, assuming data follows Hölder smoothness. This approach allows for a more refined piecewise approximation, leveraging transformers' universal approximation capacity. The study also extends to latent conditional DiTs, showing that these models offer better approximation and estimation bounds than standard conditional DiTs, especially under linear latent subspace assumptions.

strengths:
- The paper provides a detailed theoretical framework for analyzing the approximation and estimation rates of conditional diffusion transformers (DiTs) under various data assumptions.
- The work integrates classifier-free guidance into the analysis of conditional DiTs.
-  The paper provides rigorous results on score approximation and distribution estimation, including sample complexity bounds for score estimation which lead to minimax optimal estimation results.
- The work integrates classifier-free guidance into the analysis of conditional DiTs.
- Beyond theoretical contributions, the findings offer practical guidance for configuring transformer-based diffusion models to achieve optimal performance.

weaknesses:
A general weakness of this paper is its reliance on a lengthy and detailed appendix, which is not mandatory for reviewers to read. While the results are strong and represent a valuable contribution to the field, the extensive appendix makes it challenging to validate the findings within a limited review timeframe.

- The analysis relies significantly on Hölder smoothness assumptions, which may not apply to all types of data. This limitation restricts the generalizability of the theoretical results to datasets that satisfy these specific conditions. Additionally, the assumption of isotropic smoothness across input variables does not always hold, particularly in cases where the condition is discrete or irregular.

- The study does not address the high-dimensional structural dependence of the data. It is overly restrictive to assume that the ambient dimensions of practical data distributions are small and that the rates of dependence relative to the ambient dimension are sufficiently slow.

- The paper lacks empirical validation that connects the theoretical rates and dimensionality to practical datasets. For instance, it would be beneficial to clarify what smoothness refers to in the context of the given dataset. Furthermore, it is unclear how one might establish practical relevance for the proposed rates and validate them empirically.

questions:
In addition to the weaknesses outlined, please consider the following:

- Can the work be extended to a manifold setup where the rates depend on the intrinsic dimension of the data, similar to the approach taken by Oko et al. (2023)?

- Regarding the assumptions made, if one has a function with significantly less restrictive conditions than those presented in this study, what advantage does using a transformer provide over traditional statistical methods for achieving a minimax rate? This question is particularly relevant from a theoretical perspective, which is the primary focus of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721599638

thread_type:
official_review

<thread_object>
creator: reviewer_wtru

<content>
summary:
This paper studies the statistical rates of approximation and estimation for conditional diffusion transformers (DiTs) with classifier-free guidance. It provides a comprehensive analysis under four common data assumptions, showing that both conditional and latent variants of DiTs can achieve minimax optimality. It uses a modified universal approximation of the single-layer self-attention transformer model to circumvent the need for dense layers, which results in tighter error bounds for both score and distribution estimation.

strengths:
The paper is original in its theoretical approach to studying conditional diffusion transformers, offering new insights into their performance and limitations. The quality of the theoretical analysis is high, with clear outlines and complete proofs. The clear and thorough exploration of statistical limits and estimation procedures is a key strength.

weaknesses:
1. Some results seem to suffer from the curse of dimensionality, which seems not a practical bound.
2. There is an absence of empirical results to complement the theoretical analyses.
3. More practical implications of the assumptions could be added.

questions:
**1.** On the top of page 3, should the denominator of $\nu_3$ be $d_0+d_y$ instead of $d_x + d_y$?

**2.** The bound in Theorem 3.1, the first bound of Theorem 4.1, and the first bound of Theorem 4.2 seem to depend exponentially on $d_x$ or $d_0$, which seems to suffer from CoD. Is the bound tight here?

**3.** Is there any practical example that matches the assumption 3.2? If so, I’d suggest adding a brief paragraph to discuss this.

**4.** In the first bound of Theorem 3.3 and Theorem 3.4, the bound seems to be decaying in a rate of $exp(-d_x)$, is this a typo?

**5.** The second bound of Theorem 3.3 seems to be not sensitive to $d_y$. The second bound of Theorem 3.4 depends on $d_x$ and $d_y$ only through $d_x+d_y$, and is not sensitive to it. Similarly, the second bound of Theorem 4.2 only depends on $d_0$ and $d_y$ only through $d_0+d_y$. Is there any intuition behind this? 

**6.** There aren't any empirical results in the current version. I’d suggest adding a few practical examples to illustrate the validation of the bounds.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731951934909

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear Reviewers,

Thank you for your insightful feedback and the time invested in reviewing our work. We have addressed your comments and question with highest care and the updated manuscript.

Following your suggestions, we have improved the paper's readability through extensive proofreading, detailed proof polishing, and typo corrections. These revisions include additional explanations, paragraphs and sections to help readers understand our techniques and results.

Notably, we add experiments to confirm our theoretical findings. Specifically, we verify the relation in our theoretical results in terms of input data dimension $d_x$, norm bounds of parameters in DiT, and testing loss (score estimation error). Additionally, we clarify the analysis of the approximation rate and include the results under the high-dimensional and low-, mild-dimensional regions. 

--- 

## **Major Revisions**

* **Newly Added `Appendix D`:** Numerical results to validate our theoretical findings. [`WTrU`, `mSoC`]

* **Newly Added  `Appendix K.3-7`:** High-dimensional and low-, mild-dimensional analysis with approximation and estimation [`1NAW`]

* **Updated `Thm3.1, 3.2, 3.3, 3.4`:** Update rates without $e^{d_x}$ dependence and vacuous asymptotics. See updated Table 1 quoted below for a quick summary.  [`WTrU`, `1NAW`]

## **Minor Revisions**

* Proofreading the manuscript and fixing all identified typos and grammatical errors identified by reviewers and authors. [`WTrU`, `mSoC`, `GBgr`, `1NAW`]
* Restructuring and Improving the readability of all proof. [`mSoC`]

---

We hope these revisions address the reviewers' concerns and improve the overall quality of our paper.

---

### **Updated `Table 1`: Summary of Theoretical Results**

The initial data is $d_x$-dimensional, and the condition is $d_y$-dimensional. For latent DiT, the latent variable is $d_0$-dimensional. $\sigma_{t}^{2} = 1 - e^{-t}$ is the denoising scheduler. The sample size is $n$, and $0 < \epsilon < 1$ represents the score approximation error. While we report asymptotics for large $d_x, d_0$, we reintroduce the $n$ dependence in the estimation results to emphasize sample complexity convergence.

| **Assumption** | **Score Approximation** | **Score Estimation** | **Dist. Estimation (Total Variation Distance)** | **Minimax Optimality** |
|----------------|-------------------------|---------------------|--------------------------------------------------|------------------------|
| Generic Hölder Smooth Data Dist.        | $\mathcal{O}\left((\log(\frac{1}{\epsilon}))^{d_x}/{\sigma_t^4}\right)$ | $n^{-\Theta(1/d_x)} \cdot (\log n)^{\mathcal{O}(d_x)}$ | $n^{-\Theta(1/d_x)} \cdot (\log n)^{\mathcal{O}(d_x)}$ | ✘ |
| Stronger Hölder Smooth Data Dist.       | $(\log(\frac{1}{\epsilon}))^{\mathcal{O}(1)}/{\sigma_t^2}$ | $n^{-\Theta(1/d_x^2)} \cdot (\log n)^{\mathcal{O}(1)}$ | $n^{-\Theta(1/d_x)} \cdot (\log n)^{\mathcal{O}(1)}$ | ✔ |
| Latent Subspace + Generic Hölder Smooth Data Dist. | $\mathcal{O}\left((\log(\frac{1}{\epsilon}))^{d_0}/{\sigma_t^4}\right)$ | $n^{-\Theta(1/d_0)} \cdot (\log n)^{\mathcal{O}(d_0)}$ | $n^{-\Theta(1/d_0)} \cdot (\log n)^{\mathcal{O}(d_0)}$ | ✘ |
| Latent Subspace + Stronger Hölder Smooth Data Dist. | $(\log(\frac{1}{\epsilon}))^{\mathcal{O}(1)}/{\sigma_t^2}$ | $n^{-\Theta(1/d_0^2)} \cdot (\log n)^{\mathcal{O}(1)}$ | $n^{-\Theta(1/d_0)} \cdot (\log n)^{\mathcal{O}(1)}$ | ✔ |


---
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the statistical rates of approximation and estimation of conditional diffusion transformers. It's a relatively novel setting for the theoretical analyses on diffusion models. During the rebuttal phase, the authors have addressed most of the questions raised by the reviewers. Although there are still questions about the assumptions and the curse of dimensionality, I feel that the novelty of the theoretical results are enough to warrant a publication at ICLR.