GUD: Generation with Unified Diffusion

## Abstract

Abstract Diffusion generative models transform noise into data by inverting a process that progressively adds noise to data samples. Inspired by concepts from the renormalization group in physics, which analyzes systems across different scales, we revisit diffusion models by exploring three key design aspects: 1) the choice of representation in which the diffusion process operates (e.g. pixel-, PCA-, Fourier-,or wavelet-basis), 2) the prior distribution that data is transformed into during diffusion (e.g. Gaussian with covariance Σ Σ \Sigma roman_Σ ), and 3) the scheduling of noise levels applied separately to different parts of the data, captured by a component-wise noise schedule.
Incorporating the flexibility in these choices, we develop a unified framework for diffusion generative models with greatly enhanced design freedom. In particular, we introduce soft-conditioning models that smoothly interpolate between standard diffusion models and autoregressive models (in any basis), conceptually bridging these two approaches.
Our framework opens up a wide design space which may lead to more efficient training and data generation, and paves the way to novel architectures integrating different generative approaches and generation tasks.

## 1 Introduction

Diffusion-based generative models, first introduced in Sohl-Dickstein et al. (), have seen great successes in recent years since the works of Song & Ermon (); Ho et al. ().
In these models, data are transformed into noise following a diffusion process, and a transformation simulating the reverse process is learned which is then used to map noise into generated samples.
In physics, the theory of renormalization group (RG) flows has been a basic tool in the study of a wide range of physical phenomena, including phase transitions and fundamental physics, both in theoretical as well as numerical approaches.
In short, an RG flow prescribes a way of erasing the high-frequency information of a physical theory, while retaining the information relevant for the long-wavelength physics.
As such, there are clear analogs between score-based generative models and RG flows, at least at a conceptual level. Indeed, it has been known for a long time that RG flows in quantum field theories can also be described as a diffusive process (Zinn-Justin, ; Gaite, ; Cotler & Rezchikov, ; Berman et al., ).
In both cases, information gets erased along the flow and many different initial distributions get mapped into the same final distribution – a feature often referred to as “universality” in the physics literature.
In the diffusion context, the “universal” distribution is given by the chosen noise distribution, independent of the data distribution.

However, there are salient differences between the ways diffusion models and RG erase information. First, the basis: the diffusive RG process is diagonal in the frequency-basis while the standard diffusion models typically diffuse diagonally in the pixel-basis. Second, the prior distribution: the endpoint of RG is a scale-invariant distribution, often with the same second-order statistics as the distribution one starts with at the beginning of the RG flow. The standard diffusion models on the other hand indiscriminately map all data distributions to that of white noise. Third, the component-wise noising schedule: RG flows erase information sequentially from high to low frequencies, while the original diffusion model has the same noise schedule for all pixels.
In our chosen basis, we allow each component to have its own noising schedule.
These considerations lead us to the framework of generative unified diffusion (GUD) models which incorporate the freedom in design choices in the above three aspects.

Autoregressive models, such as next-token prediction models, play an increasingly dominant role in modern-day machine-learning applications such as LLMs, and seem to be distinct from diffusion models at first glance. In autoregressive models, tokens are generated one at a time, conditional on previously generated ones, while diffusion models generate information in all components simultaneously. We will show that the two can in fact be unified in our framework, which in particular allows for *soft-conditioning* generative processes. Intuitively, this means that we can condition on partial information from other components as long as that information has already been generated in the diffusion process.

## 2 Related Work

The idea that diffusion models can incorporate the multi-scale nature of the dataset through a specific choice of data representation has been exploited in interesting works including Wavelet Score-Based Generative Models (Guth et al., ), Pyramidal Diffusion Models
(Ryu & Ye, ), Cascaded Diffusion Models (Ho et al., ), and more.
In these works, the generation is sharply autoregressive
between different hierarchies of generation (between different resolutions, for instance).
In Blurring Diffusion Models (Hoogeboom & Salimans, ), diffusion models in the frequency basis was proposed.
Our framework, besides not
being restricted to these specific choices of basis, allows for an extension of these work by introducing the freedom to soft-condition the generative process.

The possibility of interpolating between the standard diffusion models and token-wise autoregressive models has recently been explored in Chen et al. () in the context of causal sequence generation. In this work, a training paradigm where diffusion models, with a chosen token-wise noising schedule, are trained to denoise tokens based on the information of only the (partially noised) previous tokens, captured in latent variables. Our work integrates the choices of data representation, the prior distribution, and the component-wise noising schedule into one framework, and therefore includes this particular paradigm as a special case. In particular, as we demonstrate in experiments, it is possible to integrate multi-scale and spatially sequential generation processes in our framework.

In practice, for applications with high-dimensional data, the diffusion generation often takes place in a lower-dimensional latent space
(Rombach et al., ; Sinha et al., ; Vahdat et al., ). The freedom to choose the basis proposed in our work is not to be understood as replacing latent space diffusion. Rather, our framework can straightforwardly be used in the latent space, leading to a latent GUD model.

## 3 Preliminaries

### 3.1 Stochastic Differential Equations

In continuous time, the general diffusion setup can be described by the following Itô stochastic differential equation (SDE):

$$ $\differential{{\bm{\phi}}}=\mathbf{f}({\bm{\phi}},t)\differential{t}+\mathbf{G }({\bm{\phi}},t)\differential{\mathbf{w}}\,,$ (1) $$

where $\differential{\mathbf{w}}$ represents a white noise Wiener process.
We use ${\bm{\phi}}\in{\mathbb{R}}^{d}$ to denote a vector.
In the above, we have $\mathbf{f}(\,\cdot\,,t):{\mathbb{R}}^{d}\to{\mathbb{R}}^{d}$ and $\mathbf{G}(\,\cdot\,,t):{\mathbb{R}}^{d}\to{\mathbb{R}}^{d\times d}$.
The reverse-time SDE is given by (Anderson, )

$$ $\differential{{\bm{\phi}}}=\left(\mathbf{f}({\bm{\phi}},t)-\nabla\cdot(\mathbf {G}\mathbf{G}^{T})({\bm{\phi}},t)-\mathbf{G}\mathbf{G}^{T}\nabla_{\bm{\phi}} \log p_{t}({\bm{\phi}})\right)\differential{t}+\mathbf{G}({\bm{\phi}},t){ \differential}{\mathbf{\bar{w}}}$ (2) $$

where $\mathbf{\bar{w}}$ is the inverse Wiener process.

The probability density $p({\bm{\phi}},t)$ corresponding to the SDE equation 1 solves the following Fokker-Planck equation (or Kolgomorov’s forward equation) (Oksendal, )

$$ ${\frac{\partial}{\partial t}}p({\bm{\phi}}(t))=-\sum_{i=1}^{d}{\partial\over \partial\phi_{i}}\left(f_{i}({\bm{\phi}},t)p({\bm{\phi}}(t))\right)+{1\over 2} \sum_{i=1}^{d}\sum_{j=1}^{d}{\partial^{2}\over\partial\phi_{i}\partial\phi_{j} }\left(\sum_{k=1}^{d}G_{ik}G_{jk}\,p({\bm{\phi}}(t))\right),$ (3) $$

where $\phi_{i}$ denotes the component of ${\bm{\phi}}$ in a given basis.

### 3.2 Renormalization Group (RG) Flows

As mentioned in the introduction, the renormalization group refers to a collection of methods in physics that aim to progressively remove the high-frequency degrees of freedom while retaining the relevant low-frequency ones. In other words, one aims to remove the irrelevant details of the physical system
without altering the physics at the larger scale one is interested in. By doing so, one hopes to be able to robustly calculate the universal macroscopic features of the physical systems.

There are many ways physicists have proposed to achieve this goal, starting with the seminal work of Kadanoff () and Wilson (; ).
How to improve the understanding and the implementation of RG flows, including efforts involving machine learning methods, remains an active topic of investigation in physics (Koch-Janusz & Ringel, ).
Here we consider the exact RG (ERG) formalism, a non-perturbative method pioneered by Polchinski () for quantum field theories. In this RG method, one implements Wilson’s idea of RG by specifying a cutoff kernel
$K_{k}(\Lambda):=K(k^{2}/\Lambda^{2})$ for a given *cutoff scale* for each frequency $k$, with the property that $K_{k}(\Lambda)\rightarrow 1$ when $k\ll\Lambda$ and $K_{k}(\Lambda)\rightarrow 0$ when $k\gg\Lambda$. With this, one erases information on frequencies much larger than $\Lambda$.
One example of such cutoff kernels is the sigmoid function.

Given a physical theory and a choice of cutoff kernel, one can define physical probability distribution $p_{\Lambda}[{\bm{\phi}}]$ that satisfies a differential equation which is an infinite-dimensional version of the Fokker-Planck equation 3, where the role of diffusion time is played by $t=-\log(\Lambda/\Lambda_{0})$ for some reference scale $\Lambda_{0}$.

### 3.3 Standard Diffusion Models

In diffusion-based generative models, a forward diffusion process that gradually transforms data samples into noise following a particularly simple SDE is inverted to transform noise into images.
A commonly used forward SDE is the finite-variance(^1^11This is referred to as the “variance-preserving” (VP) diffusion in some literature. We will reserve the term to cases when the variance is actually strictly constant throughout the diffusion process, which we will discuss in §4.4.) SDE (Song et al., ) defined as:

$$ $\mathrm{d}{\bm{\phi}}=-\tfrac{1}{2}\beta(t){\bm{\phi}}\,\mathrm{d}t+\sqrt{ \beta(t)}\,\mathrm{d}\mathbf{w},$ (4) $$

where the initial vector ${\bm{\phi}}(0)\in\mathbb{R}^{d}$ represents the data sample and $\mathrm{d}\mathbf{w}$ denotes the standard Wiener process. The function $\beta:[0,T]\to\mathbb{R}_{+}$ which determines the SDE is the predefined noise schedule.

The reverse-time SDE follows from specializing equation 2 and reads

$$ $\mathrm{d}{\bm{\phi}}=\left[-\tfrac{1}{2}\beta(t){\bm{\phi}}-\beta(t)\nabla_{ \bm{\phi}}\log p_{t}({\bm{\phi}})\right]\mathrm{d}t+\sqrt{\beta(t)}\mathrm{d} \bar{\mathbf{w}}\,,$ (5) $$

where $\mathrm{d}\bar{\mathbf{w}}$ is a reverse-time Wiener process, and $\nabla_{\bm{\phi}}\log p_{t}({\bm{\phi}})$ is the score function of the marginal distribution at time $t$.
The task for machine learning is thus to approximate the score function, which can be achieved by denoising score matching (Vincent, ) with the objective function

$$ $\mathcal{L}_{\rm DSM}=\mathbb{E}_{t,{\bm{\phi}}(0),\epsilon}\left[\lambda(t) \norm{s_{\theta}({\bm{\phi}}(t),t)-\nabla_{{\bm{\phi}}(t)}\log p_{t}({\bm{\phi }}(t)|{\bm{\phi}}(0))}^{2}\right]\,,$ (6) $$

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ is Gaussian white noise, ${\bm{\phi}}(t)=\alpha(t){\bm{\phi}}(0)+\sigma(t)\epsilon$ is the noised data at time $t$, $\lambda:[0,T]\to{\mathbb{R}}_{+}$ is a weighting function, and $\alpha(t),\sigma(t)$, and $\beta(t)$ are functions capturing the equivalent information about the noising schedule. They are defined in equation 16 by specializing $\beta_{i}=\beta$ etc.
Importantly, the choice of SDE (4) leads to an Ornstein-Uhlenbeck (OU) process and the conditional score $\nabla_{{\bm{\phi}}(t)}\log p_{t}({\bm{\phi}}(t)|{\bm{\phi}}(0))$ can be computed analytically (Song et al., ).

## 4 Methods

### 4.1 Diagonalizable Ornstein Uhlenbeck Process

Returning to the general diffusion SDE (1),
we now consider the special case in which $\mathbf{f}={\bm{F}}{\bm{\phi}}$ and ${\bm{F}}$ is ${\bm{\phi}}$-independent . This guarantees that the SDE describes a Ornstein-Uhlenbeck process admitting analytical solutions for the conditional distribution $p_{t}({\bm{\phi}}(t)\lvert{\bm{\phi}}(0))$ required for denoising score matching.
Moreover, we consider a choice of simultaneously diagonalizable $\mathbf{F}$ and $\mathbf{G}$:

$$ $\bm{F}=M^{-1}{\tilde{{\bm{F}}}}M,~{}~{}\bm{G}=M^{-1}\sqrt{\bm{\beta}}$ (7) $$

with some constant matrix $M$ and diagonal $\bm{\beta}={\rm{diag}}(\beta_{i})$ and $\tilde{\bm{F}}={\rm{diag}}(\tilde{F}_{i})$.

In terms of the parameterization

$$ $\bm{\chi}:=M{\bm{\phi}},$ (8) $$

the SDE equation 2 is equivalent to $d$ decoupled SDEs of the form

$$ $\differential\chi_{i}=\tilde{F}_{i}(t)\chi_{i}\differential t+\sqrt{{\beta}_{i }(t)}\differential{\mathrm{w}}$ (9) $$

with the reverse SDE given by

$$ $\differential\chi_{i}=\left(\tilde{F}_{i}(t)\chi_{i}-{{\beta}_{i}(t)}\nabla_{ \chi_{i}}\log p_{t}(\chi)\right)\differential t+\sqrt{{\beta}_{i}(t)}{ \differential}{\mathrm{\bar{w}}}\,.$ (10) $$

The choice of the transformation matrix $M$ is a choice of data representation in which the diagonal score-based diffusion based on the SDE equation 9 and equation 10 can be efficiently performed.

In particular, as the Wiener process is invariant under orthogonal transformations, it is convenient to view the change of basis (given by $M$) as the composition of an orthogonal ($U$) and a scaling ($S$) transformation: $M=S^{-1}U$.
In terms of the original ${\bm{\phi}}$ variables, the forward SDE then reads

$$ $\differential{{\bm{\phi}}}=F{\bm{\phi}}\,dt+U^{-1}S\sqrt{\bm{\beta}}\,{ \differential}{\mathrm{w}}=U^{-1}\tilde{\bm{F}}U{\bm{\phi}}\,dt+\sqrt{\bm{ \beta}^{\prime}}\,{\differential}{\mathrm{w^{\prime}}}$ (11) $$

where $\sqrt{\bm{\beta}^{\prime}}=U^{-1}\sqrt{\bm{\beta}}U$
and $\differential{\mathrm{w^{\prime}}}=\sqrt{\Sigma_{\mathrm{prior}}}\differential
{\mathrm{w}}$ is a Wiener process with covariance matrix $\Sigma_{\mathrm{prior}}=U^{-1}S^{2}U$.

The choice of $M$, particularly the orthogonal part $U$, captures the freedom in our unified framework to choose the basis in which the diffusion process is diagonal.
Moreover, the scaling $S$ then determines the choice of the noise (prior) distribution $p_{\mathrm{prior}}=\mathcal{N}(0,\Sigma_{\mathrm{prior}})$, which the forward process approaches at late times. Finally, note that the $\beta_{i}(t)$ can a priori be independent functions of $t$ for each component $i$. The choice of $\beta_{i}(t)$ thus captures the choice of a component-wise noising schedule.

Due to the diagonal property of the SDE, the denoising score matching loss function equation 6 for learning the Stein score $\nabla_{\chi_{i}}\log p_{t}(\chi)$ can be straightforwardly generalized to the GUD models:

$$ $\mathcal{L}_{\rm GUD}=\mathbb{E}_{t,{\bm{\chi}}(0),\epsilon}\sum_{i=1,\dots,d} \lambda_{i}(t)\Big{\lvert}s_{i,\theta}({\bm{\chi}}(t),t)-\nabla_{\chi_{i}(t)} \log p_{t}({\bm{\chi}}(t)|{\bm{\chi}}(0))\Big{\lvert}^{2}\,$ (12) $$

where the $\bm{\lambda}=(\lambda_{1},\dots,\lambda_{d}):[0,T]\to{\mathbb{R}}_{+}^{d}$ is the weighting vector.
In our experiments, we let
$\lambda_{i}(t)=\sigma_{i}^{2}(t)$,
with the aim to scale the loss to be an order-one quantity and
generalizing
the common weighting factor $\lambda(t)=\sigma^{2}(t)$ in the standard diffusion loss (6) (Song & Ermon, ).
The SDE (10) with the learned score, when discretized, leads to a hierarchical generative model with model density

$$ $\tilde{p}({\bm{\chi}}(0))=\int\left(\prod_{k=1}^{T}d^{d}{\bm{\chi}}(\tfrac{k}{ T})\right)\;\left(\prod_{\ell=0}^{T-1}p({\bm{\chi}}({\tfrac{\ell}{T}})\lvert{ \bm{\chi}}({\tfrac{\ell+1}{T}}))\right)\tilde{p}({\bm{\chi}}(1))$ (13) $$

where $T$ is the number of steps in the discretization, and the prior distribution is given by the noise distribution
$\tilde{p}({\bm{\chi}}(1))={\cal N}(0,{\bf I})$.

### 4.2 Finite-Variance Diffusion and the Signal to Noise Ratio

In the coordinate given by ${\bm{\chi}}$, we now further specialize equation 9 to the following finite-variance diffusion process:
with $\tilde{\bm{F}}(t)=-{1\over 2}{\bm{\beta}}(t)$, the corresponding SDE reads

$$ $d\chi_{i}=-{1\over 2}{\beta}_{i}(t)\chi_{i}dt+\sqrt{{\beta}_{i}(t)}d{\mathrm{w }}.$ (14) $$

Integrating the above gives

$$ $\chi_{i}(t)=\alpha_{i}(t)\chi_{i}(0)+\sigma_{i}(t)\epsilon\;,\qquad\epsilon \sim{\cal N}(0,1)$ (15) $$

where

$$ $\alpha_{i}(t)=\exp(-\frac{1}{2}\int_{0}^{t}\beta_{i}(s)\differential{s})\,, \quad\text{and}\quad\sigma_{i}(t)^{2}=1-\alpha_{i}(t)^{2}\,.$ (16) $$

It follows that the variance

$$ ${\rm Var}(\chi_{i}(t))=\alpha_{i}(t)^{2}(\Sigma^{({\bm{\chi}})}(0))_{ii}+ \sigma_{i}(t)^{2}$ (17) $$

interpolates between 1 and the data variance $(\Sigma^{({\bm{\chi}})}(0))_{ii}$, and is in particular finite at all stages of diffusion.
In the above, we have used the following notation for the data covariance matrix

$$ $(\Sigma^{({\bm{\chi}})}(0))_{ij}:={\mathbb{E}}_{p_{\rm data}({\bm{\chi}}(0))} \left[(\chi_{i}(0)-\overline{\chi_{i}(0)})(\chi_{j}(0)-\overline{\chi_{j}(0)}) \right]~{},~{}{\rm where}\quad\overline{\chi_{i}(0)}:={\mathbb{E}}_{p_{\rm data }({\bm{\chi}}_{0})}[\chi_{0,i}],$ $$

An important quantity signifying the stage of the diffusion process (for each component) is the time evolution of the ratio between the signal and the noise, captured by the *signal-to-noise ratio*,

$$ ${\rm SNR}_{i}(t):={\mathbb{E}}\left(\frac{(\alpha_{i}(t)\chi_{i}(0))^{2}}{ \sigma_{i}(t)^{2}}\right)=(\Sigma^{({\bm{\chi}})}(0))_{ii}\frac{\alpha_{i}^{2} (t)}{\sigma_{i}^{2}(t)},$ (18) $$

where the expectation is with respect to the data and the noise distribution, and we have assumed that the data mean vanishes (which can always be made to be the case by subtracting the mean).
Note that this is different from the signal-to-noise ratio quoted in some diffusion model contexts,

$$ ${{\rm snr}}_{i}(t):={\alpha_{i}^{2}(t)}/{\sigma_{i}^{2}(t)}=e^{-\gamma_{i}(t)},$ (19) $$

as this version does not take into account the magnitude of the signal in the data.
As they depend only on the schedule, we note that the functions
$\beta_{i}$, $\alpha_{i}$, $\sigma_{i}$ and $\gamma_{i}$ all contain the same information.

Figure: Figure 1: Different noising schedules $\bm{\gamma}(t)$.

At a given time $t\in[0,T]$ in the diffusion process, the information of ${\bm{\gamma}}(t)=(\gamma_{1},\dots,\gamma_{d})(t)\in{\mathbb{R}}^{d}$ is what we call the *noising state*, indicating the extent to which information in the data has been replaced by noise at that time. As a function of $t$, the evolution of the noising state traces out a path connecting the data distribution $p_{\rm data}$, corresponding to

$$ ${\bm{\alpha}}({\bm{\gamma}})=(\alpha_{1},\dots,\alpha_{d})=(1,\dots,1),$ $$

and the prior distribution $p_{\rm prior}$, corresponding to ${\bm{\alpha}}=(0,\dots,0)$, where $\alpha_{i}={\rm sigmoid}(-\gamma_{i})^{1/2}$ as in equation 19.
The different paths correspond to different
Ornstein-Uhlenbeck processes, as defined in equation 14, with different diffusion dynamics.

This highlights the fact that the freedom in component-wise noising schedules in the GUD model is fundamentally larger than the
freedom in the noising schedule in standard diffusion models, which is given by different choices of the function $\gamma(t)$ with $\gamma_{i}(t)=\gamma_{j}(t)=\gamma(t)$.
In this case, all choices of $\gamma(t)$ trace out the same diagonal path (as long as the boundary values $\gamma(0)$, $\gamma(T)$ are held fixed) and merely amount to different time parameterizations (Kingma et al., ). In contrast, different component-wise schedules generically correspond to genuinely different paths, as illustrated in the schematic Figure 1 where the continuous line corresponds to the standard diffusion schedule (with any time parametrization) and the dashed lines represent other possible component-wise schedules.

### 4.3 Unification via Soft-Conditioning

The above form of SNR clarifies an implicit hierarchical structure of the standard diffusion models: even when $\gamma_{i}(t)=\gamma(t)$ is identical for all components $i$, the components with larger amplitudes have larger signal-to-noise ratio SNR${}_{i}(t)=(\Sigma^{({\bm{\chi}})}(0))_{ii}e^{-\gamma(t)}$, and are in this sense less “noised” throughout the diffusion process. As a result, the generation process equation 13 and in particular the modeling of the probability $p({\bm{\chi}}({\tfrac{\ell}{T}})\lvert{\bm{\chi}}({\tfrac{\ell+1}{T}}))$ conditional on the previous state is implicitly a process of generating the less important features (with smaller amplitude) conditional on the more important features (with larger amplitude) that have already been partially generated. It is clear that by making more general choices of component-dependent noising schedules $\gamma_{i}(t)$ one can tune the degree of this *s*oft-conditioning property, as we will explore in the experiments below. In the extreme case when the support of $\beta_{i}(t)$ and $\beta_{i\neq j}(t)$, namely the “active time” for the $i$th resp. $j$th component, do not overlap, we arrive at autoregressive generation, in which one feature/token (or one group of features/tokens) is generated at each time, conditional on those that have been generated already. See Figure 6 for the visualization of a specific example.
In this way, the freedom to choose a component-dependent noising schedule in our GUD model enables us to interpolate between standard diffusion and autoregressive generation.

### 4.4 Whitening

A particularly interesting choice for the matrix $M=S^{-1}U$ is the orthogonal transformation $U$ that diagonalizes the data covariance matrix $\Sigma^{({\bm{\phi}})}(0)$, and the diagonal matrix $S^{-1}$ that performs a *whitening* transformation.
In other words, we choose $S$ and $U$ such that the data covariance matrix matches $\Sigma^{({\bm{\phi}})}(0)=U^{-1}S^{2}U$.
Note that $M$ is then precisely the familiar PCA transformation followed by a whitening transformation which makes the variance uniform.
In the context of diffusive generation, such a basis has the following appealing features. First, the softness of the soft-conditioning, manifested via the evolution of the signal-to-noise ratio equation 18, is now completely controlled by the component-wise schedule $\gamma_{i}(t)$, which can make the design process of the diffusion modeled more streamlined and uniform across different applications with different datasets. Second, with such a choice the covariance matrix actually remains constant throughout the diffusion process as the data covariance $\Sigma^{({\bm{\chi}})}(0)={\bf I}$ is now the same as the noise covariance, and the finite-diffusion equation 14 is *variance preserving* in the strict sense. In other words, the conditional number of the covariance matrix is always one and the generative process does not need to alter the second-order statistics. We expect this property to be beneficial in some situations for learning and discretization.

### 4.5 Noising-State Conditional Network Architecture

For the score network architecture, we follow the approach of predicting the noise $\epsilon$ given a noised image (Ho et al., ),
trained via denoising score matching (Vincent, ).
In standard diffusion models, this score network is typically conditioned on the time variable or an equivalent object such as $\gamma(t)$ (Kingma et al., ).
The introduction of a component-wise schedule in our framework suggests generalizing this by conditioning the model on the more informative component-wise noising state, represented by the component-wise noise state ${\bm{\gamma}}(t)=(\gamma_{1},\dots,\gamma_{d})(t)$.
Since this is a vector of the same dimension as the data and not a scalar, a modification of the network architecture is required.
We have implemented this by incorporating cross-attention between the data and the noising state, further details can be found in section A of the appendix.

Since any choice of the noising schedule $\gamma_{i}(t)=\gamma(t)$ in standard diffusion models can be thought of as just a reparametrization of time (Kingma et al., ) (cf. §4.2), the diffusion time $t$ itself suffices as a feature for the network to indicate the noising state. For our GUD models, this is true only for a fixed schedule choice.
By conditioning directly on $\bm{\gamma}$ instead of $t$, our score network is directly conditioned on the instantaneous noising state, and not on the totality of its path, namely the schedule ${\bm{\gamma}}(t)$.
This enables us to train a single network for a range of schedules, as we will do in the experiments described in the next section.
The set of values ${\bm{\gamma}}\in{\mathbb{R}}^{d}$ used during training bound a region, visualized schematically by the shaded area in Fig. 1.
This is implicitly the region of the values of $\bm{\gamma}\in\mathbb{R}^{d}$ where the score function has been learned. This suggests the possibility of using any particular path within the shaded region for generation, which might differ from the path used for training (indicated by the dashed lines in Fig. 1).
This feature of the GUD model may facilitate the numerical optimization of component-wise schedules in future work.

## 5 Experiments

We will now showcase the flexibility of the GUD model with some examples, and conduct preliminary investigations into the effects of these different design choices on the behavior of diffusion models and their resulting sampling quality.
An overview of the experiments, highlighting the relevant design choices, is given in the following table.

|  | §5.1 | §5.2 | §5.3 |
| --- | --- | --- | --- |
| basis | pixel, PCA , FFT | column | wavelet $\otimes$ column |
| prior | isotropic Gaussian and | isotropic Gaussian | isotropic Gaussian |
| variance-matching Gaussian |  |  |  |
| noising schedule | varying softness | varying softness | varying softness |
| and ordering variables |  |  |  |
| other applications |  | image extension |  |

### 5.1 Soft-Conditioning Schedules

Figure: Figure 2: For eight of the PCA components $\chi_{i}$ of CIFAR-10, we visualize the OU noise level ${\sigma}_{i}(t)$, the corresponding noising path ${\gamma}_{i}(t)={\rm logit}(\sigma_{i}^{2}(t))$ for the linear schedule of equation 20, and the corresponding signal-to-noise ratio. Blue dashed lines indicate chosen minimal noising/reconstruction levels. From left to right: (a) Standard diffusion where ${\gamma}_{i}={\gamma}_{j}$. (b) The schedule $\bm{\gamma}$ is chosen such that $\log{\rm SNR}_{i}(t)=\log{\rm SNR}_{j}(t)$, corresponding to a generative process with no hierarchy. (c) With whitened data, with the schedule $\bm{\gamma}$ chosen such that $\log{\rm SNR}_{i}(t)$ is identical to that in the standard diffusion case shown in column (a). (d) Hierarchy-less generation with whitened data.
Refer to caption: x1.png

First, we investigate the effect of choosing different bases, priors, and schedules by conducting experiments on the unconditional generation of CIFAR-10 images.
As an example of a simple setup, we choose linear noise schedules, which means that all $\gamma_{i}(t)$ we consider here are linear functions of the diffusion time $t$.
In the first setup, we choose the basis, given by the orthogonal transformation $U$ described in §4.1, to be the PCA basis.
In terms of prior distributions, we choose our noise to be either given by isotropic Gaussian or Gaussian with covariance matching that of the data. Alternatively, as explained in §4.1, this is equivalent to whitening the data (in the PCA basis) while using the isotropic Gaussian noise. We will therefore refer to the two choices of priors as * whitened* and * unwhitened* in what follows.

In the second setup, we use the Fourier basis and consider a two-parameter family of component-wise noising schedules, where we vary the precise ordering of the different Fourier components (given by the “ordering variables”) as well as the softness parameter of the soft-conditioning schedule.
It is important to note that these experiments only constitute a one- or two-dimensional subspace of a much larger design space. We make the above choice for the sake of concreteness and due to a limitation in computational resources.

#### Linear component-wise schedules.

For a given choice of basis ${\bm{\chi}}$, let $\Sigma_{i}:=(\Sigma^{({\bm{\chi}})}(0))_{ii}$ denote the variance of each component $\chi_{i}$. As discussed in §4.2, a choice of the schedule $\bm{\gamma}(t)$ then leads to the signal-to-noise ratio $\log\mathrm{SNR}_{i}(t)=-\gamma_{i}(t)+\log\Sigma_{i}$.
For the diffusion generative models, each component must be sufficiently noised in the forward process and sufficiently denoised in the reverse process.
It will therefore be convenient to introduce the parameters $\tilde{\gamma}_{\mathrm{denoise}}$ and $\tilde{\gamma}_{\mathrm{noise}}$
which specify the minimal levels of denoising and noising at the initial and final time, respectively:

$$ $\displaystyle\begin{split}\tilde{\gamma}_{\mathrm{denoise}}&=-\min\limits_{i} \log\mathrm{SNR}_{i}(t=0)\\ \tilde{\gamma}_{\mathrm{noise}}&=-\max\limits_{i}\log\mathrm{SNR}_{i}(t=1). \end{split}$ $$

Here and in what follows we parametrize the diffusion time to be $t\in[0,T]=[0,1]$.
The parameters $\tilde{\gamma}_{\mathrm{denoise}}$ and $\tilde{\gamma}_{\mathrm{noise}}$ have to be chosen carefully.
In particular, to employ the inverse process as a generative model, the distribution at the final time $t=1$ must be sufficiently close to the prior normal distribution from which we draw initial samples. This is not guaranteed by demanding endpoints for the SNR alone, as a low SNR can be achieved with little noise if the data magnitude is small. We therefore also require $\sigma_{i}(t=1)\geq\sigma_{\min}$, which translates to $\tilde{\gamma}_{\mathrm{noise}}\geq{\rm logit}(\sigma_{\min}^{2})-\min_{i}\log
\Sigma_{i}$.

Next, we associate an ordering variable $l_{i}$ to each component, which will determine the hierarchical structure among the components by shifting the onset of noise in the forward process. CIFAR-10, just like many other (natural) image datasets, is an example of what might be called
frequency-based datasets. We use this term to describe the datasets with a natural meaning of locality, whose covariance is approximately diagonalized in the Fourier basis and whose variance is generally decreasing with increasing frequencies (Tolhurst et al., ; Field, ). For these datasets, the hierarchical structure can naturally be specified in terms of the related notions of variance, frequencies, and resolution, as familiar from image processing.
We choose our ordering variable $l_{i}$ to capture this notion of hierarchy.

To control the level of autoregressiveness, or softness, of the soft-conditioning linear schedule, we introduce a parameter $a>0$ and define the linearly interpolating schedule

$$ $\gamma_{i}(t)=\gamma_{\min,i}+(\gamma_{\max,i}-\gamma_{\min,i})t$ (20) $$

with the endpoints given by

$$ $\begin{gathered}\gamma_{\min,i}:=\tilde{\gamma}_{\mathrm{denoise}}+\log\Sigma_ {i}+a(l_{i}-l_{\max})\,\\ \gamma_{\max,i}:=\tilde{\gamma}_{\mathrm{noise}}+\log\Sigma_{i}+a(l_{i}-l_{ \min})\,.\end{gathered}$ (21) $$

Figure: Figure 3: Dependence of model quality in terms of negative log-likelihood (left) and FID (right) on the softness parameter for the linear schedule in §5.1. The schedule is defined in PCA components and results are shown both for unwhitened and whitened data scaling (i.e. white and data-matching priors). Training on CIFAR-10 using a single score-network for each choice of scaling. Standard diffusion corresponds to $a=1$ in the unwhitened case.
Refer to caption: x2.png

Here $l_{\rm max}=\max_{i}l_{i}$ and similarly for $l_{\rm min}$.
The larger $a$ is, i.e. the smaller the softness $1/a$, the more autoregressive the schedule becomes. On the other hand, in the limit of extreme softness (small $a$) the hierarchical nature of the generative model disappears.

The parameters of our linear schedules are thus the ordering variables $l_{i}$, the softness parameter $a^{-1}$, and the SNR endpoints given by $\tilde{\gamma}_{\rm denoise}$, $\tilde{\gamma}_{\rm noise}$.
In our experiments, we fix $\tilde{\gamma}_{\rm denoise}=-7$ and $\tilde{\gamma}_{\rm noise}=\max[3,{\rm logit}(\sigma_{\min}^{2})-\min_{i}\log
\Sigma_{i}]$ with $\sigma_{\min}=0.99$.

#### Softness in PCA space.

In the first experiment,
we apply the above linear schedule in the whitened and unwhitened PCA bases.
We choose the ordering variable to be given by $l_{i}=-\log\Sigma_{i}$, which allows for the trajectory of the signal-to-noise ratio of the standard diffusion model to be reproduced at $a=1$, also when the prior has been changed to have the same covariance as the data (see Fig. 2 (c)).
We trained a single score network for the (inverse) softness parameter in the range $a\in[0.4,1.6]$ by randomly sampling $a$ at each training step.
Figure 3 shows the negative log-likelihood (NLL) and FID evaluated for different values of $a$(^2^22The FID has not converged and was still dropping at a significant rate when we terminated training due to limited computational resources. Nevertheless, the qualitative features displayed in Figure 3 appear to have stabilized.). Interestingly, the unwhitened configuration performs better when measured by NLL, but worse in terms of FID, with the standard diffusion setup ($a=1$) appearing close to optimal.
See section B of the appendix for further experimental details.

#### Ordering variables.

We also perform experiments in the Fourier (FFT) basis, for which the RG physics reviewed in §3.2 naturally suggests an ordering of noising based on the frequency $|k_{i}|$.
To test the dependence of the quality of the model on ordering parameters, we consider ordering variables
$l_{i}=(1-r)(-\log\Sigma_{i})+r(|k_{i}|+\delta)/\kappa$, parametrized by $r\in[0,1]$ and interpolating between $l_{i}=-\log\Sigma_{i}$ as in the PCA experiments and the frequencies $|k_{i}|$. The slope and offset parameters, $\kappa$ and $\delta$, are chosen such that the range of $l_{i}$ are the same at $r=1$ as at $r=0$. We trained a score network for a range of values of $r$ in addition to the softness parameter $a^{-1}$ on CIFAR-10, with evaluation results shown in Figure 6. We find the optimal performance in terms of NLL is located slightly away but close to standard diffusion. In this experiment, we choose the prior to be the isotropic Gaussian (i.e. unwhitened).

### 5.2 Sequential Generation in Real Space

Figure: Figure 4: Diffusion forward process for a single image of CIFAR-10: (a) standard diffusion, (b) variance-matching Gaussian noise with same SNR as standard diffusion, (c) column-wise sequential schedule of §5.2 with $b=0.5$, (d) combination of Haar wavelet and column-sequential schedule of §5.3 with $a=0.5$, and with variance-matching Gaussian noise.
Refer to caption: x3.png

Figure: Figure 5: The model quality with a two-parameter family of schedules controlling the softness and the ordering parameters. The right figure is the region in the box on the left and the same color map is shared. The black dots indicate the parameters corresponding to standard diffusion models.
Refer to caption: x4.png

While the previous experiment explores the GUD model in the context of multi-scale hierarchical generation, it can equally be applied to perform sequential generation in pixel space, as we will now demonstrate with a soft-conditioning column-wise generation model.
Grouping the components according to their column in the pixel space of size $L\times L$, we index the schedules according to the columns labeled by $i=1,\ldots L$.

As before, we perform experiments with a linear schedule

$$ $\gamma_{i}(t)=\mathrm{clip}_{\gamma_{\min},\gamma_{\max}}\quantity(\gamma_{ \min}+(t-t_{i})\frac{\gamma_{\max}-\gamma_{\min}}{1-b})~{},~{}~{}~{}{\rm with} ~{}~{}t_{i}=b\frac{L-i}{L-1}$ (22) $$

with a varying degree of softness/autoregressivity, now parametrized by $b$.
The clipping, defined by the clipping function ${\rm clip}_{y,z}(x):=\max(y,\min(z,x))$, has the effect of freezing the columns when the designated noising ($\gamma_{\rm max}$) or reconstruction level ($\gamma_{\rm min}$) is reached.

#### Training on PCAM dataset.

We trained separate score networks at $b=0.3$ and $b=0.5$ on the PCAM dataset (Veeling et al., ), downscaled to $32\times 32$ pixels, obtaining negative log-likelihoods of $3.90$ and $3.94\,\mathrm{bits/dim}$, respectively.

#### Image extension.

Since images are generated column-by-column from left to right, conditioned on the part on the left that has already been generated, this suggests that the generative process can be repeated. See section C of the appendix for further detail.
Figure 7 shows three examples of image strips generated in this manner, using the score network trained on $32\times 32$ images at $b=0.5$.

Figure: Figure 7: Images generated using the column-wise schedule by repeatedly adding noise to the right, using a score network that was only trained on square training data.
Refer to caption: x6.png

### 5.3 Haar wavelets

To further showcase the versatility of our unified framework, we integrate Haar wavelet decomposition with a column-wise noise schedule among the wavelet components at each hierarchical level. This extends the wavelet-conditioned score matching of Guth et al. () by including a parameter allowing for soft-conditioning, and incorporating column-wise sequential noising at each level.

Concretely, we use two parameters $a$ and $b$ to parametrize the (inverse) softness among the different levels of wavelet components and the columns within each level, respectively.
Suppose there are $\mathcal{N}$ hierarchical levels of wavelet decompositions, labeled by $i=1,\ldots,\mathcal{N}$, and there are $L_{i}$ columns in the $i$-th level, indexed by $j=1,\ldots,L_{i}$, we define the offsets

$$ $c_{i}=a\frac{\mathcal{N}-i}{\mathcal{N}-1},~{}~{}c_{ij}=b\frac{L_{i}-j}{L_{i}- 1}.$ $$

With this, we specify the linear schedule for $a,b\in[0,1]$ to be

$$ $\gamma_{ij}(t)=\mathrm{clip}_{\gamma_{\rm min},\gamma_{\rm max}}\quantity( \gamma_{\rm min}+(\gamma_{\rm max}-\gamma_{\rm min})\frac{t_{i}-c_{ij}}{1-b}),$ $$

where
$t_{i}=\mathrm{clip}_{0,1}(t-\frac{c_{i}}{1-a}).$
See Figure 4(d) for a visualization of this schedule.

We trained a score network for $a\in[0.3,0.7]$ and $b\in[0.3,0.7]$ on CIFAR-10 for $300$k steps and using $\mathcal{N}=3$ levels. Similar to the results in §5.1,
the model quality again depends on the softness parameters, with the lowest NLL value reached being $3.17\,\mathrm{bits/dim}$.

## 6 Conclusions

In this work, we proposed the GUD framework, which naturally integrates novel design freedoms in diffusion-based generation. Notably, the framework eliminates the rigid boundary between diffusive and autoregressive generation methods and instead offers a continuous interpolation between the two. This flexibility paves the way for a broad range of potential applications.

First, our experiments indicate that the choices in all three aspects we investigate in the present work – the diagonal basis, the prior distribution, the component-wise schedule – do have an influence on the final quality of the model. As a result, there is potentially vast room to improve the quality of diffusion models. In future work, we will address the question of the optimization of these design choices.

Second, the flexibility of our framework enables seamless integration of various approaches to generative models.
For instance, we illustrated in §5.3 the possibility to combine hierarchical generation (in the wavelet basis) with sequential generation, and in §5.2 how our framework can readily be used to extend images. Similarly, the inpainting, coloring, upscaling, and conditional generation tasks can all be realized and generalized within the GUD framework, via an appropriate choice of basis and component-wise schedules.

While the scope of our numerical experimentations and our ability to optimize important hyperparameters has been limited by the compute resources available to us, we believe our theoretical framework has the potential to lead to more efficient diffusion models, a wide range of applications, and novel architecture designs.

## Acknowledgements

This work has been supported by the Vidi grant (number 016.Vidi.189.182) from the Dutch Research Council (NWO).
MG was partially supported by a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 864035).

## Appendix A Noising state conditional score network architecture

Inspired by techniques in conditional generation tasks (Rombach et al., ), we introduced a cross-attention mechanism between the intermediate embeddings of the image and the component-wise noising state $\bm{\gamma}$, allowing the network to effectively modulate its predictions based on the noising state at each stage of the diffusion process.
Otherwise, we follow a U-Net architecture similar to Song et al. (),

To incorporate additional information on the structure of the data, we first concatenate the noising state $\bm{\gamma}$ with position labels specific to the application and the choice of basis. For instance, for the PCA example in §5.1 this is the negative logarithm of the variance of each component, which is also used as the ordering variable. For the experiment in Fourier space, we used the FFT frequency label $|k|$.
In §5.2-5.3 we used a sequence of integers which increments by one for each subsequent column and adjacent group of Haar wavelet components, respectively.
The concatenated inputs are then processed through MLP-Mixer layers to facilitate learned embeddings of $\bm{\gamma}$.
Along the depth of the U-Net, a single dense layer is used to reduce the spatial extent to that of the coarse images at that level, before they are input into the cross-attention.

## Appendix B Experimental Details

#### PCA and Fourier bases.

In our experiments we make use of datasets of colored images, which have pixel and color channel indices.
Among our choices of basis is the Fourier basis. The Fourier transform (specifically the fast Fourier transform) is applied independently in each color channel. To have an analogous PCA basis, we have decided to perform the same orthogonal transformation – the one corresponding the PCA basis of the color-averaged data – in each color channel. It could be interesting to investigate further choices, including the PCA transformation that mixes the color channels.

#### Training.

Unless specified otherwise, training was done with a batch size of $128$ using the Adam optimizer with a learning rate of $5\times 10^{-4}$. The validation parameters used to evaluate sample quality are exponentially moving averages updated at a rate of $0.999$.
The diffusion times for denoising score matching are sampled uniformly in $[0,1]$, and schedule parameters (where applicable) were drawn uniformly from the specified range for each training batch of samples.

The score networks for both CIFAR-10 and PCAM were trained for $300$k training steps on NVIDIA A100 and H100s.
After this time the NLL appears to have converged. However, the FID in the case of CIFAR-10 keeps decreasing with further training.
For the experiments of §5.1 training was therefore resumed and continued for another $100$k steps and with a batch size of $512$ samples.
The FID has apparently not yet reached its minimum for the shown values, and we intend to update these once we have gained access to further computational resources.

#### Dataset processing.

We have used a uniformly dequantized version of the dataset, both for training and evaluation, by first adding uniform noise to each quantized pixel value and then rescaling it to $[-1,1]$.
We have additionally removed the empirical mean of the dataset, computed on all training data.
Otherwise, the mean would have to be taken into account when defining the magnitude-sensitive SNR, instead of just the variances as discussed in §4.2, and dividing by the mean when “whitening” could lead to extremely large values when the variance of a component is much smaller than its mean.

#### Scores representations.

The scores in different data representations, e.g. the original data ${\bm{\phi}}$ and the chosen components ${\bm{\chi}}$ in the notation of §4.1, are related by

$$ $\nabla_{{\bm{\chi}}}\log p_{t}({\bm{\chi}})=SU^{\dagger}\nabla_{\bm{\phi}}\log p _{t}({\bm{\phi}})\,,$ (23) $$

and we can always go back-and-forth between the scores in both basis.

As we base our architecture on the commonly used convolutional U-net architecture as in (Song et al., ), which implicitly assumes the locality and approximately shift-symmetric properties, we let the inputs and outputs of the score network to be always represented in the original image space and not the chosen PCA, Fourier, or other basis.

#### Evaluation and sampling.

Evaluation of the negative log-likelihood was done using the ODE corresponding to the reverse SDE equation 10.
Specifically, the SDE of equation 14, i.e.

$$ $d\chi_{i}=-{1\over 2}{\beta}_{i}(t)\chi_{i}\differential t+\sqrt{{\beta}_{i}(t )}\differential{\mathrm{w}}$ (24) $$

has a corresponding deterministic ODE that produces the same marginal probabilities, given by

$$ $d\chi_{i}=-\frac{1}{2}{\beta}_{i}(t)(\chi_{i}+\nabla_{\chi_{i}}\log p_{t}({\bm {\chi}}))\differential{t}\,.$ (25) $$

We use the above ODE to compute the log-likelihoods of the data under the trained models, with the score above replaced by its learned approximation (for more details we refer to Song et al. ()).
We used $6144$ samples in each computation, averaging over $3$ different slices of the Hessian in Hutchinson’s trace estimator for each sample.
For the integrator, we used Tsitouras’ 5/4 method as implemented in Kidger () with adaptive step size and both relative and absolute tolerance of $1\times 10^{-4}$.

## Appendix C Repeated Column-Wise Generation

The real-space sequential column of §5.2 generates images conditional on the left part of the image that has already been denoised, after an initial stage in which the first columns get denoised.
This immediately suggests an application in reconstructing an image that is only partially available.
Figure 8 shows how partially noised images can be reconstructed by filling in the right hand side of the image.
Different choices of random key then generate slightly different completions.

The linear column-wise schedule in equation 22 is defined such that by integrating the diffusion process by a time $\Delta t=b/(L-1)$, the “noising front” is effectively moved by one pixel. In other words, in the forward process, a particular column after this time has reached the same SNR its left neighbor had at the previous time.
Starting from an image at noising time $t>0$, we can generate in principal infinitely long strips of texture.
First, we denoise using the learned score from $t$ to $t+k\Delta t$ with $k<L$ a positive integer.
Then, we cut off the first $k$ columns and store them for the final image.
Next, we append $k$ columns of noise drawn from the prior to the right of the image.
As long as the softness parameter $b$ was chosen sufficiently large given the particular $k$, this effectively restores the image to the noising state at the original time $t$.
This process can thus be repeated, and by concatenating the previously generated left-side columns, a connected rectangular stripe of image is constructed.
As an example, in Figure 7 we show the results with $b=0.5$ and $k=9$.
Finally, note that this procedure only works if the training data is approximately translationally invariant.

Figure: Figure 8: Reconstruction of images from the test set (left column) partially noised to $t=0.5$ (second column) using a sequential schedule in real space as described in §5.2. The different reconstructions shown on the right differ by random key used.
Refer to caption: x7.png

## Appendix D Haar wavelets

The 2D Haar wavelet transform decomposes an image $X\in\mathbb{R}^{N\times N\times C}$ into low- and high-frequency components across multiple scales.
We apply the same wavelet transform to each color channel $c=1,\dots,C$ (for us $C=3$) separately.
Therefore, to ease the notation we will suppress the color index in what follows. The wavelet transform
at level $n$ is defined recursively as follows:

#### 1. Row Transformation

Apply the 1D Haar transform along the rows:

$$ $\displaystyle L_{i,k}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(X_{2i,k}^{(n-1)}+X_{2i+1,k}^{(n-1)} \right),$ $\displaystyle H_{i,k}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(X_{2i,k}^{(n-1)}-X_{2i+1,k}^{(n-1)} \right),$ $$

where $i=0,\dots,\tfrac{N}{2^{n}}-1$ and $k=0,\dots,N-1$.

#### 2. Column Transforms

Apply the 1D Haar transform along the columns to the results of the row transformation:

$$ $\displaystyle LL_{i,j}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(L_{i,2j}^{(n)}+L_{i,2j+1}^{(n)}\right),$ (26) $\displaystyle LH_{i,j}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(L_{i,2j}^{(n)}-L_{i,2j+1}^{(n)}\right),$ $\displaystyle HL_{i,j}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(H_{i,2j}^{(n)}+H_{i,2j+1}^{(n)}\right),$ $\displaystyle HH_{i,j}^{(n)}$ $\displaystyle=\tfrac{1}{\sqrt{2}}\left(H_{i,2j}^{(n)}-H_{i,2j+1}^{(n)}\right),$ $$

where $j=0,\dots,\tfrac{N}{2^{n}}-1$.

#### 3. High-Frequency Component:

Stack the high-frequency sub-bands into a single high-frequency array at level $n$:

$$ $HF^{(n)}=\text{concat}\left(LH^{(n)},\,HL^{(n)},\,HH^{(n)}\right)\,.$ (27) $$

#### 4. Recursive Decomposition

The low-frequency component $LL^{(n)}$ becomes the input for the next level:

$$ $X^{(n)}=LL^{(n)}\,.$ (28) $$

At each level, the transform produces one array of low-frequency components $LL^{(n)}$ and one array of high-frequency components $HF^{(n)}$. This process can be recursively applied up to a desired depth ${\cal N}$, resulting in a hierarchical decomposition of the image.

After level ${\cal N}$, the original image is represented by one lowest-frequency array and ${\cal N}$ higher-frequency arrays.

For example, for level 3 one obtains three high-frequency arrays $HF^{(1)}$, $HF^{(2)}$, $HF^{(3)}$,
and one coarse array $LL^{(3)}$.
To accommodate images with multiple color channels $C$, the transform is applied independently to each channel, and the resulting components are concatenated along the channel dimension.
The factors of $\sqrt{2}$ make sure that the transform is an orthogonal transformation, whose inverse can be computed analogously.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.

The time-dependent diffusion scale term has not been investigated in much detail as far as I am aware and I believe this should be the main focus of the paper or at least more attention. What are the benefits of this compared to cascading diffusion, can cascading be seen as a case of this?
soundness: 3
strengths: The paper is well explained. 

The authors bring attention to the flexibility in the diffusion model paradigm, though as discussed below this has been discussed in many prior papers.

The authors introduce what I believe to be a novel interpretation and use case for time-varying diffusion scale timers, leading to an autoregressive type forward process, applying noise to separate components independently. A similar procedure was used for diffusion in frequency space by applying different diffusion noise scales per frequency level [1] but these were not set to 0 as described here.


[1] Blurring diffusion models, Hoogeboom et al 2022
summary: The authors describe diffusion models with a under class of dynamics and marginals than the typical scaled Ornstein Uhlenbeck or Brownian motion reference processes used in the vast majority of diffusion model papers. In particular, the authors consider a linear transformation to perform the diffusion under a change of basis; varying the variance of the prior  Gaussian marginal  to match the data distribution; considering time dependent diffusion scale terms which can lead to auto-regressive-like dynamics.
weaknesses: ## Weakness 1
While the authors attempt to unify the design of dynamics for references; two of the three ideas proposed are not novel so it is unclear what the main contributions of the paper are. 

1) Using a change of basis
Applying diffusion in a transformed space / change of basis has been done before. Although [1] focuses on  change of basis to frequency basis, section 4.1 of [1] explicitly explains how any other change of basis can be performed.  I do not see any compelling evidence to suggest one basis over another in this submission.

2) Prior distribution
Discussion of variance of prior distribution was first discussed in [3], and referred to as Technique 1. This is still using diagonal covariance. 

It is not clear how scalable learning the covariance matrix for a full high dimensional data distribution would be or if it would even be beneficial.

The time-dependent diffusion scale term has not been investigated in much detail as far as I am aware and I believe this should be the main focus of the paper or at least more attention.

## Weakness 2
The second major weakness is in limited numerical evaluation. The FID scores shown for CIFAR10 are >20; significantly far from standard diffusion model performance of <3. It is not possible to evaluate whether there any benefit to generative modelling for the proposed methods without compelling numerical support.

Whilst I am not particularly interested in SOTA generative models FID <2, for toy datasets like CIFAR10 I would expect at least FID<4 given the abundance of code available for this and the limited novelty for 2/3 methods.

## Weakness 3
It is not clear to me the theoretical soundness of using the autoregressive approach for extending existing images i.e. changing dimension from previously trained model. It seems the generative process is no longer related to the time reversal of an SDE given the dimension changes. Can this be formalised?

Minor
- Blurring diffusion models [2] was a follow up to inverse Heat Dissipation Generative Model [1]. This should be cited and discussed as it was a pioneering paper in this area.


[1] GENERATIVE MODELLING WITH INVERSE HEAT DISSIPATION, Rissanen et al 2022
[2] Blurring diffusion models, Hoogeboom et al 2022
[3]Improved Techniques for Training Score-Based Generative Models, Song and Ermon, 2020

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors explain how to select in the large design space the various parameters/hyperparameters?

Can the authors briefly position wrt the works like the ones cited in the weaknesses section?

Minor:  Figure 7 is qualitatively difficult to interpret from someone not specialized in the field. I suggest the authors to either add some extra comments or produce a similar image for a dataset which is more understandable for a generic reader.
soundness: 3
strengths: The Generative Unified Diffusion (GUD) model provides a novel unification of diffusion and autoregressive generative approaches, allowing a flexible transition between simultaneous and sequential generation processes. This ability to bridge methods expands the framework’s application to a broad spectrum of tasks, from inpainting and sequential data extension to standard generative modeling. By creating a model that can interpolate between different generative styles, GUD allows developers to tailor the generation process to specific needs, enhancing control over the structure and dependencies of generated data.

One of GUD’s most notable strengths is its capacity for component-wise noise scheduling, which enables a hierarchical and selective approach to noising different parts of the data. This flexibility allows the model to prioritize important features by applying noise schedules tailored to specific components, leading to a more efficient and accurate generative process. Combined with its support for multiple basis representations—such as pixel, PCA, Fourier, and wavelet bases—GUD is adaptable to various data types and structures, making it particularly suitable for applications that benefit from multi-scale or hierarchical data representations.

Additionally, GUD’s design includes a whitening process, which aligns the data and noise distributions, providing better variance control throughout the generative process. This feature simplifies denoising and increases model stability, potentially reducing training time by minimizing noise-related artifacts. By supporting flexible basis selection, component-wise noise control, and variance alignment, GUD allows for refined generative modeling that can adapt to diverse tasks and applications, offering a powerful tool for high-quality, customizable data generation.
summary: This work introduces the Generative Unified Diffusion (GUD) model, a framework that expands the flexibility of diffusion-based generative models by enabling diverse configurations in basis representation, noise scheduling, and prior distributions. Standard diffusion models transform noise into data through a learned reverse process. Here, GUD leverages concepts from physics, specifically renormalization group flows, allowing distinct configurations in the process, such as using Fourier, PCA, or wavelet bases, and implementing component-wise noise schedules to tune noise levels for different data parts.

The GUD framework unifies diffusion and autoregressive models, bridging differences between simultaneous and sequential generation. It introduces soft-conditioning, where the model can conditionally generate components based on previously generated data, enabling partial dependency across features. The approach supports more efficient training, flexible architectural designs, and tasks requiring conditional generation, inpainting, or sequential extensions.

A key technical innovation is in the model’s flexibility of noise schedules and priors. GUD models allow each component a unique noise schedule, enabling a range of generation hierarchies from purely autoregressive (extreme component-wise scheduling) to standard diffusion. Additionally, a whitening transformation using PCA stabilizes the variance, simplifying the denoising process.

Experiments demonstrate the framework's adaptability across various data representations, including PCA, Fourier, and wavelet bases. By controlling softness and hierarchical order in noise schedules, GUD supports both hierarchical and spatially sequential generation, showing improved performance on benchmark image generation tasks, like CIFAR-10.
weaknesses: The GUD framework is flexible, and consequently introduces significant computational complexity. Each configuration, such as basis choice (PCA, Fourier, wavelet) and component-wise noise scheduling, requires tuning, making the model resource-intensive. This complexity can hinder scalability, especially in high-dimensional data applications where each choice impacts the computational load.

Architecturally, GUD’s design adds complexity by requiring modifications like cross-attention mechanisms for conditioning on component-wise noise states. These additions complicate the implementation and increase the risk of instability during training, as standard architectures like U-Nets are not inherently optimized for GUD’s intricate conditioning needs. This limitation might however be not so crucial.

Finally, I think the authors could have expanded the comparison with  related works. As (non exhaustive) examples, non isotropic noise perturbation has been considered in [1] and optimal steady state covariance wrt the data distribution has been investigated [2].


[1] Voleti et al, Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models, NeurIPS 2022 Workshop on Score-Based Methods.

[2] Das et al, Image generation with shortest path diffusion, ICML 2023

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors summarize the experimental results from Sec. 5.1, particularly regarding how the choice of basis, prior, and noising schedule contributes to performance compared with standard diffusion models?
soundness: 4
strengths: 1. The paper addresses limitations in standard diffusion models by proposing an interesting and innovative Generative Unified Diffusion (GUD) model.
2. The theoretical foundation of the paper is solid, and the presentation is clear.
3. The analyses and designs within the GUD framework are novel and potentially valuable across multiple applications.
summary: This paper proposes Generative Unified Diffusion (GUD), an extension of standard diffusion models based on the Ornstein-Uhlenbeck process. By defining appropriate orthogonal transformations, the authors introduce novel analyses and designs within the GUD framework, including SNR analysis, soft-conditioning, whitening, and orthogonal transformations. The authors conclude with experiments that validate these designs, showcasing GUD's potential in various applications.
weaknesses: 1. **Limited empirical evaluation:** The experiments primarily serve to validate the proposed designs (pixel/PCA/FFT). While these results offer some insights, the evaluation lacks depth, particularly in quantifying each design’s impact on GUD's performance. More comprehensive quantitative and qualitative results would better demonstrate the effectiveness of each design.

2. **Limited practical application contribution:** Although the paper suggests various potential applications, it appears these may not be fully viable in practice. Providing further insights into real-world application strategies would enhance the paper's practical relevance.

3. **Missing discussion of related work:** One significant application of GUD is the component-wise scheduling for different states used in sequential generation (as outlined in Sec. 5.2). As a comparison, [1,2] also propose distinct schedules for different components. Could the authors discuss these related works or provide a comparative analysis within the GUD framework?

[1] Rolling Diffusion Models, ICML 2024  
[2] Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, NeurIPS 2024

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: 1. **Unified Framework**: The authors present a cohesive framework for diffusion generative models, broadening design options.

2. **Structured Components**: The framework is well-organized around the Ornstein-Uhlenbeck process, prior distribution choice, and component-wise noise scheduling, enhancing its theoretical foundation.

3. **Diverse Design Examples**: The paper includes examples of various diffusion designs, demonstrating the framework’s flexibility and applicability.
summary: This paper proposes a unified framework for diffusion generative models, inspired by renormalization concepts from physics, allowing for flexible design choices in representation, prior distribution, and noise scheduling. The framework introduces soft-conditioning models that blend diffusion and autoregressive approaches, potentially enabling more efficient training and versatile generative architectures.
weaknesses: 1. **Limited Experimental Scope**: The experiments are primarily conducted on CIFAR-10, a relatively small dataset. Datasets with larger images would better validate the findings and demonstrate the framework’s effectiveness in diverse settings.

2. **Insufficient Explanation of Unified Diffusion and Autoregressive Generation**: The explanation on how the framework unifies standard diffusion and autoregressive generation lacks clarity. Providing a specific example of the component-wise noise schedule would enhance understanding and illustrate this unification more concretely.

Meta-review: This paper proposes the Generative Unified Diffusion (GUD) framework, which unifies diffusion and autoregressive models by introducing flexibility in data representation, noise scheduling, and prior distributions. While the theoretical foundation is intriguing and contributes to the design space for diffusion-based generative models, the submission lacks sufficient empirical evidence to validate its practical utility.

Decision: Reject