The Missing U for Efficient Diffusion Models
Authors: Sergio Calvo Ordoñez
Chun-Wun Cheng
Jiahao Huang
Lipei Zhang
Guang Yang
Carola-Bibiane Schönlieb
Angelica I Aviles-Rivero
Venue: Accepted by TMLR ()

Abstract: Diffusion Probabilistic Models stand as a critical tool in generative modelling, enabling the generation of complex data distributions. This family of generative models yields record-breaking performance in tasks such as image synthesis, video generation, and molecule design. Despite their capabilities, their efficiency, especially in the reverse process, remains a challenge due to slow convergence rates and high computational costs. In this paper, we introduce an approach that leverages continuous dynamical systems to design a novel denoising network for diffusion models that is more parameter-efficient, exhibits faster convergence, and demonstrates increased noise robustness. Experimenting with Denoising Diffusion Probabilistic Models (DDPMs), our framework operates with approximately a quarter of the parameters, and $\sim$ 30\% of the Floating Point Operations (FLOPs) compared to standard U-Nets in DDPMs. Furthermore, our model is notably faster in inference than the baseline when measured in fair and equal conditions. We also provide a mathematical intuition as to why our proposed reverse process is faster as well as a mathematical discussion of the empirical tradeoffs in the denoising downstream task. Finally, we argue that our method is compatible with existing performance enhancement techniques, enabling further improvements in efficiency, quality, and speed.

Published in Transactions on Machine Learning Research (04/2024)
The Missing U for Efficient Diffusion Models
Sergio Calvo- 2alvo=Ordoñez.,, Chun- Cheng', Jiahao Huang", Lipei Zhang', Guang Yang Carola-Bibiane Schönlieb", Angelica I Aviles-Rivero
Oxford- Man Institute of Quantitative Finance, University of Oxford Mathematical Institute, University of Oxford Department of Applied Mathematics and Theoretical Physics, University of Cambridge Bioengineering Department and Imperial-X & National Heart and Lung Institute, Imperial College London CCaddiouscularr Research Centre, Royal Brompton Hospital London School of Biomealccal Engineering Imaging Sciences, King's 's College London
Reviewed on OpenRevicw: https://openrevieww.tt http://wwww...ttooo//oo/ooaaaaaa
Abstract
Diffusion Probabilistic Models stand as critical tool in generative modelling, cnabling the generation of complex data distributions. This family of generative models yields record- breaking performance in tasks such as image synthesis, video generation, and molecule design. Despite their capabilities, their efficiency, especially in the reverse process, remains a challenge due to slow convergence rates and high computational costs. In this paper, we introduce an approach that leverages continuous dynamical systems to design a novel denoising network for diffusion models that is more parameter-effiiint,, exhibits faster convergence, and demonstrates increased noise robustness. Experimenting with Denoising Diffusion Probabilistic Models (DDPMs), our framework operates with approximately a quarter of the paramcters, and ~ 30% of the Floating Point Operations (FLOPs) compared to standard U-Nets in DDPMs. Furthermore, our model is notably faster in inference than the baseline when measured in fair and equal conditions. We also provide a mathematical intuition as to why our proposed reverse process is faster as well as mathematical discussion of the empirical tradcoffs in the denoising downstream task. Finally, we argue that our method compatible with existing performance enhancement techniques, enabling further improvements in efficiency, quality, and speed.
1 Introduction
Diffusion Probabilistic Models, grounded in the work of (Sohl-Dickstein al., 2015) and expanded upon by (Song & Ermon 2020; Ho et al. 2020; Song et al. 2020b), have achieved remarkable results in various domains, including image generation (Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022b), audio synthesis (Kong et al., 2021; Liu et al., 2022a), and video generation (Ho et al., 2022; Ho et al., 2021). These score- score-based generative models utilise an iterative sampling mechanism to progressively denoise random initial vectors, offering a controllable trade-off between computational cost and sample quality. Although this iterative process offers a method to balance quality with computational expense, it often leans lowards the latter for stale-of-the-art results. Generating top-tier samples often demands a significant number of iterations, with the diffusion models requiring up to 2000 times more computational power compared to other generative models (Goodfellow et al., 2020; Kingma. & Welling, 2013; Rezende et al., 2014; Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018).
Recent research has delved into strategies to enhance the efficiency and speed of this reverse process. In Early- stopped Denoising Diffusion Probabilistic Models (ES-DDPMs) proposed by (Lyu al., 2022), the diffusion process is stopped early. Instead of diffusing the data distribution into a Gaussian distribution via hundreds of iterative steps, ES-DDPM considers only the initial few diffusion steps that the reverse denoising process

---

Published in Transactions on Machine Learning Research (04/2024)
starts from a non-Gaussian distribution. A similar method used in (Zheng et al., 2023b), also truncates the forward process allowing for fewer reverse steps to generate the data. Additionally (Xiao et al., 2022) reduce the sampling process overhead by modelling the denoising distribution using a complex multimodal distribution with a denoising diffusion generative adversarial network for each step. (Lu et al., 2022) propose an exact lormulation of the solution of diffusion ODEs, allowing sampling in a few steps. Continuing with faster sampling methodologics, (Zhang & Chen, 2023) present Diffusion Erponential Integrator Sampler which leverages a semilinear structure of the learned diffusion process to reduce the discretisation error and is more efficient. Another significant contribution is the Analytic-DPM framework (Bao et al., 2022). This training-free inference framework estimates the analytic forms of variance and Kullback-Leibler divergence using Monte Carlo methods in conjunction with a pre-trained score- based model. Results show improved log-likelihood and a speed-up between 20x to 80x. Furthermore, approaches that study using manifold constraints and inverse problems hypothesis for diffusion models (Chung et al. 2022; Liu et al. 2022b; Chung et al. 2023; Rout al. 2023; Lou & Ermon 2023) achieve significant performance boost. Other lines of work focused on modifying the sampling process during the inference while keeping the model unchanged. (Song et al., 2020a) proposed Denoising Diffusion Implicit Models (DDIMs) where the reverse Markov chain is altered to take deterministic jumping steps composed of multiple standard steps. This reduces the steps required but may introduce discrepancies from the original diffussio process. (Nichol & Dhariwal, 2021) proposed timestep respacing to non-uniformly select timesteps in the reverse process. While reducing the total number of steps, can cause deviation from the model's training distribution. In general, these methods provide inference-time improvements but do not accelerate model training.
A different approach trains diffusion models with continuous timesteps and noise levels to enable variable numbers of reverse steps after training (Song & Ermon, 2020). Models trained directly continuous objectives outperform discretely-trained models on continuous data where the score function is properly defined (Song et al. 2020b; Karras et al. 2022). (Kong et al., 2021) approximate continuous noise levels through interpolation of discrete timesteps, but lack theoretical grounding. Orthogonal strategics accelerate diffusion models by incorporating conditional information. (Preechakul et al., 2022a) inject an encoder vector to guide the reverse process. While elfective for conditional lasks, it provides limited improvements for unconditional generation. (Salimans & Ho, 2022) distil a teacher model into students taking successively fewer steps, reducing steps without retraining, but distillation cost scales with teacher steps. Unlike existing work, we underline that our approach diverges by parameterising the dynamics via a second-order ODE that, specifically models acceleration in the reverse process.
To tackle these issucs, throughout this paper, we construct and evaluate an approach that rethinks the reverse process diffusion models by fundamentally altering the demoising network architecture. Current literature predominantly employs U-Net architectures for the discrete denoising of diffused inputs over a specified number of steps. Many reverse process limitations stem directly from constraints inherent to the chosen denoising network. Building on the work of (Cheng et al., 2023), we leverage continuous dynamical systems to design novel denoising network that is parameter-efficient, exhibits faster and better convergence, demonstrates robustness against noise, and outperforms conventional U-Nets while providing theoretical underpinnings. We show that our architectural shift directly euhances the reverse process of diffusion models by offering comparable performance in image synthesis but an improvement in inference time in the reverse process, denoising performance, and operational efficiency. Importantly, our method orthogonal to existing performance enhancement techniques, allowing their integration for further improvements. Furthermore, we delve into a mathematical discussion to provide foundational intuition as to why it is a sensible design choice to use our deep implicit layers in a denoising network that is used iteratively in the reverse process. Along the same lines, we empirically investigate our network's performance at sequential denoising and theoretically justify the tradeoffs observed in the results. Besides our framework's compatibility with other families of diffusion models (as discussed in Section 3), the method could be leveraged for downstream tasks in other areas, for example, and not limited to, MRI reconstruction, audio generation, image segmentation, or synthetic data generation. In particular, our contributions are:
We propose new denoising network that incorporates an original dynamic Neural ODE block integrating residual connections and time embeddings for the temporal adaptivity required by diffusson models.
2

---

Published in Transactions on Machine Learning Research (04/2024)
We develop novel family of diffusion models that uses deep implicit U-Net denoising network; as alternative to the standard discrete U-Net and achieve enhanced efficiency.
We evaluate our framework, demonstrating competitive performance in image synthesis, and perceptually outperforms the baseline in denoising with approximately 4x fewer parameters, smaller memory footprint, and shorter inference times.
2 Preliminaries
This section provides a summary of the theoretical ideas of our approach, combining the strengths of continuous dynamical systems, continuous U-Net architectures, and diffusion models.
Denoising Diffusion Probabilistic Models (DDPMs). These models extend the framework of DPMs through the inclusion of a denoising mechanism (Ho et al., 2020). The latter is used an inverse mechanism to reconstruct data from a latent noise space achieved through a stochastic process (reverse diffusion). This relationship emerges from (Song et al., 2020b), which shows that a certain parameterization of diffusion models reveals an oquivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling. DDPMs can be thought of as analog models to hierarchichal VAEs (Cheng et al., 2020), with the main difference being that all latent states, Zy for t [1, have the same dimensionality as the input zo. This detail makes them also similar to normalizing flows (Rezende & Mohamed, 2015), however, diffusion models have hidden layers that are stochastic and do not need to use invertible transformations.
Neural ODEs. Neural Differential Equations (NDEs) offer a continuous-time approach to data modelling (Chen al., 2018). They are unique in their ability to model complex systems over time while efficiently handling memory and computation (Rubanova et al., 2019). A Neural Ordinary Differential Equation is specific NDE described as:
dy di (1) = ft(t)(())),
v() = yo:
(1)
where yo E Rdc x-- refers to an input tensor with any dimensions, 0 symbolizes a learned parameter vector, and So I x Bdxx...dxx + Rdi xxdxx a neural network function. Typically, Jo is parameterized by simple neural architectures, including feedforward or convolutional networks. The selection of the architecture depends the nature of the data and is subject to efficient training methods, such the adjoint sensitivity method for backpropagation through the ODE solver.
Continuous U-Net. (Cheng et al., 2023) propose a new U-shaped network for medical image segmentation motivated by works in deop implicit lcarning and continuous approaches based on neural ODEs (Chen et al., 2018; Dupont et al., 2019). This novel architecture consists of a continuous deep network whose dynamics are modelled by socond-order ordinary differential equations. The idea is transform the dynamics in the network previously CNN blocks into dynamic blocks to get solution. This continuity comes with strong and mathematically grounded benefits. Firstly, by modelling the dynamics in higher dimension, there is more flexibility in learning the trajectories. Therefore, continuous U-Net requires fewer iterations for the solution, which is more computationally efficient and in particular provides constant memory cost. Secondly, it can be shown that continuous U-Net is more robust than other variants (CNNs), and (Cheng et al., 2023) provides an intuition for this. Lastly, because continuous U-Net is always bounded by some range, unlike CNNs, the network is better at handling the inherent noise the data.
3 Related Work
A significant amount of research focuses on speeding up diffusion models. In this section, we offer an overview of these strategies and detail how our method distinctly contributes to improving diffusion model efficiency. We also argue our method's orthogonality by outlining potential ways to incorporate our framework into existing works, underscoring its compatibility and additive impact on the field.
3

---

Published in Transactions on Machine Learning Research (04/2024)
The work introduced by (Wang et al, 2022), introduces an efficient diffusion model tailored for generating diverse textures and patterns from single image, leveraging unique approach to learn the distribution of image patches, enabling high-quality synthesis with minimal data. Patch Diffusion (Wang et al., 2023), introduces a novel training approach for diffusion models, significantly reducing training time and enhancing data efficiency by learning conditional score functions on image patches of varying sizes and locations. (Zheng et al., 2023a) look into accelerating diffusion models through a novel sampling technique that uses neural operators to solve the probability flow ODE. (Arakawa et al., 2023) present proposal that uses a patch-based diffusion probabilistic model that divides the images into patches and generates them independently to reduce memory consumption during inference. Each of these methods introduces unique efficiencies within the diffusiin modelling framework, yet they all use U-Net architectures for the denoising process, therefore providing benefits which are independent of the denoising network choice. This suggests that there is room for improvement through the implementation of our method to further increase efficiency and reduce memory roquirements while preserving image quality.
(Gao et al., 2023) and (Zheng et al., 2023c) present novel approaches to enhance diffusson models, with the former introducing mask latent modelling scheme for improved learning speed and the latter focusing integrating local fcatures and global content for efficiency gains. Despite their advancements, (Gao et al., 2023) report no increase in memory efficiency, and the methodology in (Zheng et al., 2023c) has not been extensively tested across all diffusion model applications, in some cases resulting in lower quality outputs.
Furthermore, there is a branch of literature that explores applying the efficiency benefits of variational autoencoders (VAEs) diffusion models. (Preechakul et al., 2022b) presents a method combining a learnable encoder for capturing high-level semantics with a diffusion probabílistic model (DPM) as the decoder for modelling stochastic variations, showing improved efficiency and generation quality over DDIMs. However, its performance relative to newer methods and its compatibility with other acceleration techniques remains unclear, as the diffusion process is secondary to encoding. (Pandey et al., 2022) presents a framework that integrates the ability to operate in a low-dimensional latent space through VAEs while still modelling a stochastic process with diffusion mechanisms. Similarly, (Rombach et al., 2022a), apply diffusion models in the latent space of powerful pre-trained autoencoders. However, this is limited by the bottleneck of the pre-trained autoencoder. In all of these cases, their use of U-Net architecture for denoising aligns with our framework, suggesting the potential for integration to enhance efficiency further. (Vahdat et al., 2021) train Score-based Generative Models (SGM) in latent space to make them more efficient with the reduction of the dimensionality. However, although improving upon the inefficiencies of diffusion processes, it still is considcrably slower than similar techniques applicd in discrete-time frameworks.
Lastly, other works introduce general methods to improve different stages of diffusion models. Karras et al. (2022) propose an analysis that helps simplify the stages of diffuusion models that use neural networks (e.g. U-Nets) to model the score of noise level-dependent marginal distribution of the training data corrupted by noise. Dockhorn et al. (2022) introduce eritically-damper Langevin diffusion simplifying the score-matchin matching process by only needing to learn the score-function of the conditional distribution of a subset of the variables. Finally, (Pandey & Mandt, 2023) introduce a similar concept, Phase Space Langevin Diffusion (PSLD). This novel SGM enhances sample quality and optimises the speed-quality trade-off by performing diffusion in an augmented space with auxiliary variables. However, this technique is only applicable in frameworks that use fully continuous points of view of the forward and reverse processes.
Below, we describe our methodology and where each of the previous concepts plays an important role within our proposed model architecture.
4 Methodology
In standard DDPMs, the reverse process involves reconstructing the original data from noisy observations through a scries of discrete steps using variants of U-Net architecture. In contrast, our approach (Fig. 1) employs a continuous U-Net architecture to model the reverse process in locally continuous-time setting'.
The locally continuous-time setting denotes hybrid method where the main training uses discretised framework, but each step involves contimous-time modelling the image's latent, representation, driven by a neural ordinary differential equation.
4

---

Published in Transactions on Machine Learning Research (04/2024)
Noise
Diffusion Process
Input Image
T
2T
Denoising Continuous U-Net
-1)
Dymamic Blocks
2T-1
Visual representation of our framework featuring implicit deep layers tailored for denoising in the reverse process of DDPM, enabling the reconstruction of the original data from a noise-corrupted version.
Figure 1:
Unlike previous work on continuous U-Nets, focusing on segmentation (Cheng al., 2023), we adapt the architecture carry out denoising within the reverse process of DDPMs, marking the introduction of the first continuous b-seet-based denoising network. Our changes touch upon:
Model Adaptation for Denoising: We have reconfigured the architecture to better suit denoising tasks. This includes adjustments in output channels, transition from categorical cross-entropy loss to a reconstruction-bent based loss for minimising pixel discrepancies, and stride modifications to preserve spatial resolution.
Temporal Dynamics Integration: Time embeddings have been introduced, following the approach by (Ho et al., 2020), to accurately model and adapt to the diffusion process across time, enhancing the model's ability to dynamically adjust to various diffusion stages.
Architectural Innovations: Our continuous U-Net now incorporates attention mechanisms and residual comnections, aiming to capture long-range dependencies and improve noise management capabilities, marking a departure from traditional designs towards a more dynamic and efficient architecture.
Overall, our architecture is strategically rooted in capability to significantly roduce computational cost. without increasing it. This reduction achieved through the decreased necessity for storing active functions and leveraging the adjoint sensitivity method, which guarantees O(1) memory cost regardless of model complexity. This approach inherently builds reversibility into the architecture, ensuring efficient memory usage and substantially reducing computation time.
4.1 Dynamic Blocks for Diffusion
Our dynamical blocks are based on socond-ordor ODEs, therefore, we make use of an initial velocity block that determines the initial conditions for our model. We leverage instance normalisation, and include sequential convolution operations to process the input data and capture detailed spatial features. The first convolution transitions the input data into an intermediate representation, then, further convolutions refine and expand the feature channels, ensuring à comprehensive representation of the input. In between these operations, we include ReLU activation layers to enable the modelling of non-linear relationships as a standard practice due to its performance (Agarap, 2019).
Furthermore, our design incorporates a neural network function approximator block (Fig. right), representing the derivative in the ODE form [1,, 2) which dictates how the hidden state evolves over
5

---

Published in Transactions on Machine Learning Research (04/2024)
the continuous-tine variable t. Group normalisation layers employed for feature scaling, followed by convolutional operations for spatial feature extraction. In order to adapt to diffusion models, we integrate time embeddings using multi-layer perceptrons that adjust the convolutional outputs via scaling and shifting and are complemented by our custom residual connections. Additionally, we use an ODE block (Fig. left) that captures continuous-ling dynamics, wherein the evolutionary path of the data defined by an ODE function and initial conditions derived from precoding blocks.
Iopu:
Input [v.2)
ODE FUECTION A AUTBUSSRNROR
Group Nortn.
DYNAMIC BASED ODE BLOCK
Time ruah
Marbod, Integration Time
Conv. Layer
(MLP)
Softplus
Sexde Shift
ODE CINecTTO PUSC
xfr) XA
Grop Norm.
Couv. Layer
Residnal Connection
ODE Souvan
Softplus
Output
Outpul
Figure 2: Architectural components of the contimuus U-Net. On the left, the Dynamic ODE Block represents the core unit of our continuous model, detailing the inlegration of the ODE solver and function approxinator within the network structure. The right panel expands on the ODE Function Approximator, highlighting the convolutional layers, group normalisation, time embeddings, and the incorporation of scale, shift operations, and residual connections to accurately adapt the network's dynamics during the diffusion process.
4.2 A New 'U' for Diffusion Models
As we fundamentally modify the denoising network used in the reverse process, relevant to look into how the mathematical formulation of the reverse process DDPMs changes. The goal is to approximate the transition probability using our model. Denote the output of our continuous U-Net Üx,,, f, E; M), where x, is the input, is the time variable related to the DDPMs, f is the time variable related to neural ODEs and V represents the paramelers of the network including O from the dynamic blocks built into the architecture. We use the new continuous U-Net while keeping the same sampling process (Ho et al., 2020) which reads
1 vī-
co( co(e,t)) + 012 where 2 ~ I)
B
(2)
V ac
As opposed to traditional discrete U-Net models, this reformulation enables modelling the transition probability using the continuous-time dynamics encapsulated in our architecture. Going further, we can represent the continuous U-Net function terms of dynamical blocks given by:
c0(2e, (2
(3)
where,
b = (() 2/z. 1, Fig Xo. 1// = g 0g)
(4)
Here, x represents the second-order derivative of the state with respect time (acceleration), f(a) (...., 07) is the neural network parametrising the acceleration and dynamics of the system, and Dig and a/ the

---

Published in Transactions on Machine Learning Research (04/2024)
initial state and velocity. Xu is the initial value and g( 0g) is the neural network parameterising the velocity. Then we can update the iteration by 2t to xt-1 by the continuous network.
4.3 Unboxing the Missing for Faster and Lighter Diffusion Models
Our architecture outperformed DDPMs in terms of efficiency and accuracy (Table 1). This section provides mathematical justification for the performance. We first show that the Probability Flow ODE is faster than the stochastic differential equation (SDE). This is shown when considering that the SDE can be viewed the sum of the Probability Flow ODE and the Langevin Differential SDE in the reverse process (Karras et al., 2022). We can then define the continuous reverse SDE (Song et al., 2020b)
dz, (f V p + g(t)dur
(5)
We can also define the probability flow ODE as follows:
dr [f 2(t)2 lo
(6)
We can reformulate the expression by setting ( b) - g(t) = / and so, (xx) V log (x+). Substituting these into (5) and (6) yields the following equations for the SDE and Probability Flow ODE, respectively.
1 dr, = B(t) + 25, (x:)]dt + B(t)dww
(7)
B(t) dx, - jare + so, (x), t)
(8)
We can then perform the following operation:
dx; - 2 3(1(zz + 250, (xt) dt + / B(t)dd:
(9)
3(t) + s9, (x:)]dt 2 (t)ss, (2,, t)dt + v B(t)dwr
Expression (9) decomposes the SDE into the Probability Flow ODE and the Langevin Differential SDE. This indicates that the Probability Flow ODE is faster, as discretising the Langevin Differential equation is time-consumingg However, we deduce from this fact that although the Probability Flow ODE is faster, is less accurate than the SDE. This key reason for our interest in second-order neural ODEs, which can enhance both speed and accuracy. Notably, the Probability Flow ODE is form of first-order neural ODEs, utilising an adjoint state during backpropagation. But what exactly is the adjoint method in the context of Probability Flow ODE? To answer this, we give the following proposition.
Proposition
Proposition 4.1 The adjoint slate r. of probabilily flow ODE follows the first order ODE
*. T 0 (() x, se, (xt, t)]
(10)
0X.
Proof. Following (Norcliffe et al., 2020), dl respect to parameter 0 as dø 8z.m
we denote the scalar loss function be L = L(xr,), ) and the gradient
0L
daan dB
Then It,, follows:
Fi.- ftm rddt + Tta
(11)
Pto f(Xu, 0y), x/ = ]0(() so, (x),
Let. K be variable such that satisfying the following integral:

---

Published in Transactions on Machine Learning Research (04/2024)
in
K
z/dt
(12)
pnn
(x'e A(t)[2] 1 3(6] so, ( (ox, t]]l f]ll)at dt + B(x10 - f)
Then we can take derivative of K respect to 0
dK
te
x1 dl +
1 A(1)
pn
dx' 012
so, (Fr, 1)
0 (1)3()) 24 s0, (2ees ti]
di
to
do
do
00
Ox
di
of
+
B
(13)
d6
d6
Use the freedom of choice of (t)) and B, then can get the following first-order adjoint state.
Y= -/--7 028(( Jt 80, (2t, ()
(14)
0X,
As observed, the adjoint state of the Probability Flow ODE adheres to the first-order ODE, where gradients are calculated by performing backward integration of both the adjoint state, r, and the real state, I, through time. This approach not only obviates the need to store intermediate values thereby utilising a fixed amount of memory and offering substantial advantage over traditional backpropagation methods but also lays the groundwork for our model's efficiency. By repurposing the first-order adjoint method within our sccond-order neural ODE framework, significantly enhance computational efficiency. This strategic choice is rooted in the finding that the compulation cost of the second-order adjoint method is, at a minimum, comparable to that of the first-order adjoint method, with the latter often requiring less computation time and cost. Such an integration of the probability flow ODE with our model architecture directly contributes to improved accuracy and speed, leveraging the universal approximation theorem, higher diffcrentia billity,, and the expanded flexibility of second-order neural ODEs for transformations beyond homeomorphic shifts real space.
There is still final question in mind, the probability flow ODE is for the whole model but continuous U-Net optimises in every step. What is the relationship between our approach and the DDPMs? This can be answered by a concept from numerical methods. If a given numerical method has local error of O(h++1), then the global error is O This indicates that the order of local and global errors differs by only one degree. To better understand the local behaviour of our DDPMs, we aim to optimise them at each step. This approach, facilitated by a continuous U-Net, allows for more detailed comparison of the order of convergence between local and global errors.
5 Experimental Results
In this section, detail the set of experiments to validate our proposed framework.
5.1 Image Synthesis
We evaluated our method's efficacy via generated sample quality (Fig. 3). As baseline, we used a DDPM that uses the same U-Net described in (Ho et al., 2020). Samples were randomly chosen from both the baseline DDPM and our model, adjusting sampling timesteps across datasets to form synthetic sets. By examining the FID (Fréchet distance) measure as a timestep function on these datasets, we determined optimal sampling times. Our model consistently reached optimal FID scores in fewer timesteps than the C-Net-based model (Table 1), indicating faster convergence by our continuous U-Net- approach.
To compute the FID, we generated two datasets, each containing 30,000 generated samples from each of the models, in the same way as generated the images shown in the figures above. These new datasets then
8

---

Published in Transactions on Machine Learning Research (04/2024)
IDPM NU.)et)
DDPM (oU-Nes)
Figure 3: Randomly selected generated samples by our model (right) and the baseline U-Net-based DDPM (left) trained CelebA and LSUN Church.
MNIST (28 x 28)
CELEBA (64 x 64)
LSUN CHURCH (128
128)
BACKBONE
FID
+
Steps
Time (s) 4
FID +
Steps
Time (s) 4
FID
Steps
Time (s) 4
U-Net
3.61
30
3.56
19.75
100
12.48
12.28
100
12.14
U-Net
3.73
11
1.12
19.54
30
3.08
11.89
40
4.25
cU-Net
2.98
5
0.54
21.44
80
7.36
12.14
90
8.33
cU-Net" 2.55
2
0.18
21.41
15
1.37
11.77
30
2.68
Table 1: Performance metrics across datasets: FID scores (measured every 5 steps for CelebA and LSUN, and at every step for MNIST), sampling timesteps (Steps), and average generation time for both the U-Net and continuous U-Net (cU-Net) models. t indicates that the model used a. DDIM sampler at inference time rather than the traditional DDPM sampler. As shown, the efficiency benefits are maintained across the different. samplers.
directly used for the FID score computation with a batch size of 512 for the feature extraction. We also note that we use the 2048-dimensional layer of the Inception network for feature extraction as this is a common choice to capture higher-level features.
We examined the average inference time per sample across various datasets (Table 1). While both models register similar FID scores, cU-Net infers notably quicker, being about 30% to 80% faster?. Notably, this enhanced speed and synthesis capability is achieved with marked parameter efficiency discussed further in Section 5.3.
5.2 Image Denoising
Denoising is essential in diffusion models to approximate the reverse of the Markov chain formed by the forward process. Enhancing denoising improves the model's reverse process by better estimating the data's conditional distribution from corrupted samples. More accurate estimation means better reverse steps, more significant transformations each step, and hence samples closer the data. A better denoising system, therefore, also speed up the reverse process and save computational effort.
In experiments, the process of noising images is tied to the role of the denoising network during the reverse process. These networks use timesteps to approximate the expected noise level of an input image at a given time. This is done through the time embeddings which help assess noise magnitude for specific timestops. Then, accurate noise levels are applied using the forward process to a certain timestep, with images gathering more noise over time. Figure shows the process noise accumulation time, which is central the functionality of diffusion models. By visualising noise at different timesteps, we demonstrate the correlation between timestep advancement and noisc intensity. This not only validates the
images over
Note that inference times reported for both models were measured on CPU, current Python OoEsso packages utilise GPU resources effectively, unlike the highly optimised cade of conventional U-Net. convolutional layers.

---

Published in Transactions on Machine Learning Research (04/2024)
Clean Image
Noise for 50 timesteps
Noise for 150 timesteps
Noise for 400 timesteps
Figure 4: Visualisation of noise accumulation in images over incrcasing timesteps. As timesteps advance, the images exhibit higher levels of noise, showcasing the correlation between timesteps and noise intensity. The progression highlights the effectiveness of time embeddings in predicting noise magnitude specific stages of the diffusion process.
NOISING TIMESTEPS
BEST SSIM VALUE
BEST LPIPS VALUE
50
0.88 0.90
0.025
0.019
100
0.85 0.83
0.044 0.038
150
0.79 0.78
0.063 0.050
200
0.74 0.71
0.079 0.069
250
0.72 0.64
0.104 0.084
400
0.58
0.44
0.184 0.146
600
0.44
0.26
0.316
0.238
800
0.32
0.18
0.419
0.315
Table Comparative average denoising performance between U-Net (left values) and cU-Net (right values) for different noise levels over the test dataset. While U-Net predominantly achieves higher SSIM scores, cU-Net. often outperforms LPIPS evaluations, indicating differences in the nature of their denoising approaches.
progression hypothesis of noise in the diffusion process but also the effectiveness of our continuous U-Net model's time embeddings.
In our denoising study, we evaluated 300 images for average model performance across noise levels, tracking SSIM and LPIPS over many timesteps gauge distortion and perceptual output differences. Table 2 shows the models' varying strengths: conventional U-Net scores better in SSIM, while our models perform better in LPIPS. Despite SSIM being considered as metric that measures perceived quality, it has been observed to have a strong correlation with simpler measures like PSNR (Horé & Ziou, 2010) due to being a distortion measure. Notably, PSNR tends to favour over-smoothed samples, which suggests that a high SSIM score may not always correspond to visually appealing results but rather to an over-smoothed image. This correlation underscores the importance of using diverse metrics like LPIPS to get a more comprehensive view of denoising performance.
The U-Net results underscore prevalent issue in supervised denoising. Models trained paired clean and noisy images via distance-based losses often yield overly smooth denoised outputs. This is because the underlying approach frames the denoising lask as a deterministic mapping from a noisy image to its clean counterpart x. From a Bayesian viewpoint, when conditioned on æ, follows a posterior distribution:
(<>ly) <(vl=)((x) q(v)
(15)
With the L2 loss, models essentially compute the posterior mean, E[ply], elucidating the observed over- smoothing. As illustrated in Fig. (and further results in Appendix A), our model delivers consistent detail preservation even amidst significant noise. In fact, at high noise levels where either model is capable of recovering fine- grained details, our model attempts to predict the features of the image instead of prioritising the smoothness of the texture like U-Net.
10

---

Published in Transactions on Machine Learning Research (04/2024)
CLEAN IMAGE
NOISY IMAGE
DDPM (CUNET)
DDPM (UNET)
10 TIMESTEPS
40 TIMESTERS
60 TIMESTER
70 TIMESTEPS
Figure 5: Original image (left), with Gaussian noise (second), and denoised using our continuous U-Net (third and fourth). As noise increases, U-Net struggles to recover the fine-grained details such as the glasses.
Furthermore, Figures and 11 Appendix B depict the Perception-Distonn Distortion tradeoff. Intuitively, this is that averaging and blurring reduce distortion but make images look unnatural. As established by (Blau & Michaeli, 2018), this trade-off is informed by the total variation (TV) distance:
drv(px.Px) = - (x)
(16)
where Px the distribution of the reconstructed images and Px the distribution of the natural images. The perception-distorionn function PDD) is then introduced, representing the best perceptual quality for a given distortion D:
11

---

Published in Transactions on Machine Learning Research (04/2024)
Noise Steps
Best SSIM Step
Time SSIM (s)
Best LPIPS Step
Time LPIPS (s)
50
47
39
5.45
4.40
41
39
4.71
4.40
100
93
73
19.72
9.89
78
72
16.54
9.69
150
140
103
29.69
14.27
119
102
25.18
13.88
200
186
130
39.51
18.16
161
128
34.09
17.82
250
232
154
49.14
21.59
203
152
43.15
21.22
400
368
217
77.33
29.60
332
212
69.77
29.19
600
548
265
114.90
35.75
507
263
106.42
/ 35.49
800
731
284
153.38
39.11
668
284
140.26
39.05
Table 3:
Comparison of average performance for U-Net (left) and cU-Net (right) at different noise levels in terms of the specific timestep at which peak performance was at tained and time taken. These results are average across all the samples in our test set.
50 Timesteps
150 Timesteps
400 Timesteps
METHOD
SSIM
LPIPS
SSIM
LPIPS
SSIM
LPIPS
BM3D
0.74
0.062
0.26
0.624
0.06
0.977
Conv AE
0.89
0.030
0.80
0.072
0.52
0.204
DrCNN
0.89
0.026
0.81
0.051
0.53
0.227
Diff U-Net
0.88
0.025
0.79
0.063
0.58
0.184
Diff cU-Net
0.90
0.019
0.78
0.050
0.44
0.146
Table 4: Comparative average performance of various denoising methods at select noise levels across the test set. Results demonstrate the capability of diffusion-basch models (Diff U-Net and Diff cU-Net) in handling a broad spectrum of noise levels without retraining.
P(D)
min drv(p2.Px) s.t. F[A(X, S D.
(17)
Pxiy
In this equation, the minimization spans over estimators Pxjr: and A(X,8) characterizes the distortion metric. Emphasizing the convex nature of P(D), for two points (D1, P(Di)) and (D2, P(D2)), we have:
- AP(Di) + -1- 2 P(AD, + (1-11))0 D2)));
(18)
where À is scalar weight that is used to take convex combination of two operating points. This convexity underlines rigorous trade-off at lower D values. Diminishing the distortion beneath specific threshold demands significant compromise in perceptual quality. Additionally, the timestep at which each model achieved peak performance in terms of SSIM and LPIPS was monitored, along with the elapsed time required to reach this optimal point. Encouragingly, our proposed model consistently outperformed this aspect, delivering superior inference speeds and requiring fewer timesteps to converge. These promising results are compiled and can be viewed in Table 3.
We benchmarked the denoising performance of our diffusion model's reverse process against established methods, including DrCNN (Zhang el al., 2017), convolutional autoencoder, and BM3D (Dabov el al., 2007), as detailed in Table 4. Our model outperforms others at low timesteps in both SSIM and perceptual metrics. At high timesteps, while the standard DDPM with U-Net excels SSIM, our cUNet lcads perceptual quality. Both U-Nets, pre-trained without specific noise-level training, effectively demoise across a broad noise spectrum, showcasing superior generalisation compared to other deep learning techniques. This illustrates the advantage of diffusion models' broad learned distributions for quality denoising across varied noise conditions.
5.3 Efficiency
Deep learning models often demand substantial computational resources due to their parameter-heary nature. For instance, in the Stable Diffusion model (Rombach et. al., 2022b) state-of-the-art text-to-image
12

---

Published in Transactions on Machine Learning Research (04/2024)
diffusion model the denoising U-Net consumes roughly 90% (860M of 983M) of the total parameters. This restricts training and deployment mainly to high-performance environments.
The idea of our framework is to address this issue by provid- ing a plug-and-play solution to improve parameter efficiency significantly. Figure illustrates that our cUNet requires only 8.8M parameters, roughly quarter of standard UNet. Maintaining architectural consistency across comparisons, our model achieves this with minimal performance trade-offs. In fact, it often matches or surpasses the U-Net in denoising capabilities.
U-net
eU-Net
cU-Neet Net wo/A
cU cU-Nee wo/R
eU-Net wo/A/R
While our focus is on DDPMs, cUNet's modularity should make it compatible with a wider range of diffusion models that also utilize U-Net-lype architectures, making our approach potentially beneficial for both efficiency and performance across broader range of diffusion models, CUNet's efficiency, reduced FLOPs, and memory conservation (Table 5) could potentially offer a transformative advantage as they minimize computational demands, enabling deployment on personal computers and budget-friendly cloud solutions.
0m+00
1e+077
2c+-07
3e+07
Parameters
Figure 6: Total number of parameters for U- Net and continuous U-Net (cU-Net) models and variants. Notation follows Table 5.
DDPM MODEL CONFIGURATION
GFLOPS
MB
U-Net
7.21
545.5
Continuous UNet (cU-Net)
2.90
137.9
cU-Net wo/A (no attention)
2.81
128.7
cU-Net wo/R R (no resblocks)
1.71
92.0
cU-Net wo/ARR (no attention & no resblocks)
1.62
88.4
Table 5: Number of GigaFL.OPS (GFLOPS) and Megabytes in Memory (MB) for Different Models.
6 Conclusion
We explored the scalability continuous U-Net architectures, introduction attention mechanisms, residual connections, and time embeddings tailored for diffusion timesteps. Through our ablation studies, we empirically demonstrated the benefits of the incorporation of these new components, in terms of denoising performance and image generation capabilities (Appendix C). We propose and prove the viability of a new framework for denoising diffusion probabilistic models in which we fundamentally replace the undisputed U-Net denoiser in the reverse process with our custom continuous U-Net alternative. In contrast with the exiting work, this new denoising network features a unique second-order Neural ODE block with residual connections and time embeddings, enhancing efficiency our advanced diffusion models that utilize a deep implicit U-Net structure. As shown above, this modification is not only theoretically motivated, but is substantiated by empirical comparison. We compared the two frameworks image synthesis, to analyse their expressivity and capacity to learn complex distributions, and denoising in order to get insights into what happens during the reverse process at inference and training. Our innovations offer notable efficiency advantages over traditional diffusion models, reducing computational demands and hinting al possible deployment on resource-limited devices due to their parameter efficiency while providing comparable synthesis performance and improved perceived denoising performance that is better aligned with human perception. Considerations for future work go around improving the ODE solver parallclisation, and incorporating sampling techniques to further boost efficiency.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Acknowledgements
SCO gratefully acknowledges the financial support of the Oxford-Man Institute of Quantitative Finance. A significant portion of SCO's work was conducted at the University of Cambridge, where he also wishes to thank the University's HPC services for providing essential computational resources. CWC gratefully acknowledges funding from CCMT, University of Cambridge. GY was supported part by the ERC IMI (101005122), the H2020 (952172), the MRC (MC/PC/21013), the Royal Society (IEC/NSFC/211233) the NVIDIA Academic Hardware Grant Program, the SABER project supported by Boehringer Ingelheim Ltd, Wellcome Leap Dynamic Resilience, and the UKRI Future Leaders Fellowship (MR/VV23799/ 1). JH was supported in part by the Imperial College Biocnginecring Department PhD Scholarship and the UKRI Future Leaders Fellowship (MR/V023799/1) CBS acknowledges support from the Philip Leverhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC advanced career fellowship EP/ EP/ AEP/V22428/LL EPSRC grants EP CPPSS22045/1 and EP/T003553/L, /15733/Z/19/2 Z and 221633/2/20/Z, CCMI and the Alan Turing Institute. AAR gratefully acknowledges funding from the Cambridge Centre for Data-Driver Discovery and Accelerate Programme for Scientific Discovery, made possible by donation from Schmidt Futures, ESPRC Digital Core Capability Award, and CMIH and CCIMI, University of Cambridge.
/ EP 6P/T011961/1, the Wellcome Innovator Awards
References
Abien Fred Agarap. Deep learning using rectified linear units (relu), 2019. 5
Shinci Arakawa, Hidcki Tsunashima, Daichi Horita, Keitaro Tanaka, and Shigeo Morishima. Memory efficient. diffusion probabilistic models via patch-based generation, 2023. 4
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie-dmm: an analytic estimate of the optimal reverse
variance in diffusion probabilistic models. arXiv preprinl Xrx:222001 065003, 2022. 2
Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, June 2018. doi: 10.1109 /vvrr 2018.00652. URL
http://dx: doi. org/10. 10. 11199/199//... 201.. 11
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 3
Chun- Wun Cheng, Christina Runkel, Lihao Liu, Raymond H Chan, Carola- Schönlieb, and Angelica Continuous u-net: Faster, greater and noiscless. arXiu preprint (xXw:2302.00626 2023. 2,
3,
Wei Cheng, Gregory Darnell, Sohini Ramachandran, and Lorin Crawford. Generalizing variational autoen- coders with hicrarchical empirical bayes, 2020.
Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusson models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683 25696, 2022.
Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Mare L. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems, 2023. 2
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):208022))) 2007.
doi: 10.1109/T11P2007...00 9001099...9....... 12
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. 1
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with langevin diffusion, 2022. 4
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes, 2019.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer, 2023. 4
Tan Goodfellow, Jean Douget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):
139 144, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural informalion processing systems, 33:6840 6851, 2020. 1, 3, 5, 6,
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation, 2021.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.
Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference Pottern Recognition, pp. 2366-2369) 2010. doi: 10.1109/CPP.2210.557 10
Tero Karras, Miika Ailtala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565 -26777, 2022. 4, 7
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXim:1312.6114, 2013.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021. 2
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism, 2022a.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds, 2022b.
Aaron Lou and Stefano Ermon. Reflected diffusion models, 2023. 2
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022. 2
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusson models via early stop of the diffusion process. arXiv preprint NKK:222551152,, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic modols. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021. 1,
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pictro Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems, 33:5911-5921,
2020. 7
Kushagra Pandey and Stephan Mandt. complete recipe for diffusion generative models, 2023.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents, 2022. 4
Konpat Precchakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pattern Recognition, pp.
2022a.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation, 2022b. 4
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2015. 1,
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models, 2014.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022a. 4
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusion models, 2022b. 1, 12
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G. Dimakis, and Sanjay Shakkottai. Solving lincar inverse problems provably via posterior sampling with latent diffusion models, 2023.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent odes for irregulatly-sampled time series, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- text-to-image diffusion models with deep language undorstanding, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 2022. 2
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015. 1
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. orXio preprint arXi::2010002502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020.
1, 2
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint vrXiv:2011.13/50 2020b. 1, 2, 3,
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space, 2021. 4
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning a diffusion model from single natural image, 2022.
Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-officient training of diffusion models, 2023. 4
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, 2022. 2
16

---

Published in Transactions on Machine Learning Research (04/2024)
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep can for image denoising. IEEE Transactions on Image Processing, 26(7):3142-311 July 2017. ISSN 1941-0042. doi: 10.1009/th.200172262206 URL http://dx.cor doi. wttp.//o............/oo//// 110.011199/TPP.011111 2662206. 12
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator, 2023. 2
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator lcarning, 2023a.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders, 2023b. 2
Huangjic Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for elficient, reconfigurable, and variable-resolution diffusion modeling, 2023c. 4
17

---

Published in Transactions on Machine Learning Research (04/2024)
A Appendix
This appendix serves as the space where we present more detailed visual results of the denoising process for both the baseline model and our proposal.
UMPS
Trenks
Tireate::
Figure 7: Tracking intermediate model denoising predictions. The images on the left are the outputs of our continuous U-Net, and the ones on the right are from the conventional U-Net.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Treoics
Traelee: T3
Tremles
Terester-1
GMN
UNd
Figure 8: Tracking intermediate model denoising predictions: The images the left depict the outputs of our continuous U-Net, which successfully removes noise in fewer steps compared to its counterpart (middle images) and maintains more fine-grained detail. The images on the right represent outputs from the conventional U-Net.
19

---

Published in Transactions on Machine Learning Research (04/2024)
LINA
UKL
LAVS
cil LPTR:
Use LAYS LCE
Figure Tracking intermediate model denoising predictions: The images on the left depict the outputs of our continuous U-Net, which attempts to predict the facial features amidst the noise. In contrast, the images on the right, from the conventional U-Net, struggle recognise the face, showcasing its limitations in detailed feature reconstruction.
20

---

Published in Transactions on Machine Learning Research (04/2024)
B
Appendix
SSIM comparison for 100 timesteps
SSIM comparison timesteps
0,7
SSIM
0.5
0.
Timestep
Timestap
SSIM comparison for 200 timesteps
SSIM comparison for 250 timesteps
Modess
Mhatels
<UNet
cUiNct
UNet
UNet
SSIM
200
280
160
340
120
320
20
250
223
200
275
150
325
100
25
Timestep
Timestep
SSIM comparison for 400 timesteps
SSIM comparison for 600 timesteps
Stodela
Modes <Uhett
<Uieet
Ulvee
UWet
04
S51M4
0.1
320
200
Timestep
Timestep
Figure 10: SSLM scores plotted against diffusion steps for varying noise levels for one image. The graph underscores the consistently superior performance of U-Net over cU-Net in terms of SSIM, particularly high noise levels. This dominance in SSIM may be misleading due to the inherent tradeoff and the tendency of our model to predict features instead of distorting the images.
21

---

Published in Transactions on Machine Learning Research (04/2024)
LPIPS comparison timesteps
comparison 150 timesteps
07
LPIPS
0.4
0.0
Timestap
Timestop
LPIPS comparison for 200 timesteps
LPIPS comparison for 250 timesteps
Modes
Mode
<
UNet
JNlt
0.6
LPISS
0.3
0.2
382
343
130
255
375
325
100
Timestop
Timestop
LPIPS comparison for 400 timesteps
LPIPS comparison for 600 timesteps
Modes
Modeb
dURee
cUNet
2.0
1.3
Ueet
0.6
0.1
LPIPS
LPIPS
04
320
200
302
Timestep
Timestep
Figure 11: LPIPS score versus diffusion - or denoising timesteps for one image. We can observe how our continuous U-Net consistently achieves better LPIPS score and does not suffer from such significant elbow effect observed in the U-Net model in which the quality of the predictions starts deteriorating after the model achieves pcak performance. The discontinuous lincs indicate the timestep at which the peak LPIPS was achieved. Here, see that, especially for large number of timesteps, continuous U-Net seems to converge to better solution in fewer steps. This is because of the ability it has to predict facial features rather than simply settling for over-smoothed results (Figure 9).
22

---

Published in Transactions on Machine Learning Research (04/2024)
C
Appendix
In this short appendix, we showcase images generated for our ablation studies. As demonstrated below, the quality of the generated images is considerably diminished when we train our models without specific components (without attention and/or without residual connections). This leads to the conclusion that our enhancements to the foundational blocks in our denoising network are fundamental for optimal performance.
Figure 12: Representative samples from the version of our model trained without attention mechanism. The decrease in quality can often be appreciated in the images generated.
Figure 13: Representative samples from the version of our model trained without residual connections within our ODE block. We can see artefacts and inconsistencies frequently.
Figure 14: Representative samples from the basic version of our model which only includes time embeddings. As one can appreciate, sample quality suffers considerably.
23


Reviews:
[{"review_id": "ip83Tg6jp9", "reviewer": "Reviewer_DR5X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strength**\n- The proposed method is overall simple but effective. It does not require additional tuning of hyper-parameters and make the model with lighter weight. Moreover, the computation cost is reduced and the denoising is more efficient as we can see the model can output less blurry predictions of $x_0$ at high noisy-level.\n\n- The experimental results well validate the approach. Notably, the paper also provides comprehensive study to across different sampling timesteps, with different metrics, etc. \n\n### **Weakness**\nI have several concerns regarding the novelty, clarity of the method, design of the experiments and some defaults in the presentation. Please see the points below:\n\n- It seems this work is heavily built upon Cheng et al., 2023. It is not clear how this work differs from Cheng et al., 2023 and tackles the problem in diffusion models. \n\n- The SDEs can be generalized as a sum of PF ODE and a Langevin diffusion SDE is not novel, as it is already shown in the EDM paper (Karras et al., 2022). The authors may need to clarify the difference between the analysis in section 3.3 those in EDM. \n\n- It is unclear how the Unet architecture is modified from Figure 2. How does the ODE block shown in the right panel play a role in the entire network and how does the left panel illustrate the design? \n\n- The notation of x\" in Eq (4) is not consistent the one in the text below Eq (4). $X_0$,  $g(\\cdot, \\cdot)$ and $\\theta_g$ are not explained. \n\n- Can the authors specify the resolution of the images used for the training? As a study of the network architecture, it is also important the proposed design can be efficient across different dimensionality (image resolution).\n\n- In the sampling stage, the authors seem to only compare when adopting DDPM sampler. It would be interesting if the authors could show some comparison using different samplers to demonstrate the efficiency of the proposed c-U-Net.\n\n- One possible reason of the improvement in computation efficiency can be the reduction of parameter size. (Please note this is not necessarily a weakness.) If the authors would like to highlight the computation efficiency is improved, it would be better to compare GFLOPs with Unet has similar parameter size. \n\n- Some recent works have also been working on tackling the efficiency of the diffusion model architecture. How does the proposed model work compared with them? (Wang et al, 2022; Wang et al., 2023; Zheng et al., 2023a; Gao et al., 2023; Ding et al., 2023; Arakawa et al., 2023; Zheng et al., 2023b)\n\n- Some claims in the paper are misleading. For example, the authors point out \"However, models trained directly on continuous  timesteps often underperform compared to discretely-trained models (Song et al., 2020b)\" in the introduction. I did not find related evidence in the ScoreSDE paper (Song et al., 2020b). On one hand, recent empirical results have shown models on continuous time reach state-of-the-art performance in Karras et al, 2022. Moreover, in Song et al., 2020b, the connection between score sde and ddpm has been derivated. It would be better if the authors could provide additional evidence to support the claim here or clarify in order to avoid potential misunderstanding. \n\n- The review of existing literatures are not comprehensive. For example, \n  - Several works have proposed to truncate the diffusion chain and start the generation from an implicit distribution instead of standard Gaussian noise  (Xiao et al., 2021; Pandey et al., 2022; Zheng et al., 2023c; Lyu et al., 2022). \n  - Besides Bao et al., 2022, a lot of works have been working on employing advanced numerical solvers for SDE/ODE (Liu et al., 2022; Lu et al., 2022; Zhang & Chen, 2023; Karras et al., 2022). \n  - Using manifold constraints or inverse problem hypothesis for diffusion models also involves several representative papers (Lou et al., 2020; Kawar et al., 2022; Chunget al., 2022; Giannis, et al., 2023; Rout et al., 2023; Lou and Ermon, 2023)\n\n### References\n\n- Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., & Li, H. (2022). Sindiffusion: Learning a diffusion model from a single natural image. arXiv preprint arXiv:2211.12445.\n\n- Wang, Z., Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., ... & Zhou, M. (2023). Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304.12526.\n\n- Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K., & Anandkumar, A. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning (pp. 42390-42402). PMLR, 2023a.\n\n- Gao, S., Zhou, P., Cheng, M. M., & Yan, S. (2023). Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389.\n\n- Ding, X., Wang, Y., Xu, Z., Welch, W. J., & Wang, Z. J. (2022). Continuous conditional generative adversarial networks: Novel empirical losses and label input mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n- Arakawa, S., Tsunashima, H., Horita, D., Tanaka, K., & Morishima, S. (2023). Memory Efficient Diffusion Probabilistic Models via Patch-based Generation. arXiv preprint arXiv:2304.07087.\n\n- Zheng, H., Wang, Z., Yuan, J., Ning, G., He, P., You, Q., Yang, H., and Zhou, M. Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffusion modeling, 2023b\n\n- Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b\n\n- Xiao, Z., Kreis, K., & Vahdat, A. (2021). Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804.\n\n- Pandey, K., Mukherjee, A., Rai, P., and Kumar, A. DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308, 2022.\n\n- Zheng, H., He, P., Chen, W., and Zhou, M. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023c\n\n- Liu, L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.\n\n- Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.\n\n- Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023\n\n- Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidatingthe design space of diffusion-based generative models. In Proc. NeurIPS, 2022\n\n- Lou, A., Lim, D., Katsman, I., Huang, L., Jiang, Q., Lim, S. N., & De Sa, C. M. (2020). Neural manifold ordinary differential equations. Advances in Neural Information Processing Systems, 33, 17548-17558.\n\n- Kawar, B., Elad, M., Ermon, S., & Song, J. (2022). Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35, 23593-23606.\n\n- Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., & Ye, J. C. (2022). Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687.\n\n- Daras, G., Shah, K., Dagan, Y., Gollakota, A., Dimakis, A. G., & Klivans, A. (2023). Ambient Diffusion: Learning Clean Distributions from Corrupted Data. arXiv preprint arXiv:2305.19256.\n\n- Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A. G., & Shakkottai, S. (2023). Solving linear inverse problems provably via posterior sampling with latent diffusion models. arXiv preprint arXiv:2307.00619.\n\n- Lou, A., & Ermon, S. (2023). Reflected diffusion models. arXiv preprint arXiv:2304.04740.\n\n----------\n\n*Given the aforementioned concerns, my current inclination is to rate this submission below the acceptance threshold. However, I am open to reconsidering my evaluation if the authors address the highlighted issues or provide clarifications in their rebuttal.*", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a more efficient architecture for diffusion-based generative model, especially for denoising diffusion models. This approach featuring a continuous U-Net denoising network, namly c-U-Net, significantly reduces the number of parameters and computational costs, achieving faster convergence and improved noise robustness. The authors demonstrate its effectiveness through experiments and corresponding analysis, showing that their model outperforms standard U-Nets in both efficiency and quality of generated images on various datasets under the DDPM setting.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The differences and connections compared with Cheng et al., 2023 and Karras et al., 2022 need to be clarified.\n\n2. The presentation regarding the introduction and method sections needs to be improved. \n\n3. The experiments, including experimental setting, comparison with recent models and comparison across different resolutions are suggested. \n\nPlease see my detailed comments in weakness part.", "broader_impact_concerns": "No significant broader impact concerns were found in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gPX4lijyAm", "reviewer": "Reviewer_WHDr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Given, the widespread adoption of diffusion models for generative modeling, I think the main strength of the paper is the reduction in wall clock time (Table 1), and computational and memory footprint (Table 5) for generating samples as compared to the standard UNet architecture.\n2. Since the authors propose updates in the denoiser, I expect the proposed approach to be complementary to existing techniques for fast diffusion sampling and it would be interesting to benchmark the continuous denoiser with fast sampling methods like DDIM, DPM-Solver, etc.\n\nWeaknesses:\n\n1. The paper has limited methodological novelty in the sense that the authors largely reuse methodological ideas presented in another paper [1]. However, based on TMLR’s acceptance criterion, this is not a problem per se.\n\n2. See Requested Changes for more details", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: The authors introduce a continuous denoiser architecture for diffusion generative modeling by leveraging ideas from Neural ODEs. More specifically, the authors adapt the existing U-Net denoiser commonly used in diffusion modeling to a continuous U-Net based on second-order ODE dynamics. Figure 2 provides a nice intuition regarding the same. Empirical results demonstrate the effectiveness of the continuous denoiser.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n\n1. In section 3.3, the authors mention: “Our architecture outperformed DDPMs in terms of efficiency and accuracy”. For readability, It would be nice to provide a reference to a table or figure which justifies this argument. Moreover, the theoretical argument presented regarding faster but less accurate solutions using the probability flow ODE seems a bit redundant considering the analysis presented in [2]. Lastly, I had some trouble understanding the relevance of the derivation presented in Proposition 3.1 in the context of the proposed methodology. It would be nice if the authors could intuitively elaborate upon the same in their response and subsequently in the revision.\n\n2. Missing citations and discussion of existing work: Since there is a large body of work on speeding up diffusion models, I found the discussion in the introduction a bit dated as the discussion of several important directions in fast sampling in diffusion models is missing. For instance, \n   1. Discussion of training-free samplers like DPM-Solver [3], DEIS [4], EDM [2] is missing.\n   2. Discussion of methods which accelerate diffusion models by incorporating conditional information like [5, 6] or perform diffusion in the latent space [7, 8].\n   3. Discussion of more efficient ways for diffusion: EDM [2], CLD [9], PSLD[10].\n   4. The authors discuss ES-DDPM. In addition I think its worth mentioning works like Truncated DDPM [11] and Denoising Diffusion GANs [12] which are based on similar ideas of truncating the forward process of DDPMs for efficient sampling.\n\nIt is worth noting that these works are an example and the authors should look into the literature for more relevant works. Moreover, I think the paper currently lacks a dedicated related works section and discussion of these works in more detail in this section would be ideal.\n\n3. The claim in the introduction: “However, models trained on continuous timesteps often underperform compared to discretely-trained models” seems incorrect. See Section 4.4 and Table 3 in Song et al where models trained with continuous time objective outperform models trained with a discrete time objective. I see that the authors cite the Song et al. paper so perhaps they meant continuous time models outperform discrete ones?\nFurthermore, the authors mention “training must be repeated for each desired step count”. This needs further elaboration since for discrete or continuous time diffusion, as long as the number of noise levels sampled during training are fairly large (around N=1000 to 4000), we can get a decent score model. This allows sampling using a lot less noise levels during inference than used during training. Therefore, this point needs to be clarified in more detail in the paper.\n\nMinor Comments:\nThe line “In general, these methods provide inference-time…” is repeated twice in the main text.\n\nReferences\n[1] Continuous U-Net: Faster, Greater and Noiseless [Cheng et al.]\n\n[2] Elucidating the Design Space of Diffusion-Based Generative Models [Karras et al.]\n\n[3] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps [Lu et al.]\n\n[4] Fast Sampling of Diffusion Models with Exponential Integrator [Zhang et al.]\n\n[5] Diffusion Autoencoders: Toward a Meaningful and Decodable Representation\n[Preechakul et al.]\n\n[6] DiffuseVAE: Efficient, Controllable, and High-Fidelity Generation from Low-Dimensional Latents [Pandey et al.]\n\n[7] Score-based Generative Modeling in Latent Space [Vahdat et al.]\n\n[8] High-Resolution Image Synthesis with Latent Diffusion Models [Rombach et al.]\n\n[9] Score-Based Generative Modeling with Critically-Damped Langevin Diffusion [Dockhorn et al.]\n\n[10] A Complete Recipe for Diffusion Generative Models [Pandey et al.]\n\n[11] Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders [Zheng at al.]\n\n[12] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs [Xiao et al.]", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JtQxMU3XUx", "reviewer": "Reviewer_Duy9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n* There are some interesting questions about architecture raised in this manuscript, namely whether or not we can cut down the size of these existing diffusion models.\n\nWeaknesses\n---------------\n* Overall, I found it very hard to parse what the exact contribution of the paper is. I believe that it's mostly the architecture, but the section on the probability flow ODE seems to deviate from this rather heavily.\n* Why the continuous UNet? It seems like the continuous UNet just replaces the resnet structure of the internal blocks with a Neural ODE, but doing this should be very slow since one has to solve the forward + reverse differential equation for each block for training. Furthermore, Neural ODEs are constrained when compared with regular resnet blocks since they are diffeomorphic, and our UNet is supposed to model the score function, which is not diffeomorphic like a pure flow.\n* Related to the above, it would help if the authors could rewrite the paper to talk exactly about what's modelled exactly. Equations (3) and (4) seem to be doing this, but it's not clear/a bit self-contradictory. In particular, (3) says that we're modeling the noise prediction, but (4) and the paragraph afterwards say that we are modeling the step between $x_{t + 1} and x_t$.\n* The connection between the second-order solver and the probability flow ODE is not clear. The probability flow ODE (following Song \"Score-Based Generative Modeling...\") is a continuous dynamical system that arises after one learns the score (gradients) of the probability distribution through score matching (which is simulation-free). The paper here seems to be proposing a way to parameterize this ODE directly and to train it with a neural ODE training objective, but that has nothing to do with the probability flow ODE and is more generally a neural ODE objective.\n* The experimental results are extremely out of date, as the baselines seem to be from the original DDPM paper. In particular, the diffusion literature has seen a lot of updating (including different architectures, a continuous training objective, fixing up core problems in the sampling process, etc...) so the paper should compare with these other components in place.\n* There are a lot of irrelevant parts in the experimental section. For example, Figure 4 just shows a noising process, which is very well known and not relevant. Figure 5 just shows that the proposed image denoiser recovers fine details like glasses compared with the regular diffusion for an example, but this is somewhat arbitrary. In fact, the entire denoising section seems somewhat weird (especially with a lack of diffusion baselines like https://arxiv.org/abs/2201.11793). Generally, I would want the experimental section to be focused on the main claims of the paper (better image generation at a lower parameter cost), and these other tertiary experiments seem rather unnecessary given this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use continuous unets and the probability flow ODE to achieve faster and more parameter efficient sampling from diffusion models when compared with the DDPM baseline.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the novelty of the paper is somewhat unclear. Following the recommendation of one of the reviewers, please follow the following suggestion:
"The clarification provided does highlight some differences, but I recommend that the authors further emphasize the innovative aspects of their work or its applicability to broader or different contexts than those covered by existing research. This could help in underlining the unique contributions of their study more clearly."

Please update the paper accordingly.
Decision: Accept (poster)