Neural Residual Diffusion Models for Deep Scalable Vision Generation

Authors: zhiyuan  ma
liangliang  zhao
biqing  qi
bowen  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:24:04

Abstract: The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities.
Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework (Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the  fidelity and consistency of generated content and supporting large-scale scalable training.

Main paper: 
 introduction: 
 Diffusion models (DMs) [1, 2, 3, 4] have emerged as a class of powerful generative models and have recently exhibited high quality samples in a wide variety of vision generation tasks such as image synthesis [5, 6, 7, 8, 9, 10, 11] , video generation [12, 13, 14, 15, 16, 17, 18, 19] and 3D rendering and generation [20, 21, 22, 23, 24] . Relying on the advantage of iterative denoising and high-fidelity generation, DMs have gained enormous attention from the community and have been significantly improved in terms of sampling procedure [25, 26, 27, 28] , conditional guidance [29, 30, 31, 32] , likelihood maximization [33, 34, 35, 36] and generalization ability [37, 38, 39, 10] in previous efforts.However, current diffusion models still face a scalability dilemma, which will play an important role in determining whether could support scalable deep generative training on large-scale vision data and give rise to emergent abilities [40] similar to large language models (LLMs) [41, 42] . Representatively, the recent emergence of Sora [43] has pushed the intelligent emergence capabilities of generative models to a climax by treating video models as world simulators. While unfortunately, Sora is still a closed-source system and the mechanism for the intelligence emergence is still not very clear, but the scalable architecture must be one of the most critical technologies, according to the latest investigation [44] on its reverse engineering.To alleviate this dilemma and spark further research in the open source community beyond the realms of well established U-Net and Transformers, and enable DMs to be trained in new scalable deep generative architectures, we propose a unified and massively scalable Residual-style Diffusion Models framework (Neural-RDM for short) with a learnable gating residual mechanism, as shown in Figure 1 . The proposed Neural-RDM framework aims to unify the current mainstream residual-style generative architecture (e.g., U-Net or Transformer) and guide the emergence of brand new scalable network architectures with emergent capabilities. To achieve this goal, we first introduce a continuous-time neural ordinary differential equation (ODE) to prove that the generative denoising ability of the diffusion models is closely related to the residual-style network structure, which almost reveals the essential reason why any network rich in residual structure can denoise well: Residual-style neural units implicitly build an ordinary differential equation that can well fit the reverse denoising process through ever-deepening neural units, thus supporting excellent generative abilities. Further, we also show that the gating-residual mechanism plays an important role in adaptively correcting the errors of network propagation and approximating the mean and variance of data, which avoids the adverse factors of network deepening. On this basis, we further present the theoretical advantages of the Neural-RDM in terms of stability and score prediction sensitivity when stacking this residual units to a very long depth by introducing another residual-sensitivity ODE. From a dynamic perspective, it reveals that deep stacked networks have the challenge of gradually losing sensitivity as the network progressively deepens, and our proposed gating weights have advantages in reverse suppression and error control.Our proposed framework has several theoretical and practical contributions:Unified residual denoising framework: We unify the residual-style diffusion networks (e.g., U-Net and Transformer) by introducing a simple gating-residual mechanism and reveal the significance of the residual unit for effective denoising and generation from a brand new dynamics perspective.Theoretically deep scalability: Thanks to the introduction of continuous-time ODE, we demonstrate that the dynamics equation expressed by deep residual networks possesses excellent dynamic consistency to the denoising probability flow ODE (PF-ODE) [45] . Based on this property, we achieve the simplest improvement to each mrs-unit by parameterizing a learnable mean-variance scheduler, which avoids to manually design and theoretically support massively deep scalable training.Adaptive stability maintenance and error sensitivity control: When the mrs-units are infinitely stacked to express the dynamics of an overall network F θ , the main technical difficulty is how to reduce the numerical errors caused by network propagation and ensure the stability of denoising. By introducing a sensitivity-related ODE in Sec. 2.3, we further demonstrate the theoretical advantages of the proposed gated residual networks in enabling stable denoising and effective sensitivity control.Qualitative and quantitative experimental results also consistently show their effectiveness. 
neural residual diffusion models: 
 We propose Neural-RDM, a simple yet meaningful change to the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training of extremely deep networks. This framework is supported by three critical theories: 1) Gating-Residual ODE (Sec. 2.1), which defines the dynamics of the minimum residual stacking unit (mrs-unit for short) that serves as the foundational denoising module, as shown in Figure 1 (a). Based on this gating-residual mechanism, we then introduce and variance preserving scheduler DDPM [2] ), which may cause concerns about instability in denoising efficiency and quality, we introduce a parametric method to implicitly learn the mean and variance distribution, which lowers the threshold of manual design and enhances the generalization ability of models. Last but not least, to maintain the stability of the deep stacked networks and verify the sensitivity of each residual unit F θi (•) to the network F θ , we stand on the shoulders of the adjoint sensitivity method [47, 48] to propose 3) Residual-Sensitivity ODE (Sec. 2.3), which means the sensitivity-related dynamics of each latent state z i from F θi (•) to the deep network F θ . Through rigorous derivation, we prove that the parameterized gating weights have a positive inhibitory effect on sensitivity decaying as network deepening. We will elaborate on them below.Let F θi represents the minimum residual unit block i (Figure 1 (a)), f (•) denotes any feature mapper wrapped by F θi . Instead of propagating the signal z through each of vanilla neural transformation ẑ = f θ (z) directly, we introduce a gating-based residual connection for the signal z, which relys on the two learnable gating weights α and β to modulate the non-trivial transformation F θi (z i ) as,EQUATIONFor a deep neural network F θ (•) with depth L, consider two common residual stacking fashions: Flow-shaped Stacking (FS) [49, 50] and U-shaped Stacking (US) [51, 52] . For the flow-based deep stacking networks as shown in Figure 2 (a), each residual unit f (•) accepts the output z i of the previous mrs-unit as input, and obtains a new hidden state z i+1 through gating-residual connection,EQUATIONNote that Eq. 2 is a refined form of Eq. 1 in the case of flow-shaped stacking. In contrast, for the U-shaped deep stacking networks as in Figure 2 (b), each minimum residual unit contains two symmetrical branches, where the left branch receives the output z i of the previous mrs-unit's left branch as input (called the read-in branch), and the right branch performs the critical nonlinear residual transformation for readout (called the read-out branch), which can be formally described as:EQUATIONHere Eq. 3 is a refined form of Eq. 1 in the case of U-shaped stacking, αi and βi collectively denotes the gating weights from the left and right branches, F θi is the i-th minimum residual unit of the U-shaped networks, and " →" denotes the skip-connection for "z i+1 -→ z 2L-2-i ", which is computed recursively via F θ i+1:L-1 . To enable the networks to be infinitely stacked, we introduce a continuous-time Gating-Residual ordinary differential equation (ODE) to express the neural dynamics ), the samples produced by Neural-RDM (trained on JourneyDB [53] ) exhibit exceptional quality, particularly in terms of fidelity and consistency in the details of the subjects in adhering to the provided textual prompts. of these two types of deep stacking networks (δ = 1 L , L -→ ∞ denotes the number of the mrs-units),EQUATIONwhere ϕ represents the gating weights, which can be independently trained or fine-tuned without considering the parameters θ of the feature mapping network F θ (•) itself.The above-mentioned gating-residual mechanism is utilized to modulate mainstream deep stacking networks and unify them into a residual-style massively scalable generative framework, as shown in Figure 2 (c). Next, we further explore the essential relationship between residual neural networks and score-based generative denoising models from a dynamic perspective.First, inspired by the theory of continuous-time diffusion models [45, 54] , the forward add-noising process can be expressed as a dynamic process with stochastic differential equation (SDE) as,EQUATIONwhich describes a data perturbation process controlled by a mean-variance scheduler composed of µ(z t , t) and σ(t) respectively, w t denotes the standard Brownian motion. Compared with the forward process, the core of the diffusion model is to utilize a deep neural network (as deep and large as possible) for score-based reverse prediction [46, 55] . A remarkable property of this SDE is the existence of a reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45] ), which retain the same marginal probability densities as the SDE (See Appendix. A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,EQUATIONwhere ∇ z log p t (z t ) denotes the gradient of the log-likelihood of p t (z t ), which can be estimated by a score matching network F θ (z t , t). Here we re-parameterize the PF-ODE by utilizing gated weights to replace the manually designed mean-variance scheduler, in which αt,ϕ and βt,ϕ denotes the time-dependent dynamics parameters, which is respectively parameterized to represent -1 2 σ(t) 2 and µ(z t , t) by our proposed gating-residual mechanism. Note that F θ (•) is a score estimation network composed of infinite mrs-units block i (i.e., F θi ), which enables massively scalable generative training on large-scale vision data, but also presents the challenge of numerical propagation errors.To control the numerical errors in back-propagation and achieve steadily and massively scalable training, we stand on the shoulders of the adjoint sensitivity method [47, 48] Residual-Sensitivity ODE, which is utilized to evaluate the sensitivity of each residual-state z t of the mrs-unit F θi to the total loss L derived by score estimation network F θ (•) (the sensitivity is denoted as s t = dL dzt , δ denotes an infinitesimal time interval) and can be formally described by the chain rule,EQUATIONOn the basis of Eq. 7, we next continue to discuss the dynamic equation of sensitivity changing with time t. First, considering the trivial transformation f θ (•) without gating-residual mechanism,EQUATIONWe can rewrite Eq. 7 based on Eq. 8 as:EQUATIONThe Residual-Sensitivity ODE under vanilla situation then can be derived,EQUATION)According to the derived residual-sensitivity equation in Eq. 10, we further use the Euler solver to obtain the sensitivity s t0 of the starting state z t0 to network F θ (•) as,EQUATIONDue to the non-negativity of the integral and the gradient ∂f θ (zt,t)∂zt not equals to 0, we can obtain a gradually decaying sensitivity sequence:s t L > s t L-1 > • • • > s t0 .Similarly, when defining parameter-sensitivity s θ = dL dθ , the same decaying results for s θ0 can be obtained: ), the sample videos from SkyTimelapse [62] , Taichi-HD [63] and UCF101 [64] all exhibit better frame quality, temporal consistency and coherence.EQUATIONTo alleviate this problem, and enable stable training in massively deep scalable architecture, we introduce the following non-trivial solution with gating-residual transformation,EQUATIONSubstitute Eq. 13 into Eq. 7 to obtain the corrected sensitivity ŝt = dL dẑt as:EQUATIONThe non-trivial Residual-Sensitivity ODE can be derived as,EQUATION)Through the Euler solver, we can also obtain the sensitivity ŝt0 of the starting state adjusted by the gating-residual weights,EQUATIONWhere α t,ϕ and β t,ϕ adaptively modulate and update the sensitivity of each mrs-unit to the final loss, which supports being trained through minimizingL s = ||F θ (z t , t) -∇ z log p t (z t )|| 2 2 + γ • L ||α t,ϕ • ∂f θ (ẑt,t) ∂ ẑt -β t,ϕ || 22 in full-parameter training or model fine-tuning fashions. 
experiments: 
 We present the main experimental settings in Sec. Table 2 : The main results for video generation on the SkyTimelapse [62] , Taichi-HD [63] and UCF-101 [64] with 256 × 256 resolution of each frame. We highlight the best value in blue , and the second-best value in green .MSCOCO [65] and JourneyDB [53] datasets. All images are resized to 256 × 256 resolution for training. For video generation tasks, we follow the previous works [12, 60] to train None-to-Video (i.e., unconditional video generation) models on the SkyTimelapse [62] and Taichi [63] datasets, and train Class-to-Video models on the UCF-101 [64] dataset. Moreover, we follow previous works [12, 60] to sample 16-frame video clips from these video datasets and then resize all frames to 256 × 256 resolution for training and evaluation.Implementation details. We implement our Neural-RDMs into Neural-RDM-U (U-shaped) and Neural-RDM-F (Flow-shaped) two versions on top of the current state-of-the-art diffusion models LDM [30] and Latte [60] for image generation, and further employ the Neural-RDM-F version for video generation. Specifically, we first load the corresponding pre-trained models and initialize gating parameters {α = 1, β = 0} of each layer, then perform full-parameter fine-tuning to implicitly learn the distribution of the data for acting as a parameterized mean-variance scheduler. During the training process, we adopt an explicit supervision strategy to enhance the sensitivity correction capabilities of α and β for deep scalable training, where the explicitly supervised hyper-parameter γ is set to 0.35. Eventually, we utilize the AdamW optimizer with a constant learning rate of 5 × 10 4 for all models and exploit an exponential moving average (EMA) strategy to obtain and report all results.Evaluation metrics. Following the previous baselines [30, 58, 59 , 60], we adopt Fréchet Inception Distance (FID) [66] , sFID [67] and Inception Score (IS) [68] to evaluate the image generation quality and the video frame quality (except for sFID). Furthermore, we utilize a Fréchet Video Distance (FVD) [69] metric similar with FID to evaluate the unconditional and conditional video generation quality. Among these metrics, FVD is closer to human subjective judgment and thus better reflects the visual quality of the generated video content. Adhering to the evaluation guidelines proposed by StyleGAN-V [70] , we calculate the FVD scores by analyzing 2048 generated video clips with each clip consists of 16 frames.Baselines. We compare the proposed method with the recent state-of-the-art baselines, and categorize them into three groups: 1) GAN-based. BigGAN-deep [56] and StyleGAN-XL [57] for image task, MoCoGAN [71] , MoCoGAN-HD [72] , DIGAN [73] , StyleGAN-V [70] and MoStGAN-V [74] for video task. 2) U-shaped. ADM [58] and LDM [30] for image task, PVDM [75] and LVDM [12] for video task. 3) F-shaped. DiT-XL/2 [59] and Latte-XL [60] for image task, VideoGPT [76] and Latte-XL [60] (with temporal attention learning) for video task.For a more objective comparison, we maintain approximately the same model size to perform classconditional and text-conditional image generation experiments, which are shown in Table ), we can observe that the samples produced by Neural-RDM exhibit exceptional quality, particularly in terms of fidelity and consistency in the details of the subjects in adhering to the provided textual prompts, which consistently demonstrates the effectiveness of our proposed approach in deep scalable spatial learning. To further explore the effectiveness and specific contributions of proposed gating-residual mechanism in temporal learning, we continue to perform the video generation evaluations, which are shown in Table 2 . From Table 2 , we find that our model (flow-shaped version) basically achieves the best results (except for the second-best results in class-to-video evaluation). Specifically, compare with Latte-XL [60] , Neural-RDM respectively achieves an improvement of 33.3% and 42.8% in FVD scores on Sky-Timelapse and Taichi-HD datasets, which hints the powerful potential of flow-based deep residual networks in promoting generative emergent capabilities of video models. Furthermore, we exhibit a number of visual comparison results of the 16-frames video produced by Neural-RDM and baseline (Latte-XL [60] ), as shown in Figure 4 . We can observe that some generated frames from the baseline partially exhibits poor quality and temporal inconsistency.Compare with the baseline, Neural-RDM maintains temporal coherence and consistency, resulting in smoother and more dynamic video frames, which further reflects the effectiveness of proposed method in both quantitative and qualitative evaluations.To better illustrate the advantage of the gated residuals and understand the positive suppression effect for sensitivity attenuation as network deepening, we visualize the normalized sensitivity at different depths of our Neural-RDM during the training process, as shown in Figure 5 . From Figure 5 , we can observe that α and β can adaptively modulate the sensitivity of each mrs-unit to correct the denoising process as network deepening, which is consistent with Eq. 16. Moreover, we can also observe that at the beginning of training, the sensitivity scores are relatively low. As training advances, α and β are supervised to correct the sensitivity until obtaining relatively higher sensitivity scores.To explore the actual effects of different residual settings in deep training, we first perform the comparison experiments on 5 different residual variants: 1) Variant-0 (Ours): z i+1 = z i +αf (z i )+β;2) Variant-1 (AdaLN [77] ):z i+1 = z i + f (αz i + β); 3) Variant-2: z i+1 = αz i + f (z i ) + β; 4)Variant-3 (ResNet [78] ): z i+1 = z i + f (z i ); 5) Variant-4 (ReZero [79] ): z i+1 = z i + αf (z i ). We utilize Latte-XL as backbone to train each variant from scratch and then evaluate their performance for video generation. As depicted in Figure 6 , as the number of training steps increases, almost all variants can converge effectively, but only Variant-0 (Our approach) achieves the best FVD scores. We speculate that it may be because this post-processing gating-residual setting maintains excellent dynamic consistency with the reverse denoising process, thus achieving better performance.Moreover, we further perform the deeep scalability experiments, which are shown in Figure 7 . We can observe that as the depth of residual units increases, the performance of the model can be further improved, which illustrates the positive correlation between model performance and the depth of residual units and further highlights the deep scalability advantage of our Neural-RDM. 
related work: 
 Deep Residual Networks. Most common deep residual networks can be divided into two types of architectures: flow-shaped stacking (FS) and u-shaped stacking (US) architectures. As a milestone of flow-based deep residual networks, ResNet [78] has led the research of visual understanding tasks [80] .In fact, the pratices [81, 82] and theories [83, 84, 85] that introducing the highway connections [86, 87] have been studied for a long time and have demonstrated excellent advantages in dealing with vanishing/exploding gradients and numerical propagation errors in deep stacked networks. Different from ResNet, U-Net [51] is a leader of u-shaped networks and almost dominated diffusion-based generative models [2, 30] . Though achieving remarkable success, both types of CNN-based models still face concerns about training efficiency. Recent years, Transformer [49] and ViT [50] have emerged as new state-of-the-art backbones in computer vision and multimodal [88, 89, 90, 91] and have also gained prominence in various diffusion models. Among them, DiT [59] and U-ViT [52] are two representative works by respectively adopting flow-shaped and u-shaped residual stacking fashions, which have enabled many studies on deep generative models [60, 92, 93, 94, 95] . In this work, we unify the above two types of residual stacking architectures from a dynamic perspective and propose a unified and deep scalable neural residual framework with a same gating-residual ODE.Diffusion Models. Recent years has witnessed the remarkable success of diffusion models [2, 3, 4] , due to their impressive generative capabilities. Previous efforts mainly focus on sampling procedure [25, 26, 27, 28] , conditional guidance [31, 32, 96, 97, 98] , likelihood maximization [33, 34, 35, 36] and generalization ability [37, 99, 39, 10] and have gained enormous attention. Recently, a major research topic on diffusion models is scalability. DiT [59] is one of the most representative models by exploiting a scalable Transformer to train latent diffusion models for image generation.Latte [60] stands on the shoulders of DiT to further perform temporal learning for video generation. However, both Latte and DiT adopt the residual structure of Transformer by default and utilize S-AdaLN to incorporate guidance information, they generally lack: 1) attention to the residual structure and 2) study the dynamic nature of the deep generative models, and 3) ignore the error propagation issues from deeper networks and therefore are still limited by the bottleneck of massively scalable training.Overall, we practically unify u-shaped and flow-shaped stacking networks and to propose a unified and deep scalable neural residual diffusion model framework. Moreover, we theoretically parameterize the previous human-designed mean-variance scheduler and demonstrate excellent dynamics consistency. 
conclusion: 
 In this paper, we have presented Neural-RDM, a simple yet meaningful change to the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training of extremely deep networks. Further, we have explored the nature of two common types of neural networks that enable effective denoising estimation. On this basis, we introduce a parametric method to replace previous human-designed mean-variance schedulers into a series of learnable gating-residual weights. Experimental results on various generative tasks show that Neural-RDM obtains the best results, and extensive experiments also demonstrate the advantages in improving the fidelity, consistency of generated content and supporting large-scale scalable training. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: Yes, we discuss it in Social Impact but in reality our approach is largely irrelevant to safety precautions.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Yes, we have followed these license specifications and accurately stated contributions from previous work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this section, we provide mathematical intuitions for our Neural-RDMs.Continuous-time Residual Networks. For a deep neural network F θ (•) with depth L, let F θi represents the minimum residual unit block i (Figure 1 (a) ). Instead of propagating the signal z through vanilla neural transformation ẑ = f θ (z), we introduce a gating-based skip-connection for the signal z, which relys on the gating weights α and β to modulate the non-trivial transformationEQUATION)In the case of continuous time, this dynamic equation describing the change process of the signal z is called the gating-residual ODE:EQUATIONDiffusion Probability Models. The diffusion probability models are modeled as: 1) a deterministic forward noising process q(x t |x t-1 ) = N (x t ; √ α t x t-1 , (1 -α t )I) from the original image x 0 to a pure-Gaussian distribution x T ∼ N (0, I), which can be formulated in an accumulated form:EQUATION2) a iteratively predictable reverse denoising process p θ (x t-1 |x t ) = N (x t-1 ; µ θ (x t , t), Σ θ (x t , t)), which can be trained in a simplied denoising objective L simple by merging µ θ and Σ θ into predicting noise ϵ θ ,EQUATION) where t ∼ U[1, T ] is time parameters, U(•) denotes uniform distribution. Moreover, in Stable Diffusion [30] , the image x t is compressed into a latent variable z t by encoder E for more efficient training, i.e., z t = E(x t ), thus this preliminary objective is usually defined as making ϵ θ (z t , t) as close to ϵ ∼ N (0, I) as possible.Reverse Denoising ODE. A remarkable property of the SDE (Eq. 19) is the existence of a reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45] ), which retains the same marginal probability densities as the SDE (See Appendix A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,EQUATIONwhere ∇ z log p t (z t ) denotes the gradient of the log-likelihood of p t (z t ), which can be estimated by a score matching network F θ (z t , t).Dynamics Consistency. Refer to Eq. 18 and Eq. 21, we define this dynamic consistency as: For any time-dependent signal z t , the different dynamics systems describe it with the same motion path (or the same change rate of data distribution). Note that in Eq. 21, we achieve this by using a re-parameterized approach.Latent Space Projection. The latent space projection is proposed by [30] to compress the input images x 0 into a perceptual high-dimensional space to obtain z 0 by leveraging a pretrained VQ-VAE model [100] . The VQ-VAE is also used by our Neural-RDM, it consists of an encoder E and a decoder G. The mathematical definition is: Given an input image x ∈ R H×W ×3 , the VQ-VAE first compress the image x into a latent variable ẑ by encoder E, i.e., ẑ = E(x) and ẑ ∈ R h×w×d , where h and w respectively denote scaled height and width (scaled factor f = H/h = W/w = 8), and d is the dimensionality of the compressed latent variable. After going through the diffusion step described in Eq. 5 and Eq. 6, the latent variable ẑ is updated and finally reconstructed into x by decoder G,EQUATIONwhere LDM(•) represents the latent diffusion models (including Unet-based or Transformer-based), θ denotes the parameters of LDM, and π denotes the parameters of the VQVAE that are frozen to train our Neural-RDM models.Motivated by [101] , we follow [45] to give a proof: A remarkable property of the SDE (Eq. 5) is the existence of a reverse ODE (PF-ODE [45] ), which retain the same marginal probability densities as the SDE. We consider the SDE in Eq. 5, which possesses the following form:EQUATIONwhere µ(•, t) : R d → R d and σ(•, t) : R d → R d×d . The marginal probability density p t (z t ) evolves according to Kolmogorov's forward equation [102] ∂p t (z) ∂t = - d i=1 ∂ ∂z i [µ i (z t , t)p t (z t )] + 1 2 d i=1 d j=1 ∂ 2 ∂z i ∂z j d k=1 σ ik (z t , t)σ jk (z t , t)p t (z t ) . = - d i=1 ∂ ∂z i [µ i (z t , t)p t (z t )] + 1 2 d i=1 ∂ ∂z i d j=1 ∂ ∂z j d k=1 σ ik (z t , t)σ jk (z t , t)p t (z t ) .(24) Since the sub-part of Eq. 24 can be written in the following form:EQUATIONThus we can obtain:EQUATIONwhere we define μi (•) as:EQUATIONCombining Eq. 26 and Eq. 27, we can conclude that Equation Eq. 26 still describes a Kolmogorov's forward process but with σ(z t , t) := 0 as:EQUATIONWhich proves that it is actually an ODE after reverse transformation μ(•):EQUATIONwhich is essentially the same with our Denoising-Diffusion-ODE given by Eq. 6. Therefore, we demonstrate the existence of the reverse ODE and the practicality of parameterizing the mean-variance scheduler from the reverse ODE. Limitation Discussion. Although significant improvements have been achieved, Neural-RDM still has some limitations, the most important of which is that the gated residual mechanism only inhibits rather than completely avoids the sensitivity decrease and numerical propagation errors caused by the deepening of the network. If we want to completely avoid it, we may have to give up stacking-based deep network architectures, but that will lead to a significant reduction in performance. Therefore, our method chooses to continue to deepen the stacking of the network and suppress error propagation as much as possible in the trade-off between the two.A. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper addresses an essential problem of scaling up diffusion models, making use of gated residual mechanisms. The positive feedback of the reviewers, confirmed during the discussion period, indicates that the paper offers valuable insights and interesting method to achieve scaling of diffusion models. The authors are requested to include the results of the discussion of the reviewers in the final version of their paper, specifically related to the complexity of the method (empirical evaluation) as well as clarity of the presentation.

Reviews: review: paper_summary: paper_summary: This paper addresses the problem of the numerical propagation errors of progressively deeper stacked neural networks for generative models. It proposes Neural Residual Diffusion Models (Neural-RDM), which introduced a series of learnable gated residual parameters to the common architectures of deep generative networks. Evaluation of the proposed method on image generation and video generation benchmarks demonstrate its superior performance over state-of-the-art methods.
strength_weakness: strength_weakness: - It is not clear what the specific definition of the "Scalability" columns in Table 1 and Table 2 is. Could you clarify the metrics represented by the cross, tick, and double tick symbols? 
- Since the paper is targeting on addressing the scalability of deep generative models, it would be better if the scalability experiments could be enhanced. Figure 7 shows that the performance of the proposed method can be improved as the depth of the network increases. However, how does its scalability compare with baseline architectures? Also, does it show similar scalability on other tasks and datasets?
- As the authors have discussed in the limitation section, the numerical propagation errors caused by stacking network layers cannot be completely avoided. Therefore, it would be more accurate not to claim "enabling the networks to be *infinitely* stacked".
questions: questions: See Weakness.
limitations: limitations: The authors have discussed clearly about the potential limitations and social impacts in A.4 and A.5.
paper_summary: paper_summary: This work presents a framework for visual generative diffusion models, aiming at addressing the challenges associated with deep stacked networks in terms of numerical propagation errors and scalability.
strength_weakness: strength_weakness: 1. While the introduction of gated residual parameters is innovative, it may also add complexity to the model, which could potentially make it harder to train or less intuitive for practitioners to understand.

2. How about the computation requirements for this scalability model?

3. Model parameters , GFLOPs should be mentioned.
questions: questions: Listed in weakness.
limitations: limitations: N/A
paper_summary: paper_summary: The paper describes the gradual change in the state z_t of a diffusion-based neural network as an ODE, and links the flow-architecture and UNet-architecture to the parameters of the ODE. Then, it describes a way to better train this network based on the dynamics of this ODE, and proposes an alternative loss function. It then performs experiments on image and video generation with this loss function, and several variants of the architecture.
strength_weakness: strength_weakness: Some parts of the paper are not clearly written. In Figure 2, the caption does not sufficiently explain the figure. In particular, (a) and (c) look the same, the difference needs to be elaborated on (corresponding to the main text). What dotted lines mean, what coloured lines mean, etc. has to be explained, in Figures 1 and 2.

Sections 2.2 and 2.3 need to be written more clearly. In particular, the dimensions of the involved variables need to be mentioned. It is unclear whether the network is used to completely or partially denoise the input. 

Most image-based experiments seem to be finetuned experiments, which are dependent on the biases of the original network used. Section 3.5 contains experiments on models trained from scratch, which is great. It is unclear though why Variant 0 performs so much better than Variant 4, since they only have a scalar "beta" difference.
questions: questions: Is the final loss function as stated in lines 162-163? In which case, does the loss apply to each layer separately? Do you sum the losses per layer?

How was the value of gamma as 0.35 decided?

It is unclear whether the network is used to completely or partially denoise the input. It seems as though it is assumed that the network is completely denoising from noise to data (instead of transforming partially noised to slightly less noised). Because the dynamics of F mention L -> infinite. If, however, F is assumed to be the full sampling procedure of multiple denoising steps, then Figure 2 doesn't make sense. In brief, it is unclear how the full dynamics are connected with each forward pass of the network. This is especially confusing from the perspective of the UNet.

The difference between Variant 0 and Variant 4 seems to be just a scalar beta value. Why does this 1 parameter affect performance so much?
limitations: limitations: The paper states where the model works better than prior works and where it doesn't.

Comments: title: Thanks!
comment: We greatly appreciate you sparing time to read our rebuttal and give us an equally careful point-by-point response with an improved rating, and we will supplement these details to the final manuscript.
Thank you for your point-by-point response, and the clarifications.
1. Those details are very helpful, please add them to the figure in the final version!
2. While the dimensions are indeed easy to specify, it is important to specify nonetheless for completion. It helps quickly clarify several confusion, such as which dimensions scalar variables are being added to, etc. Please add them to the final version.
3. There was indeed some confusion about this because of the reasons previously stated. Your response is quite helpful, please add it to the final version. Since the same variable names such as F_theta are being specified for different architectures, it is critical to clarify such confusions in the main text.
4. Good to know, thanks for the clarification.
5. Thanks for the clarification.
6. Good to know that \gamma was calculated empirically, would be helpful to add this maybe to the supplementary.
In light of the authors' response, I increase my score.
Dear Reviewer 6rnE,

We would like to extend our appreciation for your time and valuable comments. We are eagerly looking forward to receiving your valuable feedback and comments on the points we addressed in the rebuttal. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. Thank you for your dedication to the review process.

Sincerely,

Authors
Dear Reviewer zq4Z,

We would like to extend our appreciation for your time and valuable comments. We are eagerly looking forward to receiving your valuable feedback and comments on the points we addressed in the rebuttal. Ensuring that the rebuttal aligns with your suggestions is of utmost importance.
Thank you for your dedication to the review process.

Sincerely,

Authors
We sincerely thank you for sparing time and efforts reading our rebuttal and giving a response, thank you.
Thanks for your reply. You have solved my concerns. It seems that the computational burden is not heavy.
I will keep my score.