Title: Neural Residual Diffusion Models for Deep Scalable Vision Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision generation

Diffusion models

Deep neural network
Abstract: The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities.
Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework (Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the  fidelity and consistency of generated content and supporting large-scale scalable training.

Neural Residual Diffusion Models for Deep Scalable Vision Generation

Abstract

The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer ) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities. Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework ( Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the fidelity and consistency of generated content and supporting large-scale scalable training. 1

1 Introduction

Diffusion models (DMs) [1, 2, 3, 4] have emerged as a class of powerful generative models and have recently exhibited high quality samples in a wide variety of vision generation tasks such as image synthesis [5, 6, 7, 8, 9, 10, 11], video generation [12, 13, 14, 15, 16, 17, 18, 19] and 3D rendering and generation [20, 21, 22, 23, 24]. Relying on the advantage of iterative denoising and high-fidelity generation, DMs have gained enormous attention from the community and have been significantly improved in terms of sampling procedure [25, 26, 27, 28], conditional guidance [29, 30, 31, 32], likelihood maximization [33, 34, 35, 36] and generalization ability [37, 38, 39, 10] in previous efforts.

However, current diffusion models still face a scalability dilemma, which will play an important role in determining whether could support scalable deep generative training on large-scale vision data and give rise to emergent abilities [40] similar to large language models (LLMs) [41, 42]. Representatively, the recent emergence of Sora [43] has pushed the intelligent emergence capabilities of generative models to a climax by treating video models as world simulators. While unfortunately,

Sora is still a closed-source system and the mechanism for the intelligence emergence is still not very clear, but the scalable architecture must be one of the most critical technologies, according to the latest investigation [44] on its reverse engineering.

To alleviate this dilemma and spark further research in the open source community beyond the realms of well established U-Net and Transformers, and enable DMs to be trained in new scalable deep generative architectures, we propose a unified and massively scalable Residual-style Diffusion Models framework ( Neural-RDM for short) with a learnable gating residual mechanism, as shown in Figure 1.

The proposed Neural-RDM framework aims to unify the current mainstream residual-style generative architecture (e.g., U-Net or Transformer ) and guide the emergence of brand new scalable network architectures with emergent capabilities. To achieve this goal, we first introduce a continuous-time neural ordinary differential equation (ODE) to prove that the generative denoising ability of the diffusion models is closely related to the residual-style network structure, which almost reveals the essential reason why any network rich in residual structure can denoise well: Residual-style neural units implicitly build an ordinary differential equation that can well fit the reverse denoising process through

(a) MRS-Unit

(b)  Neural Residual Denoising Models

ever-deepening neural units, thus supporting excellent generative abilities. Further, we also show that the gating-residual mechanism plays an important role in adaptively correcting the errors of network propagation and approximating the mean and variance of data, which avoids the adverse factors of network deepening. On this basis, we further present the theoretical advantages of the Neural-RDM in terms of stability and score prediction sensitivity when stacking this residual units to a very long depth by introducing another residual-sensitivity ODE. From a dynamic perspective, it reveals that deep stacked networks have the challenge of gradually losing sensitivity as the network progressively deepens, and our proposed gating weights have advantages in reverse suppression and error control.

Our proposed framework has several theoretical and practical contributions:

Unified residual denoising framework: We unify the residual-style diffusion networks (e.g., U-Net and Transformer ) by introducing a simple gating-residual mechanism and reveal the significance of the residual unit for effective denoising and generation from a brand new dynamics perspective.

Theoretically deep scalability: Thanks to the introduction of continuous-time ODE, we demonstrate that the dynamics equation expressed by deep residual networks possesses excellent dynamic consistency to the denoising probability flow ODE (PF-ODE) [45]. Based on this property, we achieve the simplest improvement to each mrs-unit by parameterizing a learnable mean-variance scheduler, which avoids to manually design and theoretically support massively deep scalable training.

Adaptive stability maintenance and error sensitivity control: When the mrs-units are infinitely stacked to express the dynamics of an overall network F θ , the main technical difficulty is how to reduce the numerical errors caused by network propagation and ensure the stability of denoising. By introducing a sensitivity-related ODE in Sec. 2.3, we further demonstrate the theoretical advantages of the proposed gated residual networks in enabling stable denoising and effective sensitivity control. Qualitative and quantitative experimental results also consistently show their effectiveness.

2 Neural Residual Diffusion Models

We propose Neural-RDM , a simple yet meaningful change to the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training of extremely deep networks. This framework is supported by three critical theories: 1) Gating-Residual ODE (Sec. 2.1), which defines the dynamics of the m inimum r esidual s tacking unit ( mrs-unit for short) that serves as the foundational denoising module, as shown in Figure 1 (a). Based on this gating-residual mechanism, we then introduce 2) Denoising-Dynamics ODE (Sec. 2.2) to further stack the mrs-units to become a continuous-time deep score prediction network F θ . Different from previous human-crafted mean-variance schedulers (e.g., variance exploding scheduler SMLD [46]

(〉(〉(∣(]

(〉(⊔(∣(]

(〉(〈(∣(]

(〉(⊔(∣(]

and variance preserving scheduler DDPM [2]), which may cause concerns about instability in denoising efficiency and quality, we introduce a parametric method to implicitly learn the mean and variance distribution, which lowers the threshold of manual design and enhances the generalization ability of models. Last but not least, to maintain the stability of the deep stacked networks and verify the sensitivity of each residual unit F θ i ( · ) to the network F θ , we stand on the shoulders of the adjoint sensitivity method [47, 48] to propose 3) Residual-Sensitivity ODE (Sec. 2.3), which means the sensitivity-related dynamics of each latent state z i from F θ i ( · ) to the deep network F θ . Through rigorous derivation, we prove that the parameterized gating weights have a positive inhibitory effect on sensitivity decaying as network deepening. We will elaborate on them below.

2.1 Gating-Residual Mechanism

Let F θ i represents the minimum residual unit block i (Figure 1 (a)), f ( · ) denotes any feature mapper wrapped by F θ i . Instead of propagating the signal z through each of vanilla neural transformation ˆ z = f θ ( z ) directly, we introduce a gating-based residual connection for the signal z , which relys on the two learnable gating weights ˆ α and ˆ β to modulate the non-trivial transformation F θ i ( z i ) as,

For a deep neural network F θ ( · ) with depth L , consider two common residual stacking fashions: Flow-shaped Stacking ( FS ) [49, 50] and U-shaped Stacking ( US ) [51, 52]. For the flow-based deep stacking networks as shown in Figure 2 (a), each residual unit f ( · ) accepts the output z i of the previous mrs-unit as input, and obtains a new hidden state z i +1 through gating-residual connection,

Note that Eq. 2 is a refined form of Eq. 1 in the case of flow-shaped stacking. In contrast, for the U-shaped deep stacking networks as in Figure 2 (b), each minimum residual unit contains two symmetrical branches, where the left branch receives the output z i of the previous mrs-unit 's left branch as input (called the read-in branch), and the right branch performs the critical nonlinear residual transformation for readout (called the read-out branch), which can be formally described as:

Here Eq. 3 is a refined form of Eq. 1 in the case of U-shaped stacking, ˆ α i and ˆ β i collectively denotes the gating weights from the left and right branches, F θ i is the i -th minimum residual unit of the U-shaped networks, and ' ↪ → ' denotes the skip-connection for ' z i +1 - → z 2 L -2 -i ', which is computed recursively via F θ i +1: L -1 . To enable the networks to be infinitely stacked, we introduce a continuous-time Gating-Residual ordinary differential equation (ODE) to express the neural dynamics

..."

"

Christina Hendricks

, dressed as a

Disney princess

, stands in full

torso on a city street, ... "

"A grand

cathedral corridor

in a

hypermaximalist style, adorned with

intricate white marble and

blue ornate

"A mystical being made of

dust

surrounded

by

vines

,

moss

,

ferns

,

foliage

, and

wild plants

..."

"A beautiful

garden

in summer

adorned with

white lilies

, featuring

a natural wreath and bushes ..."

"A handsome

fairy prince

with blue

hair delicately holds out a single

white apple

, his perfect facial ..."

Baseline

Neural-RDM

of these two types of deep stacking networks ( δ = 1 L , L - →∞ denotes the number of the mrs-units ),

where ϕ represents the gating weights, which can be independently trained or fine-tuned without considering the parameters θ of the feature mapping network F θ ( · ) itself.

2.2 Denoising Dynamics Parameterization

The above-mentioned gating-residual mechanism is utilized to modulate mainstream deep stacking networks and unify them into a residual-style massively scalable generative framework, as shown in Figure 2 (c). Next, we further explore the essential relationship between residual neural networks and score-based generative denoising models from a dynamic perspective.

First, inspired by the theory of continuous-time diffusion models [45, 54], the forward add-noising process can be expressed as a dynamic process with stochastic differential equation (SDE) as,

which describes a data perturbation process controlled by a mean-variance scheduler composed of µ ( z t , t ) and σ ( t ) respectively, w t denotes the standard Brownian motion. Compared with the forward process, the core of the diffusion model is to utilize a deep neural network (as deep and large as possible) for score-based reverse prediction [46, 55]. A remarkable property of this SDE is the existence of a reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45]), which retain the same marginal probability densities as the SDE (See Appendix. A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,

where ∇ z log p t ( z t ) denotes the gradient of the log-likelihood of p t ( z t ) , which can be estimated by a score matching network F θ ( z t , t ) . Here we re-parameterize the PF-ODE by utilizing gated weights to replace the manually designed mean-variance scheduler, in which ˆ α t,ϕ and ˆ β t,ϕ denotes the time-dependent dynamics parameters, which is respectively parameterized to represent -1 2 σ ( t ) 2 and µ ( z t , t ) by our proposed gating-residual mechanism. Note that F θ ( · ) is a score estimation network composed of infinite mrs-units block i (i.e., F θ i ), which enables massively scalable generative training on large-scale vision data, but also presents the challenge of numerical propagation errors.

2.3 Residual Sensitivity Control

To control the numerical errors in back-propagation and achieve steadily and massively scalable training, we stand on the shoulders of the adjoint sensitivity method [47, 48] to introduce another

Residual-Sensitivity ODE, which is utilized to evaluate the sensitivity of each residual-state z t of the mrs-unit F θ i to the total loss L derived by score estimation network F θ ( · ) (the sensitivity is denoted as s t = d L d z t , δ denotes an infinitesimal time interval) and can be formally described by the chain rule,

On the basis of Eq. 7, we next continue to discuss the dynamic equation of sensitivity changing with time t . First, considering the trivial transformation f θ ( · ) without gating-residual mechanism,

We can rewrite Eq. 7 based on Eq. 8 as:

The Residual-Sensitivity ODE under vanilla situation then can be derived,

According to the derived residual-sensitivity equation in Eq. 10, we further use the Euler solver to obtain the sensitivity s t 0 of the starting state z t 0 to network F θ ( · ) as,

Due to the non-negativity of the integral and the gradient ∂f θ ( z t ,t ) ∂ z t not equals to 0 , we can obtain a gradually decaying sensitivity sequence: s t L > s t L -1 > · · · > s t 0 . Similarly, when defining parameter-sensitivity s θ = d L d θ , the same decaying results for s θ 0 can be obtained:

Baseline

Neural-RDM

Baseline

Neural-RDM

Baseline

Neural-RDM

Baseline

Neural-RDM

To alleviate this problem, and enable stable training in massively deep scalable architecture, we introduce the following non-trivial solution with gating-residual transformation,

Substitute Eq. 13 into Eq. 7 to obtain the corrected sensitivity ˆ s t = d L d ˆ z t as:

The non-trivial Residual-Sensitivity ODE can be derived as,

Through the Euler solver, we can also obtain the sensitivity ˆ s t 0 of the starting state adjusted by the gating-residual weights,

Where α t,ϕ and β t,ϕ adaptively modulate and update the sensitivity of each mrs-unit to the final loss, which supports being trained through minimizing L s = ||F θ ( z t , t ) -∇ z log p t ( z t ) || 2 2 + γ · ∑ L || α t,ϕ · ∂f θ (ˆ z t ,t ) ∂ ˆ z t -β t,ϕ || 2 2 in full-parameter training or model fine-tuning fashions.

3 Experiments

We present the main experimental settings in Sec. 3.1. To evaluate the generative performance of Neural-RDM, we compare it with state-of-the-art conditional/unconditional diffusion models for image synthesis and video generation in Sec. 3.2 and Sec. 3.3 respectively. We also visualize and analyze the effects of the proposed gated residuals and illustrate their advantages in enabling deep scalable training, which are presented in Sec. 3.4 and Sec. 3.5.

3.1 Experimental Settings

Datasets. For image synthesis tasks, we train and evaluate the Class-to-Image generation models on the ImageNet [61] dataset and train and evaluate the Text-to-Image generation models on the

MSCOCO [65] and JourneyDB [53] datasets. All images are resized to 256 × 256 resolution for training. For video generation tasks, we follow the previous works [12, 60] to train None-to-Video ( i.e., unconditional video generation) models on the SkyTimelapse [62] and Taichi [63] datasets, and train Class-to-Video models on the UCF-101 [64] dataset. Moreover, we follow previous works [12, 60] to sample 16 -frame video clips from these video datasets and then resize all frames to 256 × 256 resolution for training and evaluation.

Implementation details. We implement our Neural-RDMs into Neural-RDM-U (U-shaped) and Neural-RDM-F (Flow-shaped) two versions on top of the current state-of-the-art diffusion models LDM [30] and Latte [60] for image generation, and further employ the Neural-RDM-F version for video generation. Specifically, we first load the corresponding pre-trained models and initialize gating parameters { α = 1 , β = 0 } of each layer, then perform full-parameter fine-tuning to implicitly learn the distribution of the data for acting as a parameterized mean-variance scheduler. During the training process, we adopt an explicit supervision strategy to enhance the sensitivity correction capabilities of α and β for deep scalable training, where the explicitly supervised hyper-parameter γ is set to 0.35. Eventually, we utilize the AdamW optimizer with a constant learning rate of 5 × 10 4 for all models and exploit an exponential moving average (EMA) strategy to obtain and report all results.

Evaluation metrics. Following the previous baselines [30, 58, 59, 60], we adopt Fréchet Inception Distance (FID) [66], sFID [67] and Inception Score (IS) [68] to evaluate the image generation quality and the video frame quality (except for sFID). Furthermore, we utilize a Fréchet Video Distance (FVD) [69] metric similar with FID to evaluate the unconditional and conditional video generation quality. Among these metrics, FVD is closer to human subjective judgment and thus better reflects the visual quality of the generated video content. Adhering to the evaluation guidelines proposed by StyleGAN-V [70], we calculate the FVD scores by analyzing 2048 generated video clips with each clip consists of 16 frames.

Baselines. We compare the proposed method with the recent state-of-the-art baselines, and categorize them into three groups: 1) GAN-based. BigGAN-deep [56] and StyleGAN-XL [57] for image task, MoCoGAN [71], MoCoGAN-HD [72], DIGAN [73], StyleGAN-V [70] and MoStGAN-V [74] for video task. 2) U-shaped. ADM [58] and LDM [30] for image task, PVDM [75] and LVDM [12] for video task. 3) F-shaped. DiT-XL/2 [59] and Latte-XL [60] for image task, VideoGPT [76] and Latte-XL [60] (with temporal attention learning) for video task.

3.2 Experiments on Image Synthesis with Deep Scalable Spatial Learning

For a more objective comparison, we maintain approximately the same model size to perform classconditional and text-conditional image generation experiments, which are shown in Table 1. From Table 1, it can be observed that our Neural-RDMs have obtained state-of-the-art results. Specifically,

(a)

(b)

(c)

score

250

300

200

Variant-1

Variant-2

Variant-3

Variant-0

score

500

300

400

Variant-0

Variant-1

Variant-2

Variant-3

score

3000

2500

2000

Variant-0

Variant-1

Variant-2

Variant-3

FVD

150

FVD

FVD

1500

200

100

1000

50

100

500

0

10k20k30k40k50k60k70k80k90k100k

10k20k30k40k50k60k70k80k90k100k

D

Trainingiterations

Trainingiterations

Trainingiterations

the flow-based version (i.e., Neural-RDM-F) consistently outperforms all class-to-image baselines in all three image's generative benchmarks and meanwhile obtains relatively suboptimal results on another text-to-image evaluations. It is worth noting that another Neural-RDM-U version have made up for this shortcoming and achieved optimal results, which may benefit from the more powerful semantic guidance abilities of the cross-attention layer built into U-Net. To more clearly present the actual effects of the gated residuals, we further perform qualitative comparative experiments, which are shown in Figure 3. Compared with the latest baseline (SDXL-1.0 [7]), we can observe that the samples produced by Neural-RDM exhibit exceptional quality, particularly in terms of fidelity and consistency in the details of the subjects in adhering to the provided textual prompts, which consistently demonstrates the effectiveness of our proposed approach in deep scalable spatial learning.

3.3 Experiments on Video Generation with Deep Scalable Temporal Learning

To further explore the effectiveness and specific contributions of proposed gating-residual mechanism in temporal learning, we continue to perform the video generation evaluations, which are shown in Table 2. From Table 2, we find that our model (flow-shaped version) basically achieves the best results (except for the second-best results in class-to-video evaluation). Specifically, compare with Latte-XL [60], Neural-RDM respectively achieves an improvement of 33 . 3% and 42 . 8% in FVD scores on SkyTimelapse and Taichi-HD datasets, which hints the powerful potential of flow-based deep residual networks in promoting generative emergent capabilities of video models. Furthermore, we exhibit a number of visual comparison results of the 16-frames video produced by Neural-RDM and baseline (Latte-XL [60]), as shown in Figure 4. We can observe that some generated frames from the baseline partially exhibits poor quality and temporal inconsistency. Compare with the baseline, Neural-RDM maintains tem-

1.0

0.8

0.6

0.6

0.4

0.4

0.2

0.2

18

0

12

2500

6

7500

Depth

10000

0

poral coherence and consistency, resulting in smoother and more dynamic video frames, which further reflects the effectiveness of proposed method in both quantitative and qualitative evaluations.

3.4 The Analyses of Gating Residual Sensitivity

To better illustrate the advantage of the gated residuals and understand the positive suppression effect for sensitivity attenuation as network deepening, we visualize the normalized sensitivity at different depths of our Neural-RDM during the training process, as shown in Figure 5. From Figure 5, we can observe that α and β can adaptively modulate the sensitivity of each mrs-unit to correct the denoising process as network deepening, which is consistent with Eq. 16. Moreover, we can also observe that at the beginning of training, the sensitivity scores are relatively low. As training advances, α and β are supervised to correct the sensitivity until obtaining relatively higher sensitivity scores.

3.5 Comparison Experiments of Gating Residual Variants and Deep Scalability

To explore the actual effects of different residual settings in deep training, we first perform the comparison experiments on 5 different residual variants: 1) Variant-0 (Ours) : z i +1 = z i + αf ( z i )+ β ; 2) Variant-1 (AdaLN [77]) : z i +1 = z i + f ( αz i + β ) ; 3) Variant-2 : z i +1 = αz i + f ( z i ) + β ; 4) Variant-3 (ResNet [78]) : z i +1 = z i + f ( z i ) ; 5) Variant-4 (ReZero [79]) : z i +1 = z i + αf ( z i ) .

We utilize Latte-XL as backbone to train each variant from scratch and then evaluate their performance for video generation. As depicted in Figure 6, as the number of training steps increases, almost all variants can converge effectively, but only Variant-0 (Our approach) achieves the best FVD scores. We speculate that it may be because this post-processing gating-residual setting maintains excellent dynamic consistency with the reverse denoising process, thus achieving better performance.

Moreover, we further perform the deeep scalability experiments, which are shown in Figure 7. We can observe that as the depth of residual units increases, the performance of the model can be further improved, which illustrates the positive correlation between model performance and the depth of residual units and further highlights the deep scalability advantage of our Neural-RDM.

4 Related Work

Deep Residual Networks. Most common deep residual networks can be divided into two types of architectures: flow-shaped stacking (FS) and u-shaped stacking (US) architectures. As a milestone of flow-based deep residual networks, ResNet [78] has led the research of visual understanding tasks [80]. In fact, the pratices [81, 82] and theories [83, 84, 85] that introducing the highway connections [86, 87] have been studied for a long time and have demonstrated excellent advantages in dealing with vanishing/exploding gradients and numerical propagation errors in deep stacked networks. Different from ResNet, U-Net [51] is a leader of u-shaped networks and almost dominated diffusion-based generative models [2, 30]. Though achieving remarkable success, both types of CNN-based models still face concerns about training efficiency. Recent years, Transformer [49] and ViT [50] have emerged as new state-of-the-art backbones in computer vision and multimodal [88, 89, 90, 91] and have also gained prominence in various diffusion models. Among them, DiT [59] and U-ViT [52] are two representative works by respectively adopting flow-shaped and u-shaped residual stacking fashions, which have enabled many studies on deep generative models [60, 92, 93, 94, 95]. In this work, we unify the above two types of residual stacking architectures from a dynamic perspective and propose a unified and deep scalable neural residual framework with a same gating-residual ODE.

Diffusion Models. Recent years has witnessed the remarkable success of diffusion models [2, 3, 4], due to their impressive generative capabilities. Previous efforts mainly focus on sampling procedure [25, 26, 27, 28], conditional guidance [31, 32, 96, 97, 98], likelihood maximization [33, 34, 35, 36] and generalization ability [37, 99, 39, 10] and have gained enormous attention. Recently, a major research topic on diffusion models is scalability. DiT [59] is one of the most representative models by exploiting a scalable Transformer to train latent diffusion models for image generation. Latte [60] stands on the shoulders of DiT to further perform temporal learning for video generation. However, both Latte and DiT adopt the residual structure of Transformer by default and utilize S-AdaLN to incorporate guidance information, they generally lack: 1) attention to the residual structure and 2) study the dynamic nature of the deep generative models, and 3) ignore the error propagation issues from deeper networks and therefore are still limited by the bottleneck of massively scalable training.

Overall, we practically unify u-shaped and flow-shaped stacking networks and to propose a unified and deep scalable neural residual diffusion model framework. Moreover, we theoretically parameterize the previous human-designed mean-variance scheduler and demonstrate excellent dynamics consistency.

3000

RDM-24

RDM-12

2500

RDM-28

2000

score

1500

FVD

1000

500

0

Trainingiterations

5 Conclusion

In this paper, we have presented Neural-RDM, a simple yet meaningful change to the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training of extremely deep networks. Further, we have explored the nature of two common types of neural networks that enable effective denoising estimation. On this basis, we introduce a parametric method to replace previous human-designed mean-variance schedulers into a series of learnable gating-residual weights. Experimental results on various generative tasks show that Neural-RDM obtains the best results, and extensive experiments also demonstrate the advantages in improving the fidelity, consistency of generated content and supporting large-scale scalable training.

Acknowledgments and Disclosure of Funding

This work is supported by the National Science and Technology Major Project (2023ZD0121403), National Natural Science Foundation of China (No. 62406161), China Postdoctoral Science Foundation (No. 2023M741950), and the Postdoctoral Fellowship Program of CPSF (No. GZB20230347).

References

Zhiyuan Ma, Yuzhu Zhang, Guoli Jia, Liangliang Zhao, Yichao Ma, Mingjie Ma, Gaofeng Liu, Kaiyan Zhang, Jianjun Li, and Bowen Zhou. Efficient diffusion models: A comprehensive survey from principles to practices. arXiv preprint arXiv:2410.11795 , 2024.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in NeurIPS , 33:6840-6851, 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , pages 8162-8171, 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Zhiyuan Ma, Guoli Jia, Biqing Qi, and Bowen Zhou. Safe-sd: Safe and traceable stable diffusion with text prompt trigger for invisible generative watermarking. In ACM Multimedia 2024 .

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Zhiyuan Ma, Guoli Jia, and Bowen Zhou. Adapedit: Spatio-temporal guided adaptive editing algorithm for text-based continuity-sensitive image editing. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4154-4161, 2024.

Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 , 2022.

Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 , 2024.

Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistencyaware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23040-23050, 2023.

Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi-stage high-aesthetic video generation. arXiv preprint arXiv:2401.04468 , 2024.

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 , 2024.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations , 2022.

Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305.18766 , 2023.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403.12008 , 2024.

Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning , pages 42390-42402. PMLR, 2023.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning , pages 32211-32252, 2023.

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 , 2023.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representations , 2023.

Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7346-7356, 2023.

Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion models. arXiv preprint arXiv:2205.13699 , 2022.

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning , pages 4672-4712. PMLR, 2023.

Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning , pages 14429-14460. PMLR, 2022.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems , 34:14151428, 2021.

Yuhan Li, Yishun Dou, Xuanhong Chen, Bingbing Ni, Yilin Sun, Yutian Liu, and Fuzhen Wang. Generalized deep 3d shape prior via part-discretized diffusion process. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16784-16794, 2023.

Zhiyuan Ma, Jianjun Li, Bowen Zhou, et al. Lmd: Faster image reconstruction with latent masking diffusion. arXiv preprint arXiv:2312.07971 , 2023.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and Yasmine et al Babaei. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.

Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177 , 2024.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

ML Chambers. The mathematical theory of optimal processes. Journal of the Operational Research Society , 16(4):493-494, 1965.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241, 2015.

Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a vit backbone for score-based diffusion models. In NeurIPS 2022 Workshop on Score-Based Methods , 2022.

Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems , 36, 2024.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , pages 574-584. PMLR, 2020.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2018.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in NeurIPS , 34:8780-8794, 2021.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 , 2024.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2364-2373, 2018.

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems , 32, 2019.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Mandi Luo, Jie Cao, Xin Ma, Xiaoyu Zhang, and Ran He. Fa-gan: Face augmentation gan for deformation-invariant face recognition. IEEE Transactions on Information Forensics and Security , 16:2341-2355, 2021.

Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia. Generating images with sparse representations. In International Conference on Machine Learning , pages 7958-7968. PMLR, 2021.

Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision , pages 2830-2839, 2017.

Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018.

Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3626-3636, 2022.

Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1526-1535, 2018.

Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In International Conference on Learning Representations , 2020.

Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations , 2021.

Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5652-5661, 2023.

Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18456-18466, 2023.

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 , 2021.

Yunhui Guo, Chaofeng Wang, Stella X Yu, Frank McKenna, and Kincho H Law. Adaln: a vision transformer for multidomain learning and predisaster building information extraction from images. Journal of Computing in Civil Engineering , 36(5):04022024, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence , pages 1352-1361. PMLR, 2021.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transformations in perceptrons. In Artificial intelligence and statistics , pages 924-932. PMLR, 2012.

Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks , pages 37-45, 2012.

Nicol Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical report/IDSIA , 98, 1998.

Nicol N Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade , pages 207-226. Springer, 2002.

Tommi Vatanen, Tapani Raiko, Harri Valpola, and Yann LeCun. Pushing stochastic gradient towards second-order methods-backpropagation learning with transformations in nonlinearities. In Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part I 20 , pages 442-449. Springer, 2013.

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387 , 2015.

Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. Advances in neural information processing systems , 28, 2015.

Zhiyuan Ma, Jianjun Li, Guohui Li, and Kaiyan Huang. Cmal: A novel cross-modal associative learning framework for vision-language pre-training. In Proceedings of the 30th ACM International Conference on Multimedia , pages 4515-4524, 2022.

Zhiyuan Ma, Jianjun Li, Guohui Li, and Yongjing Cheng. Unitranser: A unified transformer semantic representation framework for multimodal task-oriented dialog system. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 103-114, 2022.

Zhiyuan Ma, Zhihuan Yu, Jianjun Li, and Guohui Li. Hybridprompt: bridging language models and human priors in prompt tuning for visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 13371-13379, 2023.

Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18733-18741, 2024.

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376 , 2024.

Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233 , 2024.

Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740 , 2024.

Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations , 2023.

Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeung Shum, Wei Liu, et al. Follow-your-click: Open-domain regional image animation via short prompts. arXiv preprint arXiv:2403.08268 , 2024.

Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01186 , 2023.

Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01900 , 2024.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR , pages 10696-10706, 2022.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , pages 12873-12883, 2021.

Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-planck equations through gradient-log-density estimation. Entropy , 22(8):802, 2020.

Bernt Øksendal and Bernt Øksendal. Stochastic differential equations . Springer, 2003.

A Appendix

A.1 Theoretical Interpretations

In this section, we provide mathematical intuitions for our Neural-RDMs.

Continuous-time Residual Networks. For a deep neural network F θ ( · ) with depth L , let F θ i represents the minimum residual unit block i (Figure 1 (a)). Instead of propagating the signal z through vanilla neural transformation ˆ z = f θ ( z ) , we introduce a gating-based skip-connection for the signal z , which relys on the gating weights ˆ α and ˆ β to modulate the non-trivial transformation F θ ( z ) as,

In the case of continuous time, this dynamic equation describing the change process of the signal z is called the gating-residual ODE :

Diffusion Probability Models. The diffusion probability models are modeled as: 1) a deterministic forward noising process q ( x t | x t -1 ) = N ( x t ; √ α t x t -1 , (1 -α t ) I ) from the original image x 0 to a pure-Gaussian distribution x T ∼ N (0 , I ) , which can be formulated in an accumulated form:

2) a iteratively predictable reverse denoising process p θ ( x t -1 | x t ) = N ( x t -1 ; µ θ ( x t , t ) , Σ θ ( x t , t )) , which can be trained in a simplied denoising objective L simple by merging µ θ and Σ θ into predicting noise ϵ θ ,

where t ∼ U [1 , T ] is time parameters, U ( · ) denotes uniform distribution. Moreover, in Stable Diffusion [30], the image x t is compressed into a latent variable z t by encoder E for more efficient training, i.e., z t = E ( x t ) , thus this preliminary objective is usually defined as making ϵ θ ( z t , t ) as close to ϵ ∼ N (0 , I ) as possible.

Reverse Denoising ODE. A remarkable property of the SDE (Eq. 19) is the existence of a reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45]), which retains the same marginal probability densities as the SDE (See Appendix A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,

where ∇ z log p t ( z t ) denotes the gradient of the log-likelihood of p t ( z t ) , which can be estimated by a score matching network F θ ( z t , t ) .

Dynamics Consistency. Refer to Eq. 18 and Eq. 21, we define this dynamic consistency as: For any time-dependent signal z t , the different dynamics systems describe it with the same motion path (or the same change rate of data distribution). Note that in Eq. 21, we achieve this by using a re-parameterized approach.

Latent Space Projection. The latent space projection is proposed by [30] to compress the input images x 0 into a perceptual high-dimensional space to obtain z 0 by leveraging a pretrained VQ-VAE model [100]. The VQ-VAE is also used by our Neural-RDM, it consists of an encoder E and a decoder G . The mathematical definition is: Given an input image x ∈ R H × W × 3 , the VQ-VAE first compress the image x into a latent variable ˆ z by encoder E , i.e., ˆ z = E ( x ) and ˆ z ∈ R h × w × d , where h and w respectively denote scaled height and width (scaled factor f = H/h = W/w = 8 ), and d is the dimensionality of the compressed latent variable. After going through the diffusion step described in Eq. 5 and Eq. 6, the latent variable ˆ z is updated and finally reconstructed into ˆ x by decoder G ,

where LDM( · ) represents the latent diffusion models (including Unet-based or Transformer-based), θ denotes the parameters of LDM, and π denotes the parameters of the VQVAE that are frozen to train our Neural-RDM models.

A.2 Additional Proofs

Motivated by [101], we follow [45] to give a proof: A remarkable property of the SDE (Eq. 5) is the existence of a reverse ODE (PF-ODE [45]), which retain the same marginal probability densities as the SDE. We consider the SDE in Eq. 5, which possesses the following form:

where µ ( · , t ) : R d → R d and σ ( · , t ) : R d → R d × d . The marginal probability density p t ( z t ) evolves according to Kolmogorov's forward equation [102]

Since the sub-part of Eq. 24 can be written in the following form:

Thus we can obtain:

where we define ˜ µ i ( · ) as:

Combining Eq. 26 and Eq. 27, we can conclude that Equation Eq. 26 still describes a Kolmogorov's forward process but with ˜ σ ( z t , t ) := 0 as:

Which proves that it is actually an ODE after reverse transformation ˜ µ ( · ) :

which is essentially the same with our Denoising-Diffusion -ODE given by Eq. 6. Therefore, we demonstrate the existence of the reverse ODE and the practicality of parameterizing the mean-variance scheduler from the reverse ODE.

A.3 More Generated Images

onlile

Bdgietian

fumgeruies

yaebiey

ANDRIE

A.4 Limitations

Limitation Discussion. Although significant improvements have been achieved, Neural-RDM still has some limitations, the most important of which is that the gated residual mechanism only inhibits rather than completely avoids the sensitivity decrease and numerical propagation errors caused by the deepening of the network. If we want to completely avoid it, we may have to give up stacking-based deep network architectures, but that will lead to a significant reduction in performance. Therefore, our method chooses to continue to deepen the stacking of the network and suppress error propagation as much as possible in the trade-off between the two.

A.5 Social Impact

Potential Social Implications. We believe that Neural-RDM will bring new thinking about deep network architectures to the generative community, and hopefully promote the generative emergence capabilities of vision generation models in the open source community. In addition, we hope that more researchers can follow the powerful capabilities of residual denoising to build brand new scalable network architectures beyond the realms of well established U-Net and Transformers.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We believe that the main claims made in the abstract and introduction accurately reflect the contributions and scope of the paper, refer to theoretical proof in Sec. 2.1- 2.3 and experimental proof in Sec. 3.2- 3.5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer to Sec. A.4 for a discussion of limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided all detailed assumptions and theoretical proofs in the main content and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We are convinced that we have achieved this and promise to disclose all code details after the paper is accepted.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We are convinced that we have achieved this and promise to disclose all code details after the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We are convinced that we have achieved this, please refer to Sec. 3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have presented the results of statistical significance in the main quantitative results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported the GPU power used in the experiments in Sec. 3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we comply with all NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we have discussed social impacts in Sec. A.5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Yes, we discuss it in Social Impact but in reality our approach is largely irrelevant to safety precautions.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we have followed these license specifications and accurately stated contributions from previous work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes the gradual change in the state z_t of a diffusion-based neural network as an ODE, and links the flow-architecture and UNet-architecture to the parameters of the ODE. Then, it describes a way to better train this network based on the dynamics of this ODE, and proposes an alternative loss function. It then performs experiments on image and video generation with this loss function, and several variants of the architecture.\n\nSTRENGTHS:\nThe paper does a great job of explaining how the dynamics of z_t through the network can be unified across the UNet and Flow architectures. It then derives a loss function based on the ODE of those dynamics. The derivation itself is involved and highly useful. The paper also performs several experiments on multiple modalities to validate the proposed hypothesis of the new loss function / perspective. In particular, there are experiments in both image and video generation, each being quite strenuous in terms of academic effort and computational requirements.\n\n---\nUpdate: I increased my score after the authors' response.\n\nWEAKNESSES:\nSome parts of the paper are not clearly written. In Figure 2, the caption does not sufficiently explain the figure. In particular, (a) and (c) look the same, the difference needs to be elaborated on (corresponding to the main text). What dotted lines mean, what coloured lines mean, etc. has to be explained, in Figures 1 and 2.\n\nSections 2.2 and 2.3 need to be written more clearly. In particular, the dimensions of the involved variables need to be mentioned. It is unclear whether the network is used to completely or partially denoise the input. \n\nMost image-based experiments seem to be finetuned experiments, which are dependent on the biases of the original network used. Section 3.5 contains experiments on models trained from scratch, which is great. It is unclear though why Variant 0 performs so much better than Variant 4, since they only have a scalar \"beta\" difference.\n\nQUESTIONS:\nIs the final loss function as stated in lines 162-163? In which case, does the loss apply to each layer separately? Do you sum the losses per layer?\n\nHow was the value of gamma as 0.35 decided?\n\nIt is unclear whether the network is used to completely or partially denoise the input. It seems as though it is assumed that the network is completely denoising from noise to data (instead of transforming partially noised to slightly less noised). Because the dynamics of F mention L -> infinite. If, however, F is assumed to be the full sampling procedure of multiple denoising steps, then Figure 2 doesn't make sense. In brief, it is unclear how the full dynamics are connected with each forward pass of the network. This is especially confusing from the perspective of the UNet.\n\nThe difference between Variant 0 and Variant 4 seems to be just a scalar beta value. Why does this 1 parameter affect performance so much?\n\nLIMITATIONS:\nThe paper states where the model works better than prior works and where it doesn't.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This work presents a framework for visual generative diffusion models, aiming at addressing the challenges associated with deep stacked networks in terms of numerical propagation errors and scalability.

STRENGTHS:
1. Clear motivation

2. The authors provide a  theoretical analysis for their approach, including the use of continuous-time neural ODEs to demonstrate the relationship between residual-style network structures and generative denoising abilities.

3. Sufficient experiments: The paper is supported by extensive experimental evidence, which show that the proposed models achieve state-of-the-art performance on various generative tasks, including image and video generation.

WEAKNESSES:
1. While the introduction of gated residual parameters is innovative, it may also add complexity to the model, which could potentially make it harder to train or less intuitive for practitioners to understand.

2. How about the computation requirements for this scalability model?

3. Model parameters , GFLOPs should be mentioned.

QUESTIONS:
Listed in weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This paper addresses the problem of the numerical propagation errors of progressively deeper stacked neural networks for generative models. It proposes Neural Residual Diffusion Models (Neural-RDM), which introduced a series of learnable gated residual parameters to the common architectures of deep generative networks. Evaluation of the proposed method on image generation and video generation benchmarks demonstrate its superior performance over state-of-the-art methods.

STRENGTHS:
- The proposed method is simple and well-motivated. 
- The experiments on generation benchmarks are comprehensive, and the great results demonstrate the efficacy of the proposed method.
- The paper is clearly written.

WEAKNESSES:
- It is not clear what the specific definition of the "Scalability" columns in Table 1 and Table 2 is. Could you clarify the metrics represented by the cross, tick, and double tick symbols? 
- Since the paper is targeting on addressing the scalability of deep generative models, it would be better if the scalability experiments could be enhanced. Figure 7 shows that the performance of the proposed method can be improved as the depth of the network increases. However, how does its scalability compare with baseline architectures? Also, does it show similar scalability on other tasks and datasets?
- As the authors have discussed in the limitation section, the numerical propagation errors caused by stacking network layers cannot be completely avoided. Therefore, it would be more accurate not to claim "enabling the networks to be *infinitely* stacked".

QUESTIONS:
See Weakness.

LIMITATIONS:
The authors have discussed clearly about the potential limitations and social impacts in A.4 and A.5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14