Neural Residual Diffusion Models for Deep Scalable Vision Generation
Authors: Zhiyuan Ma
Liangliang Zhao
Biqing Qi
Bowen Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision generation
Diffusion models
Deep neural network
Primary area: diffusion_based_models

Abstract: The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities.
Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework (Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the  fidelity and consistency of generated content and supporting large-scale scalable training.

Neural Residual Diffusion Models for Deep Scalable Vision Generation
Zhiyuan Ma', Liangliang Zhao!:2 Biqing Qi Bowen Zhou 1.3-
Department of Electronic Engineering. Tsinghua University, Beijing, China 2Frontis.Al Beijing, China
SShanghai Al Laboratory, Shanghai, China
{mzyth, zhoubowen) (ttte)/ttssinnnn edu.
Abstract
The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emer- gence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effec- tively perform generative denoising lies the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities. Afterwards, we stand on the shoulders two common types of deep stacked networks to propose unified and massively scalable Neural Residual Diffusion Models framework (Neural-Re) RDDM for short), which is simple yet meaningful change to the common architecture of deep generative networks by introducing a series learnable gated residual parameters that conform to the generative dynamics. Experimental results various gener- ative tasks show that the proposed neural residual models obtain state-of-the-art scores image's 's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the fidelity and consistency of generated content and supporting large-scale scalable training.
1 Introduction
Diffusion models (DMs) [1, 2, 3, 4] have emerged as a class of powerful generative models and have recently exhibited high quality samples in wide variety of vision generation tasks such as image synthesis [5, 6, 7, 8, 9, 10, 11], video generation [12, 13. 14, 15, 16, 17, 18, 19] and 3D rendering and generation [20, 21, 22. 23, 24]. Relying on the advantage of iterative denoising and
generation, DMs have gained enormous attention from the community and have been significantly improved in terms of sampling procedure [25, 26, 27, 28], conditional guidance [29, 30, 31, 32], likelihood maximization [33, 34, 35, 36] and generalization ability [37, 38, 39, 10] in previous efforts.
However, current diffusion models still face scalability dilemma, which will play important role in determining whether could support scalable deep generative training on large-scale vision data and give rise to emergent abilities [40] similar large language models (LLMs) [41, 42). Representatively, the recent emergence of Sora [43] has pushed the intelligent emergence capabilities of generative models a climax by treating video models as world simulators. While unfortunately,
"Corresponding Author.
'Code available at
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sora is still a closed-soure source system and the mechanism for the intelligence emergence is still not very clear, but the scalable architecture must be one of the most critical technologies, according to the latest investigation [44] on its reverse engineering.
To alleviate this dilemma and spark further research in the open source community beyond the realms of well established U-Net and Transformers, and enable DMs to be trained in new scalable deep generative architectures, we propose unified and massively scalable Residnal-style Diffiusion Models framework (Neural-RDM for short) with a learnable gating residual mechanism, as shown in Figure 1.
The proposed Neural- RDMM framework aims to unify the current mainstream residual-style gen- erative architecture (e.g.. U-Ner or Transformer) and guide the emergence of brand new scalable network architectures with emergent capabili- ties. To achieve this goal, we first introduce a continuous time neural ordinary differential equation (ODE) to prove that the generative de- noising ability of the diffusion models closely related to the residual-style network structure, which almost reveals the essential reason why any network rich in residual structure can de- noise well: Residual-style neural units implicitly build ordinary differential equation that can well fit the reverse denoising process through ever-deepening neural units, thus supporting excellent generative abilities. Further, we also show that the gating-residual mechanism plays an important role in adaptively correcting the errors of network propagation and approximating the mean and variance of data, which avoids the adverse factors of network deepening. On this basis, we further present the theoretical advantages of the Neural-RDM in terms of stability and score prediction sensitivity when stacking this residual units to very long depth by introducing another deep stacked networks have the challenge of gradually losing sensitivity as the network progressively deepens, and our proposed gating weights have ad vantages in reverse suppression and error control.
dz, [2,,, + ort)dw
6₂
(.)
2
2t
8,+
dæ dt d Ji(2,, () A.o
2,
(o) MRS-Uiit block;
Neural Residual Denoising Medels
Figure I: Neural Residual-style Diffission Models framework with massively scalable gating-base based minimum residual stacking unit (mrs-unit).
ODE. From a dynamic perspective, it reveals that
Our proposed framework has several theoretical and practical contributions:
Unified residual denoising framework: We unify the residual-style diffusion networks (e.g., U-Net and Transformer) by introducing simple gating-residual mechanism and reveal the significance of the residual unit for effective denoising and generation from a brand new dynamics perspective.
Theoretically deep scalability: Thanks to the introduction of continuous-time ODE, we demonstrate that the dynamics equation expressed by deep residual networks possesses excellent dynamic con- sistency to the denoising probability flow ODE (PF-ODE) [45]. Based on this property, we achieve the simplest improvement to each mrs-unit by parameterizing learnable mean-variin variance scheduler, which avoids to manually design and theoretically support massively deep scalable training.
Adaptive stability maintenance and error sensitivity control: When the mrs-units are infinitely stacked to express the dynamics of an overall network Fa. the main technical difficulty is how to reduce the numerical errors caused by network propagation and ensure the stability of denoising. By introducing a sensitivity-rellted ODE in Sec. 2.3, we further demonstrate the theoretical advantages of the proposed gated residual networks in enabling stable denoising and effective sensitivity control. Qualitative and quantitative experimental results also consistently show their effectiveness.
2 Neural Residual Diffusion Models
We propose Neural-RDAt a simple yet meaningful change the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training of extremely deep networks. This framework is supported by three critical theories: 1) Kating-RRssduaa ODE (Sec. 2.1), which defines the dynamics of the minimum residual stacking unit (mrs-unit for short) that serves as the foundational denoising module, as shown in Figure (a). Based on this gating-residual mechanism, we then introduce 2) Menoising-Dynamtie stack the mrs-units to become a continuous time deep score prediction network Fo. Different from previous human-craftee crafted mean-variance schedulers (e.g., variance exploding scheduler SMLD [46]
ODE (Sec. 2.2) to further

---

30
2g
200
0g
56 (46)
9++
0g
((x0
[Rat-a)
Ag1
A2-
8)
69
(2.
9:
0xx
Fi
(R))
Av
x
[())
flaus
Att
9++
+6
@
Riv
& (3.0)
6,%
(i)
% x
[a)
f(aor-s-1
Xo)
4,1
S 0
Ait
26
96-1
1/-1
R2-1
R6-1
(RL
OL-1X
1156-5)
+86-3
Ap-1
#
(4)
Figure 2: Overview. (a) Flow-shaped residual stacking networks. (b) U-shaped residual stacking networks. (c) Our proposed unified and massively scalable residual stacking architecture (i.e., Neural- RDM) with learnable gating-residual mechanism. (d) Residual denoising process via Neural-RDM.
and variance preserving scheduler DDPM [2]), which may cause concerns about instability in denoising efficiency and quality, we introduce a parametric method to implicitly learn the mean and variance distribution, which lowers the threshold of manual design and enhances the generalization ability of models. Last but not least, to maintain the stability of the deep stacked networks and verify the sensitivity of each residual unit Fø, (-) to the network Fa. we stand the shoulders of the adjoint sensitivity method [47, 48] to propose 3)
ODE (Sec. 2.3), which means the sensitivity-related dynamics of each latent state z; from Fo, (-) to the deep network Fo. Through rigorous derivation, we prove that the parameterized gating weights have positive inhibitory effect on sensitivity decaying as network deepening. We will elaborate on them below.
2.1 Gating-Residus Mechanism
Let Fo. represents the minimum residual unit block, (Figure 1 (a)). f(-) denotes any feature mapper wrapped by Fe.. Instead of propagating the signal through each vanilla neural transformation aating-based based residual connection for the signal which relys on the learnable gating weights â and modulate the non-trivial transformation Fo, (z:) as,
2 = fz) directly, we introduce a gating-
2 z; + 0; (z.) + 3r-
For deep neural network Fo(-) with depth L, consider two common residual stacking fashions: Flow-shaped Stacking (FS) [49, 50] and U-shaped Stacking (US) [51, 52]. For the flwwbbsed deep stacking networks as shown in Figure 2 (a), each residual unit f(-) accepts the output z, of the previous mrs- mrs-unit input, and obtains a new hidden state 2;+1 through gating-residus connection,
2 21+1 2, [a:: (z.) + B.).
Note that Eq. 2 is refined form of Eq. in the case of flow-shaped stacking. In contrast, for the U-shaped deep stacking networks in Figure (b), each minimum residual unit contains two symmetrical branches, where the left branch receives the output z; of the previous mrs-unit's ' left. branch as input (called the read-in branch), and the right branch performs the critical nonlinear residual transformation for readout (called the read-out branch), which can be formally described as:
2; (2 + 10(0) + 2; + a Sog (221 + p! = 2: + OG Fo. 3.. (3)
read-in branch
read-out branch
Here Eq. is refined form of Eq. in the case of U-shaped stacking, â; and 3 collectively denotes the gating weights from the left and right branches, Fe, is the i-th minimum residual unit of the U-shaped networks, and "-," denotes the skip-connection for "2,+1 computed recursively via Fo To enable the networks to infinitely stacked, we introduce continuous-timg Gating Residual ordinary differential equation (ODE) to express the neural dynamics
225.-2-i". which is

---

Basdine
Neural
"Cleniis Hoskell, droadd Diveyy priceuu Mash Sil cooiaialyy
"A MApsiiial bong rede des aemaadddllg was, wos soss,ins jolage widylisu
'A haxabere Sony prine wits Mir bai boirdffiandy leddvoot stegle spyli parber
beaetifd gonko serseer adomed will When Bataiing
"A gread cochudest coerder in lopertastoalic style adonted
Figure 3: Compared with the latest baseline (SDXL-1.0 (7]). the samples produced by Neural- RDM (trained on JourneyDB [53]) exhibit exceptional quality, partieularly in terms of fidelity and consistency in the details of the subjects in adhering to the provided textual prompts.
of these two types of deep stacking networks (6 7, L + oc denotes the number of the mrs-umits),
Zitó 2,
8 2; 2, = å. Fo, (z,) + +B dz = âs (z.) + Bg.
(4)
dt
where & represents the gating weights, which can be independently trained or fine-tuned without considering the parameters 0 of the feature mapping network Fo(-) itself.
2.2 Denoising Dynamics Parameterization
The above-mentioned gating-residual mechanism is utilized to modulate mainstream deep stacking networks and unify them into residual-style massively scalable generative framework, as shown in Figure (c). Next, we further explore the essential relationship between residual neural networks and score-based generative denoising models from dynamic perspective.
First, inspired by the theory of continuous-time diffusion models [45, 54], the forward add-noising process can be expressed as dynamic process with stochastic differential equation (SDE) as,
dz, = (z,, t)dt + (t) = dz:
((z,,t) + (t) Ce, C E ((..
(5)
which deseribes data perturbation process controlled by a mean-variance scheduler composed of (=(z,t)) t) and (t) respectively. w, denotes the standard Brownian motion. Compared with the forward process, the core of the diffusion model is to utilize deep neural network (as deep and large as possible) for woorebbsed reverse prediction [46, 55J. A remarkable property of this SDE is the existence of reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45]), which retain the same marginal probability densities as the SDE (See Appendix. A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,
dzt ulz,, () 2
L (1)2 [ loggp(z))) á., Folze, 1) + Br
(6)
dl.
where V. pt Pr (z+) denotes the gradient of the log- lig-likelihood of p. (zc). which can be estimated by a score matching network Fo (z,,,) t). Here we re- ce-parmeteriie the PF-ODE by utilizing gated weights to replace the manually designed vern-variance scheduler, in which ái, and B.S denotes the time-dependent dynamics parameters, which is respectively parameterized to represent - (t)2² and u(z,, by our proposed gating-residual mechanism. Note that F=(-) is score estimation network composed of infinite mrs-units block; (i.c., Fe,), which enables massively scalable generative training on large-scale vision data, but also presents the challenge of numerical propagation errors.
2.3 Residual Sensitivity Control
To control the numerical errors in back propppaopagttion and achieve steadily and massively scalable training, stand the shoulders of the adjoint sensitivity method [47, 48] to introduce another

---

Class-to-Imadd
to-laage
Text-to-Image
Architecture
Method
Scalability
FID
sFID
ISt
FID
sFID
ISt
BigGAN-deep [56]
X
6.95
7.36
171.4
GAN
StyleGAN-XL [57]
X
2.30
4.02
265.12
ADM [58]
10.94
6.02
100.98
ADM-U
7.49
5.13
127.49
ADM-G
4.59
5.25
186.70
U-shaped
LDM-8 [30]
15.51
79.03
16.64
11.32
64.50
LDM-8-G
7.76
209.52
9.35
10.02
125.73
LDM-4
10.56
103.49
12.37
11.58
94.65
LDM-4-G
3.60
247.67
3.78
5.89
182.53
DiT-XL/2 [59]
9.62
6.85
121.50
8.53
5,47
144.26
F-shaped
DiT-XI/2-G
2.27
4.60
278.24
3.53
5,48
175.63
Latte-XL [60]
2.35
5.17
224.75
2.74
5.35
195.03
Neural-RDM-U (Ours)
3.47
5,08
256.55
2.25
4.36
235.35
Unified
Neural-RDM-FF (Ours)
2.12
3.75
295.32
2.46
5.65
206.32
Table 1: The main results for image generation on ImageNet [61] (Class-to-Image) and Jour- neyDB [53] (Text-to-lmage) with 256 x 256 image resolution. We highlight the best value in blue, and the second-best value in green The Scalability column indicates the scaling capability of the parameter scale and architecture.
Residual-Sen Sensitivity ODE, which is utilized to evaluate the sensitivity of each residual-state of the mrs-unit Fa, to the total loss C derived by score estimation network F- (the sensitivity denoted as 8, dz.' ô denotes an infinitesimal time interval) and can be formally described by the chain rule,
d. dc
dz
dz
8: = dz
8848
dzz+8
dz,
dz:
On the basis of Eq. 7, we next continue to discuss the dynamic equation of sensitivity changing with time t. First, considering the trivial transformation fo(-) without residual mechanism,
dzu+5 dz + {
144 fo(z,, t)dt.
(8)
We can rewrite Eq. based on Eq. 8 as:
8 = 8t+8 + 8 dz 1/the 1t
folze, t)dt).
(9)
The Residual-Sensitiive ODE under vanilla situation then can be derived,
0 ( fo(zz)dt)
d8:
81+6 81 lim
8145 ōz,
St Ofo(ze,()
lim 5
(10)
dt
5
8
dz.
According to the derived residual-sensitivvty equation in Eq. 10, we further use the Euler solver to obtain the sensitivity Sto of the starting state 21 network Fo(-) as,
dsi to
do f 0So(2,,)) 1) dt.
Sto
Stc
+
dt Stc
(11)
dt
8z,
Due to the non-negativity of the integral and the gradient 0fo(2..)) gradually decaying sensitivity sequence: St, > St.... ... St. Similarly, when defining
not equals to O, we can obtain
Oz.
parameter-sensitivity 89 df the same decaying results for se, can be obtained: do
80, s0L + 11 to d8e dt 88z
0.0(z1, dt.
S9 00
(12)
dt
se
5

---

Figure 4: Compared with the latest bascline (Latte-XL [60].. the sample videos from Sky SkyTime- lapse [62], [aichi-HD[63] and UCFI01 [64] all exhibit better frame quality, temporal consistency and coherence.
To alleviate this problem, and enable stable training in massively deep scalable architecture, introduce the following non-trivial solution with gating-residus residual transformation.
dž,-o dž,
(ro So(2.) + Br,d) dt.
(13)
dC Substitute Eq. 13 into Eq. 7 to obtain the corrected sensitivity ā: ds.
1+6
[oco fo(2,) + Be,0) dt).
s, S + 8146
(14)
The non-trivial Residual-Sensitinee ODE be derived as,
d3;
$et5
0fo(2..t)
lim 6
(ar, 82)
(Br,s 81).
(15)
dt
5
02.
Through the Euler solver, also obtain the sensitivity Sto of the starting state adjusted by the gating-residual weights,
1. to då.
= 8t2 - 11/ [ 8,) 8.)) 0fo(2..t) + (Br. 81)) dt.
8to
812 +
(16)
Jtc di
82.
Where as and Be adaptively modulate and update the sensitivity of each mrs-unit the final loss, which supports being trained through minimizing L, ||Fo(z,, t) V. log pc()))1 +
Σx |lass @fe(2..)) 02.
Br, 1|2 full-parameter training or model fine-tuning fashions.
Experiments
We present the main experimental settings in Sec. 3.1. To evaluate the generative performance of Neural-RDM, we compare it with state-of-the-art conditional//mcoodiiioo image synthesis and video generation in Sec. 3.2 and Sec. 3.3 respectively. We also visualize and analyze the effects of the proposed gated residuals and illustrate their advantages in enabling deep scalable training, which are presented in Sec. 3.4 and Sec. 3.5.
diffusion models for
3.1 Experimental Settings
Datasets. For image synthesis tasks, we train and evaluate the Class-to-Image generation models on the ImageNet [61] dataset and train and evaluate the Text-to-Image generation models on the

---

Frame Evaluation
None-to-Video
Class-to-Videe
Method
Scalability
SkyTimelapse
Taichi-HD (FVDL)
UCF-10) 101 (FVDL)
FIDI
IST
(FVDL)
MoCoGAN [71]
X
23.97
10.09
206.6
2886.9
MoCoGAN-HD [72]
X
7.12
23.39
164.1
128.1
1729.6
DIGAN [73]
X
19.10
23.16
83.11
156.7
1630.2
StyleGAN-V [70]
X
9.45
23.94
79.52
1431.0
MoSIGAN-V [74]
X
65.30
1380.3
PVDM [75]
29.76
60.55
75,48
540.2
1141.9
LVDM [12]
95.20
99.0
372.0
VideoGPT [76]
22.70
12.61
222.7
2880.6
Latte-XL [60]
5.02
68.53
59.82
159.60
477.97
Neural-RDM (Ours)
3.35
85.97
39.89
91.22
461.03
Table 2: The main results for video generation the Sky 3kyTimelapse [62], Taichi-HD [63] and
UCF-101 [64] with 256 256 resolution of each frame. We highlight the best value in blue and the second-best value in green
MSCOCO [65] and JourneyDB [53] datasets. All images are resized 256 x 256 resolution for training. For video generation tasks, we follow the previous works [12, 60] to train None-to-Videe (i.e., unconditional video generation) models on the SkyTimelapse [62] and Taichi [63] datasets, and train Class-to- Video models the UCF-101 [64] dataset. Moreover, we follow previous works [12, 60] to sample 16-frame video clips from these video datasets and then resize all frames to 256 x 256 resolution for training and evaluation.
Implementation details. We implement our Neural-RDMs into Neural-RDM-U (U-shaped) and Neural-RDM-F (Flow-shaped) two versions on top of the current state-of-the-ar diffusion models LDM [30] and Latte [60] for image generation, and further employ the Neural-RDM-F version for video generation. Specifically, we first load the corresponding pre-trained models and initialize gating parameters fa
B 0} of each layer, then perform full aall-parmeeter fine-tuning to implicitly learn the distribution of the data for acting parameterized mean-variana scheduler. During the training process, we adopt an explicit supervision strategy to enhance the sensitivity correction capabilities of and 3 for deep scalable training, where the explicitly supervised hyper-parameter y is set to 0.35. Eventually, we utilize the Adam optimizer with a constant learning rate x 10d for all models and exploit an exponential moving average (EMA) strategy to obtain and report all results.
Evaluation metries. Following the previous baselines [30, 58, 59, 60], we adopt Fréchet Inception Distance (FID) [66], sFID [67] and Inception Score (IS) [68] to evaluate the image generation quality and the video frame quality (except for sFID). Furthermore, we utilize Fréchet Video Distance (FVD) [69] metric similar with FID to evaluate the unconditional and conditional video generation quality. Among these metrics, FVD is closer human subjective judgment and thus better reflects the visual quality of the generated video content. Adhering to the evaluation guidelines proposed by StyleGAN-V [70], we calculate the FVD scores by analyzing 2048 generated video clips with each clip consists of 16 frames.
Baselines. We compare the proposed method with the recent state-of-the-an baselines, and categorize
them into three groups: 1) GAN-based. BigGAN-deep [56] and StyleGAN-XI [57] for image task, MoCoGAN [71], MoCoGAN-HD [72], DIGAN [73], StyleGAN-V [70] and MOSIGAN-V [74] for
video task. 2) U-shaped. ADM [58] and LDM [30] for image task, PVDM [75] and LVDM [12]
for video task. 3) F-shaped. DIT-XL/2 [59] and Latte-XL. XL [60] for image task, VideoGPT [76] and
Latte-XL [60] (with temporal attention learning) for video task.
3.2 Experiments on Image Synthesis with Deep Scalable Spatial Learning
For more objective comparison, we maintain approximately the same model size to perform class- conditional and text-conditional image generation experiments, which are shown in Table 1. From Table 1, it can be observed that our Reural-RDMr have obtained state-of-the-ar art results. Specifically,
7

---

300
Venirbo Verkord
250
400
200
RVD score
erare
2000
150
200
100
1000
50
900
10k sox 7dk Training berations
1.3 30k S0k 60k
therations
(a)
Figure 6: (a). (b), and (c) respectively illustrate the performance of the five residual structures variant models across the Sky Timelapsee [62]. Taichi-HD(63J, and UCF-101 [64].
the flow-based version (i.e., Heural-RDM- consistently outperforms all class-to-image baselines in all three image's generative benchmarks and meanwhile obtains relatively suboptimal results on another text-to-image evaluations. It worth noting that another Neural-RDM-U version have made up for this shortcoming and achieved optimal results, which may benefit from the more powerful semantic guidance abilities of the cross-attention layer built into U-Net. To more clearly present the actual effects of the gated residuals, we further perform qualitative comparative experiments, which are shown in Figure 3. Compared with the latest baseline (SDXL-1.0 [7], we can observe that the samples produced by Neural-RDM exhibit exceptional quality, particularly in terms of fidelity and consistency in the details of the subjects in adhering to the provided textual prompts, which consistently demonstrates the effectiveness of our proposed approach in deep scalable spatial learning.
3.3 Experiments Video Generation with Deep Scalable Temporal Learning
To further explore the effectiveness and specific contribu- tions of proposed gating-residus mechanism in temporal learning, we continue to perform the video generation evaluations, which are shown in Table 2. From Table 2, we find that our model (flow-shaped version) basically achieves the best results (except for the second-best sults class-to-video evaluation). Specifically, compare with Latte-XL, [60], Neural-RDM respectively achieves improvement of 33.3% and 42.8% FVD scores on Sky- Timelapse and Taichi-HD datasets, which hints the power- ful potential of flow-based deep residual networks in pro- moting generative emergent capabilities of video models. Furthermore, we exhibit a number of visual comparison results of the 16-frames video produced by Neural-RDM and baseline (Latte-XL [60]), shown in Figure 4. We can observe that some generated frames from the baseline partially exhibits poor quality and temporal inconsistency. Compare with the baseline, Neural-RDM maintains tem- poral coherence and consistency, resulting in smoother and more dynamic video frames, which further reflects the effectiveness of proposed method both quantitative and qualitative evaluations.
00
0.4 04
0.2
Depth
2500
Steps
10000
Figure 5: The sensitivity of a and / at different depths of the residual denoising network during the training process.
3.4 The Analyses of Gating Residual Sensitivity
To better illustrate the advantage of the gated residuals and understand the positive suppression effect for sensitivity attenuation as network deepening, we visualize the normalized sensitivity different depths of our Neural-RDi RDM during the training process, shown in Figure 5. From Figure 5, we can observe that a and 3 can adaptively modulate the sensitivity of each mrs-unit to correct denoising can also observe that at
process as network deepening, which is consistent with Eq. 16. Moreover, the beginning of training, the sensitivity scores are relatively low. As training advances, a and 3 are supervised to correct the sensitivity until obtaining relatively higher sensitivity scores.

---

3.5 Comparison Experiments of Gating Residual Variants and Deep Scalability
To explore the actual effects of different residual settings in deep training, first perform the
comparison experiments on 5 different residual variants: 1) Variant-O (Ours): 2i+1 = 2; fz) 3
2) Variant-1 (AdaLN [77]): Zi+1 24 + f(az; + B); 3) Variant-2: 2i-1 az; + f(x) + 3: 4)
Variant-3 (ResNet [78]): Z++1 = 2; + [zz); 5) Variant-4 (ReZero [79]): Zi+1 = 2, + az)).
We utilize Latte-XL as backbone to train each variant from scratch and then evaluate their performance for video generation. As depicted in Figure 6, as the number of training steps increases, almost all variants can converge effectively, but only Variant-0 (Our approach) achieves the best FVD scores. We speculate that it may be because this post-processing gating-residual setting maintains excellent dynamic consistency with the reverse denoising process, thus achieving better performance.
3000
RDM-4
R0M-12
2500
RDM-36
RDM-28
2000
1500
FVD
1000
500
Moreover, we further perform the deeep scalability exper- iments, which are shown in Figure 7. We can observe that as the depth of residual units increases, the performance of the model can be further improved, which illustrates the positive correlation between model performance and the depth of residual units and further highlights the deep scalability advantage of our Neural-DDM.
10k 20k 30k
40k
S0k
60k 70k aök 90k 100k
Training iteretions
Figure 7: The performance of Neural- RDM with different network depths on the UCF-101 dataset [64].
4 Related Work
Deep Residual Networks. Most common deep residual networks can be divided into two types of architectures: flow-shaped stacking (FS) and u-shaped stacking (US) architectures. As a milestone of flow-based deep residual networks, ResNet [78] has led the research of visual understanding tasks [80]. In fact, the pratices [81, 82] and theories [83, 84, 85] that introducing the highway connections [86, 87] have been studied for long time and have demonstrated excellent advantages in dealing with vanishing/exploding gradients and numerical propagation errors in deep stacked networks. Different from ResNet, U-Net [51] is leader of u-shaped networks and almost dominated diffusion based generative models [2. 30J. Though achieving remarkable success, both types of CNN- based models still face concerns about training efficiency. Recent years, Transformer [49] and VIT [50] have emerged as new state-of-the-an backbones in computer vision and multimodal [88, 89, 90, 91] and have also gained prominence in various diffusion models. Among them, DiT [59] and U-ViT [52] are two representative works by respectively adopting flow-shaped and u-shaped residual stacking fashions, which have enabled many studies on deep generative models [60, 92, 93, 94, 95]. In this work, we unify the above two types of residual stacking architectures from a dynamic perspective and propose unified and deep scalable neural residual framework with a same gating-residus ODE.
Diffusion Models. Recent years has witnessed the remarkable success of diffusion models [2, 3, 4], due to their impressive generative capabilities. Previous efforts mainly focus on sampling procedure [25, 26, 27, 28). conditional guidance [31, 32, 96, 97, 98], likelihood maximization [33, 34, 35, 36] and generalization ability [37, 99, 39, 10] and have gained enormous attention. Recently, major research topic on diffusion models is scalability. DiT [59] is one of the most representative models by exploiting a scalable Transformer to train latent diffusion models for image generation. Latte [60] stands the shoulders of DiT to further perform temporal learning for video generation. However, both Latte and DiT adopt the residual structure of Transformer by default and utilize S-AdaLN to incorporate guidance information, they generally lack: 1) attention to the residual structure and 2) study the dynamic nature of the deep generative models, and 3) ignore the error propagation issues from deeper networks and therefore still limited by the bottleneck of massively scalable training.
Overall, we practically unify u-shaped and flow-shaped stacking networks and to propose a unified and deep scalable neural residual diffusion model framework. Moreover, we theoretically parameterize the previous ddsinnde variance scheduler and demonstrate excellent dynamics consistency.

---

Conclusion
In this paper, we have presented Neural-RDM, a simple yet meaningful change to the architecture of deep generative networks that facilitates effective denoising, dynamical isometry and enables the stable training extremely deep networks. Further, we have explored the nature of two common types of neural networks that enable effective denoising estimation. On this basis, we introduce a parametric method to replace previous human- mennanddesigedd mean-variame schedulers into a series of learnable gating-residuat weights. Experimental results various generative tasks show that Neural-RDM obtains the best results, and extensive experiments also demonstrate the advantages in improving the fidelity, consistency of generated content and supporting large-scale scalable training.
Acknowledgments and Disclosure of Funding
This work is supported by the National Science and Technology Major Project (2023ZD0121403), Na- tional Natural Science Foundation of China (No. 62406161), China Postdoctoral Science Foundation (No. 2023M741950), and the Postdoctoral Fellowship Program of CPSF (No. G2B20230347).
References
[1] Zhiyuan Ma, Yuzhu Zhang, Guoli Jia, Liangliang Zhao, Yichao Ma. Mingjie Ma, Gaofeng Liu, Kaiyan Zhang, Jianjun Li, and Bowen Zhou. Efficient diffusion models: A comprehensive survey from principles to practices. arXiv preprint arXiv:2410.11795 2024.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in NeurlPS, 33:6840-6851, 2020.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171, 2021.
[4] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic ext-to--imaee diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 334444 2022.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical rext-conditional image generation with clip latents. arXiv preprint arXiv:2204.061255 2022.
[7] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955. 2023.
[8] Zhiyuan Ma, Guoli Jia, Biqing Qi, and Bowen Zhou Safe-sd: diffusion with text prompt trigger for invisible generative watermarking. In ACM Multimedia 2024.
Safe and traceable stable
[9] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[10] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22510 10, 2023.
[11] Zhiyuan Ma, Guoli Jia, and Bowen Zhou. Adapedit: Spatio-temporal guided adaptive editing algorithm for text-based continuity-sensine image editing. Proceedings of the AAA! Conference on Artificial Intelligence, volume 38, pages 4154-4161, 2024.
10

---

[12] Yingqing He. Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.1322// 2022.
[13] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.10118 2022.
[14] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope eex-to-vvieo technical report. arXiv preprint arXiv:2308..05. 06571, 2023.
[15] Andreas Blattmann,, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arxivv:3 1...2.111111111 151227 2023.
[16] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.00 109047, 2024.
[17] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consisteney- aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 200050, 2023.
[18] Weimin Wang. Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2. Multi-stage high-aesthetic video generation. arXiv preprint arXiv:2401..m 044468, 2024.
[19] Omer Bar- Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12940 2024.
[20] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
[21] Yichun Shi, Peng Wang. Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations, 2023.
[22] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[23] Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305. 18766, 2023.
[24] Vikram Voleti, Chun-Han Yao. Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthe- sis and generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. 12008, 2024.
[25] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pages 42390-42402 PMLR, 2023.
[26] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
[27] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, pages 32211-221111 1-32252, 2023.
11

---

[28] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. 2023.
[29] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12.741111,.0041, 2021.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[31] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In The Twelfth International Conference Learning Representations, 2023.
[32] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content- guiddd video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[33] Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion models. arXiv preprint arXiv:2205. /tIi::2055/3699, 2022.
[34] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning, pages 4672-4712 PMLR, 2023.
[35] Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for cosre-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pages 14429-14460. PMLR, 2022.
[36] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021.
[37] Yuhan Li, Yishun Dou, Xuanhong Chen, Bingbing Ni, Yilin Sun, Yutian Liu, and Fuzhen Wang. Generalized deep 3d shape prior via part-discretized diffusion process. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16784-16794, 2023.
[38] Zhiyuan Ma. Jianjun Li, Bowen Zhou, et Lmd: Faster image reconstruction with latent masking diffusion. arXiv preprint arXiv:2312.079711 2023.
[39] Chong Mou, Xintao Wang, Liangbin Xie. Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2296-43044 2024.
[40] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and Yasmine et al Babaei. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.
[42] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiiv:3303.08774, 2023.
[43] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Erie Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
12

---

[44] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen. Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, al. Sora: A review on background. technology. limitations, and opportunities of large vision models. arXiv preprint 11Xi::2402.17177,, 2024.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representaatonss 2020.
[46] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[47] ML Chambers. The mathematical theory optimal processes. Journal of the Operational Research Society, 16(4))493-494 1965.
[48] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241, 2015.
[52] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a vit backbone for score-based diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[53] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
[54] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[55] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574 584. PMLR, 2020.
[56] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
[57] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[58] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in NeurIPS, 34:8780-8787 2021.
[59] William Peebles and Saining Xie. Scalable diffusion models with transformers, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[60] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li. Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprins arXiv:2401. 000048, 2024.
[61] Alex Krizhevsky, Tlya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
13

---

[62] Wei Xiong, Wenhan Luo. Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2364-22622222 2373, 2018
[63] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
[64] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint arXiv: lttl::2222.000 222022 2012.
[65] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[66] Mandi Luo, Jie Cao, Xin Ma, Xiaoyu Zhang, and Ran He. Fa-gan: Face augmentation gan for deformation-invaariantt face recognition. IEEE Transactions on Information Forensics and Security, (6:2341-2355 2021.
[67] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia. Generating images with sparse representations. In International Conference on Machine Learning, pages 7958-7968. PMLR, 2021.
[68] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830-2839, 2017.
[69] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01711 2018.
[70] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3626-3636, 2022.
[71] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018
Decomposing
[72] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In International Conference on Learning Representations, 2020.
[73] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Way Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2021.
[74] Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 5652-5661, 2023.
[75] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18456-18466, 2023.
[76] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va-vae and transformers. arXiv preprint arXiv:2 10157, 2021.
[77] Yunhui Guo, Chaofeng Wang, Stella X Yu, Frank McKenna, and Kincho H Law. AdaIn: vision transformer for multidomain learning and predisaster building information extraction from images. Journal of Computing in Civil Engineering, 3605),04 2022.
14

---

[78] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings ofe IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[79] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, Rezero all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pages 1352-1361. PMLR, 2021.
[80] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9,
2015.
[81] Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear trans- formations in perceptrons. In Artificial intelligence and statistics, pages 924-932. PMLR, 2012.
[82] Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.
[83] Nicol Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical repor//IDSIA, 98, 1998.
[84] Nicol N Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade, pages 207-226. Springer, 2002.
[85] Tommi Vatanen, Tapani Raiko, Harri Valpola, and Yann LeCun. Pushing stochastic gradient towards second-orde methods backpropagation learning with transformations in nonlineari- ties. In Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings. Part I 20, pages 442 449. Springer, 2013.
[86] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint
2015.
[87] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. Advances in neural information processing systems, 28, 2015.
[88] Zhiyuan Ma, Jianjun Li, Guohui Li, and Kaiyan Huang. Cmal: A novel roos--modal associa- tive learning framework for vision- languge pre-training In Proceedings of the 30th ACM International Conference on Multimedia, pages 4515-4524, 2022.
[89] Zhiyuan Ma, Jianjun Li, Guohui Li, and Yongjing Cheng. Unitranser: unified transformer semantic representation framework for multimodal tosk-orie dialog system. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 103-114, 2022.
[90] Zhiyuan Ma. Zhihuan Yu, Jianjun Li, and Guohui Li. Hybridprompt: bridging language models and human priors in prompt tuning for visual question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 13371-13379, 2023.
[91] Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang. Bowen Zhou, and Jie Zhou Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18733-18741, 2024.
[92] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.123777 2024.
[93] Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint 2xXv:2405500423. 2024.
[94] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Vanden-Fijnden Fijnden, and Saining Xie. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. arXiv preprint arXiv: 040.0087400 2024.
15

---

[95] Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok. Ping Luo, Huchuan Lu. and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2023.
[96] Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Yeung Shum, Wei Liu, et al. Follow-your-click Open-domain regional image animation via short prompts. arXiv preprint 0rXiv:2403.08266 2024.
[97] Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided texi-to-video generation using pose-free videos. arXiv preprini arXiv:2304.01 186, 2023.
[98] Yue Ma, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, al. Follow-your-eme emoji: fine-controllabe and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01990 22900000090, 2024.
[99] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pages 10696-10706, 2022.
[100] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883, 2021.
[101] Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-planck equations through gradient-log-denstty estimation. Entropy, 22(8):802, 2020.
[102] Bernt Oksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
16

---

A Appendix
A.1 Theoretical Interpretations
In this section, we provide mathematical intuitions for our Neural-RDMs.
Continuous-time Residual Networks. For a deep neural network Fo(.) with depth L, let Fo, represents the minimum residual unit block, (Figure I (a)). Instead of propagating the signal through vanilla neural transformation å = fo(z), we introduce a gating-based skip-connection for the signal 2, which relys on the gating weights á and 3 to modulate the non-trivial transformation Fo(z) as,
2 = x + á Fo(z) + 3.
(17)
In the case of continuous time, this dynamic equation describing the change process of the signal is called the rating-residual ODE:
dz: ào. So.
(18)
dt
Diffusion Probability Models. The diffusion probability models modeled as: 1) deterministic forward noising process g ( [x, = N(x;; /0; (1 (1- (1-0a)) T) from the original image No to a pure-Gaussian distribution Xy (O, I). which can be formulated in an accumulated form:
X;
x + -1- E~ (0,1
E
(19)
2) a iteratively predictable reverse denoising process po (x (xe) (xx (x f), Eg (x, t)). which can be trained in simplied denoising objective Lsimple by merging μ4 and Eg into predicting noise Co.
Simmle = Exp. (x.. e
(20)
where t U[,, is time parameters, u()) denotes uniform distribution. Moreover, in Stable Diffusion [30], the image is compressed into latent variable by encoder & for more efficient
training, i.e., Z, = Exx),, thus this preliminary objective usually defined as making Eg Ez 1) as
close to E ~ (0, as possible.
Reverse Denoising ODE. remarkable property the SDE (Eq. 19) the existence of a reverse ODE (also dubbed as the Probability Flow (PF) ODE by [45]), which retains the same marginal probability densities as the SDE (See Appendix A.2 for detailed proof) and could effectively guide the dynamics of the reverse denoising, it can be formally described as,
dzt = u(z,,t) t) 2
1 (t)²
[V log log))(z)) = âi, Felze, t) + Sr,00
(21)
dt
where V. (z.,) denotes the gradient of the log-likelihood of pt(2)), which can be estimated by a score matching network Fo t2.
Dynamics Consistency. Refer to Eq. 18 and Eq. 21, we define this dynamic consistency as: For any time-dependent signal z,. the different dynamics systems describe it with the same motion path (or the same change rate of data distribution). Note that in Eq. 21, we achieve this by using re-pa re-parameterized approach.
Latent Space Projection. The latent space projection is proposed by [30] to compress the input images Xy into perceptual nigh-dimensiioal space to obtain 2, by leveraging pretrained VQ-VAE model [100]. The VQ-VAE is also used by our Neural-RDM, decoder G. The mathematical definition is: Given an input image
it consists of an encoder E and a H W the VQ-VAE first compress the image x into a latent variable 2 by encoder ε, i.e., 2 = E(x) and é E kxxwxx where h and w respectively denote scaled height and width (scaled factor ʃ = H/h W/v = 8), and d is the dimensionality of the compressed latent variable. After going through the diffusion step described in Eq. 5 and Eq. 6, the latent variable 2 updated and finally reconstructed into by decoder G.
- = G.
(22)
where LDM(.) represents the latent diffusion models (including Unet-based or seasformer-basedd) 0 denotes the parameters of LDM, and n denotes the parameters of the VQVAE that are frozen train our Neural-RDM models.
17

---

A.2 Additional Proofs
Motivated by [101], we follow [45] to give proof: A remarkable property of the SDE (Eq. 5) the existence of reverse ODE (PF-ODE [45]), which retain the same marginal probability densities as the SDE. We consider the SDE in Eq. 5, which possesses the following form:
dz (1,,,, ()dt + (z l)dw,,
(23)
where (t(,tt) : Rd + H and (-,t)) R + Raxd The marginal probability density pr evolves according to Kolmogorov's forward equation [102]
Opt(z)
0
8
|ps (z=))
Ow t)x
dzi
O2,02y
d
8
(z)))
aa (z (2 t)0 Do
dz
dz
Zzy
(24)
Since the sub-part of Eq. 24 can be written in the following form:
a ou (2)5 ()p((z))
0zg
j=1
d
d
Pr(ze) Tah omk (z, t)) ((((z))))))))))))))))))))))))))) +
Fik (ze, ()asM(Ze, (z,) 0zj
log Pt (zt)
1=1
j=1k=1
(z, V (z o T (z pa
1)or
(zz, Vz, (z.))(z.).
(25)
Thus we can obtain:
(z((
0 jui (z)(
1
à
oa (2 ()p
2
θzj
i Jui(Ze, ())(
0z,
1=1
1
i
p((zz)
-o (( p)()))))(( (z logpe(z))
0z,
0
(1(zztt)p((z)
0z,
(0( 1000 + (2t, (2,, (10((()(((()))
8 [ū,(Zz, ())(
(26)
Oz,
1=1
where we define (-) as:
ju(z,,t) = ((zz t) V [a t) T t)] 2 (z (z,, V 2. lo,).
(27)
2
Combining Eq. 26 and Eq. 27, we can conclude that Equation Eq. 26 still describes Kolmogorov's forward process but with (z t) == as:
dz: fz( tydt (2),t)dwe, ā a 0.
(28)
Which proves that it is actually ODE after reverse transformation (i()):
dz, t)dz,
(29)
which is essentially the same with our demonstrate the existence of the reverse ODE and the practicality of parameterizing the mean-variance scheduler from the reverse ODE.
given by Eq. 6. Therefore, we
18

---

A.3 More Generated Images
Unbile Lastietan Tourenne
Opelics Phimuugss
"A landscape featuring a wooden bridge over serene lake with majestic mountain the background."
"A cartoon-style watercolor cover illustration featuring vibrant pink and cream hydrangeas in Disney- inspired setting, complemented by typography."
"A speckled headscary, a swamp adder, and flag of India depicted in isometric illustration."
"A castle situated the mountains with an array of very high thin towers adorned with numerous arrowslits."
"The King of Pentacles stands in regal pose, surrounded by earthly riches, while a mysterious UFO hovers above, adding element of otherworldly intrigue."
"A hyper-realistic portrait of 7-year-old old English girl with mismatched eyes, blonde curly hair adorned with flowers, holding flute, radiating pure joy."
"woman captain wearing cocked hat stands ship, gazing in and fear as huge Cthulhu emerges from the water."
"A cyberpunk man with silver skin wearing helmet featuring large glass visor; holding rocket launcher in futuristic setting, depicted with hyper-realistie...
Figure The samples produced by Neural-RDe -Dral--DMM (trained on JourneyD [53]).
A.4 Limitations
Limitation Discussion. Although significant improvements have been achieved, Neural-RDM still has some limitations, the most important of which is that the gated residual mechanism only inhibits rather than completely avoids the sensitivity decrease and numerical propagation errors caused by the deepening of the network. If want completely avoid it, may have give up stacking-based deep network architectures, but that will lead to a significant reduction in performance. Therefore. method chooses continue deepen the stacking of the network and suppress propagation as much as possible in the trade-off between the two.
A.5
Social Impact
Potential Social Implications. We believe that Neural-RDM will bring new thinking about deep network architectures to the generative community, and hopefully promote the generative emergence capabilities of vision generation models in the open source community. In addition, hope that more researchers can follow the powerful capabilities of residual denoising to build brand new scalable network architectures beyond the realms of well established U-Net and Transformers.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe that the main claims made the abstract and introduction accurately reflect the contributions and scope of the paper, refer to theoretical proof in Sec. 2.1- 2.3 and experimental proof in Sec. 3.2- 3.5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer Sec. A.4 for a discussion of limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: We have provided all detailed assumptions and theoretical proofs in the main content and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We are convinced that we have achieved this and promise to disclose all code details after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: are convinced that we have achieved this and promise to disclose all code details after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
hetbbmmissiooooicyyy...........p for more details. httee//uidee//oeeuueeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.aa cttpssscccppccccccccpcccccccccct c/guidgussssseeee /eeo//eoebubiiic ssi iasionPoiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We convinced that have achieved this, please refer Sec. 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We have presented the results of statistical significance in the main quantitative
results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the GPU power used the experiments in Sec. 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeerpp.eeeeeeeeee http/pullc/ttthiceiiiiieeee
Answer: [Yes]
Justification: Yes, comply with all NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have discussed social impacts Sec. A.5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we discuss it in Social Impact but in reality approach is largely irrelevant to safety precautions.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, have followed these license specifications and accurately stated contributions from previous work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi twwode..coo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Pk22sb9BFu", "reviewer": "Reviewer_zq4Z", "summary": "This paper addresses the problem of the numerical propagation errors of progressively deeper stacked neural networks for generative models. It proposes Neural Residual Diffusion Models (Neural-RDM), which introduced a series of learnable gated residual parameters to the common architectures of deep generative networks. Evaluation of the proposed method on image generation and video generation benchmarks demonstrate its superior performance over state-of-the-art methods.", "questions": "See Weakness.", "limitations": "The authors have discussed clearly about the potential limitations and social impacts in A.4 and A.5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple and well-motivated. \n- The experiments on generation benchmarks are comprehensive, and the great results demonstrate the efficacy of the proposed method.\n- The paper is clearly written.", "weaknesses": "- It is not clear what the specific definition of the \"Scalability\" columns in Table 1 and Table 2 is. Could you clarify the metrics represented by the cross, tick, and double tick symbols? \n- Since the paper is targeting on addressing the scalability of deep generative models, it would be better if the scalability experiments could be enhanced. Figure 7 shows that the performance of the proposed method can be improved as the depth of the network increases. However, how does its scalability compare with baseline architectures? Also, does it show similar scalability on other tasks and datasets?\n- As the authors have discussed in the limitation section, the numerical propagation errors caused by stacking network layers cannot be completely avoided. Therefore, it would be more accurate not to claim \"enabling the networks to be *infinitely* stacked\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Definitions and clarifications on scalability metrics** \\\n**R1:**  Thanks for the helpful comment. We first want to clarify that the \"Scalability\" columns in Tab.1 & Tab.2 indicates the scaling capability (i.e., parameter scale and architecture stackability) of the evaluated models, meanwhile ensuring that errors do not accumulate as the network deepens, to facilitate the deep scalable training of the generative models (i.e., scaling law). Moreover, we want to re-clarify the meaning of the three metrics (i.e., _**cross**_, _**tick**_ and **_double tick_**): 1) GAN: Non-scalability (_**cross**_), limited by the instability of deep network training between the generator and the discriminator; 2) U/F-shaped: Low-scalability (_**tick**_), supporting a certain deep-scalable training, but error accumulation will occur as the network deepens during gradient update; 3) Ours: High-scalability (**_double tick_**), supporting deep-scalable training with the integration of error correction mechanism. We will make these points clearer in the final revision and hope the above clarifications can help understand our work better.\n\n**Q2: Supplements of scalability experiments**\\\n**R2:** Thanks for this valuable suggestion, we have supplemented the scalability experiments versus the baseline (i.e., **_Latte-XL_**) in following table, benchmarking on more tasks (**_T2I_**, **_N2V_** & **_C2V_**) and more datasets (**_JourneyDB_**, **_Taichi-HD_** & **_UCF-101_**). As shown in the table, in the T2I task, as the network depth increases from 12 to 32, the IS-score of our Neural-RDM shows a significant improvement compared to the baseline, which implys the superiority of the error control mechanism in supporting deep scalable training. A similar trend can be observed in the N2V and C2V tasks, which consistently validate the deep scalability advantage of our Neural-RDM. We will supplement the above results to the final version to better illustrate the performance of the proposed method.\n\n| Depths | T2I in JourneyDB (IS$\\uparrow$)  |  | N2V in Taichi-HD (FVD$\\downarrow$) |  | C2V in UCF-101 (FVD$\\downarrow$) |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | Baseline | Neural-RDM (Ours) | Baseline | Neural-RDM (Ours) | Baseline | Neural-RDM (Ours) |\n| 12 | **103.18** | 94.20 | 236.83 | **191.04** | 923.38 | **896.33** |\n| 24 | 134.62 | **167.36** | 171.91 | **128.91** | 681.31 | **625.46** |\n| 28 | 195.03 | **206.32** | 159.60 | **91.22** | 477.95 | **461.01** |\n| 32 | 211.97 | **231.06** | 121.89 | **59.71** | 376.46 | **338.08** |\n\n**Q3: Regarding the imprecise claim.** \\\n**R3:**  Thanks for this very thoughtful comment.  We will refine this claim to make it more precise in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "koJj6ucysw", "reviewer": "Reviewer_jEd7", "summary": "This work presents a framework for visual generative diffusion models, aiming at addressing the challenges associated with deep stacked networks in terms of numerical propagation errors and scalability.", "questions": "Listed in weakness.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation\n\n2. The authors provide a  theoretical analysis for their approach, including the use of continuous-time neural ODEs to demonstrate the relationship between residual-style network structures and generative denoising abilities.\n\n3. Sufficient experiments: The paper is supported by extensive experimental evidence, which show that the proposed models achieve state-of-the-art performance on various generative tasks, including image and video generation.", "weaknesses": "1. While the introduction of gated residual parameters is innovative, it may also add complexity to the model, which could potentially make it harder to train or less intuitive for practitioners to understand.\n\n2. How about the computation requirements for this scalability model?\n\n3. Model parameters , GFLOPs should be mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** **Complexity of the model after introducing gated residual parameters**\\\n**R1:** We understand the reviewer's concern, but that's actually unnecessary. Firstly, we want to highlight the introduction of these gated residual parameters is a _**simple yet meaningful**_ change to the common architecture of deep generative networks (e.g., _**U-Net**_ or _**Transformer**_), which only requires adding learnable scaling and bias tensors in each vanilla residual connection layer to fit the mean-variance scheduling dynamics of the proposed _**Denoising-Dynamics ODE**_. Secondly, we designed an error correction loss and integrated it into the conventional score matching loss at a smaller proportion (i.e., $\\gamma=0.35$), balancing the model's denoising ability and error correction performance while ensuring that the model is easy to train.\n\n**Q2: Computation requirements**\\\n**R2:** Neural-RDM focuses on helping existing or brand-new generative backbone networks support large-scale and deep-scalable training, which supports both full parameter training and partial parameter fine-tuning. In our experiments, _**2/4\\*A100 80G GPUs**_ are respectively adopted to fine-tune the learnable gated parameters for image tasks (C2I & T2I) and video tasks (N2V & C2V) and _**8\\*A100 80G GPUs**_ for the full parameter training from scratch.\n\n**Q3: Model parameters and GFLOPs**\\\n**R3:** Thanks for this helpful suggestion, we have added the details of model parameters and GFLOPs in the table below, which will be included in the final version. \n\n| Task | Method | Parameters (M) | FLOPs (G) |\n| --- | --- | --- | --- |\n| Image Generation | Baseline (LDM-4) |  264.00  |  2,759.97   |\n|  | Neural-RDM-U (Ours) | 293.11 (**+11.02%**) | 2,931.91 (**+6.63%**) |\n|  | Baseline (Latte-XL/2) | 673.68 | 428.56 |\n|  | Neural-RDM-F (Ours) | 748.06 (**+11.04%**) | 457.26 (**+6.67%**) |\n| Video Generation | Baseline (Latte-XL/2) | 673.68 | 5,572.69 |\n|  | Neural-RDM (Ours) | 748.06 (**+11.04%**) | 5,603.80 (**+0.55%**) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8S92mPc50G", "reviewer": "Reviewer_6rnE", "summary": "The paper describes the gradual change in the state z_t of a diffusion-based neural network as an ODE, and links the flow-architecture and UNet-architecture to the parameters of the ODE. Then, it describes a way to better train this network based on the dynamics of this ODE, and proposes an alternative loss function. It then performs experiments on image and video generation with this loss function, and several variants of the architecture.", "questions": "Is the final loss function as stated in lines 162-163? In which case, does the loss apply to each layer separately? Do you sum the losses per layer?\n\nHow was the value of gamma as 0.35 decided?\n\nIt is unclear whether the network is used to completely or partially denoise the input. It seems as though it is assumed that the network is completely denoising from noise to data (instead of transforming partially noised to slightly less noised). Because the dynamics of F mention L -> infinite. If, however, F is assumed to be the full sampling procedure of multiple denoising steps, then Figure 2 doesn't make sense. In brief, it is unclear how the full dynamics are connected with each forward pass of the network. This is especially confusing from the perspective of the UNet.\n\nThe difference between Variant 0 and Variant 4 seems to be just a scalar beta value. Why does this 1 parameter affect performance so much?", "limitations": "The paper states where the model works better than prior works and where it doesn't.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper does a great job of explaining how the dynamics of z_t through the network can be unified across the UNet and Flow architectures. It then derives a loss function based on the ODE of those dynamics. The derivation itself is involved and highly useful. The paper also performs several experiments on multiple modalities to validate the proposed hypothesis of the new loss function / perspective. In particular, there are experiments in both image and video generation, each being quite strenuous in terms of academic effort and computational requirements.\n\n---\nUpdate: I increased my score after the authors' response.", "weaknesses": "Some parts of the paper are not clearly written. In Figure 2, the caption does not sufficiently explain the figure. In particular, (a) and (c) look the same, the difference needs to be elaborated on (corresponding to the main text). What dotted lines mean, what coloured lines mean, etc. has to be explained, in Figures 1 and 2.\n\nSections 2.2 and 2.3 need to be written more clearly. In particular, the dimensions of the involved variables need to be mentioned. It is unclear whether the network is used to completely or partially denoise the input. \n\nMost image-based experiments seem to be finetuned experiments, which are dependent on the biases of the original network used. Section 3.5 contains experiments on models trained from scratch, which is great. It is unclear though why Variant 0 performs so much better than Variant 4, since they only have a scalar \"beta\" difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Explanation of some details in Fig.1 & 2**\\\n**R1:** Thanks. We will explain these details below: \n1) In Fig.2, (a), (b) and (c) respectively represent **Flow-shaped Networks** (e.g., Transformer), **U-shaped Networks** (e.g., U-Net) and **Unified Stacking Network (our Neural-RDM)**. As depicted in line 98-118, (a) and (b) are both special cases of (c), i.e., $f_{\\theta_i}$ in (a) corresponds to $F_{\\theta_i}$in (c), whereas $f_{\\theta_i^{(l)}}$ and $f_{\\theta_i^{(r)}}$ in (b) jointly correspond to $F_{\\theta_i}$ in (c).  Note that (c) covers both data streams mentioned earlier with a unified dynamics formula, which is a significant difference from (a).\n2) The dotted lines in Fig.1 & 2 indicate the residual connections.\n3) The coloured arrows (i.e., blue arrows) denote the data streams of the stacked networks.\n\nWe will carefully supplement these details in the final version.\n\n**Q2: Dimensions of the variables**\\\n**R2:** We first want to clarify that this paper is actually a foundational work applicable to a variety of DMs' deep-scalable training, e.g., image, video and others, thus we do not specifically emphasize the dimensions of the variables, but they are actually easy to specify, e.g., $\\boldsymbol{z}\\in\\mathbb{R}^{B \\times F \\times C \\times H \\times W}$($B$ is bach size, $F$ is video frames, $C$ is channels, $H$ and $W$ denote height and width respectively), $\\alpha\\in\\mathbb{R}^{L \\times D}$ and $\\beta\\in\\mathbb{R}^{L  \\times D}$ ($L$ is the number of network layers, $D$ is dimension of the hidden layer) in video tasks. Note that the image tasks are slightly different from the videos in frame dimensions. If necessary, we'll add them in the final version.\n\n**Q3: Whether to denoise partially or completely**\\\n**R3:** We suspect this may be a serious misunderstanding, and we want to clarify that although the Neural-RDM network ($F_{\\theta}$) is composed of a series of mrs-unit $F_{\\theta_i}$, but it is still performed for single-step denoising, i.e., $\\boldsymbol{z}(t)\\rightarrow\\boldsymbol{z}(t-1)$ for time t, as illustrated in Fig.2(d). Note that this point is completely consistent with the previous baseline LDM, DiT and Latte. Moreover, we want to further highlight: we introduce continuous-time **_Residual-Sensitivity ODE_** into the neural networks of depth L, **NOT** into the diffusion process (e.g., $\\boldsymbol{z}(0)\\rightarrow\\boldsymbol{z}(T)$). Moreover, for U-shaped stacking networks, we want to specify $\\mathcal{F} _ {\\theta _ {i}}$ stands for both $f_{\\theta_i^{(l)}}$ and $f_{\\theta_i^{(r)}}$ in the i-th residual unit, i.e., the skip-connection $\\boldsymbol{z} _ {i+1}\\rightarrow\\boldsymbol{z}_{2L-2-i}$ in UNet or U-ViT. The goal is to build a unified **_Denoising-Dynamics ODE_** under consistent dynamics formula, so as to facilitate the improvement and optimization of all stacked networks (including U-shaped and Flow-shaped) for better deep scalable training.\nWe will make these points clearer in the final revision and hope the above clarifications can help understand our work better.\n\n**Q4: Explanation for the difference in performance between variant 0 and variant 4**\\\n**R4:** We understand the reviewer's concern, but that's actually unnecessary. Firstly, we want to clarify that the $\\hat{\\beta} _ {t,\\phi}$ is not a simple scalar, but rather a series of learnable time-dependent parameters that are used to fit the mean-related scheduler (i.e., $\\mu(z_t,t)$) in our **_Denoising-Dynamics ODE_**, thus the adjustment of $\\hat{\\beta} _ {t,\\phi}$ directly impacts the denoising performance (i.e. the FVD score). More broadly, the $\\hat{\\alpha} _ {t,\\phi}$ and $\\hat{\\beta} _ {t,\\phi}$ respectively parameterized the variance- and mean-related scheduler $-\\frac{1}{2}\\sigma(t)^2$ and $\\mu(\\boldsymbol{z} _ t,t)$, which replaces human design in previous works as,\n\n$\\frac{d\\boldsymbol{z} _ t}{dt} = \\mu(\\boldsymbol{z} _ t,t)-\\frac{1}{2}\\sigma(t)^2\\cdot\\Big[\\nabla _ {z} \\log p_t(\\boldsymbol{z}_t)\\Big] = \\hat{\\alpha} _ {t,\\phi}\\cdot\\mathcal{F} _ \\theta(\\boldsymbol{z} _ t,t) + \\hat{\\beta} _ {t,\\phi}.$\n\nWhereas variant 4 contains only a scaling tensor $\\hat{\\alpha}_{t,\\phi}$, making it difficult to fit the **_Denoising-Dynamics ODE_**, thus resulting in worse results than variant 0.\n\n**Q5: Details of the Loss Function**\\\n**R5:** Yes, the final loss function with sensitivity control is as stated in lines 162-163, which can be expressed as follows,\n\n$\\mathcal{L} _ s = ||\\mathcal{F} _ \\theta(\\boldsymbol{z} _ {t}, t) - \\nabla_z \\log p _ t(\\boldsymbol{z} _ t)||^2_2 + \\gamma \\cdot \\sum _ {L}|| \\hat{\\alpha}_ {t,\\phi} \\cdot\\frac{\\partial f _ \\theta(\\hat{\\boldsymbol{z}} _ {t},t)}{\\partial\\hat{\\boldsymbol{z}} _ {t}}-\\hat{\\beta}_{t,\\phi}||^2_2.$\n\nThe total loss is obtained by summing the loss per layer, with sensitivity of each mrs-unit layer $F_ {\\theta_i}$ adaptively modulated and updated by $\\hat{\\alpha} _ {t,\\phi}$ and $\\hat{\\beta} _ {t,\\phi}$.   \n\n**Q6: Determination of hyper-parameter $\\gamma$**\\\n**R6:** The value of the $\\gamma$ is determined via hyper-parameter search experiments. As shown in the table below, we conducted extensive experiments with different values of $\\gamma$ on C2I & T2I tasks, benchmarking on ImageNet and JourneyDB datasets respectively. Based on the observation from experimental results, we select $\\gamma=0.35$ as the optimal value.\n| $\\gamma$ |  | 0.00 | 0.20 | 0.30 | 0.35* | 0.40 | 0.60 | 0.80 | 1.00 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| C2I (IS$\\uparrow$)  | Neural-RDM-U | 79.03 | 140.24 | 241.61 | **256.55** | 228.00 | 173.52 | 152.41 | 139.92 |\n|  | Neural-RDM-F | 224.75 | 247.07 | 273.09 | **295.32** | 266.91 | 263.06 | 247.71 | 235.79 |\n| T2I (IS$\\uparrow$)  | Neural-RDM-U | 64.50 | 182.52 | 269.98 | **235.35** | 208.04 | 192.67 | 189.27 | 146.35 |\n|  | Neural-RDM-F | 195.03 | 202.98 | 204.82 | **206.32** | 193.57 | 182.01 | 170.76 | 162.19 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for sparing their time and efforts reading our paper and giving many insightful comments. We notice that all reviewers hold a positive view regarding the _**well motivation**_ and _**superior performance**_ of our model. The major questions are summarized below and addressed point by point, to help understand our work better.

Meta-review: The paper addresses an essential problem of scaling up diffusion models, making use of gated residual mechanisms. The positive feedback of the reviewers, confirmed during the discussion period, indicates that the paper offers valuable insights and interesting method to achieve scaling of diffusion models. The authors are requested to include the results of the discussion of the reviewers in the final version of their paper, specifically related to the complexity of the method (empirical evaluation) as well as clarity of the presentation.
Decision: Accept (poster)