Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization
Authors: Qihao Liu
Zhanpeng Zeng
Ju He
Qihang Yu
Xiaohui Shen
Liang-Chieh Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Generation
Primary area: diffusion_based_models

Abstract: This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization.
Diffusion models have gained prominence for their effectiveness in high-fidelity image generation.
While conventional approaches rely on convolutional U-Net architectures, recent Transformer-based designs have demonstrated superior performance and scalability.
However, Transformer architectures, which tokenize input data (via "patchification"), face a trade-off between visual fidelity and computational complexity due to the quadratic nature of self-attention operations concerning token length.
While larger patch sizes enable attention computation efficiency, they struggle to capture fine-grained visual details, leading to image distortions.
To address this challenge, we propose augmenting the **Di**ffusion model with the **M**ulti-**R**esolution network (DiMR), a framework that refines features across multiple resolutions, progressively enhancing detail from low to high resolution.
Additionally, we introduce Time-Dependent Layer Normalization (TD-LN), a parameter-efficient approach that incorporates time-dependent parameters into layer normalization to inject time information and achieve superior performance.
Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, where DiMR-XL variants surpass previous diffusion models, achieving FID scores of 1.70 on ImageNet $256 \times 256$ and 2.89 on ImageNet $512 \times 512$. Our best variant, DiMR-G, further establishes a state-of-the-art 1.63 FID on ImageNet $256 \times 256$.

Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization
Qihao Liu!!2, Zhanpeng Zengl.", Ju Hell2, Qihang Yu', Xiaohui Shen', Liang-Chieh Chen
ByteDance
Johns Hopkins University
University of Wisconsin-Madison equal contribution
https://qihaa067, git.ggithhub...... io/ projeccts/DDMM
Uncurated 512 x 512 somples
DiMR (Ours)
Uncurated low- low-fidelln fidelity somples
DIMR (Ours)
DiT
Figure 1: (Top) Randomly sampled 512 x 512 images generated by the proposed DiMR. (Bottom) Random samples of the low visual fidelity 256 x 256 images generated by DiMR and DiT [41]. To detect low visual fidelity images for both models, classifier baaeiider-b-edddd rejection model is employed (with the same rejection rate). DiMR generates images with higher fidelity and less distortion than DIT.
Abstract
This paper presents innovative enhancements diffusion models by integrating a novel mullii-esooution network and time-dependent layer normalization. Diffu- sion models have gained prominence for their effectiveness in high-fidelity image generation. While conventional approaches rely on convolutional U-Net architec- tures, recent Transformer-basee and scalability. However, Transformer architectures, which tokenize input data (via "patchification"), face trade-off between visual fidelity and computational complexity due to the quadratic nature of self-attention operations concerning token length. While larger patch sizes enable attention computation efficiency, they
designs have demonstrated superior performance
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

struggle to capture fine fine-grainen grained visual details, leading to image distortions. To address this challenge, we propose augmenting the Diffusion model with the Multi- Resolution network (DiMR), framework that refines features across multiple res- olutions, progressively enhancing detail from low to high resolution. Additionally, we introduce l'me-DDeendeen Layer Normalization (TD-LN), parameter-efficient approach that incorporates time-dependent parameters into layer normalization to inject time information and achieve superior performance. Our method's 's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, where DiMR-XL variants surpass previous diffusion models, achieving FID scores of 1.70 on ImageNet 256 x 256 and 2.89 on ImageNet 512 x 512. Our best variant, DiMR-G, further establishes sfate-of-the-art 1.63 FID ImageNet 256 x 256.
Introduction
Diffusion and score-based generative models [23, 53, 55, 19, 54] have demonstrated promising results for high-fidelity image generation [7, 38, 44, 46, 48]. These models generate images through an iterative process of gradually denoising Gaussian random noise to create realistic samples Central to this process is neural network, tasked with denoising the inputs through mean squared error loss function. Traditionally, U-Net architectures [47] (enhanced with residual blocks [15] and self- attention blocks [59] at lower resolution) have been prevalent. However, recent advancements have introduced Transformer- based designs [59, 8]. offering superior performance and scalability.
In practice, Transformer-basee architectures face the challenge of balancing visual fidelity and computational complexity, primarily stemming from the self-attention operation and the patchification process employed for downsampling inputs [8] (i.e., a smaller patch size results in better visual fidelity at the cost of a longer token length and thus more computational complexity by the self- attention operation). The quadratic complexity inherent in self-attention concerning token length necessitates larger patch sizes to facilitate more efficient attention computations. However, the adoption of large patch sizes inevitably compromises the model's capacity to capture finer visual details, resulting in image distortion (i.e., low visual fidelity). This dilemma prompts DiT [41] to conduct systematic study on the impact of patch size image distortion, as depicted in Fig. 7 of their paper. Consequently, they settled on a patch size of for their final design. Similarly, U-ViT [2] opted for patch size of for input sizes of 256 x 256 and patch size of 4 for 512 x 512 images, effectively balancing the token length for different image sizes. Despite these meticulous adjustments, the generated results still exhibit discernible image distortion, as illustrated in Fig. 1.
One simplistic solution to mitigate image distortion in Transformer-bases architectures is adopting a patch size of 1, but this significantly increases computational complexity. Instead, inspired by the success of imuge cascade [20, 48] which generate images at increasing resolutions, we propose a feature cascade approach that progressively upsamples lower-resolution features to higher resolutions, alleviating distortion in image generation. In this study, we present DiMR, which enhances the Diffusion model with detail capture and computational complexity through improvements in the denoising backbone architecture. We employ multi-resolution network design that comprises multiple branches to progressively refine features from low to high resolution, preserving intricate details within the input data. Specifically, the first branch handling the lowest resolution incorporates Transformer blocks [59], leveraging the superior performance and scalability observed in prior works [2, 41], while the remaining branches utilize ConvNeXt blocks [35], which are efficient for high resolution features. The network processes inputs progressively from the lowest resolution, with additional features from the preceding resolution. The last branch refines features at the same spatial resolution as the input, effectively mitigating image distortion arising from the patchification.
network. DiMR tackles the challenge of balancing visual
Additionally, we observe that existing time conditioning mechanisms [42, 25, 7J. such as adaptive layer normalization (adaLN) [41], are parameter-intensive. intensive. In contrast, we propose a more efficient approach, Time- Deeeeneendddtt Layer Normalization (TD-LN), that integrates time-dependent parameters directly into layer normalization [1], achieving superior performance with fewer parameters.
To demonstrate its effectiveness, we evaluate DiMR on the class-conditional ImageNet generation benchmark [6]. On ImageNet 64 64, DiMR-M (133M parameters) and DiMR-L. (284M), without classifier- free guidance [18], achieve FID scores of 3.65 and 2.21, respectively, outperforming the

---

Transformer- U- U-V and U- U-ViT-L/4 by 2.20 and 2.05 FID. On ImageNet 256 x 256, DiMR- XL (505M) achieves FID scores of 4.50 without classifier-free guidance and 1.70 with classifier free guidance. Meanwhile, DiMR-G (1 (1.06B) further improves the FID scores to 3.56 without classifier- free guidance and 1.63 with classifier-free guidance. On ImageNet 512 512, DiMR-XI. (525M) achieves FIID scores of 7.93 and 2.89, without and with classifier-free guidance, respectively. These results demonstrate superior performance compared to all previous methods, despite having similar or smaller model sizes, establishing new state-of-the-art performance. In summary, our main contributions are as follows:
We develop effective strategies for integrating multi-resolution networks into diffusion models, introducing the novel feature cascade approach that captures visual details and reduces image distortions in high-fidelity image generation.
2. We propose TD-LN, a simple yet effective parameter-efficient method that explicitly encodes crucial temporal information into the diffusion model for enhanced performance.
3. We introduce DIMR, a novel architecture that enhances diffusion models with the proposed multi-resolution network and the TD-LN. DiMR demonstrates superior performance on the class-conditional ImageNet generation benchmark compared to existing methods.
2 Related Work
Diffusion models. Diffusion [53, 19] and score-based generative models [23, 55], centered around a denoising network trained to progressively produce denoised variants of the input data. They have driven significant advances across various domains [34, 29, 58, 56, 62, 40, 61], particularly excelling in high-fidelity image generation tasks [38, 44, 46, 48]. Key advancements in diffusion models include the improvements in sampling methodologies [19, 54, 26] and the adoption of classifier- free guidance [18]. Latent Diffusion Models (LDMs) [46, 41, 43, 63] address the challenges of high-resolution image generation by conducting diffusion in the lower-resolution latent space via a pre-trained autoencoder [28]. In this study, our focus lies designing the denoising network within diffusion models and examining its applicability across both pixel diffusion models and LDMs.
Architecture for diffusion models. Early diffusion models employed convolutional U-Net archi- tectures [47] as the denoising network, which were subsequently strengthened through explorations of either computing attention 39] performing diffusion directly at multiple scales [20, 13). architectures [2, 41, 14] along with other explorations [64, 57, 27] have emerged as promising alternatives, showcasing superior performance and scalability. Specif- ically, for Transformer- based architectures, U-VIT [2] treats all inputs, including time, condition, and noisy image patches, tokens and employs long-skip connections between shallow and deep transformer layers inspired by U-Net. Similarly, DiT [41] leverages Vision Transformers (ViTs) [8] systematically explore the design space under the Latent Diffusion Models (LDMs) framework, demonstrating favorable properties such as scalability, robustness, and efficiency. In this study, Network as new denoising architecture for diffusion models,
Recently, Transformer-basee
we introduce the featuring multi-branch design where each branch is dedicated to processing specific resolution.
Time conditioning mechanisms. Following the widespread usage of adaptive normalization [42] in GANS [3, 25], diffusion models similarly explore adaptive group normalization (AdaGN) [7] and adaptive layer normalization (AdaLN) [41] to encode the time information. These methods share the similarity in requiring computing a linear projection of the timestep, which significantly increases the parameter of the model. Recently, U-ViT [2] introduces new strategy simply treat time as a token and process with Transformer blocks. Even though effective, it is not feasible to treat time as input for other blocks (e.g.. ConvNeXt blocks [35]). In this study, we introduce Time- Depenndeendnt Layer Normalization (TD-LN), -efieeter-effiientt approach that explicitly encodes temporal information by incorporating time-dependent parameters into layer normalization [1].
3 Preliminary
Diffusion models [53, 19] are characterized by forward process that gradually injects noises to destroy data 2o ((z0), and reverse process that inverts the forward process corruptions. Formally,
3

---

the noise injection process is formulated as Markov chain:
T II ( (01-)) =
((1r)
where 2t for t E [1 T] is a family of random variables obtained by progressively injecting Gaussian noise into the data æu, and ((<r)|=)-1) = /a B,I) represents the noise injection schedule such that a, + 3, = 1. In the reverse process, a Gaussian model pl ( (2,) = (( ( of I) is learned approximate the ground truth reverse transition g( (a.)). This step is equivalent to predicting the denoised variant of the input æ; and thus the learning objective can be further simplified predicting the noise 6, via noise prediction network (with parameters || - <g(xx)] The condition information can be incorporated into 0), i.e., Eg (æt): min 2o.
0
the learning objective when the diffusion process is guided by the class condition [7], i.e., co (x e): min 20.0.e c,, ca, <) [12. Traditionally, learning this objective relies on the U-Net [47], with 0
the condition e encoded into the U-Net through various methods [2, 7, 41,1, 46].
Classifier-free guidance [18]. an effective approach to generating high-fidelity samples, combines the score estimates from conditional diffusion model and a jointly trained unconditional diffusion model. Formally, the classifier-free guidance encourages the sampled æ to have high p( c) by setting: ca(a;, c) eo (are; 0) + s- V log p(azle) x ca 0) + (ea(2t, e) (2 0)), where is the scale of guidance, 2 1, and setting = becomes the standard sampling. Following prior arts [2, 41], we also exploit classifier-free guidance.
4 Method
In this section, we begin by introducing the proposed Multi-Resolution Network (Sec. 4.1), which progressively refines features from low to high resolution. Next, we detail the proposed Time- Dependent Layer Normalization (Sec. 4.2). We then discuss several micro-level design enhancements (Sec. 4.3), Finally, we present the DiMR model variants, scaled for different model sizes (Sec. 4.4).
4.1 Multi-Resolution Network
Motivation. There is a trade-off between generation quality and computational complexity as depicted in the ablation study in Fig. 7 of DiT [41]. Their careful study revealed that Transformer- based diffusion models with smaller patch sizes operate at higher feature resolutions and produce better generation quality but incur higher computational costs due to the increased input size.
We conjecture that the distortion in U-ViT [2] and DiT [41] arises from their oversimplified up- sampling module, where lower-resolution feature maps are upsampled directly to the target size of the generated images via simple linear layer (for increasing channels) and pixel shuffling upsam- pling [52]. Inspired by image cascade [20, 48]-a method for generating high-resolution images by using multiple cascaded diffusion models produce images of progressively increasing resolu- tion tion-we propose feature cascade, which progressively upsample lower-resolution features to higher resolutions alleviate distortion in image generation. The feature cascade is implemented through Network, deployed the denoising network in diffusion models.
the proposed
Overview of multi-branch design. The proposed Multi-Resolution Network comprises R branches, where each branch is dedicated to process specific resolution. For the r-th branch (r E {1,.... , R)). the input features are processed by a convolution with kernel size of 2 2 and stride of 2 which effectively patchifies the input for different resolutions. The first branch (i.e., 1) downsamples the input features by factor of 2 and subsequently handles the lowest resolution features via the Transformer blocks [59], which enjoys the superior performance and scalability of self-attention operations [2, 41]. For higher resolution features, the remaining branches utilize the opera- tions [22, 50]. Intermediate features from the previous branch (last block's output) are upsampled and added with the inputs for the current branch. Following U-ViT [2), all branches employ the long skip connections and an additional x convolution in the end. The final branch (i.e., r R) refines features at the same spatial resolution as the input.
ConvNeXt blocks [35], which leverages the efficiency of large kernel depthwise-convoluue

---

ConvNeXI
Transformer Block
ConvNext Block
ConoNeXt Blook
ConsNeXi
Transfomeer Black
Conviell Block
ConoNexii Block
Depthwise Conv
TD-UN
Conrr 4x4, Stride
Conv 2x2, Strid 2
Corv 1x1,, Stride
Class Token
Position Embodding
input
TD-UN
Tine-Dependert Layer Norm
Add
Concal
1
Upeomple
Figure Model overview. We propose DiMR that enhances Diffusion models with a Multi- Resolution Network. In the figure, present the Multi-Res Reatii-Reslution Network with three branches. The first branch processes the lowest resolution times smaller than the input size) using powerful Transformer blocks, while the other two branches handle higher resolutions times smaller than the input size and the same size as the input, respectively) using effective ConvNeXt blocks. The network employs feature cascade framework, progressively upsampling lower-resolution features to higher resolutions to reduce distortion in image generation. The Transformer and ConvNeXt blocks are further enhanced by the proposed Time- Deneedeennd Layer Normalization (TD-LN), detailed in Fig. 4.
R). we define the r-th branch as a function fo.r as follows: Design details. For r E {1,..., "R.
c0 r), sa,- (Ze, Vr-1,
where the function fe.r, parameterized by 0 and r, takes as input the imput features æ, and the features from previous resolution yr-1 (also time t and condition c). The outputs of fe.- contain the intermediate features y, (last block's output, before the final x convolution) and the predicted noise eø (o c, r) for the resolution specific to r-th branch.
To process the inputs, the function forr first patchifies the input features æ,, and adds it with the upsampled features yr-1 from the previous resolution r 1. The resulting features are then processed by either a stack of Transformer blocks (when 1) ConvNeXt blocks (when of with another 3 x 3 convolution added in the end. Formally, we have:
fer Mr-1,
Convaxs (Patchiff + (p)
(2)
where Convaxa is 3 3 convolution, Patchify patchification instantiated via convolution with kernel size of 2 x 2 and stride of R Upsample is the pixel shuffling upsampling operation [52], and gø.r augmented with the long skip connections [2]. For the first branch (i.e., r 1), Yo is set to zero. The noise prediction Ea Eg( c, R) at the last branch (i.e., r = R) is used for the iterative diffusion process. We illustrate the proposed Multi-Resolution Network with three branches in Fig, 2. Note that the input features can be either raw image pixels or latent features after VAE [28], where the latent features facilitate efficient high-resolution image generation [46].
stack of Transformer blocks or ConvNeXt blocks, depending on r,
4.2 Time-Dependent Layer Normalization
Motivation. Time conditioning plays a crucial role the diffusion process. While the ConvNeXt blocks in the Multi-Resolution Network efficiently process high-resolution features, they also present new challenge: How do we inject time information into ConvNe:: blocks? ? To address this, we carry out a systematic ablation study (details in Tab. 2), starting with the U-VIT architecture [2], which encodes time information via an in-context conditioning mechanism, Time-Toker (i.e., treating time input token to Transformer). Unlike Transformer blocks, however, it is not feasible to add a time token directly ConvNeXt blocks, which can only process 2D features. As an alternative, explored the adaptive normalization mechanism, particularly the adaptive layer normalization
5

---

1.0
0.8
0.0
0.6
0.6
Variance
84
Variance
8.4
0.
93
0.2
0.0
6.0
10
10
10
PCS of Scale
PCs of Scale V22
PCs of Shift B
PCS of Shift
Figure 3: Principal Component Analysis (PCA) of learned scale and shift parameters in adaLN-
Zero [41]. We conduct PCA on the learned scale (7, 12) and shift (31, (3) parameters obtained from a parameter-heavy MLP in adaLN-Zero using pre-trained DiT-XL/2 [41] model. The vertical axis represents the explained variance ratio of the corresponding Principal Components (PCs). Our observations reveal that the learned parameters can be largely explained by two principal components, suggesting the potential to approximate them by a simpler function.
AdaI AN-Zeo [41]. Interestingly, we found AdaL-N-Zee -N-Zero to be more effective than Time-Token on the ImageNet 64 x 64 benchmark, contradicting U-ViT's findings on CIFAR-10 [30] (See Fig. 2(b) in [2]). However, AdeL sigmificantly increases model parameters (from 130.9M to 202.4M) due the Multi-Layer Perceptron (MLP) used to adaptively learn the scale and shift parameters.
To understand how time information utilized adaLN- Zero, we conducted Principal Component Analysis (PCA) on the learned scale (71:72) and shift (31, 32) parameters from parameter-heavy MLP in --ero using a pre-trained DiT-X1/2 [41] model, as shown in Fig. 3. Intriguingly, parameters can be largely explained by two principal components, suggesting that a parameter-heavy MLL might be unnecessary and that a simpler function could suffice. To address the increase parameters, we intro- duce Lime-Dependen Layer Normalization (TD-LN), a straightforward and lightweight method inject time into layer normalization. We detail the designs below.
Score
MIP
MJ"
TD-LN
observed that the learned
Layer
Soofe
dite sitee
TD-LN
Layer Norm
Lagar Noer
1hz Tance [ine Cau
Transfortrer Blook
Transformey Blook
indeL--Zerro
TD-UN
adaLN design. Building layer normalization [1], adaLN additionally learns the scale parameter ? and shift parameter 31 via an MLP from the of the embedding vectors of time and class condition c. For- mally, given the input (ignoring the dependency on for simplicity), we have:
Figure 4: Time conditioning mecha- nisms. (Left) adal.N-Zero [41] learns scale and shift parameters (Tis B., Ox., i= (1,2]) using parameter-heavy MLPs. (Right) The proposed Time-Dependee Layer Normalization (TD-LN) formulates the LN statisties as functions of time (t), (())), making it parameter- effiiieet....
71, B1 MLP(Embed()) + Embed (c)),
(4) where ?1 and 31 scale and shift the output from the layer normalization LN, the function Embed generates the embedding vectors for time and class condition and is the output. The LN has its learnable affine transform parameters 7 and (3. N- [41] introduces another scale parameter 01, obtained from the same MLP for zero initialization of residual block [12]. We note that DiT employs two sets of (71, ,1, a.) and (7,.32, a2) in Transformer block, as shown in Fig. 4.
2 = LN(#, 7: 3) + 31.
TD-LN design. In contrast, our proposed method, Lime-Dependent Layer Normalization (TD-LN), directly incorporates time f into layer normalization by formulating LN's learnable affine transform parameters ÿ and B as functions of t. Motivated by the observation that the learned parameters of adaLN-Zere can be largely explained by two principal components, we propose to model this through the linear interpolation of two learnable parameters pl and P2. Formally.
s(t) = Sigmoid w t b). (t) stt) P1 + (1 (1-s(t))- P2:
(5)
(6)
where s(t) is a transformation of time i, w and are the learnable weight and bias, and Sigmoid the sigmoid activation function. The other affine transform parameter, ((t), is formulated similarly with

---

another two parameters P3 and P1. Consequently, the proposed TD-LN is represented as follows:
(a, (())
Unlike adaLN, which learns additional re-scaling 71 and re-centering 31 variables, TD-LN directly incorporates the time-dependent 7() and (8t) into layer normalization, eliminating the need for a parameter-heavy MLP. Furthermore, TD-LN a versatile mechanism, enabling the injection of time information into both Transformer blocks and ConvNeXt blocks. In DiMR, we replace all layer normalizations with the proposed TD-LN, and treat the class condition e as input token for the Transformer blocks.
4.3 Micro-Level Design
In addition the major architectural modifications discussed earlier, we also explore several micro- level design changes to enhance model performance.
Multi-scale loss. The proposed Relti-Resolution Network comprises R branches, each dedicated to processing features at a specific resolution, naturally producing multi-scale outputs. To leverage this, we explore training the network with multi-scale loss Lmutti, which is a weighted sum of mean squared error loss at each resolution. Formally, the multi-scale loss defined as follows:
Emeulti - a, EE, (Downsample(e..r) co co(2,,,,))1
(8)
where is the loss weight for the r-th branch, and Downsample(e:. r)) downsamples the target noise G by a factor of 2 using average pooling (the R-th branch, containing no downsampling, is our final output). We set a, 1/(2"trr 2*trr, motivated by the prior work [21] which found that the signal to noise ratio increases by a factor of k22 when the noised input is average-pooled with & x kernel. Intuitively, for the intermediate outputs are scaled down quadratically based on the downsampling factor.
target output (the R-th branch) has a loss weight ar 1, and the loss weights
Gated linear unit. In the proposed Delti-Resolution Network, both Transformer and ConvNeXt blocks include an MLP block, consisting of two linear transformations with GeLU activation [16] in between. We also explore replacing the first linear layer with GeGLUU [51], an enhanced version of the Gated Linear Unit (GLU) [5] that has 2x expansion rate.
4.4 DiMR Model Variants
We now introduce the DiMR model variants, scaled appropriately for different model sizes. We present four sizes: DiMR-M (medium, 133M parameters), DiMR-L (large, 284M parameters), DiMR-XL (extra-large, around 500M parameters) and DiMR-G (giant, 1.06B parameters). Three hyperparameters R (number of branches), N (number of layers per branch), and D (hidden size per bbranch)-dffine each DiMR variant. Specifically, R determines the number of branches in the multi-resolution network. We append 2R 3R the model name indicate whether two or three branches are used. The number of layers N in the multi-resolution network is represented as tuple of R numbers, where the r-th number specifies the number of layers the r-th branch. Similarly, the hidden size D is also tuple of R numbers. We follow a straightforward scaling rule: most layers are stacked in the first branch, which processed by Transformer blocks, while the remaining branches use only half the number of layers of the first branch. Additionally, when the resolution is doubled, the hidden size is reduced by a factor two. The model variants details are presented in Tab. 4 in Sec. B in the Appendix.
5 Experimental Results
5.1 Experimental Setup
Datasets. We consider class-conditionss image generation tasks at 64 x 64, 256 x 256, and 512 x 512 resolutions on ImageNet- K [6]. For images at 64 x 64, we train DiMR on pixel space. For images at 256 x 256 and 512 x 512, following the baselines [2, 41], we utilize an off-the-shelf pre-trained variational autoencoder [28] from Stable Diffusion [46] to extract the latent representations sized at 32 32 and 64 x 64, respectively. Then we train our DiMR to model these latent representations.
7

---

Table 1: Class-conditional image generation on ImageNet 256 256 and ImageNet 512 x 512. We report training epochs, number of parameters (#Params), GFLOPs, and FID-50K with and without Classifier-Free Free Guidance (CFG). Best results are marked in bold.
(a) ImageNet 256 x 256
(b) ImageNet 512 x 512
Model
Epoch
AParams,
Gfiops
HD (wi CHG)
FID
Model
Epoch
#Params.
Gilops HID (wlo CPGIL HD]
ADM-U [7]
396
608M
742
7.49
3:94
ADM [7]
422M
23.24
7.72
LDM-4 [[6]
166
400M
104
10.56
3.60
ADM-U
1081
731M
2813
9.96
3.85
U-VI--R22 [2]
400
501M
133
6.58
2.29
U-VIT-LM [2]
400
287M
77
4.67
DIT-XI.2 [41]
1399
675M
119
9.62
2.27
U-VIT-H44 [2]
501M
133
15.70
4,05
DIMR-XI (Durs)
505M
160
4.87
1.77
DIT-XI [41]
675M
525
12.03
3.04
DIMR-XL/2R (Ours)
800
505M
160
4.50
XL/R (Ours)
206
8,56
3.23
DiMR-G/2R (Ours)
1.06B
3.56
1.63
DiMR-XL/3R (Ours)
525M
2,89
Evaluation. We measure the model's performance using Fréchet Inception Distance (FID) [17]. We report FID on 50K generated samples to measure the image quality (i.e., FID-50K). To ensure fair comparisons, we follow the same evaluation suite as the baseliness [7, 41] to compute the FID scores. We also report Inception Score [49] and Precision/Recall [31] in Sec. D as secondary metrics.
Implementation details. We use Adam W optimizer [36] with a constant learning rate of 2 x 10-4
for most experiments, except for the 64 x 64 models where we use x 10-4, A batch size of 1024 is used for most architectures. For a fair comparison with DiT [41], we train the 256 x 256 and 512 512 models for IM iterations, and also report results for 500K iterations to compare with U-ViT [2]. We train the 64 x 64 models for 300K iterations, following the U-ViT protocol. Our training hyperparameters are almost entirely retained from U-VIT [2]. We did not tune learning rates. decay/warm- up schedules, Adam 3//32 values, or weight decays. Further details on hyperparameters and configurations are provided in Sec. in the Appendix.
5.2 State-of-the-Art Diffusion Models
We compare DiMR with state-of-the-art diffusion models on ImageNet 256 x 256 and 512 x 512 in Tab. 1, and provide more comparisons with other types of generative models in Tab. 7 and Tab. 8 in the Appendix. Results on ImageNet 64 x 64 are reported in Tab. in the Appendix. More random samples of the generated images are also presented in Fig. 7 to Fig. 18 in the Appendix.
ImageNet 256 x 256. From Tab. la, we observe that our DiMR-XL/2R outperforms all previous diffusion-based models and achieves a state-of-the-art FID-5OK score of 1.70. Specifically, with a comparable model size and equal or fewer training epochs, our model surpasses previous state-of- the-art transformer- based diffusion models, including U- ViT[2] (1.77 vs. 2.29 with Classifier-Free Guidance [18] (CFG) and 4.87 vs. 6.58 without CFG) and DiT [41] (1.70 vs. 2.27 with CFG and 4.50 vs. 9.62 without CFG). Our best model, DiMR-G/2R, scales up to the Dillion-parameter level, setting a new sfate-of-the-ar art with an FID of 1.63 with CFG and 3.56 without CFG.
ImageNet 512 x 512. Our DIMR outperforms all previous diffusion-based models on ImageNet 512 x 512 and achieves state-of-the-an FID-50K score of 2.89 as shown in Tab. 1b. It is worth noting that, although both Gflops and model sizes are critical for improving performance, as discussed in the DiT paper [41], we still outperform it with only 39.2% of the GFLOPs and 77.8% of the model size, improving the FID-50K from 3.04 to 2.89. As transformers and diffusion models have demonstrated good scaling behavior, we believe that further scaling up our DiMR will lead to better performance, which we have left as future work.
5.3 Alleviating Distortion
Transformer-basee architectures encounter the challenge of balancing visual fidelity with compu- tational complexity. Despite adopting small patch size of 2, current models still struggle with distortions. To illustrate the effectiveness of DiMR in alleviating these distortions, we adopt a blassifier-based rejection model following previous work [45]. However, we diverge from previous approaches by solely using the rejection model to analyze distorted images, rather than filtering out bad images and computing metrics only on selected 'good' images. It is important to note that all metrics in our paper are computed without using the rejection model to ensure fair comparisons.
8

---

3i fortioo rate 63,
DIT FID rote 71.00
DiMR (Cors) FEDsL77
Sittrtion 29.22 2%
Figure 5: DiMR alleviates distortions and improves visual fidelity. In this figure, we randomly visualize the detected sow-fidelity images, identified by pretrained classifier, which are generated by the best models from the baselines and our DiMR. The first column reports both their FID-50K scores and the proportion of distorted images based human evaluation. DiMR demonstrates better generation performance and lower distortion rates than the baselines.
Table 2: A Allation study. Beginning with the baseline, we verify the effectiveness of each component.
Model
AdaLN-Zerro TD-LN Multi-branch GLU [51] Multi-scale Loss
FID(L)
Params.
Baseline M/4 (2]
130.9M
5.44
202.4M
7,91
217.9M
5.21
154,0M
4.86
132.9M
DIMR M/3R (Ours)
3.65
132.9M
Specifically, we randomly generate 80K images for each model and utilize pretrained Vision Transformer classifier [8] identify low-fidelity images based on the predicted probabilities. Images with probability below a threshold of 0.2 are considered low-fidelity potentially distorted. Fig. shows random samples of low-fidelity images detected by the classifier. However, we find that all detected images distorted; many classified with low probability due to classifier errors. To accurately identify distorted images among those detected by the classifier, we conduct user studies where human evaluators manually assess the images. Images generated by all three methods are merged and presented, along with their corresponding class labels, to human evaluators, who are instructed to determine whether each image is distorted (i.e., identify low-fidelity images). Each image is evaluated by five different human evaluators. We consider the proportion of distorted images generated by different models, i.e. distortion rate. We compute three distortion rates, for each model, from each evaluator based on the images they evaluate. The final distortion rate for each model is obtained by averaging the rates from all evaluators. As reported in Fig. 5, we observe that even among those low-fidelity images, only 29.2% of the images generated by DiMR are distorted, while previous methods yield much higher distortion rates 63.5% and 71.0%.
5.4 Ablation Studies
We conduct the primary ablation experiments ImageNet 64 64, progressively building on the bascline U- ViT-M//4 [2] to validate the effectiveness the proposed designs, leading to our final model, DiMR-M/SE
as presented in Tab. 2. Additionally, we explore alternative design choices on ImageNet 256 x 256 with DiMR-XL/2R, including adopting a pure convolutional architecture, replacing addition with concatenation in feature cascading, and introducing skip connections between branches, as shown in Tab. 3.
AdaL.- vs. TD-LN. Since the time token used in U-VIT cannot be adopted for ConvNeXt blocks, we first apply AdaL-N-Zero [41] to the original U-ViT and our multi-branch network. As observed in row of Tab. 2, AdaLN-Zero slightly improves the performance of U-ViT from 5.85 to

---

Table 3: Design choices. We empirically experiment with different design choices in model architec- ture (Tab 3a), feature cascade (Tab 3b) and skip-connection (Tab 3c). We report FID-50K scores with classifier-fiee free guidance (CFG) after 400 training epochs.
(a) Pure Cony v.s. Hybrid
(b) Concatenation v.S. Addition
(c) w/ v.s. w/o Skip- Colp-c-Cnnettioon
Model
branch PIDL
Model
feature cascade FID4
skip commecteeci
RID
ConeNeXi
2.09
Concatenatio
2.06
1,96
DiMR-XI/2R
DIMR-XL/2R
DIMR-XI/2R
Transformer
1.77
Addition
1.77
1.77
5.44. However, it does not perform well on ConvNeXt blocks and thus decreases the performance from 5.44 to 7.91 (row 3). Additionally, AdaLN-Zeru significantly increases the model size from 130.9M to 202.4M. In contrast, our TD-LN is more flexible and parfietter-efficienn: it efficiently provides time information to both Transformer blocks and ConvNeXt blocks, improving the FID-50K score from 7.91 to 5.21 (row 4), and also reduces the model size from 217.9M to 154.0M.
GLU further reduces model size. In Tab. 2, row shows the improvement of GLU compared with the vanilla MLP block. We observe that using GLU slightly improves the performance from 5.21 to 4.86 and further reduces the model size from 154.0M to 132.9M.
Multi-scale loss is critical for multi-resolution network. Training a presents additional challenges and can result in sub-optimal results. In Tab. 2, row illustrates that our multi-scale loss significantly enhances the performance, achieving FID-50K score of 3.65.
network
Multi-branch design improves visual fidelity and alleviates distortions in image generations. Finally, comparing the multi-branch design in row 6 (incorporating TD-LN, GLU, and multi-scale loss to facilitate training) with the baseline in row 1 reveals a significant improvement in FID-50K, from 5,85 to 3.65, with just a 1.5% increase in model size (130.9M to 132.9M) 2M). Additionally, from Fig. 5, it's evident that the multi-branch design generates images with higher fidelity and less distortion.
Transformer essential for low-resolution processing. As shown in Table 3a, replacing the Transformer blocks in the 1st (lowest-resolution) branch with ConvNeXt blocks results in DiMR variant that uses only convolutional layers. However, this configuration performs worse compared to combining Transformer blocks with ConvNeXt blocks across different resolutions. This indicates that Transformers are more effective at capturing fine-grained details, while their usage at the lowest resolution maintains manageable computational cost.
Simple addition suffices for multi-resolution feature eascading. As shown in Table 3b, a straight- forward addition operation effectively transfers information from nower-resolution features to higher- resolution features. Replacing addition with concatenation leads to slightly worse results. We also validate the necessity of adding skip-connection between branches. As shown in Table 3c, intro- ducing skip-connection not only degrades performance but also complicates the model architecture. Therefore, adopt simple upsampling followed by addition operation for feature cascading.
6 Conclusion
In this work, we introduce DiMR, which enhances diffusion models through the Multi-Resolution Network, progressively refining features from low to high resolutions and effectively reducing image distortion. Additionally, DiMR incorporates the proposed parameter-efficien Time-Dependent Layer Normalization (TD-LN), further improving image generation quality. The effectiveness of DiMR has been demonstrated on the popular class-conditional ImageNet generation benchmark, outperforming prior methods and setting new state-of-the-ar art performance on diffusion-style generative models. We hope that DiMR will inspire future designs of both denoising networks and time conditioning mechanisms, paving the way for even more advanced image generation models.
Acknowledgement: We thank Xueqing Deng and Peng Wang for their valuable discussion during Zhanpeng's internship.
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.0644 2016.
arXiv preprint
[2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words; vit backbone for diffusion models. In CVPR, 2023.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprini arXiv:1809.110% 2018.
[4] Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
[5] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, 2017.
[6] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NewrIPS, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[10] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV, 2023.
[11] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is strong image synthesizer. arXiv preprint arXiv:2303. /4389, 2023.
[12] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenct I hour. arXiv preprint arXiv: 02677, 2017.
[13] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In ICLR, 2023.
[14] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: transformers for image generation. In ECCV, 2024.
Diffusion vision
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[16] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 160.00845, 2016.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12599, 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 2020.
[20] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models high fidelity image generation. JMLR, 23(47):1-33 2022.
[21] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In /CML, 2023.
[22] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolational neural networks for mobile vision applications. arXiv preprint arXiv: 1704. 04861, 2017.

---

[23] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. JMLR, 6(4), 2005.
[24] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
[25] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[26] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. NeurIPS, 2022.
[27] Dongjun Kim, Mhich-HsiinB Lai, Wei-Hsiang Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of one-step generator from low-resolution diffusion teacher. arXiv preprint arXiv:2405. 140.:222.......,... 2024.
[28] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[29] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXivv2009.09700 2020.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS, 2019.
[32] Doyup Lec, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
[33] Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representa- tions. arXiv preprint arXiv:2312.0370/. 2023.
[34] Xiang Li, John Thickstun, Ishaan Guirajani. Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. NeurlPS, 2022.
[35] Zhuang Liu, Hanzi Mao. Chao- Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[36] llya Loshchilov and Frank Hutter. 1,05 2017.
Decoupled weight decay regularization. arXiv preprint
[37] Nanye Ma. Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, 2024.
[38] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: 22. 10741, 2021.
[39] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In ICMI,, 2021.
[40] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07466 2022.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[42] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. reasoning with general conditioning layer. In AAAI, 2018
Film: Visual
[43] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:3307.0195 2023.
[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612. 2022.
[45] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelin images with vq-vae-2. NewrIPS, 2019.
12

---

[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[47] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NewrIPS, 2022.
[49] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurlPS, 2016.
[50] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018
[51] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05222 2020.
[52] Wenzhe Shi, Jose Cahallero, Ferenc Huszár, Johannes Toiz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[53] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML,, 2015.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019.
[56] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. NewrIPS, 2021.
[57] Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 2024.
[58] Arash Vahdat, Francis Williams, Zan Gojcie, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models shape generation. NeurlPS, 2022.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[60] Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng. Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding- free image generation via bit tokens. arXiv preprins arXiv:2409. 16211, 2024.
[61] Jiarui Xu, Sifci Liu, Arash Vahdat, Wonmin Bycon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
[62] Minkai Xu, Lantao Yu, Yang Song. Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint (xXiv:2203.02923 2022.
[63] Zeyue Xue, Guanglu Song. Qiushan Guo, Boxiao Liu, Zhuofan Zong. Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture diffusion paths. NeurlPS, 2023.
[64] Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In CVPR, 2024.
[65] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge. and Yonghui Wu. Vector-quantized image modeling with improved vogan. arXiv preprint arXiv:2110.04627. 2021.
[66] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image worth 32 tokens for reconstruction and generation. NeurIPS, 2024.
13

---

Appendix
In the appendix, we provide additional information as listed below:
Sec. provides the dataset information and licenses.
Sec. B provides the DiMR model variants, scaled appropriately for different model sizes.
Sec. provides the implementation details DiMR.
Sec. D provides more comparison with other methods for class-conditional image generation on ImageNet 64 x 64, ImageNet 256 x 256, and ImageNet 512 x 512.
Sec. E provides detailed introduction and more results of the Principal Component Analysis (PCA) on the scale and shift parameters learned in adaLN-Zero.
Sec. F provides more generated image samples by DiMR.
Sec. G discusses the limitations of our method.
Sec. discusses the positive societal impacts of our method.
Sec. I discusses the potential risk of our method and safeguards that will be put in place for responsible release our models.
A Datasets Information and Licenses
The ImageNet [6] dataset, containing 1,281,167 training and 50,000 validation images from 1,000 different classes, is standard benchmark for image classification and class-conditional image generation. For the task of class-conditional image generation, the images are typically resized specified size, e.g., 64 x 64, 256 256, 512 512.
ImageNet:
License: Custom License, non-commercial. attps::/ //mpsg///mmaeeeee neteeenettteee aaee.or//acceesareeeent
Dataset website: /
B DiMR Model Variants
We introduce the DiMR model variants, scaled appropriately for different model sizes. We present four sizes: DiMR-M (medium, 132.7M parameters), DIMR-L (large, 284.00 844.0M parameters), DiMR-XL lextra-large, around 500M parameters), and DiMR- G (giant, 1.06B parameters) Three it (number of branches), N (number of layers per branch), and (hidden size per branch)-define each DiMR variant. Specifically, It determines the number of branches in the multi-resolution network. We append or 3R to the model name indicate whether two or three branches are used. The number of layers N in the multi-resolution network is represented of R numbers, where the r-th number specifies the number of layers the r-th branch. Similarly, the hidden size D is also tuple of R. numbers. We follow straightforward scaling rule: most layers stacked in the first branch, which processed by Transformer blocks, while the remaining branches only half the number of layers of the first branch. Additionally, when the resolution is doubled, the hidden size reduced by a factor two. The model variants are illustrated Tab. 4.
tuple
Table DiMR family. The specific configuration DiMR variant determined by the hyperpa- rameters R (number of branches), N (number of layers per branch), and D (hidden size per branch).
model
input size latent size
Wbranches R Wlayers N
hidden size D
Mpurams
DIMR- DiMR-M/3R
64 x 64
3
(15, 8, 8)
(768, 384, 192)
133M
DiMR-L/3R
64
(33, 17. 17)
(768, 384, 192)
DiMR- DiMR-XL/2R
256 256
32 32
2
(39, 20)
(960, 480)
505M
DiMR-XL/3R
512 x 512
64 x 64
3
(39, 20, 20)
(960, 480, 240)
525M
DiMR-G/2R
256 256
32
29)
(1152, S76)
L.06B
14

---

Table 5: Experimental setup of DiMR. Experimental settings for all DiMR variants, including model architectures, training hyperparameters, training costs, and sampler information.
Model
DiMR-MI M//3R
DiMR-L/3R
DiMR-XL/2R
DIMR XL/3R
DiMR-G/2
Parameters
133M
284M
505M
525M 512 x 512
1.06B 256 x 256
Image Resolution
64 x 61
64 x 61
256 x 256
Latent space
X
Latent shape
32 sd-vae- wae-ft-ema
64 61 sd-vae-ft vae-ft-ema
32 32 sd-vae-ft-emm
Image decoder
Number of branches R
3 Transformer
2 Transformer
3 Transformer
2 Transformer
Blocks in 1st branch
Transformer
Layers
33
39
39 960
57
Dimensions
768
768
960
1152
# Heads
12
12
16
16
16
Resolution
16 16
16 x 16
16 x 16
16 x 16
16 x 16
Loss Coeff's
1/16
1/16
1/4
1/16
1/4
Blocks 2nd branch
ConvNeXt
ConvNeXt
ConvNeXt
ConvNeXt
ConvNeXt
Layers
17
20
20
29
Dimensions
384
384
480
480
576
Kernel size
7 x
7
7 x 7
7 x
7 x 7
Resolution
32 32
32 x 32
32 x 32
32 x 32
32 32
Loss Coeffs.
1/4
1/4
1/4
Blocks in 3rd branch
ConvNeXt
ConvNeXt 17
ConvNeXt
Layers
20
Dimensions
192
192
240
Kernel size
7
7 x
x 7
Resolution
64 64
64 x 64 1
64 64 1 1024
Loss Coeff's
Batch size
1024
1024
1024
1024
Training iterations
300K
300K
1M
1M
IM
Warm-up steps
5K
5K AdamW
5K AdamW
5K
Optimizer
Adam
Adam
Adam
learning rate
3 x 10-4
3 x 10-4
x 10-4
2 x 10-4
2 10-1
Weight decay
0.03 (0.99. 0.99)
0.03
0.03
0.03
0.03
Betas
(0.99, 0.99)
(0.99, 0.99)
(0.99, 0.99)
(0.99, 0.99)
Training devices
8 A100
A100
A100
A100
A100
Training time
62 hours
80 hours
172 hours
288 hours
400 hours
Sampler
DPM-Solver
DPM-Solver 50
DPM-Solver 250
DPM-Solver 250
DPM-Solver 250
Sampling steps
C Implementation Details
We the AdamW optimizer [36] with constant learning rate of 2 10-4 for most experiments, except for the 64 x 64 models where we use x 10-4 We set the weight decay to 0.03 and the betas (0.99, 0.99) for all experiments. A batch size of 1024 is used for all architectures. For fair comparison with DiT [41]. we train the 256 x 256 and 512 x 512 models for IM iterations, and we also report results for 500K iterations compare with U-ViT [2]. We train the 64 x 64 models for 300K iterations, following the U-ViT protocol. All experiments use 5K steps for warm-up. We present the detailed experimental setup for all DiMR variants in Tab. 5.
D Additional Experimental Results
We present the full results of the proposed DiMR compared to other methods on ImageNet [6] in
terms of Fréchet Inception Distance (FID) [17], Inception Score (IS) [49], and Precision/Recall [31]. The comparisons are made on class-condition ImageNet x in Tab. 6, and with classifier-free guidance [18] on ImageNet 256 x 256 in Tab.
image generation without classifier-free guidance on
and ImageNet 512 x 512 in Tab. 8.
15

---

Table 6: Class-conditional image generation ImageNet 64 64 (w/o classifier-free guidance). Metrics include Fréchet Inception Distance (FID), Inception Score (IS), Precision, and Recall, where "4" or "f" indicate whether lower higher values are better, respectively. "Type": the type of the generative model. "Epoch": the number of epochs trained on ImageNet [6]. "#Params": the number of parameters in the model. "#Gflops": the computational cost. "Diff.": Diffusion models.
Model
Type
Epoch
#Params.
Gflops
FID(()
IS(1)
Precision(1)
Recall(1)
U-VIT-M/4 [2]
Dift.
240
131M
5.85
33.71
0.69
0.61
U-ViT-I/4 [2]
Diff.
240
287M
4.26
40.66
0.71
0.62
DiMR-M/SE (Ours)
Diff.
240
133M
54
3.65
42.41
0.74
0.59
DiMR-L/3R (Ours)
Diff.
240
284M
111
2.21
55.73
0.75
0.60
Table 7: Class-conditional image generation on ImageNet 256 x 256 (with classifier-free guid- ance). Metrics include Fréchet Inception Distance (FID), Inception Score (IS), Precision, and Recall, where "L" "f" indicate whether lower or higher values are better, respectively. We report results of GAN-based models (GAN), BERT-style masked-prediction models (Mask.), autoregressive models (AR), visual autoregressive models (VAR), and diffusion based models (Diff.). "Type": : the type of the generative model. "Epoch": the number of epochs trained on ImageNet [6]. "#Params": the number of parameters in the model. "#Gflops": the computational cost. "-re": the models utilize rejection sampling. "Mask. + Diff." the models using maasked-predicteee to improve diffusion models.
Model
Type
Epoch
#Params.
Gflops
FID(.)
IS(t)
Precision(t)
Recall(t)
BigGAN
GAN
112M
6.95
224.5
0.89
0.38
GigaGAN [24]
GAN
569M
3.45
225.5
0.84
0.61
MaskGIT [4]
Mask.
300
227M
6.18
182.1
0.80
0.51
MaskGiT-re [4]
Mask.
300
227M
300
4,02
355.6
RCG [33]
Mask.
200
502M
3.49
215.5
TYTok-S-128 [66]
Mask.
800
287M
1.97
281.8
MDT-G [10]
Mask. + Diff.
1299
676M
119
1.79
283.0
0.81
0.61
MDTTv2-G [1]
Mask. + Diff.
919
675M
119
1.58
314.7
0.79
0.65
MaskBit [60]
Mask.
1080
305M
1.52
328.6
VQGAN [9]
AR
100
TA4B
15.78
74.3
VQGAN-re [9]
AR
100
L.4B
5.20
280.3
VTVQ [65]
AR
100
1.7B
4.17
175.1
ViTVQ-re [65]
AR
1.7B
3.04
227.4
RQTran [32]
AR
50
3.8B
7.55
134.0
RQTran-re [32]
AR
50
3.8B
3.80
323.7
VAR-d16 [57]
VAR
200
310M
3,60
257.5
0.85
0.48
VAR-d20 [57]
VAR
250
600M
2.95
306.
0.84
0.53
VAR-d24 [57]
VAR
350
1.0B
2.33
320.1
0.82
0.57
VAR-d30 [57]
VAR
350
2.0B
1.97
334.7
0.81
0.61
VAR-d30-re [57]
VAR
350
2.0B
1.80
356.4
0.83
0.57
ADM-G [7]
Diff.
396
554M
4.59
186.7
0.82
0.52
ADM-G, ADM-U [7]
Diff.
208
608M
742
3.94
215.8
0.83
0.53
CDM [20]
Diff.
2158
4.88
158.7
LDM-4 [46]
Diff.
400M
3.60
247.7
DiT-L/2 [41]
Diff.
1399
458M
5.02
167.2
0.75
0.57
DIT-XL/2 [41]
Diff.
1399
675M
2.27
278.2
0.83
0.57
U-VIT-L/2 [2]
Diff.
240
287M
3.40
219.9
0.83
0.52
U-ViT-H/2 [2]
Diff.
400
501M
133
2.29
263.9
0.82
0.57
DIFFUSSM-XL [64]
Diff.
515
673M
280
2.28
259.1
0.86
0.56
SIT-XL [37]
Diff.
1399
675M
119
2.06
270.3
0.82
0.59
DIMT [14]
Diff.
400
561M
114
1.73
276.5
0.80
0.62
DIMR-XL/2R (Ours)
Diff.
400
505M
160
1.77
285.7
0.79
0.62
DiMR-XL (Ours)
Diff.
800
505M
160
1.70
289.0
0.79
0.63
DiMR-G/2R (Ours)
Diff.
800
1.IB
331
1.63
292.5
0.79
0.63
ImageNet 64 x 64. We follow the exact experimental setup of U-VIT [2] for class-conditional image generation on ImageNet x 64 without classifier- free guidance verify the effectiveness of our proposed backbone. Therefore, we focus solely on comparing against U-VIT on this benchmark. As shown Tab. 6, when both are trained for 240 epochs, the proposed DiMR-M/3R with 133M parameters achieves an FID of 3.65 and an IS of 42.41, improving upon the counterpart U- L-VIT-M/4 with 131M parameters by 2.20 in FID and 8.70 in IS. For the larger model, DiMR-L/3R with 284M parameters outperforms D-ViT-L/4 with 287M parameters by 2.05 in FID and 15.07 in IS. These
16

---

Table 8: Class-conditional image generation ImageNet 512 512 (with classifier-free guid- ance). Metrics include Fréchet Inception Distance (FID), Inception Score (IS), Precision, and Recall, where "4" "f" indicate whether lower higher values are better, respectively. We report results of GAN-based models (GAN), BERT-style masked-prediction models (Mask.), autoregressive models (AR), visual autoregressive models (VAR), and diffusion based models (Diff.). "Type": the type of the generative model. "Epoch": the number of epochs trained on ImageNet [6]. "#Params": the
number parameters in the model. "#Gflops": the computational cost.
Model
Type
Epoch
#Params.
Gflops
FID(L)
IS(1)
Precision(1))
Recall(t)
BigGAN [3]
GAN
158M
8.43
177.9
0.88
0.29
MaskGIT [4]
Mask.
300
227M
7.32
156.0
0.78
0.50
MaskGiTT-re 141
Mask.
300
227M
4.46
342.0
VAR-d30-s [57]
VAR
350
2.35B
2.63
303.2
ADM-G [7]
Dill.
422M
7.72
172.7
0.87
0,42
ADM-G, 17M-U171
Diff.
1081
731M
2813
3.85
221.7
0.84
0.53
DIT-XL/2 [41]
Diff.
599
675M
525
3.04
240.8
0.84
0.54
U-ViT-L/4 [2]
Diff.
400
287M
77
4.67
213.3
0.87
0.45
U-V [2]
Diff.
400
501M
133
4.05
263.8
0.48
DIFFUSSM-XI [64]
Diff.
236
673M
1066
3.41
255.0
0.49
DiffTT [14]
Diff.
800
561M
2.67
252.1
0.83
0.55
DIMR-XI/3R (Ours)
Diff.
400
525M
206
3.23
285.1
0.82
0.54
DiMR-XL/3R (Ours)
Dilf.
800
525M
206
2.89
289.8
0.83
0.55
consistent and significant improvements demonstrate the capability the proposed multi-Resolution Network and TD-LN in enhancing diffusion models generate high-fidelity images.
ImageNet 256 x 256. We compare DiMR with state-of-the-art generative models on ImageNet 256 256 with classifier-free guidance in Tab. 7. Compared to U-ViT [2] in fair setting, DiMR-XLL XL/2R with 505M parameters, trained for 400 epochs, significantly outperforms U- U-VIT-H/2 with 501M parameters, also trained for 400 epochs. by 0.52 in FID and 21.8 8 in IS. In comparison with the recently popular diffusion model DiT [41]], DiMR- XL/2R trained for 800 epochs consistently outperforms DİT-L/2 with 458M parameters by 3.32 in FID and 121.8 21.8.1m in IS. DiMR-XL/2R surpasses the larger variant DiT-XL/" with 675M parameters by 0.57 FID and 11.8 IS. Notably, DiT models require training for 1399 epochs, while DiMR-XL/2R achieves superior performance with only 800 epochs. Furthermore, scaling up to DiMR-G/2R sets a new state-of-the-art for ImageNet 256 256 image generation, achieving FID of 1.63 and IS of 292.5.
ImageNet 512 X 512. We compare DiMR with state-of-the-ar generative models on ImageNet 512 x 512 with classifier-free guidance in Tab. 8. Under the same training setting for 400 epochs, DiMR-XL/3R with 525M parameters outperforms U-ViT-H/4 with 501M parameters by 0.82 in FID and 21.3 in IS. Compared to DiT-XL/2 with 675M parameters, DiMR- shows slight improvements of 0.15 in FID and 49.0 in IS. Overall, DiMR-XL/3R demonstrates performance comparable other state-of-the-an generative models for ImageNet 512 x 512 image generation.
E
Additional PCA of Learned Scale and Shift Parameters in adaLN-Zero
We conduct PCA on the learned scale heavy MIP in adal.N-Zero using pre-trained DiT-XL/2 [41] model with 28 layers in depth. To conduct the analysis, we utilize the pre-trained DiT-X1/2 to generate images and collected the scale and shift parameters (tensors) produced by the MLP at different layers along the sampling steps. PCA is then performed on the collected tensors at each layer separately. The results are presented in Fig. 6, where each row of the figure displays the analysis result different depths, from top to bottom: 14, 21, 28. The vertical axis represents the explained variance ratio of the corresponding Principal Components (PCs). As observed. in most cases, the most important principal component can explain most of the variance, while starting from the 3rd principal component, it usually only accounts for less than 5% of the variance. Our observations reveal that the learned parameters, regardless of whether produced by MIP shallower layer a deeper layer, can be largely explained by two principal components, suggesting the potential to approximate them by a simpler function, TD-LN, where the linear interpolation of two learnable parameters is learned introduced in Sec. 4.2.
and shift (81, 32) parameters obtained from a parameter-
17

---

Variance Rañio
Verience Ralio
0.6
8.
9.4
0.
0.0
0.0
10
4
10
10
PCS of Scale
PCS of Scale Y2
PCS of Shift 91
PCS Shift P.
1.0
Varincc Ratio
0.8
Vanance Ratio
Vañance Ratio
0.0
Verince Ratio
0.0
0.
0.6
0.4
0.4
0.
0.2
0.2
0.0
0.0
0.0
10
10
10
PCs of Scale
PCs of Scole Y2
PCs of Shift A
PCs Shift P.
Variance Rabio
0.8
Vanaace Rabio
0.8
Veriance Ratio
0.6
0.6
8.0
0.6
Vaniance
0.4
0.4
0.4
0.2
0.2
0.2
0.
0.0
0.0
0.0
10
10
PCs Scale Y1
PCs of Scale Y2
PCs of Shift fa
PCs Shift P:
1.0
0.0
0.7
Veriance Robio
Variance Rabio
0.5
Jariance
Variance
04
0.3
0.2
02
0.
0.0
0.0
0.0
10
PCs Scale Y
PCS Scale V2
PCs of Shift 5
PCs Shift P:
Figure 6: Principal Component Analysis (PCA) of learned scale and shift parameters in adaLN-
Zero [41]. We conduct PCA the learned scale (71, 72) and shift (81, 82) parameters obtained from a parameter-heavy MLP in adaLN-Zero using pre-trained DiT-XL/2 [41] model with 28 layers in depth. Each row of the figure presents the analysis result at different depths, from top bottom: 14. 21, 28. The vertical axis represents the explained variance ratio of the corresponding Principal Components (PCs). Our observations reveal that the learned parameters can be largely explained by two principal components, suggesting the potential to approximate them by a simpler function.
F
Model Samples
We present samples from our largest variant, DiMR-XL/3R, at 512 x 512 resolution trained for 800 epochs. Fig. 7- 18 display uncurated samples from the model across range of input class labels with classifier-free guidance. It worth noting that our generated image samples exhibit high-quality and minimal image distortions.
G
Limitations
The proposed DiMR has few remaining limitations. First, it focuses on class-conditional image generation, rather than full text-to-image generation approaches. Additionally, although DiMR demonstrates great scalability in our experiments, the exploration of DiMR variants stops at the DiMR-XI/3R model with 224.8M parameters due to constraints on computational resources. In contrast, few recent methods have scaled image diffusion models to billions of parameters. We leave as future work to further explore the scaling law of our proposed DiMR enhance its image generation capabilities.
18

---

Figure 7: Uncurated 512 x 512 DiMR samples. Class label 'arctic wolf' (270)
Figure 8: Uncurated 512 512 DiMR samples. Class label 'volcano' (980)
Figure 9: Uncurated 512 512 DiMR samples. Class label = 'husky' (250)
19

---

Figure 10: Uncurated 512 x 512 DiMR samples. Class label = sulphur-crester cockatoo (89)
Figure Uncurated 512 512 DiMR samples. Class label 'cliff drop-off' (972)
Figure 12: Uncurated 512 x 512 DiMR samples. Class label bballoon' (417)
20

---

Figure 13: Uncurated 512 x 512 DiMR samples. Class label = 'lion' (291)
Figure 14: Uncurated 512 x 512 DiMR samples. Class label 'otter' (360)
Figure 15: Uncurated 512 x 512 DiMR samples. Class label = 'red panda' (387)
21

---

Figure 16: Uncurated 512 x 512 DiMR samples. Class label 'panda' (388)
Figure 17: Uncurated 512 x 512 DiMR samples. Class label 'coral reef" (973)
Figure 18: Uncurated 512 512 DiMR samples. Class label 'macaw' (88)
22

---

H
Broader Impacts
The proposed DiMR has the potential to facilitate numerous fields through its advanced image generation capabilities. In the realm of creative industries, DiMR can enhance the efficiency and creativity of artists and designers by generating eigh-fidlitty images with fewer distortions. The high-quality generated images can also contribute to research on synthetic datasets by creating realistic images, aiding in reducing the annotations required for training vision models. However, with these advancements come ethical considerations, such the risk of generating deepfakes or other malicious content, thus crucial to implement safeguards minimize potential harms.
I Safety Concerns and Safeguards
Given the powerful capabilities of DiMR, it is essential to implement robust safeguards address potential safety and ethical concerns. One primary concern is the misuse generated content, such as the creation of deepfakes, which can lead to misinformation and privacy violations. To mitigate this, it important to establish strict access controls and usage policies to prevent the misuse of these models when released. Transparency in the training data and model architecture is also critical ensure accountability and to identify potential biases that could lead to harmful outputs. By prioritizing these safeguards, we can ensure the responsible of DiMR while minimizing potential risks.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. 1 for the main claims of the paper and Sec. 4 and Sec. 5 for their illustration and validation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec. G for the discussion the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Sec. 5.1 and Tab. 5 for all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We have open-sourced the full code and trained model weights after thorough cleaning and the implementation of necessary safeguards.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. ca/
httpr//uuiiess/ooeuSuiissse onoooicooollyloooooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.cc..occc wwtpppcwccgeeeeeeccceccccceeeeee wtwggg/iicccdssai wwt.SSummisiiooooii asiionylicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Sec. 5.1 and Tab. 5 for all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following the literature [2, 41, 64, 14, 57], error bars are not reported to provide statistical significance, the metrics FID, IS, Precision, and Recall inherently account for by being computed while sampling 50K images.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Tab. for all the information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://neurippe cc cwtppblic//eteeeeeeeeeeeeeeeeeee httpbliie/thiccccuulelllllee
Answer: [Yes]
Justification: The research conducted in the paper conform the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Sec. H for the discussion on the broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Sec. for the discussion on the safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Sec. A for dataset licenses.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "eRgfA8sts7", "reviewer": "Reviewer_1gRs", "summary": "This paper works on efficient diffusion model backbone architecture by using transformer and ConvNeXt architecture respectively on small and large resolutions of the same inputs, to leverage the strength of both architectures, and alleviate the distortion problem.", "questions": "Are there any ablation studies on the number of different resolutions used? For instance, what happens if we use one small resolution with DiT and one normal resolution with a larger conv net?", "limitations": "See the weakness section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to read and follow.\n\n2. The idea is simple and effective.\n\n3. They conduct rich experiments to support the idea.", "weaknesses": "1. This work utilizes two standard architectures, incorporating only minor design elements such as the TD-LN. It would be more advantageous to propose a new, general, and elegant architecture that effectively combines the strengths of the attention layer and the convolution layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments, and we carefully address the concerns below.\n\n&nbsp;\n\n> W1: This work utilizes two standard architectures; more advantageous to propose a new, general, and elegant architecture that combines both.\n\nWe thank the reviewer for the suggestion, which we fully agree with. Additionally, we believe that our proposed DiMR is one step towards the goal, by effectively combining the strengths of the attention and convolution layers. Specifically,\n\n1. ***DiMR is clean and elegant:*** As shown in Fig. 2 of the paper, DiMR only uses the standard multi-head attention layers, depthwise convolutions, and MLP layers, along with the proposed TD-LN layers. No other complex designs or operations are involved. As a result, DiMR showcases a straightforward and effective network design.\n\n2. ***DiMR is general:*** Since only standard operations are employed in DiMR, users can design their own DiMR variant by simply changing the types, numbers, or orders of layers. Additionally, as demonstrated in Table A, DiMR also supports the variant of pure convolutions (i.e., the lowest-resolution branch also uses ConvNeXt blocks).\n\n***Table A**: DiMR supports arbitrary combination of different architectures at different branches. Experiments on ImageNet-256. We explain the performance gap between 2R and 3R in Q1 (Table B).*\n\n|                        | 1st branch (lowest resolution) | 2nd branch | 3rd branch | FID50K w/o CFG | FID50K w/ CFG |\n|------------------------|--------------------------------|------------|------------|----------------|---------------|\n| DiMR-XL/2R (pure conv) | ConvNeXt                       | ConvNeXt   | -          | 5.75           | 2.09          |\n| DiMR-XL/2R             | Transformer                    | ConvNeXt   | -          | 4.87           | 1.77          |\n| DiMR-XL/3R             | Transformer                    | ConvNeXt   | ConvNeXt   | 5.58           | 1.98          |\n\n3. ***DiMR is novel and effective:*** To the best of our knowledge, DiMR is the first work that has successfully combined transformer and convolution architectures into a single **multi-resolution** diffusion model. DiMR demonstrates the effectiveness of combining both architectures, resulting in state-of-the-art performance without any bells and whistles (even surpassing Large-DiT 3B parameters by a large margin: 1.70 vs. 2.10 on ImageNet-256 benchmark, while DiMR only uses 505M parameters).\n\n4. ***TD-LN is crucial, not minor:*** As demonstrated by the systematic analysis in the paper, TD-LN is a parameter-efficient approach that effectively injects time information into the model. We emphasize that other reviewers appraise the proposed TD-LN: Reviewer MPTv acknowledges that TD-LN is **particularly interesting, well-motivated, and innovative**, while Reviewer Wwps also thinks that TD-LN is **more efficient and effective than AdaLN-zero, due to the removal of the cumbersome MLP layer**.\n\n&nbsp;\n\n> Q1: Are there any ablation studies on the number of different resolutions used?\n\nWe thank the reviewer for the suggestion. It is noteworthy that using different number of resolutions affects the trade-off between generation performance and model speed (both inference and training time). As a result, we use two resolutions (denoted as 2R in the paper) and three resolutions (i.e., 3R) for ImageNet-256 and ImageNet-512 benchmarks, respectively.\n\nBelow, we present a careful ablation study on the number of resolutions. Note that for ImageNet-512 generations, we could not complete the full experiment within the short rebuttal period. Therefore, we followed U-ViT and reported the FID10K at every 100K iterations (i.e., 80 training epochs). Our findings indicate that one transformer branch combined with one convolution branch (i.e., 2R) usually achieves the best results. However, the Gflops also increases (since the transformer branch is operated on a higher resolution than the 3R counterpart). Conversely, employing more blocks  at smaller resolutions (i.e., 3R) reduces the computational burden but slightly degrades the performance. We will include these experiments in the revised version.\n\n***Table B**: Ablation of different numbers of resolutions on ImageNet-256*\n\n|            | Epoch | #Params. | Gflops | FID-50K w/o CFG | FID-50K w CFG |\n|------------|-------|----------|--------|-----------------|---------------|\n| DiMR-XL/2R | 400   | 505M     | 160    | 4.87            | 1.77          |\n| DiMR-XL/3R | 400   | 502M     | 48     | 5.58            | 1.98          |\n\n***Table C**: Ablation of different numbers of resolutions on ImageNet-512*\n\n|            | #Params. | Gflops | FID-10K w/o CFG at 80 epochs | FID-10K w/o CFG at 160 epochs | FID-10K w/o CFG at 240 epochs |\n|------------|----------|--------|------------------------------|-------------------------------|-------------------------------|\n| DiMR-XL/2R | 515M     | 619    | 17.56                        | 13.68                         | 12.95                         |\n| DiMR-XL/3R | 525M     | 206    | 18.74                        | 14.84                         | 13.90                         |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mgm4DAtigw", "reviewer": "Reviewer_MPTv", "summary": "This paper introduces a diffusion model named DiMR, which incorporates a Multi-Resolution Network and Time-Dependent Layer Normalization to enhance image generation quality. Traditional diffusion models, often limited by a trade-off between computational efficiency and visual fidelity, struggle with image distortion due to the coarse resolution of input data processing in Transformer-based designs. DiMR addresses this by refining image details progressively across multiple resolutions, reducing distortions significantly. The new light-weight Time-Dependent Layer Normalization technique introduced in the model embeds time-based parameters into the normalization process efficiently. Demonstrated through benchmarks on ImageNet 256x256 and ImageNet  512x512, DiMR outperforms competing models.", "questions": "- This work appears to be a cascade diffusion model based on DiT. How does it perform compared to traditional UNet-based cascade diffusion models?\n- What are the training and convergence speeds of this model, particularly in comparison to the baselines?\n- The proposed method adopted the class token similar to that used in U-ViT for injecting class conditions. Is it possible to replace this with an approach similar to Time-Dependent Layer Normalization (TD-LN)?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and easy to follow, featuring clear diagrams and detailed captions that enhance understanding.\n+ The introduction of Time-Dependent Layer Normalization (TD-LN) is particularly interesting. The use of PCA analysis to justify this approach provides a strong motivation and highlights its innovativeness.\n+ The experimental section of the paper effectively demonstrates the effectiveness of the proposed method.", "weaknesses": "- The model has a higher FLOPs count compared to a DiT of similar parameter size, raising concerns about slower training speeds.\n- It is unclear whether this cascaded approach can support multi-resolution training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments, and we carefully address the concerns below.\n\n&nbsp;\n\n> W1: The model has a higher FLOPs count compared to a DiT of similar parameter size, raising concerns about slower training speeds.\n\nThanks for the comment. To address it, we provide a training speed analysis in Table D. We note that on ImageNet-256, when comparing with DiT-XL/2, our DiMR-XL/2R only increases the computation by 41 Gflops, which is relatively negligible. Additionally, we also experiment with a much faster DiMR variant (i.e., DiMR-XL/3R) on ImageNet-256. As shown in the following table, DiMR-XL/3R, even with much smaller Gflops, still surpasses DiT-XL/2 by a large margin.\n\n***Table D**: Comparison of training speed on ImageNet-256. For training speed, we test all models with batch size of 256 on 8 A100s.*\n|                   | #Params. | Gflops | FID-50K w/o CFG | FID-50K w CFG | Total training time |\n|-------------------|----------|--------|-----------------|---------------|---------------------|\n| DiT-XL/2          | 675M     | 119    | 9.62            | 2.27          | 24.8 days           |\n| Large-DiT-3B      | 3B       | 928    | -               | 2.10          | -                   |\n| DiMR-XL/2R (ours) | 505M     | 160    | 4.87            | 1.77          | 11.8 days           |\n| DiMR-XL/3R (ours) | 502M     | 48     | 5.58            | 1.98          | 10.0 days           |\n\nFurthermore, on ImageNet-512, our DiMR-XL/3R is more compute-efficient than DiT-XL/2. Specifically, our DiMR-XL/3R has ***206*** Gflops while DiT-XL/2 has ***525*** Gflops, indicating that we are more than 2x faster at each forward pass, making ours much more training efficient for high-resolution image generations.\n\n&nbsp;\n\n> W2: Can this cascaded approach support multi-resolution training?\n\nWe thank the reviewer for the interesting idea. Our DiMR can actually perform multi-resolution training and enable multi-resolution generation with a single model. Due to the limited time of the rebuttal period, we only tried two different resolutions: 256x256 and 512x512 multi-resolution training, and we report the results in Table E. We observe that the same DiMR-XL/2R model can generate both 256x256 and 512x512 images while achieving good FID scores. For reference, we also reported the best DiT model at each resolution. Our model is comparable (even superior) to the best resolution-specific DiT models. \n\n***Table E**: Multi-resolution generation. For DiMR-XL/2R, we use the same model to generate 50K images for each resolution and compute the FID-50K score. For DiT-XL, we report their best model trained on each specific resolution.*\n|                                               | #Params. | Gflops | FID-50K on 256x256 | FID-50K on 512x512 |\n|-----------------------------------------------|----------|--------|--------------------|--------------------|\n| DiMR-XL/2R （multi-resolution generation）    | 505M     |  160   | 1.79               | 3.18               |\n| DiT-XL/2 （single 256-resolution generation） | 675M     |  119   | 2.27               | x                  |\n| DiT-XL/2（single 512-resolution generation）  | 675M     | 525    | x                  | 3.04               |\n\n&nbsp;\n\n> Q1: How does DiMR perform compared to traditional UNet-based cascade diffusion models?\n\nAs discussed in the paper, DiMR adopts a framework of *feature cascade*, instead of *image cascade* (like traditional UNet-based cascade diffusion models). Specifically, the *image cascade* approaches (e.g.,  Cascaded Diffusion Models) first generate low-resolution images by a base diffusion model, and then improve the details in the subsequent super-resolution diffusion models. By contrast, DiMR generates feature maps progressively from low-resolution to high-resolution, ***all within a single model***.\n\nBelow, Table F provides the results compared to the cascade diffusion models, i.e., Cascaded Diffusion Model (CDM) and Matryoshka Diffusion Model (MDM), a special type of cascaded diffusion model where multi-resolution images are generated at the same time via a Nested U-Net. Note that they only report results on ImageNet-256 generations. As shown in the table, our model surpasses them by a large margin.\n\n***Table F**: Comparison with cascade diffusion models on ImageNet-256.*\n|      | FID-50K without CFG | FID-50K with CFG |\n|------|---------------------|------------------|\n| CDM  | -                   | 4.88             |\n| MDM  | 8.92                | 6.62             |\n| DiMR-XL/2R (ours) | 4.50                | 1.70             |\n\n&nbsp;\n\n> Q2: What are the training and convergence speeds of this model, particularly in comparison to the baselines?\n\nPlease refer to W1 for the detailed discussion.\n\n&nbsp;\n\n> Q3: The proposed method adopted the class token similar to that used in U-ViT for injecting class conditions. Is it possible to replace this with an approach similar to Time-Dependent Layer Normalization (TD-LN)?\n\nWe thank the reviewer for the interesting question. We experimented with this idea, but it did not work. In the formulation of TD-LN (Equ (5) in the paper), the time token is considered as a scalar, instead of an embedding vector as in adaLN (Equ (3) in the paper). The diffusion (or denoising) process is a monotonical function of time, allowing us to easily encode time as a scalar. On the other hand, the class token contains more complex information, and therefore simply encoding it as a scalar in our Equ (5) does not work. In the end, we resort to U-ViT's simple yet effective strategy by feeding the class tokens into the transformer branch. A more careful exploration (e.g., adding a lightweight MLP for the class tokens) may make it work, and we leave it for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtJe6VBjfy", "reviewer": "Reviewer_RADT", "summary": "This paper proposes to replace the original pull transformer blocks in DiT with multi-resolution network. Detailly, transformer block is employed for low resolution and Conv blocks are used for the remaining higher resolution. An additional time-conditioning block is designed for Conv blocks. The effectiveness of this method is validated with numercial experiments.", "questions": "1. It is suggested to identify the real core contribution for performance gain of this paper. The ablation study of replacing the transformer block with conv block is suggested.\n2. The effectiveness of this method on larger image size and networks are concerened. The authors are suggested to give furhter discussions on these points if they wish a full surpass of the original DiT.", "limitations": "The authors mask well discussions on the limitaion of this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a multi-resolution network, consisting of both transformer and conv blocks, to boost the performamce of image generation. Such design is rational. To fit the conv block, this paper also introduce an additional time conditioning block.\n2. This method achieves better performance than DiT.", "weaknesses": "1. It is unclear what the performance gain comes from. intuitively, the multi-resolution design could already brings better performance. But this paper proposes the conbination of transformer and conv. I wonder if the design of transformer in low resolution really necessary.\n2. The scalibility of this network may be limited due to the introduction of conv blocks. For example, for network parameters larger than 1B.\n3. This paper mainly claims to solve the large computional complexityinduced by the long token length. But in the real long token length case (image 512x512), the perfomance gains of this method is marginal compared to DiT. This raises the concern of the real performance of this method on large image case, eg. >512x512.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments, and we carefully address the concerns below.\n\n&nbsp;\n\n> W1: It is unclear what the performance gain comes from. Is the design of transformer in low resolution really necessary?\n\nWe thank the reviewer for the suggestion. Below, our experiments in Table G demonstrate that ***using transformer blocks at the lowest resolution is critical for achieving good performance while maintaining efficiency.*** Replacing the transformer blocks in the lowest-resolution results in a DiMR variant that has multi-resolution design with ***pure*** convolution blocks. However, it performs worse than combing transformer blocks with convolution blocks at different resolutions.\n\n***Table G**: Ablation of different architectures on ImageNet-256.*\n\n|                                 | 1st branch (lowest resolution | 2nd branch | Epoch | FID-50K w/o CFG | FID-50K w CFG |\n|---------------------------------|-------------------------------|------------|-------|-----------------|---------------|\n| DiMR-XL/2R (pure conv)          | ConvNeXt                      | ConvNeXt   | 400   | 5.75            | 2.09          |\n| DiMR-XL/2R (transformer + conv) | Transformer                   | ConvNeXt   | 400   | 4.87            | 1.77          |\n\n&nbsp;\n\n> W2: The scalibility of this network may be limited due to the introduction of conv blocks. For example, for network parameters larger than 1B.\n\nWe thank the reviewer for the suggestion. To verify it, we trained a 1B variant of DiMR (i.e., DiMR-H/3R) on ImageNet-512 during the rebuttal period. ***As shown in Table H below, performance significantly improves when increasing the model size from 505M to 1.02B, demonstrating the scalability of DiMR.*** We note that DiMR-H/3R has been trained with only 300 epochs due to limited GPUs and the short rebuttal period. However, we still observe improvement as training progresses. We emphasize that this experiment was conducted only once due to time and GPU constraints. Therefore, better performance may be achieved with a more careful network design and optimized training hyperparameters.\n\n***Table H**: Scaling up DiMR on ImageNet-512*\n|             | Epoch | #Params. | Gflops | FID-50K w/o CFG | FID-50K w CFG |\n|-------------|-------|----------|--------|-----------------|---------------|\n| DiMR-XL/3R  | 400   | 525M     | 206.1  | 8.56            | 3.23          |\n| DiMR-H/3R   | 300   | 1.03B    | 399.6  | 7.74            | 2.86          |\n\n&nbsp;\n\n> W3: In the real long token length case (image 512x512), the performance gains of this method are marginal compared to DiT, eg. >512x512.\n\nWe thank the reviewer for the comment. To clarify, on ImageNet-512, we adopt a three-branch design, DiMR-XL/3R, where the transformer branch has a patch size of 4. This design strikes a better balance between accuracy and speed, following U-ViT-H/4. However, the design may lead to inferior performance to a model using a patch size of 2, like DiT-XL/2. As a result, we think ***U-ViT-H/4 is a more proper baseline for DiMR-XL/3R (both use the same patch size for transformer branch), as the patch size significantly affects generation accuracy.*** It is also evidenced by the Table 4 of DiT paper, where **DiT models with a patch size of 4 are 1.47 to 2.21 times worse than the same models with a patch size of 2** (e.g., DiT-XL/4 43.01  vs. DiT-XL/2 19.47 FID-50k w/o CFG). Therefore, DiT-XL/2 needs to handle heavy computational burdens to achieve good results. Specifically, ***on ImageNet-512, DiT-XL/2 is 2.55 times slower than our DiMR-XL/3R per forward pass (525 Gflops vs. 206 Gflops under similar model parameters).*** Nevertheless, DiMR-XL/3R still outperforms the best and heaviest DiT-XL/2 by 0.15 FID, and surpasses the proper baseline U-ViT-H/4 by 1.16 FID.\n\nIn addition, our model can also be improved by reducing the patch size and increasing the computational complexity. During the rebuttal period, we also tried 512 generation with 2 branches (which equal to a patch size of 2). We report the FID-10K score during training in Table I. We can observe that we can further improve performance, but it also increases computational complexity.\n\n***Table I**: ImageNet-512 generation with 2 branches (DiMR-XL/2R) will further improve performance*\n|            | #Params. | Gflops | FID-10K w/o CFG at 80 epochs | FID-10K w/o CFG at 160 epochs | FID-10K w/o CFG at 240 epochs |\n|------------|----------|--------|------------------------------|-------------------------------|-------------------------------|\n| DiMR-XL/3R | 525M     | 206    | 18.74                        | 14.84                         | 13.90                         |\n| DiMR-XL/2R | 515M     | 619    | 17.56                        | 13.68                         | 12.95                         |\n\n&nbsp;\n\n> Q1: Identify the real core contribution for performance gain of this paper. The ablation study of replacing the transformer block with conv block is suggested.\n\nPlease refer to W1 for the detailed discussion.\n\n&nbsp;\n\n> Q2: The effectiveness of this method on larger image size and networks are concerned.\n\nPlease refer to W3 for the detailed discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "susUbaYpLt", "reviewer": "Reviewer_Wwps", "summary": "The paper proposes a multi-resolution network for diffusion model which emphasizes the learning features across resolutions. The method further introduces a time-dependent layer norm to boost the model performance with fewer parameters compared to AdaLN in DiT. This proposed network demonstrates a SoTA performance on ImageNet at 1.70 and 2.89 on ImageNet 256 & 512, respectively.", "questions": "N/A", "limitations": "As pointed out in Weaknesses, the method still has a certain distortion rate which is due to the use of strided convolution (a form of patchification) at the start of each branch. I think the author should include this in the revised manuscript.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses image distortion caused by varying patch sizes by integrating a multi-resolution structure into the network. The choice of patch sizes involves a trade-off between computational complexity and model performance: larger patch sizes reduce computational complexity but degrade performance. \n- The method finds a balance between compute complexity and performance by introducing a multi-branch architecture and multi-resolution loss to decompose the learning process from low to high resolution. The idea is a substantial contribution. Through quantitative and qualitative results, the method can alleviate distortion of image generation and achieve SoTA FID scores.\n- The time-dependent layer norm is a simplified form of AdaLN with less intensive parameters by removing the cumbersome MLP layer and rearranging class embedding and time embedding for parameter-efficient time conditioning.", "weaknesses": "- The discussion between the method and cascaded methods like \"Cascaded diffusion models\" and \"Matryoshka diffusion models\" should be included to highlight the advantages of the method. \n- In table 1, it needs to be updated with some latest SoTa methods like MDTv2 with the best FID of 1.58 and PaGoDA with the best FID of 1.56 on ImageNet 256x256. Not to compete with them, but to get an overview picture of the latest methods.\n- The proposed multi-scale diffusion loss is also introduced in SimpleDiffusion. So, the authors need to mention the difference in the paper. \n- Equation 2: have the authors used concat instead of adding the upsampling features directly to the larger-resolution features of the next branch?\n- The current method only uses the output features of the previous branch and injects them into the start of the next branch. However, this design lacks interconnections between blocks from the low-resolution and high-resolution branches. Ideally, there should be skip-connections across branches. What is the motivation behind this? If authors have not tested this, it is encouraged to do so.\n- In table 2, what is the reason why using AdaLN-Zero with Multi-branch (row 3) causes a bad result? \n- The root problem of distortion boils down to the use of patch size which is the main motivation of the work. However, in the design of multiscale network, the authors also patchify the input image to the corresponding resolution for each branch. I wonder why the author do not use different-resolution noisy images as the input to each branch like Matryoshka Diffusion Models. I think this is more effective to completely remove distortion problem. As shown in Figure 5, the method still exhibits a certain level of distortion rates. \n- Sampling speed: it is valid to include a comparison of sampling time with baselines.\n- Figure 3: What is the red line meaning?\n\nMisc: L37, L130: the figure 7 of DiT paper is incorrectly linked.\n\nRef: \n- Gao, Shanghua, et al. \"MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer.\" arXiv preprint arXiv:2303.14389 (2023).\n- Kim, Dongjun, et al. \"PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher.\" arXiv preprint arXiv:2405.14822 (2024).\n- Hoogeboom, Emiel, Jonathan Heek, and Tim Salimans. \"simple diffusion: End-to-end diffusion for high resolution images.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments, and we carefully address the concerns below.\n\n&nbsp;\n\n> W1: The discussion between the method and cascaded methods like \"Cascaded diffusion models\" and \"Matryoshka diffusion models\" should be included to highlight the advantages of the method.\n\nThank you. We will add the following paragraph to the related work section.\n\n  ***Cascaded diffusion models.** To generate images with high resolution while solving the large computational complexity, it is commonly to use Cascaded diffusion models [a,b,c], where a first diffusion model is used to generate images at lower resolution, and then one or multiple diffusion models are used to gradually generate the super-resolution versions of the initial generation. Recently, Gu et al [d] propose using one diffusion model to generate images at different resolutions at the same time via a Nested U-Net and a progressive training recipe. However, the key insight of their multi-resolution design is that high-resolution images can be generated with smaller and affordable models if the low-resolution generation is used as conditioning. In contrast, we observe that different architectures (e.g., transformer and convolution layers) have different behaviors (such as performance, speeds, etc) at different resolutions. Therefore, we propose a multi-resolution network design in the ***feature space***, and use different architectures to handle features at different resolutions. This enables us to utilize the advantages of different architectures within a single diffusion model, and achieve the best performance while maintaining model efficiency.*\n  - *[a] Ho, Jonathan, et al. \"Cascaded diffusion models for high fidelity image generation.\"*\n  - *[b] Ramesh, Aditya, et al. \"Hierarchical text-conditional image generation with clip latents.\"*\n  - *[c] Saharia, Chitwan, et al. \"Photorealistic text-to-image diffusion models with deep language understanding.\"*\n  - *[d] Gu, Jiatao, et al. \"Matryoshka diffusion models.\"*\n\n&nbsp;\n  \n> W2: In table 1,  update with some latest SoTa methods.\n\nThanks for the suggestion. We will add them to the full table. However, it is noteworthy that these methods propose new strategies to train diffusion models, such as using mask latent modeling schemes or progressive training, which are orthogonal to the main focus of this work (a multi-branch network design). We believe that these strategies can also be used to train our model and achieve better performance. Additionally, PaGoDA was on arXiv (May 23, 2024) after our submission to NeurIPS.\n\n&nbsp;\n\n> W3: The proposed multi-scale diffusion loss is also introduced in SimpleDiffusion. So, the authors need to mention the difference in the paper.\n\nThanks for the suggestion. We already cited SimpleDiffusion in L230, and will discuss more in the revision. In short, the multi-scale training loss in SimpleDiffusion is used to balance the loss on different levels of details when training diffusion models for high resolutions. The multi-scale training loss used in our DiMR is to supervise different branches at different resolutions. Another crucial difference is the underlying denoising backbone, where SimpleDiffusion uses U-Net, and DiMR uses a multi-branch network.\n\n&nbsp;\n\n> W4: Have the authors used concat instead of adding the upsampling features directly to the larger-resolution features of the next branch?\n\nWe actually considered both strategies at the early stage of this project. We found that concatenation and upsampling make no difference experimentally. They achieve very similar performances as shown in Table J.\n\n***Table J**: Comparison between upsampling and concatenation on ImageNet-256*\n|                             | Epoch | #Params. | Gflops | FID-50K w/o CFG | FID-50K w CFG |\n|-----------------------------|-------|----------|--------|-----------------|---------------|\n| DiMR-XL/2R w/ upsampling    | 400   | 505M     | 160    | 4.87            | 1.77          |\n| DiMR-XL/2R w/ concatenation | 400   | 506M     | 161    | 5.01            | 2.06          |\n\n&nbsp;\n\n> W5: Ideally, there should be skip-connections across branches. What is the motivation behind this? If authors have not tested this, it is encouraged to do so.\n\nThanks for the interesting suggestion. We have experimented with skip-connections, as detailed in Table K. Specifically, we connect the input of each layer in the latter branch with the output of the corresponding layer in the former branch using a skip-connection block. This block consists of two layer normalization layers with a pixel shuffle upsampling layer in between. Our results indicate that skip-connections have a marginal, or slightly negative, influence on our DiMR model.\n\n***Table K**: Ablation on skip connections*\n|            | skip-connections | Epoch | #Params. | Gflops | FID-50K w/o CFG | FID-50K w CFG |\n|------------|------------------|-------|----------|--------|-----------------|---------------|\n| DiMR-XL/2R | NO               | 400   | 505M     | 160    | 4.87            | 1.77          |\n| DiMR-XL/2R | YES              | 400   | 542M     | 165    | 4.45            | 1.96          |\n\n&nbsp;\n\n> W6: In table 2, what is the reason why using AdaLN-Zero with Multi-branch (row 3) causes a bad result?\n\nOur observation is that AdaLN-Zero may not perform well on convolution layers, probably because MLP considers data in 1D, which may not handle the 2D structure of the convolution layers very well. Also, our multi-branch design (w/ transformer and convolution layers) may require a more careful tuning of AdaLN-Zero, which was originally designed for DiT (single-branch and pure transformer layers).\n\nAdditionally, some other papers (e.g., see Sec. 3.1 and Fig. 2b in U-ViT paper [e]) also find that AdaLN does not perform well.\n\n*[e] Bao, Fan, et al. \"All are Worth Words: A ViT Backbone for Diffusion Models.\"*\n\n&nbsp;\n\n**We address Weakness 7-10 and Limitation 1 in \"Part2 of our rebuttal to Reviewer Wwps\"**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and ACs,

&nbsp;

We thank all reviewers for their valuable comments and feedback, mentioning that our method is "**simple and effective**" (Reviewer 1gRs and MPTv), which "**alleviates distortion of image generation and achieves SoTA FID scores**" (Reviewer Wwps). Additionally, we are glad that they find the proposed TD-LN is "**particularly interesting with strong motivation and innovativeness**" (Reviewer MPTv) and "**simple with less intensive parameters**" (Reviewer Wwps).

&nbsp;

We address all the comments and questions from the reviewers in the rebuttals below, and provide detailed explanations and experimental results.

&nbsp;

For quick reference, we list new experiments provided in this rebuttal below:

- Table A shows that **DiMR supports arbitrary combination of different architectures at different branches.**
- Table B/C ablate different numbers of resolutions on ImageNet-256/512, respectively.
- Table D **demonstrates the advantage in training speed compared to the baselines.**
- Table E shows that **DiMR is capable of multi-resolution generation and is comparable to (or even superior to) the best resolution-specific DiT models.**
- Table F compares DiMR with cascade diffusion models.
- Table G ablates different architectures (pure convolution vs. transformer + convolution)
- Table H **shows the results of the 1B variant of DiMR on ImageNet-512, demonstrating the scalability of DiMR.**
- Table I shows that **increasing Gflops of DiMR will further improve performance on ImageNet-512.**
- Table J compares the difference between upsampling and concatanation.
- Table K ablates the skip connections.
- Table L **demonstrates the advantage in sampling speed compared to the baselines.**

&nbsp;

We thank the reviewers for the constructive comments, and we will include all the new results and discussions in the final version.

&nbsp;

Best,

Authors

Meta-review: The paper presents a multiresolution architecture that combines Transformer-based layers on low-resolution branches and ConvNeXt-based layers on high-resolution branches. The paper also makes an interesting observation on adaLN layers in diffusion models and proposes a simple parameter-efficient time-dependent adaLN. Overall the reviewers have found the paper well-written and acknowledged that the exhaustive experimental studies in the paper provide compelling evidence on the efficacy of the proposed architecture. The rebuttal has addressed most remaining concerns well, and thus we are happy to recommend acceptance at NeurIPS.

We strongly recommend incorporating reviewers' suggestions and the new experimental results in the final camera-ready version (or its appendices). Please also update your Table 1 with results from MDTv2 and DiffiT.
Decision: Accept (poster)