Title: Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Generation
Abstract: This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization.
Diffusion models have gained prominence for their effectiveness in high-fidelity image generation.
While conventional approaches rely on convolutional U-Net architectures, recent Transformer-based designs have demonstrated superior performance and scalability.
However, Transformer architectures, which tokenize input data (via "patchification"), face a trade-off between visual fidelity and computational complexity due to the quadratic nature of self-attention operations concerning token length.
While larger patch sizes enable attention computation efficiency, they struggle to capture fine-grained visual details, leading to image distortions.
To address this challenge, we propose augmenting the **Di**ffusion model with the **M**ulti-**R**esolution network (DiMR), a framework that refines features across multiple resolutions, progressively enhancing detail from low to high resolution.
Additionally, we introduce Time-Dependent Layer Normalization (TD-LN), a parameter-efficient approach that incorporates time-dependent parameters into layer normalization to inject time information and achieve superior performance.
Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, where DiMR-XL variants surpass previous diffusion models, achieving FID scores of 1.70 on ImageNet $256 \times 256$ and 2.89 on ImageNet $512 \times 512$. Our best variant, DiMR-G, further establishes a state-of-the-art 1.63 FID on ImageNet $256 \times 256$.

Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization

Abstract

This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization. Diffusion models have gained prominence for their effectiveness in high-fidelity image generation. While conventional approaches rely on convolutional U-Net architectures, recent Transformer-based designs have demonstrated superior performance and scalability. However, Transformer architectures, which tokenize input data (via 'patchification'), face a trade-off between visual fidelity and computational complexity due to the quadratic nature of self-attention operations concerning token length. While larger patch sizes enable attention computation efficiency, they

struggle to capture fine-grained visual details, leading to image distortions. To address this challenge, we propose augmenting the Di ffusion model with the M ultiR esolution network (DiMR), a framework that refines features across multiple resolutions, progressively enhancing detail from low to high resolution. Additionally, we introduce Time-Dependent Layer Normalization (TD-LN), a parameter-efficient approach that incorporates time-dependent parameters into layer normalization to inject time information and achieve superior performance. Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, where DiMR-XL variants surpass previous diffusion models, achieving FID scores of 1.70 on ImageNet 256 × 256 and 2.89 on ImageNet 512 × 512 . Our best variant, DiMR-G, further establishes a state-of-the-art 1.63 FID on ImageNet 256 × 256 .

1 Introduction

Diffusion and score-based generative models [23, 53, 55, 19, 54] have demonstrated promising results for high-fidelity image generation [7, 38, 44, 46, 48]. These models generate images through an iterative process of gradually denoising Gaussian random noise to create realistic samples Central to this process is a neural network, tasked with denoising the inputs through a mean squared error loss function. Traditionally, U-Net architectures [47] (enhanced with residual blocks [15] and selfattention blocks [59] at lower resolution) have been prevalent. However, recent advancements have introduced Transformer-based designs [59, 8], offering superior performance and scalability.

In practice, Transformer-based architectures face the challenge of balancing visual fidelity and computational complexity, primarily stemming from the self-attention operation and the patchification process employed for downsampling inputs [8] ( i.e ., a smaller patch size results in better visual fidelity at the cost of a longer token length and thus more computational complexity by the selfattention operation). The quadratic complexity inherent in self-attention concerning token length necessitates larger patch sizes to facilitate more efficient attention computations. However, the adoption of large patch sizes inevitably compromises the model's capacity to capture finer visual details, resulting in image distortion ( i.e ., low visual fidelity). This dilemma prompts DiT [41] to conduct a systematic study on the impact of patch size on image distortion, as depicted in Fig. 7 of their paper. Consequently, they settled on a patch size of 2 for their final design. Similarly, U-ViT [2] opted for a patch size of 2 for input sizes of 256 × 256 and a patch size of 4 for 512 × 512 images, effectively balancing the token length for different image sizes. Despite these meticulous adjustments, the generated results still exhibit discernible image distortion, as illustrated in Fig. 1.

One simplistic solution to mitigate image distortion in Transformer-based architectures is adopting a patch size of 1, but this significantly increases computational complexity. Instead, inspired by the success of image cascade [20, 48] which generate images at increasing resolutions, we propose a feature cascade approach that progressively upsamples lower-resolution features to higher resolutions, alleviating distortion in image generation. In this study, we present DiMR, which enhances the Di ffusion model with a M ultiR esolution network. DiMR tackles the challenge of balancing visual detail capture and computational complexity through improvements in the denoising backbone architecture. We employ a multi-resolution network design that comprises multiple branches to progressively refine features from low to high resolution, preserving intricate details within the input data. Specifically, the first branch handling the lowest resolution incorporates Transformer blocks [59], leveraging the superior performance and scalability observed in prior works [2, 41], while the remaining branches utilize ConvNeXt blocks [35], which are efficient for high resolution features. The network processes inputs progressively from the lowest resolution, with additional features from the preceding resolution. The last branch refines features at the same spatial resolution as the input, effectively mitigating image distortion arising from the patchification.

Additionally, we observe that existing time conditioning mechanisms [42, 25, 7], such as adaptive layer normalization (adaLN) [41], are parameter-intensive. In contrast, we propose a more efficient approach, Time-Dependent Layer Normalization (TD-LN), that integrates time-dependent parameters directly into layer normalization [1], achieving superior performance with fewer parameters.

To demonstrate its effectiveness, we evaluate DiMR on the class-conditional ImageNet generation benchmark [6]. On ImageNet 64 × 64 , DiMR-M (133M parameters) and DiMR-L (284M), without classifier-free guidance [18], achieve FID scores of 3.65 and 2.21, respectively, outperforming the

Transformer-based U-ViT-M/4 and U-ViT-L/4 by 2.20 and 2.05 FID. On ImageNet 256 × 256 , DiMRXL (505M) achieves FID scores of 4.50 without classifier-free guidance and 1.70 with classifier-free guidance. Meanwhile, DiMR-G (1.06B) further improves the FID scores to 3.56 without classifierfree guidance and 1.63 with classifier-free guidance. On ImageNet 512 × 512 , DiMR-XL (525M) achieves FID scores of 7.93 and 2.89, without and with classifier-free guidance, respectively. These results demonstrate superior performance compared to all previous methods, despite having similar or smaller model sizes, establishing a new state-of-the-art performance. In summary, our main contributions are as follows:

We develop effective strategies for integrating multi-resolution networks into diffusion models, introducing the novel feature cascade approach that captures visual details and reduces image distortions in high-fidelity image generation.

We propose TD-LN, a simple yet effective parameter-efficient method that explicitly encodes crucial temporal information into the diffusion model for enhanced performance.

We introduce DiMR, a novel architecture that enhances diffusion models with the proposed multi-resolution network and the TD-LN. DiMR demonstrates superior performance on the class-conditional ImageNet generation benchmark compared to existing methods.

2 Related Work

Diffusion models. Diffusion [53, 19] and score-based generative models [23, 55], centered around a denoising network trained to progressively produce denoised variants of the input data. They have driven significant advances across various domains [34, 29, 58, 56, 62, 40, 61], particularly excelling in high-fidelity image generation tasks [38, 44, 46, 48]. Key advancements in diffusion models include the improvements in sampling methodologies [19, 54, 26] and the adoption of classifierfree guidance [18]. Latent Diffusion Models (LDMs) [46, 41, 43, 63] address the challenges of high-resolution image generation by conducting diffusion in the lower-resolution latent space via a pre-trained autoencoder [28]. In this study, our focus lies on designing the denoising network within diffusion models and examining its applicability across both pixel diffusion models and LDMs.

Architecture for diffusion models. Early diffusion models employed convolutional U-Net architectures [47] as the denoising network, which were subsequently strengthened through explorations of either computing attention [7, 39] or performing diffusion directly at multiple scales [20, 13]. Recently, Transformer-based architectures [2, 41, 14] along with other explorations [64, 57, 27] have emerged as promising alternatives, showcasing superior performance and scalability. Specifically, for Transformer-based architectures, U-ViT [2] treats all inputs, including time, condition, and noisy image patches, as tokens and employs long-skip connections between shallow and deep transformer layers inspired by U-Net. Similarly, DiT [41] leverages Vision Transformers (ViTs) [8] to systematically explore the design space under the Latent Diffusion Models (LDMs) framework, demonstrating favorable properties such as scalability, robustness, and efficiency. In this study, we introduce the Multi-Resolution Network as a new denoising architecture for diffusion models, featuring a multi-branch design where each branch is dedicated to processing a specific resolution.

Time conditioning mechanisms. Following the widespread usage of adaptive normalization [42] in GANs [3, 25], diffusion models similarly explore adaptive group normalization (AdaGN) [7] and adaptive layer normalization (AdaLN) [41] to encode the time information. These methods share the similarity in requiring computing a linear projection of the timestep, which significantly increases the parameter of the model. Recently, U-ViT [2] introduces a new strategy to simply treat time as a token and process with Transformer blocks. Even though effective, it is not feasible to treat time as input for other blocks ( e.g ., ConvNeXt blocks [35]). In this study, we introduce Time-Dependent Layer Normalization (TD-LN), a parameter-efficient approach that explicitly encodes temporal information by incorporating time-dependent parameters into layer normalization [1].

3 Preliminary

Diffusion models [53, 19] are characterized by a forward process that gradually injects noises to destroy data x 0 ∼ q ( x 0 ) , and a reverse process that inverts the forward process corruptions. Formally,

the noise injection process is formulated as a Markov chain:

where x t for t ∈ [1 : T ] is a family of random variables obtained by progressively injecting Gaussian noise into the data x 0 , and q ( x t | x t -1 ) = N ( x t | √ α t x t -1 , β t I ) represents the noise injection schedule such that α t + β t = 1 . In the reverse process, a Gaussian model p ( x t -1 | x t ) = N ( x t -1 | µ t ( x t ) , σ 2 t I ) is learned to approximate the ground truth reverse transition q ( x t -1 | x t ) . This step is equivalent to predicting the denoised variant of the input x t , and thus the learning objective can be further simplified to predicting the noise ϵ t via a noise prediction network (with parameters θ ), i.e ., ϵ θ ( x t ) : min θ E t, x 0 , ϵ t ∥ ϵ t -ϵ θ ( x t ) ∥ 2 2 . The condition information c can be incorporated into the learning objective when the diffusion process is guided by the class condition [7], i.e ., ϵ θ ( x t , c ) min θ E t, x 0 ,c, ϵ t ∥ ϵ t -ϵ θ ( x t , c ) ∥ 2 2 . Traditionally, learning this objective relies on the U-Net [47], with

: the condition c encoded into the U-Net through various methods [2, 7, 41, 46].

Classifier-free guidance [18] , an effective approach to generating high-fidelity samples, combines the score estimates from a conditional diffusion model and a jointly trained unconditional diffusion model. Formally, the classifier-free guidance encourages the sampled x to have high p ( x | c ) by setting: ˆ ϵ θ ( x t , c ) = ϵ θ ( x t , ∅ ) + s · ∇ x log p ( x | c ) ∝ ϵ θ ( x t , ∅ ) + s · ( ϵ θ ( x t , c ) -ϵ θ ( x t , ∅ )) , where s is the scale of guidance, s ≥ 1 , and setting s = 1 becomes the standard sampling. Following prior arts [2, 41], we also exploit classifier-free guidance.

4 Method

In this section, we begin by introducing the proposed Multi-Resolution Network (Sec. 4.1), which progressively refines features from low to high resolution. Next, we detail the proposed TimeDependent Layer Normalization (Sec. 4.2). We then discuss several micro-level design enhancements (Sec. 4.3). Finally, we present the DiMR model variants, scaled for different model sizes (Sec. 4.4).

4.1 Multi-Resolution Network

Motivation. There is a trade-off between generation quality and computational complexity as depicted in the ablation study in Fig. 7 of DiT [41]. Their careful study revealed that Transformerbased diffusion models with smaller patch sizes operate at higher feature resolutions and produce better generation quality but incur higher computational costs due to the increased input size.

We conjecture that the distortion in U-ViT [2] and DiT [41] arises from their oversimplified upsampling module, where lower-resolution feature maps are upsampled directly to the target size of the generated images via a simple linear layer (for increasing channels) and pixel shuffling upsampling [52]. Inspired by image cascade [20, 48]-a method for generating high-resolution images by using multiple cascaded diffusion models to produce images of progressively increasing resolution-we propose feature cascade , which progressively upsample lower-resolution features to higher resolutions to alleviate distortion in image generation. The feature cascade is implemented through the proposed Multi-Resolution Network, deployed as the denoising network in diffusion models.

Overview of multi-branch design. The proposed Multi-Resolution Network comprises R branches, where each branch is dedicated to process a specific resolution. For the r -th branch ( r ∈ { 1 , · · · , R } ), the input features are processed by a convolution with a kernel size of 2 R -r × 2 R -r and a stride of 2 R -r , which effectively patchifies the input for different resolutions. The first branch ( i.e ., r = 1 ) downsamples the input features by a factor of 2 R -1 , and subsequently handles the lowest resolution features via the Transformer blocks [59], which enjoys the superior performance and scalability of self-attention operations [2, 41]. For higher resolution features, the remaining branches utilize the ConvNeXt blocks [35], which leverages the efficiency of large kernel depthwise-convolution operations [22, 50]. Intermediate features from the previous branch (last block's output) are upsampled and added with the inputs for the current branch. Following U-ViT [2], all branches employ the long skip connections and an additional 3 × 3 convolution in the end. The final branch ( i.e ., r = R ) refines features at the same spatial resolution as the input.

Transformer Block

ConvNeXt Block

Transformer Block

ConvNeXt Block

ConvNeXt Block

ConvNeXt Block

Conv 4x4, Stride 4

Conv 2x2, Stride 2

Conv 1x1, Stride 1

Transformer Block

C

ConvNeXt Block

ConvNeXt Block

C

C

+

+

1

2

L

C 0

…

Multi-Head

Attention

TD-LN

TD-LN

MLP

+

+

Transformer Block

U

U

Depthwise

Conv

TD-LN

MLP

+

ConvNeXt Block

+

C

U

: Concat

: Upsample

Conv 3x3

Conv 3x3

Conv 3x3

C

TD-LN

: Time-Dependent Layer Norm

: Position Embedding

: Predicted noise at different scales

: Noised input

: Add

Design details. For r ∈ { 1 , · · · , R } , we define the r -th branch as a function f θ ,r as follows:

where the function f θ ,r , parameterized by θ and r , takes as input the input features x t and the features from previous resolution y r -1 (also time t and condition c ). The outputs of f θ ,r contain the intermediate features y r (last block's output, before the final 3 × 3 convolution) and the predicted noise ϵ θ ( x t , c, r ) for the resolution specific to r -th branch.

̸

To process the inputs, the function f θ ,r first patchifies the input features x t , and adds it with the upsampled features y r -1 from the previous resolution r -1 . The resulting features are then processed by either a stack of Transformer blocks (when r = 1 ) or ConvNeXt blocks (when r = 1 ) with another 3 × 3 convolution added in the end. Formally, we have:

where Conv 3 × 3 is 3 × 3 convolution, Patchify is patchification instantiated via a convolution with a kernel size of 2 R -r × 2 R -r and a stride of 2 R -r , Upsample is the pixel shuffling upsampling operation [52], and g θ ,r is a stack of Transformer blocks or ConvNeXt blocks, depending on r , augmented with the long skip connections [2]. For the first branch ( i.e ., r = 1 ), y 0 is set to zero. The noise prediction ϵ θ ( x t , c, R ) at the last branch ( i.e ., r = R ) is used for the iterative diffusion process. We illustrate the proposed Multi-Resolution Network with three branches in Fig. 2. Note that the input features can be either raw image pixels or latent features after V AE [28], where the latent features facilitate efficient high-resolution image generation [46].

4.2 Time-Dependent Layer Normalization

Motivation. Time conditioning plays a crucial role in the diffusion process. While the ConvNeXt blocks in the Multi-Resolution Network efficiently process high-resolution features, they also present a new challenge: How do we inject time information into ConvNeXt blocks? To address this, we carry out a systematic ablation study (details in Tab. 2), starting with the U-ViT architecture [2], which encodes time information via an in-context conditioning mechanism, Time-Token ( i.e ., treating time as input token to Transformer). Unlike Transformer blocks, however, it is not feasible to add a time token directly to ConvNeXt blocks, which can only process 2D features. As an alternative, we explored the adaptive normalization mechanism, particularly the adaptive layer normalization

1.0

0.8

0.8

Ratic

o

0.8

Ratic

0.8

Ratio

0.6

Ratio

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

PCs of Scale y1

PCs of Scale Y2

PCs of Shift β1

PCs of Shift β2

AdaLN-Zero [41]. Interestingly, we found AdaLN-Zero to be more effective than Time-Token on the ImageNet 64 × 64 benchmark, contradicting U-ViT's findings on CIFAR-10 [30] (See Fig. 2(b) in [2]). However, AdaLN-Zero significantly increases model parameters (from 130.9M to 202.4M) due to the Multi-Layer Perceptron (MLP) used to adaptively learn the scale and shift parameters.

Multi-Head

Attention

Layer Norm

Layer Norm

MLP

+

Scale, Shift

Scale

Scale, Shift

Scale

MLP

+

Transformer Block

w/ adaLN-Zero

Class

Time

Time

TD-LN

Transformer Block

w/ TD-LN

Layer Norm

Multi-Head

Attention

MLP

Class

+

TD-LN

Layer Norm

+

a2

Y2β2

r(t),β(t)

a1

Y1β1

r(t),β(t)

To understand how time information is utilized in adaLNZero, we conducted Principal Component Analysis (PCA) on the learned scale ( γ 1 , γ 2 ) and shift ( β 1 , β 2 ) parameters from a parameter-heavy MLP in adaLN-Zero using a pre-trained DiT-XL/2 [41] model, as shown in Fig. 3. Intriguingly, we observed that the learned parameters can be largely explained by two principal components, suggesting that a parameter-heavy MLP might be unnecessary and that a simpler function could suffice. To address the increase in parameters, we introduce Time-Dependent Layer Normalization (TD-LN), a straightforward and lightweight method to inject time into layer normalization. We detail the designs below.

adaLN design. Building on layer normalization [1], adaLN additionally learns the scale parameter γ 1 and shift parameter β 1 via an MLP from the sum of the embedding vectors of time t and class condition c . Formally, given the input x (ignoring the dependency on t for simplicity), we have:

where γ 1 and β 1 scale and shift the output from the layer normalization LN, the function Embed generates the embedding vectors for time t and class condition c , and z is the output. The LN has its own learnable affine transform parameters γ and β . adaLN-Zero [41] introduces another scale parameter α 1 , obtained from the same MLP for zero initialization of a residual block [12]. We note that DiT employs two sets of ( γ 1 , β 1 , α 1 ) and ( γ 2 , β 2 , α 2 ) in a Transformer block, as shown in Fig. 4.

TD-LN design. In contrast, our proposed method, Time-Dependent Layer Normalization (TD-LN), directly incorporates time t into layer normalization by formulating LN's learnable affine transform parameters γ and β as functions of t . Motivated by the observation that the learned parameters of adaLN-Zero can be largely explained by two principal components, we propose to model this through the linear interpolation of two learnable parameters p 1 and p 2 . Formally,

where s ( t ) is a transformation of time t , w and b are the learnable weight and bias, and Sigmoid is the sigmoid activation function. The other affine transform parameter, β ( t ) , is formulated similarly with

another two parameters p 3 and p 4 . Consequently, the proposed TD-LN is represented as follows:

Unlike adaLN, which learns additional re-scaling γ 1 and re-centering β 1 variables, TD-LN directly incorporates the time-dependent γ ( t ) and β ( t ) into layer normalization, eliminating the need for a parameter-heavy MLP. Furthermore, TD-LN is a versatile mechanism, enabling the injection of time information into both Transformer blocks and ConvNeXt blocks. In DiMR, we replace all layer normalizations with the proposed TD-LN, and treat the class condition c as input token for the Transformer blocks.

4.3 Micro-Level Design

In addition to the major architectural modifications discussed earlier, we also explore several microlevel design changes to enhance model performance.

Multi-scale loss. The proposed Multi-Resolution Network comprises R branches, each dedicated to processing features at a specific resolution, naturally producing multi-scale outputs. To leverage this, we explore training the network with a multi-scale loss L multi , which is a weighted sum of mean squared error loss at each resolution. Formally, the multi-scale loss is defined as follows:

where α r is the loss weight for the r -th branch, and Downsample ( ϵ t , r ) downsamples the target noise ϵ t by a factor of 2 R -r using average pooling (the R -th branch, containing no downsampling, is our final output). We set α r = 1 / (2 R -r × 2 R -r ) , motivated by the prior work [21] which found that the signal to noise ratio increases by a factor of k 2 when the noised input is average-pooled with a k × k kernel. Intuitively, our target output (the R -th branch) has a loss weight α R = 1 , and the loss weights for the intermediate outputs are scaled down quadratically based on the downsampling factor.

Gated linear unit. In the proposed Multi-Resolution Network, both Transformer and ConvNeXt blocks include an MLP block, consisting of two linear transformations with GeLU activation [16] in between. We also explore replacing the first linear layer with GeGLU [51], an enhanced version of the Gated Linear Unit (GLU) [5] that has 2 × expansion rate.

4.4 DiMR Model Variants

We now introduce the DiMR model variants, scaled appropriately for different model sizes. We present four sizes: DiMR-M (medium, 133M parameters), DiMR-L (large, 284M parameters), DiMR-XL (extra-large, around 500M parameters) and DiMR-G (giant, 1.06B parameters). Three hyperparametersR (number of branches), N (number of layers per branch), and D (hidden size per branch)-define each DiMR variant. Specifically, R determines the number of branches in the multi-resolution network. We append 2R or 3R to the model name to indicate whether two or three branches are used. The number of layers N in the multi-resolution network is represented as a tuple of R numbers, where the r -th number specifies the number of layers in the r -th branch. Similarly, the hidden size D is also a tuple of R numbers. We follow a straightforward scaling rule: most layers are stacked in the first branch, which is processed by Transformer blocks, while the remaining branches use only half the number of layers of the first branch. Additionally, when the resolution is doubled, the hidden size is reduced by a factor of two. The model variants details are presented in Tab. 4 in Sec. B in the Appendix.

5 Experimental Results

5.1 Experimental Setup

Datasets. We consider class-conditional image generation tasks at 64 × 64 , 256 × 256 , and 512 × 512 resolutions on ImageNet-1K [6]. For images at 64 × 64 , we train DiMR on pixel space. For images at 256 × 256 and 512 × 512 , following the baselines [2, 41], we utilize an off-the-shelf pre-trained variational autoencoder [28] from Stable Diffusion [46] to extract the latent representations sized at 32 × 32 and 64 × 64 , respectively. Then we train our DiMR to model these latent representations.

Evaluation. We measure the model's performance using Fréchet Inception Distance (FID) [17]. We report FID on 50K generated samples to measure the image quality ( i.e ., FID-50K). To ensure fair comparisons, we follow the same evaluation suite as the baseliness [7, 41] to compute the FID scores. We also report Inception Score [49] and Precision/Recall [31] in Sec. D as secondary metrics.

Implementation details. We use AdamW optimizer [36] with a constant learning rate of 2 × 10 -4 for most experiments, except for the 64 × 64 models where we use 3 × 10 -4 . A batch size of 1024 is used for most architectures. For a fair comparison with DiT [41], we train the 256 × 256 and 512 × 512 models for 1M iterations, and also report results for 500K iterations to compare with U-ViT [2]. We train the 64 × 64 models for 300K iterations, following the U-ViT protocol. Our training hyperparameters are almost entirely retained from U-ViT [2]. We did not tune learning rates, decay/warm-up schedules, Adam β 1 / β 2 values, or weight decays. Further details on hyperparameters and configurations are provided in Sec. C in the Appendix.

5.2 State-of-the-Art Diffusion Models

We compare DiMR with state-of-the-art diffusion models on ImageNet 256 × 256 and 512 × 512 in Tab. 1, and provide more comparisons with other types of generative models in Tab. 7 and Tab. 8 in the Appendix. Results on ImageNet 64 × 64 are reported in Tab. 6 in the Appendix. More random samples of the generated images are also presented in Fig. 7 to Fig. 18 in the Appendix.

ImageNet 256 × 256 . From Tab. 1a, we observe that our DiMR-XL/2R outperforms all previous diffusion-based models and achieves a state-of-the-art FID-50K score of 1.70. Specifically, with a comparable model size and equal or fewer training epochs, our model surpasses previous state-ofthe-art transformer-based diffusion models, including U-ViT[2] (1.77 vs . 2.29 with Classifier-Free Guidance [18] (CFG) and 4.87 vs . 6.58 without CFG) and DiT [41] (1.70 vs . 2.27 with CFG and 4.50 vs . 9.62 without CFG). Our best model, DiMR-G/2R, scales up to the billion-parameter level, setting a new state-of-the-art with an FID of 1.63 with CFG and 3.56 without CFG.

ImageNet 512 × 512 . Our DiMR outperforms all previous diffusion-based models on ImageNet 512 × 512 and achieves a state-of-the-art FID-50K score of 2.89 as shown in Tab. 1b. It is worth noting that, although both Gflops and model sizes are critical for improving performance, as discussed in the DiT paper [41], we still outperform it with only 39 . 2% of the GFLOPs and 77 . 8% of the model size, improving the FID-50K from 3.04 to 2.89. As transformers and diffusion models have demonstrated good scaling behavior, we believe that further scaling up our DiMR will lead to better performance, which we have left as future work.

5.3 Alleviating Distortion

Transformer-based architectures encounter the challenge of balancing visual fidelity with computational complexity. Despite adopting a small patch size of 2, current models still struggle with distortions. To illustrate the effectiveness of DiMR in alleviating these distortions, we adopt a classifier-based rejection model following previous work [45]. However, we diverge from previous approaches by solely using the rejection model to analyze distorted images, rather than filtering out bad images and computing metrics only on selected 'good' images. It is important to note that all metrics in our paper are computed without using the rejection model to ensure fair comparisons.

DiMR (Ours)

FID=1.77

Distortion rate

=

29.2%

DiT

FID=2.27

Distortion rate

=

71.0%

U-ViT

FID=2.29

Distortion rate

=

63.5%

Specifically, we randomly generate 80K images for each model and utilize a pretrained Vision Transformer classifier [8] to identify low-fidelity images based on the predicted probabilities. Images with a probability below a threshold of 0.2 are considered low-fidelity or potentially distorted. Fig. 5 shows random samples of low-fidelity images detected by the classifier. However, we find that not all detected images are distorted; many are classified with low probability due to classifier errors. To accurately identify distorted images among those detected by the classifier, we conduct user studies where human evaluators manually assess the images. Images generated by all three methods are merged and presented, along with their corresponding class labels, to human evaluators, who are instructed to determine whether each image is distorted ( i.e ., identify low-fidelity images). Each image is evaluated by five different human evaluators. We consider the proportion of distorted images generated by different models, i.e . distortion rate. We compute three distortion rates, one for each model, from each evaluator based on the images they evaluate. The final distortion rate for each model is obtained by averaging the rates from all evaluators. As reported in Fig. 5, we observe that even among those low-fidelity images, only 29 . 2% of the images generated by DiMR are distorted, while previous methods yield much higher distortion rates of 63 . 5% and 71 . 0% .

5.4 Ablation Studies

We conduct the primary ablation experiments on ImageNet 64 × 64 , progressively building on the baseline U-ViT-M/4 [2] to validate the effectiveness of the proposed designs, leading to our final model, DiMR-M/3R, as presented in Tab. 2. Additionally, we explore alternative design choices on ImageNet 256 × 256 with DiMR-XL/2R, including adopting a pure convolutional architecture, replacing addition with concatenation in feature cascading, and introducing skip connections between branches, as shown in Tab. 3.

AdaLN-Zero vs . TD-LN. Since the time token used in U-ViT cannot be adopted for ConvNeXt blocks, we first apply AdaLN-Zero [41] to the original U-ViT and our multi-branch network. As observed in row 2 of Tab. 2, AdaLN-Zero slightly improves the performance of U-ViT from 5.85 to

(a)

Pure Conv v.s. Hybrid

Model

1st branch

FID

↓

DiMR-XL/2R

ConvNeXt

2.09

Transformer

1.77

(b)

Concatenation v.s. Addition

Model

feature cascade

FID

↓

DiMR-XL/2R

Concatenation

2.06

Addition

1.77

(c)

w/ v.s. w/o Skip-Connection

Model

skip-connection

FID

↓

DiMR-XL/2R

✓

1.96

1.77

5.44. However, it does not perform well on ConvNeXt blocks and thus decreases the performance from 5.44 to 7.91 (row 3). Additionally, AdaLN-Zero significantly increases the model size from 130.9M to 202.4M. In contrast, our TD-LN is more flexible and parameter-efficient: it efficiently provides time information to both Transformer blocks and ConvNeXt blocks, improving the FID-50K score from 7.91 to 5.21 (row 4), and also reduces the model size from 217.9M to 154.0M.

GLU further reduces model size. In Tab. 2, row 5 shows the improvement of GLU compared with the vanilla MLP block. We observe that using GLU slightly improves the performance from 5.21 to 4.86 and further reduces the model size from 154.0M to 132.9M.

Multi-scale loss is critical for multi-resolution network. Training a multi-resolution network presents additional challenges and can result in sub-optimal results. In Tab. 2, row 6 illustrates that our multi-scale loss significantly enhances the performance, achieving a FID-50K score of 3.65.

Multi-branch design improves visual fidelity and alleviates distortions in image generations. Finally, comparing the multi-branch design in row 6 (incorporating TD-LN, GLU, and multi-scale loss to facilitate training) with the baseline in row 1 reveals a significant improvement in FID-50K, from 5.85 to 3.65, with just a 1.5% increase in model size (130.9M to 132.9M). Additionally, from Fig. 5, it's evident that the multi-branch design generates images with higher fidelity and less distortion.

Transformer is essential for low-resolution processing. As shown in Table 3a, replacing the Transformer blocks in the 1st (lowest-resolution) branch with ConvNeXt blocks results in a DiMR variant that uses only convolutional layers. However, this configuration performs worse compared to combining Transformer blocks with ConvNeXt blocks across different resolutions. This indicates that Transformers are more effective at capturing fine-grained details, while their usage at the lowest resolution maintains a manageable computational cost.

Simple addition suffices for multi-resolution feature cascading. As shown in Table 3b, a straightforward addition operation effectively transfers information from lower-resolution features to higherresolution features. Replacing addition with concatenation leads to slightly worse results. We also validate the necessity of adding skip-connection between branches. As shown in Table 3c, introducing skip-connection not only degrades performance but also complicates the model architecture. Therefore, we adopt a simple upsampling followed by an addition operation for feature cascading.

6 Conclusion

In this work, we introduce DiMR, which enhances diffusion models through the Multi-Resolution Network, progressively refining features from low to high resolutions and effectively reducing image distortion. Additionally, DiMR incorporates the proposed parameter-efficient Time-Dependent Layer Normalization (TD-LN), further improving image generation quality. The effectiveness of DiMR has been demonstrated on the popular class-conditional ImageNet generation benchmark, outperforming prior methods and setting new state-of-the-art performance on diffusion-style generative models. We hope that DiMR will inspire future designs of both denoising networks and time conditioning mechanisms, paving the way for even more advanced image generation models.

Acknowledgement : We thank Xueqing Deng and Peng Wang for their valuable discussion during Zhanpeng's internship.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR , 2023.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR , 2022.

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , 2021.

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV , 2023.

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389 , 2023.

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In ICLR , 2023.

Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. In ECCV , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR , 23(47):1-33, 2022.

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML , 2023.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. JMLR , 6(4), 2005.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR , 2023.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , 2019.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NeurIPS , 2022.

Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of a one-step generator from a low-resolution diffusion teacher. arXiv preprint arXiv:2405.14822 , 2024.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS , 2019.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR , 2022.

Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. arXiv preprint arXiv:2312.03701 , 2023.

Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. NeurIPS , 2022.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV , 2024.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 , 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , 2023.

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI , 2018.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurIPS , 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. NeurIPS , 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS , 2016.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR , 2018.

Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 , 2020.

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR , 2016.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS , 2019.

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. NeurIPS , 2021.

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 , 2024.

Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. NeurIPS , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv preprint arXiv:2409.16211 , 2024.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Openvocabulary panoptic segmentation with text-to-image diffusion models. In CVPR , 2023.

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923 , 2022.

Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. NeurIPS , 2023.

Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In CVPR , 2024.

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 , 2021.

Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. NeurIPS , 2024.

Appendix

In the appendix, we provide additional information as listed below:

Sec. A provides the dataset information and licenses.

Sec. B provides the DiMR model variants, scaled appropriately for different model sizes.

Sec. C provides the implementation details of DiMR.

Sec. D provides more comparison with other methods for class-conditional image generation on ImageNet 64 × 64 , ImageNet 256 × 256 , and ImageNet 512 × 512 .

Sec. E provides detailed introduction and more results of the Principal Component Analysis (PCA) on the scale and shift parameters learned in adaLN-Zero.

Sec. F provides more generated image samples by DiMR.

Sec. G discusses the limitations of our method.

Sec. H discusses the positive societal impacts of our method.

Sec. I discusses the potential risk of our method and safeguards that will be put in place for responsible release of our models.

A Datasets Information and Licenses

ImageNet: The ImageNet [6] dataset, containing 1,281,167 training and 50,000 validation images from 1,000 different classes, is a standard benchmark for image classification and class-conditional image generation. For the task of class-conditional image generation, the images are typically resized to a specified size, e.g ., 64 × 64 , 256 × 256 , or 512 × 512 .

License: Custom License, non-commercial. https://image-net.org/accessagreement

Dataset website: https://image-net.org/

B DiMR Model Variants

We introduce the DiMR model variants, scaled appropriately for different model sizes. We present four sizes: DiMR-M (medium, 132.7M parameters), DiMR-L (large, 284.0M parameters), DiMR-XL (extra-large, around 500M parameters), and DiMR-G (giant, 1.06B parameters) Three hyperparametersR (number of branches), N (number of layers per branch), and D (hidden size per branch)-define each DiMR variant. Specifically, R determines the number of branches in the multi-resolution network. We append 2R or 3R to the model name to indicate whether two or three branches are used. The number of layers N in the multi-resolution network is represented as a tuple of R numbers, where the r -th number specifies the number of layers in the r -th branch. Similarly, the hidden size D is also a tuple of R numbers. We follow a straightforward scaling rule: most layers are stacked in the first branch, which is processed by Transformer blocks, while the remaining branches use only half the number of layers of the first branch. Additionally, when the resolution is doubled, the hidden size is reduced by a factor of two. The model variants are illustrated in Tab. 4.

C Implementation Details

We use the AdamW optimizer [36] with a constant learning rate of 2 × 10 -4 for most experiments, except for the 64 × 64 models where we use 3 × 10 -4 . We set the weight decay to 0.03 and the betas to (0.99, 0.99) for all experiments. A batch size of 1024 is used for all architectures. For a fair comparison with DiT [41], we train the 256 × 256 and 512 × 512 models for 1M iterations, and we also report results for 500K iterations to compare with U-ViT [2]. We train the 64 × 64 models for 300K iterations, following the U-ViT protocol. All experiments use 5K steps for warm-up. We present the detailed experimental setup for all DiMR variants in Tab. 5.

D Additional Experimental Results

We present the full results of the proposed DiMR compared to other methods on ImageNet [6] in terms of Fréchet Inception Distance (FID) [17], Inception Score (IS) [49], and Precision/Recall [31]. The comparisons are made on class-conditional image generation without classifier-free guidance on ImageNet 64 × 64 in Tab. 6, and with classifier-free guidance [18] on ImageNet 256 × 256 in Tab. 7 and ImageNet 512 × 512 in Tab. 8.

ImageNet 64 × 64 . We follow the exact experimental setup of U-ViT [2] for class-conditional image generation on ImageNet 64 × 64 without classifier-free guidance to verify the effectiveness of our proposed backbone. Therefore, we focus solely on comparing against U-ViT on this benchmark. As shown in Tab. 6, when both are trained for 240 epochs, the proposed DiMR-M/3R with 133M parameters achieves an FID of 3.65 and an IS of 42.41, improving upon the counterpart U-ViT-M/4 with 131M parameters by 2.20 in FID and 8.70 in IS. For the larger model, DiMR-L/3R with 284M parameters outperforms U-ViT-L/4 with 287M parameters by 2.05 in FID and 15.07 in IS. These

consistent and significant improvements demonstrate the capability of the proposed Multi-Resolution Network and TD-LN in enhancing diffusion models to generate high-fidelity images.

ImageNet 256 × 256 . We compare DiMR with state-of-the-art generative models on ImageNet 256 × 256 with classifier-free guidance in Tab. 7. Compared to U-ViT [2] in a fair setting, our DiMR-XL/2R with 505M parameters, trained for 400 epochs, significantly outperforms U-ViT-H/2 with 501M parameters, also trained for 400 epochs, by 0.52 in FID and 21.8 in IS. In comparison with the recently popular diffusion model DiT [41], DiMR-XL/2R trained for 800 epochs consistently outperforms DiT-L/2 with 458M parameters by 3.32 in FID and 121.8 in IS. DiMR-XL/2R even surpasses the larger variant DiT-XL/2 with 675M parameters by 0.57 in FID and 11.8 in IS. Notably, DiT models require training for 1399 epochs, while DiMR-XL/2R achieves superior performance with only 800 epochs. Furthermore, scaling up to DiMR-G/2R sets a new state-of-the-art for ImageNet 256 × 256 image generation, achieving an FID of 1.63 and an IS of 292.5.

ImageNet 512 × 512 . We compare DiMR with state-of-the-art generative models on ImageNet 512 × 512 with classifier-free guidance in Tab. 8. Under the same training setting for 400 epochs, DiMR-XL/3R with 525M parameters outperforms U-ViT-H/4 with 501M parameters by 0.82 in FID and 21.3 in IS. Compared to DiT-XL/2 with 675M parameters, DiMR-XL/3R shows slight improvements of 0.15 in FID and 49.0 in IS. Overall, DiMR-XL/3R demonstrates performance comparable to other state-of-the-art generative models for ImageNet 512 × 512 image generation.

E Additional PCA of Learned Scale and Shift Parameters in adaLN-Zero

Weconduct PCA on the learned scale ( γ 1 , γ 2 ) and shift ( β 1 , β 2 ) parameters obtained from a parameterheavy MLP in adaLN-Zero using a pre-trained DiT-XL/2 [41] model with 28 layers in depth. To conduct the analysis, we utilize the pre-trained DiT-XL/2 to generate images and collected the scale and shift parameters (tensors) produced by the MLP at different layers along the sampling steps. PCA is then performed on the collected tensors at each layer separately. The results are presented in Fig. 6, where each row of the figure displays the analysis result at different depths, from top to bottom: 7, 14, 21, 28. The vertical axis represents the explained variance ratio of the corresponding Principal Components (PCs). As observed, in most cases, the most important principal component can explain most of the variance, while starting from the 3rd principal component, it usually only accounts for less than 5% of the variance. Our observations reveal that the learned parameters, regardless of whether produced by an MLP at a shallower layer or a deeper layer, can be largely explained by two principal components, suggesting the potential to approximate them by a simpler function, TD-LN, where the linear interpolation of two learnable parameters is learned as introduced in Sec. 4.2.

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.0

0.0

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

PCs of Scale V1

PCs of Shift β1

PCs of Shift β2

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

2

PCs of Scale V1

4

6

8

10

2

4

6

8

10

2

PCs of Shift β1

4

6

8

10

2

PCs of Shift β2

4

6

8

10

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.0

2

PCs of Scale V1

4

6

8

10

2

4

6

8

10

2

PCs of Shift β1

4

6

8

10

2

PCs of Shift β2

4

6

8

10

1.0

0.8

0.5

0.8

0.4

0.6

0.6

0.31

0.4

0.2

0.2

0.1

0.1

0.0

0.0

0.0

0.0

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

PCs of Shift β1

PCs of Shift β2

PCs of Scale V1

F Model Samples

We present samples from our largest variant, DiMR-XL/3R, at 512 × 512 resolution trained for 800 epochs. Fig. 7- 18 display uncurated samples from the model across a range of input class labels with classifier-free guidance. It is worth noting that our generated image samples exhibit high-quality and minimal image distortions.

G Limitations

The proposed DiMR has a few remaining limitations. First, it focuses on class-conditional image generation, rather than full text-to-image generation approaches. Additionally, although DiMR demonstrates great scalability in our experiments, the exploration of DiMR variants stops at the DiMR-XL/3R model with 524.8M parameters due to constraints on computational resources. In contrast, a few recent methods have scaled image diffusion models to billions of parameters. We leave it as future work to further explore the scaling law of our proposed DiMR to enhance its image generation capabilities.

ADY

H Broader Impacts

The proposed DiMR has the potential to facilitate numerous fields through its advanced image generation capabilities. In the realm of creative industries, DiMR can enhance the efficiency and creativity of artists and designers by generating high-fidelity images with fewer distortions. The high-quality generated images can also contribute to research on synthetic datasets by creating realistic images, aiding in reducing the annotations required for training vision models. However, with these advancements come ethical considerations, such as the risk of generating deepfakes or other malicious content. It is thus crucial to implement safeguards to minimize potential harms.

I Safety Concerns and Safeguards

Given the powerful capabilities of DiMR, it is essential to implement robust safeguards to address potential safety and ethical concerns. One primary concern is the misuse of generated content, such as the creation of deepfakes, which can lead to misinformation and privacy violations. To mitigate this, it is important to establish strict access controls and usage policies to prevent the misuse of these models when released. Transparency in the training data and model architecture is also critical to ensure accountability and to identify potential biases that could lead to harmful outputs. By prioritizing these safeguards, we can ensure the responsible use of DiMR while minimizing potential risks.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Sec. 1 for the main claims of the paper and Sec. 4 and Sec. 5 for their illustration and validation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. G for the discussion on the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sec. 5.1 and Tab. 5 for all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: We have open-sourced the full code and trained model weights after thorough cleaning and the implementation of necessary safeguards.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sec. 5.1 and Tab. 5 for all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following the literature [2, 41, 64, 14, 57], error bars are not reported to provide statistical significance, as the metrics FID, IS, Precision, and Recall inherently account for it by being computed while sampling 50K images.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Tab. 5 for all the information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Sec. H for the discussion on the broader impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: See Sec. I for the discussion on the safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Sec. A for dataset licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a multi-resolution network for diffusion model which emphasizes the learning features across resolutions. The method further introduces a time-dependent layer norm to boost the model performance with fewer parameters compared to AdaLN in DiT. This proposed network demonstrates a SoTA performance on ImageNet at 1.70 and 2.89 on ImageNet 256 & 512, respectively.\n\nSTRENGTHS:\n- The paper addresses image distortion caused by varying patch sizes by integrating a multi-resolution structure into the network. The choice of patch sizes involves a trade-off between computational complexity and model performance: larger patch sizes reduce computational complexity but degrade performance. \n- The method finds a balance between compute complexity and performance by introducing a multi-branch architecture and multi-resolution loss to decompose the learning process from low to high resolution. The idea is a substantial contribution. Through quantitative and qualitative results, the method can alleviate distortion of image generation and achieve SoTA FID scores.\n- The time-dependent layer norm is a simplified form of AdaLN with less intensive parameters by removing the cumbersome MLP layer and rearranging class embedding and time embedding for parameter-efficient time conditioning.\n\nWEAKNESSES:\n- The discussion between the method and cascaded methods like \"Cascaded diffusion models\" and \"Matryoshka diffusion models\" should be included to highlight the advantages of the method. \n- In table 1, it needs to be updated with some latest SoTa methods like MDTv2 with the best FID of 1.58 and PaGoDA with the best FID of 1.56 on ImageNet 256x256. Not to compete with them, but to get an overview picture of the latest methods.\n- The proposed multi-scale diffusion loss is also introduced in SimpleDiffusion. So, the authors need to mention the difference in the paper. \n- Equation 2: have the authors used concat instead of adding the upsampling features directly to the larger-resolution features of the next branch?\n- The current method only uses the output features of the previous branch and injects them into the start of the next branch. However, this design lacks interconnections between blocks from the low-resolution and high-resolution branches. Ideally, there should be skip-connections across branches. What is the motivation behind this? If authors have not tested this, it is encouraged to do so.\n- In table 2, what is the reason why using AdaLN-Zero with Multi-branch (row 3) causes a bad result? \n- The root problem of distortion boils down to the use of patch size which is the main motivation of the work. However, in the design of multiscale network, the authors also patchify the input image to the corresponding resolution for each branch. I wonder why the author do not use different-resolution noisy images as the input to each branch like Matryoshka Diffusion Models. I think this is more effective to completely remove distortion problem. As shown in Figure 5, the method still exhibits a certain level of distortion rates. \n- Sampling speed: it is valid to include a comparison of sampling time with baselines.\n- Figure 3: What is the red line meaning?\n\nMisc: L37, L130: the figure 7 of DiT paper is incorrectly linked.\n\nRef: \n- Gao, Shanghua, et al. \"MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer.\" arXiv preprint arXiv:2303.14389 (2023).\n- Kim, Dongjun, et al. \"PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher.\" arXiv preprint arXiv:2405.14822 (2024).\n- Hoogeboom, Emiel, Jonathan Heek, and Tim Salimans. \"simple diffusion: End-to-end diffusion for high resolution images.\" International Conference on Machine Learning. PMLR, 2023.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nAs pointed out in Weaknesses, the method still has a certain distortion rate which is due to the use of strided convolution (a form of patchification) at the start of each branch. I think the author should include this in the revised manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to replace the original pull transformer blocks in DiT with multi-resolution network. Detailly, transformer block is employed for low resolution and Conv blocks are used for the remaining higher resolution. An additional time-conditioning block is designed for Conv blocks. The effectiveness of this method is validated with numercial experiments.\n\nSTRENGTHS:\n1. This paper proposes a multi-resolution network, consisting of both transformer and conv blocks, to boost the performamce of image generation. Such design is rational. To fit the conv block, this paper also introduce an additional time conditioning block.\n2. This method achieves better performance than DiT.\n\nWEAKNESSES:\n1. It is unclear what the performance gain comes from. intuitively, the multi-resolution design could already brings better performance. But this paper proposes the conbination of transformer and conv. I wonder if the design of transformer in low resolution really necessary.\n2. The scalibility of this network may be limited due to the introduction of conv blocks. For example, for network parameters larger than 1B.\n3. This paper mainly claims to solve the large computional complexityinduced by the long token length. But in the real long token length case (image 512x512), the perfomance gains of this method is marginal compared to DiT. This raises the concern of the real performance of this method on large image case, eg. >512x512.\n\nQUESTIONS:\n1. It is suggested to identify the real core contribution for performance gain of this paper. The ablation study of replacing the transformer block with conv block is suggested.\n2. The effectiveness of this method on larger image size and networks are concerened. The authors are suggested to give furhter discussions on these points if they wish a full surpass of the original DiT.\n\nLIMITATIONS:\nThe authors mask well discussions on the limitaion of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a diffusion model named DiMR, which incorporates a Multi-Resolution Network and Time-Dependent Layer Normalization to enhance image generation quality. Traditional diffusion models, often limited by a trade-off between computational efficiency and visual fidelity, struggle with image distortion due to the coarse resolution of input data processing in Transformer-based designs. DiMR addresses this by refining image details progressively across multiple resolutions, reducing distortions significantly. The new light-weight Time-Dependent Layer Normalization technique introduced in the model embeds time-based parameters into the normalization process efficiently. Demonstrated through benchmarks on ImageNet 256x256 and ImageNet  512x512, DiMR outperforms competing models.\n\nSTRENGTHS:\n+ The paper is well-written and easy to follow, featuring clear diagrams and detailed captions that enhance understanding.\n+ The introduction of Time-Dependent Layer Normalization (TD-LN) is particularly interesting. The use of PCA analysis to justify this approach provides a strong motivation and highlights its innovativeness.\n+ The experimental section of the paper effectively demonstrates the effectiveness of the proposed method.\n\nWEAKNESSES:\n- The model has a higher FLOPs count compared to a DiT of similar parameter size, raising concerns about slower training speeds.\n- It is unclear whether this cascaded approach can support multi-resolution training.\n\nQUESTIONS:\n- This work appears to be a cascade diffusion model based on DiT. How does it perform compared to traditional UNet-based cascade diffusion models?\n- What are the training and convergence speeds of this model, particularly in comparison to the baselines?\n- The proposed method adopted the class token similar to that used in U-ViT for injecting class conditions. Is it possible to replace this with an approach similar to Time-Dependent Layer Normalization (TD-LN)?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper works on efficient diffusion model backbone architecture by using transformer and ConvNeXt architecture respectively on small and large resolutions of the same inputs, to leverage the strength of both architectures, and alleviate the distortion problem.

STRENGTHS:
1. The writing is easy to read and follow.

2. The idea is simple and effective.

3. They conduct rich experiments to support the idea.

WEAKNESSES:
1. This work utilizes two standard architectures, incorporating only minor design elements such as the TD-LN. It would be more advantageous to propose a new, general, and elegant architecture that effectively combines the strengths of the attention layer and the convolution layer.

QUESTIONS:
Are there any ablation studies on the number of different resolutions used? For instance, what happens if we use one small resolution with DiT and one normal resolution with a larger conv net?

LIMITATIONS:
See the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13