Exploring DCN-like architecture for fast image generation with arbitrary resolution
Authors: Shuai Wang
Zexian Li
Tianhui Song
Xubin Li
Tiezheng Ge
Bo Zheng
Limin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image generation
deformable convolution
Primary area: generative_models
TL;DR: Exploring DCN-like architecture for fast image generation with arbitrary resolution

Abstract: Arbitrary-resolution image generation still remains a challenging task in AIGC, as it requires handling varying resolutions and aspect ratios while maintaining high visual quality. Existing transformer-based diffusion methods suffer from quadratic computation cost and limited resolution extrapolation capabilities, making them less effective for this task. In this paper, we propose FlowDCN, a purely convolution-based generative model with linear time and memory complexity, that can efficiently generate high-quality images at arbitrary resolutions. Equipped with a new design of learnable group-wise deformable convolution block, our FlowDCN yields higher flexibility and capability to handle different resolutions with a single model.
FlowDCN achieves the state-of-the-art 4.30 sFID on $256\times256$ ImageNet Benchmark and comparable resolution extrapolation results, surpassing transformer-based counterparts in terms of convergence speed (only $\frac{1}{5}$ images), visual quality, parameters ($8\%$ reduction) and FLOPs ($20\%$ reduction). We believe FlowDCN offers a promising solution to scalable and flexible image synthesis.

FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution
Shuai Wang Nanjing University
Zexian Li Alibaba Group
Tianhui Song Nanjing University
Xubin Li Alibaba Group
Tiezheng Ge Alibaba Group
Bo Zheng
Limin Wang
X
Alibaba Group
Nanjing University, Shanghai Al Lab
Figure I: Selected rebitrary-resolutie samples (384x384, 224x448, 448x224, 256x256). Generated from single FlowDCN-XL/2 model trained ImageNet 256x256 resolution with CFG 4.0.
Abstract
Arbitrayy-reoolution image generation still remains challenging task in AIGC, as it requires handling varying resolutions and aspect ratios while maintaining high visual quality. Existing transformer-based diffusion methods suffer from quadratic computation cost and limited resolution extrapolation capabilities, mak- ing them less effective for this task. In this paper, we propose FlowDCN, a purely convolution-base generative model with linear time and memory complexity, that can efficiently generate high-quality images arbitrary resolutions. Equipped with new design of learnable group-wise deformable convolution block, our FlowDCN yields higher flexibility and capability handle different resolutions with a single model. FlowDCN achieves the state-of-the-an art 4.30 sFID on 256 256 ImageNet Benchmark and comparable resolution extrapolation results, surpassing maasforrer-basee counterparts in terms of convergence speed (only = images), visual quality, parameters (8% reduction) and FLOPs (20% reduction). We believe FlowDCN offers promising solution to scalable and flexible image synthesis.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Image generation is an important task in computer vision research, which is aimed at capturing the inherent data distribution of original image datasets and generating high-quality synthetic images through sampling. Diffusion models [1, 2, 3, 4, 5] have recently emerged as a highly promising foundation for training algorithms in image generation, outperforming GAN-base models [6. 7] and suto-Reereessive models [8] by significant margin. The evolution of diffusion models is fast, transitioning from discrete forms [1] to SDE-based continuous forms [2, 3, 4, 5, 9]. In a nutshell, diffusion models incrementally degrade an image through time-dependent stochastic perturbation process and then learn the reverse process to restore the original image from its corrupted state.
Beyond theoretical advancements in diffusion models, the architecture of these models also signifi- cantly influences the quality of generated images. Many works [L, 10, 11] in the diffusion domain adopt a standard UNet architecture as the generation backbone, which consists of downsample blocks, upsample blocks, and long residual connections between these components. Inspired by the success of the vision transformer in perception tasks, DiT [12] eliminates the long residual connection in favor of pure transformer- based architecture. Through rigorous experiments, DiT demonstrates that the UNet inductive bias is not essential for achieving high performance in diffusion models [12]. Meanwhile, PixArt [13, 14] and SD3 [15] venture further by significantly increasing the number of parameters, exploring new frontiers in model architecture and its impact on image generation.
When considering the generation of images at arbitrary resolution, diffusion transformers need to confront at least two primary challenges. The first is the quadratic computation cost: the architecture of diffusion transformers employs attention mechanisms the dense nature of attention computations, high-resolution image generation inevitably leads to significant computation and memory demands, both scaling with (n²) complexity. To address the quadratic computation challenge, some methods [16] have adapted recurrent computational strategies from natural language processing. However, these adaptations do not fully capitalize on the strengths of autoregressive tasks and result in slower inference speeds due to the reduced parallelism inherent in RNN-based scanning. The second challenge resolution extrupolation: many diffusion transformers rely on absolute position embedding (APE) [17] to incorporate positional information, introducing it at the onset of the model. This approach forces subsequent layers to become overfitted to the APE for providing positional context to the attention layers, which presents significant barrier when extrapolating to different resolutions. To address this issue, FiT [18] has turned to Rotary Positional Encoding [19], incorporating RoPE2D enhance its resolution extrapolation capabilities. Nevertheless, FiT still requires a training pipeline tailored to
aggregate spatial tokens. Owing to
generation.
In contrast, convolutional models are the most common choice of visual encoders, boasting linear complexity and aggregating spatial features based on relative positions. With the support of modern convolution operators [20, 21, 22], convolutional models have demonstrated comparable performance or even surpassed transformers in perception tasks. This naturally leads us to inquire: Can modern convolutional networks achieve arbitrary- former counterparts? To answer this question, we opt for deformable convolution as the basic block for exploration in generation, owing to its superior performance in perception tasks.
generation efficiently and outperform trans-
Specifically, we propose a novel approach to decouple the scale and direction prediction of deformable convolution, giving rise to group- multiscale deformable convolution block that enables efficient multiscale feature aggregation. By leveraging this block, we introduce FlowDCN, a modern purely convolution generative model that tackles arbitrary-resolutin of convolutional deformable block, our FlowDCN yields higher flexibility and capability to handle different resolutions with single model. The experiments demonstrate that FlowDCN consistently surpasses its diffusion transformer counterparts, DiT [12] and SiT [23]. Notably, on the 256x256 ImageNet benchmark, FlowDCN achieves faster convergence, yielding SoTA sFid of 4.30 and FID of 2.13 under 1.5M steps with batch size 256, while exhibiting 20% lower latency, 8% fewer parameters, and approximately 20% fewer floating-poin ppint operations (FLOPs). On the 512x512 ImageNet benchmark, FlowDCN achieves 4.53 sFid o and 2.44 FID under 100K finetuning steps with batch size 256.
generation. Thanks to the new design
Moreover, our FlowDCN offers a significant advantage in fast arbitray generation, as only requires linear time and memory complexity. Through visualization comparisons, our FlowDCN demonstrates substantially better visual quality even at extremely small sampling steps, such as 3, 4, and 5 steps. To further enhance its visual quality, we propose Scale Adjustment, a simpler
2

---

technique for extrapolating resolution to unseen dimensions. Our results show that FlowDCN achieves comparable resolution extrapolation capabilities to highly tailored methods, underscoring its potential for generating high-quality images at various resolutions. The contributions can be summarized as:
We decouple the scale and direction priors of deformable convolution and propose a Group- wise MultiScale Deformable Block. Building upon this block, we propose FlowDCN, purely convolution-
generative model with high efficiency.
On 256x256 ImageNet benchmark, under only 1.5M training steps, our FlowDCN-XI.22 achieves 2.13 FID and SoTA 4.30 sFID with Euler solver and classifier free guidance.
On 512x512 ImageNet benchmark, under only 100K finetuning steps, our FlowDCN-XL/2 achieves 2.44 FID and 4.53 sFID with Euler solver and classifier free guidance.
We propose much simple and efficient resolution extrapolation method, deemed as Scale Adjustment. For arbitrary resolution generation, we achieve comparable results to highly tailored methods.
2 Preliminary
2.1 Linear-based Flow Matching
Flow matching [4. is simple but powerful diffusion family. We incorporate linear-based flow matching as the training framework for its simplicity. Given the image sampled æ from training distributions and the noise sampled from a Gaussian distribution, linear-based flow matching forward process interpolate 2y with r and using the following equation:
2t tr + (1 t)c.
(1)
The velocity field of linear-based flow matching [4, 5] is defined as Eq. (2). We train our FlowDCN predict the time-dependent velocity field between x and
vx(xx) -
During training, the flow matching objective directly regresses the target velocity:
C, = f Blll vo 1) - (x())) [7]
(3)
For sampling, the common ODE/SDE solver e.g.. Fuler method, Heun method can be employed.
2.2 Deformable Convolution Revisited
Given an image feature E Hx deformable convolution predicts the deformable field Ax(x) " w x Kx2 and the dynamic weights W(x) #xxxxx from the image feature x. Specifically, H and W represent the height and width of the feature spatial shape, D is the feature channel, K is the number of sampling points, and G is the number of groups in the deformable convolution operation. The deformable field and dynamic weights are computed as Eq. (4):
AP(x) xx WT
deformable X + bieeormablee
W(x) = WT weight + bweight-
(5)
For specific group g in deformable convolution, the sampling position is determined by the base feature position Po. sampling position prior Pk. and predicted deformable Aps from A x)) for the k-th sampling point. The dynamic weight wr provided from W(x). The deformable convolution aggregates K sparse spatial features according to the sampling location and dynamic weight as following:
K w x (Po + Pk + App(x)),
(Po)
(6)
km O
concat
y2 2
y).
The predefined spatial position prior Pk is initialized from the regular convolution, commonly using as the predefined value.
(..(1, (-1,- - 1), ... (). 1, 1)
3

---

Deformable convolution introduces long-range dependencies and dynamic aggregation into regu- lar convolutions, bridging the gap between convolution and multi-head self-attention [24]. Thus, deformable convolution shares the efficiency merit of convolution and the dynamics merit of the attention mechanism. In most scenarios, DCN-like architectures are more powerful than common CNNs, we provide comparison experiments of DCN and CNN of flow matching training. Notably, deformable convolution directly predicts the dynamic weights and only aggregates limited features from spatial locations, enjoying a relatively sparse computation diagram. A deformable convolu- tion operator only requires 4KWWC FLOPs for computation when employing bilinear sampling to aggregate features.
3 Method
3.1 Multi-Scale Deformable Convolution
The original deformable convolution has been widely adopted in hierarchical model architectures [25, 24, 20] for perception tasks. However, these models typically progressively downsample the feature maps to increase the reception field growth rate. In contrast, image generation tasks require outputs with more high-frequency details and low-level information. From this perspective, progressively downsampling features would lead the loss of high-free high-freuency details. One possible solution is to introduce long residual connections to generation models [11, 10]. However, in practice, this approach demands caching image features from the encoder part, which increases peak memory usage during model inference.
To strike balance between receptive fields and high-freeuuee freeurrreuenyy details, propose decoupling the deformable field into scale and direction, and introduce a novel multiscale deformable convolution. Unlike previous deformable convolutions, our approach assigns different scale priors to different groups.
Decoupling deformable field to direction and scale. The original deformable convolution directly regresses the deformable field to learn an unbounded and adaptive sampling point generator. However, the vast image spatial range poses challenge to the learning process, it leads to unstable regression of the deformable range when extrapolating from local neighbors to distant feature locations. We tackle this problem by decoupling the direction and scale of the deformable field. Specifically, reorganize the sampling point formulation in Eq. (9).
s(x) Smax sigmoid W. x),
(8)
p Po + sxx) (px + Apx(x)),
(9)
where s(x) is the learnable scale predicted from the image feature Smax is the max scale value of the given deformable convolution, we leave it as hyper-parameter only related to input resolution, thus we can manually tune according to input resolution, details are placed in Sec. 3.3.
Group-wise multi-scale deformable convolution. To keep high-resolution feature maps and own a large reception field growth rate, we propose assign different scale priors different groups. This allows deformable groups with large scale priors to aggregate long-dependency features, while those with small scale priors aggregate short-dependency features as follows:
3 (x) Smax sigmoid (W. x + +g))
(10)
p = Po + (x)) * (pk + (x))
(11)
Specifically, we initialize the scale priors with Eq. (12) and initialize W, with zeros to obtain linearly increased sigmoid( along group axis:
g+l
log(
(12)
G
3.2 Flow-based Deformable Convolutional Generative Model
We introduce our novel diffusion generation architecture, dubbed FlowDCN. Rather than directly adopting tailored architectures for image generation, such as long residuals and normalization tech- niques, aim to explore the generative capabilities of deformable convolution-basee architectures

---

Precided
Murisiate Plar
*
Uiwar and Reevaas
beformale Conviision
Oetamit Sereb
Lager
FlowDCN
Scak
DON
Pacialy
Fisted
Papedian
Sye Weight
inred slie
Precic Jon
Predecon
Caestio
Labilyy
FlowDCN Architecture. FCowDCN consists of stacked MultiScaleDCN blocks and SwiGLU blocks. We also employ RMSNorm to stabilize training.
(b) MultiScale DCN Block. Dynamic weight and scale& direction deformable field are predicted from input fea- tures, then merged with priors to form the deformable kernels extract features.
Figure 2: The Architecture of Our FlowDCN and MultiScale DCN Block.
in faithful manner. To this end, deliberately discard long residual connections and opt build pure convolution-bas
generative model, preserving the unique characteristics of DCN-like models as much as possible. For training and sampling, we leverage the powerful flow matching algorithm to align our model with the state-of-the-an SiT [23].
Deformable convolution generative model. The model architecture illustrated in Fig. 2a. We aim to build pure DCN-like generative model to explore the generation ability of DCN-like [20] architectures. To match the base resolution of model input with DiT [12] and SiT [23]. similarly
patchify the noisy input via convolution. Inspired by DiT [12]. we inject the timestep and label
conditions through adaLN-Zero [12, 26]. The basic block is formulated as Eq. (13). Drawing
inspiration from LLaMA [27, 28], we replace vanilla FFN and LayerNorm with SwiGLU and
RMSNorm, respectively. Note we also provide FFN and LayerNorm version FlowDCN for fair comparisons:
X1 x AyaLN(y, t, MultiScale-DDNNia
(13)
X x1 AddLLN(y,t, SwiGLU(x;)).
(14)
3.3 Arbitrary Resolution Sampling
We denote the training resolution as Hurain x W iain and the inference resolution as His X W Notably, our FlowDCN is capable of handling arbitrary resolution that differs from the training resolution. As reminder, the multiscale deformable convolution block aggregates features based on predicted scales and directions according Equation (Eq. (9)). In practice, the predicted scale of the multiscale deformable convolution layer is typically fitted to match the training resolution distribution. However, this limits the reception fields of the image features when encountering unseen resolution, ultimately hurting the global semantic consistency [29, 30]. To improve the global semantic consistency, we propose adjusting the scaling factor based on the relative ratio between the training resolution and inference resolution.
Adjust Smax match inference resolution. As shown in Eq. (10), Smax controls the maximum sampling range in multiscale deformable convolution. As discussed in Sec. 3.1, we treat it as resolution-deper dependeen hyperparameter. It is straightforward relative aspect ratio between train size and inference size could match the reception field between train and inference:
observe that scaling Smax with the
Host
5 (x) = sigmoid WTxx x + si)
Smax
(15)
Hmmin
Wizs
2 (x) = sigmoid (W" x + sg) Smax
(16)
W W
5

---

Operator
16 16
Runtime (ms) of Imput Shape H x X X
128
32
61
82 x:
Alteation (Math SDP)
0.92/2.1
1.162.71
10.7/28.8
12.4/35.8
Attention (Flash SDP) [34]
0.62/N
1,77/N
14,4/N
DefomCon<DDCN: (20]
0.77/1 .00
1.0/2.1
4/84
3.9/8.1
DeformCony(Shmm
0.56/0.81
1.1/1.
2.7/3.9
5.07.3
DeformCony (Titon-lang))
0.83K0.89
0.95/111
3,4/3.8
4,0/4.8
Table 1:
Op-level benchmark on standard input shape of Diffusion backbone task. FP16/FP32
results are collected on Nvidia A10 GPU. We use 32 batch sizes for benchmarking. f indicates our Triton- lang [32] implementation of DCNv4, N indicates implementation is not available.
Models
FIDI
sFID,
IST
Kernel
FID
sFID]
IS"
Pcc
s(x)
FID.
sFID]
IS
S/T:S/2
4.47
5.88
8.89
fixed
lixed
5.6
4.58
8.90
DCN-222-
5.47
4.35
8,89
5.47
4.35
8.89
fixed
learn
5.47
4.35
8,89
w/o MultScale
5.72
4.42
8.85
16
5.39
4.54
8.93
learn
fixed
6.01
4.43
8.85
w/o PriorIait
5.68
4.49
5.13
4.43
9.05
learn
learn
5.63
4.37
8.89
(a) Comparsions with SIT. Our
(b) KernelSize K of FlowDCN.
(c) Deformable fields learning set-
FlowDCN outperforms SiT by a sig- large kernel size produces better re- ting. Default achioves best results. nificant margin.
sults than small one.
Table 2: Ablation Studies and Comprasion with other flow-based method 32x32 CIFAR Dataset. In order to fully align with SiT [23]. here we replace our SwiGLU and RMSNorm with FFN and LayerNorm armed in SiT, respectively. Bold font indicates the default setting.
Experiments
We conduct experiments on 32x32 CIFARIO and 256x256 ImageNet datasets. The training batch size set to 256. Similar to SiT [23] and DiT [12], we use Adam optimizer [31] with a constant learning rate 0.0001 during the whole training. We do not adopt any gradient clip techniques for fair comparison. For 32x32 CIFARIO 10 dataset, we train our model for 25000 steps. As for 256x256 ImageNet dataset, train for 1.5M steps. We use A100 GPUs the default training hardware.
Efficient deformable convolution implementation. Although DCNv4 [20] proposes a much faster deformable convolution implementation, it is not tailored for image generation input shape. For reso- lution below 512 x 512, there are fewer spatial tokens (only 16 x 16 tokens for 256 x 256 resolution) to fully utilize sparse computation strengths, thus DCNy4 exhibits even worse latency compared to attention. To remedy high latency of deformable convolution for low-resolution scenery. we decide to leverage shared memory to reduce the latency of random sampling in deformable convolutions, deemed as DeformConv(shm)) We place the performance benchmark at Tab. 1. For high-resolution in Triton-lang DeformCoov(TTitonnbil to
scenery, We also re-implement DeformConn(DCNN) leverage the strengths of compiler [32, 33] to find suitable hyperparameters.
4.1 32x32 CIFAR Dataset
The CIFARIO 355]. comprising 50,000 32x32 small-resolution images from 10 distinct class categories, considered an ideal benchmark validate the design of our MultiScale deformable block due to its relatively small scale. We select SiT-S/2 as comparison baseline, as it also leverages the flow-matching framework. For sampling, we employ the Euler stochastic solver with 1000 sampling steps to generate images. We report the FID [36], sFID [37], and Inception Score [38] as the primary metrics to evaluate the performance of our model.
Compare with baseline SiT. We summarise the metrics of our FlowDCN and SiT in Tab. 2a. Our FlowDCN achieves 5.47 fid, surpassing its counterpart SiT with 2.0 fid margins. Additionally, our model performs slightly better in terms of sFID and Inception Scores, further demonstrating its superiority.
Group-wise multiscale design. As showed in Tab. 2a, we denote the variant of FlowDCN that uses vanilla deformable convolution instead of Multiscale deformable convolution as w/o MultiScale. Notably, the absence of group-wise multiscale deformable convolution leads to 0.25 FID perfor- mance degradation. This result demonstrates the effectiveness and power of our proposed group-wise multiscale mechanism.

---

Prior initialization. By default, we manually initialize the direction priors with predefined grids ((-1,-1), (.,, (1,, and initialize the scale priors with linearly increased scale along group axis. We also experiment with randomly initialized direction and scale priors in Tab. 2a, donated as w/o PriorInit. Random initialization shows slight performance degradation.
Sampling points. In Tab. 2b, We train FlowDCN with varying kernel sizes K and observe that the performance consistently improves as the number increases. Specifically, using 32 points to aggregate features, FlowDCN achieves FID score of 5.13 and an sFID score of 4.43. However, to maintain a relatively sparse pattern, we choose K as the default setting, striking a balance between performance and computational efficiency.
Fixed direction priors. Tab. 2c, we present the results of training FlowDCN with different prior learning settings. Notably, we find that the fixed direction prior ps: in Eq. (9) achieves better results compared to the learnable direction prior. We hypothesize that the learnable direction prior may cause the learning of the deformable field to become unstable, leading to inferior performance.
Learnable relative scale. In the s(x) column of Tab. 2c, the notation "learn" indicates that we predict relative scale of the deformable fields in addition to the learnable scale priors og ( (W" x in Eq. (10)), whereas "fixed" does not predict the relative scales s s(x) in the deformable field. Learning relative scale for each feature in Tab. 2c achieves better results of 5.47 FID.
4.2 256 256 ImageNet Dataset
Based amalysis, select the MultiScale deformable convolution with kernel size of K 9 as the basic block for our Imagenet experiments. Our default setting involves fixing the direction priors and learning relative scales from the deformable field. We manually initialize the direction priors with predefined grids and initialize the scale priors with linearly increased scales. To generate images, we employ Fuler-Maruyama solver with 250 steps for stochastic sampling. We report the FID, sFID, Inception Score, and Precision & Recall as the primary metrics to evaluate the performance of our model.
Model
FLOPs (G)
Params (M) Latency(ms)
FID
sFID
1St
SIT-S/2
6.06
33
0.026
57.64
9.05
24.78
SiT-S/2
6.06
33
0.026
57.9
8.72
24.64
FlowDCN-S/2
4.36 (-28%)
30.3 (-8.1%)
0.027
54.6
8.8
26.4
SiT-B/2
23.01
130
0.084
33.5
6.46
43.71
SiT-B/2
23.01
130
0.084
37.3
6.55
40.6
FlowDCN-B/2
17.87 (-22%)
120 (-7.6%)
0.076
28.5
6.09
51
w/o RMS & SWiGLU
17.88 (-22%)
120 (-7.6%)
0.072
29.1
6.13
50.4
DIT-L/2
80.71
458
0.291
23.3
SYT-L/2
80.71
458
0.291
18.8
5.29
72.02
FlowDCN-L/2
63.51 (-21%) 421 (-8.0%)
0.254
13.8
4.69
85
DIT-XL/2
118.64
675
0.387
19.5
SIT-XL/2
118.64
675
0.387
17.2
5.07
76.52
FLoWDCN-XL/Z
93.24 (-21%)
618 (-8.4%)
0.303
11.3
4.85
97
Table 3: Image generation metries comparisons between SIT [23], DIT [12] under 400k training
steps budgets. All metries are calculated from the sampled 50k images under 250 Euler SDE sampling steps without classifier-free guidance. 1: reproduced result. Latency(ms) is the I-NFE latency and collected from Nvidia A10 GPU with batchsize under float32.
Metrics comparison with baseline SIT. We present the performances of different-size models at 400K training steps in Tab. 8. From Small to XL-size models, our FlowDCN model family consistently outperforms its counterpart DiT [12] and SiT [23] with significant margins. Without RMS/SWiGLU, our FlowDCN- B/2 degrades with 0.6 FID gains but still surpasses SiT by large margin. In addition to its superior performance and convergence speed, our FlowDCN also boasts a remarkable 8% reduction in parameters and at least 20% reduction in FLOPs compared to DIT/SIT. This demonstrates that our FlowDCN surpasses vision transformer based generation models in multiple aspects.
7

---

ODE Solver Steps
SIT
Ours
SIT
Ours
Figure 3: Visualization Comparison with SIT. Best viewed zoomed-in. sample both our FlowDCN- XL/2 and SiT-XL/2 with Euler ODE solver under 2, 3, 4, 5, 8, 10 steps using the same latent noise. At the fewer steps sampling scenery, our FlowDCN generates slightly clearer higher-quality images.
ImtgeNet 256x256 Benchmark
Long
Total Images(M)
Total GFLOPs
FID 4 sFID
IS
P1
R1
Generative Models
Residuals
ADM-U
507
3.76
10t1
7.49
5.13
127.49
0.72 0.63
CDM [39]
4.88
158.71
LDM-4 [40]
213
2.22
10
10.56
103.49 0.71 0.62
DiT-XL/2 [2]
1792
2.13 10a
9.62
6.85
121.50 0.67 0.67
DiffusionSSM-XXLI66
X
660
1.85 x 10
9.07
5.52
118.32
0.69 0.64
SiT-XL/2[3]
X
1792
2.13
10"1
8.61
6.32
131.65
0.68 0.67
FlowDCN-X1/2
X
384
3.57
10
8.36
5.39
122.5
0.69 0.65
Classifier-free Guidance
ADM-U[10]
507
3.76
10
3.60
247.67 0.87 0,48
LDM-4 [40]
213
2.22
10
3.95
178.22 0.81 0.55
U-VIT-H/2 [11]
512
6.81
1040
2.29
247.67
0.87 0.48
DiT-XL/2 [12]
1792
2.13
10Th
2.27
4.60
278.24
0.83 0.57
DiffusionSSM-XL [16]
660
1.85
10
2.28
4.49
259.13
0.86 0.56
SiT-XL/2[233
X
1792
2.13 x 10
2.06
4.50
270.27
0.82 0.59
FTT-X1/2
X
450
4.27
9.99
249.72
0.84 0.51
FlowDCN-XL/2 (cfg=1335; ODE)
384
3.57
1040
2.13
4.30
243.46
0.81 0.57
HowDCN-XX/22 (cfg=1.375; SDE)
384
3.57
10
2.08
4.38
257.53
0.82 0.57
FlowDCN-XL/2 (cfg=1.375; ODE)
486
4.52
10
2.01
4.33
254.36
0.81 0.58
FlowDCN-XI/2 (cfg=1.375;
486
4.52
1010
2.00
4.37
263.16
0.82 0.58
Table 4: Image generation quality evaluation of and existing approaches on ImageNet 256x 256.
Total images by training steps batch size reported, and total GFLOPs by Total Images GFLOPs/maae. refers to Precision and R refers to Recall.
Comparison with other generative models. We report the final metrics of FlowDCN-XL/2 at Tab. 4. Our FlowDCN achieves much faster convergence speed with nearly total images compared its No-Long-residuals counterparts. Additionally, using Euler ODE solver and classifier-free guidance with 1.375, our FlowDCN obtains SoTA 4.30 sFID and 2.13 FID results. Training for extra 400k steps, FlowDCN will be further improved to 2.00 FID. As sFID reflects the spatial structure quality [37], better sFID shows our Flow DCN captures better structure distributions. We notice that the IS metric is lower than other models, however, there is an improvement trend along with training iterations.
Visual quality comparison with baseline SiT. We sample both our FlowDCN-XLL XL/2 and SiT-XL/2 with Euler ODE solver for 2. 3, 4. 5, 8. 10 steps, employing the same latent noise for both models. Notably, the fewer steps sampling scenario, our FlowDCN generates slightly clearer and higher- quality images. We place the generated images at Fig. and Appendix.
4.3 512 512 ImageNet Dataset
As training on high-resolution images consumes much more resources, we opt to fine-tune 100k steps from the same model trained on 256 256 resolution setting of 1.5M steps (corresponding to

---

ClassConnditonal ImageNet 512x512
Model
FID
sFID
1st
Precision
Recallf
BigGAN-dcep [6]
8.43
8.13
177.90
0.88
0.29
StyleGAN-XL [7]
2,41
4.06
267.75
0.77
0.52
ADM [10]
23.24
10.19
58.06
0.73
0.60
ADM-U [10]
9.96
5.62
121.78
0.75
0.64
ADM-G [10]
7.72
6.57
172.71
0.87
0.42
ADM-G, ADM-U
3.85
5.86
221.72
0.84
0.53
DIT-XL/2 [12]
12.03
7.12
105.25
0.75
0.64
DII-XI./2-G [12] (cf)=1.500)
3.04
5.02
240.82
0.84
0.54
SiT-XL/2-G [23] (clg=1.50)
2.62
4.18
252.21
0.84
0.57
DlowDCNN-LL/LIgg=L.33, ODE-50)
2.76
5.29
240.6
0.83
0.51
HLoNDCN-KI/210g11..75..95. SDE-250)
2.44
4.53
252.8
0.84
0.54
Table 5: Benchmarking class-conditional image generation on ImageNet 512x Our FlowDCN-
XL/2 is fine-tuned for 100k steps from the same model trained on 256 x 256 resolution setting of 1.5M steps
with Smax Adjustment
without Smix Adjustment
Figure 4: Visualization Comparison about Smax Adjustment. Here the 512 512, 256 512 and 512 x 256, three type resolution images. We employ the same latent noise as start, sampling with Euler SDE solver for 250 steps. With Smax Adjustment, sampled images consistently looks better.
FlowDCN with 384M training images of Tab. 4). Although fine-tuned with limited 100k steps, our FlowDCN demonstrated powerful performance.
Comparison with other generative models. We report the final metries of Flow on 512 x 512 ImageNet Dataset at Tab. 5. Our FlowDCN achieves much better FID and sFID) performance compared its counterparts. Using Fuler SDE solver with 250 steps and classifier-fre free guidance with 1.375, our FlowDCN obtains 4.53 sFID and 2.44 FID results. Using Euler ODE solver with 50 steps and classifier- free guidance with 1.375, our FlowDCN obtains 5.29 sFID and 2.76 FID result. As shown in Tab. 5, our FlowDCN achieves better sFID and captures better spatial structure distributions.
4.4 Arbitrary Resolution Extension
For the resolution extrapolation evaluation, we follow the setting in FIT. We select 320x320 and 224x448 as the evaluation arbitrary resolution. It is worth noting that our Flow HCN can handle arbitrary resolution within reasonable range, the reasonable range is determined by the training setting and training dataset. As our primary goal is to explore DCN-like architectures in universal image generation, Therefore, we do not employ any multiple aspect ratio training techniques like FiT[18]. Instead, directly use the FlowDCN model trained on the center-croppet 256x256 ImageNet dataset for arbitrary resolution extension experiments, showcasing the model's 's inherent capabilities. Moreover, provide resolution extension experiments with various aspect ratio training techniques in the Appendix.
do not intend to enhance the resolution extrapolation nature by data processing.
Metric comparsion. We report the evaluation results on Tab. 6. For Base-size models, our FlowDCN- B/2 achieves much better results on 320x320 resolution, with 34.4 FID and 35.7 FID using Smmx adjustment, outperforming FiT and DiT with large margin. On 224x448 resolution, our FlowDCN-

---

x (1:1)
224x448 (1:2)
320x (1:1)
224x448 (1:2)
Method
Method
FID
sFID 151
FID
sRID
1S1
HID
sFID]
HiDL sFID)
1ST
DiT-B
95.5
108.7
18.4
109.1
110.7
14.0
ADM G.U [10]
9.39
9.01
162
11.34
146
DiT-Bai
81.5
62.3
21.0
133.2
72.5
11.1
LDM-4140
6.24
13:21
220
8.55
17.62
186
DiT-Bpi
72.5
54.0
24.2
133.4
70.3
11.7
UViT H/2 [1]]
7.65
16,30
208
67.1
42.92
45.5
FTT-B
61.4
30.7
31.0
24.1
37.1
MDT- [41]
383
136
4.24
142.8
4,91
FVT-B, BYYON
44.8
38.0
44.7
41.9
42.8
45.9
DIT [12]
9.98
23.57
225
94,94
56.06
35.7
FTT FTT-BENTK
57.3
31.3
34.0
43.8
26.3
39.2
FIT-XL/2 [18]
5.42
15.41
252
7,9
19.63
215
FlowDCN-B/2
34.4
27.2
52.2
71.7
62.0
23.7
FlowDDN--L/2
5.99
9.71
238
12.8
17.9
168
Sun Adjust
35.7
29.3
51.2
81.1
40.2
21.1
FlowDCN-XL/2
5.86
13.5
275
12.9
20.6
184
(a) Metries Results on Base-Size Models
(b) Metries Results Large-Size Models
Table 6: Benchmarking resolution extrapolations on ImageNet dataset. On the Base-size Models benchmark, our FlowDCN achieves much better results on 320x320 resolution and comparable results on 224x448 resolution. On the Large-Size Models benchmark, our flowDCN shows comparable extrapolation performance to SoTA models.
B/2 achieves comparable results. Note our FlowDCN not employs any various aspect ratio training in Tab. 6, so we believe our FlowDCN-B22 can achieve better results when incorporating such training augmentations. For large-size models, report our FlowDCN-L/2 and FlowDCN-XLA with Smax adjustment in Tab. 6b, our model shows comparable results to SoTA models. Meanwhile, notice that FiT performs poorly on 256x256 resolution in Tab. which hypothesize is due to resolution-related data augmentation hurting the fitting power of original resolution distributions. Furthermore, as FiT employs the 256 x 256 reference statistics from ADM Eval Suite [0] to evaluate all resolution(even for 224 x 448), we suspect this evaluation paradigm is unreasonable.
Visual quality comparison of Smax. In Tab. 6a, We notice FlowDCN-B/2 with Smax adjustment does not exhibit better results than directly generating images, we hypothesize that FID and sFID are low level visual quality assessments, not reflecting semantic visual quality. So we also provide the visualization comparisons of our FlowDCN XL/2 with and without S max Adjustment in Fig. 4 and Appendix. With Smax Adjustment, generated images consistently look better. But not all the cases demand Smax Adjustment, some images like the bubble and the husky case in Fig. 4, still look good without Sma Adjustment. More comparison examples can be found in the Appendix.
5 Conclusion
In this paper, we have presented FlowDCN, a novel deformable convolutional network for arbitrary- resolution image generation. Our FlowDCN model leverages the strengths of both group-wise multiscale deformable convolutions and linear flow to generate high-quality images of various resolutions with high flexibility. Through extensive experiments, we demonstrate that FlowDCN outperforms the state-of-the-art transformer-based counterparts in terms of performance, convergence speed, and computational efficiency. Additionally, our model exhibits strong resolution extrapolation capabilities, achieving comparable results to previous models on arbitrary resolution without any additional training techniques. We believe that FlowDCN has great potential to become powerful tool for a wide range of image generation tasks and applications.
Limitations and Future Works
Our current implementation of deformable convolution backward is inefficient to be on par with Attention. Our primary focus remains on optimizing the training speed. Once we have made significant strides in training optimization, we plan to scale up our FlowDCN to accommodate larger model parameters and higher training resolution, paving the way for more advanced explorations.
Acknowledgement
This work is supported by the National Key R&D Program of China (No. 20222D0160900,, the National Natural Science Foundation China (No. 62076119), 19), the Fundamental Research Funds for the Central Universities (No. 02021 0202143801199 19), the Nanjing University-Chine Mobile Communications Group Co., Ltd. Joint Institute, and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
10

---

References
[1] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020. 2
[2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint
arXiv:2011.13456, 2020. 2
[3] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. Advances in Neural Information Processing Systems, 35:26565- 26577, 2022, 2
[4] Xingchao Liu, Chengyue Gong, and Qiang Liu, Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint wwXiv2220.03000) 2022. 2,
[5] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:22/0.0277:0 2022. 2,
[6] Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. /1066, 2018. 2,
[7] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1- 1-10, 2022. 2, 9.9
[8] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022. 2
[9] Michacl S Albergo, Nicholas M Boffi, and Eric Vanden- Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303. 0879777 2023.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021. 2, 10
[11] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22669-22677 2023. 2, 4, 10
[12] William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023, 2, 5, 6, 7, 8, 9, 10
[13] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu,, et al. Pixart-kalpha: Fast training of diffusion transformer for photorealistic text-to- -ottae synthesis. arXiv preprint 21X00:2310.00466, 2023. 2
[14] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Ln, and Zhenguo Li. Pixart-\sigmaa Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprini arXiv:2403..m 04449222..46..... 2024. 2
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Bocsel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.00 00300000,, 2024.
[16] Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. arXiv preprint arXiv:2311.18227. 2023. 8
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
[18] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402. /2376, 2024, 2, 8, 9, 10
[19] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. 2
[20] Yuwen Xiong, Zhiqi Li, Yuntao Chen, Feng Wang, Xizhou Zhu, Jiapeng Luo, Wenhai Wang, Tong Lu, Hongsheng Li, Yu Qiao, et al. Efficient deformable convnets: Rethinking dynamic and sparse operator for vision applications. arXiv preprint arXiv:2401 0669977006699.... 2024. 2, 6
[21] Zhuang Liu, Hanzi Mao, Chao-Yusa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 1997-119886 2022. 2
[22] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kerells to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11963-11973 2022. 2

---

[23] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Manden-Eijnded and Saining Xie. Sit: Exploring flow and diffuusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401. 087400, 2024. 2, 5, 8,
[24] W Wang, J Dai, Z Chen, Z Huang. Li, X Zhu, X Hu, Lu, L Lu, Li, et al. Internimage: Exploring large- scale vision foundation models with deformable convolutions. arxiv. arXiv preprint arXiv:2211.05778. 2022. 4
[25] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9308-9316, 2019. 4
[26] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017. 5
[27] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilama: Open and efficient foundation language models. arXiv preprini arXiv: 13971, 2023. 5
[28] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 1.xiv::3377.92888 2023, 5
[29] Yingqing He, Shaoshu Yang. Haoxin Chen, Xiaodong Cun. Menghan Xia, Yong Zhang, Xintao Wang. Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-fite free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations, 2023. 5
[30] Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Zhenyuan Chen, Yao Tang, Yuhao Chen, Wengang Cao, and Jiajun Liang. Hidiffusion: Unlocking high-rcsolution creativity and efficiency low-rcsolution trained diffusion models. arXiv preprint arXiv:231 2023. 5
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
[32] Philippe Tillet, Hsiang-T Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10-19, 2019.
[33] Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis & transformation. In International symposium code generation and optimization, 2004. CGO 2004., pages
75-86. 86. IEEE, 2004.
[34] Tri Dao. Dan Fu, Stefano Ermon. Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 15:16444-16355 2022.
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Advances in neural information processing systems, 2009. 6
[36] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6
[37] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse
representations. arXiv preprini (.XXi:2210300884 2021. 6, 8
[38] Tim Salimans, Ian Goodfellow, Wajeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[39] Jonathan Ho. Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded difffusion models for high fidelity image generation. Journal of Machine Learning Research. 23(47):1-33, 2022. 8
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684- 10695, 2022. 8, 10
[41] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 23164-23173, 2023. 10
12

---

A. Model Details
Model
Layers N
Hidden size
Groups
FlowDCN-S
12
384
FlowDCN-B
768
FlowDCN--L
24
1024
16
FlowDCN-XL
28
1152
16
Table 7: Details of FlowDCN models. We follow DiT for the Small (S), Base (B), Large (L) and XLarge
(XL) model configurations.
B. Comparisons between FlowCNN and FlowDCN on ImageNet 256 x 256
The relationship between DCN and common CNN. As Eq. (6) states, DCN introduces deformable field Ap(x) and dynamic weight w(xx). When all features shares the same static weight instead of dy- namic, and deformable field Ap(x: degrades zeros, DCN degenerates to common in most scenarios, DCN-like architectures are more powerful than common CNNs. Furthermore, the fix p& Tab. 2cindicates that we freeze the pt (not the deformable field Ap(x)) and initialize it with a predefined grid.
Why not try common CNN architecture. In many computer vision tasks, traditional CNNs have been outperformed by transformers, so we opted to explore the modern, advanced CNN variant, Deformable Convolutional Networks (DCN). Additionally, conducted small experiment where we replaced the DCN block in FCowDCN with standard 3x3 and 5x5 group-wise convolution blocks.
Model
layers groups channels Params (M)
sFID IS
SIT-S/2
12
6
384
33.0
57.64
9.05
24.78
FlowCNN
3x3
12
512
49.1
59.0
10.7
27.4
FlowCNN-55
12
384
33.1
63.0
10.9
23.6
FlowDCN-SS
12
384
30.3
54.6
8.8
26.4
Table 8: Image generation metrics comparisons between SiT, FlowDCN and FlowCNN under
400k training steps budgets.
256 (1:1)
320x 320 (1:1)
224x (1:2)
160x480 (1:3)
Method
FID.
sFID.
1S1
FIDJ
sFID.
18?
FID.
sFID)
IST
FID.
sFID.
IS1
DiT-B
44.83
8.49
32.05
95.47
108.68
18.38
109.1
110.71
14.00
122.81
DiT-B
44.83
8.49
32.05
81.48
62.25
20.97
133.2
72.53
11.11
160.4
93.91
7.30
DiT-B P
44.83
8,49
32.05
72.47
54.02
24.15
133.4
70.29
11.73
156.5
93.80
7,80
FiT-B
36.36
11.08
40.69
61.35
30.71
31.01
44.67
24.09
37.1
56.81
22.07
25.25
FIT-B VisionYaRN
36.36
11.08
44.76
38.04
44.70
41.92
42.79
45.87
62.84
44.82
27.84
F VisionNTK
36.36
11.08
40.69
57.31
31.31
33.97
43.84
26.25
39.22
56.76
24.18
26.40
FlowDCN-B
28.5
6.09
51
34.4
27.2
52.2
71.7
62.0
23.7
211
111
5.83
FloDDCN (+VAR)
7.72
15.88
69.5
Smn Adjust
23.6
7.72
62.8
30.7
19.4
68.5
37.8
22.8
54.4
53.3
22.6
31.5
Table 9: Benchmarking resolution extrapolations on ImageNet with various aspect ratio training. VAR indicates various aspect ratios training. We follow the same evaluation pipeline of FIT without using CFG.
C. Resolution Extension with Various Aspect Ratios Training
While FlowDCN, trained on fixed-resolution images, is capable of generating images of arbitrary resolution within a reasonable aspect ratio range, its performance can be improved by adopting variable aspect ratio (VAR) training instead of fixed 256x256 resolution. To ensure fair comparison with FiT, which inherently uses VAR, we train FlowDCN-B/2 model from scratch using VAR techniques. We evaluate our model using the same pipeline and reference batch FiT, without CFG.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We plan to opensource our code and implementation later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: //tp///iippp//e cc/
publ / guides
des CodeSuumm ss asion sisioollli liyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cttppcccppccccccccc pubppcplco.ooo /wwpu/gusssssee/eeeeeeee Code Submi ssissiionnn ceiionnoliinn Piyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Running experiments demands a lot of resources and time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips..... hc/ppuulccc/tthhcccccleeee.ee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Current models are only trained small datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper wi hhoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "NQBA2FLROw", "reviewer": "Reviewer_rZWQ", "summary": "This paper presents a purely deformable convolution based architecture for flow-matching based diffusion models. Such a purely convolution-based models can easily generalize to different aspect ratio/resolution during testing, which is a pain point for transformer-based models. Evaluated on ImageNet 256x256 benchmark, the proposed FlowDCN achieves promising results with fewer parameters/FLOPs and faster sampling speed.", "questions": "Please see weakness for detailed questions.\n\nAfter reading the paper, although I have some concerns in the representation and experiments, I do appreciate the efforts on purely convolution architecture for diffusion models, which naturally generalizes better to different resolutions compared to transformer-based ones. Thus my initial rating is weak accept. I expect the author can justify the usage of deformable convolution based on results from Tab. 3 (c), and how would the model perform on ImageNet 512 benchmark under \"resolution extension\" setting.", "limitations": "No other limitations as far as I am aware.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow\n\n\n- I like the simple idea of adopting convolution-based architecture to achieve a better generalization across different resolution, which seems a simple yet effective solution IMO.\n\n- The adaptation of deformable convolution and corresponding optimization is technically solid.\n\n- Extensive experiments and good performance", "weaknesses": "Although I like the idea of using a CNN style architecture for better generalization across different resolution, I do have several questions after reading the paper:\n\n- In Tab. 2(c), if I understand correctly, the deformable conv degrades back to a normal conv when p_k and s(x) are both fixed. From the table it achieves a similar performance to the final model, which makes it less convincing to use deformable convolution, which requires additional CUDA optimization etc. Taking Tab. 2 (b) into consideration as well, it seems that a normal convolution with larger kernel can do as well. Please illustrate why deformable conv is especially needed. Although normal convolution is a local op, under the setting of input size 32 x 32 (Cifar, or ImaegNet with VAE latents), a normal conv with larger kernel seems \"global\" enough IMHO.\n\n- Tab.3 and Tab.4 reports different FLOPs, which is confusing. I see Tab.4's GFLOPs seems refer to the total training compute. It would be better if they can be explicitly annotated to avoid confusion.\n\n- Although FlowDCN has a comparable or better FID compared to SiT or DiT, its Inception score falls behind, can the author provide some in-depth analysis on this?\n\n- One major advantage of conv-based models against transformer-based models is the low complexity when scaling up to larger resolution input. It would be great if the effectiveness of FlowDCN can be verified on ImageNet 512 x 512 as well besides 256 x 256. I understand the 512 experiments can be costly and do not expect to see them in the rebuttal, but would highly appreciate if at least the results of testing 256 x 256 models on 512 x 512 benchmarks can be reported, as \"arbitrary resolution extension\" is claimed to be one major advantages of FLowDCN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The relationship between DCN and common CNN.**\nAs Eq.3 states, DCN introduces a deformable field $\\Delta p(x)$ and a dynamic weight $w(x)$. When all channels share the same static weights instead of dynamic ones, and deformable field $\\Delta p(x)$ degrades to zeros, DCN degenerates to common CNN. Therefore, in general, DCN-like architectures are more flexible and powerful than common CNNs. It should be noted that, the *fixed $p_k$* in Tab. 2 (c) indicates that we freeze the $p_k$ (not the deformable field $\\Delta p(x)$) and initialize it with a predefined grid. This setting is not equivalent to the common CNN.\n\n\n**Why not try a normal convolution.** In many computer vision tasks, traditional CNNs have been outperformed by transformers, so we opted to explore a modern and advanced variant of CNN: Deformable Convolutional Networks (DCN). Additionally, we conducted a small experiment on ImageNet 256, where we replaced the DCN block in FlowDCN with a standard 3x3 group-wise convolution block. Due to limited time, the experiments with larger kernel of 5*5 is still on the way.\n| Model         | Params (M) | FID$\\downarrow$ | sFID$\\downarrow$ | IS$\\uparrow$ |\n|---------------|------------|-----------------|------------------|--------------|\n| SiT-S/2       | 33.0       | 57.64           | 9.05             | 24.78        |\n| FlowCNN-S/2   | 49.1       | 59.0            | 10.7             | 27.4         |\n| FlowDCN-S/2 | 30.3       | 54.6          | 8.8              | 26.4       |\n\n**Different annotations for Total Training GFLOPs and 1-NFE FLOPs in Tab3. & 4.** We appreciate your insightful feedback and concur with your opinions. We will clarify this point finally.\n\n**Lower IS metric.** In Tab. 3, our FlowDCN consistently achieves superior performance in terms of the IS metric compared to its counterparts. However, the final results reported in Tab. 4 surprisingly yield relatively lower IS scores compared to DiT and SiT. Notably, we observe a trend of IS improvement when extending the training iteration by an additional 400k steps, with the score increasing from 257.5 at 1.5M training steps to 263.26 at 1.9M training steps. This suggests that the limited number of training steps may be the primary cause of the lower IS metric.\n\n**ImageNet $512\\times512$ Experiments.**\nDue to the time constraint on the rebuttal, training FlowDCN-XL/2 from scratch on ImageNet at $512\\times512$ resolution is not feasible. Instead, we fine-tune our pre-trained FlowDCN-XL/2 model, which was trained on ImageNet at $256\\times256$ resolution for 1.5M steps, for just 100k steps on ImageNet 512. We adopt the same training pipeline as the original $256\\times256$ resolution setting, without incorporating advanced techniques such as lognorm sampling, aspect ratio augmentation, and random cropping. Notably, our approach achieves a remarkable 2.76 FID score with the ODE solver in 50 steps, and 2.44 FID with the SDE solver in 250 steps.\n\n| Model                                     | FID$\\downarrow$ | sFID$\\downarrow$ | IS$\\uparrow$   | Precision$\\uparrow$ | Recall$\\uparrow$ |\n|-------------------------------------------|-----------------|------------------|----------------|---------------------|------------------|\n| BigGAN-deep                               | 8.43            | 8.13             | 177.90         | 0.88                | 0.29             |\n| StyleGAN-XL                               | 2.41            | 4.06             | 267.75         | 0.77                | 0.52             |\n| ADM                                       | 23.24           | 10.19            | 58.06          | 0.73                | 0.60             |\n| ADM-U                                     | 9.96            | 5.62             | 121.78         | 0.75                | 0.64           |\n| ADM-G                                     | 7.72            | 6.57             | 172.71         | 0.87              | 0.42             |\n| ADM-G, ADM-U                              | 3.85            | 5.86             | 221.72         | 0.84                | 0.53             |\n| DiT-XL/2                                | 12.03           | 7.12             | 105.25         | 0.75                | 0.64           |\n| DiT-XL/2-G (cfg=1.50)                   | 3.04          | 5.02           | 240.82       | 0.84                | 0.54             |\n| **FlowDCN-XL/2(cfg=1.375, ODE-50)**  | 2.76            | 5.29             | 240.6          | 0.83                | 0.51             |\n| **FlowDCN-XL/2(cfg=1.375, SDE-250)** | **2.44**   | **4.53**    | **252.8** | 0.84                | 0.54             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5bdc7nIP4", "reviewer": "Reviewer_zBRq", "summary": "The paper proposes a novel convolutional-based generative model called FlowDCN. This model addresses the challenge of generation speed and arbitrary-resolution image generation, which remains difficult for transformer-based diffusion methods due to their quadratic computation cost and limited resolution extrapolation capabilities. FlowDCN introduces a group-wise multiscale deformable convolution block, enabling the model to handle varying resolutions efficiently. The paper claims that FlowDCN achieves state-of-the-art performance on the 256x256 ImageNet benchmark, surpassing transformer-based counterparts in terms of convergence speed, visual quality, parameter efficiency, and computational cost.", "questions": "The FID in Table 5 seems too high, is that because the models are trained with 400k iter and without cfg?", "limitations": "The authors have discussed the limitations and that makes sense to me.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper presents a novel approach by leveraging multiscale deformable convolution blocks, which is innovative compared to traditional transformer-based models.\n- It demonstrates impressive performance on the 256x256 ImageNet benchmark, achieving state-of-the-art FID scores with reduced parameters and FLOPs compared to other models. The efficiency is also improved with linear time and memory complexity, making it suitable for fast image generation.", "weaknesses": "It would be great to test FlowDCN on higher resolution such as 512x512 or higher, to showcase the efficiency of using a convolution-based model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**FID in Table 5 seems too high.** Yes, we follow SiT, FiT and DiT to train our FlowDCN under 400K budgets. We follow the evaluation pipeline of FiT to obtain the metrics of arbitrary resolution generation without using CFG.\n\n\n**ImageNet $512\\times512$ Experiments.**\nGiven the time constraint on the rebuttal, it is infeasible to train FlowDCN-XL/2 from scratch on ImageNet at a resolution of 512x512. As an alternative, we fine-tune our pre-trained FlowDCN-XL/2 model, which was initially trained on ImageNet at 256x256 resolution for 1.5 million steps, for an additional 100,000 steps on ImageNet 512. We utilize the same training pipeline as the original 256x256 resolution setting. Notably, our approach achieves a remarkable FID score of 2.76 with the ODE solver in just 50 steps, and 2.44 with the SDE solver in 250 steps.\n\n| Model                                     | FID$\\downarrow$ | sFID$\\downarrow$ | IS$\\uparrow$   | Precision$\\uparrow$ | Recall$\\uparrow$ |\n|-------------------------------------------|-----------------|------------------|----------------|---------------------|------------------|\n| BigGAN-deep                               | 8.43            | 8.13             | 177.90         | 0.88                | 0.29             |\n| StyleGAN-XL                               | 2.41            | 4.06             | 267.75         | 0.77                | 0.52             |\n| ADM                                       | 23.24           | 10.19            | 58.06          | 0.73                | 0.60             |\n| ADM-U                                     | 9.96            | 5.62             | 121.78         | 0.75                | 0.64           |\n| ADM-G                                     | 7.72            | 6.57             | 172.71         | 0.87              | 0.42             |\n| ADM-G, ADM-U                              | 3.85            | 5.86             | 221.72         | 0.84                | 0.53             |\n| DiT-XL/2                                | 12.03           | 7.12             | 105.25         | 0.75                | 0.64           |\n| DiT-XL/2-G (cfg=1.50)                   | 3.04          | 5.02           | 240.82       | 0.84                | 0.54             |\n| **FlowDCN-XL/2(cfg=1.375, ODE-50)**  | 2.76            | 5.29             | 240.6          | 0.83                | 0.51             |\n| **FlowDCN-XL/2(cfg=1.375, SDE-250)** | **2.44**   | **4.53**    | **252.8** | 0.84                | 0.54             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNgMMFMGdH", "reviewer": "Reviewer_yBFS", "summary": "This paper introduces FlowDCN, a novel image generation model that efficiently generates high-quality images at various resolutions. The model's core innovation is a group-wise multiscale deformable convolution block, which enhances its adaptability to various resolutions. Built on flow-based generative models, FlowDCN demonstrates improved flexibility compared to DiT, FiT, SiT. The group-wise multiscale deformable convolution operation proves more efficient than conventional attention operation and DCNv4 operation. Additionally, the model incorporates a straightforward yet effective scale adjustment method for resolution extrapolation.", "questions": "My question is mainly about the presentation and experiment of this paper, which have been described in detail in Weakness.", "limitations": "Yes, the authors candidly acknowledge several limitations in their study. First, they note that the current implementation of the backward pass for deformable convolution is inefficient, indicating potential for future optimization. Second, the research lacks experiments with higher resolution images, particularly at 512x512 pixels, which could provide valuable insights into the model's performance on larger visual data. Finally, the authors recognize the absence of experiments with larger model variants, which limits the exploration of FlowDCN's scalability. These identified shortcomings not only demonstrate the authors' critical assessment of their work but also highlight promising directions for future research and improvements to the model.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "I have summarized the following two strengths of this paper:\n1. **Techinical Contributions**: \nThe originality of this work lies in the creative application of deformable convolution to the image generation domain. By rethinking the design of this core building block, the authors have developed a new generative architecture that outperforms existing approaches. This is a novel Techinical contribution, as prior work has primarily focused on adapting transformer-based models for generation tasks.\n\n2. **Experiment Results are Complete**:\nThe authors provide a comprehensive evaluation of their new deformable convolution operator, thoroughly examining the impact of each component. They present results for FlowDCN across various resolution settings (256x256, 320x320, 224x448), demonstrating its versatility. Additionally, the paper offers an efficiency analysis comparing their deformable convolution operator with the traditional attention operator and the DCNv4 operator. The authors also provide a detailed analysis of FlowDCN's computational requirements, including FLOPs, latency, and parameter count, offering valuable insights into the model's performance and resource utilization.", "weaknesses": "I think there are two main weaknesses of this paper:\n\n1. **Presentation**:\nFirstly, I recommand the authors should consider restructuring the paper to include a separate preliminary section. This section should cover the background on Deformable Convolution Revisited (Sec 2.1) and Linear-based flow matching (Sec 2.3), as these are not novel contributions of this work but rather foundational concepts. Secondly, the addition of a related work section would greatly benefit the reader. This section should provide a concise overview of relevant literature on generative models and vision backbone architectures. Lastly, the paper would benefit from a thorough proofreading to address typographical errors. For instance, in Appendix E, the title incorrectly uses \"Uncarted\" instead of \"Uncurated\". A careful review of the entire manuscript would help eliminate such oversights and enhance the overall quality of the presentation.\n\n2. **Experiment**: The authors conducted their ablation studies primarily on the CIFAR dataset. However, it would be beneficial to extend these studies to larger datasets, such as ImageNet-1K, following the DiT. This expansion is crucial because performance characteristics can vary significantly between small and large datasets, potentially offering more comprehensive insights into the model's behavior. Furthermore, to facilitate a direct comparison with the FiT, it is recommended that the authors include evaluations at additional resolutions, specifically 160x480, 160x320, and 128x384. These additional evaluations would provide a more complete and robust comparison and offer a broader perspective on the model's performance across various image sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Presentation issues.**\nThanks for your suggestions. We will re-organize the structure of our paper in the final version. We will add a preliminary section to introduce the background on DCN and Flow Matching. We will also rewrite the related work section to comprehensively discuss the relevant literature. The final version will be carefully revised according to your comments.\n\n**Ablation issues.**\nThanks for your suggestion. We will conduct ablation studies on ImageNet256 to offer more comprehensive results into the model's behavior. Due to limited time, we cannot provide the detailed results before the rebuttal ddl.\n\n**ImageNet $512\\times512$ Experiments.**\nWe try to provide the ImageNet512 results of FlowDCN-XL/2 to show the generation power on high-resolution images. Due to the time constraint on the rebuttal, training FlowDCN-XL/2 from scratch on ImageNet at $512\\times512$ resolution is not feasible. Instead, we fine-tune our pre-trained FlowDCN-XL/2 model, which was trained on ImageNet at $256\\times256$ resolution of 1.5M steps, for just 100k steps on ImageNet 512. We adopt the same training pipeline as the original $256\\times256$ resolution setting, without incorporating advanced techniques such as lognorm sampling, aspect ratio augmentation, and random cropping. Notably, our approach achieves a remarkable 2.76 FID score with the ODE solver in 50 steps, and 2.44 FID with the SDE solver in 250 steps.\n| Model                                     | FID$\\downarrow$ | sFID$\\downarrow$ | IS$\\uparrow$   | Precision$\\uparrow$ | Recall$\\uparrow$ |\n|-------------------------------------------|-----------------|------------------|----------------|---------------------|------------------|\n| BigGAN-deep                               | 8.43            | 8.13             | 177.90         | 0.88                | 0.29             |\n| StyleGAN-XL                               | 2.41            | 4.06             | 267.75         | 0.77                | 0.52             |\n| ADM                                       | 23.24           | 10.19            | 58.06          | 0.73                | 0.60             |\n| ADM-U                                     | 9.96            | 5.62             | 121.78         | 0.75                | 0.64           |\n| ADM-G                                     | 7.72            | 6.57             | 172.71         | 0.87              | 0.42             |\n| ADM-G, ADM-U                              | 3.85            | 5.86             | 221.72         | 0.84                | 0.53             |\n| DiT-XL/2                                | 12.03           | 7.12             | 105.25         | 0.75                | 0.64           |\n| DiT-XL/2-G (cfg=1.50)                   | 3.04          | 5.02           | 240.82       | 0.84                | 0.54             |\n| **FlowDCN-XL/2(cfg=1.375, ODE-50)**  | 2.76            | 5.29             | 240.6          | 0.83                | 0.51             |\n| **FlowDCN-XL/2(cfg=1.375, SDE-250)** | **2.44**   | **4.53**    | **252.8** | 0.84                | 0.54             |\n\n**Various Aspect Ratios training Experiments.** Although FlowDCN, trained on fixed-resolution images, can generate images of arbitrary resolution within a reasonable aspect ratio range, the quality can be further enhanced by employing variable aspect ratio (VAR) training instead of a fixed $256\\times256$ resolution. Therefore, we train a FlowDCN-B/2 model from scratch using VAR techniques to ensure a fair comparison with FiT, which inherently adopts VAR. We follow the same evaluation setting of FiT,  using the same reference batch, without CFG. Note that the generated resolutions are the same with the FiT.\n\n|                       | 256x256 FID | sFID   | IS      | 320x320 FID | sFID    | IS      | 224x448 FID | sFID    | IS      | 160x480 FID | sFID    | IS      |\n|-----------------------|-------------|--------|---------|-------------|---------|---------|-------------|---------|---------|-------------|---------|---------|\n| DiT-B                 | 44.83       | 8.49 | 32.05   | 95.47       | 108.68  | 18.38   | 109.1       | 110.71  | 14.00   | 143.8       | 122.81  | 8.93    |\n| with EI            | 44.83       | 8.49 | 32.05   | 81.48       | 62.25   | 20.97   | 133.2       | 72.53   | 11.11   | 160.4       | 93.91   | 7.30    |\n| with PI            | 44.83       | 8.49 | 32.05   | 72.47       | 54.02   | 24.15   | 133.4       | 70.29   | 11.73   | 156.5       | 93.80   | 7.80    |\n| FiT-B (+VAR)          | 36.36     | 11.08  | 40.69 | 61.35       | 30.71 | 31.01   | 44.67       | 24.09 | 37.1    | 56.81       | 22.07 | 25.25   |\n| with VisionYaRN  | 36.36     | 11.08  | 40.69 | 44.76     | 38.04   | 44.70 | 41.92     | 42.79   | 45.87 | 62.84       | 44.82   | 27.84 |\n| with VisionNTK   | 36.36     | 11.08  | 40.69 | 57.31       | 31.31   | 33.97   | 43.84       | 26.25   | 39.22   | 56.76     | 24.18   | 26.40   |\n| FlowDCN-B             | 28.5        | 6.09   | 51      | 34.4        | 27.2    | 52.2    | 71.7        | 62.0    | 23.7    | 211         | 111     | 5.83    |\n| FlowDCN-B (+VAR)      | 23.6        | 7.72   | 62.8    | 29.1        | 15.8    | 69.5    | 31.4        | 17.0    | 62.4    | 44.7        | 17.8    | 35.8    |\n| with $S_{max}$ adjust | 23.6        | 7.72   | 62.8    | 30.7        | 19.4    | 68.5    | 37.8        | 22.8    | 54.4    | 53.3        | 22.6    | 31.5    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvSWXAuM7e", "reviewer": "Reviewer_LWE8", "summary": "This paper presents a new convolution-based diffusion model that is able to generate images at arbitrary resolutions when trained on fixed resolutions. It also achieves comparable performance when trained and tested on the same resolution compared to transformer backbones such as U-VIT and DiT.", "questions": "Please see the question in the weaknesses section. I will be willing to raise my rating if these questions are resolved.", "limitations": "Yes, discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This model is efficient: it achieves similar performance compared to SOTA methods but with 20% lower latency, 8% fewer parameters, and approximately 20% fewer floating-point operations.\n- It enables arbitrary-resolution generation, which is an interesting achievement. Previously, we needed to fine-tune diffusion models on specific resolutions to get good images. Now this enables more flexible generation.", "weaknesses": "- As I mentioned, what I like most about this paper is the ability for arbitrary resolution generation. However, I think more experiments regarding handling arbitrary resolutions are needed. For example, it is not very clear which part enables the proposed method to handle arbitrary resolutions. In the contribution section, the authors attribute this ability to Scale Adjustment. Does the MultiScale DCN Block also help? However, in Line 245, the authors say this ability is determined by the training setting and training dataset, which is also confusing since the training data is 256x256 ImageNet.\n- In Table 4, it would be good to see what happens if this model is trained for more iterations. Currently, the model doesn’t beat SiT-XL/2 on ImageNet 256. I wonder if the model will get better results when trained for longer iterations. (My personal experience on DCN is that it sometimes converges faster but not necessarily better.)\n- One interesting benefit of the proposed model is the ability to generate images at any resolution. However, the results are worse than FiT on the 1:2 setting (Table 5). It would be great to see if the performance improves when multiple aspect ratio training augmentations are used. In addition, what reference statistics are you using for the 1:2 setting? And what is the performance when you also use 256×256 reference statistics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Which part contributes most to arbitrary resolution generation?**\nSorry for the confusion. Basically, the fundamental contribution to the ability of arbitrary resolution generation is our MultiScale DCN Block, which equips our model with the flexibility to handle different resolutions. So, the sentence in Line 245 is miss-leading and will be revised in the final version. Smax adjustment is another technique to further improve the global semantic consistency of generated images, as observed in Fig. 4. In rebuttal experiment, we further find aspect ratio augmentation during training is helpful to improve its performance on arbitary resolution generation. This result is easy to understand as data augmentation typically contributes to a better model with higher performance.\n\n**ImageNet $256\\times256$ Long Iterations Experiments.**\nNotably, our FlowDCN-XL/2 model, trained for only 1.5 million steps, achieves comparable results to SiT, which was trained for 7 million steps. This raises the question of whether FlowDCN-XL/2 is inherently more powerful or simply converges faster. To clarify this, we extended the training of our FlowDCN-XL/2 model by an additional 400k steps. The results show that our model attains a FID score of 2.01 with the ODE solver and 2.00 with the SDE solver, significantly outperforming SiT.\n\n| Generative Models           | Total Images（M） | Total GFLOPs                   | FID $\\downarrow$ | sFID $\\downarrow$ | IS $\\uparrow$   | P $\\uparrow$  | R $\\uparrow$  |\n|-------------------------------------------------|-----------------|--------------------------------|------------------|-------------------|-----------------|---------------|---------------|\n| ADM-U                                           | 507             | $3.76\\times10^{12}$            | 3.60             | -                 | 247.67          | 0.87          | 0.48          |\n| LDM-4                                           | 213             | $2.22\\times10^{10}$            | 3.95             | -                 | 178.22          | 0.81          | 0.55          |\n| U-ViT-H/2                                       | 512             | $6.81\\times10^{10} $           | 2.29             | -                 | 247.67          | {0.87}        | 0.48          |\n| DiT-XL/2                                        | 1792            | $2.13\\times10^{11}$            | 2.27             | 4.60              | 278.24 | 0.83          | 0.57          |\n| DiffusionSSM-XL                                 | 660             | $1.85\\times10^{11}$            | 2.28             | 4.49              | 259.13          | 0.86 | 0.56          |\n| SiT-XL/2                                        | 1792            | $2.13\\times10^{11}$            | 2.06         | 4.50              | 270.27          | 0.82          | 0.59 |\n| FiT-XL/2                                        | 450             | -                              | 4.27             | 9.99              | 249.72          | 0.84          | 0.51          |\n| FlowDCN-XL/2  (cfg=1.375; ODE) | 384    | ${3.57\\times10^{10}}$ | 2.13           | 4.30     | 243.46        | 0.81       | 0.57       |\n| FlowDCN-XL/2 (cfg=1.375; SDE) | 384   | ${3.57\\times10^{10}}$ | 2.08          | 4.38            | 257.53        | 0.82      | 0.57        |\n|FlowDCN-XL/2 (cfg=1.375; ODE) | 486    | ${4.52\\times10^{10}}$ | 2.01          | 4.33            | 254.36        | 0.81        | 0.58     |\n| FlowDCN-XL/2 (cfg=1.375; SDE) | 486    | ${4.52\\times10^{10}}$ | 2.00    | 4.37            | 263.16        | 0.82        | 0.58      |\n\n\n**Various Aspect Ratios training Experiments.** While FlowDCN, trained on fixed-resolution images, is capable of generating images of arbitrary resolution within a reasonable aspect ratio range, its performance can be improved by adopting variable aspect ratio (VAR) training instead of a fixed 256x256 resolution. To ensure a fair comparison with FiT, which inherently uses VAR, we train a FlowDCN-B/2 model from scratch using VAR techniques. We evaluate our model using the same pipeline and reference batch as FiT, without CFG.\n\n|                       | 256x256 FID | sFID   | IS      | 320x320 FID | sFID    | IS      | 224x448 FID | sFID    | IS      | 160x480 FID | sFID    | IS      |\n|-----------------------|-------------|--------|---------|-------------|---------|---------|-------------|---------|---------|-------------|---------|---------|\n| DiT-B                 | 44.83       | 8.49 | 32.05   | 95.47       | 108.68  | 18.38   | 109.1       | 110.71  | 14.00   | 143.8       | 122.81  | 8.93    |\n| with EI            | 44.83       | 8.49 | 32.05   | 81.48       | 62.25   | 20.97   | 133.2       | 72.53   | 11.11   | 160.4       | 93.91   | 7.30    |\n| with PI            | 44.83       | 8.49 | 32.05   | 72.47       | 54.02   | 24.15   | 133.4       | 70.29   | 11.73   | 156.5       | 93.80   | 7.80    |\n| FiT-B (+VAR)          | 36.36     | 11.08  | 40.69 | 61.35       | 30.71 | 31.01   | 44.67       | 24.09 | 37.1    | 56.81       | 22.07 | 25.25   |\n| with VisionYaRN | 36.36     | 11.08  | 40.69 | 44.76     | 38.04   | 44.70 | 41.92     | 42.79   | 45.87 | 62.84       | 44.82   | 27.84 |\n| with VisionNTK  | 36.36     | 11.08  | 40.69 | 57.31       | 31.31   | 33.97   | 43.84       | 26.25   | 39.22   | 56.76     | 24.18   | 26.40   |\n| FlowDCN-B             | 28.5        | 6.09   | 51      | 34.4        | 27.2    | 52.2    | 71.7        | 62.0    | 23.7    | 211         | 111     | 5.83    |\n| FlowDCN-B (+VAR)      | 23.6        | 7.72   | 62.8    | 29.1        | 15.8    | 69.5    | 31.4        | 17.0    | 62.4    | 44.7        | 17.8    | 35.8    |\n| with $S_{max}$ adjust | 23.6        | 7.72   | 62.8    | 30.7        | 19.4    | 68.5    | 37.8        | 22.8    | 54.4    | 53.3        | 22.6    | 31.5    |\n\n**The Evaluation of Arbitrary Resolution** We follow the same evaluation pipeline of FiT,  using the same reference statistics of ImageNet 256x256, without CFG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work presents FlowDCN, a generative model using pure convolutions, enabling arbitrary resolution image generation. Additionally, the authors propose a new design of learnable group-wise deformable convolution block. Promising results have been demonstrated on the ImageNet generation benchmark.

Initially, the reviewers were concerned about the relationship between DCN and common CNN, experiments on ImageNet 512x512 (FID and latency), paper presentation, and so on. During the reviewer-author discussion period, the provided rebuttal successfully assuaged the reviewers' concerns. In the end, the paper received all accept recommendations (2 borderline accepts, 1 weak accept and 1 strong accept). After considering the author rebuttal and reviewer discussion/comments, the area chair agrees with this recommendation. 

Finally, the authors are strongly encouraged to incorporate the rebuttal and reviewer suggestions to their final camera-ready version.
Decision: Accept (poster)