AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
Authors: Anil Kag
Huseyin Coskun
Jierun Chen
Junli Cao
Willi Menapace
Aliaksandr Siarohin
Sergey Tulyakov
Jian Ren
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation
Hybrid Architectures
Primary area: diffusion_based_models
TL;DR: We propose a hybrid architecture with asymmetric distribution of convolution and attention blocks in different network stages to achieve superior latency-vs-performance trade-off in image recognition and generation tasks.

Abstract: Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN---a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, without performing any optimization of inference time our model shows faster execution, even when compared to works that do such optimization, highlighting the advantages and the value of our approach.

AsCAN: Asymmetrie Convolution-AAtention Networks for Efficient Recognition and Generation
Anil Kag
Huseyin Coskun
Jierun Chen"
Junli Cao
Willi Menapace
Aliaksandr Siarohin Sergey Tulyakov Jian Ren
Snap Inc.
Project Page: http:///saaarrreeeee.ggggibbbonn jmaage
Figure 1: Example images generated by our efficient text-to-image generation model based on an asymmetric architecture. It generates photo-realistic images while following long prompts.
Abstract
Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN-a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective asymmetric architecture, where the distribution of convolutional and transformer blocks asymmetric, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety tasks: recognition, segmentation, class-conditional image generation, and features superior trade-off between performance and latency. We then scale the same architecture to solve large-scale extt-to-mmage task and show state-of-the-art performance compared to the most recent public
Work done during internship at Snap Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach.
Introduction
Convolutional neural networks (CNNs) and transformers have been deployed in wide spectrum of real-world applications, addressing various computer vision tasks, egg. image recognition [1. 2] and image generation [3, 4, 5, 6, 7, 8]. CNNs encode many desirable properties like translation equivariance facilitated through the convolutional operators [9]. However, they lack the input-adaptive weighting and the global receptive field capabilities offered by transformers [10, 11]. By recognizing the potential benefits of combining these complementary strengths, research endeavors explore hybrid architectures that integrate both convolutional and attention mechanisms [12, 13, 14, 15]. Recently, such architectures witness huge success when scaled up to train large-scale text-to-image (T2I) diffusion models [16, 17, 18, 19], enabling vast range of visual applications, such as content editing [20, 21, 22, 23, 24, 25, 26] and video generation [27, 28, 29, 30].
One prominent research area for hybrid models involves the creation of building blocks that can effectively combine convolution and attention operators [31, 32). While these efforts seek to use the strengths of both operators, their faster attention alternatives only approximate the global attention, leading to compromised model performance as lacking global receptive field. Thus, they neces- sitate incorporating additional layers compensate for the capacity reduction due to the attention approximation. On the other hand, minimal effort is directed toward optimizing the entire hybrid architecture. This raises the question: Is the current macro design of hybrid architecture optimal?
In this work, we propose a simple yet effective hybrid architecture, wherein the number of convolution and transformer blocks is asymmetric in different stages. Specifically, we adopt more convolutional blocks in the early stages, where the feature maps have relatively large spatial sizes, and more trans- former blocks at the later stages. This design is verified across different tasks. For example, in Fig. 3, we demonstrate superior advantages of our model for the latency- seencc-perfommnce trade-off on ImageNet- IK [33] classification task. Particularly, our model achieves even faster speed than many works fea- turing efficient attention operations. Additionally, we scale up our architecture to train the large-scale T2I diffusion model for high-fidelity generation (Fig. Fig. 4, and Tab. 3). Furthermore, considering the high training cost for the large-scale T2I diffusion models, we introduce a multi-stage training pipeline to improve the training efficiency. Overall, our contributions can be summarized as follows:
We revisit the macro design principles of hybrid connolutiooalttrnsfforoeo propose one with asymmetrically distributed convolutional and transformer blocks.
architectures and
For the image classification task, we perform extensive latency analysis on the ImageNet- IK dataset and show our models achieve superior parpuuuhuu-pefffrmmaae Fig. 3). Notably, we show that the model runtime can be significantly accelerated even without any acceleration optimization on attention operations, Additionally, we show our pre-trained models on ImageNet-1K can be applied to downstream tasks such as semantic segmentation.
trade-offs than existing works (see
For the class-conditional generation on ImageNet-IK IK (256 x 256), our asymmetric UNet achieves similar performance as ofate-of-the-art models with half the compute resources (see Tab. 4).
For the text-to- text-to-image generation task, we demonstrate that our network can be scaled up for the large-seat scale T2I generation with a better (as in Tab. 3). Additionally, we improve the training efficiency through multi-stage training pipeline, where we first train the model on a small dataset, i.e., ImageNet-IKe for T2I generation, then fine-tune the model on large-scale dataset.
iirfoommann--latenee trade-off than existing public models
2 Related Works
Efficient Hybrid Architectures. Over the past decade, convolutional neural networks (CNNs) [34, 35, 36) have achieved unprecedented performance in various computer vision tasks [8). Despite numerous attempts to improve CNNs [39, 40, 41, 42, 43], they still face limitations, particularly in terms of their local and restrictive receptive fields. On the other hand, the vision transformer (ViT)

---

treats images as sequences of tokens [44], facilitating the computation of global dependencies between tokens to enhance the receptive field. However, ViTs come with quadratic computation complexity concerning input resolution. To address this, several studies have aimed to develop more efficient attention operations [45, 46, 47, 48, 49]. Recently, there has been growing interest in exploring models that go beyond pure convolutional or transformer blocks, combining them within a single architecture to harness the spatial and translational priors from convolutions and the global receptive fields from the attention mechanism [11, 31, 2, 12, 32]. We revisit the design choices of hybrid architectures and propose new design with a fast runtime while maintaining high performance. We reveal that optimizing the macro architecture directly allows to the original attention for superior latency- pertormeefooree trade-off compared to existing works. Most importantly, our designed networks be applied to different domains, e.g., image recognition and image generation, with both superior seeecc-performance trade-offs.
Text-to-Image Diffusion Models. The development of text-to-image models, such as GAN- based models [50, 7I, autoregressive models [51, 52], and diffusion models [3, 53], has en- abled the generation of high-fidelity images by using textual descriptions. Among them, dif- fusion models demonstrate advantages in stable training processes and the scalability of large- scale neural networks [19, 54, 55, 16, 17, 13]. Recent studies explore the enhancements of text- to-image diffusion models through various directions, such as designing new network architec- tures [56, 57, 58, 59, 60, 61, 62, 63], improving inference efficiency during multi-steps sam- pling [64, 65, 66], and accelerating the training convergence [67, 68, 69, 70, 71j. Our designed network can be scaled up for the T2I generation when modified to UNet architecture [13]. The model achieves better performance-latee latency trade-offs than open-sourced models. Furthermore, we reduce the training cost through the proposed two-stage training pipeline.
3
Methods
This section describes our network and training designs in detail. First, we justify our choices of utilizing the convolutional and transformer operations for the building blocks, which we then use to design the asymmetric architecture (Sec. 3.1). Second, we scale up the network architecture for training the text-to-image diffusion models (Sec. 3.3). We point out that it is casier to ablate the design of building blocks on the image classification task due to the lower resource requirements (traning/evaluuation compute) in comparison to image generation.
3.1
Architecture Design: Asymmetrie Convolution-AAtentian Networks (AsCAN)
Our hybrid architecture consists of mix of convolutional and transformer blocks, which enables oper- ating on different input resolutions seamlessly and allows us to be pareto efficient w.r.t. performance- compute trade-off compared to other architectures. Before delving into the exact architecture configuration, we discuss our choice of the building blocks. We use X E R k xC to represent the input feature map X that has x W spatial dimensions along with channels. We denote
Y " xw' xc' as the output of building block (convolution or transformer) and symbol for the function composition operator. In the following, we present our design choices based on ImageNet- 1K [72] classification task by varying different convolution and transformer blocks.
Convolutional Block (C). There are various choices for convolutional block that can be used in our architecture, e-g., MBConv [73], FusedMBConv [1], and ConvNeXt [74]. While MBConv block has been used in many networks [75, 31), the presence of depthwise convolutions results low accelerator utilization for high-end GPUs [1]. To better understand the latency- pereec-perforrmncee trade-off for various convolutional blocks, we experiment with the same hybrid architecture yet with different convolutional blocks. As in Tab. 1 (more experimental settings in Sec. 4. 1),, FusedMBCony has better latency on A100 and V100 GPUs than others, while maintaining high performance. Therefore, we adopt FusedMBConv (C) for the convolutional block and represent it as Y X + P o SE o C(X), where C is a full 3 x 3 convolution with 4x channel expansion followed by batch norm and GeLU non-linearity [76], SE is squeeze-ande and-zexitedeexeee operator [39] with shrink ratio of 0.25, and P is 1 x convolution to project to C channels.
Transformer Block (T), Similar to the convolution blocks, there are many choices for attention mechanism transformers, e.g., blocks containing efficient attention mechanisms like multi-axial attention [31] and hierarchical attention [32]. Tab. 2 shows that vanilla attention mechanism provides
3

---

and Recognition
Image Generation
Norm + GeLU
CONV :3 x 3
POOL FC
CONV : x
C C
CCCT
CCTT
CTTY
Norn GaLL
Geld
20
CONY x3
SE
CONY
Llnor Prej
Time Step Emnbecking
Embeediigg
SEP R
BaFE i
CC
CCTT
CCTTTT
CTTTTC
CCTTTT
CCTT
CC
(b)
(C)
Figure 2: Example AsCAN architectures for Image Classification Text-to-Image Generation.
(a): The architecture for the image classification and details of the convolutional (C) and transformer
blocks (T). AsCAN includes Stem (consisting of convolutional layers) and four stages followed by pooling and classifier.
(b): The UNet architecture for the image generation. The Down blocks (the first three blocks starting from leff) have the reverted reflection as the Up blocks (the first three blocks starting from right). (c): The details for C and T used in UNet. For the T that performs the cross attention between latent image features and textural embedding, the Q matrix comes from the textural embedding. Note that, compared to image classification, the C and T blocks for image generation only adds extra components to incorporate the input time-step and textual embeddings.
better accuracy vs throughput trade-off across different GPUs and batch sizes. Thus, we choose the vanilla attention in our transformer block (T). We express it with the following update equations:
= X Yaun + mmipi Yaun
P o A Pokv (X); Ymip = P o & PML
(1)
where X = LW o @(X), C. denotes layer normalization, @ denotes the GeLU non- -ineaiiayy [76]. A is the multi-headed self-attention function, Poκv & PMLP denote the linear projection to the QKV and MLP space, respectively, and P denotes the projection operator to the same space the input. Note that these update equations are inspired by recent works [77, 78], where the feed-forward and the self-attention operators are arranged in parallel in order to get improved throughput with marginal reduction performance.
Design Choices. Given the FusedMBConv (C) and Vanilla Transformer (T) blocks, we introduce the macro design for our hybrid architecture. For the image classification task, we follow the existing works [11, 31, 32] to utilize four-stage architecture (excluding the convolutional Stem at the beginning and classifier components at the end). However, before finalizing our architecture, we still have a design question to answer, namely, in which configuration should we arrange these building blocks? For instance, CoAtNet [1 chooses to stack convolutional blocks in the first two stages and transformer blocks in the remaining stages while Max [31] stacks convolutional and transformer blocks alternatively throughout the entire network. A formal algorithm requires evaluating all possible C and T configurations, which is computationally expensive. Even neural architecture search leads to exponential search space. Thus, we follow naive strategy that based on the following principles:
C before T. In any stage, we prefer convolutions followed by transformer blocks to capture the global dependence between the features aggregated by the convolutions, as they can capture scale and Translation-awarr information. Our ablations in Appendix Tab. 9 justify this design choice. Fixed first stage. As transformer blocks have quadratic computation complexity in terms of the sequence length, we prefer the first stage to contain only convolutional blocks to improve the inference latency.
Equal blocks in remaining stages. For ease of analysis, we fix the number of blocks in the remaining stages, iee. stages 2 to 4, to be four. Once we finalize the basic configuration, we can scale these stages similar to earlier works [31] achieve larger models.
Asymmetric va Symmetric. We refer to the architectures as symmetric whenever and blocks are distributed equally within stage. For example, configuration of COCC-CCTT-TTTT is

---

symmetric since both C and T blocks are equal within a stage. In contrast, the configuration of CCCT-CCTT-CTV asymmetric since and blocks are not equal in stages and 4.
Final Architecture. Given these design principles, we list various promising configurations for the building blocks (C & T), and analyze their inference throughput and top-1 accuracy. Tab. 2 provides these configurations along with performance and runtime on different GPUs. For a better comparison with existing works, we also add the configurations of CoAtNet and Max ViT for reference. From the results, we can draw the following conclusions:
Compared to symmetric architecture design, asymmetric distribution of C & T blocks yield better trade-off for throughput and accuracy, as compared by configurations C1-C5 vs C6-C10.
Higher number of transformer blocks in the early stages result in lower throughput, which can be observed by comparing the latency for the configurations of C2 vs C10, and C8 vs C9.
While increasing the number of transformer blocks in the network improves the throughput, it does not result in improved accuracy, as demonstrated by C6 vs C9.
Given above analysis, we prefer the CI configuration for simplicity along with better accuracy vs latency trade-off. Since transformer blocks capture global dependencies, we prefer at least some blocks in the early layers as well in conjunction with the convolutional blocks. Similarly, we prefer having few convolutional blocks in the later stages to capture the spatial, translation, or scale aware features. To be concrete, our final architecture includes a convolutional stem followed by four stages and the classifier head. In the first stage, we only keep convolutional blocks. In the second stage, we keep 75% convolutional and 25% transformer blocks. This trend is reversed in the final stage. For the third stage, we keep equal number of convolutional and transformer blocks. We visualize this configuration in Fig. 2 along with the diagrams representing the convolutional and transformer blocks.
Remarks. For simplicity, in this work, we focus on the configuration which to combine the C & blocks, and leverage vanilla quadratic attention and convolutional mechanisms. We can incorporate faster alternatives to quadratic attention to further boost the performance and latency trade-offs. We leave this exploration to future research.
3.2 Discussion
While many works in the literature focus on improving the trade-off between performance and multiply-add operation counts (MACs), most of the time, MACs do not translate to throughput gains. is primarily due to the following reasons:
Excessive use of operations that do not contribute to MACs. Such tensor operators include reshape, permute, concatenate, stack, etc. While these operations do not increase MACs, they burden the accelerator with tensor rearrangement. The cost of such rearrangement grows with the size of the feature maps. Thus, whenever these operations occur frequently, the throughput gains drop significantly. For instance, Max ViT [31] uses axial attention that includes many permute operations for window/grid partitioning of the spatial features. Similarly, SMT [79] includes many concatenation and reshape operations in the SMT-Block. It reduces the throughput significantly even though their MACs are lower than AsCAN (see Appendix Tab. 7).
MACs do not account for non-linear accelerator behavior in batched inference. Another issue is that MACs do not account for the non-linear behavior of the GPU accelerators in the presence of larger batch sizes. For instance, with small batch sizes (B=l), the GPU accelerator is not fully utilized. Thus, the benchmark at this batch size is not enough. Instead, one should benchmark at consistency between architectures.
larger batch sizes
Lack of efficient CUDA operators for specialized building blocks. Many architectures propose specialized and complex attention or convolution building blocks. While these blocks offer better HAACs--s-performane trade-offs, it is likely that their implementation relies on naive CUDA constructs and does not result in significant throughput gains. For instance, Bi-Former [80] computes attention between top-k close regions using top-k sorting operation and performs many gather operations on the queries and keys. Similarly, RMT [81] computes the Manhattan distance between the tokens in the image. It includes two separate attention along the height and width of the image. This process invokes many small kernels along with reshape and permute operations. These specialized blocks would benefit from efficient CUDA kernels.
Using siceleratorrfrienlll operators. Depending on the hardware, some operators are better than others. Depth- Dest wise separable convolutions reduce the MACs, yet they may not be efficient for
5

---

Table 1:
Analysis of the Configura- tion of Convolution and Transformer Blocks. We analyze various convolutional and transformer blocks by training hybrid architectures with different options on the ImageNet-Ik IK dataset. We provide the in- ference latency (as throughput) for differ- ent GPUs Results show that FusedMB- Conv and Vanilla Transformer blocks pro- vide better trade-off over aceuracy and latency than others.
Table 2: Analysis of Architecture Configuration. We analyze the distribution of convolution and transformer blocks by training hybrid architecture with different distributions of blocks on ImageNet-1H KK dataset [72]. It demonstrates that the design in Fig. provides better trade-off over accuracy and latency. Symbol M denotes Max ViT block [31] composed of MBCCnn[73, 75] and Multi-Axial Attention blocks, which is equivalent to CT. Note that CoAtNet [11] uses MBConv blocks compared
to the FusedMBConv blocks in our design.
Convolation- Based
Block
Params
A100
B=64
(Imuges/s))
VI00
B=16
Top-1 Acc.
Family
Our 10-OCCT-OCTT-GE (CI)
Block Configuration
Params
SSM
Inferenceiinages/
B=168=64
3224 4295
A106
V100
B=16
1144
83.4%
Top-1
Acc.
------CCT-----0000 (C2)
73M
3237 4179
1056
83.2%
MBConv
29M
3013
914
83.12%
Arymmetrie:0: 1000-000--0000 TTTT
1224
ConyNext
35M
3923
1104
82.81
(C4)
5IM
3434
4411
1182
83.1%
FusedMBCoBv
35M
4295
1148
83.44%
02-0001
000-1-0001--000 00CT-00CT (C5)
95M
31335 4066
991
82.7%
02-0800-0000 0000-0000111110000 TVTT (Ch)
5IM
3783 4998
1280
82.8%
Trunsformer-Base
Block
Params
A100 | V10U
B=64
(Imagesk)
B=16
Top-1
Acc.
S) monetric 00-0000-
00-CIV7--0 CIU--177T- 1------77771 (C9)
11--C000-0011 -17T (C7)
-0--000000---111111 -177-
(CH)
42M
30M
34M
3536 4941
3316 4091
3475 5:11
1469
1296
1293
82.7%
82.64%
82.4%
Muiti-Axial
83M
3541
630
83.59%
03-0211-0C11--00
(C10)
Hicrarchical
74M
3470
552
83.51%
CaAINer-D 00-00D-TT7T--
25M
3837 5221
976
81 11.
Vanilla
55M
4295
1148
83.44%
CoAINet-1 oc-c:0cce-
42M
2221 2907
629
83.3%
ViX- NV-HNYNWWWW-WW
1058
particular hardware. Excessive use of depth-wise separable convolutions should be avoided in favor of the full convolutions wherever possible. For instance, MogaNet [82] extensively uses depth-wise convolutions with large kernel sizes and concatenation operations. These operators reduce the multiply-addition counts, which are not necessarily efficient on high-end GPU accelerators.
3.3 Scaling Up Architecture for Image Generation
We further scale up our architecture for the image generation task, which requires more computation than the image recognition task. We train our network by utilizing the denoising diffusion probabilistic models (DDPM) [16, 17, 54] in the latent space [13]. The latent diffusion model includes variational autoencoder (VAB) [83, 84] that encodes the image x into latent z and a diffusion model éØ(-) with parameters O. We utilize UNet [18] the network for the diffusion model. For the T2I generation, we get the text embedding with Flan-T5-XX encoder [85]. We train the diffusion model following the noise prediction [16, 17]:
(2) where / is the time step and T is the total number of steps, is the added Gaussian noise, and e is the condition signal. z, is a noisy latent obtained with a pre-defined variance schedule {, E (0, 1))e with the following updates [16]:
min E1- [I.T]. |e εo(2,,
2, = V Va a VI ãee; at = Be: 6.
II a.
(3)
In the following, we discuss our design for the diffusion model and the multi-stage training pipeline.
3.3.1 Asymmetric UNet Architecture
We follow the existing literature [13, 8, 15, 86] design the UNet architecture for the T2I diffusion model. Fig. 2 gives an overview of the overall architecture. It consists of three main stages, namely, Down blocks, Middle Block, and Up blocks. The Up blocks are the reverted reflection of the Down blocks. In addition, there are skip connections to add the features from Down blocks to Up blocks. We adopt the VAE from SDXL [8] to transform the image to the latent space and carry out the diffusion in this space. Since the text-to-imet tooiiaae generation task requires additional inputs (i.e. time and text embeddings), we modify the and T blocks to incorporate these conditions. Similar to the existing literature [8], we add the time embedding to the C blocks. Additionally, we add the text embedding to the T blocks through cross-attention operation carried in parallel to the self atte-attenton operation shown in Fig. 2(c).

---

While the UNet can take arbitrary resolution as input. we notice that adding positional embeddings in the self-attention operation reduces the artifacts such as duplicate patterns. For this purpose, we add RoPE [87] embeddings for encoding the position information in the T blocks. Further, we incorporate query-key normalization using the RMSNorm [88] for stable training in lower precision (bfloat16).
3.3.2 Improved Multi-Stage Training Pipeline
Instead of training the T2I network from scratch, we first train the model on a small-scale dataset and then fine-tune the model on a much larger dataset. It effectively reduces the training cost on the large-scale scale dataset (see ablation in Appendix Sec. A.6, Fig 8). This strategy differs from existing works that perform multi-stage training using different architectures. For instance, Pix-o [69, 71] trains a class conditional image generation network and modifies to fine-tune the network for text-to-image generation. Both our pre-training and fine-tuning tasks use the same architecture for text-to-image generation. We use the AdamW optimizer [89] with 31 = 0.9 and [32 = 0.99.
Specifically, in the first stage, we train our model using ImageNet- IK [33] for the -ixttto-image generation. Following Esser el al. [90], we form the conditioned text prompt as "a photo of <class name>", where class name is randomly chosen from the label of each image. The model is trained to generate an image with a resolution of 256 x 256. the second stage, we fine-tune the model from the first stage on a much larger dataset. Here we train the model in four phases: First, we conduct training at the resolution of 256 x 256 for 300K iterations with the batch size as 16, 384 and de - 4 as the learning rate. Second, we continue the training at the resolution of 512 x 512 for 200K iterations with the batch size as 6, 144 and le as the learning rate. Third, we train the model for 1024 x 1024 for 100K iterations with the batch size as 1, 536 and 5e 5 as the learning rate. Finally, we perform the multi-aspect ratio training such that the model can synthesize images at various resolution [8]. Additionally, we adjust the added noise at different resolution [15], i.e., BT in Eq. (3) is chosen as 0.01 for 256 x 256 and 0.02 for higher resolution (see ablations in Appendix Sec. A.6). We add an offset noise of 0.05 during multi-asped ratio training similar to earlier works[8].
4 Experiments
In this section, evaluate the architectures proposed in Sec. 3.1 on image recognition and generation tasks. We also apply this design to the semantic segmentation task in Appendix Sec. A.3. We highlight the important aspects of these experiments below and provide the experimental details in the appendix.
85.3
Hogansst-
85.3
ANCAN
ALENSE
Accuracy Imegettes 1K (%)
PavATo
Top-1 Accuracy on ImageNitt 1K (%%
Hogelen
Hege
84
24
Dicestann
PNTV
eon Throughput (Images/second) Nvidia V100
500
1700
600
12n0
1800
2400
Throughput (images/sacona) Mvidia A100
Figure 3: Top-1 Accuracy Inference Latency on ImageNet-11 1K Classification. We plot the latency measured as images inferred per second on a single V100 GPU (Left)/A 100 GPU (Right) with batch-size 16 with 224 x 224 resolution. The plot compares oftte-of-hheartt art models (convolutional, transformer, hybrid architectures) against the proposed ASCAN architecture. The area of each circle proportional the model size. Our model consistently achieves better accuracy vs latency trade- offs. While some models regress between two hardware (e.g.. MaxViT-S vs SMT-B ), our model consistently achieves better accuracy latency trade-offs. We report additional baselines along with multiply-add operations count and different batch sizes in Appendix Tab. 7.
7

---

4.1 Image Recognition
We scale the AsCAN architecture discussed in Sec. 3.1 to base and large variants following earlier works [31]. We train these variants on the (mageNet-IK[72] classification task. We provide the experimental details (dataset description, architectures, training syper-parameters) in Appendix Sec. A.2. Fig. 3 plots the inference speed on a V100 GPU with batch size 16 (measured in images processed per second) and top-1 accuracy achieved on this task for various models. In addition, Appendix Tab. 7 shows the parameter count and inference speed on both V100 and A 100 GPUs along with additional details such as floating-pont multiply-ade count (FLOPs/MACC) and inference speed across different batch sizes. Below, we highlight the salient observations from these experiments.
More than 2x higher throughput across accelerators. Compared the existing hybrid architectures such as FasterViT [32] and Max ViT [31], the proposed AsCAN family achieves similar or better top-1 accuracy with more than 2x higher throughput. This trend holds true for both A A100 and V100 GPUs. For instance. on A100 with batch=16, FasterViT-L achieves 83.2% top-1 accuracy with throughput as 1123 imagesls, while AsCAN-T 83.44% with throughput as 3224 images/s. Better throughput across different batch sizes. AsCAN consistently achieves better throughput across batch sizes for both the accelerators compared baselines.
Better storage and computational footprint. AsCAN- famiily of architectures requires less number of parameters and float operations to achieve similar performance. For example, to achieve nearly 85.2% accuracy, Max VIT-L requires 212M parameters and 43.9G MACs whereas AsCAN-L requires 173M parameters and 30.7G MACs.
We achieve better latency accuracy trade-off than hybrid architectures with better alternatives to the attention mechanisms in transformer. For instance, we outperform newer architectures such as PVTv2[91], MOAT[92], EfficientViT[93]] and Scale-Aware Modulation Transformers(79).
Further, we observe similar trends when we scale these models to pre-training on ImageNet-221 [94] dataset (see Appendix Sec. A.2.3) as well as semantic segmentation task (Appendix A.3).
Table 3: GenEval Scores. We use this benchmark to compare T2I models on various aspects of generation, including counting, color attribution, position, etc. It clearly shows that our model achieves better overall score, by convincingly outperforming pipelines such as PixArt-a and SDXL.
Single Object
Two
Color
Model
Overall
Counting
Colors
Position
Objects
Attribution
SDv1.5
0.43
0.97
0.38
0.38
0.76
0.04
0.06
PixArt-a
0.48
0.98
0.50
0,44
0.80
0.08
0.07
SDv2.1
0.50
0.98
0.51
0.44
0.85
0.07
0.17
PixArt-E
0.53
0.99
0.65
0,46
0.82
0.12
0.12
SDXL
0.55
0.98
0.74
0.39
0.85
0.15
0.23
Ours
0.64
0.99
0.78
0.43
0.88
0.28
0.48
4.2
Class Conditional Generation
We apply our asymmetric UNet architecture to learn class conditional image generation on the ImageNet-IK dataset with 256 x 256 resolution. We train a smaller variant of our asymmetric UNet architecture (as in Sec. 3.3.1) with nearly 400M parameters and inject the class condition through cross-attention mechanism. We train this model using DDPM [16] (more details in Appendix Sec. A.7), and provide results in Tab. 4. As can be seen, we achieve Fréchet Inception Distance (FID) [95] close to otate-of-the-art models with nearly half the FLOPs (e.g., Ours vs. D/T-XL/2-G), and better FID than the existing work with similar computation (e.g., Ours U-VIT-L/2).
4.3 Text-to-Image Generation
Below, we evaluate T2I asymmetric UNet architecture trained using the multi-stage training.
GenEval Benchmark. We evaluate our model on the GenEval [97] benchmark that studies various aspects of an image generation model. Tab. 3 shows that our model outperforms fast training pipelines such as Pix-rt- by a significant margin. It beats even larger models such as SDXL. which consume significantly more training data and computational resources.

---

Ours
SDXL w/ refiner
PiAArt-
PhAAT-E
Firefly 3
Dalle3
Imagen2
Midjourney
super cute extremely fivffy lictle dog
dog
person light black floor made tikes, blurred with white
dog's left belge
very realistic, detailed, cat with big hat and white sunglasses, posing, typernalistic, atmospheric, In city, street, new york, oinematk, dramatic lighting, photorealiatic, MIOR de 35mm lens, zife Blur, Shutter Spend 1060, F S.G. Super Resolution
young woman standing outside alightly turned to her left. Her head alighely turned to her right, her bont the efbow and her arms are at stomach lever, She has blach helr, dart brown eyes, red lipstich, face paint with black and white squiggles on her cheeks, and facial facial expression with a slight smile, she wears an elaborate headdress made of foathers with black, orange, and white decorations, and golden eariigs. She wears black T-shirs underneath shoulder piece with elabarate feathers and decorations that are orange and shows grey ground
Black. background
A woman is standing straight near the shore of late with her back facing towards the camera, She has shoulder-fongth blando heir. She is drassed in a jechejechet with grey scart around her neck, the bachground, the lake has reflections of trees and the sky, brown trees on sky covered with gray rainy during daytime
sides
In the daytime, a young man standing slightly turned his right, while holdinng a gray rabblc behind his left showlder. His right arm bent at the albow and resting on his shoulder, while left arm down his heald turmed the right. He stubblte mustachs, a beard, gray syex blue-Brown patterns, brown jacket,
happy expression snowy forest snow-capped
He wearing grayy knitted background.
Figure 4: Qualitative Comparison against open source and commercial models. We compare our T21 model against generations from different baselines. We illustrate that many times existing models generate images with less photo-realism (either lot less details specially for PixArt-a and Pr. Further, they frequently miss the fine-grained details explicitly asked in the prompts. We highlight these mistakes in red color in the imput prompt. For instance, in the above generations (ordered A + F from top to bottom row), baselines miss details such as. (A) lack of realism (B) light blue jeans, (C) white sunglasses, (D) black, orange, and white feathers, (E) grey scarf & back towards camera, and (F) gray knitted hat with dark blue-brown patterns.
on the cartoonish side),
Qualitative Comparison. Fig. 4 shows qualitative comparison between different methods. We highlight salient differences between the baselines and our generations. Baselines such PixArt-a and PixArt-E tend to generate images with much less photo-realism and more often is much more on the cartoonish side or it has grainy artifacts. Similarly, SDXL with refiner framework is unable to adapt to long text prompts due to limitations of the CLIP text-encoder. Hence, misses many key features described in the prompts. In contrast, our model is able to follow the long prompts while adhering to the required semantics as well as photo-realism.
Human Preference study. We perform a user study to compare open-sourced models to evaluate their image-text alignment characteristics. We select 1000 prompts from our validation set and generate images from SDXL, PixArt-a, E, and our multi-aspect ratio model. We show these results in the Fig. It shows that our model convincingly outperforms both SDXL and PixArt-a generations, while it has similar image-text alignment as Prx model. To further evaluate these models, we generate 10K images from our validation set (described in Appendix A.5.2) and compute the FID [95] between the generated and original images. We show these scores in Tab. 5, which

---

Table 4: ImageNet-1K Class Conditional Generation. We train smaller variant of our T21 UNet architecture to perform class conditional generation on ImageNet- We train this model at 256 x 256 and 512 x 512. Our asymmetric architecture achieves similar FID as the state-of-the-ee models with less than half the floating point operations (e.g., Ours vs. DiT-XL/2-G), and better FID than the existing work with similar computation (e.g., Ours vs. U-ViT-1./2).
256 256
512 x 512
Throughput
Throughput A100 (B=64) samples/sec
Model
FLOPs
A100 (B=64)
FID
FLOPs
FID
samples/sce
ADM [18]
110G
10.60
LDM [13]
104G
3.60
U-ViT-L/2 [86]
77G
498
3,40
340G
86
4.67
U-ViT-H/2 [86]
133G
271
2.29
546G
45
4.05
DIT-XL/2-O0 [4]
118G
293
2.27
525G
51
3.04
Ours
556
2.41
224G
130
3,28
Ours (sampled cfg [96])
52G
556
2.23
224G
130
3.15
Table 5: Validation Data Evaluation.
(%)
23.2
31.2
60
We the validation of captioned data for computing the FID scores for comparison between different models (see details in Appendix A.5.2)
16
Oura val RDXL.
Oun PixzAri-
Ouri PinAri-3
Image-Text Alignment
FID
FID
Figure 5: Image-Texx Alignment Study. We perform user study for 1000 prompts and ask them to choose images with better image-text alignment. It shows that we out- perform SDXL and PixArt-a. While our performance is on par with PixAnt-, Tab. shows that we yield more realistic generations.
Model
Sct-B-10K
Sct-A-10K
PixArt-
46.03
20.12
PixArt-
40.01
19.25
SDXL
35.86
16.49
Ours
15.45
10.88
clearly indicates that our generations align well with the real-world images. This also evident from our carlier comparison on qualitative visualization in Fig. 4.
Resource Efficiency. We compare the resource requirements of our model against various baselines in Appendix Tab. 12. It shows that we achieve better inference latency compared to many existing models. Further, consume considerably less compute achieve much better performance than many existing baselines as illustrated by evaluations in the previous section.
Conclusion
In this work, we design hybrid architectures comprising convolutional and transformer blocks with applications many computer vision tasks. We focus on developing a simple hybrid model with better throughput and performance trade-offs. Instead of designing efficient alternatives to the convolutional and transformer (mainly attention mechanism) blocks, leverage existing vanilla attention along with the FusedMBConv block to design the new architecture, called AsCAN, Our main philosophy revolves around the uneven distribution the convolutional and transformer blocks asymmetric, in the sense that
in the different stages of the network. We refer to this distribution favors more convolutional blocks the early stages with mix few transformer blocks, while it reverses this trend favoring more transformer blocks in the later stages with fewer convolutional blocks. We demonstrate superiority of the proposed architecture through extensive evaluations across the image recognition task, class conditional generation, and sott-to-image generation.
10

---

References
[1] Mingxing Tan and Quae I.c. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pages 10096-10106. PMLR, 2021.
[2] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35:12934-12949 2022.
[3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances Neural Information Processing Systems, 0996449936494 2022.
[4] William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[5] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Texi-to-image diffusion model on mobile devices within two seconds. arXiv preprint arXiv:2306. 00980, 2023.
[6] Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realistic human generation with latent structural diffusion. arXiv preprint 10X0522310.08579, 2023.
[7] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[8] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.011 01952,, 2023.
[9] Risi Kondor and Shubhendu Trivedi. On the generalization equivariance and convolution in neural networks to the action of compact groups In International Conference on Machine Learning, pages 2747-2755, PMLR, 2018
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, In International Conference on Learning Representations, 2021.
[11] Zihang Dai, Hanxiao Liu, Qune V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
[12) Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sorgey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. In Proceedings of IEFE/CVF International Conference on Computer Vision, pages 16889-16900, 2023.
[13] Robin Rombach. Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[14] Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R Manmatha, Ashwin Swaminathan, Zhuowen Tu, Stefano Ermon, and Stefano Soatto. On the scalability ceattusion-basse text-to-image generation. arXiv preprint arXiv:2404.02883 2024.
[15] Emiel Hongehoom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. International Conference on Machine Learning, pages 13213-13232. PMLR, 2023.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbcel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6833 2020.
[17] Yang Song, Jascha Sohl-Dickse Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

---

[18] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[20] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:21 10741, 2021.
[21] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lec, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[22] Aditya Ramesh, Prafulla Dhariwal,, Alex Nichol. Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 0622 2022.
[23] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-iman -mage generation via masked generative transformers. arXiv preprint arXiv:2301.0077 2023.
[24] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song. Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.00 1.01324. 2022.
[25] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6027-6037, 2023.
[26] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311.1008, 2023.
[27] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 10.00233, 2022.
[28] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:23/2.06662 2023.
[29] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samanch Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint 1xXx:23311110009 2023.
[30] Willi Menapace, Aliaksandr Siarohin. Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag. Yuwei Fang, Alcksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-vidco synthesis. arXiv preprint 2rXiv:2402. 14797, 2024.
[31] Zhengzhong Tu, Hossein Talebi. Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transforer.. ECCV, 2022.
[32] Ali Hatamizadeh, Greg Heinrich, Hongxu Yin. Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189 2023.
[33] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248 255. Iece, 2009.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing syxtems, 25, 2012.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
12

---

[37] Licheng Jiao, Fan Zhang. Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu. A survey of deep bearning-based object detection. IEEE access, 7: 7:128837-128868. 2019.
[38] Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEFE transactions on pattern analysis and machine intelligence, 44(7):3523-3542, 2021.
[39] Jie Hu, Li Shen, and Gang Sun. guueeze-and-excitttii networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132 22417,1,,210 2018.
[40] Anil Kag and Venkatesh Saligrama. Condensing cnns with partial differential equations. In Proceedings of the JERF/CVF Conference on Computer Vision and Pottern Recognition, pages 610-619, 2022.
[41] Yifan Sun, Linan Zhang, and Hayden Schaeffer. Neupde: Neural network based ordinary and partial diffcrential equations for modeling dimendepnnentt data. In Mathematical and Scientific Machine Learning, pages 3723 PMLR, 2020.
[42] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[43] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in enns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11963 11975, 2022.
[44] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999, 2020.
[45] Sinong Wang. Belinda Z Li. Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[46] Ze Liu, Yutong Lin. Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[47] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbome with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12124- 12134, 2022.
[48] Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tz- imiropoulos, and Brais Martinez. Edgevits: Competing light weight transformers. In European Conference on Computer Vision, pages 294-311. Springer, 2022.
mobile devices with vision
[49] Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning, pages 12633-12646. PMLR, 2023.
[50] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power gans for fast large-scale text-to-image synthesis. arXiv preprint (.0XXiv23010095550 2023.
[51] Jiahui Yu, Yuanzhong Xu, Jing Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[52] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference Computer Vision, pages 89-106. Springer, 2022.
[53] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 02233099.1007..7 15807, 2023.
[54] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
13

---

[56] Susung Hong. Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 7462-7471, 2023.
[57] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaidl. Matryoshka diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[58] Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Mamuel Perez-Rua. Gentrom: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:2312.04557, 2023.
[59] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[60] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696, 2023.
[61] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. arXiv preprint arXiv:2312.0216 2023.
[62] Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image diffusion model with kmowledge-enhaneed
mixture-of-denoisee denoising--xpertss In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10135-10145, 2023.
[63] Zeyue Xue, Guanglu Song. Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
[64] Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. arXiv preprint w3x.::3332000055 2023.
[65] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2024.
[66] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian Hc, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprini urXiv:23/2.03209, 2023.
[67] Ryan Webster, Julien Rabin, Loie Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.1 12733, 2023.
[68] Aaron Gokaslan, A Feder Cooper, Jasmine Collins, Landan Seguin, Austin Jacobson. Mihir Patel, Jonathan Frankle, Cory Stephenson, and Volodymyr Kuleshov. Commoncanvas: An open diffusion model trained with creative-commons images. arXiv preprint arXiv:2310. 2023.
[69] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
[70] Pablo Pernias, Dominic Rampas. Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffuusio models. In The Twelfih International Conference on Learning Representations, 2023.
[71] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- siigaa: to-k--t-sttoon training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403. 04662, 2024.
[72] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11::211 1-222, 2015.
[73] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
[74] Zhuang Liu, Hanzi Mao, Chao-Yuse Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976 1986, June 2022.
14

---

[75] Andrew Howard. Mark Sandler, Grace Chu. Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[76] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.0880 1606.08415,
2016.
[77] Mostafa Dehghami, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480- 7512. PMLR, 2023.
[78] Hugo Touvron, Matthieu Cord, Alazeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers, 2022.
[79] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scalc-aware modulation meet transformer, 2023.
[80] Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-le routing attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[81] Qihang Fan, Huaibo Huang. Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision transformers. In CVPR, 2024.
[82] Siyuan Li, Zedong Wang. Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. Moganet: Multi-order gated aggregation network. In International Conference on Learning Representations, 2024.
[83] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.1111,, 2013.
[84] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[85] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling iestructiio-finetunn language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
[86] Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22669-22679, 2023.
[87] Jianlin Su, Murtadha Ahmed, Yu Lu. Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocopputing, 568:127063, 2024.
[88] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[89] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[90] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03200 2024.
[91] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu,, Ping Luo, and I Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(34:4154424, March 2022.
[92] Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Adam, and Liang-Chid Chich Chen. MOAT: Alternating mobile convolution and attention brings strong vision models. In The Eleventh International Conference on Learning Representations, 2023.
[93] Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. arXiv preprint 1457v:200..4756, 2022.
15

---

[94] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the 2021.
[95] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances neural information processing systems, 30, 2017.
[96] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
[97] Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating lext-to-image alignment, 2023.
[98] Ekin Dogus Cubuk, Barret Zoph, Jon Shiens, and Quoc Le. Randaugment: Practical automated data augmentation reduced search space. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18613-18624. Curran Associates, Inc., 2020.
[99] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[100] Ross
Wightman.
Pytorch image models.
https://gitthub.com.
pytorch- pytorch-image models, 2019.
[101] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059-1077 PMLR, 2021.
[102] Donghyun Kim, Byeongho Heo, and Dongyoon Han. Densenets reloaded: Paradigm shift beyond resnets and vits, 2024.
[103] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[104] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-512 5130, 2017.
[105] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://gitah .tt.tuu.ocoooooooooooooooooo.oo. open www.aaaaeemmggtttaaaiamae 2020.
[106] Tete Xiao, Yingcheng Liu. Bolei Zhou, Yuning Jiang, understanding. In European Conference on Computer Vision. Springer, 2018.
Sun. Unified perceptual parsing scene
[107] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 23-29 Jul 2023.
16

---

A Appendix
A.1 Image Recognition Architecture Details
We describe the three AsCAN variants (tiny, base, and large) used in the experiments (see Sec. 4.1) in the Tab. 6. We follow earlier works [31] to scale the tiny variant discussed in Sec. 3.1. Note that the stem consists of two convolutional layers that downsample the input to half the spatial resolution. The classifier stage projects the final feature map to the corresponding embedding size and performs adaptive average pooling to reduce the spatial dimensions to x 1, in order to apply a feed-forward layer that acts as classifier head on top of these pooled features. Our convolution blocks use the natch-norm&oe as the normalization layer while the transformer blocks leverage the layer-norm as the normalization layer. We also learn the relative positional embeddings in our attention mechanism similar to earlier works [31. 32, 46].
Table 6: Architecture Details. We provide detailed configuration of the three variants of the AsCAN proposed in this work, namely. tiny, base, and large. As per our method section, we use FusedMBBConv (C) as the convolution block and Vanilla Transformer (T) as the transformer block. We refer the reader to the Fig. 2 for their detailed description and schematic of these building blocks. We use K to denote the number of classes. Note that we hide the activation and normalization layers in the blocks. We use GeLU activation and batch-normalizztion in the Stem and the classifier stages.
Stage
Tiny
Base
Large
Conv 3 Channels 64, Stride 2
Conv 3, Channels 64, Stride 2
Conv 3 3, Chunnels 128, Stride
S0: Stem
Conv 3 x 3, Channels 64, Stride
Conv 3 3, Channels 64, Stride
Conv 3 x 3, Channels 128, Stride
C. Channels 96, Stride 2
C, Channels 96, Sirie 2
C, Chunnels 128, Strie 2
$1: Only- Only-Coav
C. Channels Stride
C, Channels 96, Stride
Channels 128, Stride
C, Channels 192, Stride 2
(C. Channels 192, Stride 2) x1
(C. Channels 256, Stride 2) x1
C. Channels 192, Stride 1
S2: Mix
(C, Channels 192, Stride 1) x3
CC. Channels 256, Stride 1) x3
C. Channels 192, Stride
(T. Channels 192, Stride 1) x2
(T. Channels 256, Stride 1) x2
T. Chunnels 192, Stride
C. Channels 384, Stride 2
(C, Channels 384, Stride 2) x1
(C. Channels 512, Stride 2) x
C. Channels 384, Stride
S3: Mix
(C, Channels 384, Stride 1) x6
(C. Channels $12, Stride 1) x6
T. Chunnels 384, Stride
(T. Channels 384, Stride 1) x
(T. Channels 512, Stride 1) x
T. Channels 384, Stride 1
C Channels 768, Stride 2
C, Channels 768, Siride 2
C, Channels 1024, Stride 2
T. Channels 768, Stride
Channels 768, Stride
Channels 1024, Stride
S4: Mix
T. Channels 768, Stride
Channels 768. Stride
T. Channels 1024, Stride I
T. Chunnels 768, Stride 1
T. Channels 768, Stride
T. Channels 1024, Stride I
Conv x Channels
Cony 1, Channels 768
Conv 1, Channels 1024
85: Classifier
Adaptive Avg Pool Feed-Forward 512x K
Adaptive Avg Pool
Adaptive Avg Pool Ferd-Forward 1024x K
Feed-Forward 768x K
A.2
ImageNet Classification (Training Procedure & Hyper-parameters)
We follow the same training strategy for our ImageNet experiments in Sec. 3.1 and Sec. 4.1. We report the top-1 1 accuracy on the single center crop image. For training ImageNet-IK models with 224 x 224 resolution, we use the AdamW optimizer with a peak learning rate of 3e 3 for 300 epochs. We use batch size of 4096 images during this training period. We follow cosine schedule for decaying the learning rate to the minimum learning rate of 5e 6. We also perform a learning rate warm-up avoid instabilities during the training. We follow 20 epoch warm-up schedule with an initial learning rate of 5e- - 7 that gets warmed up to the peak learning rate.
For all our experiments, we use standard data augmentation strategies. We use with parameters (2, 15), MixUp [99] with a = 0.8, color jittering with 0.4 as the weight, and label smoothing with 0.1 as the smoothing parameter. We also use 0.05 value for the weight decay regularization and perform an exponential model averaging with a decay value 0.9999. we adopt gradient clipping with gradient norm of 1.0 to avoid instabilities during the training of such large models. We also enable stochastic depth for regularization. We use the stochastic depth of 0.3/0.4/0. /0.5 for the three variants in our experiments.
[98]
addition,
A.2.1 ImageNet-1K
We benchmark various existing architectures on two accelerators (NVIDIA 1000 and V100 GPUs) with different batch sizes to evaluate the inference speed of single forward pass. Additionally, we
17

---

train the three AsCANvariants with the training procedure described in earlier sections. We show the full results corresponding to the Fig. 3 in the Tab. 7. We also include inference memory consumed by various recent works in Tab. 8.
A.2.2 Ablations on Architecture Configuration
One of the AsCAN design principles include prefemiig C blocks over T blocks in stage. We demon- strated various architecture configurations using this design choice in Tab. 2. For completeness, we reverse this design choice and prefer blocks over blocks in a stage. We show these configurations in Tab. 9. Using T block in stem carly layers result in lower throughput. Further, the performance of configurations with T before yields lower accuracy vs throughput trade-off.
A.2.3 ImageNet-21K
We also study the effects of pre-training the ASCAN family on larger datasets such as ImageNet- 21K [72]. Similar to earlier works, we pre-train our models the ImageNet-21K dataset. We use the pre- pre-processed version of this dataset for ease of usage [94]. Following previous works [11, 311, we train the AsCAN model for 90 epochs on the ImageNet-21K dataset and fine-tune these weights on the ImageNet- IK classification task. Similar to the ImageNet- IK experiments, we use RandAugment [98] with parameters (2.5), MixUp [99] with a = 0.2, color jittering with 0.4 as the weight and label smoothing with 0.01 as the smoothing parameter. We also use 0.01 value for the weight decay regularization. Additionally, we perform an exponential model averaging with a decay value of 0.9999 and gradient clipping with a gradient norm of 1.0 to avoid instabilities during training such large models. We also enable stochastic depth for regularization. We use the stochastic depth of 0.4/0.5/0 for the three variants in our experiments.
We report the performance in Tab. 10, where we observe similar top-1 accuracy as the other baselines while achieving much better inference throughput. This trend is similar to the one we observed in Sec. 4.1 when these models trained only on the ImageNet-IK dataset without any pre-training on the ImageNet-21K K dataset.
A.3 ADE20K Semantic Segmentation
Dataset Details. ADE20K [104] is popular scene-parsing dataset used to evaluate the semantic
segmentation performance. It consists of 20K train and 2K validation images over 1 150 fine- fine-grained semantic categories. Images resized and cropped to 512 x 512 resolution for training.
Training Procedure & Hyper-parameters. We base our semantic segmentation experiments on the widely used and publicly available mmsegmentation library [105]. We use the UPerNet [106] as our semantic segmentation architecture wherein different hybrid architectures are used as the backbones to extract the spatial feature maps. We extract the spatial feature maps at stages S2, S3, and S4, and forward these feature maps to the semantic segmentation network. All three AsCAN variants (tiny, base, large) have been initialized with the weights pre- mre-trained the ImageNet- IK task with 224 x 224 resolution. Training is performed 512 x 512 resolution. Similar to Faster ViT [32], we follow similar schedule for training these segmentation models with an Adam W optimizer with a learning rate of le - 4, and a weight decay of 0.05. We use batch size of 16 on A100 GPUs. We also use stochastic depth similar to ImageNet training for controlling overfitting during these experiments.
Experimental Setup. We train UperNet [106) as segmentation architecture on the ADE20K dataset. We use the proposed AsCAN as the backbone pre-trained on the ImageNet-1K dataset. We use the AdamW optimizer [89] with learning rate 1e - 4 and weight decay 0.05. Following earlier works [32, 74], we train the models for 160K iterations using the mmsegmentation library [105]. We train these networks on 8 NVIDIA A000 GPUs using 16 as the batch size. We compute the various inference statistics with 512 x 512 as the image resolution.
Experimental Results. We compare the performance of various backbones in Tab. 11. It shows that the proposed backbone achieves competitive mIoU while achieving nearly 1.5x faster inference latency compared to the baselines, measured using the frames per second metrics. This fact can be observed across different scaling of the backbones. For instance, the Swin-T backbone achieves latency of 44FPS while AsCAN-T achieves 64FPS as the latency with similar mloU. Thus, our
18

---

models achieve a similar performance-vs- -lttency trend at the downstream semantic segmentation task the classification benchmark.
A.4 T21 Dataset Details
We curate training dataset with careful filtration diverse data sources using first-party and licensed datasets. We depict this entire process in Fig. 6. We start by collating all these data sources to create stream of images and if available, relevant captions. Next stage, add series of filters to improve the quality of the selected images (resolution, de-duplication, aesthetic score, nsfw filtering, etc.). Since this data mostly contains very short descriptions almost non-existent descriptions of the image, we unify the captioning process using an image captioning model. We collect subset of 200K data for human annotations, where we ask the annotators add details such as the <angle shot of the image>, <image background information <human attributes>, etc. We use this labeled data fine-tune BLIP2 [107] model which is used generate long captions similar to this format.
Image
Video
Data Sources
Video Frame Extraction
Resoltationand and Asgecc Fittwing
Phush Deduplng
NSPW Score Fittering
Image Filtering
Entropy Score Fittering
Caption Text
Fitering
Watermark Score Filtering
Indecency
Viclence
Vulgar
Aniinationn
Image Annotation
Object
Captioning
Images
Final Dataset
Image Caption Pairs
Dataset
Figure 6: Curation Pipeline for the T2I Dataset. We show various stages involved in the creation of our T2I dataset.
19

---

A.5
T2I Evaluation Details.
A.5.1
Resource Usage
We compare the resource usage of various existing methods with our proposed T21 model. Tab. 12 shows the parameter count of the model, training compute cost, and the inference cost per image generation.
A.5.2 Evaluation on Internal Data
We evaluate our models against various baselines on two internally captioned datasets (Set-A-10K and Set-B-1OK). We draw these datasets from our validation set. We compute the FID between the generated 10K images against the reference images in Tab. 5. This table also shows that fast training pipelines like PixArt-a and PixArt-E suffer considerably when evaluated on real-world images compared to models like SDXL and Ours. We also point out that SDXL has been trained with a considerably larger dataset compared to ours and uses much more computational resources.
A.6 Hyper-parameter Setup for Text-to-Iman Image Generation
Since full-scale text-to-image generation task is computationally expensive task, it becomes crucial that we choose appropriate pyper-parameters and initialization of the model before training it on the entire dataset. Thus, choose a proxy task that be run a small scale and guide in the suitable training configurations.
ImageNet-1K T2I Generation Task. We resort to the ImageNet- KK dataset since it contains variety of objects (1000 objects) in different settings. Earlier works such as Pixart-o, train their transformer architecture on the ImageNet- K class-condtional generation tasks, wherein they incorporate the text condition via a cross-and cross-attention mechanism. This process requires the model forget the class condition and adapt to the text condition. In this work, along the lines of [90]. we simply convert the ImageNet-IK classification dataset to Text-to-Image dataset by adding the text corresponding to the <class name>, with the template "a photo of <class name>". Specifically, the dataset provides various different names corresponding to each class, we pick up one at random during training to learn diverse text mappings. We evaluate the models by computing the FID between the 50K generated images and the reference images from the dataset.
Embedding Pre-Comp Pre-Computation. Our T2I model performs diffusion in the latent space. To keep parity between this task and the full training, we also pre-compute the SDXL VAE embeddings of the ImageNet- dataset. Similarly, we pre-compute Flan-TS, Maa-T5-XXX -XLL embeddings for the text condition. This saves training resources since loading VAE and Flan-T5-XXL encoders consumes significant GPU memory as well as non-trivial computation time. Thus, by pre-computing these embeddings, we increase the batch size available for training the UNet model.
Architecture Details. We use similar configuration for the C blocks as in the image recognition architecture. All our convolutional operators use x 3 kernel sizes. The number of output channels in the three Down blocks are {320, 640, 1280). Since the three Up blocks are reflections of the Down blocks, their output channels are in the reverse order, i.e., {1280, 640, 320). Given that we use the SDXL VAE to convert the 3- 3-channel RGB- RGB-image into 4- channel latent space, our number of input channels for the convolution operator in the first Down block. The number of attention heads in the three T blocks in the three Down stages are (5. 10, 20). Since our T blocks are applied after C blocks, the number of output channels of the convolutional blocks along with the number of attention heads, determine the attention dimension for the transformer block. Finally, we compute the time-step and textual embeddings. The time-step integer values are converted to timestep embedding by first converting it into sinusoidal space and then projecting through two linear layers. We process the textual embeddings using two T blocks for adapting the frozen embeddings (dimension 4096) for T2I generation with just one head and the same dimension as the frozen embeddings.
Noise Level Ablation for Different Resolutions. Since our training strategy involves jumping resolutions from 256 + 512 + 1024, we ablate on the noise levels at each of these resolutions. We use the DDPM noise scheduler for injecting the noise and the level of noise is controlled by the 3r hyper-parameter. We try four different noise levels (0.01. 0.02, 0.03, 0.04). and compute the FID scores. We plot these scores in Fig. 7. We find that at resolution 256 noise level 0.01 produces the best FID score, and at remaining resolutions, noise level 0.02 works the best.
20

---

16
roise-01
roise-33 rotse
14
note 06
14
12
FID
60k
80k
100k
120k
140k
30k 40k 50k
60k
70k
80k 90k 100k 110k
Iterations
Iterations
Figure 7: Noise Level Selection for Different Resolution. We ablate on the end noise level during the
for the ImageNet-II Ik T2I task for the 256 x 256 (Leff) and 512 x 512 (Right) resolutions with different learning rate and weight decay parameters. We plot the FID scores against reference image set from the dataset.
T2I Task Initialization Ablation. We analyze initializing the 256 x 256 resolution training with and without pre-training from the ImageNet- K T2I pre-trained model. As shown in Fig. 8, pre-training helps improve the performance of the model compared to training from scratch.
50
Without ImageNet Pre-training
45
With ImageNet Pre-training
40
35
25
20k
25k
30k
35k
Iterations
Figure 8: Comparing T2I 256 256 Resolution Training with and w/o Pre-training, We report the FID score Set-B-10K for 256 256 resolution training, with and without pre-training from the ImageNet-1k T21 model.
A.7 ImageNet-1K 1K Class Conditional Generation.
We learn a class conditional image generation on the ImageNet-Il KK dataset with image resolution 256 256. We train smaller variant of asymmetric UNet architecture (as in Sec. 3.3.1) with nearly 400M parameters and inject the class condition through the cross-attention mechanism. We train this model using the DDPM scaled linear noise schedule for 1000 time steps. The training is conducted for 1000 epochs with a batch size of 2048. We use AdamW optimizer with 6e - as the learning rate, 0.01 the weight decay, and (31, 52) (0.9, 0.99). We use an 8-chanel VAE to convert the input image space into a latent space for latent diffusion. We compare our results with other class conditional models Tab. Our asymmetric architecture for this task is similar the T2I UNet architecture described earlier. To lower down the compute footprint, we reduce the number of channels {160, 320, 640) and reduce the cros-attentiio dimension from 4096 768. We evaluate our model with two configurations and report these two FID scores. In the first configuration, we apply a classifier-free guidance (cfg) scale [96] of 1.78 to generate the class-conditioned images. In the second configuration (sampled cfg), we apply the guidance in the steps [5, 30] with cfg scale in the increasing range 3.6. This sampling procedure is similar to the one proposed in MUSE [23]. We use the Heun [19] discrete scheduler for inference with 30 sampling steps.
21

---

A.8 Limitations
We develop hybrid architectures with asymmetric distribution of the convolution and transformer blocks. We show that this design paradigm is generic and applicable to many vision tasks such as image recognition, semantic segmentation, and text-to-image generation. We expect this architecture design to be widely useful for other applications. From the architecture design perspective, our building blocks are not rigid and can replaced with other efficient alternatives of the chosen C and blocks. Currently, we focus on the vanilla attention mechanism, but we can certainly incorporate faster alternatives to quadratic attention without sacrificing any efficiency advantages. Since we demonstrate an application of the asymmetric design to the UNet framework, we list out some limitations of our text-to-image generation model, which is broadly similar to various existing text-to-image models,
Extra limbs. While our model is able to generate limbs for humans and animals very well in most cases. There are instances where we can observe extra limbs in generations. It is unclear if this is due to poorly learned diffusion process, poor captioning guidance, lack of granular information in image-caption pairs, or just an artifact of the VAE encoder while converting the latent space. Artifacts in extreme resolutions. We train our T21 model with multi-aspect ratio bucketing. While it is able to adapt to many different resolutions in a graceful manner, we find that the model generates duplicate patterns in the extremes of the aspect ratio (significantly higher width than height or vice-versa). We believe this due to the lack training data in such resolutions.
While we are much better at avoiding NSFW generations compared to baselines (due to careful curation of the training data), there might be slight percentage of NSFW content, due to leakage in various filtration processes in the dataset curation. Thus, we incorporate an additional NFSW filter on top of the generation to remove such edge cases.
A.9 Societal Impact
We design the asymmetric architecture and show the advantages it brings in many applications (image recognition, semantic segmentation, and rext-to-image generation). Our architecture design targets better trade-off between efficiency (training/inference latency) while achieving achieving good performance. itself, the asymmetric design would reduce the amount of resources utilized by deep neural networks used commonly in vision tasks. The downstream applications of this architecture, especially in Text-to-Iman Image generation need to be deployed with care. We expect the community to utilize proper care w.r.t. training data well the generated output. There needs to be special checks in place while preparing the training data for T2I tasks. We specially add various filters to remove malicious and harmful content from the dataset. Further, even generated images need to be processed with some filtration to remove similar harmful and malicious content.
A.10 Text-to-Image Generation Results.
For visualization purposes, we generate more images with different prompts using multi-aspect ratio model. We present these images in Fig. 9, 10, and 11. shows that our model is able to generate wide range of images, both very realistic as well as stylistic images. In addition, it is able to grasp various concepts and compositions.
22

---

old koowloon walled city hong kong
victorian gothic steampunk outdoor mansion and woods at night unknown creatures in the woods
A tornado made of tigers crashing into a skyscrappr. painting in the style of Hokusai.
There large gray duck next the water. There are small waves in the water. There is log next to the water.
harlequin WOMAN, crazy face epic scène, half robotic, led lights,
A magical pink castle a rocky cliff near the sea
a humanoid robot performing maintenance itself by Frazetta
an eye-level full shot of a peacock standing sideways the ground there is beige stone in background, and trees and palms are visible the foreground, hyperrealistic extremely detailed
Warhammer 40k, UHD, surreal vibes
Figure 9: Example text-to-image generation results.
23

---

Table 7: Results on ImageNet-1K [72] for Classification Task. We compare the performance of asymmetric architectures, AsCAN, against baselines. We report the inference latency the throughput (images per second) that is measured by inferring images with batch size as B in half-pree paff-precision, ie., fpl6, library [100]. A similar procedure (without torch-compile) is followed to obtain the throughput on the V100 GPU.
A100 GPU using torch-compile and benchmark utility from timm
Throughput (images/s) Batch (B)
Top-1
Architecture
Resolution Params MACs
A100
VIOO
Accuracy
B=1
B=16 B=64 B=1
B=16
EfficientNet-B6 [73]
528
43M
19.0G
88
529
589
27
205
84.0%
EfficientNet-B7 [73]
600
66M
37.0G
321
24
124
84.3%
NFNet-FO [101]
256
72M
12.4G
199
2470
3348
47
813
83.6%
NFNet-F1 [101]
320
132M
35.5G
106
998
1192
26
48
84.7%
ConvNet EfficiientNetV2-5 [T]
384
24M
8.8G
112
1884
2744
34
561
83.9%
EfficientNNet2-MM [1]
480
55M
24.0G
918
26
329
85.1%
ConvNeXt-S [74]
224
50M
8.7G
174
2291
2889
56
836
83.1%
ConvNeXt-B [74]
224
89M
15.4G
167
1760
2073
.88
619
83.8%
ConvNeXt-L.
224
198M
34.4G
168
1045
1127
58
362
84.3%
RDNct-S [102]
224
50M
8.7G
102
1782
2761
53
780
83.7%
RDNet-B [102]
224
15.4G
1578
1891
48
589
84.4%
RDNet-L [102]
224
186M
34.7G
78
640
990
32
290
84.8%
VIT-B/16 [TO]
384
86M
55.4G
2T2
T006
T266
104
86
77.9%
ViT-B/32 [10]
384
307M
190.7G
893
924
54
27
76.5%
VIT
DeiT-B [103]
384
86M
55.4G
1058
488
83.1%
Swin-S [46]
221
50M
8.7G
50
841
2221
34
436
83.0%
Swin-B [46]
381
88M
47.0G
51
458
486
20
85
84.5%
PVTv2-B3[91]
221
45M
7G
49
933
3782
33
548
83.2%
PVTv2-B5[91]
221
11.8G
511
2112
17
83.8%
EfficiemtViT-B22993
221
24M
1.6G
142
2171
5132
48
782
82.
EfficientViTTB33[933]
221
49M
4G
114
1882
2860
32
552
83.5%
MOAT-2[12]]
224
73M
17.2G
132
1367
2087
38
424
84.7%
MOAT-3[92]
190M
44.9G
805
21
246
85.3%
SMT-S(79]
224
21M
4.7G
38
566
2211
43
348
83.7%
SMT-B[79]
224
32M
7.7G
27
388
1412
14
243
84.3%
MogaNct-L. [82]
224
83M
15.9G
34
523
882
24
290
84.7%
MogaNct-XI. [82]
224
181M
34.5G
32
471
576
19
210
85.1%
BiFormer-S [80]
224
26M
4.5G
45
937
2139
36
595
83.8%
BiFormer-B [80]
221
57M
9.8G
50
840
1439
28
440
84.3%
RMT-S [81]
221
4.5G
790
38
480
84.1%
Hybrid
CoAtNet-O [11]
221
25M
4.2G
214
3537
5221
61
976
81.6%
1 [11]
224
42M
8.4G
141
2221
2907
45
629
83.3%
CoAtNet-2 [11]
224
75M
15.7G
133
1718
2040
38
540
84.1%
3oAtNet-3 [11]
224
168M
34.7G
1085
37
388
84.5%
Max [31]
224
31M
5.6G
73
1098
2756
23
357
83.62%
MaxViT-S [31]
224
69M
11.7G
1019
1775
24
243
84.45%
MaxViT-B [31]
224
120M
23.4G
34
507
1012
11
164
84.95%
MaxViT-L [31]
224
212M
43.9G
34
544
759
10
123
85.17%
FasterViT-1 [32]
5.3G
1123
23
363
83.2%
Fister [32]
221
76M
8.7G
64
1112
4376
24
321
84.2%
FasterViT-3 [32]
224
160M
18.2G
46
831
3131
17
257
84.9%
FasterViT-4 [32)
224
425M
36.6G
50
800
1392
18
234
85,4%
AsCAN-T
221
55M
7.7G
3224
4295
67
1148
83.44%
Ours
AsCAN-B
224
98M
16.7G
113
1878
2393
38
590
84.73%
AsCAN-L
224
173M
30.7G
120
1381
1617
40
440
85.24%

---

Table 8: Results on ImageNet-1K for Classification Task (memory consumption). Following
Tab. 7, for recent related networks, we include memory consumed during inference with 64 batch size.
Throughput (images/s) Batch (B)
Memory (GB)
Top-1 Accuracy
Architecture
Resolution Params MACs
A100
V100
B=1
B=16 B=64 B=1
B=16
B=64
RDNet-S [102]
224
50M
8.7G
102
1782
2761
780
83.7%
12.6
ConvNet RDNet-B B [102]
224
87M
15.4G
80
1578
1891
48
589
84.4%
17.5
RDNet-L L [102]
224
186M
34.7G
78
640
990
32
290
84.8%
26.2
MOAT-2 2 [92]
224
73M
17.2G
132
1367
2087
38
424
84.7%
31.8
MOAT-3 [92]
224
190M
44.9G
70
805
962
21
246
85.3%
65.2
SMT-S [79]
224
21M
4.7G
38
566
2211
43
348
83.7%
13.5
SMT-B [79]
224
32M
7.7G
388
1412
84.3%
22.6
Hybrid
MogaNet-S [82]
224
25M
5.0G
93
1593
2455
47
740
83.4%
17.1
MogaNet-L. [82]
224
83M
15.9G
34
523
882
24
290
84.7%
46.6
MogaNet-XI [82]
224
181M
34.5G
32
471
576
19
210
85.1 %
74.9
BiFormer-S [80]
224
26M
4.5G
45
937
2139
36
595
83.8%
18.6
BiFormer-B [80]
224
57M
9.8G
50
840
1439
28
440
84.3%
27.9
RMT-S [81]
224
27M
4.5G
46
790
2439
38
480
84.1%
13.7
AsCAN-T
224
55M
7.7G
199
3224
4295
67
1148
83.44%
9.6
Ours
AsCAN-B
224
98M
16.7G
113
1878
2393
38
590
84.73%
15.2
AsCAN-L
224
173M
30.7G
120
1381
1617
40
440
85.24%
21.2
Table 9: Analysis of Architecture Configuration (with T before C). We extend Tab. 2 by ablating over the preference and blocks in a stage. Using block stem / early layers result in lower throughput. Further, the performance of configurations with T before C yields lower accuracy vs throughput trade-off.
Inference(images/)
Top-1
Block Configuration
Params
A100
V100
Acc.
B=16 B=64
B=16
(C1)
55M
3224
4295
1148
83.4%
CC-CCCT-6CTT-CCIT (C2)
73M
3217 4179
1036
83.2%
CC-CCCT-CCTT-ITI (C3)
41M
3384
4472
1224
82.9%
CC-COCT-CCOCC-TIE (C4)
50M
3434
4411
1182
83.1%
D6-CCCT-CCCT-CCCG (C5)
95M
3135 4066
991
82.7%
CC-TCCC- wwwC-CCCTTTT (TI)
56M
3029 4021
945
82.8%
CC-COCT- -ECSCTT-TCC-CTTT (T2)
57M
3100 4092
1021
82.9%
CC-CCCT-CCCT-TTTE (T3)
64M
3190 4193
1045
83.1%
--CC-CCCCC---CC-- (T4)
55M
1428
1584
487
83.1%
TE-TTTC-TTCCC TTCC-TTTC (T5)
100M
1280
1440
390
83.5%
Table 10: ImageNet-21K [72] Pre-training. We report the performance of models pre-trained on ImageNet-21K with 224 resolution and fine-tuned on ImageNet- KK dataset. report the inference latency as the throughput (images per second) that is measured by inferring images with batch size as half-precision, i.e. fp16, on an A100 GPU using torch-compile and benchmark utility from timm library [100].
Batch (B=16)
Batch (B=64)
Top-1
Architecture
Resolution
Params
MACs
Throughput (images/s)
Throughput (images/s)
Accuracy
ConvNeXi-L [74]
224
198M
34.4G
1045
1127
86.6%
MaxVTT-L [31]
224
212M
43.9G
544
759
86.7%
ViT- [32]
224
425M
36.6G
800
1392
86.6%
Ours
AsCAN-L
173M
30.7G
1381
86.7%
25

---

Table 11: Semantic Segmentation Results on ADE20K [104]. We compare different backbones on the ADE20K [104] semantic segmentation benchmark with UPerNet [106] the detection architecture. Computational and storage statistics are computed using the input resolution of 512 x 512.
Latency (frames/s)
Latency (frames/s)
MACs (G)
Backbone
Params
mloU
A100
V100
Swin-T [46]
44
23
60M
237
44.5
Faster ViT-2 [32]
47
25
47.2
AsCAN-T (Ours)
64
30
86M
264
47.3
Swin-S [46]
27
16
81M
261
47.7
FasterViT-3 [32]
48.7
AsCAN-B (Ours)
44
24
128M
311
48.9
Swin-B [46]
13
121M
301
48.1
FasterViT-4 [32]
28
15
49.1
AsCAN-L (Ours)
36
19
204M
397
50.3
Table 12: Resource Usage Comparison. We compare the training time required to learn various diffusion models in the literature to our proposal. We also include parameter count and inference latency of these models.
Train GPU Inference per image
Inference per image (1024 1024)
Model
Params
A100 days
(512 512)
RAPHAEL
3.0B
60,000
DALL.E 2
6.5B
41,667
Imagen
3.0B
7,132
13.3s
SDv1.5
0.9B
6,250
1.9s
SDXL+Refiner
2.6B
7.9s
12.2s
Wurstchen
1.0B
810
2.5s
GigaGAN
0.9B
4.783
PixArt-
0.6B
753
2.1s
7.28
Ours
2.4B
2,688
2.2s
5.5s
26

---

this imagee can object with different colors placed the surface and the bachground there doors and walls.
In this image I can see colorful threads rolls, which the surface. the backkround it looks like window.
barred owf pocking out from dense branches
A teddy bear under some furniture that appears to be turned on It's side.
bird known for its distinctive blue and orange plumage. The kingfisher perched branch, its body angled slightly to the inft as if poised take flight any moment.
cute orange kitten sliding down aqua slide. happy excited. 16mm lens in front. we see his excitoment and scared in the vibrant colors, water spleshing on the lens
dirship docked the part of city in the
vikings cave trofl, vilking army, viking Fighting the rain, city fire, rain, Richard Schmid style, matte painting. cinematic, spic
An orea whale swimmiing in the Nile River fromt of Egypsian pyramid
sky by Ivan shishkin, artstation, bloshock infinite, 35mm, cinomatic, -ar 2:1
Figure 10: Example text-to-image generation results.
27

---

twe phaffer on atr ground with treas the baohground
mind-blowhig zunnet the mountalus
lumn forest with witlding ther
thunderstorm in mystterour forest
Arachy Aenort landscape with sowering moser
raging
Digital ilustration of beach noans crafted from beige yarn, onto shore, yarr sun sots on the korizon,
antheguompphic profile the white mow ow! Crystal prizztess art axco painting, pretty
raived.
shouiders, He gray syee tooking directly into cameris, weering at blue safety hatmar and gray hoaded orange safecy detailing vortically araund choulder. blurred backgroum goong has grayy eqvibmeent on the curtain
yarn. zendy beach depicted
Intricate, elaborate, hypernatton, hypar detailed, high actant, ultra resotation, perfection,
casting warm glow. Vrom polm tress zurty pently,
18, conematic lighting, uinal ciariyy shoding, tumen super
aweatahue
grading, retouch, enhanced, P8R, Blander, V-ray, Precrente, atrush, inroal Engine cinamatia, digital pointing blur
lighting, engle
Figure 11: Example text-to-image generation results.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction regarding the efficiency and performance trade-offs of the asymmetrically distributed convolution-attentii networks, reflect the paper's ' contributions. We design these new architectures for image recognition, semantic segmentation, and text-to-image applications. We show these applications and architecture modifications and advantages in our empirical evaluations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes. We describe the limitations in Appendix Sec. A.8.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results included in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the architecture configurations used in this work in detail in the main and appendix sections. We also describe the hyper-parameter configurations used train this model (training iterations, optimizer, learning rate, weight decay, data augmentation, etc.)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
30

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release our asymmetric architecture implementation for facilitating future research in this direction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co
public/guides, / ittpssummsssioooooooooiooooeeeee onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppbcccggguueeecccccccccccccoe wwwcc//ggg/e//eeeeeeeeeee https//oooSuuuiissisiiiee siinissssoocccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental settings and details for experiments conducted in this work (see Sec. A.2, Sec. A.3, Sec. 3.3.2).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Similar to earlier works ImageNet- KK and Text-to- Image generations, do not report any error bars due to high computational resources involved in training one run.
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we describe the amount of resources required to train our models, includ- ing the hardware type. We also provide the total number of iterationsepoocss required to reproduce the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't it make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / /tt.//nurrp...e htcpppllic/tthhcccceeeeeeeeeeeee
Answer: [Yes]
Justification:
We have reviewed the code of ethics and our research work conforms to this
code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
32

---

Justification:
We discuss the societal impacts of this work in the appendix Sec. A.9,
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Yes, we discuss various safeguards both for preparing training data as well as serving the generated images for our text-to-image generation model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For our image recognition and semantic segmentation experiments, we use the widely available ImageNet- 1K and ADE20K datasets. We cite these resources in the work. For the T2I experiments, we use first party and licensed datasets. We depict this entire process in Fig. 6. We start by collating all these data sources to create stream of images and if available, relevant text captions. Next stage, we add series of filters to improve the quality of the selected images (resolution, de-duplication, aesthetic score, nsfw filtering, etc.). Since this data most contains very short description almost non-existent description
33

---

of the image, we unify the captioning process using an image captioning model. We collect subset of 200K data for human annotations, where we ask the annotators to add details such as the <angle shot of the image>, <image background information>, <human attributes>, etc. We use this labelled data to fine-tune a BI.IP2 model which is used generate long captions similar to this format.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
34

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "tMbWfuCofh", "reviewer": "Reviewer_hpXB", "summary": "This paper presents a hybrid neural network architecture that incorporates both convolution-based and vision transformer (ViT)-based building blocks for discriminative and generative modeling. The proposed convolutional block, labeled as (C), is identical to the EfficientNetV2's FusedMBConv, where a standard 3x3 convolution with a 4x dimension expansion and an extra squeeze excitation module (SE) is within, followed by a 1x1 convolution where GELU replaces SiLU. The transformer-like block, termed vanilla Transformer (T), processes the vanilla self-attention and MLP blocks in parallel, which is in contrast to the original sequential Transformer block. For ImageNet experiments, the authors follow the standard protocol to evaluate the proposed models with the ImageNet-1k top-1 accuracies, where the proposed model is empirically adjusted in the order of C and T blocks to optimize the trade-off between ImageNet-1K top-1 accuracies and computational budgets, including throughput and the number of parameters. The resulting optimal model is like asymmetric architecture, wherein the number of convolution and transformer blocks is asymmetric in different stages. Additionally, the architecture is applied to UNet-based diffusion models (i.e., DDPM), structured similarly to the ImageNet architecture, to assess the effectiveness of the proposed design. The generative model is evaluated by the FID score on generating 256x256 ImageNet-1K 2 images using class conditional generation. The optimal architecture for this turned out to be asymmetric across the UNet stages as well.", "questions": "- See the weaknesses\n- Can the authors give the performance tradeoff like Fig. 3 on an A100 GPU?", "limitations": "Limitations are provided.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper is easy to follow and well-written.\n+ Extensive experimental results are provided, proving the claim in both discriminative and generative modeling.\n+ The performance on the ImageNet-1K classification looks impressive compared with some recent models.", "weaknesses": "- The paper lacks intuition and clear reasoning for the architectural design choices. No explanation or key insights are provided on why the proposed design or employed modules perform better than other alternatives. \n  - It is unclear why the S1 stage should consist solely of two convolution layers. It seems that the traditional stem (having the stride 4) is separated into two stages (i.e., S0, S1). Are there any other reasons for this design choice?\n  - The authors claimed that the searched architecture is asymmetric, but it is unclear why an asymmetric combination of convolution and transformer blocks would perform better. The rationale behind this architectural choice needs further clarification. \n  - While the chosen FusedMBConv and Vanilla Transformer in Table 1 appear promising, it is unclear why these options outperform others. \n  - The rationale for why the generative model achieves a better FID score using modules optimized for the recognition task is not clearly explained.\n\n- The positioning of self-attention modules after convolution-based layers (in a stage or across the entire network), as seen in the architectural choices, has already been explored in previous studies [1, 2]. Therefore, the architectural results presented in Table 2 can be considered expected outcomes based on the known knowledge. The authors are encouraged to provide new insights or takeaways.\n\n- There is no held-out set for searching architectural choices. It appears that the proposed architecture was optimized over the ImageNet validation set for the best performance, but a more appropriate setup would involve using a separate held-out set.\n\n- The results from pre-training on ImageNet-21K are not as impressive as those from ImageNet-1K. This reviewer suspects that the reason may be that the architecture is highly fine-tuned to the ImageNet-1K dataset.\n\n- The ADE20K Semantic Segmentation results are not fairly compared and missing numbers for recent architectures; the competing methods enjoy smaller computational costs (e.g., for Swin) to the proposed method than the proposed models; and there are only Swin and Faster-ViT architectures for comparison.\n\n- The proposed Transformer architecture (a parallel processing architecture of self-attention and MLP)  is referred to as the \"vanilla\" block, but it is known that a typical \"vanilla\" Transformer processes self-attention and MLP sequentially. This terminology used by the authors could lead to confusion. Furthermore, the parallel architecture was proposed in [a] and employed in many works like [b]:\n  - [a] MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning, arxiv 2020\n  - [b] Simplifying Transformer Blocks, ICLR 2024\n\n- It would be beneficial to provide memory consumption during training and inference for the proposed architectures.\n\n- Comparing the proposed method with some recent works would reveal its effectiveness more clearly. The authors are encouraged to compare performance trade-offs with the following recent works [3, 4, 5, 6].\n\n[1] MobileNetV4 - Universal Models for the Mobile Ecosystem, arxiv 2024\n\n[2] Scale-Aware Modulation Meet Transformer, ICCV 2023\n\n[3] BiFormer: Vision Transformer with Bi-Level Routing Attention, CVPR 2023\n\n[4] MogaNet: Multi-order Gated Aggregation Network, ICLR 2024\n\n[5] RMT: Retentive Networks Meet Vision Transformers, CVPR 2024\n\n[6] DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for reading the paper thoroughly and providing invaluable feedback. Below, we have tried to answer your questions and concerns. While we answered some questions in the main rebuttal, we reiterate their highlights for completeness.\n\n**Stem Design Choice.** This is a popular stem design and has been utilized in related works such as MaxViT, CoAtNet, etc. We have also included ablations with some configurations that replace convolution blocks with transformers.  \n\n**Rationale behind the asymmetric design.** Existing hybrid architectures such as MaxViT, FasterViT, CoAtNet, etc., follow a symmetric design in which C and T blocks are uniformly distributed across stages or within a stage. In contrast, AsCAN architectures recommend an asymmetric distribution with a preference for more convolutional blocks in the early stages and more transformer blocks in the later stages. Further, since even this search space is large for a reasonable computational budget search, we restrict ourselves to some design choices to find reasonably performing architectures in this search space. In Table 2, we perform ablations on various combinations of these C and T blocks and show that asymmetric configurations yield better performance-latency trade-offs. It is further evaluated in Figure 3 and Table 7, which compare AsCAN with existing hybrid architectures on various accelerators.\n\n\n**Why modules optimized for recognition tasks generalize on generative tasks.** We use the ImageNet-1K task for the search problem since it is computationally cheaper for ablations when compared to other tasks and is still representative enough for the vision domain. To show its benefits, we use this simple design in other tasks without any task-specific optimizations.  In the class-conditional generation, our asymmetric architecture achieves similar FID as various state-of-the-art models with less than half the FLOPs (translating into half latency). We see similar improvements in other tasks like Text-to-Image generation. Thus, we can conclude that the search on the ImageNet-1k task generalizes to other tasks. Indeed, we would expect to achieve even better performance once we optimize these modules per task, but this process is computationally expensive. \n\n**Architectural results presented in Table 2 can be considered expected outcomes based on the known knowledge.** \n- We already compare against SMT models ([2] Scale-Aware Modulation Meet Transformer, SMT-B, and SMT-S variants), see Figure 3 and Table 7 for comparison. For instance, AsCAN-B architecture achieves $84.73\\%$ top-1 accuracy with a throughput of $590$ samples per second on a V100 GPU, while SMT-B architecture achieves $84.3\\%$ top-1 accuracy with a throughput of $243$ samples per second. Notice that many operations in SMT architectures are not high-end accelerator friendly and thus their lower FLOPs do not translate to higher throughput. Further, SMT architecture has a symmetric distribution of convolution and transformer blocks (initial stages have only C and later stages only have T). \n\n- Similarly, MobileNetV4 architectures are targeted toward mobile device applications and hence focus on layers that are mobile-friendly. For instance, having depth-wise separable convolutions does not utilize the full accelerator capacity on high-end GPUs. Further, this architecture only leverages transformer blocks sparingly in a few configurations. \n\n- Besides, we already compared against many hybrid architectures that have such a design (either C followed by T in different stages) or uniform mixing of C and T blocks within a stage. Instead, we are proposing an asymmetric distribution of these blocks in different stages. \n\n\n**Provide memory consumption for the proposed architectures.** We have included the inference memory consumption along with the numbers for the remaining new architectures requested by the reviewer. As illustrated in Table 3 (see attached rebuttal pdf), AsCAN architectures have lower memory consumption compared to other architectures. For instance, for batch-size $64$, the MogaNet-XL model consumes $74.9$GB memory while AsCAN-L consumes $21.2$GB memory. This is still lower than the memory consumed by RDNet-L model which is a purely convolutional architecture and requires $26.2$GB memory. \n\n**Compare with recently proposed works.** Thank you for listing the missing related works. We have included a performance vs throughput comparison with these works in Table~3 in the attached rebuttal pdf. We have also included these architectures in Figure 1 for the performance-latency trade-off on V100 and A100 GPUs. In this list, there are architectures that have significantly lower FLOPs but this does not result in significant gains in the throughput when measured on high-end accelerators. This is due to a large presence of operators such as permute which do not add floating point operations but require additional runtime. In contrast, AsCAN architectures still outperform these works on inference latency vs top-1 accuracy trade-offs. \n\n\n**Performance tradeoff like Fig. 3 on an A100 GPU.** Thank you for raising this point. We already report throughput and performance in the Appendix (Table 7). We have included the requested performance trade-off figure for an A100 GPU in the rebuttal pdf. This follows a similar trend as the one for V100 GPU.\n\n\n**Confusion in Transformer block naming.**  Thank you for pointing this out, we will update the terminology in the paper. \n\n**Lack of held-out set in ImageNet experiments.** We follow previous works in the experimental setup.  \n\n**Semantic Segmentation results missing recent architectures.** Thank you for the feedback. We will include other results for semantic segmentation in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1hbF8tT82J", "reviewer": "Reviewer_3CqY", "summary": "The authors present AsCAN, a hybrid architecture that combines convolutional and transformer blocks, that can applied to both visual recognition and generation. This architecture features an asymmetric distribution, with more convolutional blocks in early stages and more transformer blocks in later stages. It demonstrates favorable results in large-scale text-to-image tasks, outperforming recent models in both public and commercial domains, and provides better throughput.", "questions": "Several design choices are not thoroughly evaluated.\n- Why is C placed before T? Do you have quantitative results on placing T before C?\n- Why is the first stage fixed? We can introduce (adaptive) pooling to incorporate T in the early stage and compare it with pure convolutions.", "limitations": "Macro-level architectural ablation experiments are mostly performed on the classification task due to resource limitations. This fundamentally limits the generalizability of the conclusions to other tasks. We cannot draw any strong conclusions derived from classification experiments for the generation task.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[Intuitive Design]The authors leverage existing vanilla attention along with the FusedMBConv block to design the new architecture, called AsCAN. Their main philosophy revolves around the asymmetric distribution of the convolutional and transformer blocks in the different stages of the network; more convolutional blocks in the early stages with a mix of few transformer blocks, while it reverses this trend favoring more transformer blocks in the later stages with fewer convolutional blocks. \n\n[Extensive Experiments] The proposal is thoroughly evaluated in both classification and generation tasks, achieving state-of-the-art results in both.\n\n[Scalability] The authors show that the proposal scales well in the regime of Tiny-Large recognition models.", "weaknesses": "[Unclear explanation]\n- Please provide clearer and more intuitive explanations of how the proposal can achieve higher throughput while maintaining or improving accuracy. \n\n[Model/Data/Resolution Scalability]\n- Please demonstrate how the proposal can scale up to larger recognition models (XL/XXL/huge regime), as it currently only shows tiny to large models in the appendix.\n\n- Please illustrate how the proposal can scale up to larger generation models. Specifically, when FLOPs are matched similarly to the FLOPs shown in Table 3 (100-150G), can we achieve better FID scores?\n\n- Please illustrate how the proposal is scalable w.r.t the data size. The appendix Table 11 provides only the snapshot on 450M images.\n\n- Please illustrate how the proposal is scalable for higher resolution image processing. For example, does the current proposal perform better in both recognition and generation compared to other architectural designs when the input resolution increases (224 -> 512 -> 1k, etc.)? Especially focus on generation task.\n\n[Minor] \n- The terminology (e.g., equal blocks in remaining stages, asymmetric vs. symmetric) can be improved for better clarity.\n\n- The table formats (e.g., Table 1 and Table 2 heights are different) can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for reading the paper thoroughly and providing invaluable feedback. Below, we have tried to answer your questions and concerns. While we answered some questions in the main rebuttal, we reiterate their highlights for completeness.\n\n**Explanation of higher throughput while maintaining or improving accuracy.** \n\n- Many existing works design architectures based on parameter count and FLOPs, but this typically does not translate into inference throughput gains. Some of these issues come from using operators that do not contribute to parameter count and FLOPs but require non-trivial runtime, such as reshape, permute, etc. Others originate from the lack of efficient CUDA operators for these specialized attention and convolutional blocks. \n\n- For instance, MaxViT consists of MBConv and Axial-attention transformer blocks. MBConv block is more friendly for mobile devices due to separable convolutions but hurts throughput on high-end GPU accelerators. Similarly, Axial-attention heavily invokes the permute operations. It results in additional overhead and hurts throughput. \n\n- In contrast, we utilize building blocks that are efficient for high-end GPU accelerators, and our experimental design directly measures the inference throughput on different accelerators to search for optimal performance vs throughput trade-offs. This search favors asymmetric design with more convolution blocks in the early stages and more transformer blocks in the later stages. This helps reduce the occurrence of reshape and permute operations appearing in instances where C and T blocks are interleaved repeatedly. \n\n**Scaling to larger architectures.** There are many strategies to scale the asymmetric design. In one direction, we can train our existing models with larger input resolution to achieve better top-1 accuracy. For instance, training AsCAN-L trained with $384$ resolution results in $86.2$\\% top-1 accuracy, compared to training with $224$ resolution which yields $85.2$\\%. On the other direction, we can scale these base architectures similar to previous works (CoAtNet, MaxViT, EfficientNetV2, etc.) by proportionally scaling the width and block repetitions in the stages (S0, S1, S2, S3). For instance, we can scale to AsCAN-XL variant by increasing the number of C and T blocks as (CC, C$^{4}$T$^{2}$, C$^{8}$T$^{8}$, C$^{4}$T$^{8}$), resulting in a $340$M parameter model that achieves $86.7$% top-1 accuracy. We can similarly scale these configurations. We can obtain even larger models by scaling the stages S1, S2, and S3. \n\n\n**Scaling Text-to-Image Generation w.r.t. data size.** While it would be good to understand the model performance scaling w.r.t. data size, unfortunately, we do not have compute resources to train multiple such text-to-image generative models at any higher data scale than $450$M image-text pairs. \n\n**Class conditional generation with $512\\times512$ resolution.** We trained our small UNet variant on the $512\\times512$ generation task. We report its FLOPs, throughput, and FID scores in the rebuttal pdf (see Table 1). On this task, DiT-XL/2 with $525$G FLOPs and $51$ throughput achieves an FID score of $3.04$. Similarly, U-ViT-H/2 with $546$G FLOPs and $45$ throughput achieves an FID score of $4.05$. In contrast, our smaller model with $224$G and $130$ throughput achieves an FID score of $3.15$. Thus, it has more than twice the throughput with a similar FID score. \n\n**Class conditional $256\\times256$ generation with 100G FLOPs.** We created an additional class conditional generation model on $256\\times256$ resolution with an asymmetric model using 103G FLOPs. It achieves a throughput of $360$ samples/sec with an FID score of $2.08$ while DiT-XL/2-G model achieves a throughput of $293$ samples/sec with an FID score of $2.27$. We will include higher FLOPs models in the final version. \n\n**Design choice ablations.** We have included ablative results for configurations with T before C as well as where the first stage consists of T blocks (see Table 2 in rebuttal pdf). In the instances where the first stage consists of T, the throughput is significantly lower than in the instances where C is the first stage. Similarly, the configurations with T before C do not achieve similar accuracy vs performance trade-off as the configurations where C appears before T. \n\n**Generalizability of architecture search on ImageNet-1k**. We use the ImageNet-1K task for the architecture search problem since it is computationally cheaper for ablations when compared to other tasks and is still representative enough for the vision domain. To show its benefits, we use this simple design in other tasks without any task-specific optimizations. In the class-conditional generation, our asymmetric architecture achieves similar FID as various state-of-the-art models with less than half the FLOPs (translating into half latency). We see similar improvements in other tasks like Text-to-Image generation. Thus, we can conclude that the search on the ImageNet-1k task generalizes to other tasks. Indeed, we would expect to achieve even better performance once we optimize these modules per task, but this process is computationally expensive.\n\n**Clarity in terminology.** Thanks for pointing out the formatting and terminology issues. We will address these in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3G96meLs78", "reviewer": "Reviewer_jBHn", "summary": "Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation\nIn this paper, AsCAN combines both convolutional and transformer blocks. The authors revisit the key design principles of hybrid architectures and propose a simple and effective asymmetric architecture, where the distribution of convolutional and transformer blocks is asymmetric, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages.\n\nThis paper contains tremendous experimental results to show its efficiency. However, it has several weak points.", "questions": "Please, see the section of Weakness.", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper contains tremendous experimental results to show its efficiency.", "weaknesses": "1. The main contribution is to combine convolutional and transformer blocks to obtain the better performance with reduced latency. \nHowever, its theoretical analysis is not shown enough. The asymmetric architecture is also used to generate images. Although the intensive experiments seem to make the contents more fluent, intensive analysis in terms of theoretical points is not enough. \n\n2. The marginal performance enhancements are given in this paper. For example, Figure 3 shows the proposed architecture achieved the best optimization. However, I think that the enhancements were very small. More perspective analysis and expectation are required. \n\n3. In the image generation part, the proposed model is used as a backbone. Is there any internal analysis such as weight distribution and training characteristics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We provide additional clarifications in this response to the best of our understanding of the reviewer's concerns. \n\n**(Q.1) Lack of theoretical analysis.** \nWe propose a new design for hybrid convolutional-transformer architectures and apply this new architecture to different applications (image recognition, class conditional generation, text-to-image generation, etc.). We show that our proposal achieves significantly better performance vs latency trade-offs. We analyze the throughput on various accelerators and show the theoretical FLOPs required by all the models. Our asymmetric architectures consist of transformer blocks with a quadratic complexity in the input sequence length. It would be good if the reviewer could elaborate on the theoretical analysis required in this neural network architecture design. \n\n\n**(Q.2) Marginal performance enhancements.**\n It is unclear why the reviewer thinks performance enhancements are marginal in this work. Other reviewers have already acknowledged that performance improvements are non-trivial (see strengths mentioned by reviewers 3CqY, R6Bu, hpXB, and fxge). To be concrete, \n\n- For image recognition tasks (see Figure 3, Table 7), AsCAN architectures achieve similar accuracy with up to $2\\times$ throughput increase compared to other architectures such as MaxViT, FasterViT, ConvNeXt, CoAtNet, etc. We have incorporated other related works in the rebuttal pdf (see Table 3 and Figure 1) that show similar gains in performance. \n\n- On class conditional generation (see Table 3), our architecture achieves similar FID scores with less than half the FLOPs. We have included latency and $512\\times512$ generation tasks in the rebuttal pdf (see Table 1). \n\n- Similarly, on the Text-to-Image generation task (see Table 4, Table 5, Table 11), our models achieve much better resource efficiency and image generation quality when compared to existing baselines. \n\n\n**(Q.3) Analysis on training characteristics.**\n We have performed experiments to ablate and improve the training quality. \n\n- We analyzed the model performance with and without pre-training on the ImageNet-1K task. Appendix Figure 8 shows the evolution of the FID score with the number of training iterations. It shows that training the model without pre-training on the ImageNet-1K task results in slower convergence, and the final FID achieved by the model without pre-training is worse than the one achieved by pre-training on this task.\n\n- We analyzed the importance of noise levels between $256\\times256$ and $512\\times512$ resolution stages in Appendix Figure 7. It shows that different resolutions prefer different input noise for the diffusion process. It helps us decide the use of noise=0.01 for $256$ resolution and noise=0.02 for $512$ resolution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmrW7L55Fe", "reviewer": "Reviewer_R6Bu", "summary": "The authors propose a principled way to design hybrid architectures for a variety of tasks, including image classification, semantic segmentation, class-conditional generation, and text-to-image generation. The goal is the resulting models, called Asymmetric Convolution-Attention Networks (AsCAN), to have the following characteristics, (1) to offer favorable performance-throughput trade-offs compared to existing SOTA models, (2) to be efficient across various modern hardware accelerators, and (3) to scale efficiently in terms of both compute and the amount of training data. \n\nAs a hybrid architecture, AsCAN consists of a sequence of convolutional and transformer blocks. The reason the authors opt for this kind of architecture, is because they aim to combine the advantages of both convolutional layers, e.g., translation equivariance, and transformer layers, e.g., global dependencies.\n\nThe authors experiment with different designs on the task of image classification by using ImageNet-1K, and then they generalize their findings to image generation. First, they experiment with different existing convolutional (C) and transformer (T) blocks, concluding to FusedMBConv for C, and vanilla attention for T, based on their accuracy vs throughput trade-off on different GPUs. Then, they experiment with different distributions of C and T blocks in a classification architecture. They always use a convolutional stem, four processing stages with multiple blocks, and a classification head. They conclude in an asymmetric design, meaning that they use more C blocks in the early stages, and more T blocks in the latter stages. They follow the same principles to design the U-Net backbone of a latent diffusion model for image generation tasks. All models are scaled by adding more blocks at each stage, maintaining the asymmetric distribution between C and T blocks. To efficiently scale training to large datasets, the authors propose a multi-stage training regime, where a model is first trained on a smaller dataset, and then, in subsequent stages, is fine-tuned for fewer iterations on the larger dataset.\n\nThe authors first experiment on ImageNet-1K, showing that AsCAN models offer a better accuracy-throughput trade-off compared to SOTA baselines. The benefits in throughput are demonstrated with different GPUs and batch sizes. Then, they show that their AsCAN diffusion model performs on par with SOTA models on ImageNet-1K class conditional generation, but with considerably less FLOPs. Similarly, AsCAN demonstrates similar performance to the baselines on semantic segmentation on ADE20K, but with higher FPS. Finally, the authors train a 2.4B params AsCAN latent diffusion model on an internal dataset of 450M images, for the task of text-to-image generation. They show that their model outperforms the baselines on most aspects of the GenEval benchmark, and either outperforms or is on par with the baselines on image-text alignment, based on a human study.", "questions": "1. In NeurIPS checklist 15, the authors answer NA, however, in Section 4.3 (Ln 276-284) and Fig. 5, the authors report a user study with human subjects, doesn’t this study require approval for research with human subjects?\n\n2. In Ln 763, Section A.8, it is mentioned “we are much better at avoiding NSFW generations compared to baselines (due to careful curation of the training data)”, how is this measured in the actual output of the models?", "limitations": "The authors discuss limitations and societal impact in Sections A.8 and A.9 respectively. One thing that in my opinion could be added in the limitations, is that AsCAN sometimes require considerable more parameters to achieve favorable performance-throughput trade-offs, leading to higher memory footprint. For example, I think this is the case in Table 7 between AsCAN-L and EfficientNetV2-M, as I mentioned in a previous section.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Quality:\n\n1. The authors develop the proposed design principles in a structured way, ablating different options, and offering adequate metrics, which include accuracy, number of parameters, and throughput on 2 GPUs and different batch sizes.\n\n2. Similarly, the experiments are well-structured, using multiple baselines and metrics. For example, on image classification the authors compare against SOTA CNNs, Transformers and hybrid architectures of different sizes, and they provide accuracy, actual throughput and number of parameters. Importantly, the experiments support the main claims of the work.\n\n3. The setup of all experiments is described in detail in the Appendix to aid reproduction.\n\nClarity:\n\n1. The manuscript is very well written, and easy to follow. The authors explain their method, and provide clear Figures and Tables, with appropriate captions.\n\nSignificance:\n\n1. The proposed models show clear benefits in terms of their performance-throughput trade-offs in different tasks, thus, they contribute to the community.\n\n2. The authors provide specific design principles that they used to build AsCAN, and they could be useful to other domains as well.", "weaknesses": "Originality:\n\n1. Hybrid architectures exist already in the literature, and AsCAN are built out of pre-existing components, like FusedMBConv, with known benefits, so, this limits the novelty of the approach.\n\nQuality:\n\n1. The authors constraint their design to always have convolutional (C) blocks before Transformer (T) blocks within a stage. However, in Table 2, which includes the comparisons for the macro design of the architecture, MaxViT, which achieves the best accuracy, alternates C and T blocks. In addition, the “C before T” constraint breaks between stages, when a stage ends with T and the subsequent stage starts with C. So, it is not clear to me why this constraint is set.\n\n2. I think the authors should provide actual throughput for the class conditional generation in Table 3. If I am not mistaken, this is the only experiment that doesn’t include this metric. The authors provide FLOPs in Table 3, however, as can be seen in other results, FLOPs don’t always translate to actual timings.\n\nClarity:\n\n1. Ln 23-24, “CNNs encode many desirable properties like translation invariance”, I think it should be “translation equivariance”.\n\n2. Ln 160, the authors mention “C1 vs C10”, but I think a more direct comparison is C2 vs C10.\n\n3. Ln 161, “While increasing the number of transformer blocks in the network improves the throughput”, I think Table 2 shows that increasing transformer blocks hurts the throughput, e.g., C6 has higher throughput on A100s compared to C9.\n\n4. In Appendix A.3, I think the “Experimental Setup” part in Ln 639-644 repeats for the most part information already provided in the “Training Procedure & Hyper-parameters” part (Ln 628-638), so, the two parts can be merged.\n\n5. Some minor typos, Ln 658, “most contains”, I think should be “mostly”; Ln 607 “the full results corresponding to the Fig. 3 in the Tab. 7”, I think the “the” before “Fig.” and “Tab.” are not necessary; Ln 681, “larger much”, I think “much” is not needed; Ln 733, “for this task similar”, I think there should be “is” after “task”; Ln 770, “achieving” is repeated twice.\n\nSignificance:\n\n1. One of the main contributions of this work is the favorable performance-throughput trade-offs offered by the proposed models. Specifically, in many experiments, AsCAN achieves similar performance compared to baselines, but manage to do it with higher throughput, so, I think the efficiency of the models makes them standout. However, I think it is not clear enough from the discussion of the method or the experiments, what are the causes of this efficiency. For example, in Section 3.1 (Ln 159-160), the authors point out that increasing T blocks in early stages decreases throughput (C8 vs C9 in Table 2), however, comparing C6, C7 and C8, which have an increasing number of T blocks earlier on, we see that C7 has slightly lower throughput compared to C6 on A100s and a bit higher on V100s, while C8 has considerably higher throughput compared to both C6 and C7 on V100, and on A100 for batch size 64, and lower for batch size 16 on A100. Also, C9, which has the most T blocks, compared to C6, which has T blocks only in the final stage, has slightly higher throughput on V100 and considerably lower on A100. So, it is not clear what exactly causes the variation in throughput, and to what extent is a matter of the hardware, e.g., would the same results persist on H100s?\nIn addition, the baselines are very diverse, not only hybrid architectures, so, it becomes even harder to interpret the differences in behavior. For example, in Table 7, AsCAN-L has similar accuracy with EfficientNetV2-M, while AsCAN-L has more than $\\times 3$ parameters, and about $\\times 1.3$ MACs. At the same time, EfficientNetV2-M is optimized to take advantage of FusedMBConv, but still, AsCAN-L has almost $\\times 1.5$ throughput on A100 with batch size 64, and about $\\times 1.3$ on V100 with batch size 16. Similarly, in Table 11, AsCAN with 2.4B parameters has higher throughput compared to PixArt-$\\alpha$, which has 0.6B params. I think the significance of the paper would be higher if the reported efficiency benefits were analyzed in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for reading the paper thoroughly and providing invaluable feedback. Below, we have tried to answer their questions and concerns. While we answered some questions in the main rebuttal, we reiterate their highlights for completeness.\n\n**Originality: Limited Novelty.** \n- Our main contribution is the asymmetric distribution of convolution and transformer blocks in various stages in a hybrid architecture. \n- We show that simple design choices (Sec. 3.1) yield architectures with existing blocks that achieve state-of-the-art performance and latency trade-offs. We demonstrate that this design is easily applicable to various applications.\n- Many works design architectures based on parameter count and FLOPs, but this typically does not translate into inference throughput gains. Some of these issues come from using operators that do not contribute to parameter count and FLOPs but require non-trivial runtime, such as reshape, permute, etc. Others originate from the lack of efficient CUDA operators for these specialized attention and convolutional operators.\n- In contrast, our proposal directly measures the inference throughput on different accelerators and incorporates building blocks that yield higher throughput. \n\n**Quality: Justification of C before T constraint.** Our intuition behind this constraint comes from the following observations:\n- Interleaving C and T in a stage performs many tensor reshape operations that do not add any FLOPs. However, these operations count towards runtime, and many such operations lower the model throughput.\n- Given a feature map, C blocks can capture local and scale-aware features, while T blocks try to work out the dependency between all feature values. Thus, it would be more beneficial to perform a convolutional operator first to capture these local and scale features, followed by pairwise dependencies between all tokens.\n- It helps to narrow the search process since interleaving these blocks would result in many possibilities and be hard to evaluate computationally.\n- Further, to validate our assumptions, we have included configurations where T appears before C in Table 2 in the rebuttal pdf. In the models where the first stage consists of T, the throughput is significantly lower than in instances where C is the first stage. Similarly, models with T before C do not achieve similar accuracy vs performance trade-off as the configurations where C appears before T.\n\n**Quality: Missing throughput in the class conditional generation.** Thank you for pointing this out. We benchmark the throughput (images generated per second) on an A100 GPU for all the baselines. In the attached rebuttal pdf, table 1 shows the throughput for one forward pass of batch size $64$ for all the models. To achieve an FID score of $2.23$, our $52$G FLOPs model achieves a $556$ samples/sec throughput while state-of-the-art DiT-XL/2-G with $118$G FLOPs achieves $293$ samples/sec. It shows that our asymmetric model still has nearly double the throughput compared to other models while achieving similar FID.\n\n**Clarity.**  We have included your comments in the manuscript. These will be reflected in the final version.\n\n**Significance.**\n- *EfficientNetV2-M vs AsCAN-L.* There seems to be a bit of confusion in reading EfficientNetV2-M numbers. This model has been trained with an input resolution of $384\\times384$ and evaluated at an input resolution of $480\\times480$. Thus, even with a smaller parameter count, it has much higher FLOPs than other models with nearly $50$M parameter range. Further, all the hybrid models have been trained and evaluated at $224\\times224$ resolution. For instance, with train/test input resolution as $224$, to achieve close to $85.1\\%$ top-1 accuracy, MaxViT-L requires 212M parameters, FasterViT-4 requires 425M parameters, MOAT-3 requires 190M parameters. When we follow the EfficientNetV2-m training strategy, AsCAN-L achieves $86.2\\%$ top-1 accuracy, while the larger EfficientNetV2-L variant with $120$M parameters achieves $85.7\\%$ top-1 accuracy. \n- *How much do the results depend on hardware?* The impact of C and T blocks in a hybrid architecture results in non-linear behavior across accelerators and batch sizes. This is precisely the reason we do not rely too much on the number of floating point operations to estimate inference latency. We choose 16GB V100 and 80GB A100 GPUs as representatives of two popular RAM and accelerator designs. We expect the trend on H100 to be similar to A100 and other lower-end GPUs (such as A10G, L4, etc.) to be similar to V100. While currently we do not have access to these accelerators, we will try to include benchmarks on these accelerators in the final version. \n- *T2I Speed Up.* PixArt-$\\alpha$ is a purely transformer architecture while AsCAN is a hybrid architecture involving convolutional blocks in early stages. As we have observed in our ablative experiments over C and T distribution in Table 2 as well as configurations where T appears before C in rebuttal pdf, transformer layers in the early part of the network significantly reduce the throughput. We will add further investigations into PixArt-$\\alpha$ in the final version.\n\n**Questions: User study review process.** Thank you for noticing this. We did get approval for the user study. We did not disclose these details to preserve anonymity. We will disclose the review process in the final version.\n\n**Questions: NSFW evaluation.**  We used a set of internal prompts that are suggestive in nature, i.e., they do not explicitly ask for generating unsafe images, but rather hide these details in words. We generated images with SDXL and our model. We compared the amount of NSFW images in these two models. Almost all of the SDXL generations are NSFW while our generations are safe and do not include any nudity. \n\n**Limitations.** We have included the inference memory consumption in Table 3. We will include the discussion on parameter aspects in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YIW8jfkK0g", "reviewer": "Reviewer_fxge", "summary": "The paper performs a thorough empirical analysis of design choices made in neural networks designed for computer vision tasks. Based on reasonable assumptions, they first narrow down the design space of the building block of such networks to combinations of Fused MBConv and vanilla transformer layers followed by an empirical analysis of the chosen combinations in terms of classification accuracy, latency (on GPUs) and model size. \n\nThe top-performing combinations exhibit significantly better latency on GPUs compared to similar accuracy models optimized for efficiency, such as Swin Transformers and MaxViT. Their model demonstrates the effectiveness of a simple convolution + vanilla transformer model as compared to several efficient/hybrid attention models.\n\nInsights gained from this design are also used to design a similar model for Text-image generation (asymmetric mix of conv + transformer), which also gives a better latency-performance (FID, human preference)  trade -off than existing models (Pixart alpha, SDXL).", "questions": "See weaknesses", "limitations": "The authors have addressed limitations", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-motivated, analyzing the benefits of a simple neural network architecture across a variety of tasks (recognition, segmentation, generation) on current hardware. The insights derived are of broad interest to a wide range of researchers.\n\n2. The paper provides a thorough empirical analysis of design choices in image recognition experiments.", "weaknesses": "The contribution of the difference in training pipeline versus that of the architecture for improving image generation performance is unclear. If an empirical ablation study is infeasible for computational reasons, a more detailed discussion in Section 3.2.2 would significantly enhance clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on this work.  \n\nAs already pointed out by the reviewer, ablations on large-scale text-to-image generation tasks are computationally very expensive. Hence, we performed small ablations on the ImageNet-1K dataset. We list these ablations below. We will include a detailed discussion of these characteristics in the final version.\n\n**Training algorithm ablations.**\n\n- We analyzed the performance of the model with and without pre-training on the ImageNet-1K task. Appendix  Figure 8 shows the evolution of the FID score with the number of training iterations. It shows that training the model without pre-training on the ImageNet-1K task results in slower convergence, and the final FID achieved by the model without pre-training is worse than the one achieved by pre-training on this task. Thus, we demonstrate that pre-training on this task helps speed the convergence. \n\n- We analyzed the importance of noise levels between $256\\times256$ and $512\\times512$ resolution stages in Appendix Figure 7. It shows that different resolutions prefer different input noise for the diffusion process. It helps us decide the use of noise=0.01 for $256$ resolution and noise=0.02 for $512$ resolution. We also ablated on other hyper-parameters (learning rate, weight decay) while training on $256\\times256$ and $512\\times512$ resolution stages. \n\n**Architecture ablations.** \n- We train on the class conditional $256\\times256$ generation on ImageNet-1k. As seen in Table 3, our asymmetric architecture achieves a similar FID score at less than half FLOPs under the same training setup as other baselines. We provide additional results on this task in the rebuttal pdf (see Table 1), including inference latency and generation on the $512\\times512$ resolution. It shows that our FLOP gains also translate to the inference latency gains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all the reviewers for their constructive and detailed feedback. We included following additional experiments in the attached rebuttal pdf:
1. Figure 1 top-1 accuracy vs throughput trade-off figures for both A100 and V100 GPUs 
2. Table 1 with additional details on class conditional generation (throughput benchmark and experiments for $512$ resolution)
3. Table 2 analyzes the impact of placing $C$ blocks before $T$ in our hybrid architecture. 
4. Table 3 compares other related works pointed out by reviewer hpXB with AsCAN architectures on the ImageNet-1k classification task. We also include the memory consumption. 

Below, we address common questions and concerns raised by the reviewers. 

**(Q.1) Novelty and Intuition Behind Higher Throughput.** 
- At the heart of our proposal lies the asymmetric distribution of convolution and transformer blocks in various stages in a hybrid architecture. 
- Asymmetric architectures outperform many existing models that utilize specialized attention and convolutional operators. These works design models based on parameter count and FLOPs, but it typically does not translate into inference throughput gains. Some of these issues come from operators that do not contribute to parameter count and FLOPs but require non-trivial runtime, such as reshape, permute, etc. Others originate from the lack of efficient CUDA operators for these specialized attention and convolutional operators.
- For instance, MaxViT consists of MBConv and Axial-attention blocks. MBConv is mobile-device friendly due to separable convolutions but hurts throughput on GPUs. Similarly, Axial-attention heavily invokes the permute operations, lowering throughput.
- In contrast, our proposal directly measures the throughput on different accelerators and incorporates blocks that yield higher throughput. 
- We show the benefits of our design on multiple tasks such as recognition and generation. We show that simple design choices (Sec. 3.1) yield architectures with existing blocks (FusedMBConv and Transformer) that achieve state-of-the-art trade-offs between performance and latency. 

**(Q.2) Base architecture for search.** There are many options for a base architecture. We chose a popular design used in MaxViT and CoAtNet papers. A more thorough approach would involve a neural architecture search for the base architecture and the search for optimal components (C and T blocks). However, this search process is computationally expensive due to the exponentially large search space. Thus, for resource efficiency, it is reasonable to use an existing base architecture. We provide simple scaling strategies to obtain larger models from the base architecture. Besides, our experiments on the class conditional and text-to-image generation tasks show that asymmetric architectures achieve better performance-latency trade-offs.

**(Q.3) Generalizability of architecture search on ImageNet-1k.** We use the ImageNet-1K task for the architecture search problem since it is computationally cheaper for ablations when compared to other tasks and is still representative enough for the vision domain. To show its benefits, we use this simple design in other tasks without any task-specific optimizations.  In the class-conditional generation, our asymmetric architecture achieves similar FID as various state-of-the-art models with less than half the FLOPs (translating into half latency). We see similar improvements in other tasks like Text-to-Image generation. Thus, we can conclude that the search on the ImageNet-1k task generalizes to other tasks. Indeed, we would expect to achieve even better performance once we optimize these modules per task, but this process is computationally expensive. 

**(Q.4) Class Conditional Generation.** We have included some requested experiments on this task in Table 1 in the attached rebuttal pdf.
- *Throughput numbers.* We benchmark the throughput (images generated per second) on an A100 GPU for all the baselines. Table~1 shows the compute time for one forward pass of batch size $64$ for each of these baselines. To achieve an FID score of $2.23$, our $52$G FLOPs model achieves a $556$ samples/sec throughput while state-of-the-art DiT-XL/2-G with $118$G FLOPs achieves $293$ samples/sec. 
- *Scaling to higher resolution.* To study the scaling of the model for higher resolution generation, we train the same small asymmetric UNet architecture for class-conditional $512\times512$ ImageNet-1k generation. Similar to the $256\times256$ task, we observe more than twice the throughput while achieving similar FID scores as the baseline architectures. Note that our text-to-image model can generate images in higher resolutions such as $1024\times1024$, $1536\times768$, etc. 

**(Q.5) C before T constraint.** Our intuition behind this constraint comes from the following observations:
- Interleaving C and T in a stage performs many tensor reshape operations that do not add any cost to FLOP. However, these operations count towards runtime, and having many such operations ends up lowering the model throughput.
- Given a feature map, convolutional operators can capture local and scale-aware features, while transformer blocks try to work out the dependency between all feature values. Thus, it would be more beneficial to perform a convolutional operator first to capture these local and scale features, followed by pairwise dependencies between all tokens. 
- It helps to narrow the search process since interleaving these blocks would result in many possibilities and be hard to evaluate computationally.
-  Further, to validate our assumptions, we have included configurations where T appears before C in Table~2 in the rebuttal pdf. In the models where the first stage consists of T, the throughput is significantly lower than in instances where C is the first stage. Similarly, models with T before C do not achieve similar accuracy vs performance trade-off as the models where C appears before T.

Meta-review: After the initially review of the paper reviewers pointed merits of the paper but also raised concern. Major concerned from reviewers were addressed during the rebuttal. I tend to recommend this paper for acceptance, but invite the authors to incorporate all the reviewers feedback in their submission.
Decision: Accept (poster)