AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation

Authors: anil  kag
huseyin  coskun
jierun  chen
junli  cao
willi  menapace
aliaksandr  siarohin
sergey  tulyakov
jian  ren
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:16:56

Abstract: Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN---a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, without performing any optimization of inference time our model shows faster execution, even when compared to works that do such optimization, highlighting the advantages and the value of our approach.

Main paper: 
 introduction: 
 Convolutional neural networks (CNNs) and transformers have been deployed in a wide spectrum of real-world applications, addressing various computer vision tasks, e.g., image recognition [1, 2] and image generation [3, 4, 5, 6, 7, 8] . CNNs encode many desirable properties like translation equivariance facilitated through the convolutional operators [9] . However, they lack the input-adaptive weighting and the global receptive field capabilities offered by transformers [10, 11] . By recognizing the potential benefits of combining these complementary strengths, research endeavors explore hybrid architectures that integrate both convolutional and attention mechanisms [12, 13, 14, 15] . Recently, such architectures witness huge success when scaled up to train large-scale text-to-image (T2I) diffusion models [16, 17, 18, 19] , enabling a vast range of visual applications, such as content editing [20, 21, 22, 23, 24, 25, 26] and video generation [27, 28, 29, 30] .One prominent research area for hybrid models involves the creation of building blocks that can effectively combine convolution and attention operators [31, 32] . While these efforts seek to use the strengths of both operators, their faster attention alternatives only approximate the global attention, leading to compromised model performance as lacking a global receptive field. Thus, they necessitate incorporating additional layers to compensate for the capacity reduction due to the attention approximation. On the other hand, minimal effort is directed toward optimizing the entire hybrid architecture. This raises the question: Is the current macro design of hybrid architecture optimal?In this work, we propose a simple yet effective hybrid architecture, wherein the number of convolution and transformer blocks is asymmetric in different stages. Specifically, we adopt more convolutional blocks in the early stages, where the feature maps have relatively large spatial sizes, and more transformer blocks at the later stages. This design is verified across different tasks. For example, in Fig. 3 , we demonstrate superior advantages of our model for the latency-performance trade-off on ImageNet-1K [33] classification task. Particularly, our model achieves even faster speed than many works featuring efficient attention operations. Additionally, we scale up our architecture to train the large-scale T2I diffusion model for high-fidelity generation (Fig. 1 , Fig. 4 , and Tab. 3). Furthermore, considering the high training cost for the large-scale T2I diffusion models, we introduce a multi-stage training pipeline to improve the training efficiency. Overall, our contributions can be summarized as follows:• We revisit the macro design principles of hybrid convolutional-transformer architectures and propose one with asymmetrically distributed convolutional and transformer blocks. • For the image classification task, we perform extensive latency analysis on the ImageNet-1K dataset and show our models achieve superior throughput-performance trade-offs than existing works (see Fig. 3 ). Notably, we show that the model runtime can be significantly accelerated even without any acceleration optimization on attention operations. Additionally, we show our pre-trained models on ImageNet-1K can be applied to downstream tasks such as semantic segmentation. • For the class-conditional generation on ImageNet-1K (256 × 256), our asymmetric UNet achieves similar performance as state-of-the-art models with half the compute resources (see Tab. 4). • For the text-to-image generation task, we demonstrate that our network can be scaled up for the large-scale T2I generation with a better performance-latency trade-off than existing public models (as in Tab. 3). Additionally, we improve the training efficiency through a multi-stage training pipeline, where we first train the model on a small dataset, i.e., ImageNet-1K, for T2I generation, then fine-tune the model on a large-scale dataset.Question: the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our claims in the abstract and introduction regarding the efficiency and performance trade-offs of the asymmetrically distributed convolution-attention networks, reflect the paper's contributions. We design these new architectures for image recognition, semantic segmentation, and text-to-image applications. We show these applications and architecture modifications and advantages in our empirical evaluations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Efficient Hybrid Architectures. Over the past decade, convolutional neural networks (CNNs) [34, 35, 36] have achieved unprecedented performance in various computer vision tasks [37, 38] . Despite numerous attempts to improve CNNs [39, 40, 41, 42, 43] , they still face limitations, particularly in terms of their local and restrictive receptive fields. On the other hand, the vision transformer (ViT) treats images as sequences of tokens [44] , facilitating the computation of global dependencies between tokens to enhance the receptive field. However, ViTs come with quadratic computation complexity concerning input resolution. To address this, several studies have aimed to develop more efficient attention operations [45, 46, 47, 48, 49] . Recently, there has been a growing interest in exploring models that go beyond pure convolutional or transformer blocks, combining them within a single architecture to harness the spatial and translational priors from convolutions and the global receptive fields from the attention mechanism [11, 31, 2, 12, 32] . We revisit the design choices of hybrid architectures and propose a new design with a fast runtime while maintaining high performance. We reveal that optimizing the macro architecture directly allows us to use the original attention for a superior latency-performance trade-off compared to existing works. Most importantly, our designed networks can be applied to different domains, e.g., image recognition and image generation, with both superior latency-performance trade-offs.Text-to-Image Diffusion Models. The development of text-to-image models, such as GANbased models [50, 7] , autoregressive models [51, 52] , and diffusion models [3, 53] , has enabled the generation of high-fidelity images by using textual descriptions. Among them, diffusion models demonstrate advantages in stable training processes and the scalability of largescale neural networks [19, 54, 55, 16, 17, 13] . Recent studies explore the enhancements of textto-image diffusion models through various directions, such as designing new network architectures [56, 57, 58, 59, 60, 61, 62, 63] , improving inference efficiency during multi-steps sampling [64, 65, 66] , and accelerating the training convergence [67, 68, 69, 70, 71] . Our designed network can be scaled up for the T2I generation when modified to a UNet architecture [13] . The model achieves better performance-latency trade-offs than open-sourced models. Furthermore, we can reduce the training cost through the proposed two-stage training pipeline.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes. We describe the limitations in Appendix Sec. A.8.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 This section describes our network and training designs in detail. First, we justify our choices of utilizing the convolutional and transformer operations for the building blocks, which we then use to design the asymmetric architecture (Sec. 3.1). Second, we scale up the network architecture for training the text-to-image diffusion models (Sec. 3.3). We point out that it is easier to ablate the design of building blocks on the image classification task due to the lower resource requirements (training/evaluation compute) in comparison to image generation.Our hybrid architecture consists of a mix of convolutional and transformer blocks, which enables operating on different input resolutions seamlessly and allows us to be pareto efficient w.r.t. performancecompute trade-off compared to other architectures. Before delving into the exact architecture configuration, we discuss our choice of the building blocks. We use X ∈ R H×W ×C to represent the input feature map X that has H × W spatial dimensions along with C channels. We denoteY ∈ R H ′ ×W ′ ×C ′as the output of a building block (convolution or transformer) and • symbol for the function composition operator. In the following, we present our design choices based on ImageNet-1K [72] classification task by varying different convolution and transformer blocks.There are various choices for a convolutional block that can be used in our architecture, e.g., MBConv [73] , FusedMBConv [1] , and ConvNeXt [74] . While MBConv block has been used in many networks [75, 31] , the presence of depthwise convolutions results in low accelerator utilization for high-end GPUs [1] . To better understand the latency-performance trade-off for various convolutional blocks, we experiment with the same hybrid architecture yet with different convolutional blocks. As in Tab. 1 (more experimental settings in Sec. 4.1), FusedMBConv has better latency on A100 and V100 GPUs than others, while maintaining high performance. Therefore, we adopt FusedMBConv (C) for the convolutional block and represent it asY = X + P • SE • C(X),where C is a full 3 × 3 convolution with 4× channel expansion followed by batch norm and GeLU non-linearity [76] , SE is a squeeze-and-excite operator [39] with shrink ratio of 0.25, and P is a 1 × 1 convolution to project to C channels.Transformer Block (T). Similar to the convolution blocks, there are many choices for attention mechanism in transformers, e.g., blocks containing efficient attention mechanisms like multi-axial attention [31] and hierarchical attention [32] . Tab. 2 shows that vanilla attention mechanism provides a better accuracy vs throughput trade-off across different GPUs and batch sizes. Thus, we choose the vanilla attention in our transformer block (T). We express it with the following update equations:EQUATIONwhere X = LN • ϕ(X), LN denotes layer normalization, ϕ denotes the GeLU non-linearity [76] ,A is the multi-headed self-attention function, P QKV & P MLP denote the linear projection to the QKV and MLP space, respectively, and P denotes the projection operator to the same space as the input.Note that these update equations are inspired by recent works [77, 78] , where the feed-forward and the self-attention operators are arranged in parallel in order to get improved throughput with marginal reduction in performance.Design Choices. Given the FusedMBConv (C) and Vanilla Transformer (T) blocks, we introduce the macro design for our hybrid architecture. For the image classification task, we follow the existing works [11, 31, 32] to utilize a four-stage architecture (excluding the convolutional Stem at the beginning and classifier components at the end). However, before finalizing our architecture, we still have a design question to answer, namely, in which configuration should we arrange these building blocks? For instance, CoAtNet [11] chooses to stack convolutional blocks in the first two stages and transformer blocks in the remaining stages while MaxViT [31] stacks convolutional and transformer blocks alternatively throughout the entire network. A formal algorithm requires evaluating all possible C and T configurations, which is computationally expensive. Even neural architecture search leads to exponential search space. Thus, we follow a naive strategy that is based on the following principles:• C before T. In any stage, we prefer convolutions followed by transformer blocks to capture the global dependence between the features aggregated by the convolutions, as they can capture scale and translation-aware information. Our ablations in Appendix Tab. 9 justify this design choice. • Fixed first stage. As transformer blocks have quadratic computation complexity in terms of the sequence length, we prefer the first stage to contain only convolutional blocks to improve the inference latency. • Equal blocks in remaining stages. For ease of analysis, we fix the number of blocks in the remaining stages, i.e., stages 2 to 4, to be four. Once we finalize the basic configuration, we can scale these stages similar to earlier works [31] to achieve larger models. • Asymmetric vs Symmetric. We refer to the architectures as symmetric whenever C and T blocks are distributed equally within a stage. For example, a configuration of CCCC-CCTT-TTTT is symmetric since both C and T blocks are equal within a stage. In contrast, the configuration of CCCT-CCTT-CTTT is asymmetric since C and T blocks are not equal in stages 2 and 4.Final Architecture. Given these design principles, we list various promising configurations for the building blocks (C & T), and analyze their inference throughput and top-1 accuracy. Tab. 2 provides these configurations along with performance and runtime on different GPUs. For a better comparison with existing works, we also add the configurations of CoAtNet and MaxViT for reference. From the results, we can draw the following conclusions:• Compared to symmetric architecture design, asymmetric distribution of C & T blocks yield a better trade-off for throughput and accuracy, as compared by configurations C1-C5 vs C6-C10. • Higher number of transformer blocks in the early stages result in lower throughput, which can be observed by comparing the latency for the configurations of C2 vs C10, and C8 vs C9. • While increasing the number of transformer blocks in the network improves the throughput, it does not result in improved accuracy, as demonstrated by C6 vs C9.Given above analysis, we prefer the C1 configuration for simplicity along with better accuracy vs latency trade-off. Since transformer blocks capture global dependencies, we prefer at least some blocks in the early layers as well in conjunction with the convolutional blocks. Similarly, we prefer having few convolutional blocks in the later stages to capture the spatial, translation, or scale aware features. To be concrete, our final architecture includes a convolutional stem followed by four stages and the classifier head. In the first stage, we only keep convolutional blocks. In the second stage, we keep 75% convolutional and 25% transformer blocks. This trend is reversed in the final stage. For the third stage, we keep equal number of convolutional and transformer blocks. We visualize this configuration in Fig. 2 along with the diagrams representing the convolutional and transformer blocks.Remarks. For simplicity, in this work, we focus on the configuration in which to combine the C & T blocks, and leverage vanilla quadratic attention and convolutional mechanisms. We can incorporate faster alternatives to quadratic attention to further boost the performance and latency trade-offs. We leave this exploration to future research.While many works in the literature focus on improving the trade-off between performance and multiply-add operation counts (MACs), most of the time, MACs do not translate to throughput gains. It is primarily due to the following reasons:• Excessive use of operations that do not contribute to MACs. Such tensor operators include reshape, permute, concatenate, stack, etc. While these operations do not increase MACs, they burden the accelerator with tensor rearrangement. The cost of such rearrangement grows with the size of the feature maps. Thus, whenever these operations occur frequently, the throughput gains drop significantly. For instance, MaxViT [31] uses axial attention that includes many permute operations for window/grid partitioning of the spatial features. Similarly, SMT [79] includes many concatenation and reshape operations in the SMT-Block. It reduces the throughput significantly even though their MACs are lower than AsCAN (see Appendix Tab. 7). • MACs do not account for non-linear accelerator behavior in batched inference. Another issue is that MACs do not account for the non-linear behavior of the GPU accelerators in the presence of larger batch sizes. For instance, with small batch sizes (B=1), the GPU accelerator is not fully utilized. Thus, the benchmark at this batch size is not enough. Instead, one should benchmark at larger batch sizes to see consistency between architectures. • Lack of efficient CUDA operators for specialized building blocks. Many architectures propose specialized and complex attention or convolution building blocks. While these blocks offer better MACs-vs-performance trade-offs, it is likely that their implementation relies on naive CUDA constructs and does not result in significant throughput gains. For instance, Bi-Former [80] computes attention between top-k close regions using a top-k sorting operation and performs many gather operations on the queries and keys. Similarly, RMT [81] computes the Manhattan distance between the tokens in the image. It includes two separate attention along the height and width of the image. This process invokes many small kernels along with reshape and permute operations. These specialized blocks would benefit from efficient CUDA kernels. • Using accelerator-friendly operators. Depending on the hardware, some operators are better than others. Depth-wise separable convolutions reduce the MACs, yet they may not be efficient for [72] . It demonstrates that the design in Fig. 2 provides a better trade-off over accuracy and latency. Symbol M denotes MaxViT block [31] composed of MBConv [73, 75] and Multi-Axial Attention blocks, which is equivalent to CT. Note that CoAtNet [11] uses MBConv blocks compared to the FusedMBConv blocks in our design.particular hardware. Excessive use of depth-wise separable convolutions should be avoided in favor of the full convolutions wherever possible. For instance, MogaNet [82] extensively uses depth-wise convolutions with large kernel sizes and concatenation operations. These operators reduce the multiply-addition counts, which are not necessarily efficient on high-end GPU accelerators.We further scale up our architecture for the image generation task, which requires more computation than the image recognition task. We train our network by utilizing the denoising diffusion probabilistic models (DDPM) [16, 17, 54] in the latent space [13] . The latent diffusion model includes a variational autoencoder (VAE) [83, 84] that encodes the image x into latent z and a diffusion model εθ (•) with parameters θ. We utilize UNet [18] as the network for the diffusion model. For the T2I generation, we get the text embedding with Flan-T5-XXL encoder [85] . We train the diffusion model following the noise prediction [16, 17] :EQUATION)where t is the time step and T is the total number of steps, ϵ is the added Gaussian noise, and c is the condition signal. z t is a noisy latent obtained with a pre-defined variance schedule {β t ∈ (0, 1)} t with the following updates [16] :EQUATIONIn the following, we discuss our design for the diffusion model and the multi-stage training pipeline.We follow the existing literature [13, 8, 15, 86] to design the UNet architecture for the T2I diffusion model. Fig. 2 gives an overview of the overall architecture. It consists of three main stages, namely, Down blocks, Middle Block, and Up blocks. The Up blocks are the reverted reflection of the Down blocks. In addition, there are skip connections to add the features from Down blocks to Up blocks. We adopt the VAE from SDXL [8] to transform the image to the latent space and carry out the diffusion in this space. Since the text-to-image generation task requires additional inputs (i.e., time and text embeddings), we modify the C and T blocks to incorporate these conditions. Similar to the existing literature [8] , we add the time embedding to the C blocks. Additionally, we add the text embedding to the T blocks through a cross-attention operation carried in parallel to the self-attention operation as shown in Fig. 2(c ).While the UNet can take arbitrary resolution as input, we notice that adding positional embeddings in the self-attention operation reduces the artifacts such as duplicate patterns. For this purpose, we add RoPE [87] embeddings for encoding the position information in the T blocks. Further, we incorporate query-key normalization using the RMSNorm [88] for stable training in lower precision (bfloat16).Instead of training the T2I network from scratch, we first train the model on a small-scale dataset and then fine-tune the model on a much larger dataset. It effectively reduces the training cost on the large-scale scale dataset (see ablation in Appendix Sec. A. 6 , Fig 8) . This strategy differs from existing works that perform multi-stage training using different architectures. For instance, PixArt-α [69, 71] trains a class conditional image generation network and modifies it to fine-tune the network for text-to-image generation. Both our pre-training and fine-tuning tasks use the same architecture for text-to-image generation. We use the AdamW optimizer [89] with β 1 = 0.9 and β 2 = 0.99.Specifically, in the first stage, we train our model using ImageNet-1K [33] for the text-to-image generation. Following Esser et al. [90] , we form the conditioned text prompt as "a photo of a <class name>", where class name is randomly chosen from the label of each image. The model is trained to generate an image with a resolution of 256 × 256. In the second stage, we fine-tune the model from the first stage on a much larger dataset. Here we train the model in four phases: First, we conduct training at the resolution of 256 × 256 for 300K iterations with the batch size as 16, 384 and 4e -4 as the learning rate. Second, we continue the training at the resolution of 512 × 512 for 200K iterations with the batch size as 6, 144 and 1e -4 as the learning rate. Third, we train the model for 1024 × 1024 for 100K iterations with the batch size as 1, 536 and 5e -5 as the learning rate. Finally, we perform the multi-aspect ratio training such that the model can synthesize images at various resolution [8] . Additionally, we adjust the added noise at different resolution [15] , i.e., β T in Eq. ( 3) is chosen as 0.01 for 256 × 256 and 0.02 for higher resolution (see ablations in Appendix Sec. A.6).We add an offset noise of 0.05 during multi-aspect ratio training similar to earlier works [8] .some way (e.g., to registered users), it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 In this section, we evaluate the architectures proposed in Sec. 3.1 on image recognition and generation tasks. We also apply this design to the semantic segmentation task in Appendix Sec. A.3. We highlight the important aspects of these experiments below and provide the experimental details in the appendix. While some models regress between two hardware (e.g., MaxViT-S vs SMT-B ), our model consistently achieves better accuracy vs latency trade-offs. We report additional baselines along with multiply-add operations count and different batch sizes in Appendix Tab. 7.We scale the AsCAN architecture discussed in Sec. 3.1 to base and large variants following earlier works [31] . We train these variants on the ImageNet-1K [72] classification task. We provide the experimental details (dataset description, architectures, training hyper-parameters) in Appendix Sec. A.2. Fig. 3 plots the inference speed on a V100 GPU with batch size 16 (measured in images processed per second) and top-1 accuracy achieved on this task for various models. In addition, Appendix Tab. 7 shows the parameter count and inference speed on both V100 and A100 GPUs along with additional details such as floating-point multiply-add count (FLOPs/MACs) and inference speed across different batch sizes. Below, we highlight the salient observations from these experiments.• More than 2× higher throughput across accelerators. Compared to the existing hybrid architectures such as FasterViT [32] and MaxViT [31] , the proposed AsCAN family achieves similar or better top-1 accuracy with more than 2× higher throughput. This trend holds true for both A100 and V100 GPUs. For instance, on A100 with batch=16, FasterViT-1 achieves 83.2% top-1 accuracy with throughput as 1123 images/s, while AsCAN-T has 83.44% with throughput as 3224 images/s. • Better throughput across different batch sizes. AsCAN consistently achieves better throughput across batch sizes for both the accelerators compared to baselines. • Better storage and computational footprint. AsCAN-family of architectures requires less number of parameters and float operations to achieve similar performance. For example, to achieve nearly 85.2% accuracy, MaxViT-L requires 212M parameters and 43.9G MACs whereas AsCAN-L requires 173M parameters and 30.7G MACs. • We achieve better latency vs. accuracy trade-off than hybrid architectures with better alternatives to the attention mechanisms in transformer. For instance, we outperform newer architectures such as PVTv2 [91] , MOAT [92] , EfficientViT [93] , and Scale-Aware Modulation Transformers [79] .Further, we observe similar trends when we scale these models to pre-training on ImageNet-21K [94] dataset (see Appendix Sec. A.2.3) as well as semantic segmentation task (Appendix A.3). We apply our asymmetric UNet architecture to learn class conditional image generation on the ImageNet-1K dataset with 256 × 256 resolution. We train a smaller variant of our asymmetric UNet architecture (as in Sec. 3.3.1) with nearly 400M parameters and inject the class condition through cross-attention mechanism. We train this model using DDPM [16] (more details in Appendix Sec. A.7), and provide results in Tab. 4. As can be seen, we achieve Fréchet Inception Distance (FID) [95] close to state-of-the-art models with nearly half the FLOPs (e.g., Ours vs. DiT-XL/2-G), and better FID than the existing work with similar computation (e.g., Ours vs. U-ViT-L/2).Below, we evaluate our T2I asymmetric UNet architecture trained using the multi-stage training.GenEval Benchmark. We evaluate our model on the GenEval [97] benchmark that studies various aspects of an image generation model. Tab. 3 shows that our model outperforms fast training pipelines such as PixArt-α by a significant margin. It beats even larger models such as SDXL which consume significantly more training data and computational resources. There is a snowy forest with snow-capped trees in the background. We compare our T2I model against generations from different baselines. We illustrate that many times existing models generate images with less photo-realism (either lot less details or more on the cartoonish side), specially for PixArt-α and PixArt-Σ. Further, they frequently miss the fine-grained details explicitly asked in the prompts. We highlight these mistakes in red color in the input prompt. For instance, in the above generations (ordered A → F from top to bottom row), baselines miss details such as, (A) lack of realism (B) light blue jeans, (C) white sunglasses, (D) black, orange, and white feathers, (E) grey scarf & back towards camera, and (F) gray knitted hat with dark blue-brown patterns.Qualitative Comparison. Fig. 4 shows a qualitative comparison between different methods. We highlight salient differences between the baselines and our generations. Baselines such as PixArt-α and PixArt-Σ tend to generate images with much less photo-realism and more often it is much more on the cartoonish side or it has grainy artifacts. Similarly, SDXL with refiner framework is unable to adapt to long text prompts due to limitations of the CLIP text-encoder. Hence, it misses many key features described in the prompts. In contrast, our model is able to follow the long prompts while adhering to the required semantics as well as photo-realism.Human Preference study. We perform a user study to compare open-sourced models to evaluate their image-text alignment characteristics. We select 1000 prompts from our validation set and generate images from SDXL, PixArt-α, PixArt-Σ, and our multi-aspect ratio model. We show these results in the Fig. 5 . It shows that our model convincingly outperforms both SDXL and PixArt-α generations, while it has similar image-text alignment as PixArt-Σ model. To further evaluate these models, we generate 10K images from our validation set (described in Appendix A.5.2) and compute the FID [95] between the generated and original images. We show these scores in Tab. 5, which clearly indicates that our generations align well with the real-world images. This is also evident from our earlier comparison on qualitative visualization in Fig. 4 .Resource Efficiency. We compare the resource requirements of our model against various baselines in Appendix Tab. 12. It shows that we achieve better inference latency compared to many existing models. Further, we consume considerably less compute to achieve much better performance than many existing baselines as illustrated by evaluations in the previous section. 
conclusion: 
 In this work, we design hybrid architectures comprising convolutional and transformer blocks with applications to many computer vision tasks. We focus on developing a simple hybrid model with better throughput and performance trade-offs. Instead of designing efficient alternatives to the convolutional and transformer (mainly attention mechanism) blocks, we leverage existing vanilla attention along with the FusedMBConv block to design the new architecture, called AsCAN. Our main philosophy revolves around the uneven distribution of the convolutional and transformer blocks in the different stages of the network. We refer to this distribution as asymmetric, in the sense that it favors more convolutional blocks in the early stages with a mix of few transformer blocks, while it reverses this trend favoring more transformer blocks in the later stages with fewer convolutional blocks. We demonstrate the superiority of the proposed architecture through extensive evaluations across the image recognition task, class conditional generation, and text-to-image generation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We plan to release our asymmetric architecture implementation for facilitating future research in this direction.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Similar to earlier works on ImageNet-1K and Text-to-Image generations, we do not report any error bars due to high computational resources involved in training one run.Guidelines:• The answer NA means that the paper does not include experiments.The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, we describe the amount of resources required to train our models, including the hardware type. We also provide the total number of iterations/epochs required to reproduce the reported results. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have reviewed the code of ethics and our research work conforms to this code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the societal impacts this work in the appendix Sec. A.9. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Yes, we discuss various safeguards both for preparing training data as well as serving the generated images for our text-to-image generation model. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: For our image recognition and semantic segmentation experiments, we use the widely available ImageNet-1K and ADE20K datasets. We cite these resources in the work. For the T2I experiments, we use first party and licensed datasets. We depict this entire process in Fig. 6 . We start by collating all these data sources to create a stream of images and if available, relevant text captions. Next stage, we add a series of filters to improve the quality of the selected images (resolution, de-duplication, aesthetic score, nsfw filtering, etc.). Since this data most contains very short description or almost non-existent description Question: Does the paper describe potential incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: Our research does not involve research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Image Recognition ArchitectureWe describe the three AsCAN variants (tiny, base, and large) used in the experiments (see Sec. 4.1) in the Tab. 6. We follow earlier works [31] to scale the tiny variant discussed in Sec. 3.1. Note that the stem consists of two convolutional layers that downsample the input to half the spatial resolution.The classifier stage projects the final feature map to the corresponding embedding size and performs adaptive average pooling to reduce the spatial dimensions to 1 × 1, in order to apply a feed-forward layer that acts as classifier head on top of these pooled features. Our convolution blocks use the batch-norm as the normalization layer while the transformer blocks leverage the layer-norm as the normalization layer. We also learn the relative positional embeddings in our attention mechanism similar to earlier works [31, 32, 46] .Table 6 : Architecture Details. We provide detailed configuration of the three variants of the AsCAN proposed in this work, namely, tiny, base, and large. As per our method section, we use FusedMBConv (C) as the convolution block and Vanilla Transformer (T) as the transformer block.We refer the reader to the Fig. 2 for their detailed description and schematic of these building blocks.We use K to denote the number of classes. Note that we hide the activation and normalization layers in the blocks. We use GeLU activation and batch-normalization in the Stem and the classifier stages. We follow the same training strategy for our ImageNet experiments in Sec. 3.1 and Sec. 4.1. We report the top-1 accuracy on the single center crop image. For training ImageNet-1K models with 224 × 224 resolution, we use the AdamW optimizer with a peak learning rate of 3e -3 for 300 epochs. We use a batch size of 4096 images during this training period. We follow a cosine schedule for decaying the learning rate to the minimum learning rate of 5e -6. We also perform a learning rate warm-up to avoid instabilities during the training. We follow a 20 epoch warm-up schedule with an initial learning rate of 5e -7 that gets warmed up to the peak learning rate. For all our experiments, we use standard data augmentation strategies. We use RandAugment [98] with parameters (2, 15), MixUp [99] with α = 0.8, color jittering with 0.4 as the weight, and label smoothing with 0.1 as the smoothing parameter. We also use 0.05 value for the weight decay regularization and perform an exponential model averaging with a decay value of 0.9999. In addition, we adopt gradient clipping with a gradient norm of 1.0 to avoid instabilities during the training of such large models. We also enable stochastic depth for regularization. We use the stochastic depth of 0.3/0.4/0.5 for the three variants in our experiments.We benchmark various existing architectures on two accelerators (NVIDIA A100 and V100 GPUs) with different batch sizes to evaluate the inference speed of a single forward pass. Additionally, we the three AsCANvariants with the training procedure described in earlier sections. We show the full results corresponding to the Fig. 3 in the Tab. 7. We also include inference memory consumed by various recent works in Tab. 8.One of the AsCAN design principles include preferring C blocks over T blocks in a stage. We demonstrated various architecture configurations using this design choice in Tab. 2. For completeness, we reverse this design choice and prefer T blocks over C blocks in a stage. We show these configurations in Tab. 9. Using T block in stem / early layers result in lower throughput. Further, the performance of configurations with T before C yields lower accuracy vs throughput trade-off.We also study the effects of pre-training the AsCAN family on larger datasets such as ImageNet-21K [72] . Similar to earlier works, we pre-train our models on the ImageNet-21K dataset. We use the pre-processed version of this dataset for ease of usage [94] . Following previous works [11, 31] , we train the AsCAN model for 90 epochs on the ImageNet-21K dataset and fine-tune these weights on the ImageNet-1K classification task. Similar to the ImageNet-1K experiments, we use RandAugment [98] with parameters (2, 5), MixUp [99] with α = 0.2, color jittering with 0.4 as the weight and label smoothing with 0.01 as the smoothing parameter. We also use 0.01 value for the weight decay regularization. Additionally, we perform an exponential model averaging with a decay value of 0.9999 and gradient clipping with a gradient norm of 1.0 to avoid instabilities during training such large models. We also enable stochastic depth for regularization. We use the stochastic depth of 0.4/0.5/0.6 for the three variants in our experiments.We report the performance in Tab. 10, where we observe similar top-1 accuracy as the other baselines while achieving much better inference throughput. This trend is similar to the one we observed in Sec. 4.1 when these models trained only on the ImageNet-1K dataset without any pre-training on the ImageNet-21K dataset. Training Procedure & Hyper-parameters. We base our semantic segmentation experiments on the widely used and publicly available mmsegmentation library [105] . We use the UPerNet [106] as our semantic segmentation architecture wherein different hybrid architectures are used as the backbones to extract the spatial feature maps. We extract the spatial feature maps at stages S2, S3, and S4, and forward these feature maps to the semantic segmentation network. All three AsCAN variants (tiny, base, large) have been initialized with the weights pre-trained on the ImageNet-1K task with 224 × 224 resolution. Training is performed in 512 × 512 resolution. Similar to FasterViT [32] , we follow a similar schedule for training these segmentation models with an AdamW optimizer with a learning rate of 1e -4, and a weight decay of 0.05. We use a batch size of 16 on 8 A100 GPUs. We also use stochastic depth similar to ImageNet training for controlling overfitting during these experiments.Experimental Setup. We train UperNet [106] as segmentation architecture on the ADE20K dataset. We use the proposed AsCAN as the backbone pre-trained on the ImageNet-1K dataset. We use the AdamW optimizer [89] with learning rate 1e -4 and weight decay 0.05. Following earlier works [32, 74] , we train the models for 160K iterations using the mmsegmentation library [105] . We train these networks on 8 NVIDIA A100 GPUs using 16 as the batch size. We compute the various inference statistics with 512 × 512 as the image resolution.Experimental Results. We compare the performance of various backbones in Tab. 11. It shows that the proposed backbone achieves competitive mIoU while achieving nearly 1.5× faster inference latency compared to the baselines, measured using the frames per second metrics. This fact can be observed across different scaling of the backbones. For instance, the Swin-T backbone achieves latency of 44FPS while AsCAN-T achieves 64FPS as the latency with similar mIoU. Thus, our models achieve a similar performance-vs.-latency trend the downstream semantic segmentation task as the classification benchmark.We curate our training dataset with careful filtration on diverse data sources using first-party and licensed datasets. We depict this entire process in Fig. 6 . We start by collating all these data sources to create a stream of images and if available, relevant text captions. Next stage, we add a series of filters to improve the quality of the selected images (resolution, de-duplication, aesthetic score, nsfw filtering, etc.). Since this data mostly contains very short descriptions or almost non-existent descriptions of the image, we unify the captioning process using an image captioning model. We collect a subset of 200K data for human annotations, where we ask the annotators to add details such as the <angle shot of the image>, <image background information>, <human attributes>, etc. We use this labeled data to fine-tune a BLIP2 [107] model which is used to generate long captions similar to this format. We compare the resource usage of various existing methods with our proposed T2I model. Tab. 12 shows the parameter count of the model, training compute cost, and the inference cost per image generation.We evaluate our models against various baselines on two internally captioned datasets (Set-A-10K and Set-B-10K). We draw these datasets from our validation set. We compute the FID between the generated 10K images against the reference images in Tab. 5. This table also shows that fast training pipelines like PixArt-α and PixArt-Σ suffer considerably when evaluated on real-world images compared to models like SDXL and Ours. We also point out that SDXL has been trained with a considerably larger dataset compared to ours and uses much more computational resources.Since full-scale text-to-image generation task is a computationally expensive task, it becomes crucial that we choose appropriate hyper-parameters and initialization of the model before training it on the entire dataset. Thus, we choose a proxy task that can be run on a small scale and guide us in the suitable training configurations.ImageNet-1K T2I Generation Task. We resort to the ImageNet-1K dataset since it contains a variety of objects (1000 objects) in different settings. Earlier works such as Pixart-α, train their transformer architecture on the ImageNet-1K class-conditional generation tasks, wherein they incorporate the text condition via a cross-attention mechanism. This process requires the model to forget the class condition and adapt to the text condition. In this work, along the lines of [90] , we simply convert the ImageNet-1K classification dataset to Text-to-Image dataset by adding the text corresponding to the <class name>, with the template "a photo of a <class name>". Specifically, the dataset provides various different names corresponding to each class, we pick up one at random during training to learn diverse text mappings. We evaluate the models by computing the FID between the 50K generated images and the reference images from the dataset.Embedding Pre-Computation. Our T2I model performs diffusion in the latent space. To keep parity between this task and the full training, we also pre-compute the SDXL VAE embeddings of the ImageNet-1K dataset. Similarly, we pre-compute Flan-T5-XXL embeddings for the text condition. This saves training resources since loading VAE and Flan-T5-XXL encoders consumes significant GPU memory as well as non-trivial computation time. Thus, by pre-computing these embeddings, we increase the batch size available for training the UNet model. Architecture Details. We use similar configuration for the C blocks as in the image recognition architecture. All our convolutional operators use 3 × 3 kernel sizes. The number of output channels in the three Down blocks are {320, 640, 1280}. Since the three Up blocks are reflections of the Down blocks, their output channels are in the reverse order, i.e., {1280, 640, 320}. Given that we use the SDXL VAE to convert the 3-channel RGB-image into 4-channel latent space, our number of input channels is 4 for the convolution operator in the first Down block. The number of attention heads in the three T blocks in the three Down stages are {5, 10, 20}. Since our T blocks are applied after C blocks, the number of output channels of the convolutional blocks along with the number of attention heads, determine the attention dimension for the transformer block. Finally, we compute the time-step and textual embeddings. The time-step integer values are converted to timestep embedding by first converting it into sinusoidal space and then projecting it through two linear layers. We process the textual embeddings using two T blocks for adapting the frozen embeddings (dimension 4096) for T2I generation with just one head and the same dimension as the frozen embeddings.Noise Level Ablation for Different Resolutions. Since our training strategy involves jumping resolutions from 256 → 512 → 1024, we ablate on the noise levels at each of these resolutions. We use the DDPM noise scheduler for injecting the noise and the level of noise is controlled by the β T hyper-parameter. We try four different noise levels {0.01, 0.02, 0.03, 0.04}, and compute the FID scores. We plot these scores in Fig. 7 . We find that at resolution 256, noise level 0.01 produces the best FID score, and at remaining resolutions, noise level 0.02 works the best. We learn a class conditional image generation on the ImageNet-1K dataset with image resolution 256 × 256. We train a smaller variant of our asymmetric UNet architecture (as in Sec. 3.3.1) with nearly 400M parameters and inject the class condition through the cross-attention mechanism. We train this model using the DDPM scaled linear noise schedule for 1000 time steps. The training is conducted for 1000 epochs with a batch size of 2048. We use AdamW optimizer with 6e -4 as the learning rate, 0.01 as the weight decay, and (β 1 , β 2 ) = (0.9, 0.99). We use an 8-channel VAE to convert the input image space into a latent space for latent diffusion. We compare our results with other class conditional models in Tab. 4. Our asymmetric architecture for this task is similar to the T2I UNet architecture described earlier. To lower down the compute footprint, we reduce the number of channels to {160, 320, 640} and reduce the cross-attention dimension from 4096 to 768. We evaluate our model with two configurations and report these two FID scores. In the first configuration, we apply a classifier-free guidance (cfg) scale [96] of 1.78 to generate the class-conditioned images. In the second configuration (sampled cfg), we apply the guidance in the steps [5, 30] with a cfg scale in the increasing range [1.1, 3.6] . This sampling procedure is similar to the one proposed in MUSE [23] .We use the Heun [19] discrete scheduler for inference with 30 sampling steps.We develop hybrid architectures with asymmetric distribution of the convolution and transformer blocks. We show that this design paradigm is generic and applicable to many vision tasks such as image recognition, semantic segmentation, and text-to-image generation. We expect this architecture design to be widely useful for other applications. From the architecture design perspective, our building blocks are not rigid and can be replaced with other efficient alternatives of the chosen C and T blocks. Currently, we focus on the vanilla attention mechanism, but we can certainly incorporate faster alternatives to quadratic attention without sacrificing any efficiency advantages. Since we demonstrate an application of the asymmetric design to the UNet framework, we list out some limitations of our text-to-image generation model, which is broadly similar to various existing text-to-image models.• Extra limbs. While our model is able to generate limbs for humans and animals very well in most cases. There are instances where we can observe extra limbs in generations. It is unclear if this is due to a poorly learned diffusion process, poor captioning guidance, lack of granular information in image-caption pairs, or just an artifact of the VAE encoder while converting to the latent space. • Artifacts in extreme resolutions. We train our T2I model with multi-aspect ratio bucketing. While it is able to adapt to many different resolutions in a graceful manner, we find that the model generates duplicate patterns in the extremes of the aspect ratio (significantly higher width than height or vice-versa). We believe this is due to the lack of training data in such resolutions. • While we are much better at avoiding NSFW generations compared to baselines (due to careful curation of the training data), there might be a slight percentage of NSFW content, due to leakage in various filtration processes in the dataset curation. Thus, we incorporate an additional NFSW filter on top of the generation to remove such edge cases.We design the asymmetric architecture and show the advantages it brings in many applications (image recognition, semantic segmentation, and text-to-image generation). Our architecture design targets a better trade-off between efficiency (training/inference latency) while achieving achieving good performance. In itself, the asymmetric design would reduce the amount of resources utilized by deep neural networks used commonly in vision tasks. The downstream applications of this architecture, especially in Text-to-Image generation need to be deployed with care. We expect the community to utilize proper care w.r.t. training data as well as the generated output. There needs to be special checks in place while preparing the training data for T2I tasks. We specially add various filters to remove malicious and harmful content from the dataset. Further, even generated images need to be processed with some filtration to remove similar harmful and malicious content.A.10 Text-to-Image Generation Results.For visualization purposes, we generate more images with different prompts using our multi-aspect ratio model. We present these images in Fig. 9 , 10, and 11. It shows that our model is able to generate a wide range of images, both very realistic as well as stylistic images. In addition, it is able to grasp various concepts and compositions.There is a log next to the water. 7: Results on ImageNet-1K [72] for Classification Task. We compare the performance of our asymmetric architectures, AsCAN, against baselines. We report the inference latency as the throughput (images per second) that is measured by inferring images with batch size as B in half-precision, i.e., fp16, on an A100 GPU using torch-compile and benchmark utility from timm library [100] . A similar procedure (without torch-compile) is followed to obtain the throughput on the V100 GPU. 11: Semantic Segmentation Results on ADE20K [104] . We compare different backbones on the ADE20K [104] semantic segmentation benchmark with UPerNet [106] as the detection architecture. Computational and storage statistics are computed using the input resolution of 512 × 512. A thunderstorm in a mysterious forest Figure 11 : Example text-to-image generation results.NeurIPS Paper Checklist 


Decision: After the initially review of the paper reviewers pointed merits of the paper but also raised concern. Major concerned from reviewers were addressed during the rebuttal. I tend to recommend this paper for acceptance, but invite the authors to incorporate all the reviewers feedback in their submission.

Reviews: review: paper_summary: paper_summary: This paper presents a hybrid neural network architecture that incorporates both convolution-based and vision transformer (ViT)-based building blocks for discriminative and generative modeling. The proposed convolutional block, labeled as (C), is identical to the EfficientNetV2's FusedMBConv, where a standard 3x3 convolution with a 4x dimension expansion and an extra squeeze excitation module (SE) is within, followed by a 1x1 convolution where GELU replaces SiLU. The transformer-like block, termed vanilla Transformer (T), processes the vanilla self-attention and MLP blocks in parallel, which is in contrast to the original sequential Transformer block. For ImageNet experiments, the authors follow the standard protocol to evaluate the proposed models with the ImageNet-1k top-1 accuracies, where the proposed model is empirically adjusted in the order of C and T blocks to optimize the trade-off between ImageNet-1K top-1 accuracies and computational budgets, including throughput and the number of parameters. The resulting optimal model is like asymmetric architecture, wherein the number of convolution and transformer blocks is asymmetric in different stages. Additionally, the architecture is applied to UNet-based diffusion models (i.e., DDPM), structured similarly to the ImageNet architecture, to assess the effectiveness of the proposed design. The generative model is evaluated by the FID score on generating 256x256 ImageNet-1K 2 images using class conditional generation. The optimal architecture for this turned out to be asymmetric across the UNet stages as well.
strength_weakness: strength_weakness: - The paper lacks intuition and clear reasoning for the architectural design choices. No explanation or key insights are provided on why the proposed design or employed modules perform better than other alternatives. 
  - It is unclear why the S1 stage should consist solely of two convolution layers. It seems that the traditional stem (having the stride 4) is separated into two stages (i.e., S0, S1). Are there any other reasons for this design choice?
  - The authors claimed that the searched architecture is asymmetric, but it is unclear why an asymmetric combination of convolution and transformer blocks would perform better. The rationale behind this architectural choice needs further clarification. 
  - While the chosen FusedMBConv and Vanilla Transformer in Table 1 appear promising, it is unclear why these options outperform others. 
  - The rationale for why the generative model achieves a better FID score using modules optimized for the recognition task is not clearly explained.

- The positioning of self-attention modules after convolution-based layers (in a stage or across the entire network), as seen in the architectural choices, has already been explored in previous studies [1, 2]. Therefore, the architectural results presented in Table 2 can be considered expected outcomes based on the known knowledge. The authors are encouraged to provide new insights or takeaways.

- There is no held-out set for searching architectural choices. It appears that the proposed architecture was optimized over the ImageNet validation set for the best performance, but a more appropriate setup would involve using a separate held-out set.

- The results from pre-training on ImageNet-21K are not as impressive as those from ImageNet-1K. This reviewer suspects that the reason may be that the architecture is highly fine-tuned to the ImageNet-1K dataset.

- The ADE20K Semantic Segmentation results are not fairly compared and missing numbers for recent architectures; the competing methods enjoy smaller computational costs (e.g., for Swin) to the proposed method than the proposed models; and there are only Swin and Faster-ViT architectures for comparison.

- The proposed Transformer architecture (a parallel processing architecture of self-attention and MLP)  is referred to as the "vanilla" block, but it is known that a typical "vanilla" Transformer processes self-attention and MLP sequentially. This terminology used by the authors could lead to confusion. Furthermore, the parallel architecture was proposed in [a] and employed in many works like [b]:
  - [a] MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning, arxiv 2020
  - [b] Simplifying Transformer Blocks, ICLR 2024

- It would be beneficial to provide memory consumption during training and inference for the proposed architectures.

- Comparing the proposed method with some recent works would reveal its effectiveness more clearly. The authors are encouraged to compare performance trade-offs with the following recent works [3, 4, 5, 6].

[1] MobileNetV4 - Universal Models for the Mobile Ecosystem, arxiv 2024

[2] Scale-Aware Modulation Meet Transformer, ICCV 2023

[3] BiFormer: Vision Transformer with Bi-Level Routing Attention, CVPR 2023

[4] MogaNet: Multi-order Gated Aggregation Network, ICLR 2024

[5] RMT: Retentive Networks Meet Vision Transformers, CVPR 2024

[6] DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs, ECCV 2024
questions: questions: - See the weaknesses
- Can the authors give the performance tradeoff like Fig. 3 on an A100 GPU?
limitations: limitations: Limitations are provided.
paper_summary: paper_summary: The authors present AsCAN, a hybrid architecture that combines convolutional and transformer blocks, that can applied to both visual recognition and generation. This architecture features an asymmetric distribution, with more convolutional blocks in early stages and more transformer blocks in later stages. It demonstrates favorable results in large-scale text-to-image tasks, outperforming recent models in both public and commercial domains, and provides better throughput.
strength_weakness: strength_weakness: [Unclear explanation]
- Please provide clearer and more intuitive explanations of how the proposal can achieve higher throughput while maintaining or improving accuracy. 

[Model/Data/Resolution Scalability]
- Please demonstrate how the proposal can scale up to larger recognition models (XL/XXL/huge regime), as it currently only shows tiny to large models in the appendix.

- Please illustrate how the proposal can scale up to larger generation models. Specifically, when FLOPs are matched similarly to the FLOPs shown in Table 3 (100-150G), can we achieve better FID scores?

- Please illustrate how the proposal is scalable w.r.t the data size. The appendix Table 11 provides only the snapshot on 450M images.

- Please illustrate how the proposal is scalable for higher resolution image processing. For example, does the current proposal perform better in both recognition and generation compared to other architectural designs when the input resolution increases (224 -> 512 -> 1k, etc.)? Especially focus on generation task.

[Minor] 
- The terminology (e.g., equal blocks in remaining stages, asymmetric vs. symmetric) can be improved for better clarity.

- The table formats (e.g., Table 1 and Table 2 heights are different) can be improved.
questions: questions: Several design choices are not thoroughly evaluated.
- Why is C placed before T? Do you have quantitative results on placing T before C?
- Why is the first stage fixed? We can introduce (adaptive) pooling to incorporate T in the early stage and compare it with pure convolutions.
limitations: limitations: Macro-level architectural ablation experiments are mostly performed on the classification task due to resource limitations. This fundamentally limits the generalizability of the conclusions to other tasks. We cannot draw any strong conclusions derived from classification experiments for the generation task.
paper_summary: paper_summary: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
In this paper, AsCAN combines both convolutional and transformer blocks. The authors revisit the key design principles of hybrid architectures and propose a simple and effective asymmetric architecture, where the distribution of convolutional and transformer blocks is asymmetric, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages.

This paper contains tremendous experimental results to show its efficiency. However, it has several weak points.
strength_weakness: strength_weakness: 1. The main contribution is to combine convolutional and transformer blocks to obtain the better performance with reduced latency. 
However, its theoretical analysis is not shown enough. The asymmetric architecture is also used to generate images. Although the intensive experiments seem to make the contents more fluent, intensive analysis in terms of theoretical points is not enough. 

2. The marginal performance enhancements are given in this paper. For example, Figure 3 shows the proposed architecture achieved the best optimization. However, I think that the enhancements were very small. More perspective analysis and expectation are required. 

3. In the image generation part, the proposed model is used as a backbone. Is there any internal analysis such as weight distribution and training characteristics?
questions: questions: Please, see the section of Weakness.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The authors propose a principled way to design hybrid architectures for a variety of tasks, including image classification, semantic segmentation, class-conditional generation, and text-to-image generation. The goal is the resulting models, called Asymmetric Convolution-Attention Networks (AsCAN), to have the following characteristics, (1) to offer favorable performance-throughput trade-offs compared to existing SOTA models, (2) to be efficient across various modern hardware accelerators, and (3) to scale efficiently in terms of both compute and the amount of training data. 

As a hybrid architecture, AsCAN consists of a sequence of convolutional and transformer blocks. The reason the authors opt for this kind of architecture, is because they aim to combine the advantages of both convolutional layers, e.g., translation equivariance, and transformer layers, e.g., global dependencies.

The authors experiment with different designs on the task of image classification by using ImageNet-1K, and then they generalize their findings to image generation. First, they experiment with different existing convolutional (C) and transformer (T) blocks, concluding to FusedMBConv for C, and vanilla attention for T, based on their accuracy vs throughput trade-off on different GPUs. Then, they experiment with different distributions of C and T blocks in a classification architecture. They always use a convolutional stem, four processing stages with multiple blocks, and a classification head. They conclude in an asymmetric design, meaning that they use more C blocks in the early stages, and more T blocks in the latter stages. They follow the same principles to design the U-Net backbone of a latent diffusion model for image generation tasks. All models are scaled by adding more blocks at each stage, maintaining the asymmetric distribution between C and T blocks. To efficiently scale training to large datasets, the authors propose a multi-stage training regime, where a model is first trained on a smaller dataset, and then, in subsequent stages, is fine-tuned for fewer iterations on the larger dataset.

The authors first experiment on ImageNet-1K, showing that AsCAN models offer a better accuracy-throughput trade-off compared to SOTA baselines. The benefits in throughput are demonstrated with different GPUs and batch sizes. Then, they show that their AsCAN diffusion model performs on par with SOTA models on ImageNet-1K class conditional generation, but with considerably less FLOPs. Similarly, AsCAN demonstrates similar performance to the baselines on semantic segmentation on ADE20K, but with higher FPS. Finally, the authors train a 2.4B params AsCAN latent diffusion model on an internal dataset of 450M images, for the task of text-to-image generation. They show that their model outperforms the baselines on most aspects of the GenEval benchmark, and either outperforms or is on par with the baselines on image-text alignment, based on a human study.
strength_weakness: strength_weakness: Originality:

1. Hybrid architectures exist already in the literature, and AsCAN are built out of pre-existing components, like FusedMBConv, with known benefits, so, this limits the novelty of the approach.

Quality:

1. The authors constraint their design to always have convolutional (C) blocks before Transformer (T) blocks within a stage. However, in Table 2, which includes the comparisons for the macro design of the architecture, MaxViT, which achieves the best accuracy, alternates C and T blocks. In addition, the “C before T” constraint breaks between stages, when a stage ends with T and the subsequent stage starts with C. So, it is not clear to me why this constraint is set.

2. I think the authors should provide actual throughput for the class conditional generation in Table 3. If I am not mistaken, this is the only experiment that doesn’t include this metric. The authors provide FLOPs in Table 3, however, as can be seen in other results, FLOPs don’t always translate to actual timings.

Clarity:

1. Ln 23-24, “CNNs encode many desirable properties like translation invariance”, I think it should be “translation equivariance”.

2. Ln 160, the authors mention “C1 vs C10”, but I think a more direct comparison is C2 vs C10.

3. Ln 161, “While increasing the number of transformer blocks in the network improves the throughput”, I think Table 2 shows that increasing transformer blocks hurts the throughput, e.g., C6 has higher throughput on A100s compared to C9.

4. In Appendix A.3, I think the “Experimental Setup” part in Ln 639-644 repeats for the most part information already provided in the “Training Procedure & Hyper-parameters” part (Ln 628-638), so, the two parts can be merged.

5. Some minor typos, Ln 658, “most contains”, I think should be “mostly”; Ln 607 “the full results corresponding to the Fig. 3 in the Tab. 7”, I think the “the” before “Fig.” and “Tab.” are not necessary; Ln 681, “larger much”, I think “much” is not needed; Ln 733, “for this task similar”, I think there should be “is” after “task”; Ln 770, “achieving” is repeated twice.

Significance:

1. One of the main contributions of this work is the favorable performance-throughput trade-offs offered by the proposed models. Specifically, in many experiments, AsCAN achieves similar performance compared to baselines, but manage to do it with higher throughput, so, I think the efficiency of the models makes them standout. However, I think it is not clear enough from the discussion of the method or the experiments, what are the causes of this efficiency. For example, in Section 3.1 (Ln 159-160), the authors point out that increasing T blocks in early stages decreases throughput (C8 vs C9 in Table 2), however, comparing C6, C7 and C8, which have an increasing number of T blocks earlier on, we see that C7 has slightly lower throughput compared to C6 on A100s and a bit higher on V100s, while C8 has considerably higher throughput compared to both C6 and C7 on V100, and on A100 for batch size 64, and lower for batch size 16 on A100. Also, C9, which has the most T blocks, compared to C6, which has T blocks only in the final stage, has slightly higher throughput on V100 and considerably lower on A100. So, it is not clear what exactly causes the variation in throughput, and to what extent is a matter of the hardware, e.g., would the same results persist on H100s?
In addition, the baselines are very diverse, not only hybrid architectures, so, it becomes even harder to interpret the differences in behavior. For example, in Table 7, AsCAN-L has similar accuracy with EfficientNetV2-M, while AsCAN-L has more than $\times 3$ parameters, and about $\times 1.3$ MACs. At the same time, EfficientNetV2-M is optimized to take advantage of FusedMBConv, but still, AsCAN-L has almost $\times 1.5$ throughput on A100 with batch size 64, and about $\times 1.3$ on V100 with batch size 16. Similarly, in Table 11, AsCAN with 2.4B parameters has higher throughput compared to PixArt-$\alpha$, which has 0.6B params. I think the significance of the paper would be higher if the reported efficiency benefits were analyzed in more detail.
questions: questions: 1. In NeurIPS checklist 15, the authors answer NA, however, in Section 4.3 (Ln 276-284) and Fig. 5, the authors report a user study with human subjects, doesn’t this study require approval for research with human subjects?

2. In Ln 763, Section A.8, it is mentioned “we are much better at avoiding NSFW generations compared to baselines (due to careful curation of the training data)”, how is this measured in the actual output of the models?
limitations: limitations: The authors discuss limitations and societal impact in Sections A.8 and A.9 respectively. One thing that in my opinion could be added in the limitations, is that AsCAN sometimes require considerable more parameters to achieve favorable performance-throughput trade-offs, leading to higher memory footprint. For example, I think this is the case in Table 7 between AsCAN-L and EfficientNetV2-M, as I mentioned in a previous section.
paper_summary: paper_summary: The paper performs a thorough empirical analysis of design choices made in neural networks designed for computer vision tasks. Based on reasonable assumptions, they first narrow down the design space of the building block of such networks to combinations of Fused MBConv and vanilla transformer layers followed by an empirical analysis of the chosen combinations in terms of classification accuracy, latency (on GPUs) and model size. 

The top-performing combinations exhibit significantly better latency on GPUs compared to similar accuracy models optimized for efficiency, such as Swin Transformers and MaxViT. Their model demonstrates the effectiveness of a simple convolution + vanilla transformer model as compared to several efficient/hybrid attention models.

Insights gained from this design are also used to design a similar model for Text-image generation (asymmetric mix of conv + transformer), which also gives a better latency-performance (FID, human preference)  trade -off than existing models (Pixart alpha, SDXL).
strength_weakness: strength_weakness: The contribution of the difference in training pipeline versus that of the architecture for improving image generation performance is unclear. If an empirical ablation study is infeasible for computational reasons, a more detailed discussion in Section 3.2.2 would significantly enhance clarity.
questions: questions: See weaknesses
limitations: limitations: The authors have addressed limitations

Comments: title: Official Comment by Reviewer R6Bu
Official Comment by Reviewer R6Bu
Thank you for your reply
Response to Reviewer Comments
comment: Dear Reviewer hpXB, 

Thank you for reading our rebuttal and providing a positive rating. We sincerely appreciate your time and efforts. We would like to use this opportunity to clarify some remarks below.

**Stem Design and merging stage S1 and S0.** The stem design in our base architecture is similar to many widely used architectures such as MaxViT, CoAtNet, EfficientFormer, etc. Further, we would like to humbly mention that the argument about merging stage S1 and S0 is invalid. Since S0 is the stem and it only consists of convolution layers, while stage S1 consists of C blocks which in our case are FusedMBConv blocks. 

**Rationale for the architectural choice.** The discriminative backbone produces class labels (e.g., the ImageNet class) at the end of the last layer, while the generative backbone (UNet) produces a synthesized latent (which is later decoded into an image) with the same size as the input sampled noise. Thus, they both are fundamentally different. UNet architecture consists of three stages: down, middle and up. Since our search experiments have been based off of discriminative backbone, it is easier to mimic the down and up stages with the asymmetric distribution. The middle stage design is borrowed from the standard UNet architectures. Besides, we have used our model for semantic segmentation task as well, showing that the asymmetric design is applicable to a wider range of tasks.

**Performance-throughput trade-offs are simply due to GPU-friendly operations.** We respectfully beg to differ with the reviewer. We are not simply using the GPU-friendly operations. We also improve the latency-performance trade-off by designing new architecture. Please kindly notice that, given the amount of GPU-friendly operations, there exists tons of configurations in which these can be configured together to yield a new architecture. Our asymmetric design choices yields architectures with much better performance-vs-throughput trade-offs (as seen in Table 2 in main text and Table 2 in rebuttal pdf). In these ablations, while we can reduce the model size of the convolution dominated architectures, this would also result in lower performance compared to the current status. 

We hope the above response could help address the concern of the reviewer. Please let us know if the reviewer has other questions and we would be very happy to help answer.

Best Regards,
Authors
Sorry for the very late reply to the responses. I greatly appreciate the detailed responses to my concerns, but my concerns still remain:

>Existing hybrid architectures such as MaxViT, FasterViT, CoAtNet, etc., follow a symmetric design in which C and T blocks are uniformly distributed across stages or within a stage. In contrast, AsCAN architectures recommend an asymmetric distribution with a preference for more convolutional blocks in the early stages and more transformer blocks in the later stages.

>Asymmetric architectures outperform many existing models that utilize specialized attention and convolutional operators. These works design models based on parameter count and FLOPs, but it typically does not translate into inference throughput gains. Some of these issues come from operators that do not contribute to parameter count and FLOPs but require non-trivial runtime, such as reshape, permute, etc. Others originate from the lack of efficient CUDA operators for these specialized attention and convolutional operators.

The above authors' responses still do not give me a rationale for the architectural choice: I am still curious about why the asymmetric distribution of convolutions and transformers works well. Even though performance improvements have been observed, I feel that the design concept might not be a universal option beyond ImageNet. Additionally, the CTTTTC block in the right middle of the generative model design (in Fig. 2) suggests that the authors' claim may vary depending on the specific scenario. I believe a NeurIPS paper should provide rationale/insights into the suggested method, enabling readers to apply it to other tasks or domains. More evidence of the transferability or universality of the discriminative architecture is certainly needed. I understand that time constraints may hinder additional experiments in this discussion round, but these could be considered in future revisions.

Another concern is that the authors claim higher acc vs. throughput trade-offs of a proposed architecture as a key contribution. However, from my perspective, the higher throughput largely depends on the use of FusedMBConv and the parallel Transformer architecture, as demonstrated in Table 1. Therefore, this may not be a unique contribution of this work, as it relies on existing GPU-friendly operations. Furthermore, Table 2 does not appear to be a controlled experiment in terms of the computational budget, which is the number of parameters. I believe that constraining the number of parameters would better highlight the trade-offs for "some" convolution-dominated counterparts, as these networks in the table seem to "typically" have more parameters to compute, which could put them at a disadvantage. I believe that more controlled experiments might lead to different conclusions, so the contribution of finding an asymmetric architecture in Table 2 is somewhat diluted.

I also noticed that Stage 1 (S1) could be merged into the stem (S0), which leads to the final number of stages to three and deviates from the standard four-stage design. Are there any particular intuitions behind this architectural choice? (I recall seeing something similar designs elsewhere, but I can't remember which work it was. and there were no clear explanations for it there either)

Finally, the authors made great efforts to address my concerns, and the contribution to generative modeling is noteworthy, so I am increasing my rating to 5. The authors are encouraged to reflect all the reviewer's concerns and the responses in to the final revision.
Dear Reviewer R6Bu,

Thank you for your feedback and positive rating! We are glad to know that our responses address all of your concerns. We will include the discussed analysis in the final manuscript.

Thanks & Regards,
Authors
Thank you very much for the provided discussion, I updated my rating assuming that this analysis will be part of the final manuscript.
Dear Reviewer jBHn, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing today, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
Dear Reviewer hpXB, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing today, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
Dear Reviewer 3CqY, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing today, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
References mentioned in above discussion
- MaxViT: Multi-Axis Vision Transformer https://arxiv.org/abs/2204.01697
- Scale-Aware Modulation Meet Transformer https://arxiv.org/abs/2307.08579 
- BiFormer: Vision Transformer with Bi-Level Routing Attention https://arxiv.org/pdf/2303.08810 
- RMT: Retentive Networks Meet Vision Transformers https://arxiv.org/pdf/2309.11523 
- MogaNet: Multi-Order Gated Aggregation Network https://arxiv.org/pdf/2211.03295 
- MobileNetV4 — Universal Models for the Mobile Ecosystem https://arxiv.org/abs/2404.10518
Below are the primary reasons MACs do not translate to the throughput gains. 
- **Excessive use of operations that do not contribute to MACs**. Tensor operators such as reshape, permute, concatenate, stack, etc., are examples of such operations. While these operations do not increase MACs, they burden the accelerator with tensor rearrangement. The cost of such rearrangement grows with the size of the feature maps. Thus, whenever these operations occur frequently, the throughput gains drop significantly. For instance, 
    - MaxViT uses axial attention that includes many permute operations for window/grid partitioning of the spatial features. See Table 7 for a throughput comparison between MaxViT and AsCAN. Also, Table 1 shows that Multi-Axial attention yields significantly lower throughput when compared to vanilla transformer block.  
    - Similarly, the “Scale-Aware Modulation Meets Transformer“ (SMT-S, SMT-B) architecture includes many concatenation and reshape operations in the SMT-Block. It reduces the throughput significantly even though their MACs are lower than AsCAN (see Table 3 in the rebuttal pdf).

- **MACs do not account for non-linear accelerator behavior in batched inference.**  Another issue is that MACs do not account for the non-linear behavior of the GPU accelerators in the presence of larger batch sizes. For instance, with small batch sizes (B=1), the GPU accelerator is not fully utilized. Thus, the benchmark at this batch size is not enough. Instead, one should benchmark at larger batch sizes to see consistency between architectures. 

- **Lack of efficient CUDA operators for specialized building blocks.** Many new architectures propose specialized and complex attention or convolution building blocks. While these blocks offer new perspectives and better MACs-vs-performance trade-offs, it is likely that their implementation relies on naive CUDA constructs and does not result in significant throughput gains. For instance, 
    - Bi-Former architecture introduces Bi-Level Routing Attention (BRA), which computes regional queries and keys and constructs a directed dependency graph. It computes attention between top-k close regions. Their implementation (see Algorithm 1) uses a top-k sorting operation and performs many gather operations on the queries and keys. We believe such an implementation would benefit from writing custom efficient CUDA kernels. 
    - RMT (Retentive Networks Meet Vision Transformers) architecture extends the notion of temporal decay in the spatial domain. It computes the Manhattan distance between the tokens in the image. It includes two separate attention along the height and width of the image. This process invokes many small kernels along with reshape and permute operations. 

- **Use accelerator-friendly operators.** Depending on the hardware, some operators are better than others. Depth-wise separable convolutions reduce the MACs, but they may not be necessarily efficient for your particular hardware. Excessive use of depth-wise separable convolutions should be avoided in favor of the full convolutions wherever possible. For instance, 
    - MogaNet extensively uses depth-wise convolutions with large kernel sizes along with concatenation operations. These operators reduce the multiply-addition counts, but these are not necessarily efficient on high-end GPU accelerators. Similarly, MaxViT uses MBConv as the convolutional block. 
    - Even on mobile devices, recently proposed MobileNetV4 architectures include full convolutions in the early layers to fully utilize the mobile accelerators.

We design AsCAN  keeping above points for reference. Namely,
- We use highly efficient building blocks. FusedMBConv provides much higher throughput than MBConv blocks and invokes a full convolution followed by a point-wise projection. Both of these operations are highly efficient on a high-end GPU accelerator. Similarly, we rely on the vanilla transformer block with efficient native CUDA implementation (PyTorch has FlashAttention operators). Our ablations in Table 1 in the main text demonstrate the effectiveness of these blocks compared to other complex building blocks.
- We do not excessively use operations that do not contribute to MACs, such as window partitioning (permute, reshape) in MaxViT or concatenation operations in SMT. We further do not alternate C and T blocks within a stage to avoid multiple reshape operations between these blocks.  
- Further, compared to pure convolutional blocks such as EfficientNetV2, we heavily use transformer blocks in the later stages. The transformer blocks capture global dependencies between the features computed by convolutional layers in the early stages.  It yields better performance.

We believe the above discussion should help clarify why MACs do not translate directly to throughput/latency gains on the high-end GPU accelerators and why our design choices in AsCAN help us achieve better throughput gains than existing architectures.
Dear Reviewer R6Bu,

Thanks for your feedback and your willingness to further read our improved main text. We will include the following discussion to the revised paper.
Thank you very much for your willingness to address my comments. I will read the changes in the main text as soon as they become available, and update my review accordingly.
Dear Reviewer jBHn, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing very soon, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
Dear Reviewer 3CqY, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing very soon, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
Dear Reviewer hpXB, 

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing very soon, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,
Authors
Dear Reviewer R6Bu,

Thank you for your quick response and positive rating! In the main text, we will include a detailed discussion as to why FLOPs do not translate to throughput (latency) gains and pinpoint the causes of inefficiencies of current SOTA designs. We will also incorporate further analysis of various configurations in Table 2 to break down the impact of C and T block arrangement. 

Best Regards,
Authors
I would like to thank the authors for their detailed reply. I would also like to acknowledge their effort to address comments with additional experiments in the rebuttal pdf. I find the arguments with respect to novelty and quality convincing, however, my main concern is about the significance, which relates to getting a deeper understanding about the causes of the better performance-throughput trade-off AsCAN provide across tasks. For example, about EfficientNetV2-M, the impact of higher input resolution in computation is captured by MACs, which are still less than those of AsCAN, so, it seems to me that it would be important to have a clear discussion about the reasons that MACs/FLOPs don't translate to throughput. The authors already provide some justification in their rebuttal, by mentioning the impact of operations like reshape and permute, or the lack of efficient CUDA operators for specialized operations. If the authors expand such observations into a clear discussion where they pinpoint the causes of inefficiencies of current SOTA designs, I think the impact of the work will significantly increase, because it would allow members of the community to make more informed decisions about their designs, without the need to make numerous ablations.

About hardware, I agree that focusing solely on FLOPs is not sufficient, and throughput should be a major consideration as well. Given that, in my review, I gave a number of examples where the "C before T" design gives conflicting throughput outcomes in Table 2. So, similar to my previous point, I think AsCAN provide a valuable contribution through their favorable performance-throughput trade-off, but the contribution would be higher if it was more thoroughly discussed what are the causes of the observed behavior.

In light of such a discussion/analysis, I would be happy to increase my rating.
Dear Reviewer 3CqY, 

Thank you very much for your valuable and constructive feedback. We have included detailed explanations and additional experiments in response to your questions. As the deadline for the discussion period approaches, we would appreciate your review of these explanations to confirm that they fully meet your expectations and resolve any remaining concerns. 

Thank you once again for your insightful review.

Best regards,
Authors
Dear Reviewer hpXB, 

Thank you very much for your valuable and constructive feedback. We have included detailed explanations and additional experiments in response to your questions. As the deadline for the discussion period approaches, we would appreciate your review of these explanations to confirm that they fully meet your expectations and resolve any remaining concerns. 

Thank you once again for your insightful review.

Best regards,
Authors
Dear Reviewer R6Bu, 

Thank you very much for your valuable feedback and the positive evaluation of our work. We have included detailed explanations in response to your questions. As the deadline for the discussion period approaches, we would appreciate your review of these explanations to confirm that they resolve any remaining concerns. Let us know if you have any other questions.

Thank you once again for your insightful review.

Best regards,
Authors
We thank the reviewer for reading the rebuttal and responding promptly. We would like to clarify some remarks below.

- We appreciate that the reviewer acknowledges we achieved significantly reduced inference time. In all our experiments, we analyze the trade-off between performance (top-1 accuracy / FID score) and latency (throughput measure in images processed per second) achieved by the architectures. We would like to humbly clarify that we do not claim that we significantly improve performance. Instead, we claim to achieve significantly better performance *vs.* latency trade-offs. This can be observed by our empirical evaluations (see Table 7 in the main text, Figure 1, Table 1, and 3 in the rebuttal pdf), where we obtain significantly better throughput to achieve similar performance as other models. Vice-versa, we achieve better performance at the same throughput (see Figure 1 in rebuttal pdf). Thanks for the comments from the reviewer. We will further improve the writing of this paper to make it more clear about the claims of this paper.

- Since we focus on designing architectures suitable for GPUs, we provide throughput numbers for different batch sizes (B=1, 16, 64). At this large scale, batched inference (with B>1) makes a lot more sense since lower batch sizes (*e.g.*, B=1) do not utilize the GPU memory fully and end up returning a very non-linear behavior. Even at B=1, we still achieve much better performance *vs.* throughput trade-off than many baselines.


We hope the above response could help address the concern of the reviewer. Please let us know if the reviewer has other questions and we would be very happy to help answer.
Thanks for your rebuttal. In Tables 1 and 2, your architectures had significantly reduced the inference time on GPUs, having marginal performance enhancements. I understand the contributions in terms of reduced computations in GPUs. The ratio of enhanced throughput with batch=1 seems to be small compared to the case with batch= 16, and 64 in Table 7. I think that the reason of the throughput enhancements on GPUs could be explained in a structural point of view.