Title: AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-Image Generation

Hybrid Architectures
Abstract: Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN---a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, without performing any optimization of inference time our model shows faster execution, even when compared to works that do such optimization, highlighting the advantages and the value of our approach.


Peer reviews:
comments: SUMMARY:
The paper performs a thorough empirical analysis of design choices made in neural networks designed for computer vision tasks. Based on reasonable assumptions, they first narrow down the design space of the building block of such networks to combinations of Fused MBConv and vanilla transformer layers followed by an empirical analysis of the chosen combinations in terms of classification accuracy, latency (on GPUs) and model size. 

The top-performing combinations exhibit significantly better latency on GPUs compared to similar accuracy models optimized for efficiency, such as Swin Transformers and MaxViT. Their model demonstrates the effectiveness of a simple convolution + vanilla transformer model as compared to several efficient/hybrid attention models.

Insights gained from this design are also used to design a similar model for Text-image generation (asymmetric mix of conv + transformer), which also gives a better latency-performance (FID, human preference)  trade -off than existing models (Pixart alpha, SDXL).

STRENGTHS:
1. The paper is well-motivated, analyzing the benefits of a simple neural network architecture across a variety of tasks (recognition, segmentation, generation) on current hardware. The insights derived are of broad interest to a wide range of researchers.

2. The paper provides a thorough empirical analysis of design choices in image recognition experiments.

WEAKNESSES:
The contribution of the difference in training pipeline versus that of the architecture for improving image generation performance is unclear. If an empirical ablation study is infeasible for computational reasons, a more detailed discussion in Section 3.2.2 would significantly enhance clarity.

QUESTIONS:
See weaknesses

LIMITATIONS:
The authors have addressed limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a principled way to design hybrid architectures for a variety of tasks, including image classification, semantic segmentation, class-conditional generation, and text-to-image generation. The goal is the resulting models, called Asymmetric Convolution-Attention Networks (AsCAN), to have the following characteristics, (1) to offer favorable performance-throughput trade-offs compared to existing SOTA models, (2) to be efficient across various modern hardware accelerators, and (3) to scale efficiently in terms of both compute and the amount of training data. \n\nAs a hybrid architecture, AsCAN consists of a sequence of convolutional and transformer blocks. The reason the authors opt for this kind of architecture, is because they aim to combine the advantages of both convolutional layers, e.g., translation equivariance, and transformer layers, e.g., global dependencies.\n\nThe authors experiment with different designs on the task of image classification by using ImageNet-1K, and then they generalize their findings to image generation. First, they experiment with different existing convolutional (C) and transformer (T) blocks, concluding to FusedMBConv for C, and vanilla attention for T, based on their accuracy vs throughput trade-off on different GPUs. Then, they experiment with different distributions of C and T blocks in a classification architecture. They always use a convolutional stem, four processing stages with multiple blocks, and a classification head. They conclude in an asymmetric design, meaning that they use more C blocks in the early stages, and more T blocks in the latter stages. They follow the same principles to design the U-Net backbone of a latent diffusion model for image generation tasks. All models are scaled by adding more blocks at each stage, maintaining the asymmetric distribution between C and T blocks. To efficiently scale training to large datasets, the authors propose a multi-stage training regime, where a model is first trained on a smaller dataset, and then, in subsequent stages, is fine-tuned for fewer iterations on the larger dataset.\n\nThe authors first experiment on ImageNet-1K, showing that AsCAN models offer a better accuracy-throughput trade-off compared to SOTA baselines. The benefits in throughput are demonstrated with different GPUs and batch sizes. Then, they show that their AsCAN diffusion model performs on par with SOTA models on ImageNet-1K class conditional generation, but with considerably less FLOPs. Similarly, AsCAN demonstrates similar performance to the baselines on semantic segmentation on ADE20K, but with higher FPS. Finally, the authors train a 2.4B params AsCAN latent diffusion model on an internal dataset of 450M images, for the task of text-to-image generation. They show that their model outperforms the baselines on most aspects of the GenEval benchmark, and either outperforms or is on par with the baselines on image-text alignment, based on a human study.\n\nSTRENGTHS:\nQuality:\n\n1. The authors develop the proposed design principles in a structured way, ablating different options, and offering adequate metrics, which include accuracy, number of parameters, and throughput on 2 GPUs and different batch sizes.\n\n2. Similarly, the experiments are well-structured, using multiple baselines and metrics. For example, on image classification the authors compare against SOTA CNNs, Transformers and hybrid architectures of different sizes, and they provide accuracy, actual throughput and number of parameters. Importantly, the experiments support the main claims of the work.\n\n3. The setup of all experiments is described in detail in the Appendix to aid reproduction.\n\nClarity:\n\n1. The manuscript is very well written, and easy to follow. The authors explain their method, and provide clear Figures and Tables, with appropriate captions.\n\nSignificance:\n\n1. The proposed models show clear benefits in terms of their performance-throughput trade-offs in different tasks, thus, they contribute to the community.\n\n2. The authors provide specific design principles that they used to build AsCAN, and they could be useful to other domains as well.\n\nWEAKNESSES:\nOriginality:\n\n1. Hybrid architectures exist already in the literature, and AsCAN are built out of pre-existing components, like FusedMBConv, with known benefits, so, this limits the novelty of the approach.\n\nQuality:\n\n1. The authors constraint their design to always have convolutional (C) blocks before Transformer (T) blocks within a stage. However, in Table 2, which includes the comparisons for the macro design of the architecture, MaxViT, which achieves the best accuracy, alternates C and T blocks. In addition, the “C before T” constraint breaks between stages, when a stage ends with T and the subsequent stage starts with C. So, it is not clear to me why this constraint is set.\n\n2. I think the authors should provide actual throughput for the class conditional generation in Table 3. If I am not mistaken, this is the only experiment that doesn’t include this metric. The authors provide FLOPs in Table 3, however, as can be seen in other results, FLOPs don’t always translate to actual timings.\n\nClarity:\n\n1. Ln 23-24, “CNNs encode many desirable properties like translation invariance”, I think it should be “translation equivariance”.\n\n2. Ln 160, the authors mention “C1 vs C10”, but I think a more direct comparison is C2 vs C10.\n\n3. Ln 161, “While increasing the number of transformer blocks in the network improves the throughput”, I think Table 2 shows that increasing transformer blocks hurts the throughput, e.g., C6 has higher throughput on A100s compared to C9.\n\n4. In Appendix A.3, I think the “Experimental Setup” part in Ln 639-644 repeats for the most part information already provided in the “Training Procedure & Hyper-parameters” part (Ln 628-638), so, the two parts can be merged.\n\n5. Some minor typos, Ln 658, “most contains”, I think should be “mostly”; Ln 607 “the full results corresponding to the Fig. 3 in the Tab. 7”, I think the “the” before “Fig.” and “Tab.” are not necessary; Ln 681, “larger much”, I think “much” is not needed; Ln 733, “for this task similar”, I think there should be “is” after “task”; Ln 770, “achieving” is repeated twice.\n\nSignificance:\n\n1. One of the main contributions of this work is the favorable performance-throughput trade-offs offered by the proposed models. Specifically, in many experiments, AsCAN achieves similar performance compared to baselines, but manage to do it with higher throughput, so, I think the efficiency of the models makes them standout. However, I think it is not clear enough from the discussion of the method or the experiments, what are the causes of this efficiency. For example, in Section 3.1 (Ln 159-160), the authors point out that increasing T blocks in early stages decreases throughput (C8 vs C9 in Table 2), however, comparing C6, C7 and C8, which have an increasing number of T blocks earlier on, we see that C7 has slightly lower throughput compared to C6 on A100s and a bit higher on V100s, while C8 has considerably higher throughput compared to both C6 and C7 on V100, and on A100 for batch size 64, and lower for batch size 16 on A100. Also, C9, which has the most T blocks, compared to C6, which has T blocks only in the final stage, has slightly higher throughput on V100 and considerably lower on A100. So, it is not clear what exactly causes the variation in throughput, and to what extent is a matter of the hardware, e.g., would the same results persist on H100s?\nIn addition, the baselines are very diverse, not only hybrid architectures, so, it becomes even harder to interpret the differences in behavior. For example, in Table 7, AsCAN-L has similar accuracy with EfficientNetV2-M, while AsCAN-L has more than $\\times 3$ parameters, and about $\\times 1.3$ MACs. At the same time, EfficientNetV2-M is optimized to take advantage of FusedMBConv, but still, AsCAN-L has almost $\\times 1.5$ throughput on A100 with batch size 64, and about $\\times 1.3$ on V100 with batch size 16. Similarly, in Table 11, AsCAN with 2.4B parameters has higher throughput compared to PixArt-$\\alpha$, which has 0.6B params. I think the significance of the paper would be higher if the reported efficiency benefits were analyzed in more detail.\n\nQUESTIONS:\n1. In NeurIPS checklist 15, the authors answer NA, however, in Section 4.3 (Ln 276-284) and Fig. 5, the authors report a user study with human subjects, doesn’t this study require approval for research with human subjects?\n\n2. In Ln 763, Section A.8, it is mentioned “we are much better at avoiding NSFW generations compared to baselines (due to careful curation of the training data)”, how is this measured in the actual output of the models?\n\nLIMITATIONS:\nThe authors discuss limitations and societal impact in Sections A.8 and A.9 respectively. One thing that in my opinion could be added in the limitations, is that AsCAN sometimes require considerable more parameters to achieve favorable performance-throughput trade-offs, leading to higher memory footprint. For example, I think this is the case in Table 7 between AsCAN-L and EfficientNetV2-M, as I mentioned in a previous section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}

comments: SUMMARY:
Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
In this paper, AsCAN combines both convolutional and transformer blocks. The authors revisit the key design principles of hybrid architectures and propose a simple and effective asymmetric architecture, where the distribution of convolutional and transformer blocks is asymmetric, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages.

This paper contains tremendous experimental results to show its efficiency. However, it has several weak points.

STRENGTHS:
This paper contains tremendous experimental results to show its efficiency.

WEAKNESSES:
1. The main contribution is to combine convolutional and transformer blocks to obtain the better performance with reduced latency. 
However, its theoretical analysis is not shown enough. The asymmetric architecture is also used to generate images. Although the intensive experiments seem to make the contents more fluent, intensive analysis in terms of theoretical points is not enough. 

2. The marginal performance enhancements are given in this paper. For example, Figure 3 shows the proposed architecture achieved the best optimization. However, I think that the enhancements were very small. More perspective analysis and expectation are required. 

3. In the image generation part, the proposed model is used as a backbone. Is there any internal analysis such as weight distribution and training characteristics?

QUESTIONS:
Please, see the section of Weakness.

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present AsCAN, a hybrid architecture that combines convolutional and transformer blocks, that can applied to both visual recognition and generation. This architecture features an asymmetric distribution, with more convolutional blocks in early stages and more transformer blocks in later stages. It demonstrates favorable results in large-scale text-to-image tasks, outperforming recent models in both public and commercial domains, and provides better throughput.\n\nSTRENGTHS:\n[Intuitive Design]The authors leverage existing vanilla attention along with the FusedMBConv block to design the new architecture, called AsCAN. Their main philosophy revolves around the asymmetric distribution of the convolutional and transformer blocks in the different stages of the network; more convolutional blocks in the early stages with a mix of few transformer blocks, while it reverses this trend favoring more transformer blocks in the later stages with fewer convolutional blocks. \n\n[Extensive Experiments] The proposal is thoroughly evaluated in both classification and generation tasks, achieving state-of-the-art results in both.\n\n[Scalability] The authors show that the proposal scales well in the regime of Tiny-Large recognition models.\n\nWEAKNESSES:\n[Unclear explanation]\n- Please provide clearer and more intuitive explanations of how the proposal can achieve higher throughput while maintaining or improving accuracy. \n\n[Model/Data/Resolution Scalability]\n- Please demonstrate how the proposal can scale up to larger recognition models (XL/XXL/huge regime), as it currently only shows tiny to large models in the appendix.\n\n- Please illustrate how the proposal can scale up to larger generation models. Specifically, when FLOPs are matched similarly to the FLOPs shown in Table 3 (100-150G), can we achieve better FID scores?\n\n- Please illustrate how the proposal is scalable w.r.t the data size. The appendix Table 11 provides only the snapshot on 450M images.\n\n- Please illustrate how the proposal is scalable for higher resolution image processing. For example, does the current proposal perform better in both recognition and generation compared to other architectural designs when the input resolution increases (224 -> 512 -> 1k, etc.)? Especially focus on generation task.\n\n[Minor] \n- The terminology (e.g., equal blocks in remaining stages, asymmetric vs. symmetric) can be improved for better clarity.\n\n- The table formats (e.g., Table 1 and Table 2 heights are different) can be improved.\n\nQUESTIONS:\nSeveral design choices are not thoroughly evaluated.\n- Why is C placed before T? Do you have quantitative results on placing T before C?\n- Why is the first stage fixed? We can introduce (adaptive) pooling to incorporate T in the early stage and compare it with pure convolutions.\n\nLIMITATIONS:\nMacro-level architectural ablation experiments are mostly performed on the classification task due to resource limitations. This fundamentally limits the generalizability of the conclusions to other tasks. We cannot draw any strong conclusions derived from classification experiments for the generation task.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a hybrid neural network architecture that incorporates both convolution-based and vision transformer (ViT)-based building blocks for discriminative and generative modeling. The proposed convolutional block, labeled as (C), is identical to the EfficientNetV2's FusedMBConv, where a standard 3x3 convolution with a 4x dimension expansion and an extra squeeze excitation module (SE) is within, followed by a 1x1 convolution where GELU replaces SiLU. The transformer-like block, termed vanilla Transformer (T), processes the vanilla self-attention and MLP blocks in parallel, which is in contrast to the original sequential Transformer block. For ImageNet experiments, the authors follow the standard protocol to evaluate the proposed models with the ImageNet-1k top-1 accuracies, where the proposed model is empirically adjusted in the order of C and T blocks to optimize the trade-off between ImageNet-1K top-1 accuracies and computational budgets, including throughput and the number of parameters. The resulting optimal model is like asymmetric architecture, wherein the number of convolution and transformer blocks is asymmetric in different stages. Additionally, the architecture is applied to UNet-based diffusion models (i.e., DDPM), structured similarly to the ImageNet architecture, to assess the effectiveness of the proposed design. The generative model is evaluated by the FID score on generating 256x256 ImageNet-1K 2 images using class conditional generation. The optimal architecture for this turned out to be asymmetric across the UNet stages as well.\n\nSTRENGTHS:\n+ This paper is easy to follow and well-written.\n+ Extensive experimental results are provided, proving the claim in both discriminative and generative modeling.\n+ The performance on the ImageNet-1K classification looks impressive compared with some recent models.\n\nWEAKNESSES:\n- The paper lacks intuition and clear reasoning for the architectural design choices. No explanation or key insights are provided on why the proposed design or employed modules perform better than other alternatives. \n  - It is unclear why the S1 stage should consist solely of two convolution layers. It seems that the traditional stem (having the stride 4) is separated into two stages (i.e., S0, S1). Are there any other reasons for this design choice?\n  - The authors claimed that the searched architecture is asymmetric, but it is unclear why an asymmetric combination of convolution and transformer blocks would perform better. The rationale behind this architectural choice needs further clarification. \n  - While the chosen FusedMBConv and Vanilla Transformer in Table 1 appear promising, it is unclear why these options outperform others. \n  - The rationale for why the generative model achieves a better FID score using modules optimized for the recognition task is not clearly explained.\n\n- The positioning of self-attention modules after convolution-based layers (in a stage or across the entire network), as seen in the architectural choices, has already been explored in previous studies [1, 2]. Therefore, the architectural results presented in Table 2 can be considered expected outcomes based on the known knowledge. The authors are encouraged to provide new insights or takeaways.\n\n- There is no held-out set for searching architectural choices. It appears that the proposed architecture was optimized over the ImageNet validation set for the best performance, but a more appropriate setup would involve using a separate held-out set.\n\n- The results from pre-training on ImageNet-21K are not as impressive as those from ImageNet-1K. This reviewer suspects that the reason may be that the architecture is highly fine-tuned to the ImageNet-1K dataset.\n\n- The ADE20K Semantic Segmentation results are not fairly compared and missing numbers for recent architectures; the competing methods enjoy smaller computational costs (e.g., for Swin) to the proposed method than the proposed models; and there are only Swin and Faster-ViT architectures for comparison.\n\n- The proposed Transformer architecture (a parallel processing architecture of self-attention and MLP)  is referred to as the \"vanilla\" block, but it is known that a typical \"vanilla\" Transformer processes self-attention and MLP sequentially. This terminology used by the authors could lead to confusion. Furthermore, the parallel architecture was proposed in [a] and employed in many works like [b]:\n  - [a] MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning, arxiv 2020\n  - [b] Simplifying Transformer Blocks, ICLR 2024\n\n- It would be beneficial to provide memory consumption during training and inference for the proposed architectures.\n\n- Comparing the proposed method with some recent works would reveal its effectiveness more clearly. The authors are encouraged to compare performance trade-offs with the following recent works [3, 4, 5, 6].\n\n[1] MobileNetV4 - Universal Models for the Mobile Ecosystem, arxiv 2024\n\n[2] Scale-Aware Modulation Meet Transformer, ICCV 2023\n\n[3] BiFormer: Vision Transformer with Bi-Level Routing Attention, CVPR 2023\n\n[4] MogaNet: Multi-order Gated Aggregation Network, ICLR 2024\n\n[5] RMT: Retentive Networks Meet Vision Transformers, CVPR 2024\n\n[6] DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs, ECCV 2024\n\nQUESTIONS:\n- See the weaknesses\n- Can the authors give the performance tradeoff like Fig. 3 on an A100 GPU?\n\nLIMITATIONS:\nLimitations are provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-24"}