Title: Revisiting the Integration of Convolution and Attention for Vision Backbone
Venue: NeurIPS 2024
Accepted: yes
Keywords: convolution

attention

vision backbone
Abstract: Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus 
enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{https://github.com/rayleizhu/GLMix}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the scalability issue in vision transformers by integrating convolutions (Convs) and multi-head self-attentions (MHSAs) at different granularity levels, rather than at the finest pixel level. The authors propose using Convs for fine-grained per-pixel feature extraction and MHSAs for coarse-grained semantic slots in parallel. They introduce a pair of fully differentiable soft clustering and dispatching modules to bridge the grid and set representations, enabling effective local-global fusion. The proposed integration scheme, named GLMix, offloads fine-grained feature extraction to light-weight Convs and utilizes MHSAs in a limited number of semantic slots. Through extensive experiments, they demonstrate the efficiency and effectiveness of their approach, showing improved performance and interpretability in various vision tasks. The paper also highlights the potential of their method to induce better semantic segmentation with weak supervision.\n\nSTRENGTHS:\n1. Innovative Integration Approach: The idea of integrating Convs and MHSAs at different granularity levels is innovative and addresses the scalability issues inherent in vision transformers.\n\n2. Efficient Local-Global Fusion: The use of soft clustering and dispatching modules enables efficient local-global feature fusion, which is a significant advancement.\n\n3. Extensive Empirical Validation: The method achieves state-of-the-art results on ImageNet-1k, COCO, and ADE20K benchmarks, showing a favorable performance-computation trade-off. Furthermore，the semantic grouping effect observed in the clustering module enhances the interpretability of the model.\n\nWEAKNESSES:\n1. Complexity of Implementation: The introduction of soft clustering and dispatching modules adds complexity to the implementation, which might pose challenges for practical deployment.\n\n2. Static Number of Semantic Slots: The use of a static number of semantic slots for all images may lead to redundancy and inefficiency in certain cases.\n\n3. Limited Scope of Clustering Strategy: The clustering strategy, though effective, could be further optimized. The current implementation might still be computationally intensive for real-time applications.\n\nQUESTIONS:\n1.How sensitive is the overall performance of the GLMix model to the choice of clustering strategy? Have the authors tested alternative clustering methods, and what were the outcomes?\t\n\n2.What are the potential limitations of using a static number of semantic slots, and have the authors considered dynamic allocation of semantic slots based on the complexity of the input image? \n\n3.How adaptable is the GLMix integration scheme to different types of vision tasks, such as object detection, instance segmentation, and semantic segmentation? Are there any task-specific adjustments needed for optimal performance?\n\n4.What are the scalability limits of the GLMix models when dealing with very high-resolution images or videos?\n\nLIMITATIONS:\n1. Scalability to Higher Resolutions: While the method shows good performance on standard benchmarks, its scalability to very high-resolution images or videos remains to be fully explored.\n\n2. Hardware Inefficiency: The use of depth-wise convolutions, despite their low arithmetic intensity, might still be inefficient on certain hardware, limiting the practical applicability of the model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to leverage the strengths of convolution layers and MHSA block to improve the performance of vision transformers. They propose to apply them in parallel at different granularity, such that the convolution layers are applied to the grid of local features and MHSA to slots for global features. Slots can be considered as clustering of patch features, where patches corresponding to the same object would be associated to a slot. To create slots, they use soft clustering that enables local-global fusion across patch features such that it produces some meaningful semantic grouping effect.\nTo connect the slots back to the patch features, they propose a dispatching module, which is then fused with a convolution block os 1x1 convolution and depth-wise convolution operation (similar to Convnext). They terms this entire block as GLMix for Global-local mixing block. \n\nFinally, to validate the effectiveness of their approach, the authors propose steps to adapt it to Swin-tiny-layout module and empirically show the performance for image classification on Imagenet-1k, semantic segmentation on ADE20K, object detection and instance segmentation on MS-COCO. The authors provide a comprehensive list of baseline methods (Table 3, Table 4) for different tasks and also provide visualization of the slots attending to different regions of the image. They also discuss different experimental settings and provide additional visualizations of the slots in the Appendix.\n\nSTRENGTHS:\nThe following are the strengths of the paper:\n- The authors propose an idea to integrate the strengths of convolution i.e. ability to encode inductive bias, alog with MHSA block with ability to learn global representation, to improve vision transformers. The idea is quite interesting and while the use of convolution and transformer layers have been applied in Hybrid architectures, their use in parallel is under-studied and is an exciting topic.\n- The visualizations provided in Figure 5 of the object level representations learned by the slots and additional such visualizations in the Appendix, gives a nice insight into representations encoded by slots. Its also interesting to see the slots learning different regions of the image, not only focusing on the object, but also regions in the background that might be of interest.\n- I also really liked the through and extensive experiments, along with comparisons to different SoTA methods, not only in terms of performance but also in efficiency.  \n- The paper is also easy to understand, with exhaustive mentions of related works and their comparisons.\n\nWEAKNESSES:\nI have the following concerns about the work:\n\n- In L59, the authors state that they have applied several macro designs from SMT [32]. However, comparing the performance of SMT, SMT-S achieves similar performance as GLNet-4G i.e. 83.7% accuracy, but has much fewer parameters (20.5M vs 27M), while having similar FLOPs. Additionally, the variant SMT-B has 2G fewer flops and half the number of parameters and yet achieves the same performance  i.e 84.3% s. 84.5%. Furthermore, SMT-S is also better than GLNet-4G on object detection on MS-COCO, which using RetinaNet. The authors of SMT also provide visualizations of the attention heads clearly attending to the object, same as in GLNet. This means that the scale-aware-module (SAM) achieves a much more efficient integration of convolution with MHSA. Could the authors please comment on effectiveness of GLNet against SMT.\n- Maybe I missed it, but Im unable to understand the intuition of applying self-attention operation on the slots (L161). Instead of just having a cosine similarity between the slots and patch features as in eq(2), a softmax over the slots would ensure that the slots do not collapse to the same object. Is this the reason why authors apply MHSA on S to obtain S'? \n- Continuing from the above point, I further cannot understand the significance of the dispatching module. Could the authors please explain the intuition behind going from slots to patch features? Is it because, it would be easier to integrate with the features obtained after conv layers? \n- In Figure 4, the clustering before applying the GLMix block and after are exactly the same. Its not clear to me what is being learnt by the GLMix block.\n- A popular work that uses slots to perform object centric learning is DINOSAUR [1*], which show in Figure 6 of their paper that increasing the number of slots results in a drop in performance. The authors however find that varying the slots has almost no effect in performance. Could the authors please comment of this difference in observation? Additionally, by carefully looking in figure 5, we can observe that there are many slots that represent the same object. This leads to some redundancy in learning. Can the authors vary the slots to somewhere between 5-10 like the one performed in [1*] to see if they have the same observation?\n\n[1*] Seitzer et al., Bridging the Gap to Real-World Object-Centric Learning, ICLR 2023\n\nQUESTIONS:\nApart from the ones mentioned in the Weakness section, I have the following questions\n\n- Can the authors please clarify their architecture? I see references to Swin, ConvNext, LV-ViT, etc. But Im unsure about what a single layer of GLNet looks like? Im also curious if the authors use a pretrained backbone, because Im unsure about the performance gains with just 1 V100 GPU and training the slots over just 1 iteration. Maybe Im missing some detail here.\n- Is there any specific reason apart from the smaller feature resolution that the GLMix block is applied to the 3rd layer? \n- There are missing references to hybrid ViTs, which the authors can add and discuss.\n\n[2*] Venkataramanan et al., Skip-Attention: Improving Vision Transformers by Paying Less Attention, ICLR 2024  \n[3*] Pan et al., Less is more: Pay less attention in vision transformers., AAAI 2022  \n[4*] Mehta & Rastegari, Separable self-attention for mobile vision transformers, arxiv  2022  \n[5*] Graham et al.,  Levit: a vision transformer in convnet’s clothing for faster inference, ICCV 2021  \n\n- Could the authors also provide the throughput details for table 3,4 and 5\n\nLIMITATIONS:\nYes the authors have discussed the limitation, but one can reduce the redundancy in slots by trying to empirically understand the performance of the proposed method with fewer slots, as suggested above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the use of Convolutions (Convs) and multi-head self-attentions (MHSAs) in vision backbones. Traditionally considered alternatives, the authors question the need for both to operate at the finest pixel granularity, particularly highlighting the scalability issues this causes in vision transformers. They propose a novel approach where Convs and MHSAs work in parallel but at different granularities: Convs handle local features on a fine-grained grid, while MHSAs manage global features on a coarse-grained set of semantic slots. The integration is facilitated by soft clustering and dispatching modules that bridge these representations, enabling local-global fusion. Their method, GLMix, leverages lightweight Convs for fine details and uses MHSAs on fewer semantic slots.\n\nSTRENGTHS:\n1. The paper re-evaluates current methods of integrating Convs and MHSAs and proposes a novel integration at different granularities. This approach harnesses the advantages of Convs, such as translation equivariance, and MHSAs, such as global interactions and data adaptivity, while mitigating scalability issues related to input resolution.\n2. The paper presents a pair of fully differentiable clustering and dispatching modules that connect the set and grid representations of image features. This enables the fusion of global features from MHSAs and local features from Convs. A notable advantage of the soft clustering module is its ability to produce meaningful semantic groupings without requiring direct dense supervision.\n\nWEAKNESSES:\n1. In Line 6, this paper mentions \"the scalability issue,\" but it only contains experiments on the IN-1K size dataset. Such a dataset cannot evaluate \"scalability.\" I suggest that the authors revise this argument.\n2. The novelty that introduces clustering into attention is limited. [1,2,3,4] have already studied the clustering in attention. Please compare with these methods and highlight your novelty and contributions.\n\n[1] Xie, Y., Zhang, J., Xia, Y., Hengel, A. V. D., & Wu, Q. (2022). Clustr: Exploring efficient self-attention via clustering for vision transformers. arXiv preprint arXiv:2208.13138.\n[2] Liang, J., Cui, Y., Wang, Q., Geng, T., Wang, W., & Liu, D. (2024). Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems, 36.\n[3] Grainger, R., Paniagua, T., Song, X., Cuntoor, N., Lee, M. W., & Wu, T. (2023). PaCa-ViT: learning patch-to-cluster attention in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18568-18578).\n[4] Zeng, W., Jin, S., Liu, W., Qian, C., Luo, P., Ouyang, W., & Wang, X. (2022). Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11101-11111).\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nThis paper will address the limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}