Title: Lightweight Vision Transformer with Bidirectional Interaction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vision Transformer

Lightweight Vision Backbone

Convolution Neural Network
Abstract: Recent advancements in vision backbones have significantly improved their performance by simultaneously modeling images’ local and global contexts. However, the bidirectional interaction between these two contexts has not been well explored and exploited, which is important in the human visual system. This paper proposes a **F**ully **A**daptive **S**elf-**A**ttention (FASA) mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. Specifically, FASA employs self-modulated convolutions to adaptively extract local representation while utilizing self-attention in down-sampled space to extract global representation. Subsequently, it conducts a bidirectional adaptation process between local and global representation to model their interaction. In addition, we introduce a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability. Based on FASA, we develop a family of lightweight vision backbones, **F**ully **A**daptive **T**ransformer (FAT) family. Extensive experiments on multiple vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT accomplishes a **77.6%** accuracy on ImageNet-1K using only **4.5M** parameters and **0.7G** FLOPs, which surpasses the most advanced ConvNets and Transformers with similar model size and computational costs. Moreover, our model exhibits faster speed on modern GPU compared to other models.

Lightweight Vision Transformer with Bidirectional Interaction

Abstract

Recent advancements in vision backbones have significantly improved their performance by simultaneously modeling images' local and global contexts. However, the bidirectional interaction between these two contexts has not been well explored and exploited, which is important in the human visual system. This paper proposes a F ully A daptive S elfA ttention (FASA) mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. Specifically, FASA employs self-modulated convolutions to adaptively extract local representation while utilizing self-attention in down-sampled space to extract global representation. Subsequently, it conducts a bidirectional adaptation process between local and global representation to model their interaction. In addition, we introduce a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability. Based on FASA, we develop a family of lightweight vision backbones, F ully A daptive T ransformer (FAT) family. Extensive experiments on multiple vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT accomplishes a 77.6% accuracy on ImageNet-1K using only 4.5M parameters and 0.7G FLOPs, which surpasses the most advanced ConvNets and Transformers with similar model size and computational costs. Moreover, our model exhibits faster speed on modern GPU compared to other models.

1 Introduction

Vision Transformers (ViTs) have recently garnered significant attention in the computer vision community due to their exceptional ability for long-range modeling and context-aware characteristics. However, because of the quadratic complexity of self-attention in ViT [13], its computational cost is extremely high. As a result, many studies have emerged to improve ViT's computational efficiency and performance in various ways. For instance, some methods restrict tokens that perform selfattention to a specific region and introduce inductive bias to ViT [34; 12; 66; 55]. Further, some methods aim to transform ViT into lightweight backbones with fewer parameters and computational requirements [38; 40; 4; 30; 37], achieving promising results but still not matching the performance of the most advanced ConvNets [51]. How to design an excellent lightweight Vision Transformer remains a challenge.

In current state-of-the-art Vision Transformers, some either excel in creating local feature extraction modules [34; 12; 66] or employing efficient global information aggregation modules [57; 58], while others incorporate both [42; 41]. For instance, LVT [66] unfolds tokens into separate windows and

Modulation

Aggregation

Act

(a) local adaptive aggregation

(c) bidirectional adaptive interaction

(b) global adaptive aggregation

Local to Global:

Nose of a fox.

Local: Nose.

Global: Fox.

Global to Local:

A fox with nose.

Aggeragation

Modulation

Modulation

Modulation

Act

Act

target token

local token

global token

output token

receptive field

applies self-attention within the windows to extract local features, while PVT [57; 58] leverages self-attention with downsampling to extract global features and reduce computational cost. Unlike them, LITv2 [41] relies on window self-attention and spatial reduction attention to capture local and global features, respectively. In terms of the local-global fusion, most methods use simple local-global sequential structures [38; 40; 37; 26; 82], whereas others combine local and global representation with simple linear operations through local-global parallel structures [48; 41; 43]. However, few works have investigated the bidirectional interaction between local and global information. Considering the human visual system where bidirectional local-global interaction plays an important role, these simplistic mixing methods are not fully effective in uncovering the intricate relationship between local and global contexts. 84

In Fig. 1, we illustrate how humans observe an object and notice its body details. Using the example of a fox, we can observe two types of interaction that occur when humans focus on either the fox's nose or the entire animal. In the first type of interaction, known as Local to Global, our understanding of the local feature transforms into the "Nose of a fox." In the second type of interaction, called Global to Local, the way we comprehend the global feature changes to the "A fox with nose." It can be seen that the bidirectional interaction between local and global features plays an essential role in the human visual system. Based on this fact, we propose that a superior visual model should not only extract good local and global features but also possess adequate modeling capabilities for their interaction.

0

1

2

3

4

5

6

FLOPs(G)

74

76

78

80

82

Top-1 Acc(%)

FAT(Ours)

EdgeViT

VAN

MPViT

Shunted

Model

throughput(imgs/s)

Topl-Acc(%)

Year

LVT

1265

74.8

CVPR'2022

MobileViT-XS

1367

74.8

ICLR'2022

EdgeNeXt-XS

EdgeVIT-XS

FAT-B0

1417

1528

1932

75.0

77.5

77.6

ECCVW"2022

ECCV'2022

Ours

MPViT-T

737

78.2

CVPR'2022

MobileViT-S

898

78.4

ICLR'2022

EfficientNet-B1

EdgeNeXt-S

FAT-B1

1395

1452

1243

79.2

79.4

80.1

ECCVW'2022

ICML'2019

Ours

MPViT-XS

597

80.9

CVPR’2022

EdgeViT-S

1049

81.0

ECCV'2022

EfficientNet-B3

VAN-B1

FAT-B2

1064

995

634

81.1

81.6

81.9

ICML2019

Arxiv'2022

Ours

Shunted-S

456

82.9

CVPR*2022

ConvNeXt-S

405

83.1

CVPR'2022

EfficientFormer-L7

FocalNet-S

FAT-B3

368

358

474

83.3

83.4

83.6

NeurIPS′2022

NeurIPS'2022

Ours

In this work, our objective is to model the bidirectional interaction between local and global contexts while also improving them separately. To achieve this goal, we introduce three types of ContextAware Feature Aggregation modules. Specifically, as shown in Fig. 1, we first adaptively aggregate local and global features using context-aware manners to obtain local and global tokens, respectively. Then, we perform point-wise cross-modulation between these two types of tokens to model their

bidirectional interaction. We streamline all three processes into a simple, concise, and straightforward procedure. Since we use context-aware approaches to adaptively model all local, global, and localglobal bidirectional interaction, we name our novel module the F ully A daptive S elfA ttention (FASA). In FASA, we also further utilize a fine-grained downsampling strategy to enhance the self-attention mechanism, which results in its ability to perceive global features with finer granularity. In summary, FASA introduces only a small number of additional parameters and FLOPs, yet it significantly improves the model's performance.

Building upon FASA, we introduce the F ully A daptive T ransformer (FAT) family. The FATs follow the hierarchical design [34; 57] and serve as general-purpose backbones for various computer vision tasks. Through extensive experiments, including image classification, object detection, and semantic segmentation, we validate the performance superiority of the FAT family. Without extra training data or supervision, our FAT-B0 achieves a top-1 accuracy of 77.6% on ImageNet-1K with only 4.5M parameters and 0.7G FLOPs, which is the first model surpasses the most advanced ConvNets with similar model size and computational cost as far as we know. Additionally, as shown in Fig. 2, our FAT-B1, B2, and B3 also achieve state-of-the-art results while maintaining similar model sizes and computational costs.

2 Related Work

Vision Transformer. Since the earliest version of ViT [13] appeared, numerous works have been proposed that focus on enhancing the self-attention mechanism. Many methods restrict self-attention to a subset of tokens, sacrificing global receptive fields to reduce the computation of ViT [34; 12; 66; 61; 1]. Despite having only local perception capabilities, the context-aware nature of selfattention enables these methods to achieve excellent performance. A famous among them is the Swin-Transformer [34], which divides all tokens into windows and performs self-attention within each window, achieving highly competitive performance. In contrast, some methods downsample the K and V in self-attention to preserve the global receptive field while reducing computation by minimizing the number of token pairs involved in the calculation [57; 60; 40; 47; 68; 5]. One such method, PVT [57], leverages large-stride convolution to process K and V , effectively lowering their spatial resolution. In addition to these methods, numerous efforts have been made to introduce ViT into the design of lightweight vision backbones [26; 40; 37; 15; 38; 4]. For example, MobileViT [38] concatenates convolution and self-attention to obtain a powerful, lightweight backbone. Nevertheless, a performance gap exists between lightweight vision transformers and state-of-the-art lightweight CNNs such as NAS-based EfficientNet [51].

Local-Global Fusion. A high-performance vision backbone typically possesses exceptional capabilities for both local and global perception. The capabilities are achieved by either connecting local and global perception modules in a serial manner [58; 31; 10; 26; 40; 55; 1], as demonstrated by DaViT [10], or by simultaneously modeling local and global information within a single module [48; 41; 43], such as the inception transformer [48]. However, current approaches to fusing local and global information are overly simplistic. In serial models, the process of fusing local and global information is not adequately represented. In parallel structures, almost all methods rely on linear modules that depend entirely on trainable parameters to fuse local and global information [41; 48; 43]. These fusing approaches lack the ability to model the interaction between local and global information, which is inconsistent with the human visual system shown in Fig. 1. In contrast, our proposed FASA module models bidirectional interaction between local and global information while separately modeling each one.

Self-Attention with Downsampling. Currently, many models utilize self-attention with downsampling, a technique that was earlier used in PVT and so on to improve the computational efficiency [57; 58; 43; 41; 48; 61; 5]. PVT [57] reduces the spatial resolution of K and V using non-overlapping large stride convolutions, which decreases the number of token pairs involved in self-attention and lowers the computational cost while maintaining a global receptive field. Similarly, PVTv2 [58] uses large stride average pooling to downsample K and V . In contrast, inception transformer [48] employs large stride average pooling to downsample all three Q , K , and V , then upsamples the tokens to their original size after self-attention is applied. However, using excessively large strides or non-overlapping downsampling as in these methods may lead to significant information loss. In FASA, we propose a fine-grained downsampling strategy to alleviate this issue.

Conv Stem

Stage1

FAT Block

×

�1

Stage2

FAT Block

×

�2

Stage3

FAT Block

×

�3

Stage4

FAT Block

×

�4

Classifier

DWConv

LayerNorm

Fine-grained

downsample

MHSA

Conv

1

×

1

Sigmoid

LayerNorm

Conv

1

×

1

DWConv

Conv

1

×

1

GELU

DWConv(s=2)

BN

Conv

1

×

1

BN

DWConv

Sigmoid

Global

Bidirectional Interaction

Local

Fully Adaptive Self-Attention

CPE

ConvFFN

×

�2

Fully Adaptive Transformer (FAT) Block

Sigmoid

DWConv(s=2)

Conv

1

×

1

BN

DWConv(s=2)

BN

Fine-grained downsample

Hadamard Product

Element-wise Add

Only in in-stage block

Only in downsample block

Omitted for simplicity

×

�

×

1

3 Method

3.1 Overall Architecture.

The overall architecture of the Fully Adaptive Transformer (FAT) is illustrated in Fig. 3. To process an input image x ∈ R 3 × H × W , we begin by feeding it into the convolutional stem used in [63]. This produces tokens of size H 4 × W 4 . Following the hierarchical designs seen in previous works [48; 50; 46; 49], we divide FAT into four stages to obtain hierarchical representation. Then we perform average pooling on the feature map containing the richest semantic information. The obtained one-dimensional vector is subsequently classified using a linear classifier for image classification.

A FAT block comprises three key modules: Conditional Positional Encoding (CPE) [6], Fully Adaptive Self-Attention (FASA), and Convolutional Feed-Forward Network (ConvFFN). The complete FAT block is defined by the following equation (Eq. 1):

Initially, the input tensor X in ∈ R C × H × W passes through the CPE to introduce positional information for each token. The subsequent stage employs FASA to extract local and global representation adaptively, while a bidirectional adaptation process enables interaction between these two types of representation. Finally, ConvFFN is applied to enhance local representation further. Two kinds of ConvFFNs are employed in this process. In the in-stage FAT block, ConvFFN's convolutional stride is 1, and ShortCut = Identity . At the intersection of the two stages, the ConvFFN's convolutional stride is set to 2, and ShortCut = DownSample is achieved via a Depth-Wise Convolution (DWConv) [23] with the stride of 2 along with a 1 × 1 convolution. The second type of ConvFFN accomplishes downsampling using only a small number of parameters, avoiding the necessity of patch merging modules between the two stages, thereby saving the parameters.

3.2 Fully Adaptive Self-Attention

In Fig. 1, we present the visual system of humans, which not only captures local and global information but also models their interaction explicitly. Taking inspiration from this, we aim to develop a similar module that can adaptively model local and global information and their interaction. This leads to the proposal of the F ully A daptive S elfA ttention (FASA) module. Our FASA utilizes context-aware manners to model all three types of information adaptively. It comprises three modules:

global adaptive aggregation, local adaptive aggregation, and bidirectional adaptive interaction. Given the input tokens X ∈ R C × H × W , each part of the FASA will be elaborated in detail.

3.2.1 Definition of Context-Aware Feature Aggregation

In order to provide readers with a clear comprehension of FASA, we will begin by defining the various methods of context-aware feature aggregation (CAFA). CAFA is a widely used feature aggregation method. Instead of solely relying on shared trainable parameters, CAFA generates token-specific weights based on the target token and its local or global context. These newly generated weights, along with the associated context of the target token, are then used to modulate the target token during feature aggregation, enabling each token to adapt to its related context. Generally, CAFA consists of two processes: aggregation ( A ) and modulation ( M ). In the following, the target token is denoted by x i , and F represents a non-linear activation function. Based on the order of A and M , CAFA can be classified into various types. Various forms of self-attention [56; 58; 57; 45] can be expressed as Eq. 2. Aggregation over the contexts X is performed after the attention scores between query and key are computed. The attention scores are obtained by modulating the query with the keys, and then applying a Softmax to the resulting values:

In contrast to the approach outlined in Eq. 2, recent state-of-the-art ConvNets [22; 17; 67] utilize a different CAFA technique. Specifically, they employ DWConv to aggregate features, which are then used to modulate the original features. This process can be succinctly described using Eq. 3.

In our FASA module, global adaptive aggregation improves upon the traditional self-attention method with fine-grained downsampling and can be mathematically represented by Eq. 2. Meanwhile, our local adaptive aggregation, which differs slightly from Eq. 3, can be expressed as Eq. 4:

In terms of the bidirectional adaptive interaction process, it can be formulated by Eq. 5. Compared to the previous CAFA approaches, bidirectional adaptive interaction involves two feature aggregation operators ( A 1 and A 2 ) that are modulated with each other:

3.2.2 Global Adaptive Aggregation

The inherently context-aware nature and capacity to model long-distance dependencies of selfattention make it highly suitable for adaptively extracting global representation. As such, we utilize self-attention with downsampling for global representation extraction. In contrast to other models that downsample tokens [57; 58; 60; 48] by using large stride convolutions or pooling operations, we adopt a fine-grained downsampling strategy to minimize loss of global information to the greatest extent possible. In particular, our fine-grained downsampling module is composed of several basic units. Each unit utilizes a DWConv with a kernel size of 5 × 5 and a stride of 2, followed by a 1 × 1 convolution that subtly downsamples both K and V . After that, Q , K , and V are processed through the Multi-Head Self-Attention (MHSA) module. Unlike regular MHSA, we omit the last linear layer. The complete procedure for the global adaptive aggregation process is illustrated in Eq. 6 and Eq. 7. Fisrt, we define our fine-grained downsample strategy and the base unit of it in Eq. 6:

where (n) represents the number of base units that are concatenated and pool denotes our fine-grained downsample operator. Then, the MHSA is conducted as Eq. 7:

where the mathematical symbol ⊗ denotes the operation of matrix multiplication, W 1 is a learnable matrix.

3.2.3 Local Adaptive Aggregation

In contrast to self-attention, convolution possesses an inductive bias that facilitates extracting highfrequency local information. However, convolutional feature aggregation solely relies on parametershared convolutional kernels, resulting in a lack of context-awareness. To address this issue, we utilize a self-modulating convolutional operator that embeds context-awareness into convolution, enabling it to extract local representation adaptively. Specifically, we generate context-aware weights through the Sigmoid and combine them with DWConv to adapively aggregate local information. This process is summarized in Eq. 8, where ⊙ represents the Hadamard product, and Q is directly derived from Eq. 7 for saving parameters of linear projection:

3.2.4 Bidirectional Adaptive Interaction

As illustrated in Fig. 1, when modeling the fox and its nose separately, two types of interactions - "Local to Global" and "Global to Local" - occur between their respective features. Inspired by the human visual system, we design a bidirectional adaptive interaction process that incorporates both types of interactions. To achieve bidirectional interaction, we adopt a method similar to the one described in Sec. 3.2.3 but utilize cross-modulation instead of self-modulation. Specifically, the equation for Local to Global interaction is given by Eq. 9:

Similar to Eq. 9, Global to Local interaction is achieved by Eq. 10:

After completing the two interaction processes, the local and global representation contain information about each other. To merge these representation, we use point-wise multiplication and achieve intermingling among channels with a linear projection, as shown in Eq. 11:

It is worth noting that to get a faster and implementable model, we omit the last high-order variable in Eq. 11 and arrive at a concise expression. More details can be found in appendix. Through the combination of local adaptive aggregation, global adaptive aggregation, and bidirectional adaptive interaction, we present the comprehensive FASA module. As depicted in Fig. 3, our model, which draws inspiration from the human visual system, is both straightforward and simple. It adaptively models local, global, and the interaction between the two in context-aware manners. More discussion about bidirectional adaptive interaction can be found in the appendix.

4 Experiments

We conducted experiments on a wide range of vision tasks, including image classification on ImageNet-1K [9], object detection and instance segmentation on COCO 2017 [33], and semantic segmentation on ADE20K [80]. In addition to them, we also make ablation studies to validate the importance of each component. More details can be found in the appendix.

4.1 Image Classification

Settings. We train our models on ImageNet-1K [9] from scratch. And we follow the same training strategy in DeiT [54] for a fair comparison. Different from DeiT, we employ 20 epochs of linear warm-up, while in DeiT it is 5. The maximum rates of increasing stochastic depth [24] are set to 0.05/0.05/0.1/0.15 for B0/B1/B2/B3, respectively. More details can be found in the appendix.

Results. We compare our FAT against many state-of-the-art models in Tab. 1. The comparison results demonstrate that our FAT consistently outperforms previous models across all settings, with a particularly significant improvement in performance on lightweight models. Specifically, our FAT-B0 achieves 77.6% Top1-accuracy with only 4.5M parameters and 0.7 GFLOPs. Our FAT-B1 also surpasses EdgeNext-S by 0.7% with similar FLOPs. Furthermore, our FAT-B2 model surpasses CrossFormer-T in top1-accuracy by 0.4% while using only half the parameters. Additionally, our FAT-B3 model has also outperformed many state-of-the-art models.

Speed. As shown in Tab. 1, we present the speed comparison results of all models on V100, and our FAT performs better than its counterparts in both speed and performance. Specifically, Our FAT-B0 has a 30% higher throughput than EdgeViT-XS, and a 0.1% higher top-1 accuracy compared to EdgeViT-XS.

4.2 Semantic Segmentation

Settings. We incorporate our pretrained FATs with the Semantic FPN [27]. As in [58; 57], we use 512 × 512 random crops during training and resize the images to have a shorter side of 512 pixels

during inference. Following [57], we fine-tune the models by AdamW with the learning rate of 1 × 10 -4 and batch size of 16. We train them for 80K iterations on the ADE20K training set.

Results. The results can be found in Tab. 2. For B0/B1/B2, the FLOPs are measured with input size of 512 × 512 . While them for B3 are measured with 512 × 2048 . All our models achieve the best performance in all comparisons. Specifically, our FAT-B1 exceeds EdgeViT-XS for +1.5 mIoU. Moreover, our FAT-B3 outperforms the recent Shunted-S for +0.7 mIoU. All the above results demonstrate our model's superiority in dense prediction.

4.3 Object Detection and Instance Segmentation

Settings. We conduct object detection and instance segmentation experiments on the COCO 2017 dataset [33]. Following Swin [34], we use the FAT family as the backbones and take RetinaNet [32] and Mask-RCNN [19] as detection and segmentation heads. We pretrain our backbones on ImageNet1K and fine-tune them on the COCO training set with AdamW [36].

Results. Tab. 3 shows the results with RetinaNet and Mask R-CNN. The results demonstrate that the proposed FAT performs best in all comparisons. For the RetinaNet framework, our FAT-B0 outperforms EdgeViT-XXS by +0.7 AP, while B1/B2/B3 also perform better than their counterparts. As for the Mask R-CNN, our FAT-B3 outperforms the recent Shunted-S by +0.5 box AP and +0.6 mask AP. All the above results tell that our Fully Adaptive Transformer outperforms its counterparts by evident margins. More experiments can be found in appendix.

4.4 Comparison in Efficiency

We compared our approach with the state-of-the-art lightweight vision backbones, as shown in Tab 4. Our method achieved the best trade-off between speed and performance. The CPU is Intel i9Core and the GPU is V100. Inference throughput is measured with batch size 64. Inference latency is measured with batch size 1.

4.5 Larger Model

We scale up our FAT to 50M+ and 85M+ to match the general Vision Transformer backbones. For object detection/instance segmentation, all models use the framework of Mask-RCNN [19] with the 1x schedule. The results are shown in Tab. 5. It can be found that FAT has great scalability.

4.6 Ablation Study and Spectral Analysis

In this section, we conduct experiments to understand FAT better. The training settings are the same as in previous experiments. More experiments can be found in the appendix.

Bidirectional Adaptive Interaction. Initially, we validate the efficacy of bidirectional adaptive interaction by comparing it with three baselines for fusing local and global information: concatenation, addition and element-wise multiplication. As shown in Tab. 6, it can be observed that bidirectional adaptive interaction outperforms all baselines significantly across various tasks. Specifically, compared to the baseline, which uses the cat+linear to fuse the local and global information, our bidirectional adaptive interaction uses fewer parameters and FLOPs but surpasses the baseline by 1.0% in Top1-accuracy. Our bidirectional adaptive interaction also performs better than element-wise multiplication.

Fine-Grained Downsampling. We compared our fine-grained downsampling strategy with three other strategies: non-overlapping large stride pooling, non-overlapping large stride convolution, and overlapping large stride convolution. As shown in Tab. 6, experimental results across various tasks have demonstrated the effectiveness of our fine-grained downsampling strategy. Specifically, our fine-grained downsampling strategy surpasses directly pooling downsampling strategy by 0.4%

Positional Encoding. At the end of Tab. 6, we explore the effect of CPE, and the results in the table show that the flexible CPE also contributes to the performance improvement of the model for 0.2% in image classification.

(a)

(b)

(c)

(d)

Spectral Analysis. As shown in Fig. 4, we conduct a spectral analysis on FASA. Compared to add+linear fusion, our bidirectional adaptive interaction can more fully fuse high-frequency and low-frequency information.

5 Conclusion

In this paper, we propose a new attention mechanism named Fully Adaptive Self-Attention (FASA), which draws inspiration from the human visual system. FASA is designed to model local and global information adaptively while also accounting for their bidirectional interaction using adaptive strategies. Furthermore, we enhance the self-attention mechanism in FASA by incorporating a finegrained downsampling strategy to promote better global perception at a more detailed level. Using the flexible FASA as our foundation, we develop a lightweight vision backbone called Fully Adaptive Transformer (FAT), which can be applied across various vision tasks. Our extensive experiments on diverse asks, such as image classification, object detection, instance segmentation, and semantic segmentation, provide compelling evidence of the effectiveness and superiority of FAT. We expect to apply FAT to other vision-related pursuits, including video prediction and vision-language models.

Acknowledgment

This work is partially funded by National Natural Science Foundation of China (Grant No. 62006228, U21B2045, U20A20223), Youth Innovation Promotion Association CAS (Grant No. 2022132), and Beijing Nova Program (20230484276).

References

Moab Arar, Ariel Shamir, Amit H. Bermano, et al. Learned queries for efficient local attention. In CVPR , 2022.

Chun-Fu (Richard) Chen, Rameswar Panda, and Quanfu Fan. RegionViT: Regional-to-Local Attention for Vision Transformers. In ICLR , 2022.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, et al. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Yinpeng Chen, Xiyang Dai, Dongdong Chen, et al. Mobile-former: Bridging mobilenet and transformer. In CVPR , 2022.

Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurIPS , 2021.

Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR , 2023.

MMSegmentation Contributors. Mmsegmentation, an open source semantic segmentation toolbox, 2020.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, et al. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW , 2020.

Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Mingyu Ding, Bin Xiao, Noel Codella, et al. Davit: Dual attention vision transformers. In ECCV , 2022.

Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. 2022.

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, et al. Xcit: Cross-covariance image transformers. In NeurIPS , 2021.

Li Gao, Dong Nie, Bo Li, and Xiaofeng Ren. Doubly-fused vit: Fuse information from vision transformer doubly with local representation. In ECCV , 2022.

Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. CVPR , 2022.

Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.09741 , 2022.

Kai Han, An Xiao, Enhua Wu, et al. Transformer in transformer. In NeurIPS , 2021.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. In ICCV , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition. In CVPR , 2016.

Byeongho Heo, Sangdoo Yun, Dongyoon Han, et al. Rethinking spatial dimensions of vision transformers. In ICCV , 2021.

Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. arXiv preprint arXiv:2211.11943 , 2022.

Andrew Howard, Mark Sandler, Grace Chu, et al. Searching for mobilenetv3. In ICCV , 2019.

Gao Huang, Yu Sun, and Zhuang Liu. Deep networks with stochastic depth. In ECCV , 2016.

Huaibo Huang, Xiaoqiang Zhou, and Ran He. Orthogonal transformer: An efficient vision transformer backbone with token orthogonalization. In NeurIPS , 2022.

Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Lightvit: Towards light-weight convolution-free vision transformers. arXiv preprint arXiv:2207.05557 , 2022.

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR , 2019.

Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In CVPR , 2022.

Kunchang Li, Yali Wang, Peng Gao, et al. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In ICLR , 2022.

Yanyu Li, Geng Yuan, Yang Wen, et al. Efficientformer: Vision transformers at mobilenet speed. In NeurIPS , 2022.

Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 , 2021.

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, and Kaiming He andPiotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, et al. Microsoft coco: Common objects in context. In ECCV , 2014.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, et al. A convnet for the 2020s. In CVPR , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, and Fahad Shahbaz Khan. Edgenext: Efficiently amalgamated cnn-transformer architecture for mobile vision applications. In CADL , 2022.

Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. In ICLR , 2022.

Juhong Min, Yucheng Zhao, Chong Luo, et al. Peripheral vision transformer. In NeurIPS , 2022.

Junting Pan, Adrian Bulat, Fuwen Tan, et al. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In ECCV , 2022.

Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In NeurIPS , 2022.

Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI , 2022.

Zhiliang Peng, Wei Huang, Shanzhi Gu, et al. Conformer: Local features coupling global representations for visual recognition. In ICCV , 2021.

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR , 2020.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS , 2021.

Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Lam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. NeurIPS , 2022.

Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self-attention via multi-scale token aggregation. In CVPR , 2022.

Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng YAN. Inception transformer. In NeurIPS , 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

Christian Szegedy, Wei Liu, Yangqing Jia, et al. Going deeper with convolutions. In CVPR , 2015.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML , 2019.

Shitao Tang, Jiahui Zhang, Siyu Zhu, et al. Quadtree attention for vision transformers. In ICLR , 2022.

Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is a wave: Phase-aware vision mlp. In CVPR , 2022.

Hugo Touvron, Matthieu Cord, Matthijs Douze, et al. Training data-efficient image transformers & distillation through attention. In ICML , 2021.

Zhengzhong Tu, Hossein Talebi, Han Zhang, et al. Maxvit: Multi-axis vision transformer. In ECCV , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention is all you need. In NeurIPS , 2017.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2103.15808 , 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media , 8(3):1-10, 2022.

Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A versatile vision transformer hinging on cross-scale attention. In ICLR , 2022.

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808 , 2021.

Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR , 2022.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV , 2018.

Tete Xiao, Mannat Singh, Eric Mintun, et al. Early convolutions help transformers see better. In NeurIPS , 2021.

Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV , 2021.

Chenglin Yang, Siyuan Qiao, Qihang Yu, et al. Moat: Alternating mobile convolution and attention brings strong vision models. In ICLR , 2023.

Chenglin Yang, Yilin Wang, Jianming Zhang, et al. Lite vision transformer with enhanced self-attention. In CVPR , 2022.

Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. NeurIPS , 2022.

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers, 2021.

Rui Yang, Hailong Ma, Jie Wu, Yansong Tang, Xuefeng Xiao, Min Zheng, and Xiu Li. Scalablevit: Rethinking the context-oriented generalization of vision transformer. ECCV , 2022.

Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In Proceedings of the European conference on computer vision (ECCV) , 2022.

Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR , 2022.

Kun Yuan, Shaopeng Guo, Ziwei Liu, et al. Incorporating convolution designs into visual transformers. In ICCV , 2021.

Li Yuan, Yunpeng Chen, Tao Wang, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV , 2021.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV , 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, et al. mixup: Beyond empirical risk minimization. In ICLR , 2018.

Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware circular convolution with merits from convnets and transformer. In ECCV , 2022.

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV , 2021.

Qinglong Zhang and Yu bin Yang. Rest: An efficient transformer for visual recognition. In NeurIPS , 2021.

Zhun Zhong, Liang Zheng, Guoliang Kang, et al. Random erasing data augmentation. In AAAI , 2020.

Bolei Zhou, Hang Zhao, Xavier Puig, et al. Scene parsing through ade20k dataset. In CVPR , 2017.

Daquan Zhou, Zhiding Yu, Enze Xie, et al. Understanding the robustness in vision transformers. In ICML , 2022.

Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio-visual learning: A survey. IJAC , 18:351, 2021.

Zhuofan Zong, Kunchang Li, Guanglu Song, et al. Self-slimmed vision transformer. In ECCV , 2022.

Appendix

A Discussion About "Interaction" And "Fusion"

Generally speaking, interaction refers to the process of two or more objects or systems affecting each other. In contrast, fusion refers to combining two or more things into a single entity. As shown in Fig. 5, in contrast to all previous feature fusion methods, our proposed bidirectional adaptive interaction stands out in two significant ways:

(a) interaction

(b) fusion

cat&linear

Bidirectional:

Adaptive:





Bidirectional:

Adaptive:





:local token

:global token

:Hadamard product

1) Bidirectional Property: Our approach draws inspiration from how visual information flows bidirectionally through the human visual system. Before fusing local and global features, we model a bidirectional process between the two. Specifically, for local features, we utilize global features to generate weights that encompass global information. These weights are applied to the local features, allowing them to have the ability to capture the global context. We execute a similar process for global features. This results in strengthened local and global features, respectively. In contrast to linear fusion, our interaction returns two enhanced vectors representing local and global contexts rather than an inseparable vector mixed with both [41; 48; 43].

Adaptive Capability: Taking local features as an example, the global weights we use to enhance them are not trainable. They are generated by tokens containing global features, making them contextaware and capable of adapting to the input data. Moreover, when fusing the enhanced local and global features, we do not introduce any additional parameters but calculate their element-wise product (Hadamard product) directly. Compared with linear fusion, our context-aware fusion approach is closer to the attention mechanism [56; 17; 22]. Our ablation experiments have shown that this fusion approach not only saves parameters but also achieves better results. Specifically, for a detailed comparison between the Add+Linear, Cat+Linear, and our model, please refer to the paper's main text.

The implementation of our proposed bidirectional adaptive interaction method is very simple, yet it has achieved remarkable results compared to other methods for fusing local and global features. Through this work, we aim to reveal the critical role that features fusion plays in visual models and hope to inspire more related research.

B More experiments

We evaluate our backbones with the framework of UperNet [62] based on MMsegmentation [7]. Following the [34], we adopt AdamW to optimize the model for 160K iterations. As shown in Tab. 7, our model surpasses all its counterparts and achieves the best performance. Specifically, our FAT-B3 surpass the resent Shunted-S by +0.7 mIoU and +0.8 MS mIoU.

C Implementation Details

C.1 Architecture Details

The detailed architectures are shown in Tab. 8, where all FLOPs are measured at the resolution of 224 × 224 for image classification. For the convolution stem, we adopt four 3 × 3 convolutions and one 1 × 1 convolution to tokenize the image. Batch normalization and ReLU are used after each 3 × 3 convolution, and a layer normalization is used after the final 1 × 1 convolution. Inspired by the fact that in the vision backbone, the early stages tend to capture high-frequency local information, and the later stages tend to capture low-frequency global information, we gradually increase the DWConv's kernel sizes used in FASA. Specifically, we set the kernel size to 3 in the first stage and 9 in the last stage. The expansion ratios are set to 4, and the kernel sizes are set to 5 for all ConvFFN layers.

C.2 Image Classification

We follow the same training strategy with Swin-Transformer [34]. All models are trained for 300 epochs from scratch. To train the models, we use the AdamW optimizer with a cosine decay learning rate scheduler and 20 epoch linear warm-up. We set the initial learning rate, weight decay, and batch size to 0.001, 0.05, and 1024, respectively. We adopt the strong data augmentation and regularization used in [34]. Our settings are RandAugment [8] (randm9-mstd0.5-inc1), Mixup [75] (prob=0.8), CutMix [74] (prob=1.0), Random Erasing [79] (prob=0.25), increasing stochastic depth [24] (prob=0.05, 0.1, 0.1, 0.15 for FAT-B0, FAT-B1, FAT-B2 and FAT-B3).

C.3 Object Detection and Instance Segmentation

We adopt MMDetection [3] to implement RetinaNet [32] and Mask-RCNN [19]. We use the commonly used " 1 × " (12 training epochs) setting for the two strategies. Following [34], during training, images are resized to the shorter side of 800 pixels while the longer side is within 1333 pixels. We adopt the AdamW optimizer with a learning rate of 0.0001, weight decay of 0.05, and batch size of 16 to optimize the model. The learning rate declines with the decay rate of 0.1 at the epoch 8 and 11.

C.4 Semantic Segmentation

We adopt the Semantic FPN [27] and UperNet [62] based on MMSegmentation [7] and apply FAT pretrained on ImageNet-1K as backbone. We use the same setting of PVT [57] to train the Semantic FPN, and we use the " 1 × " schedule (trained for 80k iterations). All the models are trained with the input resolution of 512 × 512 . When testing the model, we resize the shorter side of the image to 512 pixels. As for UperNet, we follow the default settings in Focal Transformer [68]. We take AdamW with a weight decay of 0.01 as the optimizer and train the models for 160K iterations. The learning rate is set to 6 × 10 -5 with 1500 iterations warmup.

D More Ablation and Analysis Results

Other architecture design choices. We modified FAT-B3 to have the same layout as Swin-T and compared it to four baseline models: Swin-T [34], DAT-T [61], FocalNet-T [67] and Focal-T [68]. As shown in Tab. 9, our model demonstrates significant superiority, with an accuracy 1.7% higher than Swin-T and 0.8% higher than Focal-Tiny.

Contribution Isolation. To better isolate the contribution of our FASA, we compare the different self-attention mechanisms on the same backbone. We choose five kinds of self-attention mechanisms to make the comparison: CSWSA in CSwinTransformer [12], Max-SA in MaxViT [55], WSA/SWSA in Swin-Transformer [34], SRA in PVT [57] and LSA/GSA in Twins-SVT [5]. The results are shown in Tab. 10. The networks' layout and the setting of CPE and ConvFFN are the same as FATB0. It can be found that our FASA surpasses all its

counterparts. Compared to the latest Max-SA, our FASA achieved a gain of 1.9%.

Local Aggregation in FASA. To validate that the model's capability to perceive local context does not solely rely on CPE and ConvFFN, we conducted experiments by removing local adaptive aggregation and bidirectional adaptive interaction from FASA, retaining only global adaptive aggregation. The results presented in Tab. 11 demonstrate a significant decrease in model performance ( 1.4% ) when using only global

adaptive aggregation. This outcome convincingly supports the soundness of our proposed FASA model.

Additional Sigmoid. In our main text, in the Method section's Bidirectional Adaptive Interaction subsection, Eq. 11 omits the high-order variable:

in the final expression to a faster implementation. This variable represents two concatenated Sigmoid activation functions within the local adaptive aggregation phase. We attempt to include the previously omitted activation function in this section but find that having too many activation functions results in a similar performance but slower speed, as shown in Tab. 12. This result confirms the rationality of the structure we designed.

kernel sizes 3, 3, 3, 3 3, 5, 7, 7 3, 5, 7, 9 7, 7, 7, 7 9, 9, 9, 9 Kernel Size of FASA. Inspired by the notion that early stages of vision backbones tend to aggregate lower-level local features, while later stages prefer higher-level semantic ones, we adopt a staged approach and incrementally increase the DWConv's kernel size in FASA to aggregate local features. In this section, we investigate how DWConv's kernel size in FASA affects the model's image classification performance by varying the kernel sizes in different stages, as depicted in Tab. 13. Our experimental results demonstrate the effectiveness of our strategy for gradually increasing kernel size at different stages. Models using kernel sizes of 3, 5, 7, 9, and 3, 5, 7, 7 achieved the best performance.

Convolution Stem. We use the convolution stem proposed in [63]. To investigate the impact of convolution stem on our model's performance, we conducted experiments and presented the results in Tab. 14. Despite incurring some extra computational overhead, using convolution stem can significantly enhance the top1 accuracy of the model by 0.4% .

Kernel Size of ConvFFN. We conduct ablation experiments to validate the effect of different kernel sizes for DWConv in ConvFFN. As shown in Tab. 15, the kernel size of 7 × 7 achieves the best performance, but it consumes more parameters (+0.2M) and FLOPs (+0.03G). To achieve a better trade-off between the performance and computational complexity, We set the kernel size of DWConv in ConvFFN to 5.

Bidirectional Interaction. Here we ablate the local-toglobal and global-to-local, respectively, to show the effect of bidirectional interaction. The results are shown in the Tab. 16. It can be seen that the bidirectional interaction plays an important role in FAT.

E Speed Measurement

In order to accelerate the inference speed of our model, we utilized a toolkit developed for accelerating large kernel DWConv in [11]. Additionally, we employed the technique of structural reparameterization to merge the residual blocks in ConvFFN into one convolutional block. Furthermore, after omitting

the higher-order variables in bidirectional adaptive interaction, the original expression can be simplified to two SiLU activations followed by the multiplication of two branches. These measures significantly accelerated the inference speed of our model. Here, we list the most recent models in Tab. 17. As can be seen from the table, our model performs the best in terms of both speed and performance.

F Limitations and Broader Impacts

While our FAT achieves superior performance with fast processing speed, one limitation of our work is that due to computational resource constraints, we do not apply our lightweight vision backbone to unsupervised learning, video processing, or visual language tasks. Additionally, we do not pretrain our models on large-scale datasets like ImageNet-21K. However, we look forward to exploring more applications of our proposed FAT in the future.

This study is purely academic in nature, and we are unaware of any negative social impact resulting directly from our work. However, we acknowledge that our models' potential malicious use is a concern affecting the entire field. Discussions related to this matter are beyond the scope of our research.

Peer reviews:
comments: SUMMARY:
The paper proposes a new lightweight ViT structure called FAT. They use a fully adaptive self-attention mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. In addition, the paper introduces a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability.

STRENGTHS:
- The paper is well-written and easy to understand.

- The experimental results show the effectiveness of the method.

WEAKNESSES:
- The usage of conv stem, conditional positional encoding, the convFFN and the aggregation of global and local attention is not novel. Thus, the novelty of this paper primarily lies in using bidirectional interaction. However, it is un-natural to explain via human visual system of using local to global and global to local interaction. Is it really the way human look at pictures?

- In Eq.5, it is unclear why using different A1 and A2 makes the interaction 'bidirectional'. The author should further explain this.

- Lack of comparison to the state-of-the-art models. For example, the proposed FAT-B3 has the same accuracy to CMT-S [1]  but larger FLOPs and parameters.

[1] CMT: Convolutional Neural Networks Meet Vision Transformers. CVPR 2022.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
See weaknesses above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new family of lightweight vision transformers that are built on the Fully Adaptive Self-Attention module. The key innovation of these transformers lies in their bidirectional interactive module, which enhances both local and global features. The experiments primarily focus on image classification, object detection, and semantic segmentation tasks. Extensive evaluations were conducted on various common benchmarks, including ImageNet-1K, to assess the performance of the proposed transformers.\n\nSTRENGTHS:\nThe proposed method demonstrates a key innovation through bidirectional interaction between the global and local branches, leading to significant improvement as validated by comprehensive ablation studies. The study includes extensive experiments, and the ablation analysis is thorough, highlighting the quality of the research. The writing is clear and concise, effectively presenting the experimental setting and demonstration. Impressively, FAT-B0 and B1 achieve remarkable results in terms of both throughput and accuracy, providing valuable insights for future research on low-latency small models.\n\nWEAKNESSES:\n1. The scaling behavior of the method is my primary concern. While I appreciate the author's focus, the current results indicate that larger models exhibit limited improvement over the baseline in the three tasks. In my opinion, it is essential to discuss the scaling aspect for a proposed general backbone.\n2. The second issue is the lack of comparison in terms of inference time between different methods or different hardware (e.g. CPU/GPU). Based on the current results, the proposed model works very well under low parameter settings, providing the direct benefit of lighter and more efficient deployment on mobile devices. However, the author has not provided sufficient evidence to demonstrate whether the proposed method maintains efficient inference efficiency with such low computational requirements. This aspect is crucial for me in proposing a lightweight model because training speed is often not a concern for these types of models. Instead, focusing on comparing the capabilities of this model under different hardware and computational costs can significantly enhance the significance of this method.\n3. The motivation for this paper appears somewhat forced, particularly from Figure 1. I have not come across any related work discussing this phenomenon. If any such works exist, please cite them, as it would enhance the trustworthiness of your motivation.\n\nQUESTIONS:\n1. My first two questions are motivated by different considerations, and I recommend that the author concentrate on one aspect to augment the method's significance. Given the exceptional performance of the proposed model on small-scale counterparts, my particular interest lies in contrasting the inference across different hardware and CNN architectures. Nevertheless, should the author contend that the proposed model should possess broader applicability, I am also intrigued to observe its scaling behavior.\n2. I am interested in investigating the training stability of the proposed methods. It would be helpful if there are any plots illustrating the training stability.\n\nLIMITATIONS:\nThe author has discussed their limitation in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented an efficient vision transformer backbone for several tasks including classification, segmentation and detection. The key idea of this paper is on a new design of considering the interaction between local and global features.  The experiments on imagenet, AD20K, and COCO  have shown that the proposed method is comparable to most existing methods.\n\nSTRENGTHS:\nThis paper has clear contributions on the vision transformer design.\n1.  The proposed the local-global feature interaction is well motivated. \n2.  The proposed methods has been evaluated with comprehensive experiments. \n3. This paper is well written and easy to understand.\n\nWEAKNESSES:\nThere are some weaknesses in this paper:\n\n1. The area of new backbone is growing rapidly.  Although the proposed method shows marginally improvements over the experimented baselines, compared with concurrent work, it becomes not appealing. \n2.  The proposed lightvision transformer is very hard to justify as it is not as efficient as mobile vit but also not sure how it can be scaled up. \nThus it is very hard to justify. \n3.  The baseline methods are not consistent.  For example Table 1 and Table 2 pick different methods for similar comparison. For example, ParC-Net should be put into 0~5 while the author intentionally put into different comparison setting to make the number look better.\n\nQUESTIONS:\nThe experiments are not fair comparison as some of the numbers are cherry picked.  It will be more convincing to compare with Segnext, ParC-Net, MobileOne and mobileVIT clearly.  As far as the reviewer can see, the proposed method is comparable or weaker than them. \n\nIn addition, the ablation studies didn't show the bidirectional interaction is useful or not. For example, given single interaction will it be OK?\n\nLIMITATIONS:\nThe key limitation of this paper is the proposed method lack of evidence on whether it can be mobile-friendly or scaled up as VIT liked backbones.   Since the backbone is not general enough, it will has very limited usage.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new family of light-weight vision transformers named FAT, which enhances the local and global feature fusion by a bi-directional interaction between them. Experiments on image classification, object detection, and semantic segmentation are conducted.\n\nSTRENGTHS:\n1. The paper provides a good practice in designing high performance efficient ViTs, which could be a valuable reference for the future work.\n\n2. The results on several benchmark datasets outperform all the compared models.\n\nWEAKNESSES:\n1. An incremental work. There exists tons of works that focus on performing local and global features in ViTs (see Sec. 2 of the paper), this paper only proposes a simple way to enhance the feature fusion. Besides, other increments are actually a combination of existing successful practices. For example, the convolutional and non-overlapping downsample module is from PVTv2, the downsample in MHSA have also been widely used (e.g., PVT), CPE is from [1].\n\n2. Actually, a CVPR 2022 paper MixFormer [2] also proposes a bi-directional interaction of convolutional and self-attention branches. What's the pure difference between this method and MixFormer?\n\n3. It is difficult for me to evaluate the efficacy of the proposed bi-directional interactions. The paper only compares the method with simple fusion baselines, while there exist many global-local feature fusion methods, I cannot figure out which one is better according to Table 1, since many increments are added in this method. I suggest the authors to replace the fusion method in FAT with other popular methods such as the interactions in MixFormer and message token in MSG-Transformer [3], for fair comparisons.\n\n---  \n**References**\n\n[1] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR, 2023.\n\n[2] Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J. and Wang, J., 2022. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5249-5259).\n\n[3] Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W. and Tian, Q., 2022. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12063-12072).\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}