Lightweight Vision Transformer with Bidirectional Interaction
Authors: Qihang Fan
Huaibo Huang
Xiaoqiang Zhou
Ran He
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision Transformer
Lightweight Vision Backbone
Convolution Neural Network
TL;DR: A lightweight Vision Transformer with bidirectional interaction achieved by context-aware manners between local and global contexts.

Abstract: Recent advancements in vision backbones have significantly improved their performance by simultaneously modeling images’ local and global contexts. However, the bidirectional interaction between these two contexts has not been well explored and exploited, which is important in the human visual system. This paper proposes a **F**ully **A**daptive **S**elf-**A**ttention (FASA) mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. Specifically, FASA employs self-modulated convolutions to adaptively extract local representation while utilizing self-attention in down-sampled space to extract global representation. Subsequently, it conducts a bidirectional adaptation process between local and global representation to model their interaction. In addition, we introduce a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability. Based on FASA, we develop a family of lightweight vision backbones, **F**ully **A**daptive **T**ransformer (FAT) family. Extensive experiments on multiple vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT accomplishes a **77.6%** accuracy on ImageNet-1K using only **4.5M** parameters and **0.7G** FLOPs, which surpasses the most advanced ConvNets and Transformers with similar model size and computational costs. Moreover, our model exhibits faster speed on modern GPU compared to other models.

Lightweight Vision Transformer with Bidirectional Interaction
Qihang Fan 1.2, Huaïbo Huang', Xiaoqiang Zhou' 1.3 Ran Hel.2.
MAIS & CRIPAC, Institute of Automation, Chinese Academy of Sciences, Beijing, China School Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Unniversity of Science and Technology of China, Hefei, China fanqihang 159@gmail..... com, huaibo. hwt.bo.uaannrripp.c wwtpeerrrracc..... ia..iaaac ac. cn, xq5250mai1 uste. en, rhe@nlpr aaa.accco en
Abstract
Recent advancements in vision backbones have significantly improved their perfor- mance by simultaneously modeling images' local and global contexts. However, the bidirectional interaction between these contexts has not been well explored and exploited, which is important in the human visual system. This paper pro- poses Fully Adaptive Sclf-Attention (FASA) mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. Specifically, FASA employs self-modulated convolutions to adaptively extract local representation while utilizing self- attention down-samplee sampled space to extract global representation. Subsequently, it conducts a bidirectional adaptation process between local and global representation to model their interaction. In addition, we introduce fine-gra fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability. Based on FASA, we develop a family of lightweight vision backbones, Fully Adaptive Transformer (FAT) family. Extensive experiments on multiple vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT accomplishes a 77.6% % accuracy on ImageNet- using only 4.5M parameters and 0.7G FLOPs, which surpasses the most advanced ConvNets and Transformers with similar model size and computational costs. Moreover, model exhibits faster speed on modern GPU compared to other models.
Introduction
Vision Transformers (ViTs) have recently garnered significant attention in the computer vision community due to their exceptional ability for long-range modeling and context-awaree characteristics. However, because of the quadratic complexity of self-attention VIT [13. its computational cost is extremely high. As result, many studies have emerged to improve ViT's computational efficiency and performance in various ways. For instance, some methods restrict tokens that perform self- attention to a specific region and introduce inductive bias ViT (34 12 66 (5).. Further, some methods aim to transform ViT into lightweight backbones with fewer parameters and computational
requirements 38 of the most advanced ConvNets
30 37), achieving promising results but still not matching the performance
[T
How to design an excellent lightweight Vision Transformer
remains a challenge.
In current state-of thethe- Vision Transformers, some either excel in creating local feature extraction
modules 34 12 66 employing efficient global information aggregation modules [57 58), while
others incorporate both [2 For instance, LVT (66) unfolds tokens into separate windows and
Ran He is the corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Local Global: Nose of fox.
Local: Nose
Global: Fox.
Global to Local: A with nose.
Aggeragation
Modulation
Modulation
Act
Act
Modulation
Aggregation
Modulation
(a) local adaptive aggregation
(c) bidirectional adaptive interaction
(b) global adaptive aggregation
target token
local token
globul token
output token
receptive field
Figure I: Illustration of the human visual system (top) and our FASA (bottom). The human visual
system perceive both local and global contexts and model the bidirectional interaction between them. Our FASA follows this mechanism and consists three parts: (a) local adaptive aggregation, (b) global adaptive aggregation, and (c) bidirectional adaptive interaction. Our FASA models local information, global information, and local-global bidirectional interaction in context-aware manners.
applies self-attention within the windows to extract local features, while PVT 57 58 leverages self-attention with downsampling extract global features and reduce computational cost. Unlike them, LITv2 [41 relies on window self-attention and spatial reduction attention to capture local and simple local-global sequential structures [38] 40 37 26 82), whereas others combine local and global representation with simple linear operations through local-global parallel structures 48 4 43.. However, few works have investigated the bidirectional interaction between local and global information. Considering the human visual system where bidirectional local-global interaction plays an important role, these simplistic mixing methods are not fully effective in uncovering the intricate relationship between local and global contexts.
global features, respectively. In terms of the local-global fusion, most methods
84
FAT(Ours)
In Fig. I we illustrate how humans observe an object and notice its body details. Using the example of a fox, we can observe two types of interaction that occur when humans focus on either the fox's 's nose or the entire animal. In the first type of interaction, known as Lo- cal to Global, our understanding of the local feature transforms into the "Nose of fox." In the second type of interaction, called Global Local, the way we comprehend the global feature changes to the "A fox with nose." It can seen that the bidirectional interaction between local and global features plays an es- sential role in the human visual system. Based on this fact, propose that superior visual model should not only extract good local and global features but also possess adequate mod- eling capabilities for their interaction.
EdgeVil
MPVIT
Shunted
Top-1 Aco(%)
80
154
DOC
FLOP
Figure 2: Top-1 1 accuracy v.s. FLOPs on ImageNet- IK of recent SOTA CNN and transformer models. The proposed Fully Adaptive Transformer (FAT) out- performs all the counterparts in all settings.
In this work, our objective is to model the bidirectional interaction between local and global contexts while also improving them separately. To achieve this goal, we introduce three types of Context- Aware Feature Aggregation modules. Specifically, shown in Fig. I we first adaptively aggregate local and global features using context-aware manners obtain local and global tokens, respectively. Then, perform point-wise cross-modulation between these two types of tokens to model their

---

bidirectional interaction. We streamline all three processes into simple, concise, and straightforward procedure. Since we use context aware approaches to adaptively model all local, global, and local- global bidirectional interaction, we name our novel module the Fully Adaptive Self-Attention (FASA). In FASA, we also further utilize a fine-grained downsampling strategy to enhance the self-attention mechanism, which results in its ability to perceive global features with finer granularity. In summary, FASA introduces only a small number of additional parameters and FLOPs, yet it significantly improves the model's performance.
Building upon FASA, we introduce the Fully Adaptive Transformer (FAT) family. The FATs follow the hierarchical design B4 57] and serve as general general-purpose backbones for various computer vision tasks. Through extensive experiments, including image classification, object detection, and semantic segmentation, we validate the performance superiority of the FAT family. Without extra training data or supervision, our FAT-BO achieves a top-1 accuracy of 7 on ImageNet- 1 K with only 4.5M parameters and 0.7G FLOPs, which is the first model surpasses the most advanced ConvNets with similar model size and computational cost as far as we know. Additionally, as shown in Fig. 2 our computational costs.
FAT-BI, B2, and B3 also achieve state-of-the-ar results while maintaining similar model sizes and
2 Related Work
Vision Transformer. Since the carliest version of ViT [3] appeared. numerous works have been proposed that focus on enhancing the self-attention mechanism. Many methods restrict self-attentin to a subset of tokens, sacrificing global receptive fields to reduce the computation of ViT [34] [2] 66 61 1. Despite having only local perception capabilities, the context-aware nature of self- attention enables these methods to achieve excellent performance. A famous among them is the Swin-Transformer (34], which divides all tokens into windows and performs self-attention within each window, achieving highly competitive performance. In contrast, some methods downsample the K and V in self-attention preserve the global receptive field while reducing computation by minimizing the number of token pairs involved in the calculation [57] 60; 40: 47 68, 5). One such method. PVT [57], leverages large-stride convolution to process K and V. effectively lowering their spatial resolution. In addition to these methods, numerous efforts have been made to introduce ViT 4. For example, ViT (38) concatenates convolution and self self-attention to obtain powerful, lightweight backbone. Nevertheless, a performance gap exists between lightweight vision transformers and state-of-the-art lightweight EfficientNet [51].
into the design of lightweight vision backbones (66 40 37
38
CNNs such as
Local-Global Fusion. A high-performance vision backbone typically possesses exceptional ca- pabilities for both local and global perception. The capabilities are achieved by either connecting local and global perception modules serial manner [58] 31 IO, 26 40 55, M. as demonstrated by DaViT [00] or by simultancously modeling local and global information within a single mod- ule (48 41 43), such as the inception transformer [48). However, current approaches fusing local and global information are overly simplistic. In serial models, the process of fusing local and global information is not adequately represented. In parallel structures, almost all methods rely on linear modules that depend entirely on trainable parameters to fuse local and global information [41 48 43) These fusing approaches lack the ability to model the interaction between local and global informa- tion, which is inconsistent with the human visual system shown in Fig. [ In contrast, our proposed FASA module models bidirectional interaction between local and global information while separately modeling each one.
Self-Attention with Downsampling. Currently, many models utilize self-attention with down- sampling, a technique that was earlier used in PVT and so on to improve the computational ef- 4
48 6 5). PVT [57] reduces the spatial resolution of K and V using non-overlapping large stride convolutions, which decreases the number of token pairs involved in self-attention and lowers the computational cost while maintaining global receptive field. Similarly, [58) uses large stride average pooling to downsample K and V. In contrast, inception transformer 48 employs large stride average pooling to downsample all three Q, K, and V. then upsamples the tokens to their original size after self-attention is applied. However, using exces- sively large strides or non-overlapping downsampling as in these methods may lead to significant information loss. In FASA, we propose fine-graine fnn-ggrinedd downsampling strategy to alleviate this issue.
ficiency [57] 588 43
PVTv2
3

---

Fully Adaptive Trausformer (FAT)
L2
CPE
Fully Adaptive Self-Attention
ConvFFN
Local
Ridirectional Internetion
X
Conv
Layer LayerSom Nomu
Come1
Layerhom
DWCaar
Cone 1
Z
Global
JiN
Signoid
DWConvgg23
Copy x1
BSS
Product
PM 1.SiiN
Cony
Com Stett
Sugeo Block
Clacitier
Element wiss Add
Block
Only stage
L
La
Ls
L4
Only downsample block
Ouiitel simplicity
Fine-grenal dnreoompl
Figure 3: Illustration of the FAT. FAT is composed of multiple FAT blocks. A FAT block consists of CPE, FASA and ConvFEN.
Method
3.1
Overall Architecture.
The overall architecture of the Fully Adaptive Transformer (FAT) is illustrated in Fig. 3 To process an input image x E R3x H XW we begin by feeding into the convolutional stem used 63 This produces tokens of size и WW' Following the hierarchical designs seen in previous works (48] 50 46 49,, we divide FAT into four stages to obtain hierarchical representation. Then we perform average pooling the feature map containing the richest semantic information. The obtained one-dimensional vector subsequently classified using linear classifier for image classification.
4.
four
A FAT block comprises three key modules: Conditional Positional Encoding (CPE) [6], Fully Adap- tive Self-Attention (FASA), and Convolutional feed-Forwart Network (ConvFFN). The complete FAT block is defined by the following equation (Eq. I]:
X CPE(Xi.) + Xin:
Y FASA(LN(X)) +X,
Z = CONFFFN(LN(Y)) + ShortCut(Y).
Initially, the input tensor X., E RCXHxW passes through the CPE to introduce positional informa- tion for each token. The subsequent stage employs FASA to extract local and global representation adaptively, while bidirectional adaptation process enables interaction between these two types of representation. Finally, ConvFFN is applied to enhance local representation further. Two kinds of ConvFFNs are employed in this process. In the in-stage FAT block, ConvFFN's convolutional stride is and ShortCut tional stride is set to 2, and ShortCut = DownSample is achieved via a Depth- Wise Convolution (DWConv) 23 with the stride of 2 along with x convolution. The second type of ConvFFN accomplishes downsampling using only a small number of parameters, avoiding the necessity of patch merging modules between the two stages, thereby saving the parameters.
Identity. At the intersection of the two stages, the ConvFFN's convolu-
3.2 Fully Adaptive Self-Attention
In Fig. | we present the visual system of humans, which not only captures local and global informa- tion but also models their interaction explicitly. Taking inspiration from this, we aim to develop a similar module that can adaptively model local and global information and their interaction. This leads the proposal of the Fully Adaptive Self-Attentin (FASA) module. Our FASA utilizes context-aware manners model all three types of information adaptively. It comprises three modules:

---

global adaptive aggregation, local adaptive aggregation, and bidirectional adaptive interaction. Given the input tokens X RC*HxW each part of the FASA will be elaborated in detail.
3.2.1 Definition of Context-Aware Feature Aggregation
In order to provide readers with a clear comprehension of FASA, we will begin by defining the various methods of context-aware feature aggregation (CAFA). CAFA is widely used feature aggregation method. Instead of solely relying on shared trainable parameters, CAFA generates token-specific weights based on the target token and its local or global context. These newly generated weights, along with the associated context of the target token, are then used to modulate the target token during feature aggregation, enabling each token to adapt to its related context. Generally, CAFA consists of two processes: aggregation (.A) and modulation (M). In the following, the target token is denoted by I,, and JF represents a non- non-linear activation function. Based on the order of and M, CAFA can be classified into various types. Various forms of self-attention [56: 58 57 45 can be expressed as E Aggregation over the contexts X performed after the attention scores between query and key are computed. The attention scores are obtained by modulating the query with the keys, and then applying a Softmax to the resulting values:
X)),
In contrast to the approach outlined in Eq. 2 recent state-of-ther the-art ConvNets 22 17 67
utilize a different CAFA technique. Specifically, they employ DWConv to aggregate features, which are then used to modulate the original features. This process can be succinctly described using Eq.
y M(A x.),
In our FASA module, global adaptive aggregation improves upon the traditional self-attention method with fine-grat grainei downsampling and can be mathematically represented by Eq. 2 Meanwhile, our local adaptive aggregation, which differs slightly from Eq, 3 can be expressed as Eq. 4
y MMM (A(a;, A(r;, X));
In terms of the bidirectional adaptive interaction process, it can be formulated by Eq. 5 Compared to the previous CAFA approaches, bidirectional adaptive interaction involves two feature aggregation operators (A1 and A2) that are modulated with each other:
V/2 (MF( X)). A2(2,, X)), Vn M(F(A2 (r X)) A1 (2s, X)).
(5)
3.2.2 Global Adaptive Aggregation
The inherently context-aware nature and capacity model long-distance dependencies of self- attention make it highly suitable for adaptively extracting global representation. As such, we utilize self-attention with downsampling for global representation extraction. In contrast to other models that downsample tokens [57 58
60 48 by using large stride convolutions pooling operations, we adopt fine-gratd grained downsampling strategy to minimize loss of global information to the greatest extent possible. In particular, our fine-graine
downsampling module is composed of several basic units. Each unit utilizes a DWConv with kernel size of 5 x and a stride of 2, followed by a x 1 convolution that subtly downsamples both K and V. After that, Q. K, and V are processed through the Multi-Head Self-Attention (MHSA) module. Unlike regular MHSA, we omit the last linear layer. The complete procedure for the global adaptive aggregation process is illustrated in Eq. 6 and Eq. Fisrt, we define our fine-grained downsample strategy and the base unit in Eq. 6
Base Unit(X) - (BN
(6)
(X) A BNDDWCCan((asssTn
where (n) represents the number of base units that are concatenated and pool denotes our fine-grained downsample operator. Then, the MHSA is conducted as Eq. 7
Q,K, V W1 X, X Xtoobat MHSA(Q, pool()).
where the mathematical symbol ( denotes the operation of matrix multiplication, W, a learnable matrix.
5

---

3.2.3 Local Adaptive Aggregation
In contrast self-attention, convolution possesses an inductive bias that facilitates extracting high- frequency local information. However, convolutional feature aggregation solely relies on parameter shared convolutional kemels, resulting in a lack context-awareness. To address this issue, we utilize a self modulaadug convolutional operator that embeds context-awareness into convolution, enabling it to extract local representation adaptively. Specifically, we generate context-aware weights through the Sigmoid and combine them with DWConv to adapively aggregate local information. This process summarized in Eq. [] where @ represents the Hadamard product, and Q directly derived from Eq. 7 for saving parameters of linear projection:
Q' = DWWCovv(Q),
(8)
X local Q' Sigmoid(Q') = Ite-q
3.2.4 Bidirectional Adaptive Interaction
As illustrated in Fig. I when modeling the fox and its nose separately, two types of interactions "Local to Global" and "Global to Local" occur between their respective features. Inspired by the human visual system, we design a bidirectional adaptive interaction process that incorporates both types of interactions. To achieve bidirectional interaction, one described in Sec. B.2.3bbut utilize cross- modullaonotiin instead of self-modulation. Specifically, the equation for Local Global interaction given by Eq. [
adopt method similar to the
Xlocal = Xioeat O Sigmoid (X² glohai) Xfoea
e
global
(9)
Q' @ 1+e-Q 1
Xitorat "
Similar to Eq. 9 Global to Local interaction is achieved by Eq.
10
X globbat
X global
Sigmoid (Xloidd((ooo (loo))
X
global
1 e
Xlool
(10)
X global
Q I+e
1 1+c
After completing the two interaction processes, the local and global representation contain information about each other. To merge these representation, we use point-wise multiplication and achieve intermingling among channels with linear projection, as shown in Eq.
W2 & (Xgiobal O X(ocat)
(1I)
W2 & (X (X global Q' 1 1
X global
Q'O
1
Itc
It is worth noting that to get a faster and implementable model, we omit the last high-order variable in Eq. II and arrive concise expression. More details can be found in appendix. Through the combination of local adaptive aggregation, global adaptive aggregation, and bidirectional adaptive interaction, we present the comprehensive FASA module. As depicted in Fig. 3, our model, which draws inspiration from the human visual system, is both straightforward and simple. It adaptively models local, global, and the interaction between the two context-aware manners. More discussion about bidirectional adaptive interaction can be found in the appendix.
4 Experiments
conducted experiments on a wide range of vision tasks, including image classification on ImageNet-1K 9 object detection and instance segmentation tic segmentation on ADE20K [80]. In addition to them, we also make ablation studies validate the importance of cach component. More details can be found in the appendix.
COCO 2017 33), and seman-

---

4.1 Image Classification
Settings. We train our models on ImageNet- 1K [9] from scratch. And we follow the same training strategy in DeiT [54] for fair comparison. Different from DeiT, we employ 20 epochs of linear war while in DeiT it is 5. The maximum rates increasing stochastic depth [24 are set to 0.05/0.05/0..1 1/0.1/00.... 15 for BO/B O0/B1/B2/B3, respectively. More details can be found in the appendix.
Size
(40
Model
Imput
Params
FLOPs
(G)
Throughput (imal/s
Top-1
Size (M)
Input
Params
FLOPs
(G)
Throughhut (imgls)
TTp-1
12T-V/T-7
73
2242
4.3
1.1
1762
71.7
XCI--124
14
224
12.1
2,3
1194
79.4
QuadThe--Bai B-b0
52
22
3.5
0.7
885
72.0
ResT- 03
22
13.7
1.9
918
79.6
2.1
Onha-T
224
3.9
0.7
74.0
DefT.8 54
226
22.1
4.6
899
79.9
EdgeVIT-XXS
60
22
4.1
0.6
1926
74.1
QuaiThoe-Be B-bl
52
224
13.6
2.3
543
80.0
MobileViT-XS
256
RegionViT--i
LVT
66
224²
5.5
0.9
1265
71.8
Wave-MLP.T
22
17.0
2.1
1052
80.6
CPVT-T--AP
6
224
5.8
1.3
74.9
MPVITXX
83
224'
10.5
2.9
597
80.9
Eeg-NeXI-X8
256
1417
EdgeViT--8
224
1049
81.0
PVT- 37
22/2
13.2
1233
75.1
VAN-B1
22
13.9
2.5
995
81.1
VIT-C 03
224²
4.6
1.1
75.3
Swin-T [4]
226
29.0
4,5
664
81.3
VAN-BO
17
2347
1662
Croesoorme-T 5g
224'
27.8
2.9
81.5
ViL-Timx
22/2
6.7
857
76.3
ResT-8 78
221F
30.3
4.3
588
81.6
Ceil-T
72]
224²
6.4
1.2
76.4
EficiemnNer B3 51
300
12.0
1.8
633
81.6
PAT-BO
2247
PAT-82
224'
13.5
2.0
1064
T2T-VTT-12
73
22:2
6.9
1.9
1307
76.5
DAT-T [6]
221F
29
4.6
577
82.0
Rast-lite
73
224
10.5
1.4
1123
77.2
FocalNet-- 67
224F
29
4,4
610
82.1
XCII-T12
234
Rocal-T [6
224
4.9
82.2
EIgeVIT-XS
226
6.7
1528
77.5
GrusFormeerS
226
31
4,9
601
82.5
CoaT-Lite-T
64
22
5,7
1.5
1045
77.5
RegionViT A
226
31
5.3
460
82.6
STT-T FRR
83
22
15.9
1057
77.7
DaVIT-T [0
22
28
4.5
616
82.8
RegNerY.L.6FF
MPVТ-T
23
2222
5.8
1.6
737
78.2
QuafTre-B-a22
52
224
24
4.5
299
82.7
MobileViT-S
X3
2562
5.6
2,0
898
78.4
CSWin-T [
22
23
4,3
591
82.7
ParC-Net-S
PVTv2-81
5%
2222
13.1
2.1
1007
78.7
HorNer-T 46
224'
23
4.0
586
83.0
PerViT-T 33
22:22
7.6
1402
78.8
FAT-B3-ST
224
29
4,7
641
83.0
Co
Com+NeX:-5 33
EfficieniNes-11
240
7.8
0.7
1395
79.2
LITv2-M [1]
224
49
7.5
436
83.3
FAN-T-VIT 8
22
7.3
1181
79.2
ElliiieFFommre-
30
224'
82
10.2
368
83.3
EdgeNext-S
25
FAT.B]
224
7.8
1452
80.1
PAT-B3
226
29
4.4
474
83.6
Table Comparison with the state-of-the-an on ImageNet- IK classification. "FAT-B3-ST" indicates the FAT-B3 with the same layout of Swin-T.
Results. We compare our FAT against many
Semantic FPN
state-of-the--rt models in Tab. [ The com- parison results demonstrate that our FAT con- sistently outperforms previous models across all settings, with particularly significant improvement in performance on lightweight models. Specifically, our FAT-BO achieves 77.6% Topl-accuraad with only 4.5M pa- rameters and 0.7GFLOPs. Our FAT-BI also surpasses EdgeNext-S by 0.7% % with similar FLOPs. Furthermore, our FAT-B2 model sur- passes CrossFormer-T in laccurcyy by 0.4% while using only half the parameters. Addition- ally, our FAT-B3 model has also outperformed many state-of-the-arr models.
Backhone
Params(M]
FLOPS(C) mloU(()
PVTv2-B0 588
7.6
25.0
37.2
VAN BO 17
8.1
25.9
38.5
EdgeViT-XXS 40
7.9
24.4
39.7
FAT-B0
8.4
25.0
41.5
PVT-T (57
17.0
33.2
35.7
ResNet50 20
28.5
45.6
PoolFormer-SI 71
16.2
31.0
37.2
EdgeVIT-XS 40
10.6
27.7
41.4
FAT-BI
11.6
27.5
42.9
ResNer18 20
15.5
32.2
32.9
PVTv2-BI
17.8
34.2
42.5
VAN-BI
18.1
34.9
42.9
FAT-B2
17.2
32.2
45.4
Swin T
31.9
182
41.5
DAT-T
32.1
198
42.6
ScalableVIT
69
30.4
174
44.9
Speed. As shown in Tab. ! present the speed comparison results of all models on V100, and FAT performs better than its counterparts in both speed and performance. Specifically, Our FAT-BO has 30% higher throughput than EdgeViT-XS. and 0.1 % higher top-1 accuracy compared to Edge ViT-XS..
CrossFormer-
34.3
VAN-B2
17
30.3
164
46.7
CSWin-T
12
26.1
202
48.2
Ortho-S
28.0
Shuted S 47
26.1
183
48.2
FAT-B3
32.9
179
Table 2: Comparison with the state-of-the-an on ADE20K.
4.2 Semantic Segmentation
Settings. We incorporate our pretrained FATs with the Semantic FPN As in
58 57
we use
512 x 512 random crops during training and resize the images to have shorter side of 512 pixels
7

---

during inference. Following [57] we fine-tune the models by AdamW with the learning rate of 1 x 10 and batch size of 16. We train them for 80K iterations on the ADE20K training set.
Results. The results can be found in Tab. 2 For B0/B1/B2, the FLOPs are measured with input size of 512 512. While them for B3 measured with 512 x 2048. All our models achieve the best performance in all comparisons. Specifically, our FAT-B1 exceeds Edge ViT-XS for +1.5 mloU. Moreover, our FAT-B3 outperforms the recent Shunted-S for +0.7 mloU, All the above results demonstrate our model's superiority in dense prediction.
4.3 Object Detection and Instance Segmentation
Settings. We conduct object detection and instance segmentation experiments on the COCO 2017 dataset 33 Following Swin [4], use the FAT family the backbones and take RetinaNet 32 and Mask-RRCNN [19 as detection and segmentation heads. We pretrain our backbones on ImageNet- 1K and fine-tune them on the COCO training set with AdamW [3
RetinaNet 1x
Misk R-CNN 1x
Backhone
Params(M)
Al
Aprs
AFss
APM
AP2
Parans(M)
AP
AF
Al%
APM
AP30
AP
DAVI-
15
25
34.8
56.9
37.0
32.6
53.7
34.5
PVTv2-B0
57.8
QuadTree--B B-50
38.4
58.7
41.1
22.5
41.7
51.6
24
38.8
60.7
42.1
36.5
58.0
39.1
EdgeViT-XX8 4b
13
38.7
59.0
41.0
22.4
42.0
$1.6
24
39.9
62.0
43.1
36.9
19.0
39.4
FAT-BO
DFvT:S 15
32
39.2
62.2
42.4
36.3
58.9
38.6
VI 40
16
40.6
61.3
43.3
25.2
43.9
$4.6
27
41.4
63.7
45.0
38.3
60.9
41.3
61.3
43.6
60.3
MPVT-T 23
41.8
62.7
44.6
27.2
45.1
54.2
28
42.2
64.2
45.8
39.0
61.4
41.8
FAT-BI
42.5
64.0
45.1
26.9
46.0
56.7
28
43,3
65.6
47.44
39.6
61.9
42.8
PiTyi-Tiay
36.7
56.9
38.9
22.6
36.7
59.2
56.7
ResT-Small 7%
23
40.3
61.3
42.7
25.7
43.7
$1.2
33
39.6
62.9
42.3
37.2
59.8
39.7
PVTv2-B]
24
41.2
61.9
43.9
25.4
44.5
54,3
34
41.8
64.3
45.9
38.8
61.2
41.6
39.0
61.8
QuadTree-B-bl
32
24
42.6
63.6
45.3
26.8
46.1
57.2
34
43.5
65.6
47.6
40.1
62.6
43.3
EdgeViT-S( S(D0
23
43.4
64.9
46.5
26.9
47.5
58.1
33
44.8
67.4
48.9
41.0
64.2
43.8
MPVIT.XS
65.0
47.1
63.4
FAT-B2
23
44.0
65.2
47.2
27.5
47.7
58.8
33
4..2
67.9
49.0
41.3
64.66
44.0
Swin-T 34
38
41.5
62.1
44.2
25.1
44.9
55.5
48
42.2
64.6
46.2
39.1
61.6
42.0
64.4
45.2
28.0
43.1
DaviE-Tiox [1]]
39
44.0
48
45.0
41.1
CMT-S [6
44
44.3
65.5
47.5
27.1
48.3
59.1
45
44.6
66.8
48.9
40.7
63.9
43.4
57.3
48.8
28.7
51.2
65.6
45.7
QuafIree-B- Bec-8822 X2
35
46.2
67.2
49.5
29.0
50.1
61.8
45
46.7
68.5
51.2
42,4
65.7
45.7
CSWin-T T 12
42
46.7
68.6
51.3
42.2
65.6
45.4
Shunted
65.9
49.2
28.7
47.1
42.5
65.8
45.7
PAT-R3
39
45.9
66.9
49.5
29.3
50.1
60.9
49
47.6
69.7
52.3
43.1
66.4
46.2
Table 3: Comparison to other backbones using RetinaNet and Mask-RCNN on COCO val2017 object detection and instance segmentation.
Results. Tab, 3 shows the results with RetinaNet and Mask R-CNN. The results demonstrate that the proposed FAT performs best in all comparisons. For the RetinaNet framework, our FAT-BO outperforms EdgeVi ViT-XXS by +0.7 AP, while B1/B2/B3 also perform better than their counterparts. As for the Mask R- R-CNN, our FAT-B3 outperforms the recent Shunted-S by +0.5 box AP and +0.6 mask AP. All the above results tell that our Fully Adaptive Transformer outperforms its counterparts by evident margins. More experiments can be found in appendix.
4.4 Comparison in Efficiency
We compared our approach with the sfate-of-the-a art lightweight vision backbones, as shown in Tab Our method achieved the best trade-off between speed and performance. The CPU is Intel i9Core and the GPU V100. Inference throughput is measured with batch size 64. Inference latency is measured with batch size 1.
4.5 Larger Model
We scale up our FAT to 50M+ and 85M+ to match the general Vision Transformer backbones. For object detection/instance segmentation, all models use the framework of Mask-RCNN [19] with the Ix schedule. The results are shown in Tab. 5 It can be found that FAT has great scalability.
8

---

Model
Params(M) FLOPs(G) CPU(ms)
GPU(ms) J Trp(ings/s)
TopI-acc(98)
EdgeViT-XXX3
43.0
1926
74.4
ViT-XS
38
2.3
100.2
1367
74.8
tiny-MOAT-O
65
3.4
0.8
61.1
14.7
1908
75.5
FAT-BO
4.5
44.3
14.4
1932
77.6
EdgeViT-XS
40
6.7
1.1
62.7
15.7
1528
77.5
ParC-Net-S
76
5.0
1.7
112.1
15.8
1321
78.6
EdgeNext-S
37
5.6
1.3
86.4
14.2
1243
79.4
FAT-B1
7.8
1.2
62.6
14.5
1452
80.1
PuCC-RssNe500 76
23.7
4.0
160.0
16.6
1039
79.6
tiny-MOAT-2 [65]
9.8
2.3
122.1
15.4
1047
81.0
EfficientNet-B3 [51
12.0
1.8
124.2
25.4
624
81.6
FAT-B2
13.5
2.0
93.4
14.6
1064
81.9
Table 4: Comparison with other methods in efficiency.
Model
Params(M)
FLOPs(G)
Topl-ace(%)
AP
Apm
Swin-S
50
8.7
83.0
44.8
40.9
Focal-S
68
51
9.1
83.5
47.4
42.8
CMT-B
16
46
9.3
84.5
FAT-B4
52
9.3
84.8
49.7
44.8
CSwin-B
12
78
15.0
84.2
MOAT-2
655
73
17.2
84.7
CMT-L [6
75
19.5
84.8
FAT-B5
88
15.1
85.2
Table 5: Comparison with general backbones.
4.6 Ablation Study and Spectral Analysis
In this section, we conduct experiments to understand FAT better. The training settings are the same in previous experiments. More experiments can be found in the appendix.
ImageNet-1K FLOPs(G)
COCO
ADE20K mloU
Model
Params(M)
Top- 1(%)
AP6
Apm
add+linear
4.5
0.72
76.2
39.0
35.8
39.6
cat+linear
4.8
0.77
76.6
39.6
36.3
40.2
mul+linear
4.5
0.72
37.1
40.9
interaction
4.5
0.72
77.6
40.8
37.7
41.5
pool down
4.4
0.71
77.2
40.2
36.9
40.6
conv w/o overlap
4.4
0.71
77.2
40.3
40.8
conv w/ overlap
4.5
0.71
77.3
40.5
37.3
40.9
refined down
4.5
0.72
77.6
40.8
37.7
41.5
w/o conv. pos conv. pos
0.70
77.4
40.5
37.3
41.2
4.5
0.72
77.6
40.8
37.7
41.5
Table 6: Ablation of FAT
Bidirectional Adaptive Interaction. Initially, we validate the efficacy of bidirectional adaptive inter- action by comparing it with three baselines for fusing local and global information: concatenation, addition and element-wiss multiplication. As shown in Tab. 6 can be observed that bidirec- tional adaptive interaction outperforms all baselines significantly across various tasks. Specifically. compared the baseline, which uses the cat+linear to fuse the local and global information, our bidirectional adaptive interaction uses fewer parameters and FLOPs but surpasses the baseline by 1.0% in Top1- Topl-aceuracy. Our bidirectional adaptive interaction also performs better than element-wise multiplication.

---

Fine-Grained Downsampling. We compared our fine ane-grained downsampling strategy with three other strategies: non-overlapping large stride pooling. non-overlapping large stride convolution, and overlapping large stride convolution. As shown in Tab. 6 experimental results across various tasks have demonstrated the effectiveness of our fine- one-grained downsampling strategy. Specifically, our fine-grained downsampling strategy surpasses directly pooling downsampling strategy by 0.4%
Positional Encoding. At the end of Tab. 6 we explore the effect of CPE, and the results in the table show that the flexible CPE also contributes to the performance improvement of the model for 0.2 % image classification.
(a)
(b)
(c)
(d)
Figure 4: Spectral analysis from output channels of FASA. The larger magnitude has a lighter color. Pixels that are closer to the center have a lower frequency. From top to bottom, the results are from (a) local adaptive aggregation, (b) global adaptive aggregation, (c) add+linear fusion, and (d) bidirectional adaptive interaction.
Spectral Analysis. shown in Fig. we conduct spectral analysis on FASA. Compared to add+linear fusion, our bidirectional adaptive interaction can more fully fuse high-frequency and low-frequency information.
Conclusion
In this paper, we propose attention mechanism named Fully Adaptive Self-Attention (FASA), which draws inspiration from the human visual system. FASA is designed to model local and global information adaptively while also accounting for their bidirectional interaction using adaptive strategies. Furthermore, we enhance the self-attention mechanism in FASA by incorporating fine- grained downsampling strategy promote better global perception the flexible FASA as our foundation, we develop a lightweight vision backbone called Fully Adaptive Transformer (FAT), which can be applied across various vision tasks. Our extensive experiments on diverse asks, such as image classification, object detection, instance segmentation, and semantic segmentation, provide compelling evidence of the effectiveness and superiority of FAT. We expect to apply FAT to other vision-related pursuits, including video prediction and vision- lisiun-language models.
detailed level. Using
Acknowledgment
This work is partially funded by National Natural Science Foundation China (Grant No. 62006228. U21B2045, U20A20223), Youth Innovation Promotion Association CAS (Grant No. 2022132), and Beijing Nova Program (20230484276).
10

---

References
[1] Moab Arar, Ariel Shamir, Amit H. Bermano, al. Learned queries for efficient local attention. In CVPR, 2022.
[2] Chun-Fu (Richard) Chen, Rameswar Panda, and Quanfu Fan. Region Regional-to-Local Attention for
Vision Transformers. In ICLR, 2022.
[3] Kai Chen, Jiaqi Wang. Jiangmiao Pang, et al. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 39Xiv:990000755555 2019.
[4] Yinpeng Chen, Xiyang Dai, Dongdong Chen, et al. Mobile-fommer; Bridging mobilenct and transformer.
In CVPR, 2022.
[5] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua
Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, 2021. [6] Xiangxiang Chu, Zhi Tian, Bo Zhang. Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICI.R, 2023.
[7] MMSegmentation Contributors. Mmssgmmentttinn an open source semantic segmentation toolbox. 2020.
[8] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, et al. Randaugment: Practical automated data augmentation with reduced scarch space. In CVPRW, 2020.
[9] Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[10] Mingyu Ding, Bin Xiao, Noel Codella, et al. Davit: Dual attention vision transformers. In ECCV, 2022.
[11] Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han. Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in enns. 2022.
[12] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
[13] Alexey Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Trans- formers for image recognition scale. In ICLR, 2021.
[14] Alaacldin El-Nouby, Hugo Touvron, Mathilde Caron, et al. Xeit: Cross-covariance image transformers. In
NewrIPS, 2021.
[15] Li Gao. Dong Nie, Bo Li, and Xiaofeng Ren. Doubly-fused vit: Fuse information from vision transformer doubly with local representation. In ECCV, 2022.
[16] Jianyuan Gun, Kai Han, Han Wu, Chang Xu, Ychui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. CVPR, 2022.
[17] Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. preprint 220: 09744.9941, 2022.
[18] Kai Han, An Xiao, Enhua Wu, et al. Transformer in transformer. In NeurIPS, 2021.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cmn. In ICCV, 2017.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition.
In CVPR, 2016.
[21] Byeongho Heo, Sangdoo Yun, Dongyoon Han, et al. Rethinking spatial dimensions of vision transformers.
In ICCV, ICCV.2021. 2021.
[22] Qihin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style
convnet for visual recognition. arXiv preprint arXiv:2211.1 0//2/11///443. 1943, 2022.
[23]
Andrew Howard, Mark Sandler, Grace Chu, et al. Searching for mobilenetv3. In ICCV, 2019.
[24]
Gao Huang. Yu Sun, and Zhuang Liu. Deep networks with stochastic depth. In ECCV, 2016.
[25] Huaibo Huang, Xiaoqiang Zhou, and Ran He. Orthogonal transformer: An efficient vision transformer
backbone with token orthogonalization. In NeurIPS, 2022.
[26] Tao Huang, Lang Huang, Shan You, Fci Wang, Chen Qian, and Chang Xu. Lightvit: Towards light-
convolution-free vision transformers. arXiv preprint or 2xX7:2200.055557 2022.
[27] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In
CVPR, 2019.
[28] Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In CVPR, 2022.
[29] Kunchang Li. Yali Wang, Peng Gao, et al. Uniformer: Unified transformer for efficient spatial-temport
representation learning. ICLR, 2022.
[30] Yanyu Li, Geng Yuan, Yang Wen, et al. Efficientformer: Vision transformers at mobilenet speed. In
NeurIPS, 2022.
[31] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint 1rXiv:2104.05700 2021. [32] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, and Kaiming He andPiotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[33] Tsung- Yi Lin, Michael Maire, Serge Belongic, et al. Microsoft coco: Common objects in context. In ECCV, 2014.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[35] Zhuang Liu, Hanzi Mao, Yuan Wu, et al. A convnet for the 2020s. In CVPR, 2022.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In /CLR, 2019.
[37]
Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir,
Rao Muhammad Anwer, and Fahad Shahbaz Khan. Edgenext: Efficiently amalgamated enn cnn-tranfformer architecture for mobile vision applications. In CADL, 2022.

---

[38]
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose. and mobile-
vision transformer. In ICLR, 2022.
[39]
Juhong Min, Yucheng Zhao, Chong Luo, et al. Peripheral vision transformer. In NewrlPS, 2022. Junting Pan, Adrian Bulat, Fuwen Tan, al. Edgevits: Competing light-weight cnns mobile devices with vision transformers. In ECCV, 2022. Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In NeurlPS,
[40]
[41]
2022.
[42] Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in
vision transformers. In AAAI, 2022,
[43] Zhiliang Peng, Wei Huang, Shanzhi Gu, et al. Conformer: Local features coupling global representations for visual recognition. In ICCV, 2021.
[44] Mija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network
design spaces. In CVPR, 2020.
[45] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS, 2021.
[46] Yongming Rao, Wenliang Zhao, Yansong Tang. Jie Zhou, Ser-Lam Lim, and Jiwen Lu. Hornet: Efficient
high-order spatial interactions with recursive gated convolutions. NeurIPS, 2022.
[47] Sucheng Ren, Daquan Zhou, Shengfeng Hc, Jiashi Feng, and Xinchao Wang. Shunted self-attention via
multi-scale token aggregation. In CVPR, 2022.
[48] Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng YAN. Inception transformer. In NeurlPS, 2022.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICI.R, 20155
[50] Christian Szegedy, Wei Liu, Yangqing Jia, et al. Going deeper with convolutions. In CVPR, 2015. Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In
[51]
ICML, 2019.
[52]
Shitao Tang, Jiahui Zhang, Siyu Zhu, et al. Quadtree attention for vision transformers. In ICLR, 2022.
[53]
Ychui Tang. Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is wave: Phase-aware vision mlp. In CVPR, 2022.
[54] Hugo Touvron, Matthicu Cord, Matthijs Douze, al. Training data-cfficient image transformers & distillation through attention. In ICML, 2021.
[55]
Zhengzhong Tu, Hossein Talebi, Han Zhang, et al. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
[56]
Ashish Vaswani, Noam Shazeer. Niki Parmar, et al. Attention is all you need. In NeuriPS, 2017.
[57]
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions.
arXiv preprint arXiv:2103.1 15008, 2021.
[58]
Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media,
8(3):1-10, 2022.
[59]
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A
versatile vision transformer hinging cross-scale attention. In ICLR, 2022.
[60] Haiping Wu. Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. arXiv preprint 2xXiv:2103.155 15808, 2021.
[61]
Zhuofan Xia, Xuran Pan, Shiji Song, Li Frran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, 2022.
[62]
Tete Xiao, Yingeheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene undorstanding. In ECCV, 2018.
[63]
Tete Xiao, Mannat Singh, Eric Mintun, al. Early convolutions help transformers see better. In NeurlPS, 2021.
[64]
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, 2021.
[65]
Chenglin Yang, Siyuan Qiao, Qihang Yu, et al. Moat: Alternating mobile convolution and attention brings strong vision models. In ICLR, 2023.
[66]
Chenglin Yang. Yilin Wang, Jianming Zhang, et al. Lite vision transformer with enhanced self-attention.
In CVPR, 2022.
[67]
Jianwei Yang. Chunyuan Li. Xiyang Dai, and Jianfeng Gao. Focal modulation networks. NewrIPS, 2022. Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal
[68]
self-attention for local-gl local-global interactions in vision transformers, 2021.
[69] Rui Yang. Hailong Ma, Jie Wu, Yansong Tang. Xuefeng Xiao, Min Zheng, and Xiu Li. Scalablevit: Rethinking the context- oriented gemeralization of vision transformer. ECCV, 2022.
[70] Ting Yao, Yingwei Pan, Yehao Li, Chong- Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In Proceedings of the Fumpean conference on computer vision (ECCV), 2022.
[71] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yiehen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng
Yan. Metaformer actually what you need for vision. In CVPR, 2022.
[72] Kun Yuan, Shaopeng Guo, Ziwei Liu, et al. Incorporating convolution designs into visual transformers. In
ICCV, 2021.
12

---

[73] Li Yuan, Yunpeng Chen, Tao Wang, et al. Tokens-to-token vit: Training vision transformers from scratch imagenet. In ICCV, 2021.
[74]
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, et al. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
[75] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, et al. mixup: Beyond empirical risk minimization. In
ICLR. 2018.
[76]
Haokui Zhang. Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware cireular convolution with merits
from convnets and transformer. ECCV, 2022.
[77]
Pengchuan Zhang, Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale
vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, 2021.
[78]
Qinglong Zhang and hin Yang. Rest: efficient transformer for visual recognition.
NeurlPS, 2021. Random erasing data augmentation. In AAAI. 2020.
[79]
Zhun Zhong, Liang Zheng, Guoliang Kang,
180
Bolci Zhou, Hang Zhao, Xavier Puig,
Scene parsing through ade20k datasct. CVPR, 2017.
[81]
Daquan Zhou, Zhiding Yu, Enze Xie, al. Understanding the robustness vision transformers. In ICML,,
2022.
[82]
Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio-visual learning: survey.
UAC, 18:351, 2021. Zhuofan Zong. Kunchang Li, Guanglu Song, et al. Self-slimmed vision transformer. ECCV, 2022.
[83]
13

---

Appendix
A Discussion About "Interaction" And "Fusion"
Generally speaking, interaction refers the process of each other. In contrast, fusion refers to combining two or more things into a single entity. As shown in Fig. 5 in contrast to all previous feature fusion methods, our proposed bidirectional adaptive interaction stands out in two significant ways:
objects systems affecting
local token
global token
Iadamard product
cat& linear
Bidirectional: 7
Bidirectional: [X]
Adaptive:
Adaptive:
(a) interaction
(b) fusion
Figure 5: Comparison between our bidirectional adaptive interaction and traditional fusion method.
1) Bidirectional Property: Our approach draws inspiration from how visual information flows bidirectionally through the human visual system. Before fusing local and global features, we model a bidirectional process between the two. Specifically, for local features, generate weights that encompass global information. These weights are applied to the local features, allowing them have the ability capture the global context. We execute similar process for global features. This results in strengthened local and global features, respectively. In contrast to linear fusion, our interaction returns enhanced vectors representing local and global contexts rather than an inseparable vector mixed with both
utilize global features to
example, the global weights them are not trainable. They are generated by tokens containing global features, making them context- aware and capable of adapting the input data. Moreover, when fusing the enhanced local and global features, we do not introduce any additional parameters but calculate their element- wise product (Hadamard product) directly. Compared with linear fusion, context-awarr fusion approach is closer to the attention mechanism [56] [7] 22]. Our ablation experiments have shown that this fusion approach not only comparison between the Add+Linear, Cat+Linear, and our model, please refer to the paper's 's main text.
2) Adaptive Capability: Taking local features
to enhance
parameters but also achieves better results. Specifically, for detailed
The implementation of our proposed bidirectional adaptive interaction method very simple, yet it has achieved remarkable results compared Through this work, we aim reveal the critical role that features fusion plays in visual models and hope to inspire more related research.
to other methods for fusing local and global features.
B
More experiments
evaluate our backbones with framework of UperNet 622 based on MMsegmentation 7
Following the [34, we adopt AdamW to optimize the model for 160K iterations. As shown in Tab.
our model surpasses all counterparts and achieves the best performance. Specifically, our FAT-B3
surpass the resent Shunted-S by +0.7 mloU and +0.8 MS mloU.
14

---

Backbone
Params(M)
FL.OPs()
ml)U(%)
MS ml)U(%)
Swin-T
945
44.5
45,8
DAT-T 61
60
957
45.5
49.0
DaViT-T 10
60
940
46.3
UniFormer-S 29
52
955
47.0
48.5
CrossFormer-S [59
62
980
47.6
48.4
MPViT-S 28
52
943
48.3
Ortho-S 225
54
956
48.5
49.9
Shunted-S [47
52
940
48.9
49.9
FAT-B3
59
936
49.6
50.7
Table 7: Results with the framework of UperNet ADE20K. The FLOPs measured with the resolution of 512 x 2048.
Output Size Layer Name
FAT-BO
FAT-B1
FAT-B2
FAT-B3
3 x 16,
3
3 3. 32,
3 3,
32,
3 x 3, 32, s2
3 3.
48,448,4
3 x
64,
3
x
3,
64,
W
Stem
3
48.
61.
3 x 3, 32,
3
3, 48,
3 x
61,
3 x
3, 64,
s8
64,
CPE FASA ConvFFN
3.32
3 3,48
3 64
4
kernel 3, heads 2
x
kernel 3, heads 3 5,
x
kernel 3, heads
x
kemel 3, heads 2
5,
CPR
5.
5 5,96
5 5, 128 kernel hends
5 x 128
#
8
W
FASA ConvFEN
kernel 5, heads
30
x
kernel 5, heads 6
5 5,96
x
5, 128
x
kemel 5, heads
5 128
x
CPE
1600
7 192
7 7. 256 kernel heads 5 5, 256
x 256
II 75
FASA
kernel 7, heads 10
x
kernel 7, hends 12 192
x
x
kernel 7, heads
x 16
ConvFEN
x 160
5 256
CPE
x 256
9,384
9 x 512
9 9,512
92
N
W
FASA
kernel 9, heads 16
x
kernel heads 24
x
kernel 9, heads 5,512
x
kernel 9, heads 16
x
ConvFFN
256
384
5 x 5,512
1
Classifier
Fully Connected Layer, 1000
Params
4.5M
7.8M
13.5M
29M
FLOPS
0.7G
1.2G
2.0G
4.4G
Table Details about FAT's architecture.
C Implementation Details
C.1 Architecture Details
The detailed architectures shown in Tab. where all FLOPs measured at the resolution of 224 x 224 for image classification. For the convolution stem, we adopt four x 3 convolutions and one x convolution to tokenize the image. Batch normalization and ReL.U used after each x convolution, and layer normalization is used after the final x convolution. Inspired by the fact that in the vision backbone, the early stages tend capture high-frequency local information, and the later stages tend to capture low-frequency global information, we gradually increase the DWConv's kernel sizes used in FASA. Specifically, set the kernel size 3 in the first stage and 9 in the last stage. The expansion ratios are set to 4, and the kernel sizes are set to 5 for all ConyFFN layers.
C.2 Image Classification
We follow the same training strategy with Swin- Siin-Tranffommer 34 All models are trained for 300 epochs from scratch. To train the models, we use the AdamW optimizer with cosine decay learning rate scheduler and 20 epoch linear warm-up. We set the initial learning rate, weight decay, and batch size to 0.001, 0.05, and 1024, respectively. We adopt the strong data augmentation and regularization used in 34 Our settings are Runment g
Mixup 75 (prob=0.8), Cut- Mix 74 (prob=1.0), Random Frasing [79] (prob=0.25), increasing stochastic depth 24 (prob=0.05,
0. 0.1, 0.15 for FAT-BO, FAT-BI, FAT-B2 and FAT-B3).
15

---

C.3 Object Detection and Instance Segmentation
We adopt MMDetection 3 implement RetinaNet and Mask-RCNN 19
We use the commonly used "1x" x (12 training epochs) setting for the two strategies. Following [34]. during training, images are resized to the shorter side of 800 pixels while the longer side within 1333 pixels. We adopt the AdamW W optimizer with a learning rate of 0.0001, weight decay of 0.05, and batch size of to optimize the model. The learning rate declines with the decay rate of 0.1 the epoch 8 and 11.
C.4 Semantic Segmentation
We adopt the Semantic FPN [27 and UperNet [62] based on MMSegmentation and apply FAT pretrained on ImageNet-IK I as backbone. We use the same setting of PVT (57] to train the Semantic FPN, and input resolution of 512 x 512. When testing the model, we resize the shorter side of the image to 512 pixels. As for UperNet, we follow the default settings in Focal Transformer [68]. We take AdamW with weight decay of 0,01 as the optimizer and train the models for 160K iterations. The learning rate is set 10 with 1500 iterations warmup.
the "1 x" schedule (trained for 80k iterations). All the models trained with the
D More Ablation and Analysis Results
Model
Blocks
Channels
Params(M)
FLOPs(G)
Top-1 acc(
Swin-T
34
[2, 2, 6,
[96, 192, 384, 768]
29
4.5
81.3
DAT-T 61
[2, 6,
[96, 192, 384, 768]
29
4.6
82.0
FocalNet-T 62
[2, 2, 6,
196, 192, 384, 768]
28
4.4
82.1
Focal-T [68
12 6,
196, 192, 384, 768]
29
4.9
82.2
FAT-B3-ST
[2, 2]
[96, 192, 384, 768]
29
4.7
83.0
Table 9: Comparison with four baseline models when use the same layout with Swin-T.
Other architecture design choices. We modified FAT-B3 to have the same layout as Swin-T and compared it to four baseline models: Swin-T [34]. DAT-T 611, FocalNet-T [67 and Focal-T (68).
As shown in Tab. 9 our model demonstrates significant superiority, with accuracy 1.7% higher than Swin-T and 0.8% higher than Focal-Tiny.
Params (M)
FLOPs
Topp-ace
Method
Contribution Isolation. To better isolate the con- tribution of our FASA, we compare the differ- ent self-attention mechanisms bone. We choose five kinds self-attention mech- anisms to make the comparison: CSWSA in CSwin- Transformer 12]. Max-SA in Max VIT [55]. WSA/S- WSA in Swin-Traasformee (34] SRA in PVT [57
(G)
(%)
Max-SA 55
4.3
0.8
75.7
the same back-
WSA/S- W
34
4.3
0.8
76.1
SRA (7
4.4
0.7
76.2
LSA/GSA (5
4.3
0.7
76.6
CSWSA 12
4.3
0.8
76.8
FASA
4.5
0.7
77.6
and LSA/GSA in Twins-SVT 55) The results are
shown in Tab. TO The networks' layout and the setting of CPE and ConvFFN are the same as FAT- BO. It can be found that our FASA surpasses all its counterparts. Compared to the latest Max-SA, our FASA achieved gain of 1.9%.
Table 10: Comparison among different self- attention mechanisms.
Local Aggregation in FASA. To validate that the model's iss capability to perceive local context does not solely rely CPE and ConvFFN, we conducted ex- periments by removing local adaptive aggregation and bidirectional adaptive interaction from FASA, retain- ing only global adaptive aggregation. The results pre- sented in Tab. II demonstrate significant decrease in model performance (1.4%) when using only global adaptive aggregation. This outcome convincingly supports the soundness of our proposed FASA model.
local aggregation
Params (M)
FLOPs
Topl-acc (%)
(G)
w/o local
4.4
0.70
76.2
local
4.5
0.72
77.6
Table 11: Ablation of kernel size used in FASA.
16

---

Additional Sigmoid. In our main text, in the Method section's Bidirectional Adaptive Interaction subsection, Eq. 11 omits the high-order variable:
Additional Sigmoid
Params FLOPs Throughput Topl-acc (M)
(G)
(imgs/s)
(%)
Sigmoid
4.5
0.72
1815
77.6
w/o Sigmoid
4.5
0.72
1932
77.6
1
Table 12: Ablation of addtional Sigmoid.
1 +te
1tc
in the final expression faster implementation. This variable represents concatenated Sigmoid activation functions within the local adaptive aggregation phase. We attempt to include the previously omitted activation function in this section but find that having too many activation functions results in This result confirms the rationality of
a similar performance but slower speed, as shown in Tab. [2 the structure we designed.
Kernel Size of FASA. Inspired by the notion that early stages of vision backbones tend to aggregate lower-level local features, while later stages prefer higher-level seman- tic ones, we adopt a staged approach and incrementally increase the DWConv's kernel size in FASA aggregate local features. In this section, we investigate how DW- Conv's kernel size in FASA affects the model's image classification performance by varying the kernel sizes in different stages, as depicted in Tab. [3] Our experimental results demonstrate the effectiveness of our strategy for gradually increasing kernel size at different stages. Models
kerell sizes
Params (M)
FLOPs
Topl-acc
(G)
(%)
4.3
0.697
77.3
3.5.7.7
4.5
0.717
77.6
7.9.7.
4.5
0.718
77.6
7.7.77
4.5
0.737
77.5
9, 9.9
4.6
0.770
77.4
Table 13: Ablation of kernel size used in FASA.
using kernel sizes of 3, 5, 7, 9, and 3, 5, 7, achieved the best performance.
Convolution Stem. We use the convolution stem pro- posed in 63 To investigate the impact of convolution stem on our model's performance, we conducted experi- ments and presented the results in Tab. [4] Despite incur- ring some extra computational overhead, using convolu- tion can significantly enhance the topl accuracy of the model by 0.4%.
conv stem
Params (M)
FLOPs
Topl-acc
(G)
(%)
w/o stem
4.5
0.65
77.2
stem
4.5
0.72
77.6
Table 14: Ablation convolution stem.
Kernel Size of ConyFFN. We conduct ablation exper- iments validate the effect of different kernel sizes for DWConv in ConvFFN. As shown in Tab. [5] the kernel size of 7 x 7 achieves the best performance, but it sumes more parameters (+0.2M) and FLOPs (+0.03G). To achieve better trade-off between the performance and computational complexity, We set the kernel size of DWConv ConvFFN 5.
kernel sizes
Params (M)
FLOPs
Topl-acc
(G)
(%)
x
4.4
0.69
77.4
5
x
5
4.5
0.72
77.6
7
4.7
0.75
77.7
Table 15: Ablation of kernel size used in ConvFFN.
Bidirectional Interaction. Here ablate the local-to- global and global-to lo-caal,,,, respectively, to show the effect of bidirectional interaction. The results are shown in the Tab. 16 It can be that the bidirectional interaction plays important role in FAT.
E
Speed Measurement
Params (M)
FLOPs
Topl-acc (%)
Model
(G)
In order accelerate the inference speed of our model, we utilized toolkit devel- oped for accelerating large kernel DW- Conv in [1I] Additionally, we em- ployed the technique of structural re- parameterization to merge the residual blocks in ConvFFN into one convolu- tional block. Furthermore, after omitting
no interaction w/o global to local
4.5
0.72
76.2
4.5
0.72
76.9
w/o local to global
4.5
0.72
76.8
bidirectional interaction
4.5
0.72
77.6
Table 16: Ablation of bidirectional interaction.
17

---

the higher-order variables in bidirectional adaptive interaction, the original expression can be sim- plified to two SiLU activations followed by the multiplication of two branches. These measures significantly accelerated the inference speed of our model. Here, we list the most recent models in Tab. 17 As can be seen from the table, our model performs the best in terms of both speed and performance.
Throughput (igg/ss
Topl-Acc (%)
Throughput (imgs/s)
Topl-Ace (%)
Model
Year
Model
Year
1762
71.7
ICCV"2021
QuadTree- B bi 52
885
72.0
ICLR'2022
Poolformer S36
71
601
81.4
CVPR'2022
TNT:Tiny
[X]
545
73.9
NeurIPS2022
CrossFormer-T
929
81.5
ICLR 2022
EdgeVIT-XXS
40
1926
74.4
ECCV'22022
EfticientNet
634
81.6
ICMI'2019
LVT 66
1265
74.8
CVPR'2022
ResT Base 77
588
81.6
NeurIPS'2021
1367
74.8
EdgeNeXi-XS 37
VAN-RO
1417
1662
75.4
75.0
ECCYW"2022
Arsiv'2022
Coat-Lite-S [4]
540
81.9
CVPR'2022
ICCV
PoolFormerSS1 71
XCTT-T12
ResT-Lite
Vil.Tiny tr
FAT-BO
78
1123
1932
1722
857
77.6
16.7
77.2
77.1
77.2
NeurIPS'0221
NeurIPS
CVPR'2022
ICCV'2021
Ours
FocalNet-T 67
PVTv2-B2
ViL-S 77
PTT.B X
610
582
601
328
359
82.0
82.0
82.1
82.4
NeurlPS 2022
Arxiv 2022
ICCV"2021
ICCV'2021
ICLR'2U22
T21-VIT-12
73
1307
76.5
1CCV'2021
RegionViT-S
460
82.6
ICLR 2022
EdgeVIT-XS
80
1528
77.5
ECCV
WaveMLPSS
599
82.6
CVPR
CosT-Lite-T
64
1045
77.5
1CCV
CSWin-T 12
591
82.7
CVPR'2022
SIT w/o FRD
83
1057
77.7
ECCV'2022
WaveViT-S 70
482
82.7
BCCV'2022
RegNetY-1.6CFF
CVPR'2020
MPVII-T 28
737
78.2
2VP2:2022
DaVIT-T 10
616
82.8
BCCV'2022
MobileViT-S
898
78.4
ICLR'2022
HorNet-T
16
586
82.8
NeurIPS/2022
ParC-Ner-S 76
1321
78.6
ECCV'2022
FAN-S-VIT
525
82.9
ICML'2022
PVTv2-B1
1007
78.7
Arxiv'0022
FAT-B3-ST
641
83.0
Ours
NeurPS'2022
299
ICML'2021
EMicientNet-B1
CoaT-Lite-Mi (64
1395
963
79.1
79.2
ICCV'2021
ICML'2019
PoolForme--MH
Focal 68
304
301
82.2
NeurIPS:2021
CVPR'2022
EdgeNeXi-S
FAN-T-ViT
XCTT-T24
37
1181
1243
79
79.2
ECCVW-2022
NeurPS'2022
ICML'2022
QuadTree-Baa R-h2 52
Shunted 47
456
299
82.9
82.7
CVPR
ICLR 2022
FAT-BI
1452
80.1
Oues
Swin 34
410
390
83.0
ICCV-2001
ResT.S 78
918
79.6
NeurPP'2022
ConxNeXt-8
405
83.1
Shunted
957
79.8
CVPR 2022
PVTv2-B3
398
83.2
Anxiv 2022
DeiT-S 54
899
79.9
ICML:2021
RegionVIT-B B
256
83.2
JCLR'2022
ICI.R'2022
EfficienFFomme...
NeurlPS'
R5 2
710
80.4
ICLR'2022
LITv2-M M
4
436
83.3
NeurIPS-2022
WaveMI P-T
1052
80.6
CVPR2022
FocalNei-S
X
358
83.4
NeurlPS'200
PTTS 21
1042
80.9
ICCV'2001
iFormer-S
48
471
83.4
NeurlPS 2022
MPVIT-XS 28
597
80.9
CVPR'2022
CrossFormer- 59
368
83.4
ICLR 2022
EdgeViTTS
1049
ECCV'2022
WaveML.P.M
CVPR'2022
VAN-B1 [
995
81.1
Araiv '2022
FXT-B3
474
83.6
Ours
Table 17: Comparisons of speed and performance among different models. All speeds are measured
on the V100 32G with the batch size of 64.
Limitations and Broader Impacts
While our FAT achieves superior performance with fast processing speed, one limitation of our work is that due to computational resource constraints, we do not apply our lightweight vision backbone to unsupervised learning, video processing, visual language tasks. Additionally, our models on large-scale datasets like ImageNet-2IK: However, we look forward to exploring more applications of our proposed FAT the future.
do pretrain
This study is purely academic in nature, and we are unaware of any negative social impact resulting directly from our work. However, acknowledge that models' potential malicious use is concern affecting the entire field. Discussions related to this matter are beyond the scope of our research.
18


Reviews:
[{"review_id": "f0JRfSIWvH", "reviewer": "Reviewer_c9kv", "summary": "This paper proposes a new family of light-weight vision transformers named FAT, which enhances the local and global feature fusion by a bi-directional interaction between them. Experiments on image classification, object detection, and semantic segmentation are conducted.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a good practice in designing high performance efficient ViTs, which could be a valuable reference for the future work.\n\n2. The results on several benchmark datasets outperform all the compared models.", "weaknesses": "1. An incremental work. There exists tons of works that focus on performing local and global features in ViTs (see Sec. 2 of the paper), this paper only proposes a simple way to enhance the feature fusion. Besides, other increments are actually a combination of existing successful practices. For example, the convolutional and non-overlapping downsample module is from PVTv2, the downsample in MHSA have also been widely used (e.g., PVT), CPE is from [1].\n\n2. Actually, a CVPR 2022 paper MixFormer [2] also proposes a bi-directional interaction of convolutional and self-attention branches. What's the pure difference between this method and MixFormer?\n\n3. It is difficult for me to evaluate the efficacy of the proposed bi-directional interactions. The paper only compares the method with simple fusion baselines, while there exist many global-local feature fusion methods, I cannot figure out which one is better according to Table 1, since many increments are added in this method. I suggest the authors to replace the fusion method in FAT with other popular methods such as the interactions in MixFormer and message token in MSG-Transformer [3], for fair comparisons.\n\n---  \n**References**\n\n[1] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR, 2023.\n\n[2] Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J. and Wang, J., 2022. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5249-5259).\n\n[3] Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W. and Tian, Q., 2022. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12063-12072).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **On Novelty, Difference with MixFormer and Efficacy of Bi-directional interaction**\n\nWe thank the reviewer for recognizing the positive aspects of our paper, and we will address the reviewer's concerns in the following parts.\n\n**Q1: Novelty.**\n\n**R1:** Thanks. We want to highlight that the key novelty is that we construct a novel module for local-global fusion, which is simple to implement and has negligible costs. Our bidirectional interaction is inspired by the existing research on the human brain [9, 10]. In contrast to all previous feature fusion methods, our bidirectional adaptive interaction stands out in two significant ways: \n\n1. Bidirectional Property: We model a bidirectional process between the two before fusing local and global features. This results in strengthened local and global features, respectively. In contrast to commonly used linear fusion, our interaction returns two enhanced vectors representing local and global contexts rather than an inseparable vector mixed with both. \n2. Adaptive Capability: Taking local features as an example, the global weights we use to enhance them are generated by tokens containing global features, making them context-aware and capable of adapting to the input data. Moreover, we calculate their element-wise product directly when fusing the enhanced local and global features. Our context-aware fusion approach is closer to the attention mechanism than linear fusion. \n\n**Q2: Difference with MixFormer [1].**\n\n**R2:** Thanks. We believe that the key differences between our FAT and MixFormer can be summarized in the following three points:\n\n1. Different motivations: In FAT, the proposed FASA aims to establish channels for information flow from local to global and global to local, allowing local features to perceive global information and refine global features based on local features. On the other hand, MixFormer is motivated by establishing channels for information flow between an image's spatial and channel dimensions. Information flows in MixFormer enable complementary interaction between window attention (with shared parameters in the channel dimension and weak channel representation) and DWConv (with shared parameters in the spatial dimension and weak spatial representation). \n\n2. Different branches: Both branches of MixFormer perceive local features, while in FASA, we use a branch for local perception and another for global. Additionally, the limitation in MixFormer [1]'s paper mentions that replacing the window attention branch with global attention would lead to a performance decline, contrary to our model's approach, as shown in the table below.\n\n   | Model| Params(M)| FLOPs(G)|Top1-acc(%)|\n   |--|--|--|--|\n   |Conv + Global attention|4.5|0.72|77.6|\n   |Conv + Window attention|4.4|0.72|75.8|\n\n   \n\n3. Different means of fusing information: Compared to FAT, MixFormer introduces more parameters and FLOPs for better representations in the processes of spatial and channel interaction. Comparison can be found in **Q3**. It can be found that FASA gets better performance (+1.1%) with fewer FLOPs (-0.25G )and Params (-1.7M).\n\n**Q3: Efficacy of  bi-directional interaction**\n\n**R3:** Thanks. We compare our FASA with eight baselines, including the module with local-global fusion (MSG in MSG-Transformer [2], Mix Block in MixFormer [1], R2L in RegionViT [3]) and other self-attention mechanisms with good performance (CSWSA in CSwin-Transformer [4], Max-SA in MaxViT [5], WSA/S-WSA in Swin-Transformer [6], SRA in PVT [7] and LSA/GSA in Twins-SVT [8]). In all baselines, we use the same architecture but only modify the token mixer. The results in the below table can demonstrate the superiority of FASA.\n\n| Model|Params(M)|FLOPs(G)|Top1-acc(%)|\n|--|--|--|--|\n| MSG [2]|4.4|0.77|76.3|\n| Mix Block [1]|6.2|0.97|76.5|\n| R2L [3]|4.8|0.76|76.2|\n| CSWSA  [4]|4.3|0.78|76.8|\n| Max-SA [5]|4.3|0.77|75.7|\n| WSA/S-WSA [6]|4.3|0.76|76.1|\n| SRA [7]|4.4|0.71|76.2|\n| LSA/GSA [8]|4.3|0.71|76.6|\n| FASA|4.5|0.72|**77.6**|\n\n\n\n**Reference**\n\n[1] Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J. and Wang, J. Mixformer: Mixing features across windows and dimensions. In CVPR, 2022.\n\n[2Jiemin Fang , Lingxi Xie, Xinggang Wang .et, al. MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens. In CVPR, 2022.\n\n[3]Chun-Fu (Richard) Chen, Rameswar Panda, and Quanfu Fan. RegionViT: Regional-to-Local Attention for Vision Transformers. In ICLR, 2022.\n\n[4]Xiaoyi Dong, Jianmin Bao, Dongdong Chen, et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.\n\n[5]Zhengzhong Tu, Hossein Talebi, Han Zhang, et al. Maxvit: Multi-axis vision transformer. In *ECCV*, 2022.\n\n[6]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In *ICCV*, 2021.\n\n[7]Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2103.15808, 2021.\n\n[8]Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In *NeurIPS*, 2021.\n\n[9]Schwarzkopf D. Samuel and Rees Geraint. 2011 Interpreting local visual features as a global shape requires awareness  Proc.R. Soc. B.278: 2207–2215. doi/10.1098/rspb.2010.1909\n\n[10]Mannion DJ, Kersten DJ, Olman CA. Regions of mid-level human visual cortex sensitive to the global coherence of local image patches. J Cogn Neurosci. 2014 Aug;26(8):1764-74. doi: 10.1162/jocn_a_00588. Epub 2014 Feb 24. PMID: 24564470; PMCID: PMC4074231.\n\n"}, {"review_id": "mJkG488VNQ", "reviewer": "Reviewer_DKrb", "summary": "This paper presented an efficient vision transformer backbone for several tasks including classification, segmentation and detection. The key idea of this paper is on a new design of considering the interaction between local and global features.  The experiments on imagenet, AD20K, and COCO  have shown that the proposed method is comparable to most existing methods.  ", "questions": "The experiments are not fair comparison as some of the numbers are cherry picked.  It will be more convincing to compare with Segnext, ParC-Net, MobileOne and mobileVIT clearly.  As far as the reviewer can see, the proposed method is comparable or weaker than them. \n\nIn addition, the ablation studies didn't show the bidirectional interaction is useful or not. For example, given single interaction will it be OK?", "limitations": "The key limitation of this paper is the proposed method lack of evidence on whether it can be mobile-friendly or scaled up as VIT liked backbones.   Since the backbone is not general enough, it will has very limited usage. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper has clear contributions on the vision transformer design.\n1.  The proposed the local-global feature interaction is well motivated. \n2.  The proposed methods has been evaluated with comprehensive experiments. \n3. This paper is well written and easy to understand. ", "weaknesses": "There are some weaknesses in this paper:\n\n1. The area of new backbone is growing rapidly.  Although the proposed method shows marginally improvements over the experimented baselines, compared with concurrent work, it becomes not appealing. \n2.  The proposed lightvision transformer is very hard to justify as it is not as efficient as mobile vit but also not sure how it can be scaled up. \nThus it is very hard to justify. \n3.  The baseline methods are not consistent.  For example Table 1 and Table 2 pick different methods for similar comparison. For example, ParC-Net should be put into 0~5 while the author intentionally put into different comparison setting to make the number look better. \n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **On comparison, efficiency, ablation study and scale-up capability**\n\nWe thank the reviewer for recognizing the positive aspects of our work. We will address the reviewer's concerns in the following parts.\n\n**Q1:** Comparison in the paper.\n\n**R1:** Thanks. The number of parameters and the computational complexity (i.e., FLOPs) are essential when evaluating a model's size. These two indicators are equally significant. However, for mobile-friendly models, the speed (inference throughput) should also be taken into consideration. In our paper, we follow the above rules when grouping the data, primarily based on the Params and FLOPs, with inference throughput as a secondary factor. For example, ParC-Net-S (5.0M, 1.7GFLOPs) has small Params similar to FAT-B0 (4.5M, 0.7GFLOPs). However, its FLOPs are much higher than those of FAT-B0, B1 (7.8M, 1.2GFLOPs), and close to B2 (13.5M, 2.0GFLOPs). Therefore, considering both the Params and FLOPs, the most reasonable choice is to compare ParC-Net-S with FAT-B1 (5.0M, 1.7GFLOPs v.s. 7.8M, 1.2GFLOPs). So we think we haven't cherry-picked the data. But for a more rigorous comparison, we group the models strictly based on the number of parameters. The updated table can be found in the global response, and we invite the reviewer to read it. It is important to note that due to the high computational complexity and critical parameter count of ParC-Net-S, we still compare it with FAT-B1. The results in global response also show the superiority of FAT.\n\n**Q2:** Comparison with mobilevit on efficiency.\n\n**R2:** Thanks. The results of Table 1 in our paper indicate that our FATs have higher accuracy and inference throughput than the recent mobile architecture, such as MobileViT, EdgeViT, EdgeNext, and more. We invite the reviewer to read the **Q5** for more comparison. Compared to mobilevit, FAT has the advantage in efficiency (FLOPs, inference throughput, and inference latency) and accuracy.\n\n**Q3:** Whether the bidirectional interaction is useful.\n\n**R3:** Thanks. Here we ablate the local-to-global and global-to-local, respectively, to show the effect of bidirectional interaction.    The results are shown in the below table. It can be seen that the bidirectional interaction plays an important role in FAT.\n\n   |Model|Params(M)|FLOPs(G)|Top1-acc(%)|\n   |-|-|-|-|\n   |no interaction|4.5|0.72|76.2|\n   |w/o global to local|4.5|0.72|76.9|\n   |w/o local to global|4.5|0.72|76.8|\n   |bidirectional interaction|4.5|0.72|77.6|\n\n\n**Q4:** How FAT can scale up.\n\n**R4:** Thanks. We scale up our FAT to common backbone sizes (29M, 4.4GFLOPs, 52M, 9.1GFLOPs, and 88M, 15.1GFLOPs). The results are shown below. We use the framework of Mask-RCNN with 1x schedule to conduct the object detection and instance segmentation. It can be seen that FAT has a good ability to scale up. In addition, our model also demonstrates significant advantages in downstream tasks.\n\n|Model|Params(M)|FLOPs(G)|Top1-acc(%)|$AP^b$|$AP^m$|\n|-|-|-|-|-|-|\n|Swin-T|29|4.5|81.3|42.2|39.1|\n|UniFormer-S|24|4.2|82.9|45.6|41.6|\n|MOAT-0 [1]|28|5.7|83.3|--|--|\n|CMT-S [2]|25|4.0|83.5|44.6|40.7|\n|FAT-B3|29|4.4|**83.6**|**47.6**|**43.1**|\n|||||||\n|||||||\n|Swin-S|50|8.7|83.0|44.8|40.9|\n|Focal-S|51|9.1|83.5|47.4|42.8|\n|UniFormer-B|50|8.3|83.9|47.4|43.1|\n|MOAT-1 [1]|42|9.1|84.2|-|-|\n|CMT-B [2]|46|9.3|84.5|-|-|\n|FAT-B4|52|9.3|**84.8**|**49.7**|**44.8**|\n|||||||\n|||||||\n|CSwin-B|78 |15.0|84.2|-|-|\n|MOAT-2 [1]|73|17.2|84.7|-|-|\n|CMT-L [2]|75|19.5|84.8|-|-|\n|iFormer-L |87|14.0|84.8|-|-|\n|FAT-B5|88|15.1|**85.2**|-|-|\n\n**Q5:** Details comparison with other methods (mobilevit, mobileone, Parc-net, SegNext...).\n\n**R5:** Thanks, for lightweight backbones, we compare their **inference latency** on GPU/CPU and **inference throughput** on GPU, CPU is Intel i9Core, GPU is V100. **Inference throughput** is measured with batch size 64. **Inference latency** is measured with batch size 1. It can be seen that our FAT has the best speed/performance trade-off among lightweight Transformers.\n\n|Model|Params(M)|FLOPs(G)$\\downarrow$|CPU(ms)$\\downarrow$|GPU(ms)$\\downarrow$|Trp(imgs/s)$\\uparrow$|Top1-acc(%)|\n|-|-|-|-|-|-|-|\n|EdgeViT-XXS|4.1|0.6|43.0|14.2|1926|74.4|\n|MobileViT-XS|2.3|1.1|100.2|15.6|1367|74.8|\n|EdgeNext-XS|2.3|0.5|52.3|13.8|1417|75.0|\n|tiny-MOAT-0|3.4|0.8|61.1|14.7|1908|75.5|\n|MobileOne-S1 [3]|4.8|0.8|38.1|5.8|3265|75.9|\n|EfficientNet-B0|5.3|0.4|42.4|14.5|2181|77.1|\n|FAT-B0|4.5|0.7|44.3|14.4|1932|**77.6**|\n||||||||\n||||||||\n|MobileOne-S2 [3]|7.8|1.3|56.8|6.9|2312|77.4|\n|EdgeViT-XS|6.7|1.1|62.7|15.7|1528|77.5|\n|tiny-MOAT-1|5.1|1.2|80.8|14.8|1506|78.3|\n|MobileViT-S|5.6|2.0|135.6|16.2|898|78.4|\n|ParC-Net-S|5.0|1.7|112.1|15.8|1321|78.6|\n|EfficientNet-B1|7.8|0.7|61.3|18.5|1395|79.2|\n|EdgeNext-S|5.6|1.3|86.4|14.2|1243|79.4|\n|FAT-B1|7.8|1.2|62.6|14.5|1452|**80.1**|\n||||||||\n||||||||\n|MobileOne-S4 [3]|14.8|3.0|110.2|9.8|1225|79.4|\n|ParC-ResNet50|23.7|4.0|160.0|16.6|1039|79.6|\n|EdgeViT-S|11.1|1.9|95.8|16.2|1049|81.0|\n|tiny-MOAT-2|9.8|2.3|122.1|15.4|1047|81.0|\n|EfficientNet-B3|12.0|1.8|124.2|25.4|624|81.6|\n|FAT-B2|13.5|2.0|93.4|14.6|1064|**81.9**|\n\nAs for SegNext, we compare it with our general backbone. It can be seen that FAT is better than SegNext.\n\n|Model|Params(M)|FLOPs(G)|Top1-acc(%)|\n|-|-|-|-|\n|SegNext-T [4]|4.2|0.7|75.9|\n|FAT-B0|4.5|0.7|**77.6**|\n|SegNext-S [4]|14.0|2.2|81.2|\n|FAT-B2|13.5|2.0|**81.9**|\n|SegNext-B [4]|27|4.5|83.0|\n|FAT-B3|29|4.4|**83.6**|\n|SegNext-L [4]|45|8.2|83.9|\n|FAT-B4|52|9.3|**84.8**|\n\n**Reference**\n\n[1]Chenglin Yang, et al. Moat: alternating mobile convolution and attention brings strong vision models. ICLR, 2023.\n\n[2]Jianyuan Guo, et al. Cmt: Convolutional neural networks meet vision transformers. CVPR, 2022.\n\n[3]Pavan. K, et al. MobileOne: An Improved One millisecond Mobile Backbone. arXiv:2206.04040, 2022.\n\n[4]Meng-Hao Guo, et al. SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation, NeurIPS, 2022."}, {"review_id": "xAMVAZUPZ7", "reviewer": "Reviewer_cvP4", "summary": "This paper presents a new family of lightweight vision transformers that are built on the Fully Adaptive Self-Attention module. The key innovation of these transformers lies in their bidirectional interactive module, which enhances both local and global features. The experiments primarily focus on image classification, object detection, and semantic segmentation tasks. Extensive evaluations were conducted on various common benchmarks, including ImageNet-1K, to assess the performance of the proposed transformers.", "questions": "1. My first two questions are motivated by different considerations, and I recommend that the author concentrate on one aspect to augment the method's significance. Given the exceptional performance of the proposed model on small-scale counterparts, my particular interest lies in contrasting the inference across different hardware and CNN architectures. Nevertheless, should the author contend that the proposed model should possess broader applicability, I am also intrigued to observe its scaling behavior.\n2. I am interested in investigating the training stability of the proposed methods. It would be helpful if there are any plots illustrating the training stability.\n", "limitations": "The author has discussed their limitation in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method demonstrates a key innovation through bidirectional interaction between the global and local branches, leading to significant improvement as validated by comprehensive ablation studies. The study includes extensive experiments, and the ablation analysis is thorough, highlighting the quality of the research. The writing is clear and concise, effectively presenting the experimental setting and demonstration. Impressively, FAT-B0 and B1 achieve remarkable results in terms of both throughput and accuracy, providing valuable insights for future research on low-latency small models.", "weaknesses": "1. The scaling behavior of the method is my primary concern. While I appreciate the author's focus, the current results indicate that larger models exhibit limited improvement over the baseline in the three tasks. In my opinion, it is essential to discuss the scaling aspect for a proposed general backbone.\n2. The second issue is the lack of comparison in terms of inference time between different methods or different hardware (e.g. CPU/GPU). Based on the current results, the proposed model works very well under low parameter settings, providing the direct benefit of lighter and more efficient deployment on mobile devices. However, the author has not provided sufficient evidence to demonstrate whether the proposed method maintains efficient inference efficiency with such low computational requirements. This aspect is crucial for me in proposing a lightweight model because training speed is often not a concern for these types of models. Instead, focusing on comparing the capabilities of this model under different hardware and computational costs can significantly enhance the significance of this method.\n3. The motivation for this paper appears somewhat forced, particularly from Figure 1. I have not come across any related work discussing this phenomenon. If any such works exist, please cite them, as it would enhance the trustworthiness of your motivation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **On scale-up capability, efficiency, motivation, and training stability**\n\nWe thank the reviewer for recognizing the positive aspects of our paper, and we will address the reviewer's concerns in the following parts.\n\n**Q1:** scaling behavior of the method.\n\n**R1:** Thanks, we scale up our FAT to 25M+, 50M+ and 85M+ to match the general Vision Transformer backbones. For object detection/instance segmentation, all models use the framework of Mask-RCNN with the 1x schedule. The results are shown below. It can be found that FAT has great scalability. In addition, our model also demonstrates significant advantages in downstream tasks (object detection/instance segmentation). Compared to recent CMT-S, FAT-B3 has **+3.0**$AP^b$ and **+2.4**$AP^m$.\n\n|Model|Params(M)|FLOPs(G)|Top1-acc(%)|$AP^b$|$AP^m$|\n|--|--|--|--|--|--|\n|Swin-T|29|4.5|81.3|42.2|39.1|\n|UniFormer-S|24|4.2|82.9|45.6|41.6|\n|MOAT-0 [3]|28|5.7|83.3|--|--|\n|CMT-S [4]|25|4.0|83.5|44.6|40.7|\n|FAT-B3|29|4.4|**83.6**|**47.6**|**43.1**|\n|||||||\n|||||||\n|Swin-S|50|8.7|83.0|44.8|40.9|\n|Focal-S|51|9.1|83.5|47.4|42.8|\n|UniFormer-B|50|8.3|83.9|47.4|43.1|\n|MOAT-1 [3]|42|9.1|84.2|--|--|\n|CMT-B [4]|46|9.3|84.5|--|--|\n|FAT-B4|52|9.3|**84.8**|**49.7**|**44.8**|\n|||||||\n|||||||\n|CSwin-B|78 |15.0|84.2|--|--|\n|MOAT-2 [3]|73|17.2|84.7|--|--|\n|CMT-L [4]|75|19.5|84.8|--|--|\n|iFormer-L |87|14.0|84.8|--|--|\n|FAT-B5|88|15.1|**85.2**|--|--|\n\n**Q2:** Efficiency of the method.\n\n**R2:** Thanks. We compare FAT with the recent lightweight Vision Transformer and strong lightweight CNN backbone (EfficientNet, MobileOne, ParC-Net, MobileNetv2). We compare their **inference latency** on GPU/CPU and **inference throughput** on GPU. The CPU is Intel i9Core and the GPU is V100. **Inference throughput** is measured with batch size 64. **Inference latency** is measured with batch size 1. It can be seen that our FAT has the best speed/performance trade-off among lightweight Transformers.\n|Model|Params(M)|FLOPs(G)$\\downarrow$|CPU(ms)$\\downarrow$|GPU(ms)$\\downarrow$|Trp(imgs/s)$\\uparrow$|Top1-acc(%)|\n|---|---|---|---|---|---|---|\n|EdgeViT-XXS|4.1|0.6|43.0|14.2|1926|74.4|\n|MobileNetv2_1.4x [6]|6.1|0.6|37.2|11.1|2342|74.7|\n|MobileViT-XS|2.3|1.1|100.2|15.6|1367|74.8|\n|EdgeNext-XS|2.3|0.5|52.3|13.8|1417|75.0|\n|tiny-MOAT-0|3.4|0.8|61.1|14.7|1908|75.5|\n|MobileOne-S1 [5]|4.8|0.8|38.1|5.8|3265|75.9|\n|EfficientNet-B0|5.3|0.4|42.4|14.5|2181|77.1|\n|FAT-B0|4.5|0.7|44.3|14.4|1932|**77.6**|\n|     |   |   |    |    |     |   |\n|     |   |   |    |    |     |   |\n|MobileOne-S2 [5]|7.8|1.3|56.8|6.9|2312|77.4|\n|EdgeViT-XS|6.7|1.1|62.7|15.7|1528|77.5|\n|tiny-MOAT-1|5.1|1.2|80.8|14.8|1506|78.3|\n|MobileViT-S|5.6|2.0|135.6|16.2|898|78.4|\n|ParC-Net-S|5.0|1.7|112.1|15.8|1321|78.6|\n|EfficientNet-B1|7.8|0.7|61.3|18.5|1395|79.2|\n|EdgeNext-S|5.6|1.3|86.4|14.2|1243|79.4|\n|FAT-B1|7.8|1.2|62.6|14.5|1452|**80.1**|\n|     |   |   |    |     |    |   |\n|     |   |   |    |    |     |   |\n|MobileOne-S4 [5]|14.8|3.0|110.2|9.8|1225|79.4|\n|ParC-ResNet50|23.7|4.0|160.0|16.6|1039|79.6|\n|EdgeViT-S|11.1|1.9|95.8|16.2|1049|81.0|\n|tiny-MOAT-2|9.8|2.3|122.1|15.4|1047|81.0|\n|EfficientNet-B3|12.0|1.8|124.2|25.4|624|81.6|\n|FAT-B2|13.5|2.0|93.4|14.6|1064|**81.9**|\n\n**Q3:** Motivation\n\n**R3:** The motivation behind proposing bi-directional interaction in FAT is primarily inspired by existing research on the human brain. In [1], it has been indicated that the human brain has the ability to interpret local information as global shapes. Additionally, in [2], it has been pointed out that the global coordination of local information has significant effects on the processing of the human mid-level visual cortex. These two behaviors align with the examples mentioned in our Introduction, suggesting that the information flows of local to global and global to local should objectively exist in the human brain. Based on these perspectives, we separately model these two types of information flows to obtain modules more similar to the human brain. Since our goal is to design an efficient visual backbone, we did not employ more complex modeling techniques. It has been proven that even with this simple bi-directional interaction pattern, the model can provide significant improvements.\n\n**Q4:** Training stability\n\n**R4:** Thanks. We plot the \"epoch-loss\" and \"epoch-Top1acc\" for FAT-B3 in the global response. We invite the reviewer to read it. \n\n**Reference**\n\n[1]Schwarzkopf D. Samuel and Rees Geraint. 2011 Interpreting local visual features as a global shape requires awareness Proc. R. Soc. B.278: 2207–2215. doi/10.1098/rspb.2010.1909\n\n[2]Mannion DJ, Kersten DJ, Olman CA. Regions of mid-level human visual cortex sensitive to the global coherence of local image patches. J Cogn Neurosci. 2014 Aug;26(8):1764-74. doi: 10.1162/jocn_a_00588. Epub 2014 Feb 24. PMID: 24564470; PMCID: PMC4074231.\n\n[3]Chenglin Yang, et al. Moat: alternating mobile convolution and attention brings strong vision models. ICLR, 2023.\n\n[4]Jianyuan Guo, et al. Cmt: Convolutional neural networks meet vision transformers. CVPR, 2022.\n\n[5]Pavan. K, et al. MobileOne: An Improved One millisecond Mobile Backbone. arXiv:2206.04040, 2022.\n\n[6]Sandler. M, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018.\n"}, {"review_id": "gz3dA0We5V", "reviewer": "Reviewer_DXVA", "summary": "The paper proposes a new lightweight ViT structure called FAT. They use a fully adaptive self-attention mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. In addition, the paper introduces a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability.", "questions": "See weaknesses above.", "limitations": "See weaknesses above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand.\n\n- The experimental results show the effectiveness of the method.", "weaknesses": "- The usage of conv stem, conditional positional encoding, the convFFN and the aggregation of global and local attention is not novel. Thus, the novelty of this paper primarily lies in using bidirectional interaction. However, it is un-natural to explain via human visual system of using local to global and global to local interaction. Is it really the way human look at pictures?\n\n- In Eq.5, it is unclear why using different A1 and A2 makes the interaction 'bidirectional'. The author should further explain this.\n\n- Lack of comparison to the state-of-the-art models. For example, the proposed FAT-B3 has the same accuracy to CMT-S [1]  but larger FLOPs and parameters.\n\n[1] CMT: Convolutional Neural Networks Meet Vision Transformers. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **On motivation & novelty, explanation of Eq.5 and Comparison With SOTA**\n\n**Q1:** Motivation & Novelty\n\n**R1:** Thanks. The key novelty of our paper is bidirectional interaction, which is simple to implement and has negligible costs. This module is inspired by existing research on the human brain. In [1], it has been indicated that the human brain has the ability to interpret local features as global shapes. Additionally, in [2], it has been pointed out that the global coordination of local information significantly affects the processing of the human mid-level visual cortex. These two behaviors align with the examples mentioned in our Introduction, suggesting that the information flows of local to global and global to local should objectively exist in the human brain. Based on these perspectives, we separately model these two types of information flows in our bidirectional interaction. Since we aim to design an efficient visual backbone, we did not employ more complex modeling techniques. It has been proven that even with this simple bi-directional interaction pattern, the model can provide significant improvements. In contrast to all previous feature fusion methods, our bidirectional adaptive interaction stands out in two significant ways: \n\n1. Bidirectional Property: We model a bidirectional process between the two before fusing local and global features. This results in strengthened local and global features, respectively. In contrast to commonly used linear fusion, our interaction returns two enhanced vectors representing local and global contexts rather than an inseparable vector mixed with both. \n2. Adaptive Capability: Taking local features as an example, the global weights we use to enhance them are generated by tokens containing global features, making them context-aware and capable of adapting to the input data. Moreover, we calculate their element-wise product directly when fusing the enhanced local and global features. Our context-aware fusion approach is closer to the attention mechanism than linear fusion. \n\n\n\n**Q2:** Explanation of Eq.5.\n\n**R2:** Thanks, Eq.5 can be written as $y_i=\\mathcal{M}(\\mathcal{F}(\\mathcal{A}_1(x_i, X)), \\mathcal{A}_2(x_i, X))$, $\\mathcal{A}_1$ and $\\mathcal{A}_2$ denote two different feature aggregation operators, $\\mathcal{F}$ is activation function, and $\\mathcal{M}$ is modulation operator. Our bidirectional interaction includes two processes. Consider the local to global process. The global features and local features have been aggregated by attention ($\\mathcal{A}_1$) and self-modulated convolution ($\\mathcal{A}_2$), respectively. Then we use ${\\rm Sigmoid}$ ($\\mathcal{F}$) to activate global features to generate global weights. After that, we use global weights to modulate local features. The whole local to global peocess can be represented by $y_i=\\mathcal{M}(\\mathcal{F}(\\mathcal{A}_1(x_i, X)), \\mathcal{A}_2(x_i, X))$. Similarly, The global to local process also can be written as this equation, except $\\mathcal{A}_1$ and $\\mathcal{A}_2$ swap their positions. For simplicity, we only use one equation to denote these two processes in our paper. Here, we present the complete formulation of these two processes. Thus the complete bidirectional interaction can be expressed as: \n\n$y_{i2}=\\mathcal{M}(\\mathcal{F}(\\mathcal{A}_1(x_i, X)), \\mathcal{A}_2(x_i, X))$\n\n\n$y_{i1}=\\mathcal{M}(\\mathcal{F}(\\mathcal{A}_2(x_i, X)), \\mathcal{A}_1(x_i, X))$\n\nThese two processes together constitute bidirectional interaction. We thank the reviewer for pointing out this point, and we will modify it in the next version of the paper.\n\n**Q3:** Comparison with SOTA.\n\n**R3:** Thanks. Here we scale up FAT to the general backbone and compare it with recent SOTA. We also use the framework of Mask-RCNN to conduct object detection/instance segmentation with the 1x schedule. It can be seen that FAT performs better than CMT, especially in downstream tasks (object detection and instance segmentation). And when it comes to the larger model, the performance advantage of FAT will be further enhanced.\n\n|Model|Params(M)|FLOPs(G)|Top1-acc(%)|$AP^b$|$AP^m$|\n|--|--|--|--|--|--|\n|Swin-T|29|4.5|81.3|42.2|39.1|\n|UniFormer-S|24|4.2|82.9|45.6|41.6|\n|MOAT-0 [3]|28|5.7|83.3|--|--|\n|CMT-S [4]|25|4.0|83.5|44.6|40.7|\n|FAT-B3|29|4.4|**83.6**|**47.6**|**43.1**|\n|||||||\n|||||||\n|Swin-S|50|8.7|83.0|44.8|40.9|\n|Focal-S|51|9.1|83.5|47.4|42.8|\n|UniFormer-B|50|8.3|83.9|47.4|43.1|\n|MOAT-1 [3]|42|9.1|84.2|--|--|\n|CMT-B [4]|46|9.3|84.5|--|--|\n|FAT-B4|52|9.3|**84.8**|**49.7**|**44.8**|\n|||||||\n|||||||\n|CSwin-B|78 |15.0|84.2|--|--|\n|MOAT-2 [3]|73|17.2|84.7|--|--|\n|CMT-L [4]|75|19.5|84.8|--|--|\n|iFormer-L |87|14.0|84.8|--|--|\n|FAT-B5|88|15.1|**85.2**|--|--|\n\n****\n\n**Reference**\n\n[1]Schwarzkopf D. Samuel and Rees Geraint. 2011 Interpreting local visual features as a global shape requires awareness Proc. R. Soc. B.278: 2207–2215. doi/10.1098/rspb.2010.1909\n\n[2]Mannion DJ, Kersten DJ, Olman CA. Regions of mid-level human visual cortex sensitive to the global coherence of local image patches. J Cogn Neurosci. 2014 Aug;26(8):1764-74. doi: 10.1162/jocn_a_00588. Epub 2014 Feb 24. PMID: 24564470; PMCID: PMC4074231.\n\n[3]Chenglin Yang, et al. Moat: alternating mobile convolution and attention brings strong vision models. ICLR, 2023.\n\n[4]Jianyuan Guo, et al. Cmt: Convolutional neural networks meet vision transformers. CVPR, 2022."}]

Author rebuttal:
The global response contains two things:
1. Updated version of Table 1 in the paper.
2. Plots to show the training stability.

Meta-review: The reviewers acknowledge the improvements made by the authors in response to their concerns and appreciate the extra experimental results, e.g., the presented trade-off between efficiency and model performance, particularly for downstream tasks. They highlight the innovation of bidirectional interaction between global and local branches as a key strength, underscored by comprehensive ablation studies. The impressive results of FAT-B0 and B1 models in terms of both throughput and accuracy are seen as valuable insights for future research. The introduction of the FAT lightweight ViT structure, its adaptive self-attention mechanism, and fine-grained downsampling strategy are all recognized as meaningful contributions. Overall, the reviewer's reservations have been alleviated, leading to an upward revision of their scores and endorsing the paper's value in the context of low-latency small models.
Decision: Accept (poster)