SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Size (M)	Model	Input	Params (M)	FLOPs (G)	Throughput (img/s)	Top-1 (%)	Size (M)	Model	Input	Params (M)	FLOPs (G)	Throughput (img/s)	Top-1 (%)
50	T2T-ViT-7 [73]	2242	4.3	1.1	1762	71.7	10~15	XCiT-T24 [14]	2242	12.1	2.3	1194	79.4
QuadTree-B-b0 [52]	2242	3.5	0.7	885	72.0	ResT-S [78]	2242	13.7	1.9	918	79.6
TNT-Tiny [18]	2242	6.1	1.4	545	73.9	Shunted-T [47]	2242	11.5	2.1	957	79.8
Ortho-T [25]	2242	3.9	0.7	—	74.0	DeiT-S [54]	2242	22.1	4.6	899	79.9
EdgeViT-XXS [40]	2242	4.1	0.6	1926	74.4	QuadTree-B-b1 [52]	2242	13.6	2.3	543	80.0
MobileViT-XS [38]	2562	2.3	1.1	1367	74.8	RegionViT-Ti [2]	2242	13.8	2.4	710	80.4
LVT [66]	2242	5.5	0.9	1265	74.8	Wave-MLP-T [53]	2242	17.0	2.4	1052	80.6
CPVT-Ti-GAP [6]	2242	5.8	1.3	—	74.9	MPViT-XS [28]	2242	10.5	2.9	597	80.9
EdgeNeXt-XS [37]	2562	2.3	0.5	1417	75.0	EdgeViT-S [40]	2242	11.1	1.9	1049	81.0
PViT-T [57]	2242	13.2	1.6	1233	75.1	VAN-B1 [17]	2242	13.9	2.5	995	81.1
ViT-C [63]	2242	4.6	1.1	—	75.3	Swin-T [34]	2242	29.0	4.5	664	81.3
VAN-B0 [17]	2242	4.1	0.9	1662	75.4	CrossFormer-T [59]	2242	27.8	2.9	929	81.5
ViL-Tiny [77]	2242	6.7	1.4	857	76.3	ResT-B [78]	2242	30.3	4.3	588	81.6
CeiT-T [72]	2242	6.4	1.2	—	76.4	EfficientNet-B3 [51]	3002	12.0	1.8	634	81.6
FAT-B0	2242	4.5	0.7	1932	77.6	FAT-B2	2242	13.5	2.0	1064	81.9
10	T2T-ViT-12 [73]	2242	6.9	1.9	1307	76.5	25~30	DAT-T [61]	2242	29	4.6	577	82.0
Rest-lite [78]	2242	10.5	1.4	1123	77.2	FocalNet-T [67]	2242	29	4.4	610	82.1
XCIT-T12 [14]	2242	6.7	1.2	1676	77.1	Focal-T [68]	2242	29	4.9	301	82.2
EdgeViT-XS [40]	2242	6.7	1.1	1528	77.5	CrossFormer-S [59]	2242	31	4.9	601	82.5
CoaT-Lite-T [64]	2242	5.7	1.6	1045	77.5	RegionViT-S [2]	2242	31	5.3	460	82.6
SiT-Ti w/o FRD [83]	2242	15.9	1.0	1057	77.7	DaViT-T [10]	2242	28	4.5	616	82.8
RegNetY-1.6GF [44]	2242	11.2	1.6	1241	78.0	Wavet-VT-S [70]	2242	20	4.3	482	82.7
MPViT-T [28]	2242	5.8	1.6	737	78.2	QuadTree-B-b2 [52]	2242	24	4.5	299	82.7
MobileViT-S [38]	2562	5.6	2.0	898	78.4	CSWin-T [12]	2242	23	4.3	591	82.7
ParC-Net-S [76]	2562	5.0	1.7	1321	78.6	MPViT-S [28]	2242	23	4.7	410	83.0
PVTv2-B1 [58]	2242	13.1	2.1	1007	78.7	HorNet-T [46]	2242	23	4.0	586	83.0
PerViT-T [39]	2242	7.6	1.6	1402	78.8	FAT-B3-ST	2242	29	4.7	641	83.0
CoaT-Lite-MI [64]	2242	11.0	2.0	963	79.1	ConvNeXt-S [35]	2242	50	8.7	405	83.1
EfficientNet-B1 [51]	2402	7.8	0.7	1395	79.2	LITv2-M [41]	2242	49	7.5	436	83.3
FAN-T-ViT [81]	2242	7.3	1.3	1181	79.2	EfficientFormer-L7 [30]	2242	82	10.2	368	83.3
EdgeNext-S [37]	2242	5.6	1.3	1243	79.4	Ortho-S [25]	2242	24	4.5	—	83.4
FAT-B1	2242	7.8	1.2	1452	80.1	FAT-B3	2242	29	4.4	474	83.6
Results. We compare our FAT against many state-of-the-art models in Tab. 1. The comparison results demonstrate that our FAT consistently outperforms previous models across all settings, with a particularly significant improvement in performance on lightweight models. Specifically, our FAT-B0 achieves  $77.6\%$  Top1-accuracy with only 4.5M parameters and 0.7GFLOPs. Our FAT-B1 also surpasses EdgeNext-S by  $0.7\%$  with similar FLOPs. Furthermore, our FAT-B2 model surpasses CrossFormer-T in top1-accuracy by  $0.4\%$  while using only half the parameters. Additionally, our FAT-B3 model has also outperformed many state-of-the-art models.
Speed. As shown in Tab. 1, we present the speed comparison results of all models on V100, and our FAT performs better than its counterparts in both speed and performance. Specifically, Our FAT-B0 has a  $30\%$  higher throughput than EdgeViT-XS, and a  $0.1\%$  higher top-1 accuracy compared to EdgeViT-XS.
Table 1: Comparison with the state-of-the-art on ImageNet-1K classification. "FAT-B3-ST" indicates the FAT-B3 with the same layout of Swin-T.
Backbone	Semantic FPN
Params(M)	FLOPs(G)	mIoU(%)
PVTv2-B0 [58]	7.6	25.0	37.2
VAN-B0 [17]	8.1	25.9	38.5
EdgeViT-XXS [40]	7.9	24.4	39.7
FAT-B0	8.4	25.0	41.5
PVT-T [57]	17.0	33.2	35.7
ResNet50 [20]	28.5	45.6	36.7
PoolFormer-S12 [71]	16.2	31.0	37.2
EdgeViT-XS [40]	10.6	27.7	41.4
FAT-B1	11.6	27.5	42.9
ResNet18 [20]	15.5	32.2	32.9
PVTv2-B1 [58]	17.8	34.2	42.5
VAN-B1 [17]	18.1	34.9	42.9
FAT-B2	17.2	32.2	45.4
Swin-T [34]	31.9	182	41.5
DAT-T [61]	32.1	198	42.6
ScalableViT [69]	30.4	174	44.9
CrossFormer-S [59]	34.3	221	46.0
VAN-B2 [17]	30.3	164	46.7
CSWin-T [12]	26.1	202	48.2
Ortho-S [25]	28.0	195	48.2
Shuted-S [47]	26.1	183	48.2
FAT-B3	32.9	179	48.9
Table 2: Comparison with the state-of-the-art on ADE20K.
4.2 Semantic Segmentation
Settings. We incorporate our pretrained FATs with the Semantic FPN [27]. As in [58; 57], we use  $512 \times 512$  random crops during training and resize the images to have a shorter side of 512 pixels
during inference. Following [57], we fine-tune the models by AdamW with the learning rate of  $1 \times 10^{-4}$  and batch size of 16. We train them for 80K iterations on the ADE20K training set.
Results. The results can be found in Tab. 2. For B0/B1/B2, the FLOPs are measured with input size of  $512 \times 512$ . While them for B3 are measured with  $512 \times 2048$ , All our models achieve the best performance in all comparisons. Specifically, our FAT-B1 exceeds EdgeViT-XS for +1.5 mIoU. Moreover, our FAT-B3 outperforms the recent Shunted-S for +0.7 mIoU. All the above results demonstrate our model's superiority in dense prediction.
4.3 Object Detection and Instance Segmentation
Settings. We conduct object detection and instance segmentation experiments on the COCO 2017 dataset [33]. Following Swin [34], we use the FAT family as the backbones and take RetinaNet [32] and Mask-RCNN [19] as detection and segmentation heads. We pretrain our backbones on ImageNet-1K and fine-tune them on the COCO training set with AdamW [36].
Backbone	RetinaNet 1×	Mask R-CNN 1×
Params(M)	AP	AP50	AP75	APS	APM	APL	Params(M)	APb	AP50	AP75	APm
DFvT-T [15]	-	-	-	-	-	-	25	34.8	56.9	37.0	32.6	53.7
PVTv2-B0 [58]	13	37.2	57.2	39.5	23.1	40.4	49.7	24	38.2	60.5	40.7	36.2
QuadTree-B-b0 [52]	13	38.4	58.7	41.1	22.5	41.7	51.6	24	38.8	60.7	42.1	36.5
EdgeViT-XXS [40]	13	38.7	59.0	41.0	22.4	42.0	51.6	24	39.9	62.0	43.1	36.9
FAT-B0	14	40.4	61.6	42.7	24.0	44.3	53.1	24	40.8	63.3	44.2	37.7
DFvT-S [15]	-	-	-	-	-	-	32	39.2	62.2	42.4	36.3	58.9
EdgeViT-XS [40]	16	40.6	61.3	43.3	25.2	43.9	54.6	27	41.4	63.7	45.0	38.3
ViL-Tiny [77]	17	40.8	61.3	43.6	26.7	44.9	53.6	27	41.4	63.5	45.0	38.1
MPViT-T [28]	17	41.8	62.7	44.6	27.2	45.1	54.2	28	42.2	64.2	45.8	39.0
FAT-B1	17	42.5	64.0	45.1	26.9	46.0	56.7	28	43.3	65.6	47.4	39.6
PVTv1-Tiny [57]	23	36.7	56.9	38.9	22.6	38.8	50.0	33	36.7	59.2	39.3	35.1
ResT-Small [78]	23	40.3	61.3	42.7	25.7	43.7	51.2	33	39.6	62.9	42.3	37.2
PVTv2-B1 [58]	24	41.2	61.9	43.9	25.4	44.5	54.3	34	41.8	64.3	45.9	38.8
DFVt-B [15]	-	-	-	-	-	-	-	58	43.4	65.2	48.2	39.0
QuadTree-B-b1 [52]	24	42.6	63.6	45.3	26.8	46.1	57.2	34	43.5	65.6	47.6	40.1
EdgeViT-S[40]	23	43.4	64.9	46.5	26.9	47.5	58.1	33	44.8	67.4	48.9	41.0
MPViT-XS [28]	20	43.8	65.0	47.1	28.1	47.6	56.5	30	44.2	66.7	48.4	40.4
FAT-B2	23	44.0	65.2	47.2	27.5	47.7	58.8	33	45.2	67.9	49.0	41.3
Swin-T [34]	38	41.5	62.1	44.2	25.1	44.9	55.5	48	42.2	64.6	46.2	39.1
DAT-T [61]	38	42.8	64.4	45.2	28.0	45.8	57.8	48	44.4	67.6	48.5	40.4
DaViT-Tiny [10]	39	44.0	-	-	-	-	-	48	45.0	-	-	41.1
CMT-S [16]	44	44.3	65.5	47.5	27.1	48.3	59.1	45	44.6	66.8	48.9	40.7
MPViT-S [28]	32	45.7	57.3	48.8	28.7	49.7	59.2	43	46.4	68.6	51.2	42.4
QuadTree-B-b2 [52]	35	46.2	67.2	49.5	29.0	50.1	61.8	45	46.7	68.5	51.2	42.4
CSWin-T [12]	-	-	-	-	-	-	-	42	46.7	68.6	51.3	42.2
Shunted-S [47]	32	45.4	65.9	49.2	28.7	49.3	60.0	42	47.1	68.8	52.1	42.5
FAT-B3	39	45.9	66.9	49.5	29.3	50.1	60.9	49	47.6	69.7	52.3	43.1
Table 3: Comparison to other backbones using RetinaNet and Mask-RCNN on COCO val2017 object detection and instance segmentation.
Results. Tab. 3 shows the results with RetinaNet and Mask R-CNN. The results demonstrate that the proposed FAT performs best in all comparisons. For the RetinaNet framework, our FAT-B0 outperforms EdgeViT-XXS by  $+0.7$  AP, while B1/B2/B3 also perform better than their counterparts. As for the Mask R-CNN, our FAT-B3 outperforms the recent Shunted-S by  $+0.5$  box AP and  $+0.6$  mask AP. All the above results tell that our Fully Adaptive Transformer outperforms its counterparts by evident margins. More experiments can be found in appendix.
4.4 Comparison in Efficiency
We compared our approach with the state-of-the-art lightweight vision backbones, as shown in Tab 4. Our method achieved the best trade-off between speed and performance. The CPU is Intel i9Core and the GPU is V100. Inference throughput is measured with batch size 64. Inference latency is measured with batch size 1.
4.5 Larger Model
We scale up our FAT to  $50\mathrm{M}+$  and  $85\mathrm{M}+$  to match the general Vision Transformer backbones. For object detection/instance segmentation, all models use the framework of Mask-RCNN [19] with the 1x schedule. The results are shown in Tab. 5. It can be found that FAT has great scalability.
Model	Params(M)	FLOPs(G) ↓	CPU(ms) ↓	GPU(ms) ↓	Trp(imgs/s) ↑	Top1-acc(%)
EdgeViT-XXS [40]	4.1	0.6	43.0	14.2	1926	74.4
MobileViT-XS [38]	2.3	1.1	100.2	15.6	1367	74.8
tiny-MOAT-0 [65]	3.4	0.8	61.1	14.7	1908	75.5
FAT-B0	4.5	0.7	44.3	14.4	1932	77.6
EdgeViT-XS [40]	6.7	1.1	62.7	15.7	1528	77.5
ParC-Net-S [76]	5.0	1.7	112.1	15.8	1321	78.6
EdgeNext-S [37]	5.6	1.3	86.4	14.2	1243	79.4
FAT-B1	7.8	1.2	62.6	14.5	1452	80.1
ParC-ResNet50 [76]	23.7	4.0	160.0	16.6	1039	79.6
tiny-MOAT-2 [65]	9.8	2.3	122.1	15.4	1047	81.0
EfficientNet-B3 [51]	12.0	1.8	124.2	25.4	624	81.6
FAT-B2	13.5	2.0	93.4	14.6	1064	81.9
Table 4: Comparison with other methods in efficiency.
Model	Params(M)	FLOPs(G)	Top1-acc(%)	APb	APm
Swin-S [34]	50	8.7	83.0	44.8	40.9
Focal-S [68]	51	9.1	83.5	47.4	42.8
CMT-B [16]	46	9.3	84.5	-	-
FAT-B4	52	9.3	84.8	49.7	44.8
CSwin-B [12]	78	15.0	84.2	-	-
MOAT-2 [65]	73	17.2	84.7	-	-
CMT-L [16]	75	19.5	84.8	-	-
FAT-B5	88	15.1	85.2	-	-
4.6 Ablation Study and Spectral Analysis
In this section, we conduct experiments to understand FAT better. The training settings are the same as in previous experiments. More experiments can be found in the appendix.
Table 5: Comparison with general backbones.
Model	ImageNet-1K	COCO	ADE20K
Params(M)	FLOPs(G)	Top-1(%)	APb	APm	mIoU
add+linear	4.5	0.72	76.2	39.0	35.8	39.6
cat+linear	4.8	0.77	76.6	39.6	36.3	40.2
mul+linear	4.5	0.72	77.1	40.3	37.1	40.9
interaction	4.5	0.72	77.6	40.8	37.7	41.5
pool down	4.4	0.71	77.2	40.2	36.9	40.6
conv w/o overlap	4.4	0.71	77.2	40.3	36.9	40.8
conv w/ overlap	4.5	0.71	77.3	40.5	37.3	40.9
refined down	4.5	0.72	77.6	40.8	37.7	41.5
w/o conv. pos	4.4	0.70	77.4	40.5	37.3	41.2
conv. pos	4.5	0.72	77.6	40.8	37.7	41.5
Table 6: Ablation of FAT
Bidirectional Adaptive Interaction. Initially, we validate the efficacy of bidirectional adaptive interaction by comparing it with three baselines for fusing local and global information: concatenation, addition and element-wise multiplication. As shown in Tab. 6, it can be observed that bidirectional adaptive interaction outperforms all baselines significantly across various tasks. Specifically, compared to the baseline, which uses the cat+linear to fuse the local and global information, our bidirectional adaptive interaction uses fewer parameters and FLOPs but surpasses the baseline by  $1.0\%$  in Top1-accuracy. Our bidirectional adaptive interaction also performs better than element-wise multiplication.
Fine-Grained Downsampling. We compared our fine-grained downsampling strategy with three other strategies: non-overlapping large stride pooling, non-overlapping large stride convolution, and overlapping large stride convolution. As shown in Tab. 6, experimental results across various tasks have demonstrated the effectiveness of our fine-grained downsampling strategy. Specifically, our fine-grained downsampling strategy surpasses directly pooling downsampling strategy by  $0.4\%$
Positional Encoding. At the end of Tab. 6, we explore the effect of CPE, and the results in the table show that the flexible CPE also contributes to the performance improvement of the model for  $0.2\%$  in image classification.
Figure 4: Spectral analysis from 8 output channels of FASA. The larger magnitude has a lighter color. Pixels that are closer to the center have a lower frequency. From top to bottom, the results are from (a) local adaptive aggregation, (b) global adaptive aggregation, (c) add+linear fusion, and (d) bidirectional adaptive interaction.
Spectral Analysis. As shown in Fig. 4, we conduct a spectral analysis on FASA. Compared to add+linear fusion, our bidirectional adaptive interaction can more fully fuse high-frequency and low-frequency information.
In this paper, we propose a new attention mechanism named Fully Adaptive Self-Attention (FASA), which draws inspiration from the human visual system. FASA is designed to model local and global information adaptively while also accounting for their bidirectional interaction using adaptive strategies. Furthermore, we enhance the self-attention mechanism in FASA by incorporating a fine-grained downsampling strategy to promote better global perception at a more detailed level. Using the flexible FASA as our foundation, we develop a lightweight vision backbone called Fully Adaptive Transformer (FAT), which can be applied across various vision tasks. Our extensive experiments on diverse asks, such as image classification, object detection, instance segmentation, and semantic segmentation, provide compelling evidence of the effectiveness and superiority of FAT. We expect to apply FAT to other vision-related pursuits, including video prediction and vision-language models.
This work is partially funded by National Natural Science Foundation of China (Grant No. 62006228, U21B2045, U20A20223), Youth Innovation Promotion Association CAS (Grant No. 2022132), and Beijing Nova Program (20230484276).
[73] Li Yuan, Yunpeng Chen, Tao Wang, et al. Tokens-to-token vit: Training vision transformers from scratch onImagenet. In ICCV, 2021.
[74] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[75] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, et al. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[76] Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware circular convolution with merits from convnets and transformer. In ECCV, 2022.
[77] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, 2021.
[78] Qinglong Zhang and Yu bin Yang. Rest: An efficient transformer for visual recognition. In NeurIPS, 2021.  
[79] Zhun Zhong, Liang Zheng, Guoliang Kang, et al. Random erasing data augmentation. In AAAI, 2020.
[80] Bolei Zhou, Hang Zhao, Xavier Puig, et al. Scene parsing through ade20k dataset. In CVPR, 2017.  
[81] Daquan Zhou, Zhiding Yu, Enze Xie, et al. Understanding the robustness in vision transformers. In ICML, 2022.
[82] Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio-visual learning: A survey. IJAC, 18:351, 2021.
[83] Zhuofan Zong, Kunchang Li, Guanglu Song, et al. Self-slimmed vision transformer. In ECCV, 2022.
A Discussion About "Interaction" And "Fusion"
Generally speaking, interaction refers to the process of two or more objects or systems affecting each other. In contrast, fusion refers to combining two or more things into a single entity. As shown in Fig. 5, in contrast to all previous feature fusion methods, our proposed bidirectional adaptive interaction stands out in two significant ways:
Figure 5: Comparison between our bidirectional adaptive interaction and traditional fusion method.
1) Bidirectional Property: Our approach draws inspiration from how visual information flows bidirectionally through the human visual system. Before fusing local and global features, we model a bidirectional process between the two. Specifically, for local features, we utilize global features to generate weights that encompass global information. These weights are applied to the local features, allowing them to have the ability to capture the global context. We execute a similar process for global features. This results in strengthened local and global features, respectively. In contrast to linear fusion, our interaction returns two enhanced vectors representing local and global contexts rather than an inseparable vector mixed with both [41; 48; 43].
2) Adaptive Capability: Taking local features as an example, the global weights we use to enhance them are not trainable. They are generated by tokens containing global features, making them context-aware and capable of adapting to the input data. Moreover, when fusing the enhanced local and global features, we do not introduce any additional parameters but calculate their element-wise product (Hadamard product) directly. Compared with linear fusion, our context-aware fusion approach is closer to the attention mechanism [56; 17; 22]. Our ablation experiments have shown that this fusion approach not only saves parameters but also achieves better results. Specifically, for a detailed comparison between the Add+Linear, Cat+Linear, and our model, please refer to the paper's main text.
The implementation of our proposed bidirectional adaptive interaction method is very simple, yet it has achieved remarkable results compared to other methods for fusing local and global features. Through this work, we aim to reveal the critical role that features fusion plays in visual models and hope to inspire more related research.
We evaluate our backbones with the framework of UperNet [62] based on MMsegmentation [7]. Following the [34], we adopt AdamW to optimize the model for 160K iterations. As shown in Tab. 7, our model surpasses all its counterparts and achieves the best performance. Specifically, our FAT-B3 surpass the resent Shunted-S by  $+0.7$  mIoU and  $+0.8$  MS mIoU.
Backbone	Params(M)	FLOPs(G)	mIoU(%)	MS mIoU(%)
Swin-T [34]	60	945	44.5	45.8
DAT-T [61]	60	957	45.5	49.0
DaViT-T [10]	60	940	46.3	—
UniFormer-S [29]	52	955	47.0	48.5
CrossFormer-S [59]	62	980	47.6	48.4
MPViT-S [28]	52	943	48.3	—
Ortho-S [25]	54	956	48.5	49.9
Shunted-S [47]	52	940	48.9	49.9
FAT-B3	59	936	49.6	50.7
Table 7: Results with the framework of UperNet on ADE20K. The FLOPs are measured with the resolution of  $512 \times 2048$ .
Output Size	Layer Name	FAT-B0	FAT-B1	FAT-B2	FAT-B3
H/4 × W/4	Conv Stem	3 × 3, 16, s2	3 × 3, 24, s2	3 × 3, 32, s2	3 × 3, 32, s2
3 × 3, 32, s2	3 × 3, 48, s2	3 × 3, 64, s2	3 × 3, 64, s2
3 × 3, 32, s1	3 × 3, 48, s1	3 × 3, 64, s1	3 × 3, 64, s1
3 × 3, 32, s1	3 × 3, 48, s1	3 × 3, 64, s1	3 × 3, 64, s1
1 × 1, 32, s1	1 × 1, 48, s1	1 × 1, 64, s1	1 × 1, 64, s1
H/4 × W/4	CPE	3 × 3, 32	3 × 3, 48	3 × 3, 64	3 × 3, 64
FASA	kernel 3, heads 2	kernel 3, heads 3	kernel 3, heads 2	kernel 3, heads 2
ConvFFN	5 × 5, 32	5 × 5, 48	5 × 5, 64	5 × 5, 64
H/8 × W/8	CPE	5 × 5, 80	5 × 5, 96	5 × 5, 128	5 × 5, 128
FASA	kernel 5, heads 5	kernel 5, heads 6	kernel 5, heads 4	kernel 5, heads 4
ConvFFN	5 × 5, 80	5 × 5, 96	5 × 5, 128	5 × 5, 128
H/16 × W/16	CPE	7 × 7, 160	7 × 7, 192	7 × 7, 256	7 × 7, 256
FASA	kernel 7, heads 10	kernel 7, heads 12	kernel 7, heads 8	kernel 7, heads 8
ConvFFN	5 × 5, 160	5 × 5, 192	5 × 5, 256	5 × 5, 256
H/32 × W/32	CPE	9 × 9, 256	9 × 9, 384	9 × 9, 512	9 × 9, 512
FASA	kernel 9, heads 16	kernel 9, heads 24	kernel 9, heads 16	kernel 9, heads 16
ConvFFN	5 × 5, 256	5 × 5, 384	5 × 5, 512	5 × 5, 512
1 × 1	Classifier	Fully Connected Layer, 1000
Params	4.5M	7.8M	13.5M	29M
FLOPs	0.7G	1.2G	2.0G	4.4G
Table 8: Details about FAT's architecture.
C.1 Architecture Details
The detailed architectures are shown in Tab. 8, where all FLOPs are measured at the resolution of  $224 \times 224$  for image classification. For the convolution stem, we adopt four  $3 \times 3$  convolutions and one  $1 \times 1$  convolution to tokenize the image. Batch normalization and ReLU are used after each  $3 \times 3$  convolution, and a layer normalization is used after the final  $1 \times 1$  convolution. Inspired by the fact that in the vision backbone, the early stages tend to capture high-frequency local information, and the later stages tend to capture low-frequency global information, we gradually increase the DWConv's kernel sizes used in FASA. Specifically, we set the kernel size to 3 in the first stage and 9 in the last stage. The expansion ratios are set to 4, and the kernel sizes are set to 5 for all ConvFFN layers.
C.2 Image Classification
We follow the same training strategy with Swin-Transformer [34]. All models are trained for 300 epochs from scratch. To train the models, we use the AdamW optimizer with a cosine decay learning rate scheduler and 20 epoch linear warm-up. We set the initial learning rate, weight decay, and batch size to 0.001, 0.05, and 1024, respectively. We adopt the strong data augmentation and regularization used in [34]. Our settings are RandAugment [8] (randm9-mstd0.5-inc1), Mixup [75] (prob=0.8), CutMix [74] (prob=1.0), Random Erasing [79] (prob=0.25), increasing stochastic depth [24] (prob=0.05, 0.1, 0.1, 0.15 for FAT-B0, FAT-B1, FAT-B2 and FAT-B3).
C.3 Object Detection and Instance Segmentation
We adopt MMDetection [3] to implement RetinaNet [32] and Mask-RCNN [19]. We use the commonly used "1×" (12 training epochs) setting for the two strategies. Following [34], during training, images are resized to the shorter side of 800 pixels while the longer side is within 1333 pixels. We adopt the AdamW optimizer with a learning rate of 0.0001, weight decay of 0.05, and batch size of 16 to optimize the model. The learning rate declines with the decay rate of 0.1 at the epoch 8 and 11.
C.4 Semantic Segmentation
We adopt the Semantic FPN [27] and UperNet [62] based on MMSegmentation [7] and apply FAT pretrained on ImageNet-1K as backbone. We use the same setting of PVT [57] to train the Semantic FPN, and we use the "1×" schedule (trained for 80k iterations). All the models are trained with the input resolution of  $512 \times 512$ . When testing the model, we resize the shorter side of the image to 512 pixels. As for UperNet, we follow the default settings in Focal Transformer [68]. We take AdamW with a weight decay of 0.01 as the optimizer and train the models for 160K iterations. The learning rate is set to  $6 \times 10^{-5}$  with 1500 iterations warmup.
D More Ablation and Analysis Results
Model	Blocks	Channels	Params(M)	FLOPs(G)	Top-1 acc(%)
Swin-T [34]	[2, 2, 6, 2]	[96, 192, 384, 768]	29	4.5	81.3
DAT-T [61]	[2, 2, 6, 2]	[96, 192, 384, 768]	29	4.6	82.0
FocalNet-T [67]	[2, 2, 6, 2]	[96, 192, 384, 768]	28	4.4	82.1
Focal-T [68]	[2, 2, 6, 2]	[96, 192, 384, 768]	29	4.9	82.2
FAT-B3-ST	[2, 2, 6, 2]	[96, 192, 384, 768]	29	4.7	83.0
Table 9: Comparison with four baseline models when use the same layout with Swin-T.
Other architecture design choices. We modified FAT-B3 to have the same layout as Swin-T and compared it to four baseline models: Swin-T [34], DAT-T [61], FocalNet-T [67] and Focal-T [68]. As shown in Tab. 9, our model demonstrates significant superiority, with an accuracy  $1.7\%$  higher
than Swin-T and  $0.8\%$  higher than Focal-Tiny.
Contribution Isolation. To better isolate the contribution of our FASA, we compare the different self-attention mechanisms on the same backbone. We choose five kinds of self-attention mechanisms to make the comparison: CSWSA in CSwinTransformer [12], Max-SA in MaxViT [55], WSA/SWSA in Swin-Transformer [34], SRA in PVT [57] and LSA/GSA in Twins-SVT [5]. The results are shown in Tab. 10. The networks' layout and the setting of CPE and ConvFFN are the same as FATB0. It can be found that our FASA surpasses all its
Method	Params (M)	FLOPs (G)	Top1-acc (%)
Max-SA [55]	4.3	0.8	75.7
WSA/S-WSA [34]	4.3	0.8	76.1
SRA [57]	4.4	0.7	76.2
LSA/GSA [5]	4.3	0.7	76.6
CSWSA [12]	4.3	0.8	76.8
FASA	4.5	0.7	77.6
Table 10: Comparison among different self-attention mechanisms.
counterparts. Compared to the latest Max-SA, our FASA achieved a gain of  $1.9\%$ .
Local Aggregation in FASA. To validate that the model's capability to perceive local context does not solely rely on CPE and ConvFFN, we conducted experiments by removing local adaptive aggregation and bidirectional adaptive interaction from FASA, retaining only global adaptive aggregation. The results presented in Tab. 11 demonstrate a significant decrease in model performance  $(1.4\%)$  when using only global