SlowGuess/NeurIPS_2023_OCR huggingface dataset:
local aggregation	Params (M)	FLOPs (G)	Top1-acc (%)
w/o local	4.4	0.70	76.2
local	4.5	0.72	77.6
Table 11: Ablation of kernel size used in FASA.
adaptive aggregation. This outcome convincingly supports the soundness of our proposed FASA model.
Additional Sigmoid. In our main text, in the Method section's Bidirectional Adaptive Interaction subsection, Eq. 11 omits the high-order variable:
$$
\frac {1}{1 + e ^ {- Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}}}}
$$
Additional Sigmoid	Params (M)	FLOPs (G)	Throughput (imgs/s)	Top1-acc (%)
Sigmoid	4.5	0.72	1815	77.6
w/o Sigmoid	4.5	0.72	1932	77.6
in the final expression to a faster implementation. This variable represents two concatenated Sigmoid activation functions within the local adaptive aggregation phase. We attempt to include the previously omitted activation function in this section but find that having too many activation functions results in a similar performance but slower speed, as shown in Tab. 12. This result confirms the rationality of the structure we designed.
Kernel Size of FASA. Inspired by the notion that early stages of vision backbones tend to aggregate lower-level local features, while later stages prefer higher-level semantic ones, we adopt a staged approach and incrementally increase the DWConv's kernel size in FASA to aggregate local features. In this section, we investigate how DWConv's kernel size in FASA affects the model's image classification performance by varying the kernel sizes in different stages, as depicted in Tab. 13. Our experimental results demonstrate the effectiveness of our strategy for gradually increasing kernel size at different stages. Models using kernel sizes of 3, 5, 7, 9, and 3, 5, 7, 7 achieved the
Table 12: Ablation of additional Sigmoid.
kernel sizes	Params (M)	FLOPs (G)	Top1-acc (%)
3, 3, 3, 3	4.3	0.697	77.3
3, 5, 7, 7	4.5	0.717	77.6
3, 5, 7, 9	4.5	0.718	77.6
7, 7, 7, 7	4.5	0.737	77.5
9, 9, 9, 9	4.6	0.770	77.4
est performance.
Convolution Stem. We use the convolution stem proposed in [63]. To investigate the impact of convolution stem on our model's performance, we conducted experiments and presented the results in Tab. 14. Despite incurring some extra computational overhead, using convolution stem can significantly enhance the top1 accuracy of the model by  $0.4\%$ .
Table 13: Ablation of kernel size used in FASA.
conv stem	Params (M)	FLOPs (G)	Top1-acc (%)
w/o stem	4.5	0.65	77.2
stem	4.5	0.72	77.6
Kernel Size of ConvFFN. We conduct ablation experiments to validate the effect of different kernel sizes for DWConv in ConvFFN. As shown in Tab. 15, the kernel size of  $7 \times 7$  achieves the best performance, but it consumes more parameters  $(+0.2\mathrm{M})$  and FLOPs  $(+0.03\mathrm{G})$ . To achieve a better trade-off between the performance and computational complexity, We set the kernel size of DWConv in ConvFFN to 5.
Table 14: Ablation of convolution stem.
kernel sizes	Params (M)	FLOPs (G)	Top1-acc (%)
3 × 3	4.4	0.69	77.4
5 × 5	4.5	0.72	77.6
7 × 7	4.7	0.75	77.7
Bidirectional Interaction. Here we ablate the local-to-global and global-to-local, respectively, to show the effect of bidirectional interaction. The results are shown in the Tab. 16. It can be seen that the bidirectional interaction plays an important role in FAT.
E Speed Measurement
In order to accelerate the inference speed of our model, we utilized a toolkit developed for accelerating large kernel DWConv in [11]. Additionally, we employed the technique of structural reparameterization to merge the residual blocks in ConvFFN into one convolutional block. Furthermore, after omitting
Table 15: Ablation of kernel size used in ConvFFN.
Model	Params (M)	FLOPs (G)	Top1-acc (%)
no interaction	4.5	0.72	76.2
w/o global to local	4.5	0.72	76.9
w/o local to global	4.5	0.72	76.8
bidirectional interaction	4.5	0.72	77.6
Table 16: Ablation of bidirectional interaction.
the higher-order variables in bidirectional adaptive interaction, the original expression can be simplified to two SiLU activations followed by the multiplication of two branches. These measures significantly accelerated the inference speed of our model. Here, we list the most recent models in Tab. 17. As can be seen from the table, our model performs the best in terms of both speed and performance.
Model	Throughput (imgs/s)	Top1-Acc (%)	Year
T2T-ViT-7 [73]	1762	71.7	ICCV'2021
QuadTree-B-b0 [52]	885	72.0	ICLR'2022
TNT-Tiny [18]	545	73.9	NeurIPS'2021
EdgeViT-XXS [40]	1926	74.4	ECCV'2022
LVT [66]	1265	74.8	CVPR'2022
MobileViT-XS [38]	1367	74.8	ICLR'2022
EdgeNeXt-XS [37]	1417	75.0	ECCVW'2022
VAN-B0 [17]	1662	75.4	Arxiv'2022
ViL-Tiny [77]	857	76.7	ICCV'2021
PoolFormer-S12 [71]	1722	77.2	CVPR'2022
XCit-T12 [14]	1676	77.1	NeurIPS'2021
ResT-Lite [78]	1123	77.2	NeurIPS'2021
FAT-B0	1932	77.6	Ours
T2T-ViT-12 [73]	1307	76.5	ICCV'2021
EdgeViT-XS [40]	1528	77.5	ECCV'2022
CoaT-Lite-T [64]	1045	77.5	ICCV'2021
SiT w/o FRD [83]	1057	77.7	ECCV'2022
RegNetY-1.6GF [44]	1241	78.0	CVPR'2020
MPViT-T [28]	737	78.2	CVPR'2022
MobileViT-S [38]	898	78.4	ICLR'2022
ParC-Net-S [76]	1321	78.6	ECCV'2022
PVTv2-B1 [58]	1007	78.7	Arxiv'2022
PerViT [39]	1402	78.8	NeurIPS'2022
CoaT-Lite-Mi [64]	963	79.1	ICCV'2021
EfficientNet-B1 [51]	1395	79.2	ICML'2019
FAN-T-ViT [81]	1181	79.2	ICML'2022
EdgeNeXt-S [37]	1243	79.4	ECCVW'2022
XCit-T24 [14]	1194	79.4	NeurIPS'2021
FAT-B1	1452	80.1	Ours
ResT-S [78]	918	79.6	NeurIPS'2021
Shunted-T [47]	957	79.8	CVPR'2022
DeiT-S [54]	899	79.9	ICML'2021
QuadTree-B-b1 [52]	543	80.0	ICLR'2022
RegionViT-Ti [2]	710	80.4	ICLR'2022
WaveMLP-T [53]	1052	80.6	CVPR2022
PiT-S [21]	1042	80.9	ICCV'2021
MPViT-XS [28]	597	80.9	CVPR'2022
EdgeViT-S [40]	1049	81.0	ECCV'2022
VAN-B1 [17]	995	81.1	Arxiv'2022
Model	Throughput (imgs/s)	Top1-Acc (%)	Year
Swin-T [34]	664	81.3	ICCV'2021
PoolFormer-S36 [71]	601	81.4	CVPR'2022
CrossFormer-T [59]	929	81.5	ICLR'2022
EfficientNet-B3 [51]	634	81.6	ICML'2019
ResT-Base [78]	588	81.6	NeurIPS'2021
FAT-B2	1064	81.9	Ours
Coat-Lite-S [64]	540	81.9	ICCV'2021
DAT-T [61]	577	82.0	CVPR'2022
PVTv2-B2 [58]	582	82.0	Arxiv'2022
PiT-B [21]	328	82.0	ICCV'2021
FocalNet-T [67]	610	82.1	NeurIPS'2022
ViL-S [77]	359	82.4	ICCV'2021
CrossFormer-S [59]	601	82.5	ICLR'2022
RegionViT-S [2]	460	82.6	ICLR'2022
WaveMLP-S [53]	599	82.6	CVPR'2022
CSWin-T [12]	591	82.7	CVPR'2022
WaveViT-S [70]	482	82.7	ECCV'2022
VAN-B2 [17]	531	82.8	Arxiv'2022
DaViT-T [10]	616	82.8	ECCV'2022
HorNet-T [46]	586	82.8	NeurIPS'2022
FAN-S-ViT [81]	525	82.9	ICML'2022
FAT-B3-ST	641	83.0	Ours
DeiT-B [54]	299	81.8	ICML'2021
Focal-T [68]	301	82.2	NeurIPS'2021
PoolFormer-M48 [71]	304	82.5	CVPR'2022
QuadTree-B-b2 [52]	299	82.7	ICLR'2022
Shunted-S [47]	456	82.9	CVPR'2022
MPViT-S [28]	410	83.0	CVPR'2022
Swin-S [34]	390	83.0	ICCV'2021
ConvNeXt-S [35]	405	83.1	CVPR'2022
PVTv2-B3 [58]	398	83.2	Arxiv'2022
RegionViT-B [2]	256	83.2	ICLR'2022
EfficientFormer-L7 [30]	368	83.3	NeurIPS'2022
LITv2-M [41]	436	83.3	NeurIPS'2022
FocalNet-S [67]	358	83.4	NeurIPS'2022
iFormer-S [48]	471	83.4	NeurIPS'2022
CrossFormer-B [59]	368	83.4	ICLR'2022
WaveMLP-M [53]	359	83.4	CVPR'2022
FAT-B3	474	83.6	Ours
Table 17: Comparisons of speed and performance among different models. All speeds are measured on the V100 32G with the batch size of 64.
While our FAT achieves superior performance with fast processing speed, one limitation of our work is that due to computational resource constraints, we do not apply our lightweight vision backbone to unsupervised learning, video processing, or visual language tasks. Additionally, we do not pretrain our models on large-scale datasets like ImageNet-21K. However, we look forward to exploring more applications of our proposed FAT in the future.
This study is purely academic in nature, and we are unaware of any negative social impact resulting directly from our work. However, we acknowledge that our models' potential malicious use is a concern affecting the entire field. Discussions related to this matter are beyond the scope of our research.