Asymmetric Factorized Bilinear Operation for Vision Transformer
Authors: Junjie Wu
Qilong Wang
Jiangtao Xie
Pengfei Zhu
Qinghua Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Transformer
Channel Mixer
Factorized Bilinear Operation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: As a core component of Transformer-like deep architectures, a feed-forward network (FFN) for channel mixing is responsible for learning features of each token. Recent works show channel mixing can be enhanced by increasing computational burden or can be slimmed at the sacrifice of performance. Although some efforts have been made, existing works are still struggling to solve the paradox of performance and complexity trade-offs. In this paper, we propose an Asymmetric Factorized Bilinear Operation (AFBO) to replace FFN of vision transformer (ViT), which attempts to efficiently explore rich statistics of token features for achieving better performance and complexity trade-off. Specifically, our AFBO computes second-order statistics via a spatial-channel factorized bilinear operation for feature learning, which replaces a simple linear projection in FFN and enhances the feature learning ability of ViT by modeling second-order correlation among token features. Furthermore, our AFBO presents two structured-sparsity channel mapping strategies, namely Grouped Cross Channel Mapping (GCCM) and Overlapped Cycle Channel Mapping (OCCM). They decompose bilinear operation into grouped channel features by considering information interaction between groups, significantly reducing computational complexity while guaranteeing model performance. Finally, our AFBO is built with GCCM and OCCM in an asymmetric way, aiming to achieve a better trade-off. Note that our AFBO is model-agnostic, which can be flexibly integrated with existing ViTs. Experiments are conducted with twenty ViTs on various tasks, and the results show our AFBO is superior to its counterparts while improving existing ViTs in terms of generalization and robustness.

Published as a conference paper at ICLR 2025
ASYMMETRIC FACTORIZED BILINEAR OPERATION FOR VISION TRANSFORMER
Junjie Wu' Qilong Wang" Jiangtao Xie Pengfei Zhu'
Qinghua Hu!
Tianjin University DDalian University of Technology (wd)-, glwang, zhupengfet, huganghha)jtjju.ddb..... jiamgtaoxio emmal dll... .nd..cn
ABSTRACT
As a core component of Transformer-like deep architectures, feed-forward net- work (FFN) for channel mixing responsible for learning features of cach token. Recent works show channel mixing can be enhanced by increasing computational burden
be slimmed the sacrifice of performance. Although some efforts have been made, existing works are still struggling to solve the paradox of perfor- mance and complexity trade-offs. In this paper, propose an Asymmetrie Fac- torized Bilinear Operation (AFBO) to replace FFN of vision transformer (ViT), which attempts efficiently explore rich statisties of token features for achieving better performance and complexity trade-off. Specifically, our AFBO computes second-order statistics via spatial-channel factorized bilinear operation for fea- ture learning, which replaces a simple linear projection in FFN and enhances the feature learning ability of ViT by modeling second-order correlation among to- ken features. Furthermore, our AFBO presents two structured-spene -sparsitty channel mapping strategies, namely Grouped Cross Channel Mapping (GCCM) and Over- lapped Cycle Channel Mapping (OCCM). They decompose bilinear operation into grouped channel features by considering information interaction between groups, significantly reducing computational complexity while guaranteeing model per- formance. Finally, our AFBO is built with GCCM and OCCM in an asymmetric way, aiming achieve better trade-off. Note that our AFBO is model-agnostic, which can be flexibly integrated with existing ViTs. Experiments are conducted with twenty ViTs on various tasks, and the results show our AFBO is superior to its counterparts while improving existing ViTs in terms of generalization and robustness.
INTRODUCTION
In recent years, transformer-lik like architectures have attracted a mass of research interests and achieved remarkable performance various computer vision tasks (Dosovitskiy et al. 2021 Fang et al. 2021 (Strudel et al., 2021 Neimark et al.) 2021). As one of core components, a feed-forward Neimark
network (FFN) with two fully-comnected layers is generally used as channel mixer to learn token features, which is proven to significantly influence performance of vision transformer (VIT) (Dong cct al. 2021] Yu et al.] 2022). Intuitively, a two-layer FFN is a concise yet naive learning scheme that fails to fully consider the rich information lying in token features, which generally achieves sub-optimal solutions in terms of both efficiency and effectiveness (Fang et al.) 2024| Xu et al.)
2024
Sridhar al. 2023). However, existing works pay less attention improving FFN module of ViT compared to the modifications on self-attention mechanism (i.e., another core component of
ViT) Huang et al. 2021 Liu et al. 2021 Kitaev et al. 2020 Graham al. 2021 Srinivas al.
202
Touvron et al. 2021bb.
Recently, some works have made attempts to improve channel mixing module of transformer-lin ar-
chitectures (Fang et al. 2024:) Xu et al. 2024; Sridhar et al. 2023 Lietal. (021). As pioneer work,
exploit Gated Linear Units (GLU) (Dauphin al. (017) to improve FFN of transformer in natural language processing tasks. Subsequently. EVA-02 (Fang
Shazeer (Shazeer] (020) introduces
et al. 2024) extends GLU to ViT by employing SiLU activation (Hendrycks & Gimpel 2016).
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Y E R
Y & Rd
Add Norm
Linear
Liner
Feed Forward
RMM
2; RM
Norm
Activation
Acthation
Cycle shift
Multi Head Attention
[c) OCCM: Overlapped Cycle Channel Mapping
Linear
Linear
Linear
Input
Embeddiing
X Rd
R
Split
Input
(a)
(b)
(d) GCCM: Grouped Cross Channel Mapping
Figure 1: Comparison of (a) original FFN with (b) proposed Asymmetrie Factorized Bilinear Operation (AFBO) for VIT. Specifically, our AFBO computes second- -oder statistics via spatial- channel factorized bilinear operation (SCFBO) for feature abstraction instead of the first linear pro- jection in FFN. Particularly, our SCFBO first approximates second-order statistics via rank-1 de- composition, and further decomposes mapping operations into Spatial Modeling (SM) and Channel Mapping (CM). For better performance and complexity trade-off, our AFBO is constructed by two structured -parrstty channel mapping strategies (i.c., (c) Grouped Cross Channel Mapping and (d) Overlapped Cycle Channel Mapping) in asymmetric manner. The details refer to Sec. 3.2
Fang al. 2024) generally require to reduce the hidden dimension of FFN, which potentially limits performance of VIT (Noshad et al. 2019 /Sridhar et al. 2023). Besides, some researchers propose to use some extra modules inheriting from convolutional neural networks (CNNs) to enhance FFN of VIT (Li et al.,
To guarantee computational efficiency, existing GLU variants (haazeer 2020
2021 Guo et al. 2022a| Zhou et al.) 2022; Cao et al. 2023), such as depth- wise (DW) convolution, channel attention (Hu et al. 22200; Wang et al. [202 Zhou et al. 2022), inverted residual FFN (San-) dler et al. 2018) and GRN (Liu et al. 2022 Woo et al.] 2023). Although these extra modules can improve performance of ViT, they undoubtedly increase computational cost, especially for large models. In contrast, some recent works (Xu et al., 2024; Sridhar et al.] 2023) focus on slimming FFN reduce computational cost of transformer- like architectures by carefully decreasing hidden dimension to reduce computational cost or employing a block diagonal linear projection regularized by channel covariance attention. Despite the aforementioned slimming methods can reduce com- putational cost of FFN, they are limited to improve performance of ViT. Therefore, existing FFN variants of ViT still struggle solve the paradox of performance and complexity trade-off.
To address the above issue, we propose novel Asymmetric Factorized Bilinear Operation (AFBO) efficiently explore and exploit rich information of token features, which replaces FFN of Trans- former for achieving better performance and complexity trade-off. Specifically, motivated by the success second-order modeling in improving deep CNNs (Lin et al., 2015) Gao et al. 2016: Li Factorized Bilinear Operation (SCFBO) to efficiently compute second-order statistics of token features, which replaces the first linear projection in the FFN for feature learning. Particularly, our SCFBO first approximates
et al. 2017 Wang et al. 2017 2021a), our AFBO presents a Spatial--haamne
second-order statistics via a rank- decomposition (Gao et al.) 2016 Li al. 2017 Mnih & Hinton 2007]. which is further decomposed into spatial modeling and channel mapping. For spatial mod- eling, we capture spatial correlation in local neighborhoods using various methods, including local pooling, DW convolution, and spatial attention. For channel mapping, two point-wise convolutions followed by dot product are used to model the channel correlation of each token feature. As such, our AFBO enhances the feature learning ability Transformer by modeling second-order correla- tion among token features. Since channel mapping usually leads to high computational complexity especially with large hidden dimension, our AFBO presents two structured-spee -sparrssyy channel map- ping strategies, namely Grouped Cross Channel Mapping (GCCM) and Overlapped Cycle Channel Mapping (OCCM), which divide channel features into several groups and perform bilinear operation by considering information interaction between groups. Based on the proposed structured-sparsity channel mappings, we can construct several AFBO variants that significantly reduce parameters and FLOPs while maintaining high performance. Particularly. OCCM and GCCM are helpful for im- proving performance and reducing complexity, and our AFBO asymmetrically built with GCCM and OCCM achieve better performance and complexity trade-off. Note that model-aggnstic
2

---

Published as a conference paper at ICLR 2025
AFBO can be flexibly integrated with existing Transformer-lite architectures, achieving better per- formance at low cost of complexity. The comparison of original FFN with our AFBO is illustrated in To evaluate our AFBO, experiments conducted on various vision tasks (i.c., image clas- sification on ImageNet- K (Krizhevsky et al. 2017) and its nut-of-distribution variants (Hendrycks)
Fig. [
et al. 2021b [Hendrycks & Dietterich] 2019 Hendrycks et al.] [202aa] Recht et al. 2019), object detection and instance segmentation on MS COCO (Lin et al. (2014) with twenty ViT models (c.g.,
DeiT (Touvron al. 2021a), Swin Transformer (Liu et al. |l.] [202]) and PoolFormer (Yu et al. 2022
The contributions of this work can be summarized as follows:
(1) This paper proposes a novel Asymmetric Factorized Bilinear Operation (AFBO) as an effective alternative of FFN for ViTs. Particularly, our AFBO efficiently explores rich statistics of token features and shows great potential to achieve better performance and complexity trade-off for ViTs.
(2) To this end, our AFBO presents spatial-channel factorized bilinear operation to efficiently model second-order statisties of token features and introduces two structured-sparsity channel map- pings reduce model complexity (i.e., parameters and FLOPs) while guaranteeing performance.
(3) Our AFBO is model-agnostic and can be flexibly integrated with existing Transformer-like archi- tectures. Extensive experiments various tasks by using different ViT' show our AFBO is superior to its counterparts, while improving existing ViTs terms of generalization and robustness.
2 RELATED WORK
Building upon the remarkable success of transformer architecture in natural language process-
ing (Vaswani et al. 2017). Vision Transformer (ViT) (Dosovitskiy et al. [202T] is initially intro-
duced to extend transformer models to vision tasks (Krizhevsky al.) 2017] Fang al. 1021
Strudel et al. 2021; Neimark et al. (021). Subsequently, lot of VIT variants
Wang et al. 2022
Liu et al. 2021 Vaswani et al. 2021 Ding al.) 2022 Huang et al.) 2023 Ho et al., 2019] Lu
et al. (021) have been studied to improve VIT, especially for its self-
mechanism. Most
of these methods aim to reduce computational complexity of self-attention since its complexity is quadratic to token numbers. Among them, some works (Liu et al., [021: Ding et al.
2022) focus
on developing a local self-attention mechanism and its shifted/haloed version to add the interaction across different local windows. Besides, SOFT (Lu et (001)) replaces the dot- dot-product similarity by proposing a softmax-free transformer with linear space and time complexity. Additionally, in- corporation of self-attention with convolution
hybrid backbone has been studied for enhancing locality of ViTs (Guo et al., 2022a Wu et al.) [021 Dai et al. 2021: A'Accoli et al. 2021: Li et al., 2023 There also exist some works (Yu et al.) 2022) challenge necessity of self-attention, and they propose some substituting operations (e.g.. pooling and identity mapping) for attention while achieving comparable performance. Distinguished from the aforementioned works, our AFBO fo- cuses on improving FFN module of ViT for achieving better efficiency and effectiveness trade-off.
Some recent works (Fang al., 2024 Xu et al. 2024; Sridhar et al. 2023 Li al. 202T) have been studied to improve channel mixing module (i.e., FFN) of VIT. For example, EVA-02 (Fang al., (024) extends GLU (Shazeer) 2020) to ViT. It employs a two-branch structure to replace
FFN. which ensures computational efficiency by reducing the hidden dimension. Besides, some works (Xu et al. 2024; Sridhar et al. 2023) focus on reducing the computational complexity of FFN
by developing some slimming schemes. Additionally, some studies (Li et al. 2021 Cao al. 2023
Guo et al. 2022a) enhance the performance of FFN by introducing extra modules (e.g., channel
attention (Hu et al. 2020; Wang et al.) 2020) DW convolution). TransNext (Shi 2024) enhances
FFN by combining SwiGLU (Shazeer 2020 Fang et al. (024)) with DW convolution. Unlike these
methods, our AFBO attempts explore rich statistics of token features to improve FFN module of
ViT, which achieves better performance and complexity trade-off than existing counterparts (refer to comparisons in Table 3).
3 PROPOSED METHOD
In this section, first briefly revisit the original FFN for ViT, and describe details of our Asym- metric Factorized Bilinear Operation (AFBO). Finally, three AFBO variants are introduced and we compare them in terms of performance and complexity for model selection.
3

---

Published as a conference paper at ICLR 2025
3.1 REVISITING FFN FOR VIT
As shown in Fig. D (a), the original FFN involved of two fully-connected (linear) layers with an activation function is used perform channel mixing in transformer block. Let X BdxN be d-dimensional features of N tokens, the output of FFN (i.e., Y E RdxNN can be written as
Y
(W (o()), Z = 1x1
(Wi.bi) (x),
(I)
1x1
where X1x1 indicates point-wiss convolution with kernel size of 1 x 1. { E RR b, Wd
and (W2 E TR Maxd b2 E Rd) respectively are weight parameters of two point-wise convolutions (w.r.f fully-connected layers), and M is the expansion ratio of hidden dimension. Z E R MdMx means the intermediate features, and (-) is the activation function (e.g., GeLU (Hendrycks & Gim- pel 2016)). From Eq. (T). one can see that FFN performs feature learning through two linear pro- jections. which do not fully consider the rich information lying in token features. A naive scheme to enhance performance of ViTs to increase the hidden dimension of FFN, but it also brings more computational cost. By considering feature learning ability heavily influences performance, reason- able exploration of information lying in token features is a potential solution to improve ViT.
3.2
ASYMMETRIC FACTORIZED BILINEAR OPERATION (AFBO) FOR VIT
3.2.1
SPATIAL-CHANNEL FACTORIZED BILINEAR OPERATION
Previous works (Lin et al. 2015 Wang al. 202 show that the appropriate (global) modeling of second-order statistics lying in features can effectively improve deep CNNs. Inspired by these works, we incorporate idea of second-order modeling into FFN of ViT by modifying Eq. [ as
= WZ + b2, Z (X"') + bb,
where X indicates the outer product of features X, capturing second-order statisties of X. T indicates the transposition operation. However, such strategy in Eq. (2) suffers from two issues: (I) x generates global representation, neglecting local information of each token; (2) representa-
tion size of T X is d x d, leading to a heavy computational burden, i.e., W1 I x
To address above issues, our AFBO approximates computation of second-order statistics via a rank- I decomposition, as suggested in (Gao et al., 2016 Li et al. 2017 Mnih & Hinton 2007). As such, our AFBO replaces the first linear projection in FFN by bilinear operation:
by) Y = 81x1
(o(2),, 2 = KxK
b.)
(x) b2)
(X)],
(3)
where {W1 E R b RM and (W₂ BdxMax B2 E RM4 weight parameters of two convolutions, respectively. And @x* indicates convolution operation with kernel size of K x K. (W3 E Maxd, by E Id are weight parameters of point-wise convolution of @1x1, and indicates the dot product.
For computational efficiency, we further present a spatial-channel factorized bilinear operation (SCFBO), which decomposes @ K into spatial modeling operation @x* K and channel map- ping (i.e., point-wise convolution 81x1). According to Eq. 33 our SCFBO is formulated as
Y
@ ix1
bs)
(a (Z)),
2
x Kxk
WW (Wi (X)
01x1
Kxk
@ 0ix1
(W)
(x)),
E RJ are parameters of 01x1 and & Øx* K. respectively. Particularly, spatial modeling @ k can be achieved by DW convolution, local pooling or spatial attention (Woo et al. 2018)) with kernel size of K x K. As such, our AFBO with SCFBO in Fig. (b) has two-branch structure with spatial modeling and channel mapping.
where (Wo E dd be E Md and ( E RRRK Je
As for Eq. bilinear representation 2 is followed by an activation function a (e.g., SiLU (Elfwing et al.] 2018) and GeLU (Hendrycks & (016)) However, such bilinear operation potentially hurts the performance of such an activation function. Specifically, for a special case of parameter- shared branch in Eq. (4). Ż = f(x)² and f indicates spatial-channe factorized mapping. Mean- while, both outputs of SiLU and GeL.U are power functions of inputs, which are more sensitive to
4

---

Published as a conference paper at ICLR 2025
1al
(b)
(c)
Figure 2: Diagram of mapping matrices for various channel mapping strategies. (a) Fully-connected layer with dense parameters. (b) Block-wise channel mapping with block-diagonal sparsity con- straint (G = 4). (c) Our Grouped Cross Channel Mapping (GCCM) (G 4). (d) Our Overlapped Cycle Channel Mapping (OCCM) (G 4).
probable noise of 2()² than one of f(X), potentially decreasing the final performance. Therefore, as shown in Fig. | we switch dot product with activation function for our SCFBO practice. More experimental comparisons can refer Table
3.2.2 TTRUCTURED-SPARSIII CHANNEL MAPPING
By considering hidden dimension (Md) plays a crucial role performance of VIT (Noshad al. 2019 Srrihhar et al. 2023). we are likely to adopt a large expansion ratio M (e.g.. M = 4) in channel mapping (i.e., point- wise convolution 81x1 with parameters of (W E Rd Md b E RMdd) to guarantee performance, which subsequently leads to high computational complexity. Therefore, we further present two structured-sparsity channel mapping strategies to decrease computational complexity of SCFBO while guaranteeing performance. Given x; be features of i-th token in X, channel mapping performs a linear projection with parameters of { [W: E Rd Md b R Mdj as
2, WXx; b As illustrated in Fig. 2 (a), W is generally fully-connected layer with M d2 parameters. As commonly used strategy, group channel mapping in Fig. 2 (b) has been studied reduce computational complexity of deep CNNs (Chollet, 2017; Krizhevsky al.) [0017] SSnndlrr et al. 2018), where W. can be regarded as incorporation of a block-diagon sparsity constraint:
Ws(1) 0
W:(2)
W bw
(5)
:
0
W!(G)
where (i) E Mdd/Gxd/G and G indicates number of group. Clearly, computational complexity of block-wise channel mapping in Eq. (5) is 1/G of the original WS. Although model efficiency, block-wise channel mapping suffers from inferior performance due to the absence of information interaction among different groups (refer to comparison Table I
To guarantee both efficiency and effectiveness, we present two structured-sparsity channel mapping strategies, namely Grouped Cross Channel Mapping (GCCM) and Overlapped Cycle Channel Map- ping (OCCM). As shown in F non-overlapped groups, respectively. Then, GCCM adopts a parameter-shared linear mapping for i-th and i+Gith groups (i E [1, G)) of outputs by using different groups of inputs. As such, differ- ent groups of inputs are fed into same linear mapping to realize information interaction between groups. Particularly, mapping matrix of our GCCM can be written as
(c), our GCCM divides the inputs and outputs into G and 2G
W1(1)
0
(:(
0
(2
W
(6)
go
W²(2)
0
W:(G)
Wo(1)
0
0
5

---

Md)), while computational cost is Published as a conference paper at ICLR 2025
Table 1: Comparisons of various channel mixers in terms of complexity, performance, and inference latency. Particularly, the parameter numbers in dashed boxes and the results of Top-1 accuracy are
reported with instantiation of DeiT-T (Touvron al. 2021a) on ImageNet- IK.
Melhod
FLOP)
(op-1(())
Lasency (ms)
PEN
((2M4")
5.7
0(4Ma")
1.3
72.2
11.4
GLU
O(aMa²)
O'UMa")
Group FEN
O O(2M) Md)
5.7
O 3M," 2Md2)
1.3
Group
01 Md)
Of a3ja2 2Md
69.8
SCFBO-GC
23 Md)
O aMra 2Md
1.1
72.2
SCFBO-OC
O 22260--1 Da Md)
6.7
D( 4MM0 1382 2Md)
Q
1.5
75.6
14.5
AFBO
O
0g Md)
6.0
o( (MaMa
02 2M
1.3
74.6
13.2
where (i( E MMd/2Gxa/G and our GCCM in Eq. (6) is only 1/2G of the original W in term of computational complexity. Furthermore, as illustrated in Fig. 2 (d), our OCCM proposes to use overlapped inputs realize information interaction between groups in cycle manner.
[W W.L,). (1), 0
W Wn(()]
0
[W [WC) (2),
(2)
0
W
DC
[. GG 1), W R 1)] 0
(G)
(W)
where our OCCM sets the overlapped range by half of dimension of grouped features. (W) R. Md/4GxafC x W ((((g) Mddddxxd/G are parameters of g-th group for channel mapping, whose numbers are 2/G of those in the original W. From Eq. (6) and Eq. 7 we can see that our GCCM and OCCM can efficiently perform information interaction between groups by introducing appropri- ate structured-spane sparsiity constraints channel mapping. Note that our AFBO can control the trade-off between performance and complexity by adopting various group numbers and overlap ratios.
3.3 AFBO VARIANTS
Based on the proposed GCCM and OCCM, we can implement SCFBO in Eq. 44 in different ways, resulting in three AFBO variants, i.e., SCFBO-GC, SCFBO-OC, and asymmetric SCFBO (AFBO).
SCFBO-GC construct SCFBO-GC by realizing channel mappings of two branches lying in SCFBO with GCCM (6), which has parameter number of O 2M
C2 + Mdd) and computational cost
(FLOPs) of ( 4 2Md2 with group number of G. G
SCFBO-OC When channel mappings of two branches lying in SCFBO are replaced by OCCM 7 we construct SCEBO-OC with parameter number of O 2M l) + Mdd and computational cost
G
(FLOPs) of O 4M 1) 2Md?), where is group number of OCCM.
G
Asymmetric SCFBO (AFBO) As hybrid scheme, replace left and right branches of SCFBO with OCCM (5) and GCCM (6), respectively. leads to an asymmetric SCFBO, i.e., AFBO. Specif-
ically, parameter number of AFBO is O( Md2 O(2Md G. 2M(GG- 2M where G and G2 are group number of GCCM and OCCM.
M(G2-1)d
G
G2
G2
Model Selection via Complexity Analysis To further analyze performance and complexity of three AFBO variants, we compare them with the original FFN, GLUU (Shazeer, 2020), and Group FFN in Eq. ( As listed in Table our AFBO variants have similar or lower computational complexity than their counterparts from a theoretical perspective. By instantiating our AFBO variants with G, and G2 = 4 with DeîT-T (Touvron et al. 202 our SCFBO-GC achieves similar performance with FFN but has fewer parameters and FLOPs. SCFBO-OC improves FFN by 3,4% and brings extra model complexity. Therefore, and complexity trade-off. which is used throughout all experiments. Finally, our AFBO respectively brings 3.8% and 2.0% gains over Group FFN and GLU, while having comparable model complexity.
AFBO combines with GCCM and OCCM to achieve performance

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
In this section, we first describe implementation details of our AFBO, and make comparison on ImageNet- 1K (Krizhevsky et al. (017) and its variants, i.e., ImageNet-C (Hendrycks & Dietterich, 2019), ImageNet-A (Hendrycks et al. (021b), ImageNet-R (Hendrycks et 2022() and ImageNet- Sketch (Recht et al. 2019). Besides, we transfer our models to object detection and instance seg- mentation on MS COCO (Ln et al. 2014). Finally, we conduct ablation studies on ImageNet- I..
4.1 IMPLEMENTATION DETAILS
As plug-in module, we apply our AFBO to various ViTs and MLP-Mixer architectures by replac- ing the original FFN modules, including DeiT Touvron et al. (021a), Swin ViT (Liu et al. 2021 PoolFormer Yu et al. 2022), LVT Yang et al. 2022). PVTv2 Wang et al. 2022). CycleMLP (Chen et al. (023), HireMLP-Tiny (Guo et al. 2222bb) GC VIT (Hatamizadeh et al. 2023) and VisionL-
LaMA (Chu et al. 2024). To match the original FFN in terms of model complexity, we set G₁ and G2 to 2 and for AFBO, respectively. For spatial modeling, we use 3x3 DW convolution default setting. To train models on ImageNet- K, we adopt exactly same strategies as the original works with 224 224 inputs. For evaluation on object detection and instance segmentation, we adopt Mask R-CNN (He et al. 2017) and RetinaNet (Lin et al. 2020) as baseline detectors, where Poolformer-SW YYu al.] 2022) and Swin-T (Liu et 2021) along with FPN (Lu al., 2020) are used as backbone models. All detectors are implemented using MMDetection toolkit (Chen et al.) 2019) with the default settings. Specifically, the shorter side of input images is resized to 800, and all the models are optimized using SGD with weight decay of le-4, momentum of 0.9 and mini-batch size of 16. The learning rate initialized to 0.01 and is decreased by factor of 10 after 8 and epochs, respectively. All programs implemented by PyTorch (Paszke et al. 2019) and run server with A6000 GPUs. The source code is available at
4.2 IMAGE CLASSIHCATION ON IMAGENET-LE 1K
We first validate the effectiveness of our AFBO by comparing state-of-the-art (SOTA) models and its counterparts on ImageNet-IK: Besides, we directly adopt the trained models to ImageNet- C (Hendrycks & Dietterich 2019), ImageNet-A (Hendrycks al. [2021b), ImageNet-R (Hendrycks) et al. 2021a) and ImageNet- (Recht et al. 2019) to verify the robustness of our AFBO.
4.2.1 COMPARISON WITH SOTA
To verify generalization of our AFBO, we compare 20 SOTA models with and without AFBO. As shown in Table our AFBO enhances the performance of all models on both IN-1K and 4 out- of-distribution (OOD) variants, while having less or comparable model complexity (i.e., parameters and FLOPs). Specifically, for tiny ViT models, AFBO respectively achieves 1.3% and 2.4% gains over LVT and DeiT-T on IN-IK, while bringing clear improvement on four OOD variants. For small VIT models, AFBO obtains 0.5%~11.9 and 0.6%~5.6% gains on IN-1K and four OOD variants, respectively. Notably, AFBO achieves 0.5% gains over the recently proposed GC ViT- XT and Pyramid VisionLLaMA-S on IN-IK, and brings more than 1.5% gains on IN-A Besides, AFBO can also improve MLP-Mixer models (i.e.. CycleMLP and HireMLP) with comparable model complexity. For middle and large ViT models, AFBO respectively brings 0.7% and 0. 0.1% gains for DeiT-B and Swin-B - on IN- IK, but has less model complexity. In particular, AFBO still brings clear improvement (0.6%~3..7) 3.7%% for medium and large models on four OOD variants. In terms of model latency, AFBO brings extra affordable inference time over the original models. These results above clearly demonstrate that AFBO can help existing ViT's achieve hetter performance and complexity trade-off, which provides promising solution to improve generalization and robustness of ViTs.
4.2.2 COMPARISON WITH COUNTERPARTS
To further evaluate the effectiveness of our AFBO, we compare with several counterparts, includ- ing SwiGLU (Fang et al. 2024). ConvNeXt block (Liu et al. 2022). IMLP (Xu et al.] 2024). SCHEME (Sridhar et al. 2023) and ConvGLU (Shi 2024), where all experiments are conducted on IN-IK and IN-R R..D.. by using DeiT (Touvron et al. 202 Poolformer (Yu et al. (022) and Swin-
T (Liu al. 2021) backbone models. Since source code is unavailable, we duplicate the results of
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparisons with various widely used vision models on ImageNet-IK (IN-IK) and four variants, including ImageNet-C (IN-C), ImageNet-A (IN-A), ImageNet-Robustnes (IN-R) and ImageNet-Sketch (IN-SK). For comparison, the original MLP modules for all baseline models are replaced by proposed AFBO module. Among OOD datasets, IN-C calculates the mean corrup- tion error (mCE) as metric, where the smaller mCE means the better robustness of the models under corruptions. All other benchmarks use Top-1 accuracy the metric if no special illustration.
Params. FLOPs
Latency
IN-1K
IN-C
IN-A
IN-R
IN-SK
Method
(M)
(G)
(ms)
(*%)
(1%)
(1%)
(1%)
(1%)
LVT Yang et al. 2022
0.8
7.7
74.8
75.0
7.5
34.6
23.0
AFBO (ours)
-0.3
+0.0
9.1
76.1
73.3
8.5
35.1
23.7
DeiT-T Touvron al. 2021a
5.7
1.3
11.4
72.2
71.1
7.3
32.6
20.2
AFBO (ours)
+0.3
+0.0
13.2
74.6,2.4)
66.2(u.0)
8.3
(8.5(5.0)
(2.1
PoolFormer-S12 Yu al. 2022
11.9
1.8
77.2
69.8
7.0
37.7
25.4
AFBO (ours)
+0.1
+0.1
11.2
79.1
65.0
8.9
42.5
27.6
GC ViT-XXT (Hatumizadch al.
2023
12.0
2.1
11.1
79.9
72.9
19.0
41.9
29.2
AFBO (ours)
+0.3
+0.2
12.5
81.2
(2.5
22.4(a.4)
43.5(L.))
31.6(14)
PVTv2-B1 Waan al. 2022
14.0
2.1
11.5
78.7
65.1
14.6
41.7
28.9
AFFBO (ours)
-0.1
+0.2
13.2
80.1
63.6
(6.7(2.1
44.1
31.1
CycleMI.P-BI (Chen et al.
2023
15.0
78.9
64.5
11.6
41.6
29.1
AFBO (ours)
-0.2
+0.1
13.4
80.0
62.3
14.2g2.))
44.3
30.4
HireMLP-Tiny Guo al. 2022b
17.0
2.1
12.8
78.9
65.3
12.8
41.5
29.0
AFBO (ours)
-0.2
+0.1
15.1
80.2.13)
62.9(2.1)
15.6
43.8(2.1)
(0.5
GC VIT-XT (Hatamizadeh al.
2023
20.0
13.2
82.0
75.3
26.7
44.3
32.0
AFBOO (ouns)
+0.4
+0.2
14.9
82.5
74.8
29.2
44.9
32.6(0.6)
Pyramid-VisionLLLame LaMA Chu et
2024
22.0
2.6
14.8
81.6
58.1
23.5
41.8
28.9
AFBO (ours)
+0.1
+0.1
16.6
(2.1
(6.1
25.3
45.3
(2
Poolformer-S24
Yu 2022
21.4
3.4
14.3
80.3
62.2
14.5
41.4
28.9
AFBO (ours)
+0.6
+0.3
16.4
81.5
57.8
18.0g..)
44.5
30.8(L.))
DeiT-S Touvron al. 202
22.0
12.9
79.8
54.6
19.8
41.9
29.4
AFBO [ours)
-0.3
-0.1
15.2
81.1
53.3
20.8
45.3
31.9(2.))
Swin-T (Liu al.
2021
28.0
4.5
12.6
81.2
62.0
21.7
41.3
29.0
AFBO [ours]
-0.5
-0.1
15.5
82.1
56.4...6
26.0
45.8(a.)
31.7(2.0)
PoolFormer-536 Yu et 2022
30.9
16.6
81.4
60.0
18.5
42.1
30.2
+FFBO (ours)
+0.8
-0.5
18.8
81.9
(7.1
21.8
43.8(L.))
31.6(L4)
Swin-S (Liu
2021
50.0
8.7
25.2
83.0
54.9
32.9
44.9
32.0
AFBO (ours)
-1.9
-0.2
29.9
83.3
(2.1
33.3
47.4(2.1)
(4.5(2.))
Wyramid-VisionI..aale
Chu al.
2024
56.0
31.3
83.2
52.1
33.5
46.0
33.5
AFBO (ours)
+0.4
+0.2
32.9
83.5
49.5(2.0)
35.2
48.1
35.8(2.3)
PoolFormer-M36
al. 2022
56.2
29.3
82.1
58.3
23.8
43.3
30.6
AFBO (ours)
+1.0
+0,9
33.6
82.5
54.6
27.5
45.2
(2.)
PoolFoomer-M488
Yu al. 2022
73.5
11.6
31.6
82.5
54.9
25.7
43.7
30.9
AFBO (ours)
+1.4
+1.2
35.7
83.0
52.8
32.0
46.6(2.0)
32.0(L.)
DeiT-B Touvron
2021a
86.0
17.6
35.7
81.8
48.5
27.4
44.9
32.0
-2.1
-0.2
38.8
82.5
45.3
31.1
47.4(2.))
(3.3
Swin-B et 2021
87.8
15.4
34.2
83.3
$4.4
35.8
46.6
32.4
AFBO (ours)
-3.1
-0.4
37.3
83.6
51.5
364(0.6)
49.0
34.0
Pyramid-Visionl V.ss-iii.o...MM....... Chu al. 2024
99.0
18.0
40.1
83.6
54.4
38.7
47.8
34.6
AFBO (ours)
+0.5
+0.2
42.9
83.7
52.0
40.2
50.2 36.2
SCHEME (Sridhar et al. 2023) ) and IMLP (Xu et al. Then, we re-implement SwiGLU, ConvNeXt block and ConyGLU for the compared backbones us- ing available source code. As shown in Table S IMLP has less model complexity but inferior performance. SCHEME is also difficult to balance performance and complexity. Particularly, our AFBO achieves higher accuracy than SCHEME- 12, while having less model complexity. Compared to SwiGLU, ConvNeXt block and ConvGLU, AFBO has similar model complexity, while benefit- ing better generalization and robustness. Specifically, AFBO achieves 0.3%~2.00 and 0.4%~~..19 gains over SwiGLU, ConvGLU and ConvNeXt block on IN-1K and IN-R, respectively. Above results clearly show our AFBO achieve better performance and complexity trade-off again.
2024) from the original papers for comparison.
4.3 ОВJECT DETECTION AND INSTANCE SEGMENTATION ON COCO
Furthermore, we evaluate generalization ability of AFBO on object detection and instance segmen- tation tasks. As shown in Table 4 when RetinaNet is used as detector, AFBO brings 1.7% and 0.6% gains over Poolformer-S11 and Swin-T in terms of AP, respectively. For Mask R-CNN,
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparisons with various counterparts ImageNet-IK (IN-IK) and ImageNet- Robustness (IN-R), where the results of "FFN". "IMLP" and "SCHEME" are duplicated from the original works (Xu al. 2024 Srichar 2023). We re-implement "ConvNext", "SwiGLU" and "ConvGLU" by using publicly available source code.
Backhone
Method
Parameter
F1.OPs
IN-IK
IN-R
FFN
5.7 M
1.3
72.2%
32.6%
ConvNeXi Block Liu al.
2022
5,9 M
1.3 G
73.6%
34.2%
IMLP Xu al.
2024
5.0 M
1.1 G
72.6%
33.5%
DeiT-T
Touvron et al. a0
SwiGLU Fang cf 2024
5.7 M
1.3 G
72.6%
33.4%
ConvGI.U Shi
2024
5.9 M
1.4 G
74.2%
37.5%
AFBO (ours)
6.0 M
1.3G
74.6%
38.5 %
FFN
22.0 M
4.6
79.8%
41.9%
IMLP Xu et al. 2024
18.8 M
3.9
80.0%
N/A
DeiT-S
Touvron ct al.
2021a
SwiGLU Fang el 2024
22.0 M
4.6G
80.4%
42.5%
ConvGLU Shi 2024
21.6
4.5G
80.6%
44.7%
AFBO (ours)
21.7 M
4,5
81.1%
45.3
FFN
11.9 M
188
77.2%
37.7%
IMLP Xu et 2024
9.8
1.5G
77.2%
N/A
SCHEME-12 Snichar et al.
2023
16.7 M
2.6
78.5%
N/A
Pool-S12 Yu et al.
2022
SCHEME-44 (Sidhar ct al.
2023
7.2 M
1.0
73.0%
N/A
ConvGIJU (Shi 2024
12.0 M
1.9
78.1%
41.7%
AFBO (ours)
12.0 M
1.9
79.1%
42.5%
FFN
21.4 M
3.4
80.3%
41.4%
IMLP
2024
17.2 M
2.7
80.7%
N/A
Pool-S24 Yu et al.
2022
SCHEME:12 Sridhar
2023
30.8 M
4.9
80.5%
N/A
ConvGLU Shi
2024
22.0 M
3.5
81.1%
43.8%
AFBO (ours)
22.0 M
3.7G
$1.5 %
44.5% %
FFN
28.0 M
4.5
81.2%
41.3%
IMLP Xu 2024
24.3 M
3.9
81.5%
Swin-T Liu 2021
SCHEME Sridhar
2023
19.7 M
3.1
79.6%
SCHEME-1 12 Sridhar atal. 2023
36.9
5.9.
81.7%
ConyGLU 2024
28.5
4.7.G
81.8%
44.8%
AFBO (ours)
27.5
4.4
82.1 %
45.8%
AFBO respectively brings 1.2% and 0.8% gains over Poolformer-S12 and Swin-T T on object de- tection tasks, while achieving 1.2% and 1.3% improvement on instance segmentation, respectively. Besides, Poolformer-S12 and Swin-T with our AFBO are superior to other compared backbone mod- els that share similar parameters. These results demonstrate that our AFBO can be well generalized to various tasks, e.g., object detection and instance segmentation.
4.4 ABLATION STUDIES IMAGENET- 1K
In this subsection, conduct ablation studies assess the effects of components and various configurations of our AFBO. All experiments are performed on IN-IK with the backbone of DeiT-T.
Effect of Different Core Components. Our AFBO involves two core components, i.e., spatial modeling (SM) and channel mapping (CM). To assess their effect, we construct various variants, in- cluding AFBO with only CM (AFBO-C), AFBO with only SM (AFBO-S), AFBO with CM followed by SM (AFBO-CS), and AFBO with SM followed by CM (AFBO-SC), The results are presented in Table where we can see that both SM and CM modules improve DeiT-T. Besides, AFBO-C su- perior to AFBO-S with fewer parameters, indicating that CM is more important than SM. AFBO-CS achieves further gains (0.9%~11.35 by combining CM with the subsequent SM, meaning that SM and CM are complementary. Compared with AFBO-CS. AFBO-SC decreases model complexity, but achieves inferior accuracy. More analysis on SM and CM can refer to the appendix.
Effeet of Various Configurations. Furthermore, we evaluate the effect of microcosmic configura- tions on our AFBO. Specifically, we analyze how position and type of activation function (α) impact final performance. In AFBO, be placed before after the dot product operation, indicated by pre-act and post-act, respectively. For the type of activation function, we compare ReLU (Nair & Hinton 2010), GELU and SiLU (Hendrycks Gimpel 2016). As listed in Table
post-act

---

Published as a conference paper at ICLR 2025
Table 4: Results of object detection and instance segmentation on MS COCO, where all backbones are pretrained on ImageNet- and 1 x learning schedule used. A and A pm denote bounding box AP and mask AP, respectively. TP denotes Throughput, i.e., images per second.
Params. Flops TP
AP (%)
APso (%6
APrs (%)
APs (%)
APи (%)
APL (%)
(M) (G) (HZ)
RetinaNet 1x
ResNet-18 (He et al. 60
21.3
173
29.4
31.8
49,6
33.6
16.3
34.3
43,2
Puolfformer-S12
2022
21.7
189
18.5
36,2
56.2
38.2
20.8
39.1
48,0
22.0
189
17.9
37.9
57.2
40.1
21.30.6
41.0g
50.2g.2
PVT-TTiny (Wang al. 2021b
23.0
201
23.5
36.7
$6.9
38.9
22.6
38.8
50.0
PVT-Small waang
202th
34.2
258
16.4
40.4
61.3
43.0
25.0
42.9
55.7
ResNet- 50 cHa 2010
37.7
220
26.3
36.3
55.3
38.6
19.3
40.0
48.8
Swin-T (Lireral, 2021
38.5
245
20.4
41.5
62.1
44,2
25.1
44.9
55.5
Swin-T+AFBO
38.0
245
19.6
42.1o.6
62.6p.5
44.9u.7
25.4g.3
45.5u.6
56.40.9
Params. (M)
Flops
TP
AP
AP
50
AP
APm
A
AP
(G)
(HZ)
(%)
(%)
(%)
(%)
(%)
(%)
Mask R-CNN 1x
ResNet-18 (He
2016
31.2
198
19.6
34.0
54.0
36.7
31.2
51.0
32.7
PoolFoorer--112 et
2022
31.6
207
13.0
37.3
59.0
40.1
34.6
55.8
36.9
SerrmerrS11+Ate B0
32.0
207
12.4
38.6
60.2,
42.4
35.812
56.3g
38.2
PVT-Tiny (Wang et al.
b
32.9
225
14.9
36.7
59.2
39.3
35.1
56.7
37.3
Twins- WVES
44.0
228
14.4
42.7
65.6
46.7
39.6
62.5
42.6
CMT-S (Guo al. 2022a
44.5
249
N/A
44.6
66.8
48.9
40.7
63.9
43.4
PVT-Small (Wae
a.. 2021b
44.1
282
12.4
40.4
62.9
43.8
37.8
60.1
40.3
ResNet50 (He ctal al 2010
44.2
246
17.0
38.0
58.6
41.4
34.4
55.1
36.7
Swin-T (Liu etal al 2021
48.0
267
14.3
43.7
66,6
47.7
39.8
63.3
42.7
Swin-T+ATH
47.5
267
13.6
44.5o.8
67.20.6
48.2a.5
41.1
63.7p.4
42.90.3
Table 5: Effect core components on AFBO.
Table 6: Effect of different configurations.
Method
Description
Params. Top-1
OCCM GCCM
Top-1
Latency
Baseline
5.7 M
72.2%
SiL.U 73.5%
13.2 ms
Pre-act
AFBO-C
Only Channel Mapping
5.7 M
73.7%
ReLU 73.7% 12.6 ms
V
SiLU 74.2%
13.2 ms
AFBO-S
Only Spatial Modeling
6.0 M
73.3%
SiLU 74.6%
13.2 ms
AFBO-CS
Channel Mapping followed by Spatial Modeling
6.0 M 74.6%
Pre-act
GeLU 74.6%
ReL.U. 73.7%
13.1
12.6 ms
Spatial Modeling followed by Channel Mapping
ReLU 74.2%
12.7 ms
AFBO-SC
5.8
73.4%
/
SiLU 74.3%
13.1 ms
strategy is clearly inferior to pre-act, whose reason is analyzed in Sec. Sec.3.2.1 The similar philosophy also observed GLU (Shazeer 2020). Besides, ReLU is inferior to SiLU and GELU in terms of accuracy while having slightly less model latency. Additionally, the combination of activation function with OCCM branch outperforms one of GCCM and both them. It may be caused by the fact that OCCM has better feature learning ability than GCCM (i.e., SCFBO-OC performs better than SCFBO-GC ing ability, leading an inferior performance. As contrast, ReLU is more tolerated for dot product operation. In this work, SiLU activation with OCCM is used as the default setting.
Table and the dot product two activation functions may hurt feature learn-
5 CONCLUSION
This paper made an attempt solve the paradox of performance and complexity trade-off in improv- ing FFN of ViTs. To this end, we proposed an Asymmetric Factorized Bilinear Operation (AFBO), whose core idea to explore rich statistics of token features improve feature learning ability of Transformer in efficient way. Specifically, our AFBO presents spatial-channel factorized bi- linear operation to model second-order statistics for feature learning, while two structured-sparsic channel mappings are developed to reduce computational complexity while guaranteeing perfor- mance. Extensive experiments on various tasks by using several ViT models clearly demonstrate our AFBO has good ability achieve performance and complexity trade-offs. In the future, will adopt our AFBO to foundation models and other applications (c.g., semantic segmentation (Zhou al. 2017) and visual question answering (Antol al. 2015)).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62276186, Grant 61925602, Grant 62222608, and Grant 62436002; in part by the National Science and Technology Major Project under Grant 2022ZD01 16500; and in part by the CAAl-Huawei MindSpore Open Fund under Grant CAAIXSJLLJ-202-000CC and in part by the Haihe Lab of ITAI under Grant 22HHXCJC00002.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV, pp. 2425-2433, 2015.
Guiping Cao, Shengda Luo, Wenjian Huang, Xiangyuan Lan, Dongmei Jiang. Yaowei Wang, and Jianguo Zhang. Strip-MLP: Efficient token interaction for vision MLP. In ICCV, pp. 1494-150% 2023.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLaa detection toolbox and benchmark. CoRR, abs/1906.077 155, 2019.
Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. CycleMLP: A MLP-Like architecture for dense visual predictions. IEEE Trans. Pattern Anal. Mach. Intell., 45 112::14284114300, 2023.
Zekai Chen, Fangtian Zhong, Qi Luo, Xiao Zhang, and Yanwei Zheng. modeling for edge computing. In WASA, pp. 393-405, 2022.
Edge Efficient visual
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pp- 1800-1807, 2017.
Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. VisionLLaMA: A unified LLaMA interface for vision tasks. CORR, abs/2403.00522, 2024.
Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In NeurIPS, pp. 3965-3977, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. ViT: Improving vision transformers with soft convolutional inductive biases. In ICML, pp. 2286-2296, 2021.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, pp. 133-941,22 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling your kernels to 31x31: Revisiting large kernel design in CNNs. CVPR, pp. 11953-11965, 2022.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. ICML, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.

---

Published as a conference paper at ICLR 2025
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. In NeurlPS, pp. 26183-26197, 2021.
Yuxin Fang. Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. Image Vis. Comput., (49:105171, 2024.
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In CVPR, pp. 317-326, 2016.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeViT: a vision transformer in convnet's clothing for faster inference. In ICCV. pp. 12249, 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. CMT: convolutional neural networks meet vision transformers. In CVPR, pp. 22165-1217,, 2022a.
Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision MLP via hierarchical rearrangement. In CVPR, pp. 816-826, 2022b.
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In ICML, pp- 12633-12646, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pp. 2980-2988, 2017.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint 1xXiv:6666084/11 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pp. 8320-8329, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pp. 13262-15271 2021b.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. CoRR, abs/1912.1218D) 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The INaturalist species classification and detection dataset. In CVPR, pp. 8769-8777 8778, 2018.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-exciiation networks. IEEE Trans. Pattern Anal. Mach. Intell., 42()):2011 2020.
Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle trans- 2021.
former: Rethinking spatial shuffle for vision transformer. CoRR,
Zilong Huang, Xinggang Wang. Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, and Thomas S. Huang. CCNet: Criss-cross attention for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 45(6):6896-690) 2023.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. Commun. ACM, 60(6):84 90, 2017.
12

---

Published as a conference paper at ICLR 2025
Kunchang Li, Yali Wang. Junhao Zhang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 45(10))
2023.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Factorized bilinear models for image recognition. In ICCV, pp. 2098-2106, 2017.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. VicalViT: Bringing locality to vision transformers. CoRR, abs/2104.05707 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pp. 740-755, 2014.
Tsung-V Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell., 42(2):318-327, 2020.
Tsung-Yu Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear CNN models for fine-grained visual recognition. In ICCV, pp. 1459,1457, 2015.
Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong. Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. In ICLR, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9992- 10002, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pp. 11966-11976, 2022.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-Amold Networks. In ICLR, 2025.
Chang Lu, Xiaochun Lei, Junlin Xie, Xiaolong Wang, and XiangBoge Mu. Panoptic feature pyramid network applications intelligent traffic. In CIS, pp. 40-43, 2020.
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. SOFT: softmax-free transformer with linear complexity. In NeurlPS, pp- 21297-21309, 2021.
Andriy Mnih and Geoffrey E. Hinton. Three new graphical models for statistical language mod- elling. In ICML, pp. 641-648, 2007.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In ICCVW, pp. 3156-3165, 2021.
Morteza Noshad, Yu Zeng, and Alfred O. Hero III. Scalable mutual information estimation using dependence graphs. ICASSP, pp. 2962-2966, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang, Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high- performance deep learning library. In NeurIPS, pp.
2019.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5500, 2019.
13

---

Published as a conference paper at ICLR 2025
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chichhhhhhhh Chen. MobileNetV2: Inverted residuals and linear bottlenecks. CVPR, pp. 4510-4520, 2018.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based localization. In ICCV, pp. 618-626, 2017.
Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020.
Dai Shi. Transnext: Robust foveal visual perception for vision transformers. In CVPR, pp. 17773- 17783, 2024.
Deepak Sridhar, Yunsheng Li, and Nuno Vasconcelos. SCHEME: scalable channer mixer for vision transformers. CoRR, abs/2312.00411 2023.
Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, pp. 16519-16529, 2021.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pp. 7242-7252, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICMI, pp. 10347-10357, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 32-42, 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake A. Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, pp. 12894-12904, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECA-Net: Efficient channel attention for deep convolutional neural networks. In CVPR, pp. 11531-11539, 2020.
Qilong Wang, Jiangtao Xie, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep CNNs meet global covariance pooling: Better representation and generalization. IEEE Trans. Pattern Anal. Mach. Intell., 43(8:25522 2597, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, pp. 548-558. 2021b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVT v2: Improved baselines with pyramid vision transformer. Compul. Vis. Media, 8(3):115447) 2022.
Yan Wang, Lingxi Xie, Chenxi Liu, Siyuan Qiao, Ya Zhang, Wenjun Zhang, Qi Tian, and Alan L. Yuille. SORT: Second- Order response transform for visual recognition. ICCV, pp. 1378,1377, 2017.
Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. CBAM: convolutional block attention module. In ECCV, pp. 3-19, 2018.
14

---

Published as a conference paper at ICLR 2025
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: co-designing and scaling convnets with masked autoencoders. In CVPR. pp. 16133-16142, 2023.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In ICCV, pp. 22-31, 2021.
Yixing Xu, Chao Li, Dong Li, Xiao Sheng. Fan Jiang, Lu Tian, and Ashish Sirasao. Improving MLP module in vision transformer. In Submitted The Twelfth International Conference on Learning Representations, 2024.
Chenglin Yang, Yilin Wang, Jianming Zhang. He Zhang, Zijun Wei, Zhe Lin, and Alan L. Yuille. Lite vision transformer with enhanced self-attention. In CVPR, pp. 11988-11998, 2022.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, pp. 10809-10819, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, pp. 5122-5130, 2017.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and José M. Álvarez, Understanding the robustness vision transformers. ICMI, pp. 27378- 27394, 2022.
A APPENDIX
Table 7: Results of AFBO with different groups numbers G1 and G2. f indicates higher better,
and J indicates lower is better.
GL
G2
Param (M)
Flops
Latency (ms)
IN-IK (%)
IN-A (1%)
IN-R (1%)
IN-SK (1%)
IN-C (1%)
(G)
2
6.2
1.4
13.7
74.8
8.3
38.8
23.6
66.0
2
6.0
1.3
13.2
74.6
8.3
38.5
23.1
66.2
2
5.8
1.3
13.1
74.2
8.1
36.7
22.7
67.1
2
2
5.6
1.3
12.9
73.0
7.6
34.6
21.0
70.4
4
5.7
1.3
12.8
73.5
8.1
36.9
21.4
68.9
A.1 EFFECT OF GROUP NUMBERS OF GCCM AND OCCM
In this subsection, we assess the effect of group numbers G1 G2 by conducting experiments with a backbone of De (Touvron et al. 202 on five datasets, including IN-1K, IN-A, IN-R, IN-SK, and IN-C. The results are given in Table where we can see that larger numbers of G2 bring higher accuracies and more computational costs. In contrast, larger numbers of G1 decrease both accuracies and computational cost. Particularly, performances of varying group numbers G1 and G2 consistent across four benchmarks. To balance efficiency and effectiveness,
respectively set G1 = and G2 = 3 of GCCM and OCCM for LVT (Yang et al. 2022). Swin (Liu et af., 202T)) and MLP-Mixers (Chen et 2023 Guo al. (022bb as the default settings. For the other remaining models, we set G1 and G2 4 for GCCM and OCCM, respectively.
A.2 EFFECT OF DIFFERENT SPATIAL MODELING STRATEGIES
The spatial modeling (SM) of our AFBO can be achieved by various strategies, including parameter- free operations (i.e., spatial max-pooling and avg-pooling), spatial attention and DW convolution.
To investigate the effect different spatial modeling strategies on our AFBO, we compare them ImageNet-1K using the DeiT-T backbone. As shown in Table 8 parameter-free operations, despite their computational efficiency, result in poor performance. We implement spatial attention as suggested in (Woo et al. 2018), which has fewer parameters than DW convolution but also achieves
15

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of different SM.
Table 9: Effect of kernel size on SM.
Backbone
Method
Params.
Top-1
Kernel Size
Params.
Top-1
Max poolioo
5.7
66.4%
5.7 M
73.7%
Avg-pooling
5.7
50.0%
3
6.0 M
74.6%
DeïT-T
Spatial Attention
5.8
71.0%
6.2
74.7%
DW Conv.
6.0 M
74.6%
6.5
74.9%
Table 10: Comparison of DW convolution (Conv) with original convolution as spatial modeling of AFBO in terms model complexity, where DeiT-T and Swin-T are used as backbones.
Backbone
Method
Params. (M)
Flops (G)
Latency (ms)
DeiT-T
DW Conv (3x3)
6.0
1.3
13.2
DeiT-T
Conv (3x3)
13.3
1.7
18.6
Swin-T
DW Conv (3x3)
27.5
4.4
15.5
Swin-T
Conv (3x3)
64.8
4.6
28.3
inferior performance. Therefore, we use DW convolution as the default spatial modeling strategy of AFBO.
Furthermore, we explore the effect of different convolution kernel sizes on DW convolution. In Table [ compare results of DW convolution with different kernel sizes. We observe that larger kernel sizes lead to greater performance gains, but also increase the computational cost. Besides, better balance between performance and efficiency is achieved when the kernel size Therefore, we set the kernel size of as default setting of DW convolution in our AFBO.
3.
Additionally, we compare DW convolution with original convolution for spatial modeling. As shown in Table TO when DW convolution is replaced by the original convolution with a kernel size of 3x3, parameters increase more than times for DeiT-T (Touvron et al., 2021a) and Swin-T (Liu al.) 2021] models. Meanwhile, model latency (inference time per image) increases by about 50%~88%% Therefore, DW convolution can significantly reduce the computational cost and guarantee the effi- ciency of the whole model, compared with the original convolution.
A.3 COMPARISON OF GLU WITH LARGE EXPANSION RATIO OF HIDDEN DIMENSION
Since hidden dimension of FFN heavily affects performance of ViTs, we compare it with GLU by using an expansion ratio of .M ically, experiments are conducted on ImageNet- IK (IN-1K) with backbones of Poolformer-S11 Poolformer-S22 and Swin-T. As shown in Table TT] for an expansion ratio of M = 4, GLU suffers from much higher computational cost, while performance still lags behind our AFBO in terms of both generalization and robustness on IN-IK and IN-R. These results clearly show that our AFBO can achieve better performance and complexity trade-off for ViTs.
further evaluate the effectiveness of our AFBO. Specif-
A.4 COMPARISON WITH BLOCK-WISE CHANNEL MAPPING
In this subsection, we provide more comparisons of our proposed channel mapping strategies with block-wise channel mapping (i.e., Group Convolution) by varying group number G and expansion ratio of hidden dimension M. For comparison, we use our AFBO with only CM by excluding the SM modules, and conduct experiments on ImageNet- 1K dataset by employing DeiT-T and DeiT- S (Touvron et al.) 2221a) as the basic backbones. As shown in Table [3 block-wise channel map- ping is clearly inferior to our AFBO with only CM for similar model sizes. While increasing the hidden dimension can result in further performance improvements (along with increase in pa- rameters), block-wise channel mapping (Group Convolution) still lags behind our AFBO with only CM by about 1%~2%. These results clearly demonstrate that interaction between channel groups is crucial for effective channel mapping, and our AFBO module can efficiently facilitate information interaction among different groups.
16

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of AFBO with GLU by using expansion ratio of hidden dimension M on IN-IK and IN-R.
Backhone
Method
Parameter
FLOPPs
IN-IK Acc.
IN-R Acc.
Inference
Pool-S12
Yu ct al. 2022
GLU (M = 4)
16.6
2.1
78.5 %
37.9%
11.2 ms
AFBO (ours)
12.0 M
1
79.1 %
42.5%
Pool-S24
et al. 2022
GLU (M 4)
30.9
4.0.G
80.5 %
41.3
19.7ms
AFBO (ours)
22.0 M
3.7 G
81,5 %
44.5%
16.4 ms
Swin-T Liu 2021
GLU (M 4)
36.9 M 27.5 M
4.6
81.3 %
44.0 %
19.8 ms
AFBO (ours)
G
82.1 %
45.8
15.5 ms
Table 13: Comparisons of AFBO with block-
wise channel mapping using DeïT-T and DeiT-$ on IN-IK.
Table 12: Comparison of different backbones GLUE benchmarks.
Backbone
Method
M Params.
Acc.
Backbone
CoLA RTE MNLI QNLI
Bascline
5.7
72.2%
GPT
54.3
63.2
82.1
86.4
Group Convolution-A
5.7 M
70.8%
GPT+AFBO
56.8
65.3
82.7
86.9
DefFT
Group Convolution-B
4
5.0 M
69.8%
BERT-base
54.8
67.2
83.5
90.1
Group Convolution-C
6.1 M
71.7%
BERT+AFBO
57.0
68.3
84.8
90.5
AFBO with only CM
5.7
73.7%
BERT-large
60.6
73.7
85.9
91.8
Baseline
22.0 M
79.8%
BERT-larg++AFBO
61.2
74.8
86.5
92.2
Group Convolution-A
22.0 M
79.7%
DeiT'S
Group Convolution-C
23.7 M
80.0%
AFBO with only CM
21.7 M
80.8%
Table Comparison our AFBO and FULL channel mapping on IN-IK.
Backbone
Method
Params (M)
Flops (G)
Latency (ms)
Accuracy (%)
DeiT-T
FULL
7.7
1.6
13.8
74.7
DeiT-T
AFBO
6.0
1.3
13.2
74.6
Swin-T
FULL
36.9
4.6
19.8
81.3
Swin-T
AFBO
27.5
4.4
15.5
82.
A.5 COMPARISON WITH FULL CHANNEL MAPPING
We implement Eq. as the FULL channel mapping method, and conduct additional experiments with the backbones of DeiT-T (Touvron al. 2021a) and Swin-T (Liu et al. (021) IN-IK dataset. The results are listed in Table 14 For De T, the FULL implementation achieves slightly higher accuracy (74.7 % 74.6 %) over our AFBO method at much higher 6.0 M) and FLOPs (1.6 G vs. 1.3 G), and model latency (13.8 ms vs. 13.2 ms). In the case of Swin- our AFBO outperforms the FULL implementation by 0.8% in terms of accuracy while having much fewer parameters (27.5 M vs. 36.9 M) and model latency. Therefore, FULL implementation limited bring further accuracy improvement, but introduces much more computational cost.
of parameters (7.7 M
A.6 COMPARISON OF VIT WITH CHEBYSHEV KOLMOGOROV-ARNOL NETWORKS (KAN)
In this subsection, we compare with ViT modified by KAN (Liu et al. 2025). whose implementation
and results publicly available the website As shown in Table 15 AFBO demonstrates superior performance across different DeiT (Touvron et al., 2021a) variants. Particularly, our AFBO superior KAN by 1.5%, 1.4%, and 1.0% on IN-1K with backbones of DeiT-T, DeiT-S, and DeiT-B, respectively. In terms of model robustness on OOD, AFBO also demonstrates superior performance IN-R across different DeiT variants. Specifically, AFBO outperforms KAN by 4.8%, 3.6%, and 2.8% on IN-R with the backbones of DeiT-T, DeiT-S, and DeiT-B, respectively. These results verify the effectiveness of AFBO.
///RADDM..... md
17

---

Published as a conference paper at ICLR 2025
Table 15: Comparison of KAN and AFBO with DeiTT backbones IN-1 K and IN-R.
Backbone
Method
Accuracy on IN-IK
Accuracy on IN-R
DeïT-T
KAN
73. 1%
33.7%
DeiT-T
AFBO
74.6% (1.5 %)
38.5% (4.8 %)
DeïT-$
KAN
79.7%
41.7%
DeïT-S
AFBO
81.1 % (1.4 %)
45.3% (3.6 %)
DeîT-B
KAN
81.5%
44.6%
DeiT-B
AFBO
82.5% (1.0 %)
47.4% (2.8 %)
Table 16: Comparison of AFBO and the original FFN in terms of performance and CPU inference speed, where Edge ViT-xxs is used as backbone.
Method
Params (M)
FLOPs (G)
CPU (ms)
IN-IK Acc. (%)
Edge ViT-xxs
4.1
1.3
74.4 42.2
Edge ViT- AFBO
4.2
1.3
47.1
76.5
A.7 COMPARISON OF FOR MOBILE MODEL
In this subsection, we conduct experiments by using Edge (Chen et al. 2022 (i.e., well-known lightweight model designed for mobile devices) and comparing on an Intel() Xeon(R) Gold 5218 CPU @ 2.30GHz. As shown in Table 16 AFBO has comparable parameters and FLOPs while bringing 4.9 ms (about 10%) extra model latency. Particularly, our AFBO achieves a substantial accuracy improvement of 2.15% 1..
A.8 GENERALIZATION ON NLP TASKS
We conduct experiments by applying AFBO to NLP tasks, where our AFBO is adopted to three widely used NLP models, including GPT (Radford) 2018), BERT-base (Devlin et al., 2019). and
BERT-large (Devlin et al.) 2019). All models are evaluated on four GLUE (Wang et al. 2019) benchmark tasks, l.e., CoLA, RTE, MNLI, and QNLI As shown in Table 12 our AFBO brings consistent improvement over three original models across all tasks. Particularly. AFBO enhances the performance of GPT, BERT-base, and BERT-large with notable gains in CoLA and MNLI tasks. The results above clearly suggest that AFBO can be generalized well to the NL.P domain.
Table 17: Comparisons with different models on INAT2019.
Method
Parameter
Top-I Acc.
Swin-T (Lu
2021)
28.0 M
71.0%
+AFBO
27.5 M
72.7%
Pyramid-
Pyyraid-Visionl..a VisionI...aMa Chu et al. 2024
22.0 M
68.4%
+AFBO
22.1 M
69.2%
Pyramid- LisionLLaMA-L Chu et al.
2024
99.0 M
74.5%
+AFBO
99.5
75.1%
A.9. CLASSIFICATION ON INATURALIST 2019
To further demonstrate the generalization ability of our AFBO, we transfer the pre-trained models to a long-tailed large-species classification task by using iNa12019 (Horn al. 2018). Specifically, we fine-tune the pre-trained backbones of Swin Transformer (Liu et al. 2021) and VisionLLaMA (Chu cet al., 2024) with our AFBO on iNat2019. As listed in Table 17 our AFBO consistently enhances the performance of these models. Specifically, AFBO improves the performance of Swin-T by 1.7% while reducing model size. Besides, AFBO achieves performance gains of 0.8% and 0.6% over the recently proposed Pyramid- VisionLLaMA-S and Pyramid- VisionLLaMA--L respectively. These results clearly demonstrate the generalization ability of our AFBO on the challenging long-tailed large-species classification task.
18

---

Published as a conference paper at ICLR 2025
A.10 QUALITATIVE ANALYSIS OF AFBO
For qualitative analysis of our AFBO, we visualize the attention maps in the 12th block generated by the original Swin-T model and Swin-T with AFBO by using Grad-CAM (Selvaraju al. 2017). The visualization results are shown in Fig [ and Fig E From them we can see that our AFBO focuses on more discriminative and informative regions, and thus realizes high-accuracy classification on IN-IK. Furthermore, we compute the entropy for the features in the 12th block generated by the original Swin-T model and Swin-T+AFBO. As shown in Table 18] the feature entropy of Swin-T with AFBO is larger than the original Swin-T, which indicates that the features of Swin-T+AFBO are denser than ones of Swin-T, which consistent with the visualization results. Therefore, features of Swin-T with AFBO involve more redundant information, which has proven very important for model robustness (Lin et al. 2024). It accounts for the clear improvement brought by our AFBO on OOD benchmarks.
Table 18: Comparison AFBO and the original FFN in terms of performance and feature entropy, where Swin-T is used as backbone.
Model
Params.
Flops
IN-IK
Feature Entropy
IN-A
IN-R
IN-SK
IN-C
(M)
(G)
(%)
(1%)
(1%)
(1%)
(4%)
Swin-T
28.0
4.6
81.2
2.7
21.7
41.3
29.0
62.0
Swin-T+AFBO
27.5
4.4
82.1
4.7
26.0
45.8
31.7
56.4
19

---

Published as a conference paper at ICLR 2025
GroundTruth
Swin
Swin+AFBO
goblet
3
beer glass
SQLiARED
SolLARED
EQUARES
STRUL HACAT
BOOGE
cocktail shaker
oil filter
steam locomotive
D
OOO
O
Figure 3: Visualization attention maps in the 12th block generated by the original Swin-T model and Swin-T with AFBO by using Grad-CAM (Selvaraju et al. 2017).
20

---

Published as a conference paper at ICLR 2025
GroundTruth
Swin
Swin +A OO
Mountain lion
padlock
harvester
apanese spaniel
Pencil box
Figure 4: Visualization of attention maps in the 12th block generated by the original Swin-T model and Swin-T with AFBO by using Grad-CAM (Selvaraju et al. 2017)
21


Reviews:
[{"review_id": "D7KFoXMkdh", "reviewer": "Reviewer_QBni", "summary": "The paper proposes an Asymmetric Factorized Bilinear Operation block as a drop-in replacement for the FFN in Vision Transformers. It consists of a Grouped Cross Channel Mapping and Overlapped Cycle Channel Mapping to perform cross-channel interactions between channel groups to improve the performance of ViTs. Experiments are conducted on ImageNet variants, Object Detection and Instance Segmentation tasks across different architectures.", "questions": "Table 2 shows that the proposed method is not very effective when scaling up the model as the performance improvements diminish. \nIf the hyperparameters for AFBO are fixed, why do the parameters/FLOPs increase for some and decrease for other models in Table 4?\n\nAlthough Table 4 shows that the proposed method works for object detection and segmentation, it also increases the parameters and FLOPs. It makes it hard to compare. Can increasing the parameters and FLOPs of the baseline model also achieve similar improvement?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed AFBO block introduces two new modules, GCCM and OCCM, to enhance ViT models performance. It is compatible with various ViT models, and extensive experiments across different ViT architectures and tasks demonstrate the method's effectiveness.", "weaknesses": "Although the proposed method is inspired by bilinear models, the final implementation seems engineered to balance accuracy and efficiency. Particularly, L237-244 mentions that SiLU, GELU activation functions cause a performance drop and use this to motivate the design of the proposed AFBO module. Did the authors experiment with other activation functions like ReLU? \n\nMissing inference speed analysis and comparison with FFN and AFBO. Table 1 only shows the latency of AFBO block but not the inference speed of the full model. Group convolutions are known to be slower. Table 1 already shows that AFBO is ~2ms slower than the standard FFN and so the overall speed might be significantly slower than the baseline model. Can the authors provide the full inference speed analysis?\n\nThe GCCM and OCCM modules contain group numbers G1 and G2 that are fixed. The authors do not discuss how to determine the optimal number of groups. Different tasks or datasets may have different optimal group numbers.\n\nMissing FLOPs for Table 4.\n\nAlthough the paper shows experiments with multiple ViTs, it is missing the important baseline of the standard ViT backbone. It is also missing the ablation experiment of implementing equation (4) in a standard way. Although it is expensive, this experiment will show the performance difference when using the proposed GCCM and OCCM modules instead of the naïve implementation.\n\nMissing qualitative analysis and comparison when using the proposed AFBO module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADuPkyLvoi", "reviewer": "Reviewer_vUSe", "summary": "The paper introduces the Asymmetric Factorized Bilinear Operation (AFBO), an approach to replace the feed-forward network (FFN) in Vision Transformers (ViTs). AFBO aims to improve the feature learning ability of ViTs by efficiently exploring rich statistics of token features, leading to better performance and complexity trade-offs. The paper proposed SCFBO,  a method to compute second-order statistics via a factorized bilinear operation, replacing the simple linear projection in FFN. Two strategies, Grouped Cross Channel Mapping (GCCM) and Overlapped Cycle Channel Mapping (OCCM), are introduced to reduce computational complexity while maintaining model performance. Experiments conducted on various vision tasks using different ViT models demonstrate AFBO's superiority in terms of generalization and robustness.", "questions": "1. I would suggest the authors to simplify the second paragraph in Introduction, highlighting the motivation more directly.\n2. A minor in Figure 1, Line 074, the period.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. AFBO improves ViTs by modeling second-order correlations among token features, leading to better feature representations.\n2. The structured-sparsity channel mapping strategies reduce computational complexity, making AFBO more efficient.\n3. AFBO achieves comparable or better performance than existing FFN variants on various vision tasks.", "weaknesses": "I would recommend this paper to be accepted. I only have some questions:\n1. Could the authors provide some visualization of feature interactions to demonstrate the effectiveness of the proposed method.\n2. Why use the DW conv to replace the original conv for spatial modeling, could the authors discuss this more specifically?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h2LSkr66lK", "reviewer": "Reviewer_HXbZ", "summary": "The paper introduces a novel operation, AFBO, designed to replace the feed-forward network (FFN) in vision transformers (ViTs). The core contribution is the proposal of a spatial-channel factorized bilinear operation that models second-order statistics of token features, aiming to enhance feature learning while reducing computational complexity. The paper also presents two structured-sparsity channel mapping strategies, GCCM and OCCM, to further decrease model complexity. Experiments conducted across various tasks and models demonstrate the effectiveness of AFBO in improving performance and complexity trade-offs.", "questions": "1.\tAlthough the paper focuses on vision tasks, a brief discussion on how AFBO might generalize to other domains, such as natural language processing and multimodal representation learning, would be beneficial. It would be better to provide experiments on popular multimodal models to prove its effectiveness. \n2.\tI have a few more questions on the reasons behind the design of channel attention. Why using OCCM on the on side with activation and GCCM on the other side? How will the number of Group G1 and G2 affect the results for OCCM and GCCM and how do you choose these hyperparameters as ablation study and discussion is missing.\n3.\tAs AFBO aims to introduce the spatial correlation of CNN into ViT, should there be more significant performance gain in pure ViT(such as DeiT) than transformers considering similar correlations(such as Swin-T), and furthermore, is this structure powerful enough to introduce 2-d correlations so that there is no need for improvement in attention module before.\n4.\t Authors are recommended to discuss and compare other FFN modification methods, like KAN.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel AFBO operation that serves as a more useful component for ViT visual tasks. \nThe experiments are thorough, containing most visual backbones and visual tasks.\nThe paper is well-written with rather clear explaination.", "weaknesses": "The novelty of this work is rather limited as only FFN’s first layer is conducted change with spatial and channel attention which are normal methods for feature transformation. \nWhile the paper demonstrates the effectiveness of AFBO, there could be more discussion on the theoretical underpinnings of why the strategies of AFBO work well. \nThe performance improvement of the proposed method is relatively insufficient, and it also brings additional computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7HT4Llyei", "reviewer": "Reviewer_QMDZ", "summary": "This paper introduces an Asymmetric Factorized Bilinear Operation (AFBO) as a replacement for the  traditional channel-mixer (FFN/MLP) in ViT architectures, with the goal of improving accuracy-complexity tradeoff. The AFBO first employs a spatial-channel factorized bilinear operation to capture second-order statistics of the input features. To manage computational load, the authors also develop two structured-sparsity channel mapping strategies (OCCM and GCCM), which preprocess features prior to the factorized  bilinear operation in AFBO. The experimental results show that AFBO has better performance than traditional FFN/MLP module in various ViT and CNN-ViT hybrid architectures,  and is competitive to its counterparts.", "questions": "For primary concerns, please refer to the weaknesses noted above. Below are several additional minor questions:\n\n1. Given that ReLU activation is considerably faster than SiLU/GeLU and widely used, why did the authors not consider ReLU activation? Also, providing results with ReLU in Table 6 would offer a clearer comparison.\n\n2. What are the specific configurations for feature-level transformations during the proposed channel mapping operations? Are these operations implemented through split-concat operations, in-feature transformations, or masking? Variations in these implementations could lead to significant differences in practical speed and CUDA compatibility.\n\n3. Since the authors emphasize optimizing the accuracy-efficiency trade-off as a primary contribution, is the proposed method operable on mobile devices? Given the reliance on self-defined feature-level transformations, can these operations be effectively exported to ONNX formats?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed AFBO utilizes a factorized bilinear approach [1] to enhance FFN performance, presenting a relatively new perspective for channel-mixing mechanisms.\n\n2. The motivation of AFBO are clearly conveyed, with the Factorized Bilinear Operation thoroughly explained. Additionally, the authors provide visualizations of their channel mapping strategies.\n\n3. The comparative experiments are well-structured and extensive. The authors conduct replace-and-play experiments across 20 ViT-like architectures of varying scales, demonstrating the utility and impact of the proposed AFBO.\n\n[1] Li Y, Wang N, Liu J, et al. Factorized bilinear models for image recognition[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2079-2087.", "weaknesses": "1. While the authors provide theoretical complexity metrics (e.g., FLOPs and parameter counts), the on-device speed of the proposed components is not well illustrated. Metrics such as latency and throughput, which are widely adopted for evaluating actual operating efficiency, are not fully reported. In Table 1, the authors include limited latency results, showing that the proposed method has slower runtime compared to the original FFN and GLU. However, the specific configurations for these latency comparisons are unclear, and on-device speed metrics are omitted for the all remaining experiments. These makes it difficult to evaluate the efficiency contribution of the proposed method.\n\n2. The AFBO framework introduces two complex channel mapping schemes to reduce the effective channel width during mixing, leveraging feature redundancy. However, instead of these complex mappings, why not explore a simpler partial-channel projection approach, as commonly used for complexity reduction [2,3]? The paper lacks experiments demonstrating that the proposed channel mapping strategies outperform this simpler method.\n\n3. Overall, the AFBO design is intricate, which may hinder fast inference. The AFBO framework employs two complex channel mapping operations with multiple linear layers, along with split-concatenate operations for feature preprocessing, followed by two depth-wise convolutions for spatial modeling and a SiLU activation function. Both depth-wise convolution and SiLU activation are known to have good theoretical efficiency but may perform slowly in practice. In contrast, the original FFN employs only two linear layers and a single activation function, highlighting a potential disadvantage in AFBO's real running speed.\n\n4. The authors claim that the Factorized Bilinear Operation better captures second-order statistics than traditional FFN. However, the paper lacks intuitive visualizations that could illustrate the improved feature modeling achieved by AFBO over FFN, as well as theoretical/statistical comparisons of features learned by both methods. This weakens the claimed contributions of the Factorized Bilinear Operation.\n\n5. Compared to the main experiments, the ablation studies lack clarity. For example, in Table 5, the paper presents several AFBO variants, but the structural details of each variant are unclear. In the AFBO-S configuration, which uses only spatial modeling, it is unclear whether channels are split into two parts for processing by two depth-wise convolutions or treated as a single unit. Similarly, in AFBO-C, which employs only channel modeling, it is unclear if different channel mappings involve activation and multiplication or if features pass directly to the final linear projection. Furthermore, there are no experiments directly validating the effectiveness of the Factorized Bilinear Operation, raising questions about whether the accuracy gains stem from the added depth-wise convolutions for spatial encoding rather than the bilinear operation itself.\n\n6. The detailed configurations of the proposed AFBO are not presented clearly. Readers may need to re-read sections multiple times to fully grasp the architecture. For instance, in Figure 1, feature dimensions are not annotated, making it challenging to understand each operation's role. In part (b) of Figure 1, after the left spatial modeling (SM) module, the use of product and positional embedding-like markers is unclear, with no definition provided in the figure or caption. It appears that positional embedding is applied in AFBO, which seems incongruous with the methodological statements.\n\n7. This paper defines the proposed methods and variants using numerous long and similar abbreviations (e.g., AFBO, SCFBO, OCCM, GCCM, SCFBO-GC, SCFBO-OC, AFBO-C, AFBO-S, AFBO-CS, AFBO-SC), which are difficult to remember and require repeated reading for clarity.\n\nIn conclusion, although this paper demonstrates FFN improvements in its comparative experiments, there is insufficient explanation for the mechanism behind these improvements, and some figures and tables lack clarity. The experiments do not offer direct evidence that the factorized bilinear operation and channel mapping schemes are truly effective compared to simpler alternatives. Moreover, the omission of on-device speed comparisons, with only theoretical complexity provided, raises questions about the method's actual efficiency improvements in practical settings. It is recommended that the authors address these concerns to strengthen the paper.\n\n[2] Chen J, Kao S, He H, et al. Run, don't walk: chasing higher FLOPS for faster neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 12021-12031.\n\n[3] Han K, Wang Y, Tian Q, et al. Ghostnet: More features from cheap operations[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 1580-1589.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)