Memory Efficient Transformer Adapter for Dense Predictions
Authors: Dong Zhang
Rui Yan
Pingcheng Dong
Kwang-Ting Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Transformer
Vision Transformer
Transformer
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In this paper, we propose META, a straightforward and high-speed ViT adapter that enhances the model's memory efficiency and reduces memory access time by minimizing inefficient memory access operations.

Abstract: While current Vision Transformer (ViT) adapter methods have shown promising accuracy, their inference speed is implicitly hindered by inefficient memory access operations, e.g., standard normalization and frequent reshaping. In this work, we propose META, a simple and fast ViT adapter that can improve the model's memory efficiency and decrease memory time consumption by reducing the inefficient memory access operations. Our method features a memory-efficient adapter block that enables the common sharing of layer normalization between the self-attention and feed-forward network layers, thereby reducing the model's reliance on normalization operations. Within the proposed block, the cross-shaped self-attention is employed to reduce the model's frequent reshaping operations. Moreover, we augment the adapter block with a lightweight convolutional branch that can enhance local inductive biases, particularly beneficial for the dense prediction tasks, e.g., object detection, instance segmentation, and semantic segmentation. The adapter block is finally formulated in a cascaded manner to compute diverse head features, thereby enriching the variety of feature representations. Empirically, extensive evaluations on multiple representative datasets validate that META substantially enhances the predicted quality, while achieving a new state-of-the-art accuracy-efficiency trade-off. Theoretically, we demonstrate that META exhibits superior generalization capability and stronger adaptability.

Published as a conference paper at ICLR 2025
MEMORY EFFICIENT TRANSFORMER ADAPTER FOR DENSE PREDICTIONS
Dong Zhang Rui Yan Pingcheng Dong', Kwang-Ting Cheng'
'The Hong Kong University of Science and Technology
2 AI Chip Center for Emerging Smart Systems (ACCESS), NNajjing University fdongz, timcheng) eust rulyane edu, en; pinghhenn donge ust .hk
ABSTRACT
While current Vision Transformer (ViT) adapter methods have shown promising accuracy, their inference speed is implicitly hindered by inefficient memory access operations, e.g., standard hormalization and frequent reshaping. In this work, we propose META, a simple and fast ViT adapter that improve the model's 's mem- ory efficiency and decrease memory time consumption by reducing the inefficient memory access operations. Our method features a memory-efficient adapter block that enables the common sharing of layer normalization between the self-attention and feed-forward network layers, thereby reducing the model's reliance on nor- malization operations. Within the proposed block, the cross-shaped self-attention is employed to reduce the model's frequent reshaping operations. Moreover, we augment the adapter block with lightweight convolutional branch that can en- hance local inductive biases, particularly beneficial for the dense prediction tasks, e.g., object detection, instance segmentation, and semantic segmentation. The adapter block is finally formulated a cascaded manner to compute diverse head features, thereby enriching the variety of feature representations. Empirically, extensive evaluations on multiple representative datasets validate that META sub- stantially enhances the predicted quality, while achieving new state-of-the-art accuracy-efficiency trade-off. Theoretically, we demonstrate that META exhibits superior generalization capability and stronger adaptability.
INTRODUCTION
State-of-the-at computer vision models typically follow the intuitive paradigm of pre-training on large single-mode general datasets, followed by fine-tuning on local task-specific datasets to achieve promising accuracy (Chen et al., 2021; Zhang et al., 2024; Radford et al., 2021; He et al., 2022; Zhang et al., 2023). However, this defacto paradigm requires downstream models to load the entire pre-trained model for fine-tuning, which may result in several undesirable drawbacks such poor structural ficxibility, large optimization gaps, and limited applicability in certain scenarios (Jic & Deng, 2023; Du al., 2022; Jie & Deng, 2022). More importantly, these drawbacks have become more prominent and urgent problems that need to be addressed for dense prediction tasks, particu- larly in the current context of significant model size growth (Gao et al., 2023; Zhang al., 2022b; Radford et al., 2021; Zhang et al., 2023; Kirillov et al., 2023).
Recently, with the increasing dominance of Vision Transformer (ViT) architectures (Han et al., 2022; Liu et al., 2021; Khan et al., 2022), following the parameter-efficient transfer learning mech- anism (Houlsby et al., 2019), ViT adapter has become central approach to learning vision-specific inductive biases from pre-trained ViT models (Hu et al., 2022; Jie & Deng, 2023; Chen et al., 2022b; Ma et al., 2024; Luo et al., 2023; Shao et al., 2024), successfully addressing the drawbacks associ- ated with the pre-training followed by fine-tuning paradigm. The progressive ViT adapter enables downstream VIT models to achieve promising accuracy levels that are comparable to, or even higher than, those achieved by fine-tuning the entire model. In particular, thanks to the utilization of plain ViT models, which contain unique moolti-oddal information. adapters based on these models can ef- fectively promote downstream models to learn beneficial semantic-rich feature representations (Tou- vron et al., 2021; I; Hu et al., 2022; Kirillov et al., 2023; Dosovitskiy et al., 2020). For example, ViT adapter under the plain VIT models has been successfully applied in multiple computer vision tasks,

---

Published as a conference paper at ICLR 2025
e.g., object detection (ODet) (Li et al., 2022b), instance segmentation (ISeg) (Liu et al., 2024), and semantic segmentation (SSeg) (Xie et al., 2021).
Despite the significant progress made by existing ViT adapters (Chen al., 2022b; Liu et al., 2024; Jie & Deng, 2023; Marouf et al., 2024), their inference speed is still somewhat unfavorable, which limits their implementations on edge computing devices (Dong al., 2024a) and applications for real-time recognition scenarios (Marouf et al., 2024). Recently, it has been revealed that in addition to computation and parameter complexity (e.g.. FLOPs and #Params.), inefficient memory access operations (Marouf et al., 2024; Liu et al., 2023), such as standard normalization and frequent re- shaping operations, play a critical role in hindering the ViT's inference speed (He & Hofmann, 2024; Pan et al., 2022; Fournier et al., 2023; Shi et al., 2023). In other words, the inefficient planning of memory access may cause delays and prevent models from fully utilizing the computing power of GPUs or CPUs, resulting in a significant negative impact on the speed of ViT models (Liu et al., 2023; Dao et al., 2022; Venkat et al., 2019; Gu et al., 2021; Ivanov et al., 2021). However, these inefficient memory access operations are usually overlooked factors during the network design pro- cess for dense prediction tasks. In this work, we explore how to solve this problem and accelerate the inference speed of ViT adapters in downstream models. Our solution is to decrease memory time consumption by reducing layer normalization and frequent reshaping operations.
We propose simple and fast memory effi- cient transformer adapter (META) that can improve the model's memory efficiency and decrease memory time consumption by re- ducing the inefficient memory access opera- tions. As illustrated in Figure 1, our META demonstrates advantages over existing ViT models and ViT adapters in terms of mem- ory consumptions and inference time costs. The main contribution of this work is the proposal of memory-efficiet adapter block that shares normalization operations between the self-attention and feed-forwe forward network layers, which exist in parallel manner (Ref. Sec. 3.2). Within this block, the gross-shaped self-attention is employed to reduce the reliance for frequent reshaping operations. Consequently, the proposed two reverse designs are capable of significantly reducing the memory consumption of the ViT adapter, resulting ea in an improved inference speed. Moreover, to enrich local inductive biases for dense predictions, a lightweight convolutional branch is introduced into the memory-efficien adapter block. In the pro- cess of interacting with the ViT backbone, a cascaded mechanism is further proposed to compute different head features, which can enhance the diversity of the obtained feature representations. We conduct extensive experiments for ODet, ISeg and SSeg on two challenging datasets, namely MS- COCO (Lin et al., 2014) 4) and ADE20K (Zhou et al., 2017), for evaluating our META. The obtained results demonstrate that META substantially enhances prediction quality, attains a new state-of- the-art accuracy level, reduces the number of parameters and memory consumption requirements, and achieves faster inference speeds. Theoretically, we also prove that META can exhibit superior generalization capability and stronger adaptability compared to existing ViT adapter methods
VII
VII Adapter
ME ME
Training Params.
2 High
E Low
E Low
pplicattinn gaps
High
Low
Low
Memory consumptions
High
High
Low
Inference time costs
G High
R High
R Low
Qualitative performance comparisons of differ- ent models with respect to training parameters, application gaps, memory access costs, and inference time costs.
Figure 1:
2 RELATED WORK
Vision Transformer (ViT). Current ViT models for state-of-the-art computer vision tasks can be roughly divided into two camps: plain ViT models, which are designed to learn general vision fea- tures (e.g., ViT (Dosovitskiy et al., 2020), DeiT (Touvron et al., 2021) and TDE Transformer (Tou- vron et al., 2021)). and hierarchical ViT models, which are designed learn vision- -spcifi features (e.g., Swin Transformer (Liu et al., 2021) and PVT (Wang et al., 2021)). Both camps of ViT models have their pros and cons. For example, the hierarchical VIT models can learn powerful vision- specific feature representations, which makes them usually perform better than the plain VIT models on accuracy (Han et al., 2022; Khan et al., 2022; Kirillov et al., 2023). However, their unvarnished disadvantage is the lack of multi-modal pre-training information for dense predictions (Chen et al., 2022b). On the other hand, although the plain VIT models have weak prior assumptions, which re- sults in lower accuracy compared to the hierarchical ViT models, their multi- pre- pre-training in-
Due to page limitations, the theoretical analysis will been provided in the supplementary materials.
2

---

Published as a conference paper at ICLR 2025
formation has great potential to provide semantic-rich feature representations for downstream mod- els (Guo et al., 2022; Zhang et al., 2023; He et al., 2022; Zhang et al., 2024; Jia et al., 2022). In this paper, we focus on the plain ViT models for dense predictions. Our contribution is the proposal of straightforward and memory efficient ViT adapter, starting from the reduction of inefficient memory access operations.
VIT for Dense Predictions. Dense predictions, including ODet, ISeg, and SSeg, aim at predicting a semantic mask where each pixel/object in the given image is assigned a class label (Ranftl et al,, 2021: Zhang et al., 2023; Vandenhende et al., 2021). For ISeg, the obtained mask also distinguishes different objects of the same class. With the development of ViT technologies, dense prediction models based on ViT have become one of the default choices for state-of-the-art methods (Liu et al., 2021; Wang et al., 2021; Dosovitskiy et al., 2020). These methods can be mainly divided into two categories: those based on Transformers and those that incorporate a combination of CNN layers. The former, e.g., SegFormer (Strudel et al., 2021), HRViT (Gu et al., 2022), Segvit (Zhang et al., 2022a), Swin UNet (Hatamizadeh et al., 2021), and ISTR (Hu et al., 2021), extract features from given image and connect with task-specific head networks to achieve specific recognition purposes. The latter compensates for the lack of local inductive bias in the pure ViT architecture by introducing local convolutional layers, and representative methods include ConFormer (Peng et al., 2021), CVT (Wu et al., 2021), NextViT (L.i et al., 2022a), and CAE-GReaT (Zhang et al., 2023). While the former
capture long- range dependencies and achieve favorable recognition capability, they require sufficient training samples to optimize the model, and further improvement is needed on memory access and inference speed. To this end, we propose a solution to the problem of limited inference speed caused by frequent memory consumption using a pre-trained ViT model.
VIT Adapters. Adapter methods are originally derived from the NLP tasks, such as PALS (Stickland & Murray, 2019) and NLP adapter (Houlshy et al., 2019). These methods add small number of trainable modules to a pre-trained network, so that downstream fine-tuning models can quickly adapt to specific datasets and tasks via a parameter-efficient transfer learning manner (Hu et al., 2022; Houlsby et al., 2019; Chen et al., 2022b; Shen et al., 2023). In the computer vision domain, VIT adapters have also adopted the similar paradigm (Jie & Deng, 2023; Chen et al., 2022b; Shao et al., 2024; Luo et al., 2023). ViT adapter method is used to fine-tune large-scale plain ViT models with a small number of trainable parameters and achieve promising accuracy (Jie & Deng, 2023; Li et al., 2022b; Chen et al., 2022b). Inspired by (He & Hofmann, 2024; Jie & Deng, 2022; Chen et al., 2022b; Liu et al., 2023; Mercea et al., 2024), we propose an efficient solution the memory efficiency and minimize memory time consumption of ViT adapters. We also provide theoretical analysis to demonstrate the superior generalization and adaptability of our method.
enhance
3 MEMORY EFFICIENT TRANSFORMER ADAPTER (META)
3.1 OVERALL ARCHITECTURE AND CONFIGURATIONS
As illustrated in Figure 2, the network takes an arbitrary RGB image I E #xx33 as input and predicts a semantic mask O E "xx xC' as output. H and W denotes the image height and width, respectively, and C' denotes the class size of the used dataset. Following (Jie & Deng, 2023; Li et al., 2022b; Chen et al., 2022b), the whole network mainly consists of two parts: the upper part a pre-trained plain ViT model (e.g., ViT (Li et al., 2021)), which consists of 4 blocks. The i-th = 2, 3, 4) block contains a set of semantic features F with the spatial size of 1/4, 1/8, 1/16, and 1/32 of I, respectively. The lower part is a trainable VIT adapter, which includes spatial prior module as in (Chen et al., 2022b) for I's encoding, where the encoded features within each block denote F sp. as well as a set of cascaded memory efficient adapter (MEA) injectors (Ref. See. 3.3) and cascaded MEA extractors (Ref. See. 3.4) that act on each ViT backbone block. The "cascaded"' refers to a computational scheme that is incorporated into the different heads of the proposed injec- tors/extractors, which can enhance the diversity of the obtained feature representations. Concretely, for the i-th trainable MEA injector, it injects the updated task-specific features back into the i-th VIT block and into the i-th extractor (as shown in Figure 2 (b)). For the i-th trainable MEA extrac- tor, it extracts the i-th general features for the (i+ 1) injector (as shown in Figure 2 (c)). This process is repeated until the final ViT block is reached. Due to the incorporation of strategies aimed at reducing inefficient memory access operations in our method, which exhibits enhanced memory efficiency and reduced memory time consumption compared existing ViT adapters, consequently
3

---

Published as a conference paper at ICLR 2025
Postion Embediding hi
The pre pe-troined WT model
Patch Embedding
Block
Block
Block
Deroe Prediction Head Netwwrrk
The adapter
Input Image
Spatial Prfor Module
C-MEA Injector
C-MEA Extroctor
C-MEA Injector
C-MEA Extractor
C-MEA Injectirr
C-MEA Extractor N
Fist
(0)
pers f
Shored Norm
---- yrs
the i-th MEA
#bf (SN)
MLF Layer
DWConm
Concar & & propetionn
Part
10
the h-ch MEA
GLUE
Chammet Woukk-Wioe
Shaped Attention
Non-Linear Activation
DWCComV
fox (0)
GIUE
(NN))
Projection
MLP Layer
x DWVConv
[a] Memory Efficient Adapter (MEA)
(b) Cascaded MEA Injector
(c) Cascaded MEA Extractor
Figure 2: Overall architecture of META. Our primary contribution is the proposal of MEA block in (a), which serves as the fundamental component for the injector in (b) and extractor in (c).
leading to accelerated inference speed. We will provide a detailed explanation of the basic compo- nent and mechanism of the MEA block in the following See. 3.2 and how can be deployed in the MEA injectors and MEA extractors to form a cascaded scheme.
3.2
MEA BLOCK
The MEA block aims to interact the features extracted from the ViT backbone and the spatial prior module. Our block consists of the attention (i.e., Attn) branch, the feed-forward network (i.e., FFN) branch, and the lightweight convolutional (i.e., Conv) branch. The three branches exist in a parallel form, which is beneficial for computing in GPUs. As shown in Figure 2 (a), consider arbitrary block, the input is Fsp and F.a. and the output F is obtained by concatenating the features from the Attn, FFN, and Conv branches along the channel dimension, and then passing them through feature projection layer. Therefore, this process can be formulated as:
F (Coonat (CooooC(CCCC.F....... sp: Fest): (is):F(F ((F sp. Fvit); C(F.p; Fos);; F. Fpi Fva)),
(1)
Attn Branch
FEN Branch
Cony Branch
where A(.), F(-), and C(-) denote the operation of the Atte, FFN, and Conv branch, respectively. In MEA, these three branches exist in a parallel manner. Concat(-) denotes the feature concatenation operation along the channel dimension, and Convaxa(-) denotes a x convolution with the output channel size of 256 for feature projection. In particular, to construct an memory efficient block, following (He & Hofmann, 2024; Wang & Komatsuzaki, 2021), the Atnn and FFN branches are subjected to a shared layer normalization operation, which leads to a decrease in memory time consumption associated with the normalization operations.
Attn Branch. In our work, we adopt the cross-shaped self-attention (CSA) to reduce the model's access to the memory unit, which allows for the computation of the attention matrix across different spatial dimensions without requiring the input tensor be frequently reshaped (Dong et al., 2022; Tu et al., 2022). Specifically, CSA first performs self-attention separately along the horizontal and vertical dimensions of the given features. The outputs A (F sp: Fuit) and Ay sp: Fost) of these two parallel groups are then concatenated along the channel dimension, followed by a feature projection operation via xx3 convolution with the output channel size of 256, to form the output A(F sp, Fois) ) For the horizontal self-attention, the input features F Fep and Fus are first subjected to the shared layer normalization, and then divided into M ove-llerlapping horizontal stripes, where each stripe has the spatial size of s x W (i.e., = H /M). Then, the self- atteenteeoo (Vaswani et al., 2017) is performed on each stripe, which can be formulated as:
biz All ( sp, F F.it) = SA(LN(FFT WQ), LN(F W LN(FF) WV)), vär
where LN(-) denotes the shared layer normalization operation. SA(.) denotes the classical single head self-attention (Vaswani et al., 2017). m denotes the m-th stripe and m = 2.., M. Wº, W and WV are used to project the input features into the queries, keys, and values spaces, re- spectively. The sub-attention results of different stripes are merged together to form the output
4

---

Published as a conference paper at ICLR 2025
Ar (Fsp. sp. Foit) of the horizontal group. In equation 2, we choose Fsp queries and Fea as keys and values for the sake of example. In our injector and extractor Sec 3.3 and See 3.4, the roles Esp and F Fost be swapped depending the specific requirements. The attention computation along the vertical dimension Av (F
Fost) is also performed a similar fashion.
FFN Branch. Following the common setting (Vaswani et al., 2017; Tolstikhin et al., 2021), to enable interaction between the obtained features in the channel dimension, our FFN branch consists of two MLP layers and a non-linear activation layer (Saxe et al., 2013; He & Hofmann, 2024), where the MLP layer consists of sequential arrangement of two 3 x 3 convolutional layers.
The imput of the FFN branch is Esp and F esst. which are sequentially processed by feature concate- nation along the channel dimension, 3 x 3 convolution with the output channel size of 256, and the shared layer normalization. This process can be expressed as:
F(F spr Fost Tem
Tem LN(Convaxx LN[Convsss(Connatt
(Concat(F.pp Fva))).
The temporary F(F spp Foir Tem are then passed through MLP layer, a non-linear activation layer (Saxe et al., 2013), and another MLP layer to obtain the output F(F sps Foit). Hence, this can be expressed as:
F(F sp, F.it) = MLP(NLA(MLP(FFE) F Foit Peit))em))):
(4)
where MLP(-) denotes the MLP layer operation, and NLA(-) denotes the non-linear activation layer (Saxe et al., 2013; He & Hofmann, 2024).
Conv Branch. Empirical evidence has demonstrated that incorporating local inductive biases into VIT models can provide notable benefits for visual tasks involving dense prediction (Peng et al., 2021; Wu et al., 2021; Li et al., 2022a; Zhang et al., 2023). To achieve this goal, we introduce a lightweight convolutional branch into the ViT adapter. Specifically, the Conv branch also takes into F and Feiz as the input, which consists of three 1 x depth-wise convolutions concatenated in series, with GLUE layer used activate the features between every two convolutions (Saxe al., 2013; He & Hofmann, 2024). The computation process of the Conv branch is formulated as:
(5) where DC(.) denotes the 1 x 1 depth-wise convolution with the output channel size of 256, and GLU(-) denotes the GLUE layer. With the help of shared layer normalization and CSA, MEA becomes a memory-efficient module with rich local inductive biases. In the following sections, we will provide detailed instructions on how deploy our MEA block injectors and extractors.
c(Fep, Foic) DC(CLLL(DCCGLUUD(((CCCeet Foi))))),
3.3 CASCADED MEA INJECTOR
As illustrated in Figure 2 (b), the cascaded MEA injector adopts the paradigm of self-attention in its computational process, where F. is used as the query, and Fi generated by the last extractor used the key and value. Particularly, to enhance the diversity of the obtained attention maps, cascaded mechanism is proposed to compute different head features in the cascaded MEA injector. Specifically, we first divide the given features into H parts along the channel dimension in a multi- head manner, following the classic self self-attention (Vaswani et al., 2017), where H is set to 16 in our
work. In the computation process of each head, the output of the h-th head J,, and F h.
vit is added into the input features of the next (h++1 1)-th head ph+1,i-1 and ph+ to be used the calculation of wit
subsequent self-attention features, where h = 1,2.... ..., H. The cascaded process continues until the feature from the last head is included in the computation. Finally, the features obtained from these heads concatenated along the channel dimension and projected through 3 x convolution layer before being outputted as the output the i-th cascaded MEA injector. Besides, since the MEA block has only one output, F h. F in our cascaded MEA injector.
ap
3.4 CASCADED MEA EXTRACTOR
As illustrated in Figure (c), following (Marouf et al., 2024; Chen et al., 2022a; Dong et al., 2024b), our i-th cascaded MEA extractor of the h-th head takes the generated by the injector and ph input, where F A.s is used the query, ph is used the key and value. The output of the
5

---

Published as a conference paper at ICLR 2025
i-th cascaded MEA extractor is Flp-- Similarly, the cascaded mechanism is applied following the cascaded MEA injector until the feature from the last head is included in the computation.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
Datasets. To facilitate a fair result comparison with existing methods, conduct experiments, in- cluding the ablation analysis, on two commonly used datasets: MS-COCO (Caesar et al., 2018) for ODet and ISeg, and ADE20K (Zhou al., 2017) for SSeg. Due to page limitations, the implemen- tation details of these datasets will be given in the supplementary materials.
Evaluation metrics. The commonly adopted average precision (AP) and mean intersection-over- union (mloU) are used to assess the model accuracy for ODet (AP*)/ISeg (AP") and SSeg, respec- tively. Besides, to evaluate the efficiency, model Parameters (#P), floating point operations (FLOPs), memory consumption (MC) of the adapter, and frames per second (FPS) are also adopted. The re- ported inference results are measured by A A100 GPUs with per-GPU batch size 2.
4.2 EXPERIMENTS ON OBJECT DETECTION (ODET) AND INSTANCE SEGMENTATION (ISEG)
Baselines and settings. As in (Chen et al., 2022b; Xiong al., 2024; Jie & Deng, 2022; Marouf et al., 2024), Mask R-CNN (He et al., 2017), Cascade Mask R-CNN (Cai & Vasconcelos, 2019), ATSS (Zhang al., 2020), and GFL (Li al., 2020) are employed as the baseline models, where the pre-trained VIT (Li et al., 2022b) is used as the backbone. All the baseline models are pre-trained on ImageNet-Ie 1k by default (Deng al., 2009). Unless otherwise specified, these baselines are set up to be consistent with their papers and the settings of the ViT-Adapter (Chen et al., 2022b) method.
Comparisons with state-of-the-art (SOTA) methods. Result comparisons with SOTA methods with Mask R-CNN (He et al., 2017) for ODet and ISeg are shown in Table 1. From this table, we can obtain the following observations and conclusions: i) Compared
the experimental results of the Mask R-CNN (He et al., 2017) model with ViT (Li et al., 2021), 1), our proposed META can con- sistently improve accuracy for ODet and ISeg across different model scales (e.g., ViT-T/S/B/L (Li et al., 2021)), while only adding a small number of training parameters. Even with different training schedules (i.e., 1x, and x with MS), our method can also improve the model performance, demon- strating the plug-an plug-and-pil play advantage of META. iï) Compared to the SOTA ViT- AiT-Adapter (Chen et al., 2022b) and LoSA (Mercea et al., 2024), META can achieve a new accuracy-efficiency trade-off. For example, in settings with the strong VIT-B (Li et al., 2021) the backbone, performance gain of 0.5%APm under 1 x training schedule and 0.7%APP under 3x training sched- ule with MS while reducing 4.9M model parameters, when compared to ViT-Adapter (Chen al., 2022b). iii Even with stronger pre-trained models. META still improves performance on the base- line models and surpasses existing methods on accuracy, parameters and memory. For example, on the ImageNet- 22k pre-trained weights from (Steiner et al., 2021). our method achieves per- formance gain of 0.8%/0.7%%l
method achieves
under 1 1x/3x training schedule while reducing 8.2M parameters compared to ViT-Adapter (Chen et al., 2022b), which validates its strong learning ability and flexi- ble adaptability. iv) Compared to SOTA ODet and ISeg methods, META also has very competitive performance, For example, compared to the strong ViTDet-B (Li al., 2022b) model, META-T has 2.2%APb and 3. 1AA" gains under the 1x training schedule, and has META-T has 4.9%APb and Compared to SOTA ViT adapter methods such
2. 2.7%APm gains under the 3x training schedule. as AdaptFormer (Chen et al., 2022a), FacT-TK (Jie & Deng, 2023), and LoSA (Mercea et al., 2024), method exhibits superior efficiency in terms of reduced parameter count, decreased FLOPs, and lower MC. Particularly, META only utilizes 62% of the MC of LoSA while achieving superior pre- diction accuracy. Therefore, the experimental results and conclusions above demonstrate that our method achieves better accuracy and higher efficiency in dense prediction tasks.
Superiority performance under different baselines. In addition to Mask R-CNN, we also choose Cascade Mask R-CNN (Cai & Vasconcelos, 2019), ATSS (Zhang et al., 2020), and GFL (Li et al., 2020) as the baselines as Chen et al. (2022b); Mercea et al. (2024). We explore the effectiveness and superiority performance of META these baseline models, where the 3x training with MS

---

Published as a conference paper at ICLR 2025
#P
FLOPs MC
1x schedule
3x +MS schedule
Methods
(M)
(G)
(GB)
AP
AP
AP 50 AP
AP
AP AP
AP
ViT-T (Li et al., 2021)
26.1
205
NA
35.5
33.5
54.9
35.1
44.9 37.0
59.6
39.0
ViTDet-T (Li et al., 2022b)
26.6
209
NA
33.5
54.7
35.2
40.4
60.1
39.3
Via--Aapter T (Chen et 2022b)
28.1
"208"""
15.2
37.1
"5.77"
"3.99
"9.99"46.0
46.0041000"6
"44.1"
AdaptFormer-T (Chen al., 2022a)
27.9
210
17.7
34.2
55.6
35.9
45.2.37.5
60.5
40.8
FacT-TK-T (Jie & Deng, 2023)
27.7
213
16.1
36.6
34.5
55.5
36.7
46.3
41.2
64.7
44.0
LOSA-T (Mercea al., 2024)
27.6
208
13.0
41.4
38.0
60.1
40.5
46.1
39.5
64.2
44.3
META-T
27.4
206
8.1
43.1
38.6
60.8
41.1
47.5
65.0
44.8
ViT-S (Li et al., 2021)
43.8
245
NA
40.2
37.1
59.9
39.3
44.0
63.4
42.2
ViTDet-S (L.i et al., 2022h)
43.8
255
NA
40.6
37.1
60.0
38.8
44.5 40.1
63.6
42.5
ViT-Adaptern (Chen et al., 2022b)
47.8
251
15.2
44.7
39.9
62.5
42.8
48.2
42.8
66.4
45.9
AdaptFormer-S (Chen et al., 2022a)
47.5
255
17.7
41.2
41.2.37.8
60.3
41.0
45.0
64.2
43.4
FaeT-TK-S (Jie & Deng, 2023)
46.8
259
16.1
41.5
37.8
60.7
41.2
45.2.118
65.0
42.9
LoSA-S (Mercea et al., 2024)
46.1
249
13.0
39.8
62.6
42.5
42.5
66.5
46.0
META-S(0us)
45.2
247
8.1
45.5
40.5
63.4
43.5
49.5
43.3
66.9
46.6
ViT-B (Li al., 2021)
113.6
719
NA
42.9
39.4
62.6
42.0
45.8
41.3
65.1
44.4
ViTDet-B (Li et al., 2022b)
121.3
800
NA
43.2
39.2
62.7
41.4
46.3
41.6
65.3
44.5
Vit-Adapier-B (Chen al., 20226)
120.2
730
15.2
41.0
65.1
44.9
49.6
43.6
667.7"
46.9
AdaptFormer-B (Chen et al., 2022a)
118.5
733
17.7
44.5
40.3
63.1
43.0
46.2
42.4
66.2
45.7
FacT-TK-B (Jic Deng, 2023)
118.0
735
16.1
44.9
41.2
64.1
43.2
47.6
67.5
46.3
LoSA-B (Mereea al., 2024)
117.2
722
13.0
45.1
41.8
64.6
44.0
48.6
43.8
67.9
47.0
META-B
115.3
720
8.1
42.3
66.0
45.7
51.2
44.3
68.2
47.5
ViT-L (Li et al., 2021)
337.3 1.907
NA
41.5
65.6
44.6
48.3
43.4
67.9
46.6
ViTDet-I. (Li et al., 2022h)
350.9 1,900
NA
46.2
65.8
44.1
49.1
44.0
68.5
47.6
ViT-Adapter-L' (Chen et al., 2022b)
347.9 1,927
15.2
48.7
43.3
67.0
46.9
52.1
46.0
70.5
49.7
AdaptFormer-L. (Chen al., 2022a)
345.7
1,924
17.7
46.3 42.6
45.8
49.0
68.1
FacT-TK-L (Tie & Deng, 2023)
342.1
1,925
16.1
46.5
42.9
66.5
46.0
48.8
44.5
68.2
47.0
LoSA-L' (Mereea al., 2024)
341.3 1,922
13.0
50.2
43.5
66.8
47.1
53.2
46.2
71.0
48.9
META-L (Ours)
339.7
1,905
8.1
51.8
44.1
67.6
47.8
55.3
46.7
71.3
50.4
Table 1: Result comparisons with SOTA methods under Mask R-CNN (He et al., 2017) on the val set of MS-COCO. & denotes the model is pre-trained on ImageNet- 22k as (Steiner et al., 2021). "MS" denotes the multi-scale training strategy. "NA" denotes not applicable.
strategy is used. The experimental results are given in Table 2. We can observe that META can consistently improve performance across different basclines, and exhibits more accurate and more efficient advantages compared to SOTA ViT adapter methods. For example, based the Cascade Mask R-CNN (Cai & Vasconcelos, 2019), META-S can achieve up to 44.8% APM with only 83M parameters, 797G FLOPs, and 8.1GB MC, which is 1.3% APM higher, 3M fewer parameters, 4G fewer FLOPs, and 7.1GB fewer MC compared to the competitive ViT-Adapter-T method (Chen al., 2022b). Besides, META-B schedule, with 5M fewer parameters and 5G fewer FLOPs than the ViT-Adapter-B (Chen et al., 2022b) model. On ATSS and GFL, our method also achieves competitive 43.2% and 44.0% APM respectively, which are 0.7% and 0.9% APM higher than the competitive vision adapter ViT-Adapter- (Chen et al., 2022b). terms of efficiency, compare with the SOTA LoSA (Mercea et al., 2024) method, our method has fewer parameters (33M v.s. 35M), fewer FLOPP (265G v.s. 268G under ATSS, 279G v.s. 284G under GFL), and fewer MC (8. v.s. 13.0GB under ATSS and GFL), indicating its higher computation and memory efficiency.
achieve 1.7% AP% APm gain on the 3x training
4.3 EXPERIMENTS ON SEMANTIC SEGMENTATION (SSEG)
Baselines and Settings. Following (Chen et al., 2022b; Jie & Deng, 2023; 2022), we select Semantic FPN (Kirillov et al., 2019) and UperNet (Xiao et al., 2018) as baseline models, where the Semantic FPN is trained for 80k iterations and the UpenNet trained for 160k iterations as in (Wang al., 2021; Liu et al., 2021). Unless otherwise specified, the training and inference settings are set up to be consistent with the VIT-Adapter (Chen et al., 2022b).

---

Published as a conference paper at ICLR 2025
Methods
FLOPs MC
FPS
AP
AP
APM 50
AP 75
Cascade Mask R-CNN (Cai & Vasconcelos, 2019)
3x
+MS schedule
Swin-T (Liu et al., 2021)
86
745
NA
15.3
50.5
43.7
66.6
47.1
VIT-S (Li al., 2021)
82
795
NA
16.5
47.9
42.8
62.1
44.8
ViT-Adapter-S (Chen et al., 2022b)
86
801
13.2
13.1
51.5
43,5
63.0
48.2
LoSA-S (Mereca al., 2024)
84
799
13.0
12.0
53.5
43.8
64.1
48.1
META-S
83
797
8.1
14.7
54.3
44.8
66.8
49.5
Swin-B (Liu et al., 2021)
145
982
NA
11.6
51.9
45.0
68.4
48.7
VTT-B (Li et al., 2021)
151
1,100
NA
13.0
50.1
42.7
63.9
44.1
ViT-Adapter-B (Chen et al., 20226)
158
1,106
15.2
8.6
52.1
43.6
64.3
45.2
LoSA-B (Mercea et al., 2024)
155
1,104
13.0
9.0
52.6
43.5
63.8
44.7
META-B:Ous
153
1,101
8.1
11.9
53.8
44.1
65.2
45.6
ATSS (Zhang al., 2020)
3x +MS schedule
Swin-T (Liu et al., 2021)
36
215
NA
17.1
47.2
41.2
54.8
45.5
VIT-S (Li ct al., 2021)
32
NA
18.0
45.2
40.5
52.0
41.8
-iT-Adptee--S (Chen et at., 2022b)
36
272
15.2
14.2
49.6
42.5
"55.1
46.5
LoSA-S (Mercea et al., 2024)
35
268
13.0
16.0
50.3
41.9
54.4
45.0
META-S(0ur
33
265
8.1
16.8
54.9
43.2
55.6
47.4
GFL (Li ct al., 2020)
3x +MS schedule
Swin-T (Liu et al., 2021)
36
251
NA
17.5
47.6
40.8
54.2
46.0
VIT-S (Li et al., 2021)
32
275
NA
18.1
46.0
40.2
52.9
45.1
VIT FfT-Adapter-S' (Chen et al., 20226)
"36
288
15.2
15.33"
550.0"
43.1
54.5"
47.1
LoSA-S (Mercea al., 2024)
35
284
13.0
16.2
51.2
43.3
54.7
47.5
META-S
33
279
8.1
17.6
55.6
44.0
55.2
48.5
Table Result comparisons with SOTA methods under Cascade Mask R-CNN (Cai & Vasconcelos, 2019), ATSS (Zhang et al., 2020), and GFL (Li et al., 2020) on the val set of MS-COCO.
Comparisons with state-of-the-art methods. We show the SSeg experimental results of META under different settings and compare them with SOTA SSeg methods in Table 3. We can observe that i) Our method can consistently improve performance across different baselines, model scales, training strategies, and pre-training weights (including IN-1K, IN-22K, and MM), while having fewer model parameters and memory consumption compared to the advanced ViT-Adapter (Chen et al., 2022b) and LoSA (Mercea et al., 2024) methods. This demonstrates the strong applicability and learning ability of META, which can solve the problem of limited applicability in certain scenar- ios of downstream models in the traditional pre-training and fine-tuning strategy at the application level. äi Compared SOTA methods, META also can achieve a new SOTA accuracy-com trade-off. The parameter from META is only about 0.2% of the overall model parameter count, indicating minimal impact on the total parameter count. Furthermore, the memory consumption of our method accounts for only 55.14% of the SOTA LoSA (Mercea et al., 2024). iii On larger model scales, META achieves higher efficiency with fewer parameters (e.g.. -1.4M on META-T and -4.2M on META-B). This indicates that our method is suitable for fine-tuning large ViT models.
4.4 ABLATION ANALYSIS
In our ablation analysis for ISeg and ODet, we adopt Mask R-CNN (He et al., 2017) under the 3x training schedule with MS as the bascline and report the experimental results the val set of MS- COCO (Caesar et al., 2018), where the ImageNet- Ik pre-trained ViT-B (Li et al., 2022b) is used as the backbone. For SSeg, choose UperNet (Xiao et al., 2018) with 160k iterations the baseline, where the ImageNet- 1k pre-trained ViT-B (Li et al., 2022b) is used as the backbone. We report the single-scale testing results on the val set of ADE20K (Zhou et al., 2017). Due page limitations, we only present the results of the effectiveness of each component in the main paper. Other aspects, such as result comparisons with other variants of ViT adapter methods, will be provided in the supplementary materials as part of the ablation analysis.
Effectiveness of each component. The experimental results of gradually adding META compo- nents the pre-trained ViT-B (Li et al., 2022b) model are shown in Table 4. First, that directly using the Attn branch as the adapter on the pre-trained VIT-B (Li et al., 2022b) model not only fails to improve performance but also significantly decreases APm and Apb (ie.., from
observe
8

---

Published as a conference paper at ICLR 2025
Semantic FPN 80k
UperNet 160k
Methods
Pre- Tra-nrainnng
#P
MC mloU (wo (we/M)) (MS)
#P
MC
mloU mloU
PVT-Tiny (Wang et al., 2021)
IN-1k
17.0
NA
36.6
37.3
43.2
NA
38.5
39.0
ViT-T (Li et al., 2021)
IN-1k
10.2
NA
39.4
40.5
34.1
NA
41.7
42.6
ViT-Adapter-T (Chen et al., 2022b)
IN-IK
12.2 14.0
41.7
42.1
36.1
14.0
42.6
43.6
LoSA-T (Mercea al., 2024)
IN-IK
11.1 10.7 40.8
41.9
36.3
10.7
42.1
43.7
META-T:00e))
IN-Ik
10.8
5.9 42.2
43.0
35.0 5.9 43.1
44.1
Twins-SVT-S (Chu et al., 2021)
IN-IK
28.3 NA 43.2
44.1
54.4 NA 46.2
47.1
VIT-S (Li al., 2021)
IN-IK
27.8 NA 44.6
45.8
53.6 NA 44.6
45.7
ViT-Adapter-S (Chen al. 20226)
IN-IK
31.9
14.0"46.1 46.1
46.6
57.6 14.0 46.2
47.1
LoSA-S (Mercea et al., 2024)
IN-1k
29.8 10.7 45.7
46.7
54.9 10.7 45.8
46.6
META-S(Our)
IN-1k
29.4 5.9 46.9
49.5
54.7 5.9 47.2
48.0
Twins-SVT-L (Chu al., 2021)
IN-Ik
103.7 NA 46.7
45.5
133.0
48.8
50.2
ViT-B (Li et al., 2021)
IN-Ik
98.0 NA 46.4
47.6
127.3 NA
46.1
47.1
ViT-Adapter-B (Chen al., 2022b)
IN-Ik
104.6 14.0 47.9
48.9
133.9
48.8
49.7
LoSA-B (Mereea et al., 2024)
IN-Ik
103.5 10.7 48.3
49.2
131.2
47.3
48.5
META-B.0ur))
IN-1k
100.4 5.9 48.7
49.7
129.1 5.9 49.4
50.5
Swin-L' (Liu et al., 2021)
IN-22k
234.0 NA
234.0
52.1
53.5
ViT-Adapter-B' (Chen et al., 2022b)
IN-22k
104.6 14.0 50.7
51.9
133.914.0 51.9
52.5
ViT-Adapter-L' (Chen al., 2022b)
IN-22k
332.014.0 $2.9
53.7
363.8 14.0 53.4
54.4
LoSA-L (Mercea et al., 2024)
IN-22k
310.5 10.7 52.3
54.0
338.5 10.7 53.0
54.5
META-L (Ours)
IN-22k
307.7 5.9 53.7
54.5
336.3 5,9 54.0
55.2
ViT-Adapter--.." (Chen al., 2022b)
MM
332.0
54.2
54.7
363.8 36.0
14.0 55.0
55.4
LoSA-L' (Mercea et al., 2024)
MM
310.1 10.7 53.3
54.5
341.0 55.1
55.5
META-L (Oars)
MM
307.7 5.9 54.7
55.2
336.3 5.9 55.6
55.9
Table 3: Result comparisons with SOTA methods the val set of ADE20K. "_" denotes there is
such a result in its paper. "IN" and "MM" denotes ImageNet and Multi Modti-Modal, respectively.
ISeg and ODet results on the val set of MS-COCO (Caesar ct al., 2018)
VTT-B (Li et al., 2021) Attn Branch. Branch, Cony Branch, Cascade AP" #P
FLOPsMC
41.345.81136
719
NA 11.5
32.133..11399
".5""1151 113
43.446.5114.4
719
7.5 11.3
30.831.2114.0
719
0.4 11.4
<
14.148.1115..
719
7.5 10.2
43.649.31144
720
8.1 11.0
44.351.2115.3
11.1
SSeg results the val set of ADE20K (Zhou et al.,
2017)
UperNet (Xiao et al., 2018)AAtt Branch. FFN Branch. Conv Branch. Cascade
mloU
#P
FLOPsMCFPS
46.1
127.3
1.841
NA 25.4
33.7
128.0
1,843
5.11 25.0
46.8
128.5
1,846
5.1 24.9
31.1
128.1
1,841
0.3 25.0
47.9
129.0
1,847
23.5
48.0
128.6
1,846
5.9 25.0
49.4
129.1
1,847 5.9 24.9
Table Effectiveness of each component META. "NA" denotes not applicable.
41.3% to 32.1% and from 45.8% to 33.6%). This indicates that using attention alone as the feature interaction method in the adapter is not enough. Then, on this basis, when we use both the Atn branch and FFN branch as adapter components (where the layer normalization is used as shared manner), the model's accuracy is significantly improved, surpassing the baseline model by 2.1% 2.1% AP"/0.77: 7% AP and achieving 43.4% APm1/46.5% Apm, At the same time, the model only has small increase in parameters and no increase in FLOPs. The same conclusion can be drawn from the SSeg task. Furthermore, the independent utilization of the FFN branch the adapter compo- nent does not yield an improvement in accuracy. Instead, it significantly detracts from the model's 'ss accuracy across all three tasks evaluated, After that, continuing to add the Conv branch using the

---

Published as a conference paper at ICLR 2025
cascaded mechanism bring sustained performance improvement, e.g., 0.7% and 0.2% APm, and 1.6% and 2.8% AP", respectively. Finally, when using all components in the adapter, the model for ISeg and ODet achieves the best performance by 44.3% APm and 51.2% APh with 115.3M param- eters, 720G FLOPs, 8.1 GB MC, and 11.1 FPS. The model for SSeg achieves the best accuracy by 49.4 mloU with 129.1M parameters, 1847G FLOPs, 5.9GB MC, and 24.9 FPS. In addition, results in the ablation analysis reveal that the increase in the model's FLOPs and MC is mainly attributed the utilization of the "cascade" mechanism. Despite this, this mechanism can lead to significant improvements in both accuracy and speed.
5 CONCLUSION
We proposed a simple and fast META that aims at improving the ViT model's memory efficiency and reduce memory time consumption by minimizing layer normalization and frequent reshaping operations. The main contribution of this work was the proposal of the MEA block that shares the layer normalization between the self-attention and feed-forward network layers, which exist parallel manner. Within the proposed block, the cross-shaped self-attention was employed to reduce the requirements for frequent reshaping operations. Moreover, lightweight local convolutional branch is introduced into META to enrich local inductive biases. Experimental results on object detection, instance segmentation, and semantic segmentation tasks demonstrated that META can achieve new state-of-the-art accuracy-cos! trade-off and at faster inference speed. In addition to the experimental validation, we also theoretical proved that META exhibits superior generalization capability and stronger adaptability compared to current ViT adapters. In the future, will explore the effectiveness of META on more ViT architectures, such as designing memory-efficient ViT models for wider range of computer vision tasks. Besides, reducing the model's memory footprint is identified as promising research direction to be explored for L.MMs.
ACKNOWLEDGMENT
The authors would like to thank all anonymous reviewers for their positive comments and constructive suggestions. This work was partially supported by the Hong Kong SAR RGC General Research Fund under Grant 16208823 and ACCESS AI Chip Center for Emerging Smart Systems, sponsored by InnoHK funding, Hong Kong SAR. Rui Yan was supported by the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20230302), Nanjing University Inte- grated Research Platform of the Ministry Education Top Talents Program (Grant No. 14380125).
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218, 2018 8.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence, 43(5):1443-1444 1498, 2019. 6, 7,8
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun- jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pp. 12299-12310, 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, pp. 16664-16678, 2022a. 6,6,7
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2022b. 1, 2, 3, 6, 8.9
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, pp. 9355-9366, 2021.
10

---

Published as a conference paper at ICLR 2025
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NewrlPS, pp. 16344-16353 2022. 2
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 008-255, 2009.
Peiyan Dong, Lei Lu, Chao Wu, Cheng Lyu, Geng Yuan, Hao Tang, and Yanzhi Wang. Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile. In NeurIPS, 2024a. 2
Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. In NeurIPS, 2024b.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, pp. 2124-121344 2022. 4
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. 1, 3
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. ar arxiv, 2022. 1
Quentin Fournier, Gaétan Marceau Caron, and Daniel Aloise. A practical survey faster and lighter transformers. ACM Computing Surveys, 55(14s):1-40, 2023. 2
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better linnua- models with feature adapters. International Journal of Computer Vision, pp. 1-15, 2023.
Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Mingjie Liu, Zixuan Jiang, Ray Chen, and David Z Pan. Towards memory-efficient neural networks via multi-level in situ generation. In ICCV, pp- 9229-52383 2021. 2
Jiaqi Gu, Hyoukjun Kwon, Dilin Wang. Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z. Pan. Multi-scale high-resolution vision transformer for semantic segmen- tation. In CVPR, pp. 12094-12103, 2022.
Meng-Hao Guo, Tian-Xing Xing Xu, fiang-Jiang Liu, Zheng-Ning Liu, Peng- Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational Visual Media, 83):331-368, 2022. 3
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE transactions on paltern analysis and machine intelligence, 45(1) 45(1):87-110, 2022. 1,2
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang. Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation brain tumors in mri images. In MICCAI Workshop, pp. 272-284, 2021. 3
Bobby He and Thomas Hofmann. Simplifying transformer blocks. In ICLR, 2024. 2,3,4,5
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961- 2969, 2017. 6, 7
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022. 1,3
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, pp. 2790-2799, 2019. 1,3
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 1, 1,3
11

---

Published as a conference paper at ICLR 2025
Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv, 2021. 3
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: case study on optimizing transformers. In MLSys, pp- 71 71122 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pp. 709-727, 2022. 3
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv, 2022. 1, 6, 7
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In AAAI, pp. 1060-1068, 2023. 3, 6,7
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, ACM computing surveys, 54(10s): 2022.
1,
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In CVPR, pp. 6399-64009 2019. 7
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, pp. 4015-4026, 2023. 1,2
Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv, 2022a. 3.5
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detec- tion. In NeurlPS, pp. 11002-21012, 2020. 6,
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv, 2021. 3, 6, 7, 8, 9
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In ECCV, pp. 296. Springer, 2022b. 2, 3, 6, 7,8
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014. 2
Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang. Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In CVPR, pp. 14420-14430, 2023. 2,
Yifei Liu, Mathias Gehrig, Nico Messikommer, Marco Cannici, and Davide Scaramuzza. Revisiting token pruning for object detection and instance segmentation. In WACV, pp.
2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012- 10022, 2021. 1, 2, 3, 8,
Anwei Luo, Rizhao Cai, Chenqi Kong, Xiangui Kang, Jiwu Huang, and Alex C Kot. Forgery-aware adaptive vision transformer for face forgery detection. arXiv, 2023. 1,3
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654. 2024.
Imad Eddine Marouf, Enzo Tartaglione, and Stéphane Lathuilière. Mini but mighty: Finetuning vits with mini adapters. In WACV, pp. 1732-1741, 2024. 5,
12

---

Published as a conference paper at ICLR 2025
Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab. Time-memory meee---mmoro parameter-efficien visual adaptation. In CVPR, pp. 5536-5545, 2024. 3, 6, 8,
Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In NeurlPS, pp. 14541-14554, 2022.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang. Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In ICCV, pp- 367-376, 2021. 3,5
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., pp. 8748-8763. 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pp. 12179-12188 2021.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv, 2013. 5
Rui Shao, Tianxing Wu, Liqiang Nic, and Ziwei Liu. ceepaake-adapted Dual-level adapter for deepfake detection. International Journal of Computer Vision, 2024. 1,3
Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang, Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang. and Q Tian. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yulong Shi, Mingwei Sun, Yongshuai Wang, Rui Wang, Hui Sun, and Zengqiang Chen. Evit: An eagle vision transformer with bi-fovea self-attention. arXiv, 2023.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research (TMLR), 2021. 6,
Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adap- tation in multi-task learning. In ICMI., pp. 5986-5995, 2019.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pp. 7262-7272 2021.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. In NeurlPS, pp. 24261-24272, 2021.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers. & distillation through attention. In ICML, pp. 10347-10357, 2021. 1,2
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, pp. 459-479, 2022. 4
Simon Vandenhende. Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: survey. IEEE Transactions on Patlern Analysis and Machine Intelligence, 44(7):3614-3633, 2021. 3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017. 4,5
Anand Venkat, Tharindu Rusira, Raj Barik, Mary Hall, and Leonard Truong. Swirl: High- performance many-core cpu code generation for deep neural networks. The International Journal of High Performance Computing Applications, 1375) 2019. 2
Ben Wang and Aran Komatsuzaki. Gpt-j-bb: billion parameter autoregressive language model,
2021. 4
13

---

Published as a conference paper at ICLR 2025
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pp. 568-578, 2021. 2, 3, 7,9
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt:
Introducing convolutions to vision transformers. In ICCV, pp. 22-31, 2021. 3,5
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pp. 418 434, 2018. 8,
Enze Xic, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, pp. 12077-12090, 2021. 2
Yuwen Xiong, Zhiqi Li, Yuntao Chen, Feng Wang, Xizhou Zhu, Jiapeng Luo, Wenhai Wang, Tong Lu, Hongsheng Li, Yu Qiao, et al. Efficient deformable convnets: Rethinking dynamic and sparse operator for vision applications. arXiv, 2024. 6
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. NeurIPS, pp. 1971-4982, 2022a.
Dong Zhang, Chengting Zuo, Qianhao Wu, Liyong Fu, and Xinguang Xiang. Unabridged adjacent modulation for clothing parsing. Pattern Recognition, 127:108594, 2022b.
Dong Zhang, Yi Lin, Jinhui Tang, and Kwang-Ting Cheng. Cae-great: Convolutionall effi- cient graph reasoning transformer for dense image predictions. International Journal of Computer Vision, pp. 1-19, 2023. 3,
Dong Zhang, Pingcheng Dong, Xinting Hu, Long Chen, and Kwang-Ting Cheng, Boundary and relation distillation for semantic segmentation. arXiv, 2024. 1,
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In CVPR, pp- 9769-9768, 2020. 6,
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 633-641, 2017. 2, 6, 9
14


Reviews:
[{"review_id": "Ex5BEcCXnj", "reviewer": "Reviewer_AfWa", "summary": "This paper proposes META, an efficient ViT Adapter that enhances ViT in dense prediction tasks. The adapter block MEA provides the local bias required for image tasks to ViT by introducing conv branches, and significantly reduces memory time consumption by minimizing reshape operations on tensors in the adapter. In classic dense prediction tasks such as Object Detection, Instance Segmentation, and Semantic Segmentation, META outperforms previous adapter methods in terms of fewer parameters and lower memory consumption. Ablation experiments were conducted to verify the effectiveness of the three modules in the MEA block and the improvement of the model with the MEA cascade.", "questions": "1. In Table 2 and Table 3, models of different sizes have the same Memory Consumption (MC). What specific quantity does MC describe, and what measurements lead to this phenomenon?\n2. For different sizes of variants of META, are there differences in the implementation details?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method proposed in this work is simple but effective, achieving higher performance and efficiency in various classic detection and segmentation frameworks.\n2. The paper provides clear and understandable descriptions of the details of each module in the MEA block, with the design purposes of each module being clear and effective.", "weaknesses": "1. There is still space on the main text pages, but the implementation parameters of the model are not clarified, such as the number of cascades. Different designs of each size are also not specified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOa84c4nZc", "reviewer": "Reviewer_ZAUt", "summary": "This paper explores the limitations of Vision Transformer (ViT) adapters in dense prediction tasks, particularly focusing on the issues of memory inefficiency and slow inference speed caused by frequent reshaping operations and normalization steps. The paper proposes a novel ViT adapter named META, which introduces a memory-efficient adapter block that enables the sharing of normalization layers between the self-attention layer and the feed-forward layer. Furthermore, a lightweight convolutional branch is added to enhance the adapter block. Ultimately, this design achieves a reduction in memory access overhead.", "questions": "In your experimental section, you have conducted in-depth explorations of the three tasks: object detection, instance segmentation, and semantic segmentation. To more intuitively demonstrate the specific improvements brought by your model in handling these tasks, are there any relevant visualization results to support this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a simple and fast ViT adapter named META, which addresses the critical yet underexplored issue of memory inefficiency. The quality of this paper is supported by theoretical foundations and empirical validations across various tasks and datasets, demonstrating that META outperforms state-of-the-art models in terms of accuracy and memory usage. The paper is structured clearly, with detailed architectural descriptions and clear explanations of the proposed motivation.", "weaknesses": "In the Atte Branch discussed in this paper, the adoption of the cross-shaped self-attention (CSA) mechanism is a pivotal factor in effectively reducing the frequent reshaping operations of the model. However, the current analysis lacks an in-depth comparison and discussion between CSA and other efficient attention mechanisms, failing to fully elaborate on why the selection of CSA achieves the current experimental results. \n\nThe ablation analysis in this paper are currently limited to the results of instance segmentation on the MS-COCO dataset, whereas your previous experimental work also encompassed the tasks of object detection and semantic segmentation. Therefore, the current ablation analysis regarding the components of the proposed module has certain limitations in terms of generalization. To more comprehensively evaluate the effectiveness and universality of the module components, I recommend conducting corresponding experimental validations for all three tasks of object detection, instance segmentation, and semantic segmentation, thereby ensuring the accuracy and applicability of the conclusions obtained.\n\nIn this paper, there is an inconsistency in the presentation, specifically between Formula (1) and part (a) of Figure 2, which do not align accurately. Although you have explained later in the text that the channel concatenation step for Fsp and Fvit is omitted in the formula, this omission may still lead to misunderstandings among readers. To ensure clarity and accuracy of the content, we recommend that the two should correctly correspond to each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbeE7GRO1H", "reviewer": "Reviewer_oVw4", "summary": "This paper proposes a Memory-Efficient Transformer Adapter, termed META, which reduces memory access costs by sharing layer normalization across multiple modules and substituting standard self-attention with cross-shaped self-attention. Meanwhile, META divides the feature map into smaller parts along the channel dimension and processes these smaller features sequentially. Thereby further reducing memory requirements. Experiment results in object detection and instance segmentation indicate that META achieves better accuracies.", "questions": "Please see weakness section. After the rebuttal, I have addressed many of my concerns and now support the acceptance of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. META introduces a cross-shaped self-attention mechanism and a cascaded process, both of which are grounded in the principles of dividing the entire feature into multiple smaller features to reduce memory costs.\n2. META incorporates local inductive biases by introducing convolutions into the FFN and an additional lightweight convolutional branch. This enables META to achieve better performance in extensive experimental evaluations.", "weaknesses": "1. Insufficient Motivation (1): META claims that the inference speed of previous adapters is hindered by inefficient memory access operations such as normalization and frequent reshaping, but it lacks experimental analysis to support this claim. It is recommended to provide a detailed breakdown of inference time to show the proportion of inefficient memory access operations in META and previous methods.\n2. Insufficient Motivation (2): META aims to decrease memory access costs by reducing frequent reshaping operations. However, I do not observe any reduction. First, the input for attention and layer normalization is $x\\in R^{B\\times L\\times C}$, where B, L, and C denote the batch size, number of tokens, and channels, respectively. In contrast, the convolution accepts input in the format $x\\in R^{B\\times C\\times H\\times W}$. The MEA block mixes many convolutions, layer normalization, and attention. This may result in multiple tensor reshaping operations. Second, the cross-shaped self-attention mechanism divides the features into non-overlapping horizontal/vertical stripes, further compounding the need for tensor reshaping operations. I conjecture that the observed lower memory access costs during experiments are due to the segmentation of the entire feature into multiple smaller features, instead of reducing tensor reshaping operations. I'd like to see a thorough analysis of memory costs associated with each operation in META and previous approaches. This will help clarify where the memory saving comes from.\n3. The results of the ablation study presented in Table 4 indicate that convolutional layers are primarily responsible for the observed improvements (FFN also includes MLP composed of two 3x3 convolutional layers). This raises the question: to what extent does the Attention Branch contribute to these improvements? Consider conducting an additional ablation study that includes the ViT-B along with the FFN Branch, maintaining the same configuration as described in Line 435, but excluding the Attn Branch.\n4. The proposed META is relatively sophisticated and comprises numerous layers (e.g., the cascaded injector includes 16 layers), making it less practical for low-performance hardware. On which hardware do you measure FPS? It is recommended to compare META with other methods on less powerful GPUs such as the V100, rather than A100 or H100.\n5. In Table S3, how do you compare other efficient attention methods? Do you only replace the attention mechanism in the ViT-adapter with other attention mechanisms? Please provide further details regarding the experimental setup.\n6. Other minor comments.\nLine 150：The spatial prior requires clarification, is the spatial prior module utilized here identical to that in the ViT-adapter [1]?  \nLine 96: TDE Transformer, DeiT is more frequently used.  \nLine 182: The term \"which\" appears to ambiguously refer to the prior module rather than the MEA block; it would be beneficial to provide clarification.\nLine 188: In Equation 1, \"Concat\" is a widely recognized abbreviation for concatenation.\nLine 199: \"Attn\" is a more commonly accepted abbreviation for attention compared to \"Atte\".  \nLine 223: Should \"respectively\" be replaced with \"sequentially\"?  \nLine 166 Table S2 in Supplementary Materials: Do you mean separate normalization for different modules? The use of \"common\" may introduce ambiguity.\n\n[1] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)