Skip-Attention: Improving Vision Transformers by Paying Less Attention
Authors: Shashanka Venkataramanan
Amir Ghodrati
Yuki M Asano
Fatih Porikli
Amir Habibian
Venue: ICLR 2024 poster (2024)
Keywords: vision transformers
efficiency
redundacy in attention maps
improved throughput
Primary area: general machine learning (i.e., none of the above)

Abstract: This work aims to improve the efficiency of vision transformers (ViTs).  While ViTs use computationally expensive self-attention operations in every layer, we identify that these operations are highly correlated across layers --  a key redundancy that causes unnecessary computations. Based on this observation, we propose SkipAT a method to reuse self-attention computation from preceding layers to approximate attention at one or more subsequent layers. To ensure that reusing self-attention blocks across layers does not degrade the performance, we introduce a simple parametric function, which outperforms the baseline transformer's performance while running computationally faster. We show that SkipAT is agnostic to transformer architecture and is effective in image classification,  semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS. We achieve improved throughput at the same-or-higher accuracy levels in all these tasks.

Published as a conference paper at ICLR 2024
IMPROVING VISION TRANSFORM- ERS BY PAYING LESS ATTENTION
Shashanka Venkataramanan
Amir Ghodrati Qualcomm Al Research*
Yuki M. Asano University of Amsterdam
Qualcomm Al Research
Fatih Porikli
Amirhossein Habibian Qualcomm AI Research?
Qualcomm AI Research?
ABSTRACT
This work aims to improve the efficiency of vision transformers (ViTs). While ViTs use computationally expensive self-attention operations in every layer, we identify that these operations are highly correlated across layers a key redun- dancy that causes unnecessary computations. Based on this observation, we pro- pose SKIPAT, method reuse self-attte self-attention computation from preceding layers to approximate attention at one or more subsequent layers. To ensure that reusing self-attention blocks across layers does not degrade the performance, we intro- duce simple parametric function, which outperforms the baseline transformer's performance while running computationally faster. We show that SKIPAT is agnostic to transformer architecture and is effective in image classification, se- mantic segmentation, image denoising, and video denoising. We achieve im- proved throughput the same-or-higher accuracy levels in all these tasks. Code can be found at wtttt:t////// //gs::///tt thub. comm Qua comm AT- AI-research/ skip-attention
INTRODUCTION
The transformer architecture (Vaswani al., 2017) has become an important and highly influential model family, due to its simplicity, scalability, and its wide range of applications. While originally stemming from the domain of natural language processing (NLP), with the advent of the Vision transformer (ViT) (Dosovitskiy et al., 2020), this has become a standard architecture in computer vision, setting various state-of-the-art (SoTA) performances learning, semantic segmentation, object detection and video understanding (Caron et al., 2021; Liu al., 2021; Carion et al., 2020; Liang al., 2022: Girdhar al., 2019).
tasks ranging from representation
However, the original formulation of the transformer includes a quadratic computational complexity with respect the number of input tokens. Given that this number typically ranges from 142 for image classification all the way to 128² 16K for image denoising. this constraint on memory and compute severely limits its applicability. To tackle this problem, there have been three sets of approaches. The first leverages redundancies across input tokens and simply reduces computation by efficient sampling, e.g., dropping or merging redundant tokens (Tang et al., 2022: Fayyaz et al., 2022; Yin et al., 2022). This, however, means that the final output of the VIT is not spatially continuous and can thus not be used beyond image-level applications such as semantic segmentation or object localization. The second set of approaches aims to cheaply estimate the attention computation, but generally at the cost reduced performances (Yu et al., 2022; Chu et al., 2021).
In this work, propose novel, far unexplored approach to solving this problem: simply ap- proximating the computationally expensive blocks of the transformer with a much faster, simpler parametric function. To arrive this solution, we first thoroughly analyse the crucial multi-head self-attention (MSA) block of the VIT. Through this analysis, we find that the attention of the CLS
"Wor done during internship Qualcomm AI Research
1 'qqul contribution
Qualcomm AI Research initiative of Qualcomm Technologies, Inc

---

Published as a conference paper at ICLR 2024
Image OmeageNee-11 1M
Selkoueendeed (rragaNae-71) 7XQ)
(ADE20K)
(Pascul VOC2012)
(SID0)
300 Te
Trrughout (ww.sraaaaaea...oooo X
CIU--bours
Throughput (raagesled)
Mocel vanuat
FLOR% (4)
Figure/ 1: Performance of SKIPAT across 5 different tasks. Our novel SKIPAT method achieves superior accuracy vs. efficiency trade-off over the baseline transformer on a wide array of tasks. Circle areas proportional to parameter count.
tokens to the spatial patches has very high correlation across the transformer's blocks, thus leading to unnecessary computations. This motivates our approach to leverage attention from an early part of the model and simply reuse it for deeper blocks basically "skipping" subsequent SA calculations instead of recomputing them at every layer.
Based on this, we go one step further and explore if the entire MSA block of layer can be skipped by reusing the representation from previous layers. We find that a simple parametric function inspired from ResneXt's depth-wing wise convolutions (Xie et al., 2017) can outperform the baseline performance -while being computationally faster in terms of throughput and FL.OPs. Previous works that use convolutions for improving efficiency transformers have proposed merging convolution layers with transformer blocks (Graham et al., 2021), refining MSA representations by introducing convo- lutions inside MSA blocks (Zhou et al., 202 la;b), or replacing MSA blocks with MLP layers (Pan et al., 2022c). In contrast, we propose to leverage redundancies across MSA blocks and approxi- mate them wholly using parametric functions. SKIPAT is general-purpose and can be applied to a VIT in any context: Figure shows that our novel parametric function achieves superior accuracy vs. efficiency trade-off compared to the baseline transformer on a wide variety of tasks, datasets, and model sizes. SKIPAT is architecture agnostic and can be applied to isotropic, hierarchical, and hybrid transformer architectures resulting in superior performances than the baseline.
In summary, our main contributions are follows:
1. We propose a novel plug-in module that can be placed in any ViT architecture for reducing the costly (n²)) Self-Attention computations
2. We show that SKIPAT is agnostic transformer architecture and achieves state-of-the-ar
performances in throughput at same-or-bette bettet accuracies for ImageNet, Pascal- VVC2222,,
SIDD, DAVIS and ADE20K (in the latter of which we obtain 40% speedup)
3. We further demonstrate the generality of our method by obtaining 26% reduction in self- supervised pretraining time (at no downstream accuracy loss) and by demonstrating supe- rior on-device latency
4. Finally, analyse the sources of performance gains and extensively ablate method to provide a model family which can be used for trading off accuracy and throughput
A [CLS]
A[CLs
A
(CLS]
A
[CLS)
ACLS)
A
A
[CIS]
[CLS]
0.48
0.63
0.84
0.95
0.94
0.78
0.95
0.93
0.97
0.62
0.91
0.44
0.72
0.87
0.92
0.96
0.85
0.91
0.97
0.97
0.84
0.94
Figure 2: Attention correlation. Mean of the attention heads from the CLS token of a pretrained VIT-T/16 at different layers from the validation set of ImageNet- IK. Numbers below each attention map indicates the cosine similarity of A cLS) with [ccus]
2 RELATED WORK
Great effort has been made to improve the efficiency of vision transformers (ViT) (Dosovitskiy et al., 2020) from multiple aspects:
2

---

Published as a conference paper at ICLR 2024
Token sampling improves the efficiency either by restructuring images during the tokenization step (Yuan et al., 2021; Han et al., 2021). pruning the redundant tokens over training (Kong et al,, 2022: Tang et al., 2022) or dynamically at inference (Yin et al., 2022; Rao et al., 2021; Fayyaz al., 2022; Chen et al., 2021). Despite their effectiveness in reducing the computational cost in image classification, token sampling methods are hardly applicable to dense prediction tasks, e.g. semantic segmentation and image denoising, where the output image should be spatially continuous. Our approach is complementary to these methods and achieves on-par or higher performance on both classification and dense prediction tasks.
Hybrid architectures such as Uni- former (Li et al., 2022). Mobile- VIT (Mehta Rastegari, )0 and
How do they improve efficiency?
TToken Hybrid Ellicient SKIPAT
others (Liu et al., 2022; Pan et al., 2022a; Mehta Rastegari, 2022), in- corporate efficient convolutional mod- ules into vision transformers. These ar- chitectures achieve this by employing MobileNet blocks in Uniformer, Mo- bileNetV2 blocks in VobileViT, ing stacks of convolutions in the image tokenization step (Graham al., 2021; Wu et al., 2021). Other approaches dis- entangle high and low-frequency representations in MSA blocks (Pan al., 2022b) or replace MSA blocks in the early layers of the network with MLP layers (Pan et al., 2022c). In our work, we use convolutions to accelerate the computation of vision transformers. However, instead of crafting custom blocks, as done in (Mehta & Rastegari, 2021; Pan et al., 2022a; Mehta & Rastegari, 2022; Li et al., 2022; Pan et al., 2022c), we approximate entire MSA block using convolutions. This enables us to apply our method across isotropic, hierarchical, and hybrid transformer architectures. We com- pare SKIPAT with the existing methods relevant to improving the efficiency of vision transformers in Table and show that among the listed methods, only SKIPAT shows all the listed properties.
sampling network attention
Does the method satisfy each property?
Improve throgghuut during inference? Generaiize dense prediction tuskr?
Tackle quadratic complexity of self-artention? Generalize different transformer backbones!
X
Table 1: SKIPAT vs. vision transformers. Compari-
us-
son between SKIPAT and methods that improve the effi ciency of vision transformers. Among the listed methods, only SKIPAT satisfies all the listed properties.
Efficient attention methods aim to reduce the quadratic complexity of the self-attention operation in vision transformers. Several approaches have been proposed, such global downsampling of key and value embeddings (Wang et al., 2021a; 2022a; Wu et al., 2021), performing self-attention in local windows (Liu et al., 2021), alternating between local and global self-attentions (Chu al., 2021; Mehta & Rastegari, 2021; Pan et al., 2022a), or replacing self-attention with a simple pool- ing (Yu et al., 2022). However, redueing self-attention to a local neighborhood limits their ability to model long-range dependencies, leading to significant performance degradation with only moderate speedups (Zhang et al., 2021). In addition, some methods, such as cyclic shift in Swin (Liu et al., 2021), lack efficient support, thus reducing actual efficiency gains in terms of latency. In contrast, our method relies on a few blocks with strong, yet inefficient self-attention operators and lighter, accurate attention estimators in other blocks. As the estimators only use standard convolutional op- erations, our method translates to actual latency gains. The approach of using convolution layers is similar to (Zhou et al., 2021b;a), that introduce convolutions inside MSA to refine attention maps. However, adding a convolution operation at every layer increases computation overhead. Addition- ally, (Xiao al., 2019; Wang al., 2021b; Ying et al., 2021) observed redundancies in attention maps for NLP tasks. Instead of copying attention maps, we propose an efficient parametric function that achieves high throughput while maintaining high model performance in vision tasks.
3 SKIP-ATTENTION
3.1 PRELIMINARIES
Vision Transformer. Let x E hxxxx be an input image, where h x w is the spatial resolution and e is the number of channels. The image is first tokenized into n hw/pp non-overlapping patches, where p x p is patch size. Each patch is projected into an embedding E Rd using linear layer to obtain the tokenized image:
Zo (21; (.......2)) 2n) B
(1)
3

---

Published as a conference paper at ICLR 2024
Here, ";" denotes row-wise stacking. Positional embeddings are added to Zo to retain positional
information. The token embeddings are then input to a L = {l..... : L} layer transformer whose
output denoted as Zr In the supervised setting, a Iearnable token [ I2 is prepended to the tokenized image (1) as Zo = (= (z/eus); Zo) E (x+++)xdd
Transformer Layer. Every layer of the transformer consists of a multi-head self attention (MSA) block followed by a multi-layer perceptron (MIP) block. the MSA block, the input, Z--1 E Raxd for ( E L. is first projected into three learnable embeddings {Q,K, V) E 12ad The attention matrix A, is calculated as
QK* Vd
1 A:
E R
where (.) denotes the row-wise softmax operation. The "multi-head" in MSA is defined by con- sidering attention heads where each head a sequence x π reprojected back to x d using linear layer which is combined with the value matrix
d matrix. The attention heads are
MSA = AV E Raxd
The output representations from the MSA block is then input to the MLP block which comprises two linear layers separated by a GeLU activation (Hendrycks & Gimpel, 2016). At a given layer I, the computational flow of representations through transformer block is denoted as
2, - Z + Z-11
(4)
2 MLP(Z.) Z
(5)
Both the MSA and MLP blocks have residual connections with layer normalization (LN) (Ba et al., 2016). While MSA blocks in each layer of the transformer learn representations independently, in the next subsection, we show that empirically there exist high correlation across these layers.
3.2 MOTIVATION: LAYER CORRELATION ANALYSIS
Attention-map correlation. The MSA block in ViT encodes the similar- ity of each patch to every other patch an n x n attention matrix. This operator computationally expensive with On²?) complexity (2). As ViTs scale up. i.e., as n increases, the complexity grows quadratically and this operation becomes a bottleneck. Recent NLP works (Vig & Belinkov, 2019; Vig, 2019) have shown that self-attention across adjacent layers in SoTA language models exhibit very high correlation. This raises the question is it worth compute self-attention at every layer of a vision transformer?
(a) OKA ((
(b) CKA of 2(
Figure 3: CKA analysis of A and MSA across differ- ent layers of pretrained VIT-T/16. Vanilla ViT-T/16 has high correlation across both attention maps (layer 3 to 10) and MSA (layer 2 to 8)
To address this question, we analyze the correlation of the self-attention maps across different layers of ViT. As shown in Figure 2, the self-attention maps from the class token, A (cLss) exhibit high cor- relation especially in the intermediate layers. The cosine similarity between A A and A can be as high as 0.97, as indicated the bottom of each attention map in Figure 2. We observe similar behavior from other token embeddings, which we analyze in the supplementary material. To quan- tify this correlation, we compute the Centered Kernel Alignment (CKA) (Kornblith et al., 2019; Cortes et al., 2012) between ! and A [CLSS] for every i, E L across all validation samples of ImageNet- IK. CKA measures the similarity between representations obtained from intermediate layers of the network, where a high value of CKA indicates high correlation between the represen- tations. From Figure 3 (a), we observe that ViT-T has a high correlation across A (Cr.)) especially from layers 3 through 10.
4

---

Published as a conference paper at ICLR 2024
SKPAT parometric function c
+
FC
Bation
skip
41-1
reshape
PXT
ECA
RX2d
Vixyix22d
vixxyfozdd
axZd
Figure 4: SKIPAT framework We illustrate SKIPAT on VIT (Dosovitskiy et al., 2020). The SKIPAT parametric function (Φ) uses representations of the MSA block (in solid color) 2 as imput, which undergoes series of transformations. An element-wise summation (+) with the output of the MLP block from layer --1 1 and ZMSA is used as input the MLP block layer The MSA operation (crossed out) is thus not computed and is discarded from the computational graph. With SKIPAT the total number of layers remains unchanged.
Feature correlation. In ViTs, the high correlation is not just limited A Atcus, but the represen- tation from MSA blocks, ZMSA, also show high correlation throughout the model (Raghu et al., 2022). To analyze the similarity across these representations, we compute the CKA between ZWM and MAAA for every i.j C. We observe from Figure (b), that ZMSA also have high similarity across adjacent layers of the model especially in the earlier layers, i.e., from layer 2 through 8.
3.3 IMPROVING EFFICIENCY BY SKIPPING ATTENTION
Based on our observation of high representation similarity across MSA blocks of transformer (subsection 3.2), we propose to leverage the correlation across both the attention matrix and the representations from the MSA block to improve the efficiency of vision transformers. Instead of computing the MSA operation (3) independently at every layer, we explore simple and effective strategy to utilize dependencies across the features from these layers.
In particular, we propose to skip MSA computation one or more layers of a transformer by reusing representations from its adjacent layers. We term this operation as Skip Attention or SKIPAT. As the compute and memory benefit from skipping the entire MSA block is greater than skipping just the self-attention operation (O(ndd nd²) vs. Onndd), in this paper focus former. However, instead of directly reusing features, i.e., copying the features from the source MSA block to one or more adjacent MSA blocks, we introduce parametric function. The parametric function ensures that directly reusing features does not affect the translation invariance and equivariance in these MSA blocks and acts as a strong regularizer to improve model generalization.
SKIPAT parametric function Let Raxd > nxd denote the parametric function that maps output of the MSA block from - 1 to as 2,MSA = + (ZM Here, ZMSA is the approximation of ZMSA The parametric function can be as simple as an identity function, where Aww is directly reused. Instead of computing MSA operation at 1, When using an identity function, due to the absence of MSA operation at the relation across tokens no longer encoded in the attention matrix, which affects representation learning. To mitigate this, we introduce the SKIPAT parametric function inspired from ResNeXt (Xie et al., 2017) as shown in Figure to encode local relations among tokens. The SKIPAT parametric function consists of two linear layers and depth-wise convolution (DwC) (Chollet, 2017) in between, as follows:
Z,MSA the input to the MLP block l.
2,M := ECA (FCCFDFCC(FCC((211
(6)
In the case of supervised learning, we first separate the CLS embeddings from ZMA R(++1)xd into class embeddings ZMIS Rd and the patch embeddings Z E Raad The patch em- beddings are then input to the first linear layer FC, R - R which expands the channel dimension. This is followed by DwC IR v before the DwC operation, we spatially reshape the input matrix to a feature tensor. Han er al. (Han et al., 2022) shows that the behavior of depth-wiss wise convolution operation resembles local attention, which helps learn translation equivalent representations and also reduces the complexity of the para- metric function. The output of the DwC is then flattened back to vector and fed the last FC layer
IR v with kernel
Note that
5

---

Published as a conference paper at ICLR 2024
FC2: R + Rx which reduces the channel dimension back to its initial dimension d. We use GeLU activations after FC1 and DwC. Following (Wang et al., 2020), we use efficient channel attention module (ECA) after FC2 to enhance the cross-channel dependencies. The ECA module first aggregates the features along the channel dimension using global average pooling (GAP). A 1 x 1 convolution with adaptive kernel size proportional to channel dimension is applied followed by sigmoid activation. This operation of the ECA module enhances cross-channel dependencies. We then concatenate the embedding of the class-token with the output of the ECA to obtain 2MM
SKIPAT framework. The overall framework of SKIPAT is illustrated in Figure 4. SKIPAT can be incorporated into any transformer architecture which we empirically show in section 4. Depending on the architecture, one can skip the MSA operation in one or more layers of the transformer. In VīT. as we empirically observe that representations from the MSA block. ZMSA have high correla- tions from layer 2 through 7 (subsection 3.2). we employ the SKIPAT parametric function these layers. This means that we use the ZAA as input to the SKIPAT parametric function and skip MSA operations in layers 3-8. Instead, the features from the output of the SKIPAT parametric function used as input to the MLP block. The computation flow of representations is now:
Z 1 M + Zi-1
Z1 + MLP( (i) + Ze
(8)
Due to the presence of residual connections in the MSA and MLP blocks, which is standard in ViT (Dosovitskiy et al., 2020), the MLP blocks layer through 8 learn representations inde- pendently and cannot be discarded from the computational graph. It is important to note that, with SKIPAT the total number of layers in ViT remain unchanged, but there are fewer MSA blocks.
Complexity: MSA vs. SKIPAT The self aelf-attention operation involves three operations. Firstly, the token embeddings are projected into query, key and value embeddings, secondly, attention matrix A is computed as dot product between Q and K and finally, the output representations are computed as dot product between A and V. This results in a complexity of O + n² ndd). Since d n, the complexity of MSA block can be reduced to (n²d).
The SKIPAT parametric function consists of two linear layers and one depth-wise convolution, which results in a (2nd + r'nd) complexity, where r X is the kernel size of the DwC operation. The overall complexity of SKIPAT can be reduced to Ondd²) since ,2 < d. Thus, SKIPAT has fewer FLOPs than MSA block as O(nd)) < O(??d) when n increases as transformers scale up.
4 EXPERIMENTS
4.1 COMPARISON WITH STATE-OF-THER OF-T
Image Classification We use isotropic transformer architectures like ViT-T/16 (Dosovitskiy et al., 2020), VIT-S/16 (Dosovitskiy et al., 2020), ViT-B/16 (Dosovitskiy et al., 2020), hierarchical archi- tectures like PvT-T (Wang et al., 2021a), PvT-S (Wang et al., 2021a) and hybrid architectures like LIT-T (Pan et al., 2022c) and LIT-S (Pan et al., 2022c) as our backbone comparisons, we follow the experimental settings in (Touvron et al., 2021), (Wang et al., 2021a) and (Pan et al., 2022c) train ViT, PvT and LIT respectively. For VIT. we evaluate SKIPAT against SoTA methods: ViT (Yin et al., 2022), ATS (Fayyaz et al., 2022), PS-ViT (Tang et al., 2022), and Rev-Vit (Mangalam et al., 2022). To the best of our knowledge, these are all the works that improve the efficiency of ViT without modifying its underlying architecture.
ImageNet- K. For fair
From Table 2a. we observe that SKIPAT achieves the best accuracy vs. efficiency trade-off compared to all SoTA methods on different transformer backbones. Notably, we outperform different variants of VIT by 0.1% to 0.4% and improve throughput by 19,, to 25% Interestingly, SoTT methods achieve lower accuracy or are on-par with the baseline. Since SKIPAT uses parametric function to skip computing MSA blocks, our reduction in number of parameters and in FLOPs is comparable to the SoTA. Dehghnai et al. (Dehghani et al., 2022) highlight the significance of using throughput as a metric to measure model efficiency: as the reduction in FLOPs does not necessarily correspond to improvements in latency, as it does not take into account the degree of parallelism or other hardware details. In line with this argument, we observe that while SoTA methods such as ATS (Fayyaz et al., 2022) achieve large reduction in FLOPs, they have lower throughput when compared to SKIPAT.

---

Published as a conference paper at ICLR 2024
BACKBONE METHOD TOP-11
PARAM|
GFLors THROUGHPUTT
(%)
(x10")
(IM/S x10")
VI
72.8
5.7
1.2
5.8
VIT-T/16
A-VIT
71.0
5.7
6,3
ATS
72.7
5.7
PS-VTT
72.6
0.7
6.6
METHOD
224 224
384
384
SKIPAT
73.3
5.8
1.1
6.9
VII-I/16
WT
79.8
22.4
4.6
3.2
V/T + SKIPAT
4.76
15.22
A-VIT
78.6
22.4
3.6
3,4
V/T
ATS
79.7
22.4
3.3
(h) On-device latency (msec)
PS-VIT
79.4
2.6
3.9
RIT-VIT
79.8
22.4
3.6
SKIPAT
METHOD
JACCARDt
CORLoCt
81.8
87.3
17.6
1.2
VIT-T
32.2
39.5
SVITE
81.6
$2.0
11.5
1.3
VIFT SKIPAT
38,0
41.5
VIT-B/16
Rev- VIT
81.5
87.3
17.6
1.2
VIT-S
29.0
40.6
PS-VIT
VIT-S SKIPAT
34,0
41.2
SKIPAT
VIT-B
33.6
36.4
PYFT
PVFT SKIPAT
75.1
1.5
VIT-B SKIPAT
36.8
37.2
76.1
12
1.8
PVTS
PVI-S
SKIPAT
80.1
23
3.4
1.6
baseine
LIT-2-8
82.0
28
3.7
1.4
LITT
LITT
81.1
3.6
1.3
SKIPAT
81.4
18
3.4
1.4
LIT'S
LITS
81.5
27
4.1
1.3
SCPAT
(a) Image classification on ImageNet-IK.
(c) Unsupervised object discovery
Table 2: (a) Accuracy efficiency comparison of SKIPAT with SoTA methods for image resolution 224 x 224. For all the methods, we measure throughput (image/see) with a batch size 1024 on a single NVIDIA A100 GPU, averaged over the validation set of ImageNet- - Additional compar- isons are give in Table 7. (b) On-device latency of vanilla ViT vs. SKIPAT for different image res- olutions Samsung Galaxy $22 powered by Qualcomm Snapdragon 8 Gen 1. (c) Unsuperivsed object discovery using Jaccard similarity and Correct Localization (Corlooc,, on the validation set of Pascal VOC2012. Image sources (from left to right): valley train (licensed under CC BY-SA 4.0), fishing boat (licensed under CC BY-SA 4.0), near Snowshill (licensed under CC BY-SA 4.0)
We also observe from Table 2a that SKIPAT improves the performance of pyramid architectures PvT- by 1.0% and improves throughput by 19%. On average, SKIPAT outperforms variants PvT with 20% gain in throughput. We also observe that SKIPAT enhances the performance of hybrid archi- tectures LIT with an average gain of 12% in throughput. Additionally, LIT-S + SKIPAT achieves the same accuracy baseline S but with fewer parameters, FLOPs, and 7% gain in throughput. Thus, we show the ability of SKIPAT to generalize to different transformer backbones.
Visualizing attention maps and MSA correlation. We analyze the effect of the SKIPAT parametric function by visualizing the mean of attention heads of the CLS token from the last four layers of ViT-T/16. From Figure 5a, we observe that while attention maps from vanilla VIT (last two layers) do not solely attend to the object, the attention maps from SKIPAT accurately focuses on the object. It interesting note that, the attention maps from SKIPAT are also capable of attending multiple objects in the image (Figure 5a: second example). The CKA of the representations from the MSA block in Figure 5b, shows that MSA has lower correlation across layers except between the layers where the MSA operation is skipped (layer 3 to 8). However, unlike vanilla ViT (Figure 3 (b)) the correlation from each layer to every other layer quite low. This shows that our SKIPAT parametric function acts as a strong regularizer and thus improves the representations of the model.
Unsupervised object discovery. We further analyze whether pretrained ViTs can attend to seman- tically meaningful regions of the image when evaluated on different dataset without fine-tuning it. We follow (Caron et al., 2021) 1), and visualize the segmentation masks produced from the final layer of the pretrained SKIPAT on the Pascal- VOC12 (Everingham et al.). From Table 2(c).we observe that while vanilla ViT-S/16 does not accurately attend to the object, SKIPAT is able to localize ob- jects quite accurately without any fine-tuning. To quantify this observation, using Jaccard similarity
7

---

Published as a conference paper at ICLR 2024
METHOD
BACKBONE
MloUt
GFLOPS] THROUGHPUTT
ResNer- 101 (Yaet 2022)
40.7
261
24.1
Semantic FPN (Kinilov ai. 2019)
PoolFormer-
2022)
42.0
PoolFormer-M36 (16 et 2023)
42.4
271
5,4
ResNet. 18 (Ho 2016)
39.9
886
17.1
ResNer- 101 (He at ot., 2016)
44.9
1031
12.0
Swin-T (Liu 2021)
45.8
945
14.2
ConvNeXi-T (Liu 20223
46.7
939
13.7
UperNet (Xiau et al., 2018)
VITT (Dosuviiskiy stal.. 2020)
37.3
212
24.1
VIF + SKIPAT
40.6
173
34.7
VTT S (Dosovitslay al,
44.4
360
19.5
VIT-S + SKIPAT
45.3
283
27.2
VIT-B (Dosoviskkiy al.,
45.6
787
11.1
VIT-B SKIPAT
46.3
633
15.5
Table 3: Semantic Segmentation on ADE20K. All models are pretrained on ImageNet-IK and fine-
tuned on ADE20K. Following Swin (Liu et al., 2021) and ConvNeXt (Liu et al., 2022), we report mloU with multi-scale testing. FLOPs and throughput are calculated on the input size of 2048 512. Throughput of all models are measured with batch size of on single NVIDIA A100 GPU, averaged over 100 forward passes.
and CorLoc (Melas-Kyriaze et al., 2022). As shown in Table 2(c), SKIPAT outperforms different variants of vanilla ViT with significant gap in terms of Jaccard similarity and CorLoc.
Ag
[CLS
[C.s]
(CLS)
A12
[CLS]
baseline
SKIPAT
baseline
PM
CM
SKIAAT
C2
(a) Visualizing A |cts]
(b) CKA analysis of SKIPAT
Figure 5: (a) Mean of the attention of different heads from A from last four layers of ViT-T/16 on the validation of ImageNet-I vanilla ViT. (b) CKA analysis of SKIPAT shows that 2SA has lower correlation between layers. The high correlation is between consecutive layers 2 through 8, where the MSA operation is skipped.
I K. Attention maps shows SKIPAT localizes the object better than
Performance mobile device. To verify the efficiency of SKIPAT low-power devices,
measure its inference time (averaged over 20 iterations) on a Samsung Galaxy S22 device powered by Qualcomm Snapdragon 8 Gen 1 Mobile Platform" with Qualcomm Hexagon TM processor for image resolutions of 224 X 224 and 384 x 384 using VIT-T/16. The inference is performed on Neural Processing Unit in 8-bit precision. As shown Table 2b, SKIPAT improves the runtime by 19% for image size of 224 x 224. The gain is even larger at 34% for image resolution 384 x 384, since the number of token increases. Thus, skipping computationally-hennt MSA blocks increases throughput by large margins and is confirmed even on mobile hardware.
Semantic Segmentation on ADE20K We show the performance of SKIPAT to dense prediction tasks such as semantic segmentation on ADE20K (Zhou et al., 2017). We follow (Liu et al., 2022; 2021) and use MMSegmennation (Contributors, 2020). We observe from Table 3, that SKIPAT out- performs all variants of ViT with 15% fewer FLOPs and 25% improved throughput. Interestingly, SKIPAT-S (ViT-S + SKIPAT) achieves 8% higher mloU while being faster than VIT-T. Furthermore, SKIPAT-S has comparable mloU with Swin-T (Liu et al., 2021) whilst having 3x fewer FLOPs
"Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
8

---

Published as a conference paper at ICLR 2024
METHOD
PSNRT
SSIMT
GFLOPS]
THROUGHPUTT
UNet (Rnneherger et 2015)
39.65
35
DAGL (Mou al, 2021)
255
Deam (Ren al., 2021)
39.47
0.957
145
MPRNet (Zamir al., 2021)
39.71
0.958
573
NBNet (Cheng
39.75
0,959
91
Restormer (Zamir et al. 2022)
40.02
0.960
140
Uformer-1
(Wang (oaneer-TTWWWggg
20226)
39,66
12
17.6
Uformer
Uformer-S S (Wang et al., 20226)
39.77
0.959
44
15.1
Ulormer S SKIPAT
39.84
0.960
39
18.9
Uformer-B B (Wang 20221)
39.89
0.960
89
9.2
Uformer B SKIPAT
39,94
0,960
77
10.9
Table 4: Image denoising on SIDD dataset using PSNR and SSIM (Wang et al., 2004) as the evalua- tion metrics the RGB space. FLOPs and throughput are calculated on the input size 256 x 256, on single NVIDIA V100 GPU, averaged over the test set of SIDD.
and being 1.7x faster. Comparing to fully convolution-bant architectures, SKIPAT-T (ViT-T + SKIPAT) on par with R8 in mloU while having 4.7x fewer FLOPs and being 1.8x faster.
Image Denoising SKIPAT can also generalize to low-level tasks such as image denoising on SIDD (Abdelhamed et al., 2018b), which consists of images with real-world noise. We ap- ply SKIPAT to Uformer (Wang et al., 2022b), SoTA image denoising model, which is U-shaped hierarchical network with Swin transformer blocks as the encoder and decoder. Detailed implemen- tation of SKIPAT on Uformer is the Appendix. Following the settings in (Wang et al., 2022b), observe in Table 4 that SKIPAT outperforms the baseline Uformer variants with the 25% higher throughput on average. Furthermore, we observe that SKIPAT-B (Uformer-B + SKIPAT) achieves comparable performance with Restormer (Zamir et al., 2022), in terms of PSNR and SSIM, while having 2x fewer FLOPs. Thus, we show the ability of SKIPAT to generalize to different tasks and also across architectures. Experiments on video denoising are provided in the Appendix.
4.2 ABLATIONS
All ablations are performed using ViT- T/16 on ImageNet- K for 100 epochs to reduce the training time. Unless spec- ified, following SKIPAT we skip the
FUNCTION 4
KERNEL
CHANNEL EXPANSION
TOP-1f
THROUGHPUTS (img/scc x10")
(%)
VITTT
65.8
5.8
MSA blocks from layer 3 through 8 for all ablations. Additional ablations are provided in the supplementary material.
IDENTITY CONY
61.1
8.5
5 x
65.4
5.2
DWC
5
65.6
7.8
Parametric function d. We study the effect of different parametric functions. As discussed in subsection 3.3, T can be
SKIPAT
5
2
67.7
6.9
7X
67.4
6.6
0.5
64.4
7.4
simple as identity function, where we directly reuse representations from a previous MSA block into one of more subsequent MSA blocks. From Table 5, using an identity function results in 4.7% drop in top-1 accuracy while be- ing 47% faster than baseline VIT. Using convolution DwC (Chollet, 2017) with kernel size 5 x 5 parametric function leads to the same performance as the baseline. However, DwC is 0.2% better and 50% faster than convolution, and 34% faster than the baseline. SKIPAT parametric function outperforms all.
7.2
2
67.7
6.9
Table 5: Ablations using VIT-T/16 on ImageNet-IK 1K for 100 epochs. We measure throughput (image/sec) with batch size of 1024 averaged over the validation set of ImageNet-1K:
single NVIDIA A100 GPU,
Kernel size. By default SKIPAT uses a DwC with kernel size x 5. As shown in Table 5, using x 3 kernel is faster than default SKIPAT by 6%, but is 0.6% worse in accuracy. A larger kernel size has poor accuracy and lower throughout. Irrespective the kemnel size, SKIPAT outperforms the baseline ViT-T by least 1,4%, showing its ability to encode cross-token interactions.

---

Published as a conference paper at ICLR 2024
Channel expansion. In the SKIPAT, the first linear layer FC1, expands the channel dimension from -> 2d. Table shows the impact of channel dimension, L.e., when the channel expansion ratio of FC₁ is 1.0 (d d) and 0.5 (d + d/2). We observe that while the lower channel expansion ratio improves the throughput, it performs worse than default SKIPAT. This could be due to sub-optimal representations encoded by the DwC due fewer filters.
CONCLUSION
We proposed SKIPAT, plug-in module that can be used in any ViT architecture to reduce self- attention computations. SKIPAT leverages the dependency across MSA blocks and bypasses atten- tion computation by re-using attention from previous MSA blocks. We introduced a simple and light parametric function that does not affect the inductive bias encoded in MSA. The SKIPAT func- tion captures cross token relations and outperforms the baseline while being computationally faster in terms of throughput and FLOPs. We plugged SKIPAT in different transformer architectures and showed its effectiveness on 7 different tasks.
REFERENCES
AbdeIrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smart- phone cameras. In CVPR, 2018a.
Ahdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. high-quality denoising dataset for smart- phone cameras. In CVPR, 2018b,
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. 16Xiv:1607.064550 2016.
Layer normalization.
arXiv preprint
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 1,
Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic quad-aua regularization algorithms for computed imaging. In ICIP, 1994. 1,2
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. NeurlPS, 2021. 3
Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, Shuaicheng Liu. Nhnet: Noise basis learning for image denoising with subspace projection. In CVPR, 2021. 9
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 5,
Xiangxiang Chu, Zhi Tian, Yuging Wang, Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NewriPS, 2021. 1,3
MMSegmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and bench- 2020. 8,
mark. https://ggthhbb com/ wwwprennnwnnnaaaannneeeeeeeeeeee
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based centered alignment. JMI.R, 2012. 4
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurlPS, 2021.
Mostafa Dehghani, Anurag Arna,, Lucas Beyer, Ashish Vaswani, and Tay. The efficiency misnomer. In ICLR, 2022. 6
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. image database. In CVPR, 2009.
Imagenet: large-scale hicrarchical
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2020. 1,2,5 5, 6, 3,4
10

---

Published as a conference paper at ICLR 2024
M. Everingham. L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PAS- CAL Visual Object Classes Challenge 2012 (VOC2012) Results.
http://www.pasee
7, 1,4
Mohsen Fayyaz, Soroush Abbasi Koohpayegani. Farnoush Rezaci, Sunando Sengupta, Hamid Reza Vaezi Joze, Hamed Pirsiavash, and Juergen Gall. Adaptive token sampling for efficient vision transformers. In ECCV, 2022. 3, 6
Ecuadorian fishing boat. CC BY-SA 4.0.
https:// ctopss:/ccommos.... www.s//wwa....geeeee..a......
Ecuadorian Ecuadorian_ fishing_ boat. jog. 7
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR, 2019.
Benjamin Graham, Alaacldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In ICCV, 2021. 2,
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NewrIPS, 2021. 3
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In ICLR, 2022. 5
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. ICCV, 2015.
Kaiming He Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian eror linear units (gelus). arXiv preprint arXiv: 2016.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enahling faster vision transformers via soft token pruning. In ECCV, 2022.
Simon Korbliihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019, 4
Kunchang Li, Yali Wang. Gao Peng, Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified
transformer for efficient spatiotemporal representation learning. In ICLR, 2022. 3, 1,2
Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. arXiv preprint 2XXi::2201.12288 2022. 1,3
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 3,
Zhuang Liu, Hanzi Mao, Chao-Yun Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 3,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 1
Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichtenhofer, and Jiten- dra Malik. Reversible vision transformers. In CVPR, 2022. 6,
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purposee and mobile-friendly vision
transformer. In ICLR, 2021. 3
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. preprint arXiv:2206.02666 2022.
arXiv
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surpris- ingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022. 8
11

---

Published as a conference paper at ICLR 2024
Chong Mou, Jian Zhang, and Zhuoyuan Wu. Dynamic attentive graph learning for image restoration. In ICCV, 2021. 9
Sheep near Snowshill. BC-B 4.0.
org. wttp.r.......ooo///////666666/6.
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight enns on mobile devices with vision transformers. In ECCV, 2022a. 3
Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchi- cal pooling. In ICCV, 2021. 3
Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. NeurlPS, 2022b. 3
Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI, 2022c. 2, 3,
Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alexander Sorkine- Hornnngg and Lue Van Gool. 2017 davis challenge on video object segmentation. arXiv:1704.00675. 2017.
Maithra Raghu, Thomas Unterthiner, Simon Korbllith, Chiyuan Zhang, and Alexey Dosovitskiy. vision transformers see like convolutional neural networks? In NeurlPS, 2022. 5
Yongming Rao. Wenliang Zhao, Benlin Liu. Jiwea Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NewrIPS, 2021.
Chao Ren, Xiaohai He, Chuncheng Wang, and Zhibo Zhao. Adaptive consistency prior based deep network for image denoising. In CVPR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Ychui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In CVPR, 2022. 1,3, 6
Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In CVPR, 2020. 1, 3,
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021. 6,
Gregory Vaksman, Michael Flad, and Peyman Milanfar. Patch craft: Video denoising by deep modeling and patch matching. In ICCV, 2021.
Kangra valley train. CC BY-SA 4.0. https: Valley_train, ipg.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 1
Jesse Vig. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714 2019. 4
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. arXiv preprint arXiv:1906.0428. 2019. 4
Qilong Wang. Banggu Wu. Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. CVPR, 2020.
Wenhai Wang, Enze Xie, Xiang Deng-Ping Fan, Kaitao Song, Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transtormer: A versatile backbone for dense prediction without convolutions. In CVPR, 2021a. 6,6
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022a. 3
Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. Evolving attention with residual convolutions. In ICMI. 2021b. 3
12

---

Published as a conference paper at ICLR 2024
Zhendong Wang. Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. CVPR, 2022b. 9,
Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from visibility to structural similarity, IEEE Transactions on Image Processing, 2004."
Ross
Wightman.
Pytorch image models.
https::///.ttub..om//o htt.:th....com..ooooooeeeetee
pytorch- image-modols, 2019.
Haiping Wu, Bin Xiao, Noel Codella. Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. ICCV, 2021." 3
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for understanding. In ECCV, 2018. 8, 1
Tong Xiao, Yingiao Li. Jingho Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast trans- former. IJCAI, 2019. 3
Saining Xie, Ross Girshick, Piotr Dollár. Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. CVPR, 2017. 2,5
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, 2021. 3
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens efficient vision transformer. CVPR, 2022. 3,6, 6
Chengxuan Ying, Guolin Ke, Di He, and Tie-Yan Liu. Lazyformer: Self attention with lazy update. arXiv preprint arXiv:2102. 12702, 2021.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. CVPR, 2022. 1,
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021. 3
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahhaz Khan, Ming- Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021. 9
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Huuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022. 9
Pengehuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, 2021. 3
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. through ade20k dataset. In CVPR, 2017. 8,
Scene parsing
Daquan Zhou, Bingyi Kang, Xiaojic Jin, Linjic Yang, Xianchen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 200iv /1886 2021a. 2,3
Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transformens. arXiv preprint arXiv:2106.0377 2021b. 2,
3
13

---

Published as a conference paper at ICLR 2024
6 IMPLEMENTATION DETAILS
6.1
ImageNet-IK: Image classification. We train SKIPAT the ILSVRC-2012 dataset (Deng et al., 2009) with 1000 classes (referred as ImageNet-IK). We follow the experimental settings of DelT (Touvron et al., 2021) and use the codebase from the timm library (Wightman, 2019) to train ViT-T, VIT-S and ViT-B. We use the default 16 x 16 patch size, using an image resolution of 224 x 224 with total number of tokens n 196. We train baseline ViT and SKIPAT for 300 epochs from scratch on 4 NVIDIA A100 GPUs using batch sizes of 2048 for ViT-T and 1024 for ViT-S and ViT-B.
ImageNet-1K: Self-supervised learning. We follow the experimental settings of DINO (Caron et al., 2021) and pre-train DINO and SKIPAT on ImageNet-IK using ViT-S/16 the backbone. While likely the hyperparameters could be tuned further for our proposed SKIPAT method, we same hyper-parameters for both the baseline and ours, yielding a conservative estimate of model's 's performance. We pre-train both methods from scratch for 100 epochs using 4 NVIDIA A100 GPUs. For linear-probing, we freeze the backbone from the pre-training stage and fine-tune the classifier for 100 epochs, exactly as done in (Caron et al., 2021).
Pascal-VOC2011 Unsupervised object segmentation. We the Pascal VOC 2012 (Evering- ham et al.) validation set for this experiment, containing 1449 images. We follow DINO and obtain unsupervised segmentation masks by thresholding the averaged self-attention map (extracted from the last layer of a pretrained ViT/SKIPAT model) to keep 80% of the mass. The Jaccard similarity J between a predicted mask, P. and ground-trute truth mask, G. defined as:
GnP
J(P,G) GUP
We report Jaccard similarity, averaged over all the samples.
Semantic segmentation. We evaluate SKIPAT ADE20K (Zhou al., 2017), widely-used semantic segmentation dataset, covering 150 semantic categories. The dataset includes 20K and 2K images in the training and validation set, respectively. Different variants of SKIPAT are evaluated using UperNet (Xiao et al., 2018) as the backbone. We use our ImageNet-IK pretrained model to initialize the backbone and Kaiming (He et al., 2015) initialization for other layers. We use AdamW (Loshchilov & Hutter, 2017), with an initial learning rate of 6e 5. weight decay of le 2, and linear warmup of 1500 iterations. All models are trained for 160K iterations with a batch size of 16 using MMSeemennation repo (Contributors, 2020). We keep the same hyper-parameters for SKIPAT and VIT.
ADE20K:
SIDD: Image denoising. We follow the experimental settings in Uformer (Wang et al., 2022b) and train SKIPAT on the Smartphone Image Denoising Dataset (SIDD) (Abdelhamed et al., 201 8a) which consists of real-world noise. The training samples are first randomly cropped to 128 128 patches and input to the model, which is trained for 250 epochs using batch size 32. The model is then evaluated on images of size 256 x 256.
DAVIS: Video denoising. We further apply our model to the temporal task of video denoising. We adopt the same U-shape encoder-decoder based architecture of UFormer. As the encoder and decoder backbone, we use UniFormer (Li et al., 2022). We train the model on noise level a = 30 using Charbonnier loss (Charbonnier et al., 1994) on patches of 7 x 128 x 128 using muitiple- input, multiple-output (MIMO) paradigm (Liang et al., 2022) (i.e., the model outputs reconstructed frames from 7 input frames). During inference, video is divided into 3D patches of x 128 128 with an overlap of 10 pixels. Each patch is fed to the model and the outputs are merged to obtain the final denoised video. Following (Tassano al., 2020), PSNR is calculated averaged over videos. as image denoising.
We use the same training
6.2 ARCHITECTURE
Image Classification. All baseline VIT variants have 12 layers in total, which remains unchanged with SKIPAT. Following the CKA analysis of MSA in Figure 3(b) of our main paper, we skip computing the MSA blocks in layer 3 through 8 for all VIT variants and retrain it from seratch.

---

Published as a conference paper at ICLR 2024
WSA
+
MLP
WSA
+
MLP
+
WSA
MLP
x
Input image
WSA
12
WSA
@ i++1
MLP
2,
2SSA
++
8
+
MLP
B
WSA
MLP
MLB
Denoised image
skip
skip
Figure 6: Framework of SKIPAT Uformer Instead of standard MSA block in VIT, Uformer uses window self-attention (WSA) block similar to Swin Transformer. We skip WSA block in the layers close to the bottleneck.
Image Denoising. We apply SKIPAT Uformer (Li et al., 2022) SoTA image denoising model. Uformer is U-shaped hierarchical network with Swin transformer blocks as the encoder and de- coder, and skip connections between them. In SKIPAT, we skip window self-attention (WSA) block in each decoder block by reusing attention of the corresponding encoder block via SKIPAT paramet- ric function. Let ZWws. IR denote the output of the WSA block at layer from the encoder and z R denote the output of the layer 1 - 1 from the decoder of Uformer. The input to the WSA block (which is skipped) at layer I the decoder given by
ZySa @< (Z Z
RX22
Here, "." denotes concatenation along the channel dimension. We show the framework of SKIPAT on Uformer in Figure 6
6.3 VIDEO DENOISING
We apply model the temporal task of video denoising. As encoder and decoder backbone, use UniFormer (I.i et al., 2022), a U-shaped hybrid encoder-decoder architecture with 3D convo- lutions and spatio-temporal global seff-attention blocks. The encoder of UniFormer comprises two 3D convolution layers followed by two spatio-temporal transformer layers with global self-attention (MSA) blocks. A downsampling operation is used after every layer in the encoder. The decoder is symmetric to the encoder with two transformer layers followed by two 3D convolution layers with an upsampling operation between each layer. Similar to Uformer, skip connections are used between encoder and decoder. Similar to image denoising, we skip MSA blocks in the decoder, however, sim- ply adopt naive SKIPAT, where we reuse window self-attention matrix, A, of the corresponding encoder block using Identity function. Let Af RR denote the self-attention matrix at layer / from the encoder. The self-attention in the decoder stage at layer I is given by A4 7 T(Af) E Raxn where (.) the identity function. We empirically observe that reusing attention works better in this task, and shows the ability of our method to be applied for different scenarios. We follow the experimental settings in (Tassano et al., 2020) and train SKIPAT on DAVIS (Pont-Tuset et al., 2017) dataset. We train using Charbonnier loss (Charbonnier al., 1994) patches of 128 128 using a multiple-input multiple-output (MIM)) paradigm (i.e. the model outputs 7 reconstructed frames from 7 input frames) for noise level 30. From Table 6, observe that SKIPAT performs on par
2

---

Published as a conference paper at ICLR 2024
METHOD
Dast (Tassuno et al., 2020)
PaCNet (Vaksman et al., 2021)
VRT (Liang al., 2022)
UniPormer et al., 2022)
UniFormer+ SKIPAT
PSNRT T
34.04
34,79
36.52
35.24
35.16
41.9
Table Video denoising Quantitative comparison (average RGB channel PSNR) with state-of-the- art methods for video denoising on DAVIS, with additive noise level 30. FLOPPs are calculated per frame per patch size of 256 256.
BACKBONE
METHOD
TOP-1 (%)
PARAM (x10%)
GFLOPS
TUROUGHPUT (img/sec x x10))
T2T-VT (Yuan al., 2021)
71.7
5.8
1.I
ComNNeXi (isa) (1 Lu et al., 2022)
72.7
5.7
5.8
VIT (Dosovitskly al., 2020)
72.8
5.7
1.2
5.8
(Yin al., 2022)
6.3
Dynamic VIT (Rao et al., 2021)
70.9
0.9
6.1
(Chen al., 2021)
71.7
SPVIT [Kong al., 2022)
72.7
5.7
0.9
6.7
ATS (Fayyar al., 2022)
72.7
5.7
0.9
6.1
PS-VIT (Tang al. 2022)
72.6
6.6
HVT (Pan et al., 2021)
70.2
5.7
0.7
7.2
SKIPAT
72.9
5.8
1.1
6.9
ComNNext T (Liu 2022)
82.1
29.0
4.5
2.6
ConNNeXt (iso) (Liu et al., 2022)
79.7
22.4
4.3
3,3
Swin-T (Liu 2021)
81.3
2.5
121-VT Yuam et 2021)
80.7
21.5
5.2
CoArNet-O (Dai al., 2021)
81.6
25
4.2
Poolformes-S24 (Ya et al., 2022)
80.3
21.0
3,4
Twinc-5VT.S (Chu al., 2021)
81.7
24.0
2.8
MobileViT-S (Mehta & Rastogari, 202 I)
78.4
5.6
2.0
(Wang
S/T
(Dusovitskiy al., 2020)
3,2
A-VIT (Yin et al., 2022)
78.6
22.4
3.6
3.4
Dynamic VIT (Rao et al., 2021)
78.3
23.1
3.4
3.6
SVITE (Chen al., 2021)
80.2
13.1
2.7
3.5
ATS (Fayyaz et al., 2022)
79.7
22.4
2.9
3,3
PS-VIT (Tang al., 2022)
3.9
SPVIT (Kong et 2022)
2022)
79.3
22.1
2.7
3.5
(Mangalam
2022)
79.8
HVTiPan et al., 2)
78.0
22.5
2,4
4.1
UniFormer 5 (Li el 2022)
82.9
3.6
1.8
EdgeViT-S (Pan al., 2022a)
81.0
SKIPAT
80.2
4.0
3.8
Swin-S (Lio 2021)
83.5
15.4
1.0
Twins-SVI-B (Chu et al. 2021)
83.2
56.U
8.6
(Wang
ConvNeXi (iso) (Liu
2022)
82.0
87.3
16.9
1.3
V/T-B/16
VT (Dusoviskkiy al., 2020)
81.8
87.3
17.6
1.2
SVITE (Chen et al., 2021)
81.6
52.0
11.5
1.3
Rev-ViT (Mangalam
2022)
81.5
87.3
17.6
1.2
PS-VIT (Tang et al., 2022)
81.5
9.8
1.6
SKIPAT
82.2
86.7
15.2
1.5
Table 7: Image classification on ImageNet-IK. Accuracy efficiency comparison of SKIPAT with SoTA methods for image resolution 224 x 224. For all the methods, we measure throughput (im- age/sec) with batch size of 1024 single NVIDIA A100 GPU, averaged over the validation set of ImageNet- -..
with baseline Uniformer, while having 17% fewer FLOPs. This shows that SKIPAT can generalize temporal tasks.
7 ADDITIONAL EXPERIMENTS
3

---

Published as a conference paper at ICLR 2024
Image classification. Here extend our SoTA comparison with methods that go beyond vanilla VIT architectures. These methods include hierarchical (Swin, PVT, Poolformer, Mobile Twins- SVT) and Hybrid (ConvNext, CoAT) architectures. We provide the complete set of SoTA methods that improve the efficiency of VIT either by token sampling (extending Table 1 in our main paper), using hybrid architectures window self-attention blocks in Table 7. Apart from methods that perform efficient token sampling, none of the other methods are directly comparable because they modify the underlying architecture of ViT, either by using window self-attention blocks or reducing the overall number of transformer layers.
Self-Supervised Learning with DINO Next, we show the generality of SKIPAT its use in the backbone for self-supervised representation learning (SSL), using DINO (Caron et al., 2021). Since, SSL methods are quite expensive in the pretraining stage in terms of compute and training time, we illustrate that SKIPAT achieves comparable performance to using a ViT but with shorter training time. Following the experimental settings of DINO (Caron et al., 2021), we use ViT-S/16 (Dosovit- skiy et al., 2020) as our student and teacher networks with SKIPAT parametric function. We pretrain both baseline and ours using DINO for 100 epochs. We observe that SKIPAT achieves almost the same performance as fully trained DINO with around 26% less training time (73.3% in 96 GPU- hours vs. 73.6% in 131 GPU-hours). When trained on 100 epochs, we observe that SKIPAT outper- forms DINO by 0.5% (74.1% vs. 73.6%).
Unsupervised segmentation of DINO. We follow DINO (Caron et al., 2021) and evaluate the performance of baseline DINO vs. SKIPAT on unsupervised object segmentation on Pascal- VOC2012 (Everingham et al.) dataset. We follow the experimental setting as discussed in section and observe that baseline DINO has a Jaccard similarity of 45.3 while SKIPAT achieves 44.7. While SKIPAT outperforms DINO on image classification by 0.5%, we achieve comparable performance in terms of unsupervised object segmentation.
ADDITIONAL ABLATIONS
Reusing self-attention. As mentioned in Subsection 3.3, we skip the 7MSA in SKIPAT as the compute and memory benefit from skipping the entire MSA block is greater than skipping just the self-attention operation. Here we study the effect of skipping just the self-attention operation. Let A:-1 denote the self-attention matrix at layer ( 1, then the self-attention matrix at layer / is given by Å, = T(A Similar SKIPAT we skip computing the self-attention matrix from layers through 8. As parametric function +, we use an identity mapping and train ViT-T/16 from scratch ImageNet- K. We observe from Table 9, that skipping the self-attention matrix results in a top-1 accuracy of 63. 93.2% which is 2.1% higher than the skipping ZMSA with an identity function (61.1% Table of main paper). However, skipping self-attention matrix results in 20% decrease in throughput (8500 + 6800 images/sec) as compared to using an identity function to skip MSA block. It is interesting to note that skipping self-attention matrix results in a lower drop in performance as compared to skipping MSA block. However, applying a parametric function to skip self-attention can be challenging due to the properties of the self-attention matrix, and we leave this to future work.
for 100 epochs
SKIPAT in pretrained model. As mentioned in subsection 6.2, we train SKIPAT with all variants of VIT from scratch. For completeness, we also study the effect of skipping the self-attention matrix and the MSA block on a pretrained VTT-T using an Identity function, without retraining. We observe from Table 9 that skipping the self-attention computation in layers 3 through 8, results in a top-1 accuracy of 53.9%, while skipping MSA blocks results in top-1 accuracy of 47.8% It is interesting to note that the drop in top-1 accuracy from skipping self-attention is merely 19% (72.8 + 53.9) on average and does not result in an extremely large drop as one might expect. This shows that there indeed exists high correlation across self-attentio and MSA which SKIPAT utilizes to improve the efficiency of the ViTs.
Skipping MSA in alternate configuration. tion in the layers 3 - 8, we study the effect of skipping MSA operation at I E {3, 5, 7, 9}. (3, 4, 5, 5, (3, 6, 9, 10} instead of default (3, 4, 5, 6, 7, 7,8} in Table 8. We ob- serve the default configuration outperforms the baseline ViT by 1.9% (65.8 vs. 67.7%) while being computationally faster in terms of throughput.
Instead of skipping the MSA opera-
4

---

Published as a conference paper at ICLR 2024
METHOD
LAYERS
TOP-1
THROUGHPUT
(imglsc x x10")
VIT-T
65.8
5.8
SKIPAT
[3,4,5,6]
67.4
6.3
[3,5,799
67.5
6.3
(3.4.5.6.7889.10)
64.2
SKIPAT (defnult)
(3,4,5,6.7.8)
Table 8: Ablations using ViT-T/16 ImageNet- IK for 100 epochs studying different layers that can be approximated.
METHOD
TRAINING
TOP-1 (%)
THROUGHPUT
A MSA
63.2
6800
61.1
8500
53.9
6800
MSA
X
47.8
8500
Table 9: Ablations on the effect of skipping the self-attention, A, and the MSA block, ZMS In the first rows, models trained for 100 epochs. In the last T/16 and simply skip computations in blocks 3-8 during inference. For all the experiments with use Identity function as 4
pretrained ViT-
CKA ANALYSIS OF ATTENTION FROM VIT-T
As discussed in Section 3.2 of our main paper, analyze the CKA of the self- attention matrix for all tokens between different layers of ViT-T/16 pretrained on ImageNet-IK. K. Since in the super- vised setting A E TR (==++))(===)) we first remove the CLS token to obtain AP E R We then compute the CKA of Af" for L. We visualiza- tion the attention maps for two random patches Figure We observe simi- lar correlation patterns as observed for CLS token. We akso observe from Fig- ure 7, that there exists a high correla- tion across all the tokens from the self- attention matrix. Thus, reusing self- attention from different layers of the VIT can improve the overall throughput while yielding comparable accuracy as the baseline ViT.
6:32
047
8:54
0:45
3:47
041
2
ANV
27T
54
10
027
34
CAL
Figure 7: CKA analysis of A for all tokens from pretrained vanilla VIT-T/16 on the validation set of I We observe a high correlation for all to- kens in A from layers 8.
5

---

Published as a conference paper at ICLR 2024
Iayer
layer
Iayer
layor
Tayer
layer
layer
layer
layer
layer
Tnyer
layer
AlMa
Figure 8: Mean of the attention heads from the CLS, n//2hh and n patch of a pretrained ViT-T/16 from the validation of ImageNet- K. n 196 number of patches.


Reviews:
[{"review_id": "NSxyaCwH4L", "reviewer": "Reviewer_VJcB", "summary": "A core component of the vision transformer is the self-attention layer, which is quadratic in the number of tokens. Following similar insights in (Raghu et al. 2022), the authors observe that self-attention operation is redundant at least in the intermediate layers, i.e. there is high correlation between:\n* (CLS -> token) Attention maps between  at layer $L$ and layer $L - 1$.\n* MSA representations between layer $L$ and layer $L - 1$.\n\nLeveraging this insight, the authors propose to replace the more computationally intensive attention operation with a lightweight refinement module termed SkipAt. More specifically, MSA at depth $L$ is replaced with inverted bottleneck layers (depthwise convolutions sandwiched between two dense layers). SkipAt layer $L$ takes the output of SkipAt layer $L - 1$ as input, as opposed to Multi-Head Self-Attention (MSA) at layer $L$ which takes the output of the MLP layer $L - 1$ as input.\n\nThe authors show experiments on classification, segmentation, unsupervised object discovery and image (+video) denoising.  They plugin their SkipAt technique and attain improved throughput and in most cases, improved accuracy.", "questions": "## Major Requests:\n-----------\n\n* The main motivation of SkipAt is that the output representations of MSA in vision transformers are redundant. So, the paper claims that just refining the outputs of the previous MSA layers is sufficient. However, SkipAt consists of depthwise convolutions which are also quite powerful modules themselves, so it is unclear if the throughput gains come just by the convolutions rather than the SkipAt formulation. I suggest that the authors run the couple of ablations below, If these ablations reach lower accuracy, it would be convincing that the SkipAt formulation is responsible for the accuracy gains.\n  * Replace all layers with SkipAt instead of just layers from 3 through 8. According to the authors hypothesis, since layers 9 though 12 have lesser correlation, using SkipAt at these layers should hurt accuracy.\n  * To show the importance of skipping the attention blocks, in Eq 7) the authors can replace $\\phi(Z_{l-1}^{MSA})$ with just $\\phi(Z_{l-i})$. This will give more evidence that skipping the attention block is necessary.\n\n* The authors test their module on Ti, B and S which all have 12 layers so they recommend to use SkipAt layers from 3 through 8. How do they recommend tuning these for larger depths?\n* In page 6, authors say that $n >> d$ and so $O(n^d)$ term dominates. I would suggest the authors add that this is specific to dense prediction tasks, since for image classification even for a S model (d=384, and n=196), so the claim that n >> d is not general.\n* Are the throughput increase in Fig a) significant? What do the error bars look like?\n\n## Minor Comments:\n-----------\nThese are just nice to have and are not likely to influence my final rating.\n\n* The authors use the efficient channel module. But, the ablation is missing from Table 5.\n* The figures from Raghu et al, indicate that there might be redundancy across the MLP layers as well. Does it make sense to have a Skip-MLP module?\n\n## Minor suggestions:\n------\n\nSome suggestions to improve presentations:\n\n* The authors can consider making the numbers in Figure 3 and Figure 5 bigger.\n* It is clear by comparing Fig 3 b) and Fig 5 b), that Fig 5 b) has lower correlation. The authors can also have a line plot where the x-axis is the layer id and y-axis is the average correlation of layers before it. If we plot the baseline ViT and the ViT with SkipAt on the same graph, it will make the comaprison even clearer. \n* In Fig 1), are the circles to indicate #params necessary since they are roughly the same? It gives the impression that the improvements are not significant even if they are, since the circles overlap", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The approach is simple and effective.\n* The authors have tested their SkipAt approach on a number of tasks. Their approach improves over similar Vision Transformer backbones and leads to improved throughputs.\n* The writing is crisp and clear.", "weaknesses": "Some experiments can be added which decouple the improvements obtained with convolutions vs the SkipAt formulation. I initially rate this  above bordeline. If the authors can convincingly answer my questions, I am happy to increase the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aYk8uXAV2x", "reviewer": "Reviewer_iYVP", "summary": "This paper improves the efficiency of Vision Transformer by replacing some attention layers with a compute-efficient parametric function, ie, convolutional feed-forward layer. The idea is motivated by a clear observation and analysis that attention patterns tend to be redundant between different layers, indicating a strong correlation. With the novel design, the authors validated the framework on various architecture and datasets. Comprehensive experiments have shown the advantage of their method.", "questions": "Can the authors specify more on the experimental setting of applying SKIPAT into hierarchical ViTs? I can understand that SKIPAT works for layer 2 to 8 in plain ViTs. But it is not intuitive to me how to select the layers to skip in PVT, LIT, etc.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is very clear, accompanied by strong analysis in the attention patterns. \n2. The figures and visualizations can clearly demonstrate their method. The overall presentation is good to me.\n3. Experiments are comprehensive, including different architectures, datasets, tasks, which strongly demonstrate that the proposed method is general.\n4. The performance gain is also consistent across different settings.", "weaknesses": "1. Based on the analysis in Section 3.2, it makes sense for the authors to apply their method from layer 2 to 8. However, it is not convincing for different pretrained ViTs to skip layer 2 to 8 as well if considering different training objectives or pretrained datasets. Thus, it would be better for the authors to study if other pretrained ViTs (MAE [A], DINOv2 [B], SAM [C]), have the same phenomenon.\n\n2. Introducing convolution into ViTs has shown to be effective in related works [D], which is intuitive to me to achieve performance gain for SKIPAT. In this paper, SKIPAT adopts convFFN as a parametric function to replace MSAs, which still needs to be trained from scratch in order to achieve efficiency gain. It would be promising if this parametric function can be used as a drop-in replacement for existing large ViTs.\n\n\n[A] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[B] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[C] Kirillov, Alexander, et al. \"Segment anything.\" ICCV (2023).\n\n[D] Wang, Wenhai, et al. \"Pvt v2: Improved baselines with pyramid vision transformer.\" Computational Visual Media 8.3 (2022): 415-424.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKpQku4Cq7", "reviewer": "Reviewer_f6zn", "summary": "This paper presents a new attention mechanism, named skip attention, aiming to reduce the computional cost of vision transformers. It is based on a simple observation that the attention maps of  adjacent  transformer blocks share similar patterns. Authors propose to reuse the attention maps of the current block in the next several ones by introducing a series lightweight operations, like linear transformations and efficient channel attention.", "questions": "I care more about the novelty of this paper as the originality of this paper has been mentioned in a previous work. If the authors make this clear, I would like to raise the ranking score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written. In the introduction section, the authors clearly explain the motivation of this paper, which is originally from the visualization of the attention maps of ViTs. The presentation is also clearly. It is easy for readers to follow the work.\n\n- The results are good. When applied different versions of ViTs, the proposed method receives clear improvement over the baselines.", "weaknesses": "- It seems that the motivation of this paper has been mentioned in Zhou et al. (Refiner: Refining self-attention for vision transformers). They observe that reusing the attention maps in the next transformer block does not brings performance drop. The authors should more clearly explain the differences between this paper and the work mentioned above.\n\n- The baselines used in this paper are not recently proposed. The results are already not state-of-the-art compared to recent works, like CMT (CVPR'2022). I would like to see how would the performance go when the proposed approach is applied to recent state-of-the-art ViT models as they mostly did not change the self-attention part.\n\n- Many ViT models are based on window self-attention, which is original proposed in Swin Transformer (ICCV'2021). The authors have shown that the proposed method works well for the original self-attention. So, how would the performance go when the proposed method is applied to ViTs with window self-attention.\n\n- In my view, one of the important functionalities of this paper is to compress vision transformers. Maybe the authors can show more comparisons with methods for compressing ViTs, like DynamicViT and EViT. As I found the proposed approach can improve the baselines' performance with even less computations. This may better highlight the strength of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)