RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
Authors: Xuwei Xu
Yang Li
Yudong Chen
Jiajun Liu
Sen Wang
Venue: ICML 2025 poster (2025)
Keywords: structural reparameterization
vision transformer
efficient
Primary area: deep_learning->attention_mechanisms

Abstract: We reveal that feedforward network (FFN) layers, rather than attention layers, are the primary contributors to Vision Transformer (ViT) inference latency, with their impact signifying as model size increases. This finding highlights a critical opportunity for optimizing the efficiency of large-scale ViTs by focusing on FFN layers. In this work, we propose a novel channel idle mechanism that facilitates post-training structural reparameterization for efficient FFN layers during testing. Specifically, a set of feature channels remains idle and bypasses the nonlinear activation function in each FFN layer, thereby forming a linear pathway that enables structural reparameterization during inference. This mechanism results in a family of **RePa**rameterizable **Vi**sion **T**ransformers (RePaViTs), which achieve remarkable latency reductions with acceptable sacrifices (sometimes gains) in accuracy across various ViTs. The benefits of our method scale consistently with model sizes, demonstrating greater speed improvements and progressively narrowing accuracy gaps or even higher accuracies on larger models. In particular, RePa-ViT-Large and RePa-ViT-Huge enjoy **66.8%** and **68.7%** speed-ups with **+1.7%** and **+1.1%** higher top-1 accuracies under the same training strategy, respectively. RePaViT is the first to employ structural reparameterization on FFN layers to expedite ViTs to our best knowledge, and we believe that it represents an auspicious direction for efficient ViTs. Source code is available at https://github.com/Ackesnal/RePaViT.

RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
Xuwei Xu
Yang Li
Yudong Chen
Jiajun Liu
Sen Wang
Abstract
N
We reveal that feedforward network (FFN) layers, rather than attention layers, the primary con- tributors to Vision Transformer (ViT) inference latency, with their impact signifying as model size increases. This finding highlights a critical opportunity for optimizing scale ViTs by focusing on FFN layers. In this work, propose novel channel idle mecha- nism that facilitates post-training structural repa- rameterization for efficient FFN layers during testing. Specifically, a set of feature channels remains idle and bypasses the nonlinear activa- tion function each FFN layer, thereby form- ing linear pathway that enables structural repa- rameterization during inference. This mecha- nism results in family of ReParameterizable Vision Transformers (RePaViTs), which achieve remarkable latency reductions with acceptable sacrifices (sometimes gains) accuracy across various ViTs. The effectiveness of our method scale consistently with model sizes, demonstrat- ing greater speed improvements and progressively narrowing accuracy gaps even higher accu- racies larger models. In particular, RePa- ViT-Large and RePa- ViT- enjoy 66.8% 5% and 68.7% speed-ups with +1.7% % and +1.1% % higher top-1 accuracies under the same training strat- egy, respectively. RiPaViT is the first to employ structural reparameterizztioon on FFN layers to ex- pedite VIT's best knowledge, and we believe that represents auspicious direction for effi- cient ViT's. Source code is available at https: / //github. cwm weem//ckksnaa/eeen PateesanaaRRPP........
N
RePa
Vanilla
Lineer
BatchNorm
efficiency of large-
Activation
Act.
Act.
RePa Linear 3
Lineer
Lineer
RePa Linear
LayerNorm
Attention
Attention
LayarNorm
LayerNorm
LayerNorm
Patch
Embed.
Embed.
Embed.
(a)
(b) RePaVIT Block
(c) Reparameterized
RePaViT Block
Figure I. RePaVIT architecture. (a) represents the vanilla VIT block. (b) illustrates our channel idle mechanism for FFN layers during training, where only subset of channels are activated while the rest bridge linear pathway. (c) shows the reparameterized RePaViT block during testing. where the number of parameters and computational complexity are significantly reduced.
1. Introduction
Vision Transformer (ViT) (Dosovitskiy et al., 2021) and its advanced variants (Touvron et al., 2021; Liu et al., 2021; Ryoo et al., 2021; Yu et al., 2022c; Liu et al., 2022: De- hghani al., 2023) have achieved outstanding performance in various computer vision tasks. However, the high compu- tational cost and memory demand of ViTs hinder their wide deployment in real-world scenarios, especially in computing assource-constraing environments.
School Electrical Engincering and Computer Science, The University of Queensland, Brisbane, Australia. *RCC Train- ing Centre for Information Resilience (CIRES), The University of Queensland, Brisbane, Australia. "DATA61, CSIRO, Pul- lenvale, Brisbane, Australia.. Correspondence to: Jiajun Liu <ryan.liu@ data6 .esiro..uu.. au>, Sen Wang wang.wan@@@nn uq. edu.au>.
To improve efficiency for ViTs, several techniques have been developed, such as token pruning (Rao et al., 2021: Liang et al., 2021: Kong et al., 2022a;b; Fayyaz al., 2022) and
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterizatioon Feedforward Network Layers
GSEEN 80.% Arc)
LN-VIT-M (1. BAAGHAAAAAAAAA
ReIT--N-VIT-M
Top-1 accuracy (%)
RoPa--N-VITSS A1 AIGGAAA MLATAA
82
RePa-Swin-BR Base
DeiT-Base
RePa-Deff-Base
LV-VI--S
Deil--Saall
80
ViT-aae DASONNACC
RePa-DeeTTSSmall the GAMACA 2..% Arc)
(14) (M7GMMA M.35
Model Size
20M
S0M
100M
100.0
150.0
200.0
300.0
$00.0
1000.0
1500.0
2000.0
Throughpur (Imagesisccond)
Figure 2. Performance comparison of RePaViT's and their vanilla backbones. RePaViT's (red circled) consistently achieve greater accelerations and smaller accuracy gaps when model sizes increase, showing the potential effectiveness in expediting large-scale ViTs. It also worth noting that RcPa-Ve LaPa-ViTearrge not only improves inference speed by more than 50% but also raises accuracy by 1.7%.
token merging (Bolya et al., 2023: Zong et al., 2022; Marin et al., 2023; Xu et al., 2024b; Kim et al., 2024) methods that gradually reduce the number of image tokens the layer goes deep; hybrid architectures (Mehta & Rastegari, 2022a; Chen et al., 2022a; Maaz et al., 2022; Li et al., 2022: Zhang et al., 2023) that embed efficient convolutional neural networks (CNNs) into ViTs; and network pruning (Yu et al., 2022b;a; Yu & Xiang. 2023; Zhang et al., 2024; He & Zhou, 2024) methods that remove less important parameters while preserving performance. Meanwhile, knowledge distillation methods (Touvron et al., 2021; Hao et al., 2022; Wu et al., 2022; Chen et al., 2022b) are introduced to further optimize efficient ViTs' performance.
ear activation function in between (i.e., Figure (a)). The two linear projections be potentially merged via struc- tural reparameterization to reduce complexity during testing. Notably, reducing FFN complexity is particularly critical for improving the efficiency of ViTs. Despite their straight- forward structure, FFN layers account for more than 60% of the total computational complexity in ViT models (Li et al., 2022; Mehta & Rastegari, 2022b). Furthermore, we observe that FFN layers contribute substantial portion of the total latency in ViTs, with this contribution scaling up as the model size grows, as shown in Figure 3. These observations reflect the urgent demand for techniques to optimize FFN layers, especially for large-scale ViTs.
Despite growing interest in efficient ViTs, existing ap- proaches often overlook structural reparameterization (Ding al., 2019; 2021b; b; Zhu et al., 2023), powerful network simplification technique widely used in CNNs. Structural reparameterizatioo enables networks to adopt different struc- tures during training and inference by merging convolutions or adjacent BatchNorm (loffe & Szegedy, 2015) and convolution via linear algebra operations. This process allows a complex architecture during training to be compressed into simpler structure for inference, thereby improving efficiency. Some recent research (Vasu et al., 2023a; Guo et al., 2024) has investigated structural reparam- eterization for ViTs by integrating elements from CNNs into VITs and subsequently reparameterizing only these CNN components. However, little attention has been given to directly applying structural reparameterization to the intrin- sic architecture of ViTs, particularly their fundamental building blocks.
To facilitate structural reparameterization for FFN layers, in this work, we propose an innovative channel idle mech- anism. Specifically, in each FFN layer, only small sub- set of feature channels undergo the activation function to provide necessary nonlinearity while the rest channels re- main idle. as shown in Figure 1(b). Consequently, these idle channels bridge a linear pathway through the activation function, enabling structural reparameterization during infer- ence. Moreover, inspired by Yao et al. (2021), we substitute the LayerNorm (Lei Ba al., 2016) with BatchNorm (loffe & Szegedy, 2015) and add another BatchNorm before the second linear projection. These BatchNorms can be repa- rameterized into their adjacent linear projection weights, which allows further reparameterization of the shorteut.
With the proposed channel idle mechanism, a family of Re PeParaaeterizable Vision Transformers (RePaViTs) are developed, whose FFN layers can be reparameterized to condensed structures during inference Figure 1(c) shows. Extensive experiments on various VIT's have validated the effecctivnness of our method, demonstrating its potential to enhance the applicablity of ViTs in resource- environments. Moreover, as Figure 2 illustrates, the ex-
Among these building blocks, feedforward network (FFN) layers represent a promising yet underexplored target for applying structural reparameterization. A typical FFN layer consists of two consecutive linear projections with a nonlin-
2

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
perimental results further indicate that our method delivers more significant acceleration and narrower performance disparity as the model complexity increases. In particular, RePaViT accelerates VireL and ViT-Huge models by -68% speed gain while even improving accuracy by 1-2% compared to their vanilla versions. This also demonstrates a transformative contribution, as many practical large-scale foundation models for computer vision tasks utilize ViTs as their backbones, such as CLIP (Radford et al., 2021; Cherti et al., 2023) and SAM (Kirillov et al., 2023). Moreover, our RePaViT achieves better trade-offs between speed improve- ment and accuracy compared to state-of-the-ar network pruning methods.
such as FFN layers in ViTs. Our work is the first to apply structural reparameterization FFN layers for ViTs.
2.2. Structural Reparameteriiation
Structural reparameterization is an effective network sim- plification technique that is typically employed in multi- branch CNNs (Ding et al., 2019; Guo et al., 2020; Ding et al., 2021a;b). It converts an over- over-paramenn parameterized network block into a compressed structure during testing, thereby reducing the model complexity and increasing the speed for the inference stage. For instance, after reparameterizing its mualti-raanch convolutions and shorteuts into a single branch, RepVGG-BO (Ding et al., 2021b) achieves 71% speed-up with no accuracy loss. Although some recent studies claim to adopt structural reparameterization for enhancing ViTs' efficiency (Vasu et al., 2023a; Wang et al., 2024; Tan et al., 2024), they primarily construct a hybrid architecture con- sisting of both convolutions and self-attentions and only perform reparameterization on the convolutional part. A recent state-of-the th-tt-- art method, SLAB (Guo et al., 2024), proposes to progressively substitute LayerNorms in ViTs with BatchNorms and reparameterize BatchNorms into lin- car projection weights. Unlike these methods, we are the first to apply structural reparameterization on FFN layers.
To our best knowledge, RePaViT is the first method that successfully applies structural reparameterization on FFN layers for efficient ViTs, and achieves significant acceler- ation while having positive gains in accuracy instead of accuracy drops on large and huge ViTs with the same train- ing strategies.
2. Related Work
2.1. Efficient Vision Transformer Methods
Vision Transformer (ViT) (Dosovitskiy et al., 2021) adapts the Transformer (Vaswani et al., 2017) architecture for com- puter vision, achieving success on various computer vision tasks. However, ViT suffers a substantial computational complexity. To alleviate the computational burden, several techniques that focus on structural design for efficient ViTs have been proposed. Spatial-wise token reduction methods are developed to identify less important tokens and subse- quently prune (Rao et al., 2021; Liang al., 2021; Kong et al., 2022a; Fayyaz et al., 2022; Xu et al., 2022; Meng et al., 2022; Tang et al., 2022; Xu et al., 2023) or merge (Bolya et al., 2023; Zong et al., 2022; Marin et al., 2023; Xu et al., 2024b; Kim et al., 2024) them during inference. As a result, the number of tokens participating in the self-attention com- putation is reduced. Meanwhile, hybrid architectures that combine self-attentions with computationally efficient con- volutions (Graham et al., 2021; Mehta & Rastegari, 2022a; Chen et al., 2022a; Li et al., 2022; Cai et al., 2023; Vasu et al., 2023a; Zhang et al., 2023; Shaker et al., 2023) are introduced to reduce the computationally expensive self- attention operations while introducing regional biases into ViTs. In addition to hybrid ViTs, MetaFormer (Yu et al., 2022c) figures out that ViTs benefit from their architectural design, which consists of one token mixer layer and one multi-layer perception layer, and the token mixer can be replaced by more efficient operations, such as average pool- ing (Yu et al., 2022c) or linear projection (Tolstikhin et al., 2021). However, these approaches overlook the structural reparameterization method, which can effectively compress network that contains consecutive linear transformations,
3. Method
3.1. Latency Analysis
To understand the significance of improving efficiency for FFN layers, we profile the latencies of major components in several representative VIT models in Figure 3, including DeiT (Touvron et al., 2021). Swin Transformer (Liu et al., 2021) and ViT (Dosovitskiy et al., 2021). Figure 3 illustrates that FFN layers constitute a substantial portion of the total processing time, which escalates quickly as the model size increases. For instance, in the DeiT-Small model, FFN lay- ers contribute to approximately 32.8% of the inference time, while in the DeiT-Base model, this proportion increases to 45.1 45.1%. Moreover, the percentage of FFN layers' latency in the large-scale ViT-Largee model rises to 53.8%, more than half of the total inference time.
This phenomenon arises because scaling up ViTs typically involves increasing the number of channels, whereas the number of tokens tends to remain constant. Mcanwhile, the computational complexity of an FFN layer, quantified as O is quadratic to the number of feature chan- nels. Consequently, as the model expands, the FFN layers become significantly more computationally expensive. In conclusion, optimizing FFN layers becomes considerably important for minimizing the overall computational costs for large ViTs.
3

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
RoPa-DeiT-Saall Saal-
Patch Embalding
MIISA
RePa-DeT--Bee Deit-Base
FFN
Repurameteried FFN
RePaSwin-Small
Swin
RePa- Basc
Swis- Bane
ViT-Large
Latenry (ms)
Figure 3. Latency analysis. Visualization the runtime latencies of patch embedding. MHSA and FFN layers. Notably, as the model size increases, the proportion of latency attributed to FFN layers also rises. Our method effectively reduces the latency of FFN layers and obtains increasingly better performance on larger models, demonstrating a scalable acceleration FFN layers.
3.2. Channel Idle Mechanism for FFN Layers
We further define the channel idle ratio as 0 = v which represents the percentage of feature channels keeping inactivated in the FFN layer. μ is set to by default in the following experiments unless otherwise noted, leading to the default @ 1- (e.g., 0 0.75 when p 4, indicating 75% channels are idling when the expansion ratio is 4).
As Figure ((a) illustrates, a typical FFN layer consists of two linear projections with a nonlinear activation function in between. Given an input the number of tokens and C denotes the number of feature channels, the FFN layer process can be formulated as
NCC where N represents
Y = FFN(LN(X)) + X Act(LN(X) w!1) W + X, (1)
wou
3.3. Structural Reparameterization for FFN layers
where nhn E Cx w RBC are the linear projec- tion weights, LN(-) is LayerNorm (Lei Ba et al., 2016) and Act(.) is usually the GELU (Hendrycks & Gimpel, 2016) activation function. p is the FFN expansion ratio, which is usually set to 4. The biases are omitted for simplicity since they are inherently linear and do not interfere with the repa- rameterization process. Unfortunately, due to the nonlinear activation function, the structural reparameterization cannot directly merge the two linear projection weights W and WaOt via linear algebra operations.
With the channel idle mechanism defined in Equation 2, we are able to simplify the FFN layer by structural reparameter- ization during the testing stage. Firstly, we reparameterize the BatchNorms into their corresponding linear projection weights as
W
W
ox + ex 7/pcls
(3)
Wout
OOt
Of + Exaa
Inspired by ShuffleNetv2 (Ma et al., 2018) which keeps a group of channels idle in grouped convolutions and shuffles channels for information exchange, we propose simple yet effective channel idle mechanism to enable reparameteriza- tion in FFN layers. Specifically, this mechanism maintains large subset of feature channels inactivated in an FFN layer, thereby bridging a linear pathway through the nonlinear acti- vation function in the corresponding FFN layer. In addition, we substitute LayerNorm with BatchNorm (BN) (loffe & Szegedy, 2015) to enable post-training reparameterization of normalization and shorteut for the FFN layer. As a result, our channel idle mechanism during the training stage can be formulated as
where ys, a²s and es are the empirical means, empirical variances and constants from the frozen BatchNorm layers, respectively. With the reparameterized projection weights W and WOw, the output Y in Equation can be reformu- lated as
Y =Attt( 1
Out Pisc :l
(4)
+ XW b. -C-1:
W I+C 1.p. :) 1 X.
Then, we further reparameterize the weights as
W Wht #C+1:pC] W |pC+1pC :] + I.
Out
(5)
By substituting Equation 5 into Equation 4, we obtain the updating function for the FFN layer during the testing stage with three reparameterized weights as
x = BN((X) Wt". X Co 1sac)), X pC+1:p))ss
(6) As Figure 1(c) shows, after reparameterization, the two massive linear projections converted into three smaller linear transformations with fewer parameters and all the normalizations are merged into linear projection weights.
Z Act( YW 1wwc)) ( Minc YYW.
== BN(( + XX,
where the activation function is only applied on μC (u p) feature channels. The (p- C) idling feature channels construct a linear route as presented in Figure 1(b).
4

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
3.4. Computational Complexity Analysis
summing them:
Number of parameters: The vanilla FFN layer's param- eters are mainly derived from the two linear projection weights W E RCxC and W E R totalling 2pC². In contrast, with our channel idle mechanism, the weights are reparameterized into three terms; an input weight Wi. loucy E RC an output weight W (D:CC, : RRCxC and reparameterized weight W E RCxe The to- tal number of parameters is effectively reduced from 2pC² to (2 + 1)
Cony Rep
WCon +
C
On the contrary, demonstrated in Equation 5, RePaViT reparameterizes two consecutive projection weights W and W by multiplying them:
WATN Rep
WFEN WEEN
(8)
In the above example, W and W have been padded to the same shape, and the reparameterization processes of BatchNorm and biases are omitted for simplicity.
Consequently, in the reparameterized FFN layer, the parame- ter count is diminished to 1-0+ of the original parameter count, where 0 is the aforementioned idle ratio. For instance, when p = 4 and f = 0.75, the number of parameters in an FFN layer declines to 37.5% post-parameterization. This reduction significantly simplifies the model, diminishing its memory consumption.
It is also worth noting that our channel idle mechanism can- not be regarded as special case of dual- bual-branch structure in RGpVGG. In RepVGG, all branches must be linear so that they can be reparameterized, whereas in our approach, one branch is linear while the other one nonlinear.
Computational complexity: The computational complex- ity of the vanilla FFN layer is C²) while the com- putational complexity is significantly reduced to ((2 + C)) in our reparameterized FFN layer. The computa- tional complexity reduction ratio for an FFN layer is also 1- 0+
4. Experiments
4.1. Datasets, Training and Evaluation Settings
We mainly train and test RePaViT' for the image classifica- tion task on the widely recognized ImageNet- Ik (Deng et al., 2009) dataset, following the data augmentations and training recipes proposed by Touvron et al. (2021) as the standard practice. In line with Yao et al. (2021), the maximum learn- ing rate set to 4 x 10-3 with 20 epochs of warmup from 10-6, The default batch size and total training epochs are 4096 and 300, respectively. For dense prediction tasks, we follow the configurations from MMDetection (Chen et al., 2019) and MMSegmentation (Contributors, 2020) to finetune RePa ViTs on MSCOCO (Lin et al., 2014) and ADE20K (Zhou et al., 2017) datasets for object detection and segmentation tasks, respectively. All the models are trained from scratch on NVIDIA H100 GPUs. To ensure fair comparisons, we measure the throughput of all the models on the same NVIDIA A6000 GPU with the same environ- ments and a fixed batch size of 128. FlashAttention (Dao et al., 2022) is used for self-attention computation during inference measurement by default. More implementation details on the training settings are provided in Appendix A.
It is worth noting that, due to the elimination of normaliza- tions and shortcuts in the FFN layer, the inference speed gain is more than the computational complexity reduction.
3.5. Comparison against VGG-stte Reparameterization
RepVGG (Ding et al., 2021b) introduces structural repa- rameterization into CNNs, where multi-branch convolutions are merged into a single-branch convolution through linear operations convolution kernels. While RePaViT draws inspiration from RepVGG, there are significant differences between our structural reparameterization approach and the Rep VGG-style reparameterization:
Different targets:
Existing works using RepVGG-style reparameterization for efficient ViTs (Vasu et al., 2023a;b) introduce CNN components into ViTs and only reparam- eterize those convolutional components. In contrast, our method directly targets existing FFN layers in ViTs, aim- ing to improve the efficiency of standard ViT architectures rather than designing an entirely new backbone. Thus, the application objectives are fundamentally distinct.
4.2. Classification Results
Backbones: We choose four ViT backbones, including a representative plain-structured VIT (DeiT (Touvron et al., 2021)), a representative hierarchical-strruuuurdd ViT (Swin Transformer (Liu et al., 2021)), plain ViT trained with token labelling (LV-VIT (Jiang et al., 2021)), and large-scale VIT (Dosovitskiy et al., 2021). The FFN layers in these models are embedded with the channel idle mechanism and are all trained from scratch solely on the ImageNet- Ik dataset by supervised learning.
Different reparameterization solutions: Another differ- ence is that RepVGG reparameterizes horizontally across parallel convolutional kernels, while RePaViT reparame- terizes vertically on consecutive linear projection weights. Mathematically, VGG reparameterizes two parallel convolutional branches with kernels W.Conv and CCnn by
5

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
Table 1. Performance comparisons among RePaViTTs and their vanilla backbones. For the "RePa" column, x and v stands for the RePaViT model pre- and post-rparameetcrization respectively. The decimals after model names (i.e. 0.50 and 0.75) represent the channel idle ratios (0). When the backbonn architecture fixes, our method consistently achieves greater accelerations and complexity reductions while narrowing the accuracy gap as the model size grows.
Table 2. Comparison with state-of-the-art nefwork pruning methods for efficient VTTs. --" indicates that the statistic is either missing or irreproducible. Our method demonstrates significantly higher speed-ups compared to pruning methods while achieving competitive or even higher top-1 1 accuracics across various ViT backbones.
Compl.
Spood
Top-1 acc.
Backbonc
Method
#MParam. 1 (GMACs)
Model
RePa #MParam
Complexity (GMACs)
Speed (imagew/ecoomd)
Top-1 accuracy
improv.
WDPruning
13.3
2.6
+18.3%
78.4%
X-pruner
24
78.9%
RePo-Der-TanyD..
Deil-Tiay
5.7
5.7
1.1
1.1
3435.1
2397.9 9
69.4% (-2.7%
72.1%
DeiT-Small
DC-VT
LPVIT
22.1
16.6
2.3
3.2
+16.3%
+20.0%
80.7%
78.6%
/
4.4 (-22.8%)
0.8 (-27.3%)
4001.2 (+-16.8%
RePaNiTVO050
3.2
+23.00%
78.9%
DeiT-Small
22.1
4.3
1410.3
79.8%
RePaViT/0075
13.2
2.5
+42.1%
77.0%
SSaal-eiT-Small0.5
x
22,1
4,3
1000.9
78.9% (-0.9%)
WDPruning
55.3
9.9
+18.2%
80.8%
16.7 ( 24.4%)
3.2
25.6%)
1734.7 23.00%
X-pruner
8.5
81.0%
DeiT-Base
86.6
16.9
418.5
81.8%
DeiT-Base
DC-VIT
12.7
+18.45%
81.3%%
86.6
16.9
336.6
LPVIT
86.6
8.8
+18.8%
80.8%
$1.1 (-11.0%)
9.9 (-41.4%)
660.3
(+-57.8%)
81.3% (-0.5%)
RePaNiTVO 50
65.3
12.7
+28.655
81.4%
RePaViI10075
10.6
+57,8%
81.3%
ViT-Large
304.3
304,5
59.8
59.7
124.2
102.7
80.3%
WDPruning
32.8
6.3
+15.3%
81, 31.8%
178.4 (-11.45) 34.9
(-11.5%)
207.2
(+66.8%)
82.0% (+1.7%)
Swin-Small
RePaViTVO.SO 50
X-prnner
37.8
6.0
64
+20.7%
82.8%
82.0%
Vug-Huge
632.2
124.3
61,5
80.3%
RePaViTT0.75
5.1
+33.1%
81.4%
RePa-ViT 007
632.5
124.4
53.0
81.4% (+1.19)
DC-VIT
66.4
11.5
+14.9%
83.8%
369.9 (- 41.3% 72.6 ( (1.00%)
103.8
(1-08.7%)
Swin-Base
I IPPTT
87.8
11.2
+8.9%
81.7%
Swin-Tiny
28.3
4.4
804,4
81.2%
RePaVIT/O.5 50
66.8
11.5
+19.6%
83.4%
28.3
4.4
614.9
RePaVi770.75
9.0
+42.4%
82.6%
RoPa-Swin-Tny00.7
78.4% (-2.8%)
17.5
2.6 (-40.9%)
1020.4 (+26.9%%
Swin-Small
49.6
8.6
471.7
83.0%
RePa-Swin-Small0..5
x
49.7
8.6
363.
81.4% (-1.6%
Table 3. Comparison against the state-of-the-art repa-
29.9 ( 39. (9.7
5.1
40.7%)
627.8 (1 (133.11)
rameterization method for ViTs. With similar num- her of parameters, RePaViT obtains both faster inference speeds and higher accuracies than SLAB (Guo et al., 2024).
Swin- Base
87,8
15.2
326.6
83.5%
RePa-Swin- SePP-SWin-Bnss00..5 Base/0.75
87.9
15.2
249.4
82.6% (-0.9%)
$2.8 (-30.9%)
9.0
(0.8%)
(+-43.2%)
Compl
Speed (umg/s)
Top-1 acc.
IVVISS
26.2
6.1
86666
81.4%
Model
#MParam. J (GMACs)
26.2
6.1
725.4
RePo-LV-VIT-S0075
81.6% (+0.29%)
19.1 (-27 19%)
4.7 (-23.0%)
(+28.2%)
SLAB DeiT--BTaee
86.6
17.1
387.0
78.9%
LV-VT-M
55.8
11.9
457.6
83.6%
79.5
15.5
452.3
81.1%
RrP--LN-VT--/0.75
55.9
11.9
396.6
83,5% (-0.1%)
SLAB-S L-B--win-Base
87.7
15.4
299.9
83.6%
40.1 (- 176
8.8 16.1%)
(1-40 40.0%)
RePa-Swin Base/.5s
80.8
14.0
356.3
83.7%
Reparameterization results: Table 1 presents the image classification performance of RePaViT's before and after reparameterization, and compares with their vanilla back- bones. Due to the nature of linear algebra operations, the pre- and postrreppramettrrizaioon accuracies are the same.
preserves accuracy to a remarkable extent. In particular, on the LV- model, we facilitate a 40.0% increase in the inference speed with negligible 0.1 1% decrease in accuracy.
Notably, RePaVIT yields -68% speed-up and even 1-2% higher accuracy on ViT-Larre and ViT-Huge models, indicating its potential on large-scale foundation models. This insight demonstrates the practical value of RePa ViT in accelerating large-scale models without compromising performance, making it an effective solution for large-scale real-world applications requiring both speed and precision.
In general, our innovative channel idle mechanism remark- ably enhances these models' computational efficiency and throughput while preserving their accuracy. We observe that with the same backbone architecture, RePaVIT achieves more substantial acceleration with a narrowing accuracy gap when the model size increases. For example, employ- ing DeiT as the backbone, the smaller DeiT-Tiny model witnesses a 16.5% speed-up at the cost of a 2.7% accuracy loss. However, when scaled up to the DeiT-Base model, our approach delivers a 57.8% throughput improvement, with only marginal 0.5% drop in accuracy. This pattern is consistent across various models. In cases where the backbones include additional regularizations during train- ing, our method not only accelerates performance but also
4.3. Comparison Against Network Pruning
While several network pruning methods for efficient ViTs fo- cus on reducing the number of parameters and the theoretical computational complexity during inference, our approach differs fundamentally from these methods. We provide a comparison with of and representative net- work pruning techniques in Table 2, including WDPruning (Yu al., 2022a), X-Pruner (Yu & Xiang, 2023), DC-ViT
6

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
Table Sensitivity of channel idle ratio 0. The performance of RePaVIT on plain (DeiT (Touvron et al., 2021)) and hierarchical (Swin (Liu et al., 2021)) ViTs with various is reported. 0= represents vanilla backbone. 0=1.00 implies the nonlinear activation being re- moved from the model. The results show a significant accuracy drop when 8 surpasses 0.75.
Table 5. Ablation study on train-time reparameterization. "Training RePa" stands for reparameterizing the model before training. v for "BatchNorm RePa" represents that the Batch- Norm before a linear projection reparameterized into pro- jection weight. "-" under top-1 accuracy means training failure. Overall, training with parameters and reparameterizing
Idle ratio
Compl. (GMACs)
Speed (img/s)
Top-1 acc.
ing testing yields better performance.
Backbone
#MParam.
Training BatchNorm
Training #MParam.
Top-1 accuracy
Model
RePa
RePa
1.00
2.6
0.5
5810.1
48.6%
0.75
3.5
0.6
4470.8
64.2%
3.5
59.6%
Deiy-Tiny
0.50
4.4
0.8
4001.2
69.4%
RePa-DeiT-Tiny1O.1
64.3%
0.25
5.3
1.0
3575.6
71.9%
X
64.2%
5.7
1.1
3435.1
72.1% %
V
13.2
75.0%
1.00
9.6
1.8
2612.9
63.9%
GePa-DeiT-Small0..7
V
X
13.2
75.7%
0.75
13.2
2.5
2003.7
77.0%
X
22.1
77.0%
DeiT-Small
0.50
16.7
3.2
1734.7
78.9%
0.25
20.3
1489.7
80.3%
V
V
51.1
1410.3
79.8%
RePa-DeiT-Bascoe
86.6
81.3%
1.00
37.0
7.1
878.7
73.7%
0.75
51.1
9.9
660.3
81.3%
178.4
DeiT-Batt Base
0.50
65.3
12.7
538.0
81.4%
RePa- GGPa-ViT-1.arge/0.77
178.5
80.6%
0.25
79,5
15.5
452.3
81.1%
304.5
82.0%
86.6
16.9
418.5
81.8%
V
V
17.5
77.1%
1.00
13.2
1.9
1180.1
67.6%
RePa-Swin-Tings 7
17.5
78.0%
0.75
17.5
2.6
1020.4
78.4%
28.3
78.4 %
Swin-Tiny
0.50
21.8
3.3
905.9
80.5%
Y
29.9
79.3%
0.25
26.1
4.0
844.8
81.4%
RdPa-Swin-Small00.75
30.0
79.1%
28.3
4.4
804.4
81.2%
81.4%
1.00
22.1
3,7
745.0
72.5%
V
52.8
79.6%
0.75
29.9
5.1
627.8
81.4%
RePa-Swin-Base100
52.9
80.3%
Swin-Small
0.50
37.8
6.5
569.2
82.8%
87.9
82.6 %
0.25
45.7
7.9
514.5
83.1 %
49.6
8.6
471.7
83.0%
19.1
RePa-LV-VIT-S00.7
81.3%
1.00
38.8
6.5
539.0
75.5%
0.75
52.8
9.0
467.6
82.6%
X
X
26.2
81.6%
Swin-Base
0.50
66.8
11.5
390.6
83.4%
V
40.1
0.25
80.8
14.0
356.3
83.7%
RePa-LV-VIT-M00.5
40.2
87.8
15.2
326.6
83.5%
55.9
83.6%
(Zhang et al., 2024), and LPVIT (Xu et al., 2024a). Due to unavailable or incomplete code repositories of certain state- rely the performance
cialized hardware and software support for sparse matrix operations. So our method can bring more speed-upt ups in general computing environments.
of-the-art pruning methods, statistics reported in the original papers and align efficiency optimization using speed improvements for fairness.
Scaling effectiveness on larger models: Compared with network pruning methods, RePa Vit yields more accelera- tions and smaller performance gaps larger models even with the same channel idle ratio 0. This underscores the important practical value of RePaViT on large foundation models for vision tasks.
Table 2 shows that the structural reparameterization ap- proach of RePaViT achieves significantly greater inference acceleration compared to network pruning methods. More- over, the effectiveness of our method increases model size grows. For example, while the state-of-the-art DC-VIT achieves speed improvements of approximately 15-20% across all backbones, RePaViT provides 19.6% to 57.8% speed improvements when the model scales up. These re- sults highlight two key advantages of our method:
4.4. Comparison Against State-of-The-Aa Method
Table 3 compares our RePaVIT approach against SLAB (Guo et al., 2024), recent state-of-the-art method introduc- ing progressive reparameterized BatchNorms for ViTs. For fair comparisons with similar model sizes, the performance of RePaViTs with 6=0.25 is used. The results indicate that our reparameterization strategy offers a better trade-off be-
Computing environment friendly: Our reparameterized model is dense and structurally regular, making efficient general- purpose hardware without requiring spe-

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
Table Performance on dense prediction tasks. Results on the 1 x training schedule are presented. The latencies (ms) per image are reported for throughput comparisons.
NetinaNet
Mask R-CNN
UperNet
Model
Latency (ms)
Latency (ms)
Latency (ms)
AP1
APwtt APrst
AP&t
AP A
APt
APsa't APtst
APst APart APr.1
+ mloUt
Swin-Small
61.7
37.2
56.9
39.6
22,4
40.5
49.4
62.5
45.5
67.8
49.9
28.6
49.2
60.4
36.3
47.6
RePu-Swin-SSall
53.8 (-12 815)
38.3
57.9
40.7
21.8
42.0
51.6
53.8 (-10.95)
43.6
65.8
47.8
27.1
47.0
57.3
32.1
(-11.00)
45.7
Swin-Base
82.0
38.9
59.5
41.3
24.3
43.6
54.4
82.6
45.8
67.6
50.3
28.7
48.9
61.7
45.6
48.1
RePa-Swin-Baan
66.7: 750
39,8
60.0
42.1
25.3
43.7
53.8
69.4
10.0%)
44.8
67.0
49.4
29.0
48.5
58.4
38.6
16.45%
46.9
tween efficiency and accuracy. For example, when utilizing DeiT-Base Base as the backbone, our method not only achieves a higher speed and fewer parameters but also surpasses SLABB by a 2.2% higher accuracy.
achieves better performance than training with less parame- ters for ViTs, which aligns with the findings in Vasu et al. (2023a;b). Meanwhile, train-time overparameterization also helps to stabilize the training process for large models. For instance, when trained with reparameterized structure, RePa- DeiT-Basee RePa-ViT-Large, RePa-LV-ViT-S and RePa-LV- ViT-M all suffer training collapse and fail to converge.
4.5. Sensitivty of Channel Idle Ratio
In Section 3.2, we define the channel idle ratio 0 as the percentage of feature channels keeping idle in the activation. Table illustrates the influence of 0 on the performance of ViTs. Overall, a larger 9 represents more channels idling in the FFN layer, leading to smaller number of parameters, lower computational complexity, and higher inference speed pestaarepramtttiiizaaio..
4.7. Dense Predictions
Table 6 presents the results of two downstream tasks. Firstly, the ImageNet- 1k pre-trained RePa-Swin models are inte- grated with one-stage detector RetinaNet (Lin et al., 2017) and two-stage detector Mask R-CNN (He et al., 2017) for the object detection task on the MSCOCO dataset with x training schedule (i.e., 12 epochs). Remarkably, our RePa-Swin-Bass model achieves up to 18.7% latency re- duction at even a higher average precision (AP) with Reti- naNet when compared to its vanilla backbone. RePA-Swin- Base also obtains a similar performance with 16.0% less latency with Mask R-CNN. Secondly, UperNet (Xiao et al., 2018) is leveraged for the semantic segmentation task on the ADE20K dataset with RePa-Swin models as backbones. Similarly, RePa-Swin-Base achieves 15.4% latency reduc- tion with merely 1.2% mIoU loss.
Remarkably, when 0 exceeds 0.75, which is the default idle ratio for ViTs, there is an obvious decline in the top- I accuracies. For instance, when setting to 1.0 (i.e., no channels being activated), the RePa-DeiT-Base accuracy drops from 81.8% to 73.7%. Similarly, the RePa-Swin- Base model witnesses its accuracy decline from 83.5% to 75.5% with 0 = 1.0. For smaller models, such performance collapse can be more severe. This outcome demonstrates that while reducing the proportion of nonlinear components can significantly enhance the model's efficiency, preserving sufficient nonlincarities also crucial for performance.
Overall, the experimental results downstream tasks re- flect a consistent trend that the performance disparities narrowing and the acceleration gains are escalating as the backbone model sizes grow. This aligns with the observa- tions in Section 4.2 well, which further proves the scalable acceleration capability of our channel idle mechanism.
It is noteworthy that, with proper 8, ViTs can achieve even better performance with fewer parameters and faster inference speeds. For example, DeiT-Small, Swin-Tiny, Swin-Small and Swin-Base models all enjoy higher top-1 accuracy when 0=0.25.
4.6. Ablation Study
4.8. Self-supervised Learning Experiments and Others
We ablate the structural reparameterization process during training. Instead of training the full 2pC² linear project weights and then reparameterizing them during testing, we directly train the reparameterized weights with a reduced size of (24 + 1)C². Specifically, in our experiments, the numbers of parameters for single FFN layer before and after reparameterization are 8 (i.e., م=4) and 3C2 (i.e., μ=l), respectively. Table indicates that training with more parameters (i.e., train-tine time overparameterization)) generally
Given that large foundation models are typically trained us- ing self-supervised learning strategies, we evaluate RePa VIT under self-supervised training (i.e., DINO (Caron et al., 2021)) and language-guider contrastive learning (i.e., CLIP (Radford et al., 2021)). The experimental results are pro- vided in Appendix B. Notably, when applied to CLIP mod- els, RePaViT improves zero-shot top-1 accuracy by 0.8% while achieving a 24.7% speed improvement, demonstrating its effectiveness in optimizing large foundation models.
8

---

RePaVIT:
Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
5. Conclusion
Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
In this paper, we investigate the latency compositions of ViTs and observe that FFN layers significantly contribute to the overall latency. The observations highlight the critical need for accelerating FFN layers to enhance the efficiency of ViTs, where structural reparameterization emerges potential solution. We introduce a novel channel idle mech- anism to facilitate the reparamcterization of FFN layers during inference. The proposed mechanism is employed on various ViT backbones, resulting a family of RePaViT's RePaViTs demonstrate consistent scalability with more ac- celerations and narrower aceuracy disparities as the back- bone model size escalates. Notably, RePaVIT achieves curacy gains while improving the inference speed on large- scale ViT backbones. These unprecedented results mark a disruptive and timely contribution to the community and establish RePaViT as significant addition to the toolkit for accelerating large foundation models. We believe that RePaViT presents promising direction for expediting ViTs and we invite the community to further explore its effective- ness on even larger foundation models.
Chen, K., Wang. J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang. J., Shi, J.. Ouyang, W., Loy, C. C., and Lin, D. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 10Xiv:1906.07155, 2019.
Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., and Liu, Z. Mobile-former: Bridging mobilenet and transformer. In CVPR, 2022a.
Chen, Y., Wang, S., Liu, J., Xu, X., de Hoog, F., and Huang, Z. Improved feature distillation via projector ensemble. NeurIPS, 2022b.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
Contributors, M. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https:// g tithub..com com/ cotomm/ppemmmlll.m hetmmmaab/mmnn www.aaaaeeeeeeatttiiomomoo 2020.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficiett exact attention with io-awareness. In NeurIPS, 2022.
Acknowledgement
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heck, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers 22 billion parameters. In ICML, 2023.
This research was partially supported by the Australian Government through the Australian Research Council's In- dustrial Transformation Training Centre for Information Resilience (CIRES) project number IC200100022, CSIRO's Research Plus Science Leader Project R-91559, and Aus- tralian Research Council Discovery Projects DP230101753 and DECRA DE200101610.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, I. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Ding, X., Guo, Y., Ding, G., and Han, Acnet: Strengthen- ing the kernel skeletons for powerful cnn via asymmetric convolution blocks. In ICCV, 2019.
References
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C.,, and Hoffman, J. Token merging: Your vit but faster. In ICLR, 2023.
Ding, X., Zhang, X., Han, J., and Ding, G. Diverse branch block: Building convolution as an inception-like unit In CVPR, 2021a.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NewrlPS, 2020.
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. Repvgg: Making vgg-style convnets great again. In CVPR, 2021b.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, In ICLR, 2021.
Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Multi-scale linear attention for high-resolution dense pre- diction. In ICCV, 2023.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J.,

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
Fayyaz, M., Koohpayegani, S. A., Jafari, R., Sengupta, S., Joze, H. R. V., Sommerlade, E.,, Pirsiavash, H., and Gall, Adaptive token sampling for efficient vision transformers. In ECCV, 2022.
Lei Ba, J., Kiros, R., and Hinton, G. E. Layer normaliza- tion. arXiv preprint arXiv: (od0::607004550 2016.
Li, Y. Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., and Ren, J. Efficientformer: Vision trans- formers at mobilenet speed. In NeurlPS, 2022.
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jégou, H., and Douze, M. Levit: a vision transformer in convnet's clothing for faster inference. ICCV, 2021.
Liang, Y., Chongjian, G., Tong, Z.. Song, Y., Wang, J., and Xie, P. Evit: Expediting vision transformers via token reorganizations. In ICLR, 2021.
Guo, J., Chen, X., Tang, Y., and Wang, Y. Slab: Efficient transformers with simplified linear attention and progres- sive re-parameterized batch normalization. In ICML,, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014.
Guo, S., Alvarez, J. M., and Salzmann, M. Expandnets: Lin- car over- over-varameeee parameterization to train compact convolutional networks. In NeurIPS, 2020.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In ICCV, 2017.
Hao, Z., Guo, J., Jia, D., Han, K., Tang, Y., Zhang, C., Hu, H., and Wang, Y. Learning efficient vision transformers via fine-graine fineggrained manifold distillation. In NeurlPS, 2022.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S. and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In ICCV, 2017.
Liu, 7.., Hu, H., Lin, Y., Yao, 7., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
He, Y. and Zhou, J. T. Data-independent module-aware pruning for hierarchical vision transformers. In ICLR, 2024.
Loshchilov, and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: 160..0845,, 2016.
Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient emn architecture design. In ECCV, 2018.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S. W., Anwer, R. M., and Shahbaz Khan, F. Edgenext: efficiently amalgamated cnn-transformer architecture for mobile vision applications. In ECCV, 2022.
Jangg Z.-H., Hou, Q., Yuan, L., Zhou, D., Shi, Y., Jin, X., Wang, A., and Feng, J. All tokens matter: Token labeling for training better vision transformers. In NeurlPS, 2021.
Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A.. Raste- gari, M., and Tuzel, O. Token pooling in vision trans- formers for image classification. In WACV, 2023.
Kim, M., Gao, S., Hsu, Y-C.., Shen, Y., and Jin, H. Token fusion: Bridging the gap between token pruning and token merging. In WACV, 2024.
Mchta, S. and Rastegari, M. Mobilevit: light- pprpoo,, and mobile-friendly vision transformer. In ICLR, 2022a.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y. al. Segment anything. In ICCV, 2023.
Mehta, S. and Rastegari, M. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.026880 2022b.
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning, In ECCV, 2022a.
Meng, L., Li, H., Chen, B.-C., Lan, S., Wu, Z., Jiang, Y.GG., and Lim, S.-N. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
Kong, Z., Ma. H., Yuan, G., Sun, M., Xie, Y., Dong, P., Meng, X., Shen, X., Tang, H., Qin, M., et al. Peeling the onion: Hierarchical reduction data redundancy for efficient vision transformer training. In AAAI, 2022b.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
10

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Xu, K., Wang. Z.,, Chen, C., Geng, X., Lin, J., Yang, X., Wu, M., Li, X., and Lin, W. Lpvit: Low-power semi- structured pruning for vision transformers. In ECCV, 2024a.
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurlPS, 2021.
Ryoo, M., Piergiovanni, A., Amab, A., Dehghani, M., and Angelova, A. Tokenlearne: Adaptive space-time tok- enization for videos. In NeurlPS, 2021.
Xu, X., Li, C., Chen, Y., Chang, X., Liu, J., and Wang, No token left behind: Efficient vision transformer via dynamic token idling. In AJCAI, 2023.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. In NeurlPS Data Centric A/ Workshop, 2021.
Xu, X., Wang, S., Chen, Y., Zheng, Y., Wei, Z., and Liu, J. Gip-vit: Efficient vision transformers via graph-based token propagation. In WACV, 2024b.
Xu, Y., Zhang, Z., Zhang, M.. Sheng, K., Li, K., Dong, W., Zhang, L.,, Xu, C., and Sun, X. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In AAAI, 2022.
Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.-H., and Khan, F. S. Swiftformer: Efficient additive attention for wransformerbbaedd real-time mobile vision applica- tions. In ICCV, 2023.
Yao, Z. Cao, Y., Lin, Y., Liu, Z., Zhang, Z., and Hu, H. Leveraging batch normalization for vision transformers. In ICCV, 2021.
Tan, Z., Li, X., Wu, Y., Chu, Q., Lu, L., Yu, N., and Ye, J. Boosting vanilla lightweight vision transformers via re- purammtterizationn In ICLR, 2024.
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S. Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In ICIRR 2020.
Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., and Tao, D. Patch slimming for efficient vision transformers. In CVPR, 2022.
Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. In AAAI, 2022a.
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, I..,
Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mip-mixer: An all-mlp architec- ture for vision. In NeurlPS, 2021.
Yu, L. and Xiang, W. X-pruner: explainable pruning for vision transformers. In CVPR, 2023.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In ICML, 2021.
Yu, S., Chen, T., Shen, J., Yuan, H., Tan, J., Yang, S., Liu, J., and Wang, Z. Unified visual transformer compression. ICLR, 2022b.
Vasu, P. K. A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. Fastvit: A fast hybrid vision transformer using structural reparameterization. In ICCV, 2023a.
Yu, W., Luo, M., Zhou, P., Si, C.,, Zhou, Y., Wang. X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In CVPR, 2022c
Vasu, K. A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. Mobileone: An improved one millisecond mobile backbone. In CVPR, 2023b.
Zhang, H., Zhou, Y., and Wang, G.-H. Dense vision trans- former compression with few samples. In CVPR, 2024.
Zhang, J., Li, X., Li, J., Liu, L., Xue, Z., Zhang, B., Jiang, Z., Huang, T., Wang, Y., and Wang. C. Rethinking mobile block for efficient attention-based models. In ICCV,
Vaswani, A. et al. Attention is all you need. In NeurIPS, 2017.
Wang, A., Chen, H., Lin, Z., Han, J., and Ding, G. Repvit: Revisiting mobile enn from vit perspective. In CVPR, 2024.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In CVPR, 2017.
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., and Yuan, L. Tinyvit: Fast pretraining distillation for small vision transformers. In ECCV, 2022.
Zhu, A., Wang, Y., Li, W., and Qian, P. Structural reparame- terization lightweight network for video action recogni- tion. In ICASSP. 2023.

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
Zong, Z., Li, K., Song, G., Wang, Y., Qiao, Y., Leng. B., and Liu, Y. Self-slimmed vision transformer. In ECCV, 2022.
12

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameteriiatioo Feedforward Network Layers
A. Training Settings
All RePaViTs are rigorously trained on the ImageNet-II Ik dataset (Deng et al., 2009), following the same data augmentations proposed by DeiT (Touvron al., 2021). Consistently, the total number training epochs standardized at 300. In an effort to accommodate the substitution of LayerNorm with BatchNorm, we have increased the batch size 4096. Additionally, the Lamb optimizer (You et al., 2020) has been selected to ensure stable training with large batch size. Learning rates dedicatedly configured for different backbone architectures, and a cosine scheduler (Loshchilov & Hutter, 2017) is utilized for learning rate adjustment throughout the training period. Detailed training settings are provided in Table 7.
Table Training settings of RePaViTs for the image classification task.
Batch
Base learning rate
Min learning rate
Warmup learning rate
Weight decay
Drop path rate
Model
Epochs
Optimizer
Scheduler
size
RePa-DeiT--inn
RePa-DeiT-Small
4 10-3
0.10
RePa-DeiT-Ban
RePa-ViT-LLarge
1 x 10-3
5 x 10-5
0,30
RePa-ViTE Hug-He
Cosine scheduler
300
4096
Lamb
1 10
0.05
RePa-Swin-Tiny
RePa-Swin-Small
10-3
RePa-Swin Base
0.10
RePa-LV-ViT-S,
1024
1 x 10-3
1 10-5
RePa-LV-ViTA VIT-M
B. Self-Supervised Learning Performance
Large foundation models with superior performance are usually trained with self-supervised learning techniques. To demonstrate the potential applicability of RePaViT with self-supervised learning, we first validate our method using DINO (Caron et al., 2021) and report the performance in Table 8. We adopt the same training settings as outlined in DINO. Even with self-supervised learning, RePa ViTs still exhibit substantial efficiency enhancement.
Notably, there is a consistent trend observed in Section 4.2 that when the model size increases, our method yields greater speed improvements and smaller accuracy gap. For example, RePa-Vitl Vil-Small achieves 39.4% increase in speed (1779.6 image/second vs 1277.0 image/second) with 2.6% drop in accuracy (74.4% vs 77.0% when using linear classifier. In the case of employing a larger backbone model, RePa-ViTe ViT-Base realizes a more significant acceleration of 57.2% (623.0 image/second vs 396.2 image/second) with a smaller accuracy loss of 1.2% (77.0% vs 78.2%). These results indicate a high adaptability of our RePaViT using different learning paradigms.
Table 8. RePaViT performance DINO models (Caron et al., 2021).
Compl. (GMACs) +
Speed (img/s)
A-NN
Linear top-1 acc.
Model
#MParam.
t
top-1 acc.
ViT-Small
21.7
4.3
1277.0
72.8%
77.0%
RoPa- Pita-ViT--mall/0.77
12.8 ( 41.1%)
2.5
41.9%)
1779.6 (9.4)%)
69.6%
74.4%
ViT- Base
85.8
16.9
396.2
76.1%
78.2%
RePa-Ver RePa-ViT-BBe Bas-Base00 50.4 ( 41.3%)
9.9
41.4%)
623.0 (1-57 2%%
74.1 14.
77.0%
Next, we evaluate RiPaVIT on a more advanced aanuuag-guided contrastive learning framework, specifically CLIP (Radford al., 2021). We adopt the open-source OpenCLIP framework (Cherti al., 2023) and train all models the LAION-400 -40M dataset (Schuhmann et al., 2021), with total of 3B seen data points. All training configurations strictly follow the default settings of OIenCIIP. The zero-shot classification performance on the ImageNet-1K K validation set is presented in Table 9,
For the smaller CLIP-ViT-B/3 model, our RePa-CLIP- CELI--VITBB/33 VTT-B/32 achieves a 26.8% speed increase with a negligible 0.3% accuracy drop. On the larger CLIP- CLIP-ViT-B/IC model, our method improves inference speed by 24.7% while achieving a 0.8% gain in zero-shot classification top-1 accuracy. These results demonstrate the effectiveness of RePaViT in enhancing the
13

---

RePaVIT: Scalable Vision Transformer Acceleration via Structural Reparameterization Feedforward Network Layers
Table RePaViT performance on CLIP models (Radford et al., 2021). models are trained on LAION 400M dataset with
seen samples in total.
Model
Idle ratio 0
#MParam. +
Complexity (GFLOPs)
Speed (image/sevond)
Top-1 accuracy
CLIP-ViT-B/32
87.9
4.4
3860.2
57.1%
DEPa-CLI--ii-BB22
0,50
66.6
(-24.2%)
3.4 (-22.7%)
4893.5 (++26.8%)
56.8% (-0.3%)
D-P-CCLIPVVIT-BAI
0.75
52.4 (-40.4%)
2.6 (-40.9%)
5812.3 (+550.6%)
53.2% (-3.9%)
CLIP-ViT-B/16
86.2
17.6
824.2
62.7%
RePa-CLIP-ViT-BII
0.50
64.9
(-24.7%)
13.4 (-23.9%)
1027.9 (+24.7%)
63.5% (+00.8%
RePa-CI.I-ViT-BB16
0.75
50.8 (-41.1%)
10.6 (-39.8%)
1161.5 (++40.9%)
61.0% 1.7%)
efficiency large foundation models trained with language-guided contrastive learning. We anticipate our method to be applied to large foundational vision models in future work.
C. Limitations
Despite the exceptional performance of RePaFormers on large backbone models, there is a notable decrease in accuracy as the model size shrinks. For example, as demonstrated in Table 4, the accuracy of dePa-Deii-Timy decreases significantly from 72.1 to 64.2%. This performance drop primarily attributed to the reduced nonlinearity in the backbone, which is a consequence keeping channels idle. In smaller models, both the number of layers and the number of feature channels limited, resulting in substantially fewer activated channels compared to larger models. After applying the channel idle mechanism with high idle ratio (e.g., 75%), tiny models would lack sufficient non- non-linear transformations. However, as the model size increases, both the number of layers and feature channels expand, enhancing the model's robustness and mitigating the impact of reduced nonlinearity.
In conclusion, while our method may not be optimally suited for tiny models, it significantly enhances the performance of large VIT models. We sincerely invite the research community to further investigate and validate the effectiveness of our approach on large foundational models, such as SAM (Kirillov al., 2023) or GPT (Radford al., 2019; Brown et al., 2020). This exploration could provide valuable insights into the scalability and adaptability of method across various advanced computational frameworks.
14


Reviews:
[{"review_id": "N0cEeEV1ga", "reviewer": "Reviewer_NyjX", "summary": "The paper introduces RePaViT, a method for accelerating Vision Transformers by applying structural reparameterization specifically to FFN layers. The key observation is that FFN layers significantly contribute to ViT inference latency, especially as the model scales. To address this, the authors propose a \"channel idle\" mechanism, maintaining a subset of channels inactive, forming a linear path that can be structurally reparameterized during inference. Experimental results demonstrate substantial speed-ups and even accuracy gains compared to the original ViTs. Notably, RePaViT achieves superior efficiency compared to state-of-the-art pruning and reparameterization methods, validating its practical value for real-world applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer NyjX for the recognition of our work, especially on\n\n* clear novelty\n\n* significant real-world application potential\n\n* and thorough and convincing experiments. \n\nWe would like to answer and clarify the questions as below:\n\n---\n\n___Q1: What motivated the default choice (0.75)? How sensitive is this choice in terms of generalization to other datasets/tasks beyond those tested?___\n\nA1: We thank for this insightful question.\n\nFirstly, we would like to clarify that the channel idle ratio $\\theta$ is a __static architectural hyperparameter__ designed to control the trade-off between model efficiency and performance. It is not meant to be universally optimal but rather user-configurable based on specific hardware constraints and application requirements. __More importantly, the choice of channel idle ratio is highly related to the model size__. As shown in Table 4, larger models (e.g., DeiT-Base) tolerate higher idle ratios with minimal accuracy drop. In particular, when the model size grows to DeiT-Base level and beyond, 0.75 idle ratio can result in high performance, which motivates our choice.\n\nSecondly, **as for the method-level generalization across tasks**, we would like to emphasize that RePaViT has been evaluated on four diverse tasks: (1) image classification on ImageNet-1K, (2) object detection on MS COCO, (3) semantic segmentation on ADE20K, and (4) zero-shot image classification using CLIP on LAION-400M. These results collectively demonstrate the method’s robustness and applicability across different computer vision problems.\n\nThirdly, **as for the model-level generalization across datasets**, we have provided zero-shot classification results in Table 9, Appendix B, where RePaViT is applied to CLIP models with different idle ratios. Since CLIP is pretrained on the large-scale unlabelled LAION-400M dataset and directly evaluated on the ImageNet-1K validation set without finetuning, this setup reflects the model's generalizability on unseen data. Notably, RePa-CLIP-ViT-B/16 with an idle ratio of 0.5 can outperform the vanilla model by 0.5%.\n\nFinally, **on the sensitivity of the idle ratio**, Table 4 presents a detailed analysis. We observe that performance remains stable or improves slightly up to a certain point, and then degrades as the idle ratio increases. Specifically, when the idle ratio exceeds 0.75, a noticeable accuracy drop occurs. We hypothesize that excessive pruning may cause insufficiency of nonlinearity in the model, negatively affecting the model representation capacity.\n\nWhile our current study has provided extensive experiment results demonstrating the generalizability of our method and the sensitivity of idle ratio choice, we acknowledge that further evaluation could offer deeper insights into its robustness. However, conducting such experiments would require substantial additional time that can extend beyond the current rebuttal phase. We still welcome specific task/dataset suggestions from Reviewer NyjX and are happy to include additional results in the revised version.\n\n---\n\n___Q2: Training Stability: Can you discuss any observed training instabilities or convergence issues with larger channel idle ratios or model scales, and how these might be mitigated in practice?___\n\nA2: We appreciate this insightful question.\n\nFortunately, we have not observed instability or convergence issues when training RePaViT with large idle ratios or larger models. As shown in Tables 1 and 4, and according to our training logs, RePaViTs converge smoothly across all configurations. We have also provided our source code and detailed instructions in the supplementary material for reproducing our results.\n\nHowever, __reparameterizing before training can cause instability__, particularly when BatchNorm is merged early. Table 5 shows that while such reparameterization reduces training time, it degrades performance and may destabilize training—especially for large models—since normalization layers are crucial for mitigating issues like internal covariate shift, ensuring consistent signal scales across layers, which is particularly important in larger models.\n\n---\n\n___W1: Batch norm instead of Layer norm may limit the training efficiency.___\n\nA3: We would like to clarify that BatchNorm is a dedicated architectural choice to facilitate a further reparameterization of normalization layers and shortcuts into the backbone. It enhances computational efficiency during inference, and we consider the increase in training time a worthwhile trade-off for the efficiency benefits achieved. Nonetheless, our method is still compatible with LayerNorm.\n\n---\n\nWe hope our responses address your questions and concerns, and would sincerely appreciate your consideration of raising the recommendation score.", "claims_and_evidence": "The primary claims—that FFN layers dominate ViT latency, and that structural reparameterization of these layers significantly improves computational efficiency—are strongly supported by comprehensive experiments.", "methods_and_evaluation_criteria": "The proposed method—structural reparameterization applied directly to FFN layers via a channel idle mechanism—is clearly defined and justified by latency analyses. Evaluation criteria are appropriate and rigorously applied. Experiments with standard benchmarks like ImageNet, MSCOCO, and ADE20K provide credible validation.", "theoretical_claims": "No proofs in the paper.", "experimental_designs_or_analyses": "The experimental design, including various model sizes and comparative baselines (vanilla ViT, pruning methods, reparameterization methods like SLAB), is robust. However, additional transparency about how exactly throughput measurements were standardized across different methods would enhance reproducibility.", "relation_to_broader_scientific_literature": "RePaViT effectively positions itself within current literature by contrasting clearly with:\n\n1. Network pruning methods: emphasizing higher practical efficiency and hardware-friendliness.\n2. Other reparameterization methods: highlighting methodological differences (vertical versus horizontal reparameterization) and targeting intrinsic ViT architectures rather than hybrid or CNN-augmented structures.\n\nThe paper situates its contribution well by emphasizing originality in applying structural reparameterization directly to FFN layers.", "essential_references_not_discussed": "The paper provides comprehensive references, and there were no obvious omissions of essential references directly relevant to the proposed method.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of structurally reparameterizing FFN layers is original and clearly presented.\n2. Clear real-world application potential, especially given that many vision foundation models use ViT backbones.\n3. Thorough and convincing experimentation demonstrating scalability, accuracy trade-offs, and latency improvements.\n\nWeaknesses:\nBatch norm instead of Layer norm may limit the training efficiency.", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "1.  What motivated the default choice (0.75)? How sensitive is this choice in terms of generalization to other datasets/tasks beyond those tested?\n\n2. Training Stability: Can you discuss any observed training instabilities or convergence issues with larger channel idle ratios or model scales, and how these might be mitigated in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZACJArXyrT", "reviewer": "Reviewer_d1Yn", "summary": "This paper proposes a novel structural reparameterization method -- RePaViT that targets the feedforward network (FFN) layers of Vision Transformers (ViTs) to accelerate inference. The key idea is a channel idle mechanism—during training, only a subset of FFN channels are activated (with the others kept “idle”), which creates a linear shortcut that can later be merged into a simplified, reparameterized structure at test time. This approach not only reduces computational complexity (both in terms of parameter count and FLOPs) but, in many cases, also improves or preserves accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer d1Yn's detailed and careful review comments. We thank Reviewer d1Yn for pointing out the strengths of our work, including\n\n* clear and understandable presentation\n\n* solid and comprehensive experiments\n\n* high performance\n\n* and novel from existing methods.\n\nWe would like to answer the question as below:\n\n---\n\n___W2&Q2: I am curious about the performance of the proposed method when combined with other efficiency methods (e.g., quantization, pruning).___\n\nA2: We thank for this insightful question on the combination of our method and other kind of existing efficient methods.\n\n* __Pruning__\n\n  Since our method primarily focuses on reducing channel-wise complexity for ViTs, we decide to test our method combined with a spatial-wise token reduction method, ToME, rather than network pruning methods. The results of ToMe-RePa-ViTs with different token reduction numbers (r) are shown in the Table below. In general, our method combines with token reduction method well and yields more significant improvement in the trade-off between accuracy and efficiency.\n\n  __Notably, ToMe-RePa-ViT-Large/0.75 with reduction number 8 achieves more than 200% acceleration (374.7 imgs/s vs 107.7 imgs/s) with merely 0.8% accuracy drop.__\n\n|Backbone|r|Reparam|#MParam.|GMACs|Speed (imgs/s)|Accuracy|\n|-|-|-|-|-|-|-|\n|ToMe-RePa-DeiT-Small/0.75|0 (No ToMe)|×|22.1|4.6|1279.1|77.1%|\n||0 (No ToMe)|√|13.2|2.9|1975.5|77.1%|\n||1|√|13.2|2.7|1983.3|77.0%|\n||2|√|13.2|2.6|2077.5|76.9%|\n||3|√|13.2|2.4|2118.0|76.8%|\n||4|√|13.2|2.2|2225.3|76.6%|\n||5|√|13.2|2.1|2286.5|76.4%|\n||6|√|13.2|2.1|2370.1|76.1%|\n||7|√|13.2|2.0|2438.9|75.9%|\n||8|√|13.2|1.9|2542.3|75.4%|\n|ToMe-RePa-DeiT-Base/0.75|0 (No ToMe)|×|86.6|17.6|393.8|81.8%|\n||0 (No ToMe)|√|51.1|10.6|659.5|81.4%|\n||1|√|51.1|9.6|660.6|81.5%|\n||2|√|51.1|9.3|678.9|81.4%|\n||3|√|51.1|9.0|697.4|81.4%|\n||4|√|51.1|8.7|738.8|81.3%|\n||5|√|51.1|8.4|746.9|81.2%|\n||6|√|51.1|8.1|773.3|80.9%|\n||7|√|51.1|7.8|812.0|80.6%|\n||8|√|51.1|7.5|843.7|80.1%|\n|ToMe-RePa-ViT-Large/0.75|0 (No ToMe)|×|304.5|59.8|102.7|82.0%|\n||0 (No ToMe)|√|178.4|34.9|207.2|82.0%|\n||1|√|178.4|32.8|208.6|82.0%|\n||2|√|178.4|30.7|220.9|81.9%|\n||3|√|178.4|28.6|236.3|81.9%|\n||4|√|178.4|26.5|257.1|81.8%|\n||5|√|178.4|24.4|278.9|81.8%|\n||6|√|178.4|22.3|304.9|81.6%|\n||7|√|178.4|20.2|337.2|81.5%|\n||8|√|178.4|18.1|374.7|81.2%|\n\n* __Quantization__\n\n  For simplicity, we employed TensorRT to perform FP32-to-FP16 quantization on our pretrained and reparameterized models. TensorRT provides a comprehensive ecosystem of tools designed to deliver high-performance deep learning inference via _post-training_ quantization. The evaluations before and after reparameterization of our RePaViT and RePaSwin models were conducted on the same hardware platform mentioned in the paper (NVIDIA A6000). \n\n  __Remakably, when quantized to FP16, our method presents more than 200% acceleration. And the acceleration even signifies with reparameterization, demonstrating the effectiveness of combining quantization with our method in real-world scenarios.__ \n\n\nModel| Reparam | FP16 Imgs/s | FP16 Acc | FP32 Imgs/s | FP32 Acc\n-|-|-|-|-|-\nRePa-DeiT-Small/0.75|×|  11945.84 (+210.3%)  |  76.41%|  3849.89|  76.41%\n||√|  18329.13 (+248.4%) |  76.40%|  5260.94|  76.41%\nRePa-DeiT-Base/0.75|    × |   3135.53 (+158.3%)  |  81.31%|  1213.87|  81.31%\n||    √ |   5441.13 (+189.8%) |  81.32%|  1877.24|  81.32%\nRePa-ViT-Large/0.75 |    × |    935.75 (+172.6%) |  81.96%|   343.29|  81.96%\n||    √ |   1650.12 (+210.8%) |  81.97%|   530.94|  81.95%\nRePa-Swin-Tiny/0.75 |    × |   7144.35 (+154.8%) |  78.43%|     2804.70 |  78.44%\n||    √ |  13351.39 (+202.8%)  |  78.43%|   4408.99 |  78.43%\nRePa-Swin-Small/0.75 |    × |   4225.21 (+152.0%) |  81.56%|    1676.96 |  81.56%\n||    √ |   7782.22 (+191.7%) |  81.56%|    2667.86 |  81.56%\nRePa-Swin-Base/0.75 |   × |    2670.20 (+152.8%) |  82.58%|    1056.28 |  82.59%\n||  √  |4873.75 (+205.7%)  |  82.58%|    1594.54 |  82.58%\n\n__In conclusion, our method combines with quantization and pruning methods well, exhibiting more significant enhancement in performance-efficiency trade-off.__ And the integration of TensorRT indicates the potential of deploying on edge devices.\n\n---\n\n___W1&Q1: How does RePaViT perform in deployment scenarios on edge devices?___\n\nA1: We thank the Reviewer for this question, which is indeed interesting and essential for demonstrating the real-world practicality of our RePaViT. \n\nUnfortunately, due to strict equipment management policies in our organization and time limits, we were unable to successfully apply for access to an edge device during the rebuttal period. As a result, we could not provide real-world inference speed measurements on actual edge hardware. We would like to report detailed performance metrics upon deployment to the Jetson AGX Orin platform in the follow-up discussion phase.\n\n---\n\nWe hope our responses address your concerns and would sincerely appreciate your reconsideration of raising the score.", "claims_and_evidence": "Claim: FFN layers are the major bottleneck in ViTs, and optimizing them can yield significant latency reductions.  \nEvidence: The latency analysis shows that FFN layers account for a growing portion of inference time as model size increases.  \n\nClaim: The channel idle mechanism and subsequent reparameterization can reduce both parameters and computation while maintaining accuracy.  \nEvidence: Extensive experiments on various backbones (DeiT, Swin, LV-ViT, ViT-Large/Huge) illustrate that RePaViT achieves up to 68% speedup and, in some cases, even higher accuracy compared to the original models.", "methods_and_evaluation_criteria": "Methods: \nIn each FFN layer, only a fraction of the channels pass through the nonlinear activation while the rest follow a linear path. During inference, the activated and idle branches are merged using structural reparameterization (with BatchNorm merging) to form a more compact FFN. The approach is applied on standard ViT architectures trained from scratch on ImageNet-1K, and evaluations span image classification, object detection, and semantic segmentation.   \nEvaluation:\nThe paper reports throughput (images per second), parameter counts, FLOPs, and top-1 accuracy on ImageNet-1K. It further compares performance on dense prediction tasks (MS COCO, ADE20K) and benchmarks against network pruning and alternative reparameterization methods (e.g., SLAB).", "theoretical_claims": "I do not think there are any theoretical claims. Most of the designs are empirical based.", "experimental_designs_or_analyses": "Experiments:\nThe authors conduct comprehensive experiments across multiple ViT backbones (plain and hierarchical) on ImageNet-1K, as well as on downstream tasks like object detection (MS COCO) and semantic segmentation (ADE20K).   \nAnalysis:\nDetailed tables compare the original and reparameterized models in terms of speed, accuracy, and computational cost.", "relation_to_broader_scientific_literature": "Its focus on directly optimizing FFN layers through a novel channel idle mechanism distinguishes it from existing methods that mainly target attention layers or use pruning strategies.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The visualization is clear and understandable.  \n2. The experiments are solid and comprehensive.  \n3. The reparameterized models achieve good efficiency and accuracy.\n\nWeaknesses:\n1. Need more discussion about edge devices or resource limited scenarios.\n2. Need more discussion about combination with other efficiency methods.", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "1. How does RePaViT perform in deployment scenarios on edge devices?\n2. I am curious about the performance of the proposed method when combined with other efficiency methods (e.g., quantization, pruning).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JB6J1ZOtni", "reviewer": "Reviewer_8Sgy", "summary": "This paper proposes RePaViT, a method for accelerating Vision Transformers (ViTs) through structural reparameterization of the feedforward network layers. Specifically, this paper argues that the computation costs of FFN layers cannot be ignored, thus a structural reparameterization method on FFN layers are designed. By using the channel idle mechanism, the RePaViT reduces the computation complexity in the inference stage.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 8Sgy's recognition of our method's high performance and would like to address the concerns raised:\n\n---\n\n__C1: Table 2 tries to show the advantages of RePaViT comparing with pruning methods. However, the number of model parameters of most of pruning methods are not provided, which means it is hard to directly compare them with the RePaViT.__\n\nA1: We acknowledge the difficulty in directly comparing RePaViT with certain pruning methods due to the unavailability of parameter counts in their publications and/or unavailability of source code:\n1. __X-pruner [1] does not report the number of parameters in its paper and has not released source code so far.__ So, we're unable to provide its number of model parameters.\n2. __DC-ViT [2] does not report the number of parameters in its paper and had not provide its source code before the submission deadline of ICML__. As a result, we're unable to provide its number of model parameters in our submission. However, we would like to reproduce its code and report the updates in the table below. Notably, DC-ViT's model size and computational complexity depends on both the number of pruned blocks and a special MLP compression ratio. Unfortunately, the authors haven't released the MLP compression ratios for reproducing their reported results (except 0.751734186 for DeiT-Base) and searching for exactly the same ratios would require a search process exceeding the rebuttal phase. So we can only provide the number of parameters and complexity that yields the closest performance as reported.\n3. __LPViT [3], while its source code is available, has not released the hardware benchmarking code so far.__ It still employs masking to simulate pruning without reducing the actual parameter count, which remains equivalent to the original model. Our investigation on the source code confirm this. And this is also why LPViT only reports sparsity in its paper. \n\nTo facilitate a more comprehensive comparison, we have reproduced several pruning baselines using their latest released code. The results are summarized in the table below (italic for updated, bold for best). \n\n|**Backbone**|**Method**|**#MParam. ↓**|**Compl. (GMACs) ↓**|**Speed improv. ↑**|**Top-1 acc. ↑**|\n|-|-|-|-|-|-|\n|**DeiT-Small**|WDPruning|13.3|2.6|+18.3%|78.4%|\n|| X-pruner|-|2.4|-|78.9%|\n|| DC-ViT|_16.6_|_3.2_| +20.0%| 78.6%|\n|| LPViT|_22.1_|__2.3__| +16.3%| **80.7%** |\n| | RePaViT/0.50 | 16.7 | 3.2| +34.9%              | 79.1%            |\n|                | RePaViT/0.75 | **13.2**       | 2.9                  | **+54.4%**          | 79.6%            |\n| **DeiT-Base**  | WDPruning    | 55.3           | 9.9                  | +18.2%              | 80.6%            |\n|| X-pruner     | -              | **8.5**              | -                   | 81.0%            |\n|  | DC-ViT       | _65.1_         | _12.7_               | _+18.4%_            | 81.3%            |\n|   | LPViT        | _86.6_         | 8.8                  | +18.8%              | 80.8%            |\n|   | RePaViT/0.50 | 65.3           | 12.7                 | +21.8%              | **81.4%**        |\n|                | RePaViT/0.75 | **51.1**       | 10.6                 | **+67.5%**          | **81.4%**        |\n| **Swin-Small** | WDPruning    | 32.8           | 6.3                  | +15.3%              | 81.3%            |\n|                | X-pruner     | -              | 6.0                  | -                   | 82.0%            |\n|  | RePaViT/0.50 | 37.8           | 6.4                  | +28.8%              | **82.8%**        |\n|   | RePaViT/0.75 | **29.9**       | **5.1**              | **+41.2%**          | 81.6%            |\n| **Swin-Base**  | DC-ViT       | _66.4_         | _11.5_               | +14.9%              | **83.8%**        |\n|   | LPViT        | _87.8_         | 11.2                 | +8.9%               | 81.7%            |\n| | RePaViT/0.50 | 66.8           | 11.5                 | +24.5%              | 83.4%            |\n|  | RePaViT/0.75 | **52.8**       | **9.0**              | **+49.6%**          | 82.6%            |\n\nWe would like to emphasize that RePaViT still achieves the best trade-off in terms of model efficiency and accuracy.\n\n---\n\n__C2: Most of baseline methods are proposed in 2021. More recent baselines may make the experimental results stronger.__\n\nA2: We respectfully disagree with this comment. All baseline methods compared in our study are published after 2021 as shown below:\n\n| Method    | Conference | Year |\n| --------- | ---------- | ---- |\n| WDPruning | AAAI       | 2022 |\n| X-Pruner  | CVPR       | 2023 |\n| DC-ViT    | CVPR       | 2024 |\n| LPViT     | ECCV       | 2024 |\n| SLAB      | ICML       | 2024 |\n\nAnd the backbones choices (DeiT/ViT/Swin/LV-ViT) follow the convention in token pruning and network pruning works.\n\n---\n\nWe sincerely appreciate your review comments, hope our responses satisfactorily address your concerns, and kindly request a consideration of recommendation score.", "claims_and_evidence": "The paper's claims are supported by some evidence, particularly through performance comparisons and ablation studies. However, there are still some problems remain. The details are listed below: \n\nPros:\n\n1. Table 1 shows that the RePaViT can reduce the number of parameters, therefore improve the inference speed. \n\n2. Table 3 shows that the RePaViT has better inference speed and accuracy than the previous re-parameterization methods. \n\n3. Ablation studies shows that importance of the idle ratio and the training time overparameterization. \n\n\n\nCons:\n1. Table 2 tries to show the advantages of RePaViT comparing with pruning methods. However, the number of model parameters of most of pruning methods are not provided, which means it is hard to directly compare them with the RePaViT.\n\n2. Most of baseline methods are proposed in 2021. More recent baselines may make the experimental results stronger.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper make sense for the problem or application at hand.", "theoretical_claims": "This paper does not include theoretical proofs.", "experimental_designs_or_analyses": "The experimental designs are valid to support the method.", "relation_to_broader_scientific_literature": "The key contribution of the paper is a novel re-parameterization methods on FFN layers, which has some relationship with RepVGG(Ding et al., 2021), and the authors analyzed the main differences with RepVGG in 3.5", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers support acceptance of this paper for the interesting and experimentally supported idea presented in this paper. So I m in favor of accepting this paper to ICML
Decision: Accept (poster)