ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
Authors: Haoran You
Huihong Shi
Yipin Guo
Yingyan Celine Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Efficient Vision Transformer; Multiplication-reduced networks; Hardware acceleration
TL;DR: We propose to reparametrize the pre-trained ViT with mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication-reduced model, dubbed ShiftAddViT for end-to-end inference speedups.

Abstract: Vision Transformers (ViTs) have shown impressive performance and have become a unified backbone for multiple vision tasks. However, both the attention mechanism and multi-layer perceptrons (MLPs) in ViTs are not sufficiently efficient due to dense multiplications, leading to costly training and inference. To this end, we propose to reparameterize pre-trained ViTs with a mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication-reduced model, dubbed $\textbf{ShiftAddViT}$, which aims to achieve end-to-end inference speedups on GPUs without requiring training from scratch. Specifically, all $\texttt{MatMuls}$ among queries, keys, and values are reparameterized using additive kernels, after mapping queries and keys to binary codes in Hamming space. The remaining MLPs or linear layers are then reparameterized with shift kernels. We utilize TVM to implement and optimize those customized kernels for practical hardware deployment on GPUs. We find that such a reparameterization on (quadratic or linear) attention maintains model accuracy, while inevitably leading to accuracy drops when being applied to MLPs. To marry the best of both worlds, we further propose a new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication or its primitives as experts, e.g., multiplication and shift, and designing a new latency-aware load-balancing loss. Such a loss helps to train a generic router for assigning a dynamic amount of input tokens to different experts according to their latency. In principle, the faster the experts run, the more input tokens they are assigned. Extensive experiments on various 2D/3D Transformer-based vision tasks consistently validate the effectiveness of our proposed ShiftAddViT, achieving up to $\textbf{5.18$\times$}$ latency reductions on GPUs and $\textbf{42.9}$% energy savings, while maintaining a comparable accuracy as original or efficient ViTs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddViT.

ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformers
Haoran You", Huihonn Shi", Yipin Guo", and Yingyan (Celine) Lin
Georgia Institute of Technology, Atlanta, GA [haoran. you, celine.lin) @ gatech.edu, eic-lab groups gatech.edu
Abstract
Vision Transformers (ViTs) have shown impressive performance and have become a unified backbone for multiple vision tasks. However, both the attention mechanism and multi-layer perceptrons (MLPs) in ViTs are not sufficiently efficient due to dense multiplications, leading to costly training and inference. To this end, we pro- pose to reparameterize pre-trained ViTs with mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication mulliplication-cduueed reduced model, dubbed ShiftAdd which aims to achieve end-to-end inference speedups on GPUs without requiring training from scratch. Specifically, all MatMuls among queries, keys, and values are reparameterized using additive kernels, after mapping queries and keys to binary codes in Hamming space. The remaining MLPs or linear layers then reparameterized with shift kernels. We utilize TVM to implement and optimize those customized kernels for practical hardware deployment on GPUs. (quadratic or linear) attention maintains
We find that such reparameterization model accuracy, while inevitably leading to accuracy drops when being applied MLPs. To marry the best of both worlds, we further propose new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication its primitives experts, e.g., multiplication and shift, and designing new aitency-aware load- hadlbblanniig loss. Such loss helps to train generic router for assigning a dynamic amount of input tokens different experts according to their latency. In principle, the faster the experts run, the more imput tokens they are assigned. Extensive experiments on various 2D/3D Transformer-be based vision tasks consistently validate the effectiveness of our proposed ShiftAddViT, achieving up to 5.18x latency reductions on GPUs and 42.9% energy savings, while maintain- ing a comparable accuracy as original or efficient ViT's. Codes and models are available at https: // //t//ggittt //gggtt.bbb.e com/GATECH GATECH- WAATECCEECC/Sh-FE.. BIC/ShiftAddVif.
Introduction
Vision Transformers (ViTs) have emerged as powerful vision backbone substitutes to convolutional neural networks (CNNs) due to their impressive performance [57, 16]. However, ViTs' state-of-the-art (SOTA) accuracy comes at the price of prohibitive hardware latency and energy consumption for both training and inference, limiting their prevailing deployment and applications [34, 51], even though pre-trained ViT models are publicly available. The bottlenecks are twofold based on many profiling results: attention and MLPs, e.g., they account for 36%/63% of the FLOPs and 45%/46% of the latency when running DeiT-Base [55] on an NVIDIA Jetson TX2 GPU [72, 19]). Previous efficient VIT solutions mostly focus macro-architecture designs [21, 39, 32, 55, 61] and linear attention
optimization [59, 50, 5, 13, 6, 73]. However, they pay less attention on reducing the underlying dominant multiplications as well as on co-optimizing both attention and MLPs.
We identify that one of the most effective yet still missing opportunities for accelerating ViTs is to reparameterize their redundant multiplications with mixture of multiplication primitives, i.e.,
'Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

bitwise shifts and adds. The idea is drawn from the common hardware design practice in computer architecture or digital signal processing. That is, the multiplication can be replaced by bitwise shifts and adds [67, 22]. Such a hardware-inspired approach can lead to an efficient and fast hardware implementation without compromising the model accuracy. As such, this paper embodies the shift&add idea in ViT's towards new type of multiplication-reduee previous ShiftAddNet [70] which requires full and slow training and dedicated hardware accelerator support, this work starts from pre-trained ViTs to avoid the tremendous cost of training from scratch and targets end-to-end inference speedups, i.e., accelerating both attention and MLPs, on GPUs.
model. Moreover, unlike
The aforementioned shift&add concept uniquely in- spires our design of naturally gerdware-friendiee ViTs, but it also presents us with three challenges to address. First, how to effectively reparameterize ViTs with shifts and adds? Previous ShiftAddNer [70] reparameterizes CNNs by cascading shift layers and add layers, lead- ing to doubled number of layers or parameters. The customized CUDA kernels of shift and add layers also
Table I: Hardware cost under 45mm CMOS.
OPs
Format
Energy (pJ)
Area (pm²)
FP32
3.7
7700
FP16
0.9
1640
Mult.
INT32
3.1
3495
INT8
0.2
282
FP32
1.I
4184
FP16
0.4
1360
Add
suffer from much slower training and inference than P [42] on GPUs. Both motivate a new easy-to- accelerate reparameterization method for ViTs. Second, how to maintain the accuracy after reparameterization? It is expected to see accuracy drops when multiplica- tions are turned into shift and add primitives [8, 11j. Most works compensate for the accuracy drops by enlarging model sizes leveraging the much improved energy efficiency advantages of shifts and adds [70], e.g., up to 196x unit energy savings than multiplications as shown in Tab. 1. While in ViTs, where input images are split into non-overlapping tokens, one can uniquely leverage the inherent adaptive sensitivity among input tokens. In principle, the essential tokens containing target objects are expected be processed using more powerful multiplications. In contrast, tokens with less important background information can be handled by cheaper primitives. Such a principle aligns with the spirit of recent token merging [3] and input adaptive methods [45, 68, 74] for ViTs. Third, how can we balance the loading and processing time for the aforementioned important and less- impootant input tokens? For ViTs, using mixture of multiplication primitives can lead to varying processing speeds for sensitive and non-sensitive tokens. This needs to be balanced; otherwise, intermediate activations may take longer to synchronize before progressing to the next layer.
INT32
0.1
137
INT8
0,03
36
INT32
0.13
157
Shift
INT16
0.057
73
INT8
0.024
34
To the best of our knowledge, this is the first attempt to address the above three challenges and to incorporate the &hift&add concept from the hardware domain in designing cnlliplication-redueedd ViTs using a mixture of multiplication primitives. Specifically, we make the following contributions:
We take inspiration from hardware practice reparameterize pre-trained ViTs with a mixture of complementary multiplication primitives, i.c., bitwise shifts and adds, to deliver a new type of multipliiction--edueee network, dubbed ShiftAdd All MatMuls in attention are reparameterized by additive kernels, and the remaining linear layers and MLPs are reparameterized by shift kernels. The kernels are built in TVM for practical deployments.
We introduce new mixture of experts (MoE) framework for ShiftAddViT to preserve accuracy post-reparameterizatinn Each expert represents either multiplication or its primitives, such as shifts. Depending on the importance of given input token, the appropriate expert is activated, for instance, multiplications for vital tokens and shifts for less crucial ones.
We introduce a latency-aware load-balancing loss term within our MoE framework dynami- cally allocate input tokens to each expert. This ensures that the number of tokens assigned aligns with the processing speeds of the experts, significantly reducing synchronization time.
We conduct extensive experiments to validate the effectiveness of our proposed ShiftAddViT. Results on various 2D/3D Transformer-basee efficiency, achieving up to 5.18x latency reductions on GPUs and 42.9% energy savings, while maintaining comparable or even higher accuracy than the original ViTs.
vision models consistently show its superior
2
Related Works
Vision Transformers (ViTs). ViTs [16, 55] split images into non-overlapping patches or tokens and have outperformed CNNs on a range of vision tasks. They use simple encoder-only architecture, primarily made up of attention modules and MLPs. The initial ViTs necessitated costly pre-training
2

---

on expansive datasets [52], and subsequent advancements like DeiT [55] and T2T- [75] alleviated this by refining training methodologies or crafting innovative tokenization strategies. Later on, a surge of new ViT architecture designs, such as PVT [60], CrossViT [7]. PiT [26], MVîT [18), and Swin- Transformer [35], has emerged to improve the aceuracy-efficiency trade-offs using a pyramid-like architecture. Another emergent trend revolves around dynamic ViT's like Dynamic ViT [45], A- ViT [68], ToME [3], and MIA-Former [74], aiming to adaptively eliminate non-essential tokens. Our MoE framework resonates with this concept but addresses those tokens using low-cost multiplication primitives that run concurrently. For edge deployment, solutions such as VeViT [21], Mobile ViT [39], and Efficient ViT [5, 50] employ efficient attention mechanisms or integrate CNN feature extractors. This approach propels VITs to achieve speeds comparable to CNNs, e.g., MobileNets [47]. In contrast, our proposed Shif is the first to draw from shift&add hardware shorteuts for reparameterizing pre-trained ViT's, boosting end-to-end inference speeds and energy efficiency without hurting accuracy. Therefore, ShiftAddViT is orthogonal to existing VIT designs and can be applied on top of them.
Tremendous efforts have been dedicated to designing efficient ViTs. For instance, to address the costly self-attention module, which possesses quadratic complexity in relation to the number of input tokens, various linear attentions have been introduced. These can be broadly divided into two categories: local attention [35, 2, 56] and kernel-based línear attention [30, 13, 65, 38, 5, 34, 1]. For the former category, Swin [35] computes similarity measures among neighboring tokens rather than all tokens. QnA [2] disseminates the attention queries across all tokens. Max ViT [56] employs block attention and integrates dilated global attention to capture both local and global information. Regarding the latter category, most methods approximate the softmax function [13, 30, 4] or the complete self-attention matrix [65, 38] using orthogonal features or kernel embeddings, allowing the computational order to switch from (QK)V to QKV). Furthermore, a few efforts target reducing the multiplication count in ViTs. For instance, Ecoformer [34] introduces a novel binarization paradigm through kernelized hashing for Q/K, ensuring that MatMuls in attentions become solely accumulations. Adder Attention [51] examines the challenges of integrating adder operations into attentions and suggests including an auxiliary identity mapping. ViT [58] incorporates spatial shift operations into attention modules. Different from previous works, our ShiftAdd focuses on not only attention but also MLPs towards end-to-end VIT speedups. All MatMuls can be transformed into watitiions/ccumllatttiiiiin
using quantization, eliminating the need for complex kernelized hashing (KSH) as in [34]. For the remaining linear layers in the attention modules or MLPs. they are converted to bitwise shifts or Moßs, rather than employing spatial shifts like [58] for attentions.
Multiplication-eess NNs. Numerous efforts aim to minimize the dense multiplications, which primary contributors to the time and energy consumption in CNNs and ViTs. In the realm of CNNs, binary networks [14, 29] binarize both activations and weights; AdderNet [8, 66, 62] fully replaces multiplications with additions with a commendable accuracy drop: Shift-based networks use either spatial shifts [64] or bitwise shifts [17] to replace multiplications. Recently, techniques initially developed for CNNs have found applications in ViTs. For instance, BiTs [36, 25] introduce binary transformers that maintain commendable accuracy; Shu et al. and Wang al. also migrate the add or shift idea to ViTs with the scope limited to attentions [51, 58]. Apart from network designs, there are also neural architecture search efforts made towards both accurate and efficient networks [71, 31]. As
compared to the most related work, i.e., ShiftAddNet [70] and AhifAA [71], our ShiftAddViT for the first time enables &hift&ad idea in ViTs without compromising the model accuracy, featuring three more characteristics that make it more suitable for practical applications and deployment: (1) starting from pre-trained ViTs to avoid the costly and tedious training from scratch; (2) focusing on optimization for GPUs rather than on dedicated hardware such as FPGAs/ASICs: (3) seamlessly
compatible with the MoE framework to enable switching between the mixture of multiplication primitives, such token-leve routing and parallelism are uniquely applicable and designed for ViTs.
3 Preliminaries
Self-attention and Vision Transformers. Self-attention a core ingredient of Transformers [57, 16]. and usually contains multiple heads H with each measuring pairwise correlations among all input tokens to capture global global-context information. can be defined as below:
fx(X): fK (X)7
Attn X) Con ,H,) Wo. where H, =,=Softmax
(X). (1)
Vdh
where h denotes the number of heads. Within each head, input tokens E R of length n and dimension d will be linearly projected to query, key, and value matrices, i.c. Q K, V e gaxds
3

---

Binary code
Shift
B(-)
MoE
Router
Add
Shift
Imput Image
Shift
B(-)
Binary code
Patchily
O(N)
Mult.
Shift
K
Add
Linear Attention
Concat
Shift
((N)
DWConv
N Patch
Embedding
(a) Inputs
(b) ShiftAddViT Attention
(c) ShiftAdaViT MLP
Figure 1: An illustration of the network architecture overview of ShiftAddViT.
through three linear mapping functions, fo XWW@ fK XW K: fv XWv. where dx d/h is the embedding dimension of each head. The results from all heads are concatenated and projected with weight matrix Wo E Rdxd to generate the final outputs. Such an attention module followed by MLPs with residuals to construct one transformer block that can be formulated as below:
X Attn(LaayrNorr(X)) + X, XMLP MLP(LayerNorm(Xx:a
X
(2)
Shift and Add Primitives. Direct hardware implementation of multiplications is inefficient. Shift and add primitives serve as "shortcuts" for streamlined hardware designs. Shift operations, equivalent to multiplying by powers of two, offer significant savings. As shown Tab. 1, such shifts can reduce energy use by up to 23.8x and chip area by 22.3x compared to multiplications [33, 70], given the INT32 data format and 45nm CMOS technology. Similarly. add operations, another efficient primitive, can achieve up to 31.0x energy and 25.5 x area savings relative to multiplications, Both primitives have driven numerous network architecture innovations [8, 17, 70, 71].
The Proposed ShiftAddViT
Overview. Given the widespread availability of pre-trained ViT model weights, we aim to fine-tune and transform them into multipliiatioonreduued ShiftAddViTs in order decrease runtime latency and enhance energy efficiency. To achieve this, we must reparameterize both the attentions and MLPs in ViTs shift and add operations. Instead of using cascaded shift layers and add layers as in ShiftAddNet [70] or designing specific forward and backpropagation schemes like AdderNet [8, 51], we lean on the simple yet effective approach of ViTs' long-dependency modeling capability. We propose replacing the dominant MLPs/Linearr/Maate intact. This approach permits ShiftAddViTs to adapted from existing ViTs without the need for training from scratch. As illustrated in Fig. 1, for attentions, we transform four Linear layers and two MatMuls into shift and add layers, respectively. For MIPss, direct replacement with shift layers results in significant accuracy degradation. Hence, we consider designing dedicated MoE framework incorporating blend of multiplication primitives, such as multiplication and shift, to achieve both accurate and efficient final models. These steps transition pre-trained ViTs into ShiTtAddViTs, achieving much reduced runtime latency while maintaining a comparable or even higher accuracy.
with shifts and adds, leaving the attention
4.1 Shift Reparameterization Towards Multiplication--eddeed Networks
This subsection describes how we reparameterize pre-trained VITs with hardware-efficient shift and add operations, including detailed implementation and the corresponding sensitivity analysis.
Reparameterization of Attention. There are two categories of layers in attention modules: MatMuls and Linears, that can be converted to Add and Shift layers, respectively. In order to reparameterize MatMuls Add layers, we consider performing binary quantization on one operand during MatMuls, (MAC) operations between
e.g., Q or K for the MatMul of QK, so that the hawppll-ann-acccmulll two matrices will be replaced with merely energy- meerg-efffccient add operations [34]. Furthermore, to build ShiftAddVTT on top of efficient linear attentions [59, 34, 38], we exchange the order of MatMuls from (QK)V to Q(KV) for achieving linear complexity w.r.t. the number of input tokens. this way, the binary quantization will be applied to K and Q as illustrated in Fig. (b), leaving the more sensitive V branch as high precision. This also allows us to insert lightweight depthwise convolution
(DWConv) to V branch in parallel to enhance local feature capturing capability with negligible

---

X12
113
1
1
0 112
X13
X X
X
012
Oz
>02
X2t
Ozz
O21
O2z
X2a
(a) Shift Layers
(b) Add Layers
Figure 2: Illustration of both Shift and Add layers, where X and refer to inputs and outputs.
overhead (<1% of total MACs) following recent SOTA linear attention designs [65, 73]. On the other hand, for reparameterizing the left four Linears in the attention module with Shift layers, we resort to sign flips and power-of-two shift parameters to represent the Linear weights [17, 70].
We illustrate the implementation Shift and Add layers in Fig. 2 and formulate them as follows:
Oshift XT Ws = Σxxx
x Ex S. Osae ΣxT 1( + 0),
(3)
where X refers to input activations, Ws and WA are weights in Shift and Add layers. re- spectively, For Shift weights Ws = 2P, P = roog (abs(W))). and 2 represents bitwise shift towards left (P > 0) right (P < 0). Both s and P are trainable during the training. For Add weights, we have applied binary quantization and will skip zeros directly, the remaining weights correspond to additions or accumulations. Com- bining those two weak players, i.c., Shift and Add, leads to higher representation capability as also stated in ShiftAddNet [70], highlight that our ShiftAddViT seamlessly achieves such combination for ViTs, without the necessity of cascading both them as a substitute for one convolutional layer.
sign(W) E {-1,1} denotes sign flips,
Reparameterization of MLPs. MLPs are often overlooked but also dominate ViTs' runtime latency apart from attentions, especially for the small or medium amount tokens [19, 78, 43J. One natural thought is to reparameterize all MLPs with Shift layers defined above. However, such aggressive design indispensably leads to severe accuracy drops, e.g., 41.1111111. 18% for PVTvI-T [60], (DWConv used between two MLPs in PVTv2 are kept). We resolve this by proposing achieve speedups with comparable accuracy, as shown in Fig. (c) and will be elaborated in Sec. 4.2.
MoE framework to
Sensitivity Analysis. To investigate whether the above reparameterization methods work as expected, we break down all components and conduct sensitivity analysis. Each of them applied ViTs with only 30 epochs of finetuning. As summarized in Tab. 2, applying lin- ear attention (LA), Add, or Shift to attention layers makes little influence on accuracy. But applying Shift to MLPs may lead to severe accuracy drops for sensitive models like P-T. Also, making MLPs more effi-
Table 2: Sensitivity analysis of reparame-
terizing ViTs with Shift and Add.
Acc. of) PVT [60, 61]
Components
Apply
PVTv2-B0
PVTvI-T
70.50
75.10
MSA
71.25
76,21
LA+Add
70.95
75.20
Attention
Shift
70.96
76.05
Shift
70.28
73.92
MLPs
MoE
70.86
74.81
cient contribute a lot to the energy efficiency (c.g., occupy 65.7% on -VTv2-BO), Both the accuracy
and efficiency perspectives motivate to propose reparameterizatioon scheme for MLPs.
4.2 ShiftAddViT: Mixture of Experts Framework
This subsection elaborates the hypothesis and method of our ShiftAddViT MoE framework.
Hypothesis. We hypothesize there are important yet sensitive input tokens that necessitate the more powerful networks otherwise suffer from accuracy drops. Those tokens are likely to contain object information that directly correlates with our task objectives as validated by [3, 74, 45, 68]. While the left tokens less sensitive and can be fully represented even using cheaper Shift layers. Such an adput-adaptive nature of ViTs calls for a framework featuring a mixture of multiplication primitives.
Mixture of Multiplication Primitives. Motivated by the sensitivity analysis in Sec. 4.1, we consider two experts Mult. and Shift for processing important yet sensitive tokens and insensitive tokens, respectively. As illustrated in Fig. (c), we apply the MoE framework to compensate for the accuracy drop. Each input token representation x will passed to one expert according to the gate values p; G(x) in routers, which are normalized via softmax distribution p; e elts and will be jointly trained with model weights. The output can be formulated y- E G(x) Ex(x), where gate function G(x) = p((x) - (px(x) 2 (x),, Vj # i), n and F, denote number of experts and i-th expert, respectively. Specifically, there are two obstacles when implementing such MoE in TVM
5

---

Dynamic Input Allocation. The input allocation is determined dynamically during runtime, and the trainable router within the MoE layers automatically learns this dispatch assignment as the gradients decay. We know the allocation only when the model is executed and we receive the router outputs. Therefore, the shape of expert input and corresponding indexes are dynamically changed. This can be handled by Py'Torch with dynamic graphs while TVM expects static input shape, To this end, we follow and leverage the compiler support for dynamism as proposed in Nimble [49] on top of the Apache TVM to handle the dynamic input allocation.
Parallel Computing. Different experts are expected to run in parallel, this can be supported by several customized distributed training frameworks integrated with Py PyTorh, e.g., FosterMoE [24], and DeepSpeed [44]. However, it remains nontrivial to support this using TVM. One option to simulate is to perform modularized optimization to mimic parallel computing. in particular, we optimize each expert separately and measure their latency. the maximum latency among all experts will be recorded and regarded as the latency of this MoE layer, and the aggregated total of the time spent for each layer is the final reported modularized model latency. To avoid any potential confusion between real-measured wall-clock time, i.e., no parallelism assumed, and simulated modularized latency, i.e., ideal parallelism assumed, reported both for models containing MoE layers in Sec. 5 to offer more insights into the algorithm's feasibility and cost implications.
Latency-aware Load balaa-calnnng Loss. The key to our MoE framework is the design of routing function to balance all experts towards higher accuracy and lower latency. Previous solutions [20, 23] use homogeneous experts and treat them equally. While in our MoE framework, the divergence and heterogeneity between powerful yet slow Mult. and fast yet less powerful Shift experts incur one unique question: How orchestrate the workload of each expert to reduce the synchronization time? Our answer is a latency-aware load- hal-hhann loss, which ensures (1) all experts receive the expected weighted sums of gate values; and (2) all experts assigned the expected number of input tokens. Those two conditions are enforced by adopting importance and load balancing loss as defined below:
LIMP = SCV (fas
(e Pi(x))ii) (xxexxxxx))))
LLOAD SCV (la.E E 9: (x))
where SCV denotes the squared coefficient of variation of given distributions over experts (also used in [46, 48]): a refers to the latency-aware coefficient of i-th expert and is defined by (Lat)/(E, Laty) runtime latency of i-th expert, Lat;; (x() is the probability that the gate value of i-th expert outweighs the others, i.e., topl expert, and is given by P (p, + Py (x), Vjj A i). Note that this probability is discrete, we use a noise proxy following [48, 23] to make it differentiable. The above latency- importance loss and lola-balanced loss help to balance gate values and workload assignments, respectively, and can be integrated with classification loss Lats as the total loss function L(X) = Las(( + A (CIne (X) + LOOAD (X) for training ViTs and gates simultaneously. À is set as 0.01 for all experiments.
because that the expected assignments are inversely proportional
5
Experiments
5.1
Experiment Settings
Models and Datasets. Tasks and Datasets. We consider two representative 2D and 3D Transformer- based vision tasks to demonstrate the superiority of the proposed ShiftAddViT, including 2D image classification ImageNet dataset [15] with 1.2 million training and 50K validation images and 3D novel view synthesis (NVS) task on Local Light Field Fusion (LLFF) dataset [40] with eight scenes. Models, For the classification task, we consider PVTv1 [60], PVTv2 [61], and DeiT [55]. For the NVS task, we consider Transformer-based GNT [53] with View- and Tay-Transformer models.
Training and Inference Details. For the classification task, we follow Ecoformer [34] to initialize the pre-trained ViTs with Multihead Sclf-Attention (MSA) weights, based on which we apply our reparameterization two-stage finetuning: (1) convert MSA to linear attention [73] and reparameterize all MatMuls with add layers with 100 epoch finetuning, and (2) reparameterize MLPs or linear layers with shift or MoE layers after finetuning another 100 epoch. Note that we follow PVTv2 [61] and Ecoformer to keep the last stage as MSA for fair comparisons. For the NVS task, we still follow the two-stage finetuning but do not convert MSA weights to linear attention to maintain the accuracy. All experiments are run on a server with eight RTX A5000 GPUs with each having 24GB GPU memory.
Baselines and Evaluation Metrics. Baselines, For the classification task, we reparameterize and compare our ShiftAddViT with PVTv1 [60], PVTv2 [61], Ecoformer [34], and their MSA variants. For the NVS task, we reparameterize top of GNT [53] and compare with both vanilla NeRF [41]

---

Table 4: Comparisons between ShiftAddViT and baselines. We show the breakdown analysis of ShiftAddViT with two kinds of Q/K K quantization. The throughput or latency is measured with a
batch size of 32 or 1, where denotes that the numbers are measured and reported after optimizing
Vdi using TVM. We report both real-device and modularized latency for models with MoE.
Linear
Add
PVTY2-B0 [61]
PVTvi-T [00)
Methods
KSH Quant.
MoE
Acc. (%)
Lal. (IIS)
(img.s)
Acc. (%)
(ms)
(img-/S)
MSA
X
X
70.77
4.62
9899
76.21
4.73
903
[61]
x
70.50
6.25
2227
75.10
5.78
1839
PVT-MoE
x
X
(MLPs)
70.82
12.46
1171
75.27
10.91
834
Ecoformer [34]
X
X
70.44
7.82
1348
NaN
7.43
1021
X
X
71.19
6.13
2066
75.50
5.78
1640
J
X
70.95
1.07|
2530
75.20
1.42
1683
ShiftAddVIT
X
(Atta)
X 70.53
1.047
24471
74.77
1.39
16477
(with KSH [34]
(Atta))
(MLP4)
70.16
1.399/1.
N/A
74.44
1.91//1.21
N/A
Quant. [27]
(Both)
70.38
1.59//1.00
NIA
74.73
2.12
N/A
to binarize Q/K)
8
X
71.36
6,34
2014
75.64
5:48
1714
71.04
1.00"
26131
75.18
1.20
1907
(Bath)
X
68.57
0.971
2736
73.47
1,18
1820'
x
X
(Both)
70.59
V1.12
NIA
74.93
1.97//1.02"
N/A
denotes the modularized latency simulated by separately optimizing each expert/router with ideal parallelism.
Table 5: Comparisons between ShiftAddViT and baselines. We apply our Shift&AAd techniques on top of SOTA Transformer-based NeRF model, GNT [53], and report averaged PSNR (f). SSIM (f) and LPIPS (1) on the LLFF dataset [40] across eight scenes. In addition, we also show the results on two representative scenes, Orchids, and Flower, and the rendering latency and energy measured on an
Eyeriss-like accelerator. More results for other scenes are available in the Appendix C.
LLFF Averaged
Orchids
Flower
Lat.
Energy
Methods
Add
Shift
MoE
PSNR
SSIM LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM LPIPS
(s)
(J)
NeRF [41]
26.50
0.811
0.250
20.36
0.641
0.321
27.40
0.827
0.219
683.6
1065
GNT [53]
27.24
0.889
0.093
20.67
27.32
0.893
0.092
1071
1849
26.85
0.874
0.116
20.74
0,730
0.182
28.02
0,891
0.089
1108
1697
/(Bath)
x
26.85
0.875
0.116
20.78
0.730
0,182
28.05
0.892
0.088
568.5
844.0
ShifAAdVIT
((kttn)
(MLPa)
26.92
0.876
0.114
20.73
0.731
0.180
28. 20
0.894
0.087
746.6
1093
/(Bath)
0.881
0.107
20.84
0.746
0.169
0.896
0.083
531.2
995.6
and GNT [53]. Evaluation Metrics, For the classification task, we evaluate the ShiftAddViT and baselines in terms of accuracy, GPU latency and throughput measured an RTX 3090 GPU. For the NVS task, we evaluate all models in terms of PSNR, SSIM [63], and LPIPS [76]. We also measure and report the energy consumption of all the above models based accelerator [12, 77], which calculates not only computational but also data movement energy.
Eyeriss-like hardware
5.2 ShiftAddViT over SOTA Baselines 2D Tasks
To evaluate the effectiveness of our proposed techniques, we apply the ShiftAddViT idea to five commonly used ViT models, including DeiT [55] and various variants of PVTv1 [60] and PVTv2 [61]. We compare their perfor- mance with baselines on the image classifica- tion task. Tab. highlights the overall compar- ison with the most competitive baseline, Eco- former [34], from which we see that ShiftAd- dviT consistently outperforms all baselines in terms of accuracy-efficieccy tradeoffs, achiev- ing 1.74x 5.18x latency reduction on GPUs and 19.4% ~ 42.9% energy savings measured on the Eyeriss accelerator [12] with comparable even better accuracy (10.04% ~ 10.20%). Note
Table 3: Overall comparison between ShiftAddViT and the most competitive baseline on five models.
Latency
Energy
Models
Methods
Acc. (%)
(ms)
(mJ)
Ecoformer [34]
70.44
7.82
33.64
PVTv2-B0
Shif'AddViT
70.59
1.51
27.13
Ecoformer [34]
NaN
7.43
93.47
PVTV1-
ShiftAddViT
74.93
1.97
72.59
Ecolormer [34]
78.38
8.02
106.2
PVTv2-B1
ShiftAddViT
78.49
2.49
85.34
Ecoformer [3:4]
81.28
15.43
198.2
PVTv2-B2
ShiftAddViT
81.32
4,83
163.9
MSA [55]
72.20
5.12
66.88
DefTT
ShiftAddvir
72.40
2.94
38.21
that we apply MoE to both linear and MLP layers. Moreover, we further provide comparisons of a variety of ShiftAddViT variants and PVT with or without multi-head self-attention (MSA) and MoE in Tab. 4 and 6. We observe that our ShiftAddViT can be seamlessly integrated with linear attention [73, 65]. simple or advanced Q K quantization [34, 27], and the proposed MoB framework, and 2.47
latency reductions and throughput improvements after the TVM optimization on an RTX 3090 as compared to MSA, PVT, and PVT+ MoE (note: two Mult. experts) basclines, respectively, under comparable accuracies, i.c.. +0.5%.
achieving up to 3.06x/4.144
7

---

Table 6: Comparisons between ShiftAddViT and baselines on PVTv2-B1 and PVTv2-B2. The
throughput or latency is measured with a batch size of 32 or 1 on an RTX 3090 GPU.
Linear
Add
PVTx2-B1 [6]]
PVTx2-B2 [61]
Methods
Atta
KSH
Quant.
Shift
MoR
Acc. (%)
Lat. (115)
(img//))
Acc. (%)
Lat. (nus)
I. (img/s)
MSA PVT [61]
78.83
4.77
821
9.08
508
/
x
x
X
78.70
6.75
1512
82,00
12.20
921
Ecoformer [34]
/
x
X
78.38
8.02
874
81,28
15.43
483
78.80
1373
81.40
9.21
796
/
/
x
X
78.70
1,701
9231
81.31
3,18
5181
ShiftAddVTT
/
x
(Attn)
(MLPs)
78.20
2.467/1.46
N/A
81.06
4.17//3.22
N/A
(with KSH [34]
/
(Both)
78.38
2.644/1448*
N/A
81.32
4.83'/3.35*
N/A
Quant. [27]
x
X
78.93
6,49
1327
81.55
11.82
782
to binarize Q/K)
8
X
78.70
1.571
9421
81.33
2.82/
5531
8
(Bath)
77.55
10211
79.97
2.931
60pt
8
(Both)
78,49
2.49//1.35
N/A
81.01
4.55
N/A
Training Wall-clock Time. To offer more insights into the training cost, we collect the training wall-clock time, which is 21 %% 25% less than training from scratch as the original Transformer model and more than 50x less than the previous ShiftAddNet [70] did on GPUs. For example, training the PVTvI-Tiny model from scratch takes 62 hours while our finetuning only needs 46 hours.
Linear Attention vs. MSA. We ensure batch size (BS) is for all latency measurements and find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA. The reasons are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWCоnv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits. To validate both points, we measure the latency of PVTv2-BB T-2-00 with various BS and input resolutions as shown in the Appendix F, the results indicate that linear attention's 's benefits show up under larger BS or input resolutions.
5.3 ShiftAddViT over SOTA Baselines on 3D Tasks
We also extend our methods to 3D NVS tasks and built ShiftAddViT on top of Transformer-based GNT models [53]. We test and compare the PSNR, SSIM, LPIPS, latency, and energy of both ShiftAddViT and baselines, i.e. vanilla NeRF [41] and GNT [53], on the LLFF dataset across eight scenes. As shown in Tab. 5, our ShiftAddViT consistently leads to better accuracy accuracy-efficiency tradeoffs, achieving 22.3% even better generation quality (10.55/10.19 averaged PSNR across eight scenes), compared to NeRF and GNT baselines, respectively. Note that GNT costs more than NeRF because of an increased number of layers. In particular, ShiftAddViT even achieves better generation quality (up to 10.88 PSNR) than GNT on some representation scenes, e.g., Orchid and Flower, as highlighted in Tab. 5, noted with corresponding colors. The
latency reductions and 20.8%/54.3%% energy savings under comparable or
where the first, second, and third ranking performance full results of eight scenes are supplied in the Appendix C. Note that we only finetune ShiftAddViT for 140K steps on top of pre-trained GNT models instead of training up 840K steps from scratch.
5.4 Ablation Studies of ShiftAddViT
Performance Breakdown Analysis. To investigate how each of the proposed techniques contributes to the final performance, we conduct ablation studies among various kinds of ShiftAddViT variants on PVTv1-T, PVTv2- 1/B2 models to gradually break down and show the advantage of each component. As shown in Tab. 4 and 6, we have three general observations: (1) ShiftAddViT is robust to the binarization method of Q/K, e.g., it is compatible with either KSH [34] or vanilla binarization [27], achieving average 3.03x latency reductions than original PVT under comparable accuracies (+0.5%), and both 3.29 x/1.32x latency/energy reductions and 0.04% ~ 0.20% accuracy improvements over Ecoformer [34]] (2) vanilla binarization methods work better than KSH in our ShiftAddViT framework in terms of both accuracy (10.05% ~ 10.21%) 9) and efficiency (on average 55.9% latency reductions and 15.8% throughput boosts). Moreover, KSH requires that Q and K are identical while vanilla binarization [27] does not have such a limitation. That explains why applying vanilla quantization to the linear attention layers results in an accuracy improvement of on average 0.15% as compared to using KSH Also note that our adopting linear attention following [73] works better (f0.29%) than PVT [60, 61]: (3) Replacing linear layers in attention modules or MLPs with Shift layers lead to an accuracy drop while our proposed MoE can help to compensate for it. Specifically, adopting Shift layers leads to on average 1.67% accuracy drop while MoE instead improves on average 1.37% accuracy. However, MoE hurts the efficiency of both the PVT baseline
8

---

0.15
0.05
Pyforch MatMuls
Latency (ms)
0.12
TVM FakeeShifts
Pytorch FakeShift
TVM MatShifts (Ours)
TVM MatMuls
Latency (ms)
0.04
TVM MatAdds (Ours)
TVM MatMuls
03
58
0.30
0.03
10
0.08
0.0
8.
0.05
29
70
20
0.01
9.
0.03
0:10
0.0
0.00
0.00
[8. K. MI
KB.K. MI
IB. C
1B. M k1
[B.
(..HK [8. 786,
S12)
5.6, 16)
Figure 4: MLP/Linear latency speedups using shifts, where
Figure 5: MatMuls latency speedups using adds, where inputs are of shape B H x K x M. weights are of shape B x H x K N.
inputs are of shape B x K x M, weights are of shape K x N. All dimensions are set w.r.t the real dimensions in PVTs [60].
and our SdiftAddViT without eustomized system acceleration, e.g., PVT+MoE increases 94% latency and reduces 51% % throughput than PVT on GPUs, due to limited parallelism supports, especially for TVM. We report the modularized latency by separately optimizing each expert demonstrate the potential of MoE's double- double-winning accuracy (t0.94% ~ f2.02%) and efficiency (115.4% ~ 142.7%).
In addition, we want to clarify that the seemingly minor latency improvements of adopting shifts in Tab. 4 and 6 are due to full optimization of the compiled model as a whole (e.g., 6.34ms + Ims for PVTv2-BO) on GPUs with sufficient chip area. Most gains are concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1 and the Appendix F. Under the same chip areas, latency savings of adopting shifts are more pertinent, e.g., PVTv2-BO with shift or MoE achieve 3.9 ~ 5.7x or 1.6 ~ 1.8x speedups.
Apart from 2D tasks, we also report the performance breakdown on 3D NVS tasks, as shown in Tab. 5. The above three ob- servations still hold except for the Shift layer, as we find that ShiftAddViT with all linear layers or MLPs replaced by Shift layers achieve even better PSNR (10.13 10.20), SSIM (70.005 10.007), and LPIPS (J0.007 (0.009) than other vari- ants. In addition, we profile on an Eyeriss Figure 3: Energy breakdown on an Eyeriss accelerator. accelerator to show the energy breakdown of both ShiftAddViT and baselines. As shown in Fig. 3, our ShiftAddViT reduced 42.9% and 40.9% energy on top of DeiT-T and GNT, respectively. Among them applying Add layers lead to 93.8% and 63.8% energy reductions than original MatMuls, Shift layers help to reduce 28.5% and 37.5% energy than previous linear/MLP layers. This set of experiments validates the effectiveness of each component in our proposed ShiftAddViT framework.
FeR MadMuls
Adds
Linears
Shifts
Others
DeiT
Saving
Ours
GNT
40. 9% Saving
100
Energy Percentage (%)
Speedups of Shifts and Adds. Apart from the overall comparison and energy reduction profiling, we also test the GPU speedups of our customized Shift and Add kernels, as shown in Fig. and 5, respectively. We can see that both customized kernels achieve faster speeds than PyTorch and TVM baselines. For example, our MatAdds achieve on average 71.4x speedups than PyTorch and TVM MatMuls, respectively. Our MatShifts achieve on average 2.35x/3.00 x/1.16x speedups than Py PoTcch takeShifs [17], TVM FakeShifts, and TVM MatMuls, respectively. More comparisons and speedup analysis under larger batch sizes are supplied in the Appendix A. Note that those comparisons, we use the default einsum operator for the Py PoTorh MatMuls, floating- point multiplication with power-of-twos for FakeShift, and implement the MatAdds and Mat.Shift using TVM [10] ourselves. We compare those customized operators with their multiplication counterparts in the same setting for fair comparisons. The speedup of MatAdds is because replace with accumulation only. The speedup of MatShifts is mainly attributed to the bit reductions (INT32 and INT8 for inputs and shift signs/weights) and thus data movement reductions instead of computations which are almost fully hidden behind data movements.
multiplication-an -at---aaadd---naaniieeeeaaaaa
MoE's Uniqueness in ShiftAddViT. Our contributions are the cnatiplicationn-eddcee ViTs and a new baldncal MoE framework. They are linked in one organic whole. Unlike previous MoE works where all experts are of the same capacity and there is no distinction between tokens, our MoE layers combine unbalanced experts, i.e., multiplication and shift, with hypothesis to divide important object tokens and background tokens. Such new MoE design offers softer version of

---

Image
Token Gate
Image
Token Gate
Image
Token Gate
Image
Token Gate
Figure 6: Visualization of the token dispatches in MoE routers/gates given images from the validation
set, where yellow color denotes Mult. experts while blue color refers to Shift experts.
ShistAddViTs,, i.e., instead of aggressively replacing all multiplications with cheaper shifts and adds, we keep the powerful multiplication option to handle importance tokens for maintaining accuracy while leaving all remaining unimportant tokens being processed by cheaper bitwise shifts, winning a better accuracy and efficiency tradeoff.
Ablation studies of Latency-aware Load-balancing Loss (LL-Loss). To demonstrate the effectiveness of the proposed LL-Loss in our ShiftAdd MoE frame- work, we conduct ablation studies on two PVT models as shown in Tab. 7. We find that ShiftAddViT w/ LL- Loss achieves better eeeuracy-efficienne tradeoffs, e.g., 14.6% latency reductions while maintaining comparable accuracies (+0.1% 1%) when considering Mult. and Shift experts. The latency reduction could be larger if we consider more unbalanced experts with distinct runtimes. This set of experiments justifies the effectiveness of our L.L.-Loss in the MoE system.
Table 7: Ablation studies of the ShiftAd-
dVIT w/ w/o LL-Loss PVT models.
Norm.
Models
Methods
Acc. (%)
Latency
w/o LL-Loss
70.38
100%
PVTv2-B0
w/ LL-Loss
70.37
85.4%
w/o LL-Loss
74.73
100%
PVTv1-T
w/ LLLLLsss
74.66
85.5%
Visualization of Token Dispatches MoE. To validate our hypothesis that important yet sensitive tokens require powerful experts while other insensitive tokens can be represented with much cheaper Shift experts, we visualize the token dispatches in MoE routers/gates of the first MLP layer in our ShiftAddViT PVTv2-BO model as shown in Fig. 6. We see that our designed router successfully identifies the important tokens that contain object information, which are then dispatched to more powerful Mul tolltipliaati tokens. This set of visualization further validates the hypothesis and explains why our proposed MoE framework can effectively boost our ShiftAddViT towards better rccuracy-efficeeng trade-offs.
experts, leaving unimportant tokens, e.g., background, cheaper Shift
5.5 Limitation and Societal Impact Discussion
We made firm step show the practical usage of the proposed serdware-inspiree ShiftAddViT. While this relies on dedicated TVM optimization, the full potential can be unraveled with cus- tomized hardware accelerators toward naturally hardware-efficient ViTs. Also, the unbalanced MoE framework shows its generalizability but highly demands system support with ideal parallelism.
6 Conclusion
In this paper, we for the first time propose a hardware-inspired multiplication-redueel ViT model dubbed ShiftAddViT. It reparameterizes both attention and MLP layers in pre-trained ViTs with mixture of multiplication primitives, e.g., bitwise shifts and adds, towards end-to-end speedups on GPUs and dedicated hardware accelerators without the need for training from scratch. Moreover, novel mixture unbalanced experts framework equipped with a new latency-aware load boad-balancing loss is proposed in pursuit of double-winning accuracy and hardware efficiency. We use multiplication or its primitives as experts our ShiftAddViT cases. Extensive experiments on both 2D and 3D Transformer-be based vision tasks consistently validate the superiority of our proposed ShiftAdd as compared to multiple ViT baselines. We believe that this paper opens up new perspective on designing energy-efficien ViT inference based on widely available pre-trained ViT models.
A Acknowledgement
The work is supported in part by the National Science Foundation (NSF) through the RTML program (Award number: 1937592) and CoCoSys, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.
10

---

References
[1] Alaacidin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeck, al. Xcit: Cross-covariance image trans- formers. Advances in neural information processing systems, 34:20014-20027, 2021.
[2] Moab Arar, Ariel Shamir, and Amit H Bermano. Learned queries for efficient local attention. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pages 10841-10852, 2022.
[3] Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann. Token merging: Your vit but faster. arXiv preprint ar axXivv2210.04461.. 2022.
[4] Daniel Bolya, Cheng-Yane Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. arXiv preprint arXiv:2209 07444, 2022.
[5] Han Cai, Chuang Gan, and Song Han. Efficientvit; Enhanced linear attention for high-resolution low- computation visual recognition. arXiv preprint arXiv:2205.1477 2022.
Beidi Chen, Tri Dao, Fric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying
sparse and low-rank attention. Advances Neural Information Processing Systems, 34:17413-17426. 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.1488 2021.
[8] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 1468-1477, 2020.
[9] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoforner: Searching transformers for visual recognition. In Procoedings of the IEEE/CVF International Conference Computer Vision, pages 12270-12280, 2021.
[10] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Coze, al. Tvm: An automated end-to-end optimizing compiler for deep 2018.
learning. arXiv preprint
[11] Xinghao Chen, Chang Xu, Minjing Dong. Chunjing Xu, and Yunhe Wang. An empirical study of adder neural networks for object detection. Advances Neural Information Processing Systems, 34:6894-6905, 2021.
[12] Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energy- efffgeeefficient recontigurable accelerator for deep convolutional neural networks. IEEE journal of solid-state circuits, 52(1):127 138, 2016.
[13] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song. Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[14] Matthicu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Haniv,, and Yoshua Bengio, Binarized neural networks: Training deep neural networks with weights and activations constrained to+ I or-1. arXiv preprint 1rXiv:1602.02333 2016.
[15] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[16] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Ncil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[17] Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Heary Tian, and Joey Yiwei Li. Deepshift: Towards multiplication-les less neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2359-2368, 2021.
[18] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the JEEE/CVF International Conference Computer Vision, pages 6824 6835, 2021.

---

[19] Hongxiang Fan, Thomas Chau, Stylianos I Venieris, Royson Lee, Alexandros Kouris, Wayne Luk. Nicholas D Lane, and Mohamed S Abdelfattah. Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design. 2022 55th JEEE/ACM International Symposium on Microarchitecture (MICRO), pages 599-615. IEEE, 2022.
[20] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal Machine Learning Research, 23(1):5232-527)) 2022.
[21] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint axXiv2204.01133 2021.
[22] Bahe Gwee, Joseph Chang, Yiqiong Shi, Chinn- Chua, and Kwen- wien-Siong Chong. low- voltage micropower asynchronous multiplier with shift add multiplication approach. IEEE Transactions on Circuits and Systems I: Regular Papers, 56(7): 1697):1349-1359 2008
[23] Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang. Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of- expert training system. arXiv preprint arXiv: 113.:22003.1322, 2021.
[24] Jiaao He, Jidong Zhai. Tiago Antunes. Haojie Wang. Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 120-134, 2022.
[25] Yefei He, Zhenyu Lou, Luoming Zhang, Weijia Wu, Bohan Zhuang, and Hong Zhou. Bivit: Extremely compressed binary vision transformer. arXiv preprint arXiv:2211.0000 2022.
[26] Byeongho Heo. Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In International Conference on Computer Vision (ICCV), 2021.
[27] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengin. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):68696689) 2017.
[28] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 2704-2713, 2018.
[29] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local binary convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 19-28, 2017.
[30] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autorcgressive transformers with linear attention. In International Conferenee Machine Learning, pages 5156 5165. PMLR, 2020.
[31] Wenshuo Li, Xinghao Chen, Jinyu Bai, Xuefei Ning, and Yunhe Wang. Searching for energy-efficient hybrid adder-convolution neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1943-1952, 2022.
[32] Yanghao Li. Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEFF/CVP Conference on Computer Vision and Pattern Recognition, pages 4804- 4814, 2022.
[33] Xiaocong Lian, Zhenyu Liu, Zhourui Song, Jiwu Dai, Wei Zhou, and Xiangyang Ji. High-performance fpga-based cnn accelerator with block-floating-poin arithmetic. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 27(8): 1874)):874-111 1885, 2019.
[34] Jing Liu, Zizheng Pan, Haoyu Hc, Jianfei Cai, and Bohan Zhuang. Ecoformer: Energy-saving attention with linear complexity. In NeurIPS, 2022.
[35] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[36] Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad, Bit: Robustly binarized multi distilld transformer. arXiv preprint 13006 2022.
12

---

[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[38] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang. and Li Zhang. Soft: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297-21309) 2021.
[39] Sachin Mchta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendee vision transformer. arXiv preprint arXiv:2110.02178, 2021.
[40] Ben Mildenhall. Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines, ACM Transactions on Graphics (TOG), 38(4):1-14) 2019.
[41] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-006, 2021.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perffrmance deep learning library. Advances in neural information processing systems, 32, 2019.
[43] Zheng Qu, Liu Liu, Fengbin Tu, Zhaodong Chen, Yufei Ding, and Yuan Xie. Dota: detect and omit weak attentions for scalable transformer acceleration. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 14-26, 2022.
[44] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next- generation ai scale. In International Conference Machine Learning, pages 18332 18346. PMLR, 2022.
[45] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems. 1937711193, 2021.
[46] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 14:8583-8595, 2021.
[47] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4510-4520, 2018
[48] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated gated mixture-of-experts layer. arXiv preprint arXiv:1701.0653 2017.
[49] Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. Nimble: Efficiently compiling dynamic neural networks for model inference. Proceedings of Machine Learning and Systems, 3:208-222, 2021.
[50] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with lincar complexities. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 3531-3539, 2021.
[51] Han Shu, Jiahao Wang. Hanting Chen, Lin Li, Yujiu Yang, and Yunhe Wang. Adder attention for vision transformer. Advances in Neural Information Processing Systems, 14:19899-19900 19909, 2021.
[52] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. ICCV, 2017.
[53] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations, 2023.
[54] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint Stii::2111.00000 2020.
13

---

[55] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 11347-10354 PMLR, 18 24 Jul 2021.
[56] Zhengzhong Tu, Hossein Talebi, Han Zhang. Feng Yang. Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel. October 27. 2022, Proceedings. Part XXIV, pages 459-479. Springer, 2022.
[57]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
[58] Guangting Wang. Yucheng Zhao, Chuanxin Tang, Chong Luo, and Weajun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 2423-2430, 2022.
[59] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
[60] Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backhone for dense prediction without convolutions. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 568-578, October 2021.
[61] Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 833):415-424, 2022.
[62] Yunhe Wang, Minggiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2101.100 10015, 2021.
[63] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[64] Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9127-9135.
2018.
[65] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström- algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138- 14148. 2021.
[66] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing XU, and Yunhe Wang- Kernel based progressive distillation for adder neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 12322-12333 12333. Curran Associates, Inc., 2020.
[67] Ping Xue and Bede Liu, Adaptive equalizer using bin power-of-two quantizer. IEEE transactions on acoustics, speech. and signal processing. 34(6):1603 1611, 1986.
[68] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809-10818, 2022.
[69] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight- straight-through estimator in training activation quantized neural nets. arXiv preprint 190.5.190.. 2019.
[70] Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li. Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. Advances in Neural Information Processing Systems, 33:2771 12783, 2020.
[71] Haoran You. Baopu Li, Shi Huihong, Yonggan Fu, and Yingyan Lin. Shiftaddnas: Hardware- insppire search for more accurate and efficient neural networks. In International Conference on Machine Learning. pages 25566-25580. PMLR, 2022.
14

---

[72] Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu. Yang Zhao. Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator design. In The 29th IEEE International Symposium on Heigh-Performacce Computer Architecture (HPPA 2023), 2023.
[73] Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang. Haoqi Fan, Peter Vajda, and Yingyan Lin. Castling-vit: Compressing self-attention via switching towards angular attention during vision transformer inference. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023.
[74] Zhongzhi Yu, Yonggan Fu, Sicheng Li, Chaojian Li, and Yingyan Lin. Mia-former: efficient and robust vision transformers via multi-graines input-adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 8962-8970, 2022.
[75] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to- to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[76] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metrie. In Proceedings of the IEEE conference on computer yision and pattern recogmition, pages 586-595, 2018.
[77] Yang Zhao, Chaojian Li, Yue Wang. Pengfei Xu, Yongan Zhang, and Yingyan Lin. Dnn-chip predictor: analytical performance predictor for dnn accelerators with various dataflows and hardware architectures. In International Conference on Acoustics, Speech, and Signal Processing, pages 1593-1597, 05 2020.
[78] Zhe Zhou, Junlin Liu, Zhenyu Gu, and Guangyu Sun. Energon: Toward efficient acceleration of transform- using dynamic sparse attention. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(1):136-149, 2022.
15

---

SaW TVM Sakeshiit
TVW Matk
TVM MatShift (Ours)
PyTorch ManMul
2.5
MatMul
Latency (ms)
Latency (ms)
TVM MarAdd (Ours)
2.0
0.10
1.5
0.05
0.5
0.03
K.
K.K.
NJ
h. 32.
H.K. M.
0.. 3258, 92,
3138, 512] 12, 784, (22, 3135, 54 126]
Figure 7: MLP/Linear latency speedups using shifts, where imputs are shape All dimensions are set w.r.t. the real dimensions PVTs [60].
Figure 8: KatMuls latency speedups using adds, where inputs are of shape Bx H XxM, M, weights are of shape B x H x
M. weights are shape N.
A More Speedup Comparisons of Shift and Add
Apart from the speedup analysis when the batch size is one, we also conducted GPU speedup tests on our customized Shift and Add kernels when using larger batch sizes, such 32. as depicted in Fig. 7 and 8, respectively. The results demonstrate that both our customized kernels consistently outperform the PyTorch and TVM baselines in terms speed. For instance, our MatAdds achieve an average speedup of 2.80 x/1.55x compared to Py PyTocch and TVM MatMuls, respectively. Similarly, our MatShifts achieve an average speedup of 3.38 x /1.21 X compared to TVM FakeShifts [17] and TVM MatMuls, respectively. It is important to note that in these comparisons, we ensured fair conditions by using the default einsum operator for PyTorch MatMuls, employing floating-point multiplication with power-of-twos for FakeShift, and implementing the MatAdds and MatShi ts ourselves using TVM [10]. We compared these customized operators with their multiplication counterparts in the same setup to ensure fair evaluation. The observed speedup of MatAdds is due to the replacement of
with accumulation only. On the other hand, the speedup of MatShifts is primarily attributed to the reduction in bit precision (INT32 and INT8 for inputs and shift signs/weights respectively), resulting in reduced data movement instead of computations. These computational improvements almost entirely hidden behind data movements.
B More Visualization of ShiftAddViT MoE Framework
In order to validate our hypothesis that important yet sensitive tokens require powerful experts, and other insensitive tokens can be represented by cheaper Shift experts, we have conducted additional token dispatch visualizations in the MoE routers/gates of the first MLP layer in our ShiftAdd PVTv2-BO model, as illustrated in Fig. 9. These visualizations provide further evidence that our designed routers successfully identify the crucial tokens that contain object information. Consequently, these tokens are dispatched to more powerful Mult. experts, while less important tokens, such as background tokens, are assigned to cheaper Shift experts. This collection of visualizations further reinforces our hypothesis and elucidates why our proposed MoE framework effectively enhances the accuracy-effiie cccuracy-effccenccy trade-offs of our ShiftAddViT model.
C More Results on 3D Tasks
We have extended our methods to 3D NVS tasks and have built ShiftAddViT on top of Transformer- based GNT models [53]. In the main paper, we tested and reported the averaged PSNR, SSIM, LPIPS, as well as latency and energy of both ShiftAddViT and baselines, i.c., the vanilla NeRF [41] and GNT [53], the LLFF dataset across eight scenes. Here, we further supply the full results of eight scenes. As shown in Tab. 8, 9, 10, our ShiftAddViT consistently leads to better accuracy- efficiency tradeoff's in terms of the three metrics mentioned above, achieving comparable or even better generation quality (10.55/L00.1 19 averaged PSNR across eight scenes), as compared to NeRF and GNT baselines, respectively. In particular, ShiftAddViT achieves better generation quality (up to 10.88 PSNR) than GNT on some representation scenes, e.g.. Orchid and Flower. In all tables, the first, second, and third ranking performances are noted with corresponding colors.
In addition, we supply more qualitative rendering visualization on three scenes, Horns, Orchids, and Flowers, shown in Fig. 10. We see that our ShiftAddViT generates more clear images than vanilla
16

---

Image
Token Gate
Image
Token Gate
Image
Token Gate
Image
Token Gate
Figure Visualization of the token dispatches in MoE routers/gates given images from the validation set, where yellow color denotes Multiplication ication experts and blue color refers to Shift experts.
Horns
Flowers
Orchids
(a) NeRF
(b) GNT
(c) Ours
(d) Ground Truth
Figure 10: Qualitative comparisons between (a) NeRF [41], (b) GNT [53], (c) ShiftAddViT (Ours),
and (d) ground truth when tested on the LLFF dataset for single-scene -cenee rendering.
NeRF [41] and comparable consistently show that our ShiftAddViT does not hurt rendering quality when improving rendering speeds and energy efficiency.
better details than GNT [53].
Quantitative and qualitative results
More Results on NLP Tasks
We also test our proposed optimization MHSA and MLPs NLP tasks. particular, we apply our methods to Transformer models and Long Range Arena (LRA) benchmarks consisting of sequences ranging from IK to 16K tokens [54]. The results are shown in Tab. 11. We see that our proposed ShiftAdd Transformers consistently show superior performance in terms of both model accuracy (+0.45% ~ +5.79 -5.79% and efficiency (1.5x 11.5x latency reductions and 2.2x 16.4 energy reductions an Eyeriss-lik -ike accelerator) compared to original Transformer and other linear
17

---

Table Comparisons between ShiftAddViT and baselines. We apply our Shift& Add techniques on top of the SOTA Transformer- based NeRF model, GNT [53], and report averaged PSNR (f) on the
LLFF dataset [40] across eight scenes.
PSNR (f)
LLFF Dataset
Methods
Add
Shift
MoE
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
NeRF [41] GNT [53]
32.70
25.17
20.92
31.16
20.36
27.40
26.80
27.45
32.96
24.31
22.57
32.28
20.67
27.32
28.15
29.62
31.68
24.63
22.34
31.79
20.74
28.02
26.99
28.62
((oth))
x
31.64
24.63
22.34
31.73
20.78
28.05
27.03
28.61
ShiRAAdViT
((Attn)
/(MLPs)
31.85
22.37
31.85
20.73
28.20
27.08
28.62
((Bohh)
32.12
24.35
22.42
32.15
20.84
18.14
27.32
29.09
Table 9: Comparisons between ShiftAddViT and baselines. We apply Shift&Add techniques on top of the SOTA Transformer-basel NeRF model, GNT [53], and report averaged SSIM (f) on the LLFF dataset [40] across eight scenes.
SSIM (1) on LLFF Dataset
Methods
Add
Shift
MoE
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
NeRF [41] GNT [53]
0.948
0.690
0.641
01827
0.828
0.963
0.846
0.852
0.934
0.752
0.893
0.936
0.935
0,949
0.839
0.838
0,922
0.730
0.891
0.916
0,911
((Both)
0,948
0.839
0.839
0,923
0.730
0.892
0.917
0.911
ShiftAdViT
((Attn)
/(LPPs)
0,950
0.840
0.841
0.923
0.731
0.894
0.917
0.912
X
((Both)
x
0,955
0.836
0.841
0,930
0.746
0.896
0.923
0.922
Table 10: Comparisons between ShiftAddViT and baselines. We apply our Shift& techniques on
top of the SOTA Transformer- based NeRF model, GNT [53], and report averaged LPIPS (L) on the LLFF dataset [40] across eight scenes.
LPIPS (L) on LLFF Dataset
Methods
Add
Shift
MoE
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
NeRF [41] GNT [53]
0.178
0.280
0.316
0.171
0.321
0.219
0.249
0.268
0,060
0,116
0.109
0,061
0.153
0,092
0.080
0.076
X
0,087
0,141
0.129
0,080
0.182
0,089
0.107
0.110
((Bohh)
x
0,088
0,142
0.128
0,080
0.182
0,088
0.106
0,110
ShiftAddViT
((Attn)
(MLPaa x
0,086
0.140
0.127
0.079
0.180
0.087
0.104
0.109
X
(Bath)
0.076
0.138
0.125
0.070
0.169
0.083
0.097
0.097
attention baselines, which means that our shift and add reparameterizatiio and baad-ealncced MoE ideas are generally applicable to Transformer models and are agnostic to domains and tasks.
E More Experiment Settings
For the classification task, we first replace the original attention layers in PVTv1 [60] and PVTv2
[61] with the standard MSA, and then finetune the MSA-based variants on top of the ImageNet-1k-
pretrained weights for 100 epochs with learning rate (Ir) of 5 x 10 following Ecoformer [34].
After that, we reparameterize the MSA-based models through two-stage finetuning process: (1)
convert MSA to linear attention [73] and reparameterize MatMuls with add layers via 100 epochs of
finetuning with a base Ir of 1 10 5, and (2) reparameterize MLPs linear layers with shift or MoE
layers through another 100 epochs of finetuning with a base Ir of 1 x 10-5, All models are trained using eight RTX A5000 GPUs with total batch size of 256, and we use the AdamW optimizer [37] with cosine decay Ir scheduler. All other hyperparameters are the same those in Ecoformer [34] and PVTv2 [61]. For the NVS task, we apply reparameterization on top of the pretrained models from GNT [53] using similar two-stage fine-tuning process, except that we do not convert MSA to linear attention for maintaining accuracy. Specifically, we first (1) reparameterize MatMuls with add layers and then (2) reparameterize MLPs or linear layers with shift MoE layers. Both stages are finetuned for 140K steps with a base Ir of 5 x 10 - and we sample 2048 rays with 192 coarse points sampled per ray in each iteration. All other hyperparameters are the same those in GNT [53], including the use of the Adam optimizer with an exponential decay Ir scheduler.
Scaling Factor Clarification. For shift layers, we reparameterize shifts following DeepShift-PS [17] and not use scaling factor. A straight-through estimator (STE) [69] is adopted to make the shift layer differentiable. For Shi with KSH [34], there is no scaling factor needed set of
18

---

Table 11: Comparisons with proposed ShiftAdd Transformer with baselines Long Range Arena (LRA) benchmarks. Latency and energy are measured on an Eyeriss-like like accelerator with batch size of 1.
Models
Accuracy (%)
Averuge Latency (ms)
Average
Text (4K)
Listops (2K)
Retricval (4K)
Image (IK)
Average
Energy (mJ)
Transtormer
65.02
37.10
79.35
38.20
54.92
84,34
139.83
Reformer
64.88
19.05
78.64
43.29
51.47
11.19
19.04
Linformer
55.91
37.25
79.37
37.84
52.59
12.13
19.68
Performer
63.81
18.80
78.62
37.07
49.58
11.93
18.74
ShifLAdd- Transformer
37.15
82.02
Table 12: Latency comparisons of PVTv2-BO with different attention types when measured under various batch sizes (BS) and input resolutions on an RTX 3090 GPU. "Linear SRA" " donates the linear spatial-reduction attention in PVTv2.
Pytorch Latency (ms))
Attention Types
Input Resolution = [224, 224)
Input Resolution = (448, 448)
B
BS=2
BS
B8=8
BS= 16
BS= 32
BS
BS=1
BS=2
BS=
BS
8
8.48
16.61
32.49
64.31
16.20
126.81
PVTv2 (Lincar SRA)
6.25
6.33
6.73
6.75
6.79
11.68
22.22
6.36
6.40
6.84
12.96
Atlention
6.75
13.56
14.02
hash functions is applied convert Q/K to binary codes. For ShiftAddViT with Quant. [27], leverage layer-wise Quant. for both Q & K, the scaling factor can be multiplied after add operations. It be efficiently implemented following [28].
F More Ablation Studies
Linear Attention vs. MSA under Various Batch Sizes and Input Resolutions. We find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA in the Tab. 4 and 6. To investigate the hidden reasons, we further measure the latency of PVTv2-BO with various BS and input resolutions, the results are summarized in the Tab. 12. From that, we observe that linear attention's 's benefits show up under larger BS or input resolutions. The hidden reasons for this phenomenon are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWCony, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits.
The Advantage of Using Shift. Replacing Multiplication based MLPs with shift layers signifi- cantly reduces latency, energy, and chip area costs. Our shift kernel offers an average speedup of 2.35x/3.07 x/1.7%1111111 16x compared PyTorch FakeShift, TVM FakeShift, and TVM MatMuls, respec- tively, as shown in Fig. 3. The seemingly minor latency improvements in Tab. 4 and 6 are due to the compiled model as whole (e.g., 6.34ms + ms for PVTv2-BO) on GPUs with sufficient chip area. Most gains concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1. Under the same chip areas, latency savings are more pertinent, e.g., PVTv2-BO w/ shift or MoE achieve 3.9 ~ 5.7x x and 1.6 1.8x speedups, respectively, as summarized Tab. 13.
full optimization
Table 13: Comparisons of our ShiftAddViT with baselines in terms of task accuracy, latency measured with batch size of on an RTX 3090 GPU, and latency profiled on the Eyeriss accelerator under the same hardware area constraint.
Add
Allention
Shift
Accuracy (%)
GPU Latency (ms))
(Under the Same Chip Area)
Eyeriss Latency
PVTv2-B0
N0.77
4.62 (Pstorohi
60.50
ShinaddViT (PVTV--B0)
/im MLP)
68.57
71.04
0.97
L.00
15.37
2.77
/(Alnn MLP)
70.59
1.51 L.12
9,77
PVTvZ.BI
X
78.33
4.77 (PyTorch)
147.77
ShinAddyi
78.70
1.57
38.64
(PVTv2-B1)
AiAlt MLP)
77.55
1.54
12.42
/1Att & MLP)
78.49
2.49 1.35*
33.71
19


Reviews:
[{"review_id": "RowLZIe4WS", "reviewer": "Reviewer_FYmk", "summary": "This work proposes ShiftAddViT, which is an efficient ViT reparameterization with a mixture of complementary multiplication primitives, such as bitwise shifts and adds. \nThe alternative parameterization (quantization) is carefully examined to allocate to different components (MHSA, MLP) in ViT. \nIn addition, the authors propose a mixture of experts (MoE) framework to classify input tokens and assign different primitives to best preserve accuracy. \nThe MoE framework is guided by latency-aware load-balancing loss. \nMultiple empirical results are demonstrated including 2D ViT in image classification task, as well as GNT for NVS task. ", "questions": "Please refer to concerns raised in weaknesses. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow. \n2. Though shift and add reparameterization is not completely new (binary, ternary or power-of-2 quantization), this is a very pioneer work to systematically investigate layer sensitivity and accuracy impact, allocation strategy, and hardware implementations. The hardware benchmarks with considerable latency and energy savings are impressive. \n3. The experimental analysis is strong. In addition to conventional classification task, the authors also provide results on NVS task. ", "weaknesses": "1. Since this work proposes a collection of analysis and optimizations on MHSA and MLP, reducing the computation complexity and energy consumption, I wonder if this work is portable to NLP tasks, especially for LLMs with more tokens? \n2. I cannot fully follow the token dispatching method mentioned in Section 4.2 and Figure 6. The authors did not discuss the methodology for input allocation and parallel computing issues in detail (line#227). Intuitively, if the important and sensitive tokens (yellow) in Figure 6 are fed into powerful experts (MULT.), while the rest are into SHIFT experts, are they still visible to each other (i.e. global receptive field)? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive comments and constructive suggestions. Below are our detailed responses to your concerns.\n\n**W1: I wonder if this work is portable to NLP tasks, especially for LLMs with more tokens?**\n\nWe follow your suggestion to test our proposed optimization of MHSA and MLPs to NLP tasks. In particular, we apply our methods to Transformer models and Long Range Arena (LRA) benchmarks consisting of sequences ranging from 1K to 16K tokens [1]. The results are shown in the table below as well as in ***Table 1 of the attached PDF in our global response***.\n\n| Models | Listops (2K) | Retrieval (4K) | Text (4K) | Image (1K) | Average Accuracy | Latency (ms) | Energy (mJ) |\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| Transformer | 37.10 | 79.35 | 65.02 | 38.20 | 54.92 | 84.54 | 139.83 |\n| Reformer | 19.05 | 78.64 | 64.88 | 43.29 | 51.47 | 11.19 | 19.04 |\n| Linformer | 37.25 | 79.37 | 55.91 | 37.48 | 52.59 | 12.13 | 19.68 |\n| Performer | 18.80 | 78.62 | 63.81 | 37.07 | 49.58 | 11.93 | 18.74 |\n| **ShiftAdd-Transformer** | 37.15 | 82.02 | 66.69 | 35.62 | **55.37** | **7.38** | **8.53** |\n\nThe results consistently show the superior performance of our proposed ShiftAdd Transformers in terms of both model accuracy (+0.45% ~ +5.79%) and efficiency (1.5x ~ 11.5x latency reductions and 2.2x ~ 16.4x energy reductions on an Eyeriss-like accelerator) as compared to original Transformer and other linear attention baselines, which means that our shift and add reparameterization and load-balanced MoE ideas are generally applicable to Transformer models and are agnostic to domains and tasks. We thank the reviewer for this comment as this set of experiments also enlarges the impact of our work.\n\n[1] Long Range Arena: A Benchmark for Efficient Transformers, ICLR 2021\n\n---\n\n**W2: How to understand the dispatching method in Section 4.2 and Figure 6? Discuss the methodology for input allocation and parallel computing issues in detail (line#227), are input tokens of multiplication experts and shift experts still visible to each other (i.e., global receptive field)?**\n\nThanks for pointing out this question, we will clarify each point one by one.\n\n***How to understand the dispatching.*** As for the dispatching method, our hypothesis is that important and sensitive tokens are expected to be handled by powerful multiplication experts while the rest are into cheaper shift experts as you also pointed out. The trainable router within the MoE layers automatically learns this dispatch assignment as the gradients decay and the loss converges to minima in loss landscapes. This learning process is guided by our proposed latency-aware load-balancing training loss function (integrated with classification loss) in Section 4.2. Figure 6 visualizes the actual learned dispatching pattern in our practically trained model to verify our hypothesis.\n\n***Elaborate input allocation and parallel computing.*** Sorry for not fully expanding it due to the limited space. We elaborate on these two points below.\n\n1. *For input allocation*, it is determined dynamically during runtime, we know the allocation only when the model is executed and we receive the router outputs. Therefore, the shape of expert input and corresponding indexes are dynamically changed. This can be handled by PyTorch with dynamic graphs while TVM expects static input shape. That is why we leverage the compiler support for dynamism as proposed in Nimble [2] on top of the Apache TVM to handle the dynamic input allocation. \n\n2. *For parallel computing*, it means that different experts are run in parallel, this can be supported by several customized distributed training frameworks integrated with PyTorch, e.g., FasterMoE [3], and DeepSpeed [4]. In contrast, it remains nontrivial to support this in the TVM community. One option to simulate is to perform modularized optimization to mimic parallel computing, in particular, we optimize each expert separately and measure their latency, the maximum latency among all experts will be recorded and regarded as the latency of this MoE layer, and the aggregated total of the time spent for each layer is the final reported modularized model latency. To avoid any potential confusion between real-measured wall-clock time, i.e., no parallelism assumed, and simulated modularized latency, i.e., ideal parallelism assumed, we reported both for models containing MoE layers as shown in Tables 4 and 6 to offer more insights into the algorithm’s feasibility and cost implications.\n\n***Maintain global receptive field.*** Good point, the input tokens of different experts are still visible to each other because we only split tokens for MLP layers while keeping the attention mechanism untouched, so that all tokens still attend to each other in attention layers to ensure the global receptive field. In addition, for PVTv2 models, there are depthwise convolutions in the middle of two MLP layers to exchange information between two experts’ outputs.\n\nWe appreciate the reviewer for raising the potential confusion readers may have and will clarify all these points in the final revised manuscript.\n\n[2] Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference, MLSys 2021\n\n[3] FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models, ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming 2022\n\n[4] DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale, ICML 2022\n"}, {"review_id": "DyylYU7dtK", "reviewer": "Reviewer_hFZt", "summary": "This paper introduces a novel reparameterization method for efficient Vision Transformers (ViT). The method replaces the heavy multiplication operations in ViT with a combination of shift and add operations. By mapping queries and keys to binary codes in Hamming space and reparameterizing multi-layer perceptrons (MLPs) or linear layers, the multiplication operations in the model are effectively reduced. Experimental results demonstrate that ShiftAddViT achieves efficient performance on various 2D/3D Transformer visual tasks while achieving latency reduction and energy savings.", "questions": "Overall, I am satisfied with the results and motivation.   I am particularly concerned about whether this method can improve inference performance for larger models. The bottleneck for small models is not particularly high, to begin with, and compared to larger models, the performance of small models will undoubtedly be lower. Therefore, if the authors' method can enable larger models to achieve a significant increase in inference speed, it would greatly enhance the significance of the approach.", "limitations": "The author discussed the limitation in their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a clear motivation for each proposed compound. For example, the reparameterization of multiplications is inspired by hardware design practices and the concept of multiple experts (MOE) for dynamically routing tokens into multiplication or shift groups. The three questions posed by the authors clearly indicate the specific problems they are addressing. The writing and demonstration are clear and concise. In terms of results, this work delivers some impressive findings in regard to both efficiency and accuracy. Extensive ablation experiments are conducted to validate the proposed approach further.", "weaknesses": "1. A primary concern is the limited validation of the method proposed by the authors, which is solely performed on small-scale models, with the largest model encompassing 30M and 4G FLOPs. I am particularly interested in the performance of larger-scale models. Naturally, I acknowledge the constraints imposed by computational resources. However, considering the hardware specifications disclosed by the authors, it is apparent that training models of ViT-Base size or similar are feasible.\n\n2. Secondly, my concern lies in the stability of the training process and the associated wall-clock time. Obtaining information regarding the overall training cost and wall-clock time from the authors would offer valuable insights into the algorithm's feasibility and cost implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns.\n\n**W1: Most experiments are small-scale models (largest: 30M parameter and 4G FLOPs), how about the performance of larger-scale models like ViT-Base size or similar?**\n\nThank you for your suggestion. As you advised, we further examine our proposed method on large-scale models, including PVTv2-B5 (Params: 82M; FLOPs: 11.8G) and DeiT-Base (Params: 86M; FLOPs: 17.6G) on ImageNet. The results are shown in the table below as well as in ***Table 6 of the attached PDF in our global response***. We can see that the proposed ShiftAddViT consistently performs better in terms of accuracy-efficiency tradeoffs, achieving 18.4% ~ 65.7% and 28.9% ~ 70.3% latency/energy reduction with comparable accuracy ($\\pm$0.5%).\n\n| Models | Methods | Accuracy (%) | Latency (ms) | Energy (mJ) |\n|---|---|:---:|:---:|:---:|\n| DeiT-Base | Linear Attention | 83.1 | 8.43 | 625.74 |\n| DeiT-Base | **ShiftAddViT** | 82.9 | 6.88 | 185.80 |\n| PVTv2-B5 | Linear SRA | 83.8 | 39.80 | 482.94 |\n| PVTv2-B5 | **ShiftAddViT** | 83.3 | 13.66 | 343.37 |\n\nAs we mainly target mobile and small-to-medium model scale scenarios previously, this new set of large model experiments further validates the scalability and potential of our ShiftAddViT on large model settings. We will incorporate the new results into the revised manuscript.\n\n---\n\n**W2: Stability of the training process and the associated wall-clock time?**\n\nThe process of training or finetuning after reparameterizing Transformer models using shifts and adds demonstrates stability. We conducted experiments involving two-step finetuning. In the initial step, we transformed Multi-Head Self Attention (MSA) into linear attention and reparameterized all Matrix Multiplications (MatMuls) with additive layers. This was followed by finetuning to restore accuracy. In the subsequent step, we reparameterized MLPs or linear layers using shift or MoE layers, again finetuning for accuracy recovery. \n\nAs suggested, we collect the training or finetuning wall-clock time, which is 21% ~ 25% less than training from scratch as the original Transformer model and >50x less than the previous ShiftAddNet [1] did on GPUs. For example, training the PVTv1-Tiny model from scratch takes 62 hours while our finetuning only needs 46 hours. We will add this information to the final revised manuscript to offer more insights into the training cost implications.\n\nIn addition, we would like to clarify that our main goal is efficient inference and deployment, as validated by the reported real-measured inference wall-clock times on RTX 3090 GPU in the submitted manuscript. The training or finetuning cost savings are by-products as we support to reparametrize ShiftAddViTs based on the weights from already available pre-trained checkpoints of original ViTs.\n\n[1] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS 2020"}, {"review_id": "1e8B2vOVfj", "reviewer": "Reviewer_MmZd", "summary": "This paper proposes a new type of multiplication-reduced model ShiftAddViT, which use the additive kernels to reparameterize the batched GEMM in the attention block and uses the shift kernels to reparameterize other MLPs or linear layers. In this way, it can reduce energy-intensive multiplications. The authors utilize TVM to implement and optimize those kernels and achieve a speedup on GPUs, and also energy savings.", "questions": "About shift and add:\n- I think the Shift method and the power-of-two quantization should be similar. But there is no scaling factor in the Shift operation (equation 3). In this way, how to convert weights in pre-trained models to the shift layers format? Is the nearest neighbor method directly adopted? Wouldn't the reconstruction error be too large without a scaling factor?\n- How many bits of the P is used in the shift layers? And during finetuning, is the STE method used to solve the gradient backpropagation for s and P?\n- Both two binary methods KSH[32] and Quant[26] use the scaling factors, but those scaling factors are not shown in Figure 1. During inference, where to multiply those scaling factors? Just after the add operation?\n- How many bits are used for the input activations of the ShiftAddViT attention? Since only the fixed-point activations can be shifted, the input activation of shift layers needs to be quantized to an integer first. However, it seems that the article does not mention anything related to activation quantization. And there is no step related to the activation quantization in the two-stage finetuning process. In addition, if the activations are quantized, under the shift&add paradigm, where to multiply the scaling factor from activation quantization? \n- How to quantize the input of the last Shift in ShiftAddViT Attention to integer values? Those input activations should also be fixed-point numbers, not floating-point.\n\nAbout the MoE framework:\n- In line 223, how to get the gate value $p_i$ based on the input token $x$? $p_i(x):= e^{p_i}/\\Sigma_j^ne^{p_j}$ ? Is this a typo?\n\nAbout the experiments:\n- In Table 4, why the latency of PVT is larger than MHA? Is it because of the different batch sizes? For a fair comparison, it is better to use the same batch size here, since the model with batch_size=1 always gets worse throughput on GPUs. And it is better to show the speedup from the Linear Attention under the fair comparison.\n- What’s the difference between line 5 and line 10 in Table 4? They have the same configs, but different results.\n- Based on Table 4 and Table 6, “Shift” may cause an accuracy decrease, and it is necessary to introduce the “MoE” method to compensate for the model accuracy, which leads to an increase in latency. So overall, there is no advantage to using the “shift” method in terms of the model accuracy and latency. Does the “shift” method only for energy reduction?\n- Experiments use the Eyeriss accelerator to get energy consumption. What are the configs of this accelerator? What data type is used for this accelerator? When counting energy, is the energy of memory access counted?", "limitations": "The authors discussed the limitations of the implementation of their work. It is necessary to use a customized hardware accelerator to fully leverage the benefits of “Shift” and “Add”.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed ShiftAddViT achieves significant acceleration across different models on different tasks.\n- Experiments show the real speedup on GPUs to demonstrate the effectiveness of the proposed method.\n", "weaknesses": "- Many implementation details of the proposed method are not clear enough.\n- It is unfair for the comparison with DeiT. The proposed method first modified the model architecture, and then introduced the shift and add kernel to improve the model efficiency. When compared with DeiT, the architectures of those two models are not the same. The comparison should be made under the condition that the model architectures are kept the same.\n- Based on Table 3 and Table 4, it appears that the “Shift” was not finally used in the classification task. And in Table 4, compared with the method that only enables “Add”, the final result (highlighted with red background) does not have any advantages in both model accuracy and latency. This seems to demonstrate that the “Shift” and “MoE” methods are not effective on classification tasks.\n- Table 4 and Table 5 lack the result of method “Quant.”+”Shift”+”MoE”. It would be better if these results could be provided to see the impact on model accuracy and latency when all the proposed methods are adopted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Implementation details are not clear enough**\n\nSorry for not making it clear enough to you. We supplied more settings to Sec. 4 in the Appendix as we focused on the motivation and high-level idea in the main paper. We will clarify more and release code&models upon acceptance.\n\n**W2: For DeiT results, the architectures should be the same?**\n\nYes, for fair comparisons, we compare to both MSA (original DeiT-T) and linear attention (LA; same architecture as ours) in ***Table 5 of the attached PDF in global response***.\n\nWe see that ShiftAddViT consistently works better, reducing 43 ~ 65% and 16 ~ 43% latency/energy with comparable accuracy ($\\pm$0.2%). Here LA achieves better accuracy than MSA because we adopted the Norm. (in Attn) and DWConv (in MLP) following TransNormer [EMNLP'22] & EfficientViT [ICCV'23]. The new operators also explain the increased latency despite linear complexity.\nThe reason for building on LA is that Q/K is less sensitive than V/Attn (see our reply to Reviewer qgW6's W1).\n\n**W3: The shift was not finally used? No advantage of final models? Shift and MoE are not effective?**\n\n**The shift was not finally used?** We humbly clarify that the shift layer was also used in our MoE framework, i.e., Mult. expert + Shift expert, and thus was also used in the final model in Tables 3 & 4. I.e., MoE can be thought of as a soft alternative to the pure shift for parameterizing linear layers or MLPs to achieve better accuracy instead of being an orthogonal technique.\n\n**The advantage of using shift.** Replacing Mult.-based MLPs with shift layers significantly reduces latency, energy, and chip area costs. Our shift kernel offers an average speedup of 2.35x/3.07x/1.16x compared to PyTorch FakeShift, TVM FakeShift, and TVM MatMuls, respectively (Figure 3). The seemingly minor latency improvements in Tables 3 & 4 are due to full optimization of the compiled model as a whole (e.g., 6.34ms → 1ms for PVTv2-B0) on GPUs *with sufficient chip area*. Most gains are concealed by data movements and system-level schedules. \nAdopting shift layers substantially lowers energy and chip area usage (Table 1). ***Under the same chip areas***, latency savings are more pertinent, e.g., PVTv2-B0 w/ shift or MoE achieve 3.9 ~ 5.7x and 1.6 ~ 1.8x speedups, respectively, as summarized in ***Table 2 of the attached PDF in our global response***.\n\n**MoE to compensate for accuracy drop.** MoE is a soft alternative to adopt both Mult. and shifts to achieve a better accuracy-efficiency trade-off, i.e., on average +1.36% accuracy gain with 46% ~ 53% shift benefits preserved.\n\n**W4: Lack the results of “Quant. + Shift + MoE”.**\n\nWe conduct the requested experiment for PVTv2-B1:\n1. Quant.: 78.70%\n2. Quant. + Shift (Attn & MLPs): 77.55%\n3. Quant. + Shift (Attn) + MoE (MLPs): 78.23%.\n\nThis is also consistent with our previously reported results in lines 5-7 of Table 6 when adopting KSH instead of Quant: \"KSH + Shift (Attn) + MoE (MLPs)\": 78.20%.\n\nWe humbly clarify that MoE is an alternative to Shift and they cannot be applied for the same layer. Let us know if we misunderstand your question.\n\n**Q1: Implementation details about the shift & add?**\n\n**Scaling factor for shifts? How to convert pre-trained weights?** We reparameterize shifts following DeepShift-PS [CVPRW'21] and do not use a scaling factor.\n\n$W_{shift} = S*2^P$, where $S = sign(W); P = round(log_2(abs(W)))$\n\nAs both S & P are trainable during the finetuning, the reconstruction loss will be reduced.\n\n**Bit allocation? STE used?** We adopt 4 bits for P and yes, STE is used following DeepShift.\n\n**Where to multiply scaling factors for KSH and Quant.?** For KSH, there is no scaling factor needed as a set of hash functions is applied to convert Q/K to binary codes. For Quant., we leverage layer-wise Quant. for both Q & K, the scaling factor can be multiplied after add ops. It can be efficiently implemented following Sec. 2.2 of [1].\n\n**Input activation Quant. for shifts? Last Shift?** The input activation of **all** shift layers is rounded to 16 bits of fixed-point precision format following DeepShift. We use layer-wise Quant. so the scaling factor can be multiplied after shift ops.\n\n[1] Quant. & Training of NNs for Integer-Arithmetic-Only Inference, CVPR'18\n\n**Q2: How to get gate value in MoE?**\n\nSorry for the confusion as we merge both gate and softmax into one equation. The gate itself is a linear layer and p = G(x) is the output gate value of dimension 2 (number of experts). The softmax is for normalizing p, and for efficiency, we use argmax to select one expert.\n\n**Q3: Experiment details?**\n\n**PVT slower than MSA? Different batch sizes (BS)?** We ensured BS=1 for all latency measurements and also found this counterintuitive phenomenon.\n\nThe reasons are two folds: (1) linear attention introduces extra ops, e.g., normalization, DWConv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large → limited benefits.\n\nTo validate both points, we measure the latency of PVTv2-B0 with various BS and input resolutions as shown in ***Table 3 of the attached PDF in our global response***: Linear attention’s benefits show up under larger BS or input resolutions.\n\n**Difference between lines 5 & 10 in Table 4?** For line 5, we use KSH where Q & K are identical following Ecoformer; For line 10, Q & K are independent as we directly quantize both.\n\n**Shift advantages? Only energy savings?** No, using shifts has comprehensive benefits. Please refer to our reply to your W3 for the analysis.\n\n**Configs of Eyeriss? What data types? Memory access?** The configs like bit allocations are matched with our algorithm (e.g., INT32 for adds; INT16 for shifts). Data type and unit energy are reported in Table 1. We do count the memory access costs. More details can be found in DNN-Chip-Predictor [ICASSP'20] of which the contribution is the simulator of Eyeriss."}, {"review_id": "PPTRKORgq5", "reviewer": "Reviewer_qgW6", "summary": "The authors propose re-parameterizing ViTs to speed up inference without a full retraining. To do this, they introduce a new operation ShiftAddViT, which works well when applied to attention. For the MLP in a transformer, the authors use a mixture of experts. This operation reduces latency and energy usage, while maintaining accuracy.", "questions": "- I would've thought that the ShiftAddViT technique can really be applied to any operation (Clearly, that's not the case, as you added a modification for MLPs.) Why doesn't this work out of the box for any matrix multiplication?\n- What is \"ideal parallelism\" in the footnote for Table 4? Are these latencies not actually measured end-to-end? If this is the case, do the baseline latency measurements also use the same assumption of ideal parallelism? How fast/slow is the MoE kernel when plugged into the network and *entire network is measured end-to-end?", "limitations": "See above.\n\nThe evaluation is thorough, and the method is clearly presented. Despite the questions and objections I had above, I feel this is a cogently stated idea and clearly a thoroughly-investigated problem. I do still have some questions about the method and would like to know the answers, but the idea alone is definitely worth publishing.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The premise of leveraging power-of-two multiplications (a shift) is clever and a well-stated intuition (justified by Table 1 and also in L150-156). The additional energy analysis and chip-area statistics build ethos very well.\n- The paper presents a very thorough set of experiments on ImageNet classification and novel view synthesis. The latency analysis is thorough and well-documented / reasoned.", "weaknesses": "- My only qualm is that (it appears) this relies on the existence of an effective binary quantization algorithm in L176. The results convince me that this binary quantization is acceptable, but this seems aggressive (as the authors mention later, when replacing the MLP).\n- From the introduction, I initially thought this re-parameterization would be arithmetically identical to the original. After reading the methods, it appears this isn't the case? In which case, wouldn't you need some fine-tuning? If so, how costly is the fine-tuning? If I'm wrong, this may be because I don't quite understand Figure 2 -- how is this related to the original matmul?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns.\n\n**W1: The only qualm is that binary quantization seems too aggressive?**\n\nWe thank the reviewer for pointing out this. We examined the sensitivity of different parts in attention blocks, where V and attention scores are much more sensitive than Q or K. Quantizing or binarizing V or attention scores yield a significant accuracy drop, e.g., a 3.6% drop when quantizing V in DeiT-T. While binarizing Q and K to convert add layers can mostly maintain the accuracy. The rationale behind this phenomenon is that Q and K mainly serve for similarity measurement while attention scores and V are the actual activations that need higher precision to maintain feature richness. That is also why we perform binarization for K in KV multiplication and Q in the Q(KV) multiplication.\n\nOur baseline Ecoformer also adopts binarization to Q and K but relies on a set of learned hash functions to map identical Q/K matrices during training. In our ShiftAddViT framework, we find that the standard layer-wise quantization is also robust for quantizing Q and K and they do not need to be identical.\n\n---\n\n**W2: The re-parameterization would not be arithmetically identical to the original? Need finetuning? How costly is the finetuning?**\n\nYes, you are right. We are not making our model arithmetically identical to the original ViTs. Instead, we inherit the pre-trained weight to parameterize shift or add layers. For example, we follow the below equation to reparameterize the shift weights based on the inherited weights:\n\n$W_{shift} = S*2^P$, where $S = sign(W); P = round(log_2(abs(W)))$\n\nWe do need finetuning to mitigate the reconstruction loss and recover the accuracy. For example, both the above S and P are trainable during fine-tuning.\n\nIf we directly reparameterize multiplication-based models with mathematical equivalent shifts **and** adds, from an algorithm perspective, there will be extreme non-uniform weight distributions (see our reply to your Q1) that are hard to quantize; from a hardware perspective, currently efficient multiplication implementations already use shift and add units. There will be no energy or latency savings. Our current solution approximates multiplications with either shifts **or** adds, thus saving more hardware resources.\n\n\nAs suggested, we collect the training or finetuning wall-clock time, which is 21% ~ 25% less than training from scratch as the original Transformer model and >50x less than the previous ShiftAddNet [1] did on GPUs. For example, training the PVTv1-Tiny model from scratch takes 62 hours while our finetuning only needs 46 hours. We will add this information to the final revised manuscript to offer more insights into the training cost implications.\n\nIn addition, we would like to clarify that our main goal is efficient inference and deployment, as validated by the reported real-measured inference wall-clock times on RTX 3090 GPU in the submitted manuscript. The training or finetuning cost savings are by-products as we support to reparametrize ShiftAddViTs based on the weights from already available pre-trained checkpoints of original ViTs.\n\n[1] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS 2020\n\n---\n\n**Q1: Why the proposed ShiftAddViT can not work out of the box for any matrix multiplication? Why add a modification to MLPs?**\n\nGood question. With W1, we are clear that our solution replaces multiplications with either shifts **or** adds instead of mathematically equivalent shifts **and** adds.\n\nThe reason why ShiftAddViT cannot work out of the box is that shift- and add-based models prefer different weight distributions from multiplication-based models. For example, while repeated additions can in principle replace any multiplicative mapping, they do so in a very parameter-inefficient way. A rough example is that input 8 multiplied by weight 2 can lead to output 16, while if we only use additions to get this output, we need to add weight 8, which is much larger than the previous weights. In contrast, power-of-2 quantization in shift layers is efficient but cannot span the entire continuous space of multiplicative mappings.\n\nIt is also commonly observed in the multiplication-less network community that shift- or add-based models suffer from slight accuracy drops due to less expressiveness [2] and demand additional techniques to fully recover the accuracy. This is why we propose to adopt a softer version, i.e., the mixture of experts, to mitigate the accuracy drop while preserving the benefits as much as possible to the best of our knowledge.\n\n[2] AdderNet: Do We Really Need Multiplications in Deep Learning? CVPR 2020 Oral\n\n---\n\n**Q2: What is \"ideal parallelism\" in the footnote for Table 4? Do the baseline latency measurements also use the same assumption? How fast/slow is the MoE kernel when plugged into the network and the *entire* network is measured end-to-end?**\n\nThe \"ideal parallelism\" refers to a simulated scenario to mimic parallel computing, in particular, we optimize each expert separately and measure their latency, the maximum latency among all experts will be recorded and regarded as the latency of this MoE layer, and the aggregated total of the time spent for each layer is the final reported modularized model latency.\n\nSince our baselines do not adopt the mixture of experts (MoE) so there is no need to assume \"ideal parallelism\". We only report the modularized latency for our models with MoE layers for readers’ reference, those latencies are not used for comparison.\n\nWe want to clarify that all our comparisons are made under fair conditions, i.e., we report the end-to-end wall-clock inference time in Table 3 and use them as our final latency when compared with all baselines instead of assuming the ideal parallelism (only providing a reference number). We will make this point clear in the final revised manuscript."}, {"review_id": "u38jRugqRq", "reviewer": "Reviewer_urMu", "summary": "This paper proposes an efficient accelerating framework for vision transformer, ShiftAddViT, which reparameterizes pre-trained ViTs with a mixture of complementary multiplication primitives and MoE designs. Specifically, All MatMuls in self-attention modules are reparameterized by additive kernels, and the remaining linear layers and MLPs are reparameterized by shift kernels. Besides, they develop a new MoE system for maintaining accuracy after reparameterization, and use a latency-aware load-balancing loss term to assign a dynamic amount of input tokens to each expert. Extensive experiments on various 2D/3D Transformer-based vision models demonstrate their superiority and efficiency.", "questions": "1. What is the most important contributions in this paper？Multiplacation-less solution and MoE seems indepedent, It is a common sence that MoE could boost the performance on various tasks, improving accuracy with MoE seems unnecessary in this paper. \n\n2. Re-parameterizing usually refers to one kind of mathematical equivalence transformations, such as RepVGG [1], AC-Net [2], what is the re-parameterizing in this paper?\n\n3. ShiftAddNet [3] provides the first scheme for the replacing of shift and additive opration with multiplication, so what is the weakness for comination of ShiftAddNet  and ViT architectures, could you show the comparision with ShiftAddViT.\n\n4. When quantized to 4/8-bit, original ViT models could also achieve high performance and better acceleration, could you compare the quantized ViT with ShiftAddViT? \n\n5. More complex task is better, such as detection and segementation.\n\n[1] Ding X, Zhang X, Ma N, et al. Repvgg: Making vgg-style convnets great again[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 13733-13742.\n\n[2] Ding X, Guo Y, Ding G, et al. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1911-1920.\n\n[3]You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783.", "limitations": "This paper provides the the limitation and societal impact discussion for their proposed technique.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written and easy to follow. Converting multiplication operations to additive and shift is a promising technique in model compression and accleration, this paper provides a systematic design solutions for vision transformer architectures, which is interesting.\nExperiments in this paper are sufficient, including 2D and 3D task., and they provides detailed latency comparison for different methods, which much helps prove their effectiveness.", "weaknesses": "ShiftAddViT is a complex system design solution, where the acceleration comes from the replacing of additive and shift operations with multiplication, the maintaining accuracy relies on the MoE design. And ShiftAddViT utils TVM to implement and optimize their customized kernels for practical hardware deployment on GPUs. In this paper, the contributions of multiplation less and MoE are much independent, this paper  seems not much suitable for algorithm-based conferences like NeurIPS/CVPR/ICML.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns.\n\n**W1: The contributions of multiplication less and MoE are independent? Not very suitable for algorithm conferences like NeurIPS/CVPR/ICML?**\n\nWe thank the reviewer for acknowledging our system design and practical deployment on GPUs. \n\n*As for the contribution*, we want to clarify that multiplication-less and MoE are linked in one organic whole. Unlike previous MoE works where all experts are of the same capacity and there is no distinction between tokens, our MoE layers combine unbalanced experts, i.e., multiplication and shift, with a hypothesis to divide important object tokens and background tokens as also visualized in Figure 6. Such a new MoE design offers a softer version of ShiftAddViTs, i.e., instead of aggressively replacing all multiplications with cheaper shifts and adds, we keep the powerful multiplication option to handle importance tokens for maintaining accuracy while leaving all remaining unimportant tokens being processed by cheaper bitwise shifts, winning a better accuracy and efficiency tradeoff.\n\n*As for the venues*, we humbly clarify that the topic of multiplication-reduced network design is suitable and of great interest to algorithm conference audiences. For example, AdderNet [CVPR’20 Oral], DeepShift [CVPRW’21], ShiftAddNet [NeurIPS’20], ShiftAddNAS [ICML’22], and Ecoformers [NeurIPS’22] adopt shift or add layers to replace multiplication-based operations in CNNs or Transformers. TVM optimization for real hardware deployment is used by previous works like HAWQ-V3 [ICML’21] as well.\n\n---\n**Q1: What are the most important contributions? Multiplication-less solution and MoE seem independent? MoE’s uniqueness in this work?**\n\nOur contributions are the multiplication-reduced ViTs and a new load-balanced MoE framework for a soft alternative to ShiftAddViTs. Also, we provide the first-time systematic investigation of layer sensitivity, accuracy impact, allocation strategy, and hardware implementations when considering shift&add-based ViTs as also acknowledged by Reviewer FYmk.\n\nThe multiplication-less solution and MoE are effectively combined, and the load-balanced and input-adaptive MoE design is uniquely applicable to our ShfitAddViTs, please also refer to our reply to your W1 for a detailed analysis.\n\n---\n**Q2: What is the re-parameterizing in this paper?**\n\nSorry for not making it clear enough. We are not making our model arithmetically identical to the original ViTs. Instead, we inherit the pre-trained weight to parameterize shift or add layers. For example, we follow the below equation to reparameterize the shift weights based on the inherited weights:\n\n$W_{shift} = S*2^P$, where $S = sign(W); P = round(log_2(abs(W)))$\n\nWe do need finetuning to mitigate the reconstruction loss and recover the accuracy. For example, both the above S and P are trainable during finetuning.\n\nIf we directly reparameterize multiplication-based models with mathematical equivalent shifts **and** adds, from an algorithm perspective, there will be extreme non-uniform weight distributions that are hard to quantize (see our reply to Q1 of Reviewer qgW6); from a hardware perspective, currently efficient multiplication implementations already use shift and add units. There will be no energy or latency savings. Our current solution replaces multiplication with either shift **or** add layers, thus saving more hardware resources.\n\n---\n**Q3: What is the weakness of applying ShiftAddNet to ViTs? Show the comparison with it.**\n\nWeakness of ShiftAddNet and qualitative comparisons (We have also included these points in Sec. 2 & 4):\n\n| ShiftAddNet | ShiftAddViT |\n|:---|:---|\n| Adopts cascaded shift layers and add layers → a doubled number of layers, parameters, and FLOPs | Keeps the original number of layers/parameters |\n| Not suitable for the MatMuls among Q/K/V in attentions as those matrices are all activations | Well supports attention in Transformers |\n| Has to train the whole model from scratch | Can inherit the pre-trained weights of ViTs |\n| No speedups on GPUs (much slower training & inference) | Provides GPU optimizations |\n| Not compatible with MoE | Compatible with MoE for switching between mixture multiplication primitives |\n\n**Quantitative comparisons.** We apply ShiftAddNet on top of PVTv2-B0 and show the comparison in ***Table 7 of the attached PDF in our global response***. We see that ShiftAddViT achieves 54%, >50x, and 59% parameters, GPU latency, and energy savings at much more stable training (ShiftAddNet suffers from loss NaN when being applied to ViTs).\n\n---\n**Q4: Compare the quantized ViT with ShiftAddViT?**\n\nSure, we compare our ShiftAddViT with the latest 4-bit Transformer quantization work [1] and show the results in ***Table 8 of the attached PDF in our global response***. *In terms of latency*, we see that our method achieves on average 4.7x and 2.4x GPU speedups than 4-bit quantized attentions and MLPs, respectively. *In terms of accuracy*, [1] reveals that existing 4-bit fully quantized training algorithms still have around a 1 ~ 2.5% accuracy drop on server tasks while our ShiftAddViT achieves comparable or even higher accuracy than original ViTs.\n\n[1] Training Transformers with 4-bit Integers, arXiv’23\n\n---\n**Q5: The more complex task is better.**\n\nWe follow the suggestion to extend the PVTv2-B0 (ShiftAddViT) backbone to detection (follow ViTDet [ECCV'22]) and segmentation (follow PVTv2) tasks as shown in ***Table 4 of the attached PDF in our global response***.\n\n*For the detection task*, the ShiftAdd-based backbone achieves 63.7% and 58.0% latency and energy reductions while keeping comparable mAP. *For the segmentation task*, the ShiftAdd-based backbone achieves 71.7% and 66.6% latency and energy reductions while keeping comparable mIoU.\n\nWe also extend our method to NLP tasks, please refer to our reply to Reviewer FYmk's W1."}]

Author rebuttal:
**Dear ACs and Reviewers,**

First of all, we deeply appreciate the time and effort spent by you in providing the reviews, and truly value your effort, especially considering the substantial scale of a conference like NeurIPS.

We are immensely grateful for the positive feedback our paper has received. The accolades, including remarks about its excellent soundness and presentation, pioneering systematic investigation, impressive achievements in latency and energy savings, and the clarity and conciseness of both the writing and demonstrations, alongside the extensive and thorough experiments, are all deeply gratifying. It's particularly encouraging that these aspects have garnered unanimous appreciation from the reviewers.

Despite the commendations, we have also received inquiries from reviewers requesting additional experiments and further clarification. We have supplied the requested experiments and provided detailed clarifications to raised questions as summarized below.

---
**To summarize, the following experiments have been supplied:**

- *Extend to NLP tasks and more token scenarios*
  - We have included the results in our rebuttal response to *Reviewer FYmk's W1*.
  - Also refer to *Table 1* of the attached PDF for the organized result table.
- *Extend to larger models like ViT-Base or similar*
  - We have included the results in our rebuttal response to *Reviewer hFZt’s W1*.
  - Also refer to *Table 6* of the attached PDF for the organized result table.
- *DeiT comparison under the same architecture*
  - We did not provide a result table in our response to *Reviewer MmZd's W2* due to length limitations.
  - Please refer to *Table 5* of the attached PDF for the organized result table.
- *Ablation studies of adopting shift or not*
  - We did not provide a result table in our response to *Reviewer MmZd's W3* due to length limitations.
  - Please refer to *Table 2* of the attached PDF for the organized result table.
- *Ablation studies of varying batch sizes and input resolutions*
  - We did not provide a result table in our response to *Reviewer MmZd's Q3* due to length limitations.
  - Please refer to *Table 3* of the attached PDF for the organized result table.
- *Add baseline: compare to ShiftAddNet*
  - We did not provide a result table in our response to *Reviewer urMu’s Q3* due to length limitations.
  - Please refer to *Table 7* of the attached PDF for the organized result table.
- *Add baseline: compare to quantized ViTs*
  - We did not provide a result table in our response to *Reviewer urMu’s Q4* due to length limitations.
  - Please refer to *Table 8* of the attached PDF for the organized result table.
- *Extend to more complex tasks, such as detection and segmentation*
  - We did not provide a result table in our response to *Reviewer urMu’s Q5* due to length limitations.
  - Please refer to *Table 4* of the attached PDF for the organized result table.

---
**To summarize, the following questions have been clarified:**

- *MoE dispatching and the methodology for input allocation and parallel computing*
  - We clarify this in our response to *Reviewer FYmk’s W2 and qgW6’s Q2*.
- *Training stability and wall-clock time*
  - We clarify this in our response to *Reviewer hFZt’s W2 and qgW6’s W2*.
- *Implementation details*
  - Scaling factors, bit allocation, and gate mechanism in ShiftAddViT
    - We clarify this in our response to *Reviewer MmZd’s Q1 and Q2*.
  - Difference between line 5 and line 10 in Table 4, i.e., KSH vs. Quant
    - We clarify this in our response to *Reviewer MmZd’s Q3*.
  - Configs of Eyeriss accelerator
    - We clarify this in our response to *Reviewer MmZd’s Q3*.
  - Lack the results of “Quant. + Shift + MoE”.
    - We clarify this in our response to *Reviewer MmZd’s W4*.
- *The advantages and rationale of adopting shifts and MoE; The contributions of multiplication less and MoE are independent?*
  - We clarify this in our response to *Reviewer MmZd’s W3/Q3 and urMu’s W1/Q1*.
- *Why PVT or Linear attention is slower than MSA*
  - We clarify this in our response to *Reviewer MmZd’s Q3*.
- *Binary quantization seems too aggressive*
  - We clarify this in our response to *Reviewer qgW6’s W1*.
- *The re-parameterization would not be arithmetically identical to the original?*
  - We clarify this in our response to *Reviewer qgW6’s W2/Q1 and urMu’s Q2*.
- *Suitable for algorithm-based conferences like NeurIPS/CVPR/ICML?*
  - We clarify this in our response to *Reviewer urMu’s W1*.

---

Regarding Reviewer MmZd's questions about implementation details, we've clarified concisely within the rebuttal's length limitations. We are open to providing further details if any points still seem unclear. To enhance reproducibility, we will release both the codebase and pre-trained models, enabling others to replicate our results effectively.

We would be much appreciated if you could check our rebuttal response and expect the new experiments and clarifications can solve your concerns. We are always willing to be involved in the discussion with you, please let us know if our responses do not resolve your concerns so that we can further clarify, thanks!

Best regards,

Paper 9955 Authors

Meta-review: All reviewers support the acceptance of this paper. The initial concerns have been addressed during the rebuttal.
Decision: Accept (poster)