Title: ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
Venue: NeurIPS 2023
Accepted: yes
Keywords: Efficient Vision Transformer; Multiplication-reduced networks; Hardware acceleration
Abstract: Vision Transformers (ViTs) have shown impressive performance and have become a unified backbone for multiple vision tasks. However, both the attention mechanism and multi-layer perceptrons (MLPs) in ViTs are not sufficiently efficient due to dense multiplications, leading to costly training and inference. To this end, we propose to reparameterize pre-trained ViTs with a mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication-reduced model, dubbed $\textbf{ShiftAddViT}$, which aims to achieve end-to-end inference speedups on GPUs without requiring training from scratch. Specifically, all $\texttt{MatMuls}$ among queries, keys, and values are reparameterized using additive kernels, after mapping queries and keys to binary codes in Hamming space. The remaining MLPs or linear layers are then reparameterized with shift kernels. We utilize TVM to implement and optimize those customized kernels for practical hardware deployment on GPUs. We find that such a reparameterization on (quadratic or linear) attention maintains model accuracy, while inevitably leading to accuracy drops when being applied to MLPs. To marry the best of both worlds, we further propose a new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication or its primitives as experts, e.g., multiplication and shift, and designing a new latency-aware load-balancing loss. Such a loss helps to train a generic router for assigning a dynamic amount of input tokens to different experts according to their latency. In principle, the faster the experts run, the more input tokens they are assigned. Extensive experiments on various 2D/3D Transformer-based vision tasks consistently validate the effectiveness of our proposed ShiftAddViT, achieving up to $\textbf{5.18$\times$}$ latency reductions on GPUs and $\textbf{42.9}$% energy savings, while maintaining a comparable accuracy as original or efficient ViTs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddViT.

ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformers

Abstract

Vision Transformers (ViTs) have shown impressive performance and have become a unified backbone for multiple vision tasks. However, both the attention mechanism and multi-layer perceptrons (MLPs) in ViTs are not sufficiently efficient due to dense multiplications, leading to costly training and inference. To this end, we propose to reparameterize pre-trained ViTs with a mixture of multiplication primitives, e.g., bitwise shifts and additions, towards a new type of multiplication-reduced model, dubbed ShiftAddViT , which aims to achieve end-to-end inference speedups on GPUs without requiring training from scratch. Specifically, all MatMuls among queries, keys, and values are reparameterized using additive kernels, after mapping queries and keys to binary codes in Hamming space. The remaining MLPs or linear layers are then reparameterized with shift kernels. We utilize TVM to implement and optimize those customized kernels for practical hardware deployment on GPUs. We find that such a reparameterization on (quadratic or linear) attention maintains model accuracy, while inevitably leading to accuracy drops when being applied to MLPs. To marry the best of both worlds, we further propose a new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication or its primitives as experts, e.g., multiplication and shift, and designing a new latency-aware load-balancing loss. Such a loss helps to train a generic router for assigning a dynamic amount of input tokens to different experts according to their latency. In principle, the faster the experts run, the more input tokens they are assigned. Extensive experiments on various 2D/3D Transformer-based vision tasks consistently validate the effectiveness of our proposed ShiftAddViT, achieving up to 5.18 × latency reductions on GPUs and 42.9% energy savings, while maintaining a comparable accuracy as original or efficient ViTs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddViT .

1 Introduction

Vision Transformers (ViTs) have emerged as powerful vision backbone substitutes to convolutional neural networks (CNNs) due to their impressive performance [57, 16]. However, ViTs' state-of-the-art (SOTA) accuracy comes at the price of prohibitive hardware latency and energy consumption for both training and inference, limiting their prevailing deployment and applications [34, 51], even though pre-trained ViT models are publicly available. The bottlenecks are twofold based on many profiling results: attention and MLPs, e.g., they account for 36%/63% of the FLOPs and 45%/46% of the latency when running DeiT-Base [55] on an NVIDIA Jetson TX2 GPU [72, 19]). Previous efficient ViT solutions mostly focus on macro-architecture designs [21, 39, 32, 55, 9, 61] and linear attention optimization [59, 50, 5, 13, 6, 73]. However, they pay less attention on reducing the underlying dominant multiplications as well as on co-optimizing both attention and MLPs.

We identify that one of the most effective yet still missing opportunities for accelerating ViTs is to reparameterize their redundant multiplications with a mixture of multiplication primitives, i.e.,

bitwise shifts and adds. The idea is drawn from the common hardware design practice in computer architecture or digital signal processing. That is, the multiplication can be replaced by bitwise shifts and adds [67, 22]. Such a hardware-inspired approach can lead to an efficient and fast hardware implementation without compromising the model accuracy. As such, this paper embodies the shift&add idea in ViTs towards a new type of multiplication-reduced model. Moreover, unlike previous ShiftAddNet [70] which requires full and slow training and dedicated hardware accelerator support, this work starts from pre-trained ViTs to avoid the tremendous cost of training from scratch and targets end-to-end inference speedups, i.e., accelerating both attention and MLPs, on GPUs.

The aforementioned shift&add concept uniquely inspires our design of naturally hardware-friendly ViTs, but it also presents us with three challenges to address. First , how to effectively reparameterize ViTs with shifts and adds? Previous ShiftAddNet [70] reparameterizes CNNs by cascading shift layers and add layers, leading to a doubled number of layers or parameters. The customized CUDA kernels of shift and add layers also suffer from much slower training and inference than PyTorch [42] on GPUs. Both motivate a new easy-toaccelerate reparameterization method for ViTs. Second , how to maintain the accuracy after reparameterization? It is expected to see accuracy drops when multiplica-

tions are turned into shift and add primitives [8, 11]. Most works compensate for the accuracy drops by enlarging model sizes leveraging the much improved energy efficiency advantages of shifts and adds [70], e.g., up to 196 × unit energy savings than multiplications as shown in Tab. 1. While in ViTs, where input images are split into non-overlapping tokens, one can uniquely leverage the inherent adaptive sensitivity among input tokens. In principle, the essential tokens containing target objects are expected to be processed using more powerful multiplications. In contrast, tokens with less important background information can be handled by cheaper primitives. Such a principle aligns with the spirit of recent token merging [3] and input adaptive methods [45, 68, 74] for ViTs. Third , how can we balance the loading and processing time for the aforementioned important and less-important input tokens? For ViTs, using a mixture of multiplication primitives can lead to varying processing speeds for sensitive and non-sensitive tokens. This needs to be balanced; otherwise, intermediate activations may take longer to synchronize before progressing to the next layer.

To the best of our knowledge, this is the first attempt to address the above three challenges and to incorporate the shift&add concept from the hardware domain in designing multiplication-reduced ViTs using a mixture of multiplication primitives. Specifically, we make the following contributions:

We take inspiration from hardware practice to reparameterize pre-trained ViTs with a mixture of complementary multiplication primitives, i.e., bitwise shifts and adds, to deliver a new type of multiplication-reduced network, dubbed ShiftAddViT . All MatMuls in attention are reparameterized by additive kernels, and the remaining linear layers and MLPs are reparameterized by shift kernels. The kernels are built in TVM for practical deployments.

Weintroduce a new mixture of experts (MoE) framework for ShiftAddViT to preserve accuracy post-reparameterization. Each expert represents either a multiplication or its primitives, such as shifts. Depending on the importance of a given input token, the appropriate expert is activated, for instance, multiplications for vital tokens and shifts for less crucial ones.

We introduce a latency-aware load-balancing loss term within our MoE framework to dynamically allocate input tokens to each expert. This ensures that the number of tokens assigned aligns with the processing speeds of the experts, significantly reducing synchronization time.

We conduct extensive experiments to validate the effectiveness of our proposed ShiftAddViT. Results on various 2D/3D Transformer-based vision models consistently show its superior efficiency, achieving up to 5.18 × latency reductions on GPUs and 42.9% energy savings, while maintaining a comparable or even higher accuracy than the original ViTs.

2 Related Works

Vision Transformers (ViTs). ViTs [16, 55] split images into non-overlapping patches or tokens and have outperformed CNNs on a range of vision tasks. They use a simple encoder-only architecture, primarily made up of attention modules and MLPs. The initial ViTs necessitated costly pre-training

on expansive datasets [52], and subsequent advancements like DeiT [55] and T2T-ViT [75] alleviated this by refining training methodologies or crafting innovative tokenization strategies. Later on, a surge of new ViT architecture designs, such as PVT [60], CrossViT [7], PiT [26], MViT [18], and SwinTransformer [35], has emerged to improve the accuracy-efficiency trade-offs using a pyramid-like architecture. Another emergent trend revolves around dynamic ViTs like DynamicViT [45], AViT [68], ToME [3], and MIA-Former [74], aiming to adaptively eliminate non-essential tokens. Our MoE framework resonates with this concept but addresses those tokens using low-cost multiplication primitives that run concurrently. For edge deployment, solutions such as LeViT [21], MobileViT [39], and EfficientViT [5, 50] employ efficient attention mechanisms or integrate CNN feature extractors. This approach propels ViTs to achieve speeds comparable to CNNs, e.g., MobileNets [47]. In contrast, our proposed ShiftAddViT is the first to draw from shift&add hardware shortcuts for reparameterizing pre-trained ViTs, boosting end-to-end inference speeds and energy efficiency without hurting accuracy. Therefore, ShiftAddViT is orthogonal to existing ViT designs and can be applied on top of them.

Tremendous efforts have been dedicated to designing efficient ViTs. For instance, to address the costly self-attention module, which possesses quadratic complexity in relation to the number of input tokens, various linear attentions have been introduced. These can be broadly divided into two categories: local attention [35, 2, 56] and kernel-based linear attention [30, 13, 65, 38, 5, 34, 1]. For the former category, Swin [35] computes similarity measures among neighboring tokens rather than all tokens. QnA [2] disseminates the attention queries across all tokens. MaxViT [56] employs block attention and integrates dilated global attention to capture both local and global information. Regarding the latter category, most methods approximate the softmax function [13, 30, 4] or the complete self-attention matrix [65, 38] using orthogonal features or kernel embeddings, allowing the computational order to switch from ( QK ) V to Q ( KV ) . Furthermore, a few efforts target reducing the multiplication count in ViTs. For instance, Ecoformer [34] introduces a novel binarization paradigm through kernelized hashing for Q / K , ensuring that MatMuls in attentions become solely accumulations. Adder Attention [51] examines the challenges of integrating adder operations into attentions and suggests including an auxiliary identity mapping. ShiftViT [58] incorporates spatial shift operations into attention modules. Different from previous works, our ShiftAddViT focuses on not only attention but also MLPs towards end-to-end ViT speedups. All MatMuls can be transformed into additions/accumulations using quantization, eliminating the need for complex kernelized hashing (KSH) as in [34]. For the remaining linear layers in the attention modules or MLPs, they are converted to bitwise shifts or MoEs, rather than employing spatial shifts like [58] for attentions.

Multiplication-less NNs. Numerous efforts aim to minimize the dense multiplications, which are primary contributors to the time and energy consumption in CNNs and ViTs. In the realm of CNNs, binary networks [14, 29] binarize both activations and weights; AdderNet [8, 66, 62] fully replaces multiplications with additions with a commendable accuracy drop; Shift-based networks use either spatial shifts [64] or bitwise shifts [17] to replace multiplications. Recently, techniques initially developed for CNNs have found applications in ViTs. For instance, BiTs [36, 25] introduce binary transformers that maintain commendable accuracy; Shu et al. and Wang et al. also migrate the add or shift idea to ViTs with the scope limited to attentions [51, 58]. Apart from network designs, there are also neural architecture search efforts made towards both accurate and efficient networks [71, 31]. As compared to the most related work, i.e., ShiftAddNet [70] and ShiftAddNAS [71], our ShiftAddViT for the first time enables shift&add idea in ViTs without compromising the model accuracy, featuring three more characteristics that make it more suitable for practical applications and deployment: (1) starting from pre-trained ViTs to avoid the costly and tedious training from scratch; (2) focusing on optimization for GPUs rather than on dedicated hardware such as FPGAs/ASICs; (3) seamlessly compatible with the MoE framework to enable switching between the mixture of multiplication primitives, such token-level routing and parallelism are uniquely applicable and designed for ViTs.

3 Preliminaries

Self-attention and Vision Transformers. Self-attention is a core ingredient of Transformers [57, 16], and usually contains multiple heads H with each measuring pairwise correlations among all input tokens to capture global-context information. It can be defined as below:

where h denotes the number of heads. Within each head, input tokens X ∈ R n × d of length n and dimension d will be linearly projected to query, key, and value matrices, i.e., Q , K , V ∈ R n × d k ,

Q

K

V

Shift

Shift

Shift

Add

Add

Patchify

...

N

Patch

Embedding

...

...

...

...

DWConv

Shift

Binary code

Binary code

(b) ShiftAddViT Attention

Shift

Mult.

Router

MoE

Concat

...

...

...

...

(c) ShiftAddViT MLP

Linear

Attention

(a) Inputs

through three linear mapping functions, f Q = XW Q , f K = XW K , f V = XW V , where d k = d/h is the embedding dimension of each head. The results from all heads are concatenated and projected with a weight matrix W O ∈ R d × d to generate the final outputs. Such an attention module is followed by MLPs with residuals to construct one transformer block that can be formulated as below:

Shift and Add Primitives. Direct hardware implementation of multiplications is inefficient. Shift and add primitives serve as 'shortcuts' for streamlined hardware designs. Shift operations, equivalent to multiplying by powers of two, offer significant savings. As shown in Tab. 1, such shifts can reduce energy use by up to 23.8 × and chip area by 22.3 × compared to multiplications [33, 70], given the INT32 data format and 45nm CMOS technology. Similarly, add operations, another efficient primitive, can achieve up to 31.0 × energy and 25.5 × area savings relative to multiplications. Both primitives have driven numerous network architecture innovations [8, 17, 70, 71].

4 The Proposed ShiftAddViT

Overview. Given the widespread availability of pre-trained ViT model weights, we aim to fine-tune and transform them into multiplication-reduced ShiftAddViTs in order to decrease runtime latency and enhance energy efficiency. To achieve this, we must reparameterize both the attentions and MLPs in ViTs to shift and add operations. Instead of using cascaded shift layers and add layers as in ShiftAddNet [70] or designing specific forward and backpropagation schemes like AdderNet [8, 51], we lean on the simple yet effective approach of ViTs' long-dependency modeling capability. We propose replacing the dominant MLPs/Linears/ MatMuls with shifts and adds, leaving the attention intact. This approach permits ShiftAddViTs to be adapted from existing ViTs without the need for training from scratch. As illustrated in Fig. 1, for attentions, we transform four Linear layers and two MatMuls into shift and add layers, respectively. For MLPs, direct replacement with shift layers results in significant accuracy degradation. Hence, we consider designing a dedicated MoE framework incorporating a blend of multiplication primitives, such as multiplication and shift, to achieve both accurate and efficient final models. These steps transition pre-trained ViTs into ShiftAddViTs, achieving much reduced runtime latency while maintaining a comparable or even higher accuracy.

4.1 ShiftAddViT: Reparameterization Towards Multiplication-reduced Networks

This subsection describes how we reparameterize pre-trained ViTs with hardware-efficient shift and add operations, including a detailed implementation and the corresponding sensitivity analysis.

Reparameterization of Attention. There are two categories of layers in attention modules: MatMuls and Linears, that can be converted to Add and Shift layers, respectively. In order to reparameterize MatMuls to Add layers, we consider performing binary quantization on one operand during MatMuls , e.g., Q or K for the MatMul of QK , so that the multiply-and-accumulate (MAC) operations between two matrices will be replaced with merely energy-efficient add operations [34]. Furthermore, to build ShiftAddViT on top of efficient linear attentions [59, 34, 38], we exchange the order of MatMuls from ( QK ) V to Q ( KV ) for achieving linear complexity w.r.t. the number of input tokens. In this way, the binary quantization will be applied to K and Q as illustrated in Fig. 1 (b), leaving the more sensitive V branch as high precision. This also allows us to insert a lightweight depthwise convolution (DWConv) to V branch in parallel to enhance local feature capturing capability with negligible

0

1

0

1

0

1

X 11

X21

X 12

X 22

X 13

X 23

O11

O21

O12

O22

X

=

X 12

O11

X 11

+ X 13

O12

X 22

O21

X 21

+ X 23

O22

=

=

=

=

(b) Add Layers

2 -1

2 -2

2 1

2 0

X 11

X 21

X 12

X 22

X 13

X 23

O 11

O21

O 12

O22

X

=

X 11

O11

X 11

+

O12

X 21

O21

X 21

+

O22

=

=

=

=

X 12

X 12

X 22

X 22

>>1

<<1

>>1

<<1

+

+

>>1

>>1

X 13

X 13

X 23

X23

>>2

>>2

>>1

>>1

(a) Shift Layers

overhead ( < 1% of total MACs) following recent SOTA linear attention designs [65, 73]. On the other hand, for reparameterizing the left four Linears in the attention module with Shift layers, we resort to sign flips and power-of-two shift parameters to represent the Linear weights [17, 70].

We illustrate the implementation of Shift and Add layers in Fig. 2 and formulate them as follows:

̸

✶ where X refers to input activations, W S and W A are weights in Shift and Add layers, respectively. For Shift weights W S = s · 2 P , s = sign ( W ) ∈ {-1 , 1 } denotes sign flips, P = round ( log 2 ( abs ( W ))) , and 2 P represents bitwise shift towards left ( P > 0 ) or right ( P < 0 ). Both s and P are trainable during the training. For Add weights, we have applied binary quantization and will skip zeros directly, the remaining weights correspond to additions or accumulations. Combining those two weak players, i.e., Shift and Add , leads to higher representation capability as also stated in ShiftAddNet [70], we highlight that our ShiftAddViT seamlessly achieves such combination for ViTs, without the necessity of cascading both of them as a substitute for one convolutional layer.

Reparameterization of MLPs. MLPs are often overlooked but also dominate ViTs' runtime latency apart from attentions, especially for the small or medium amount of tokens [19, 78, 43]. One natural thought is to reparameterize all MLPs with Shift layers as defined above. However, such aggressive design indispensably leads to severe accuracy drops, e.g., ↓ 1.18% for PVTv1-T [60], (DWConv used between two MLPs in PVTv2 are kept). We resolve this by proposing a new MoE framework to achieve speedups with comparable accuracy, as shown in Fig. 1 (c) and will be elaborated in Sec. 4.2.

Sensitivity Analysis. To investigate whether the above reparameterization methods work as expected, we break down all components and conduct sensitivity analysis. Each of them is applied to ViTs with only 30 epochs of finetuning. As summarized in Tab. 2, applying linear attention ( LA ), Add , or Shift to attention layers makes little influence on accuracy. But applying Shift to MLPs may lead to severe accuracy drops for sensitive models like PVTv1-T. Also, making MLPs more effi-

cient contribute a lot to the energy efficiency (e.g., occupy 65.7% on PVTv2-B0). Both the accuracy and efficiency perspectives motivate us to propose a new reparameterization scheme for MLPs.

4.2 ShiftAddViT: Mixture of Experts Framework

This subsection elaborates on the hypothesis and method of our ShiftAddViT MoE framework.

Hypothesis. We hypothesize there are important yet sensitive input tokens that necessitate the more powerful networks otherwise suffer from accuracy drops. Those tokens are likely to contain object information that directly correlates with our task objectives as validated by [3, 74, 45, 68]. While the left tokens are less sensitive and can be fully represented even using cheaper Shift layers. Such an input-adaptive nature of ViTs calls for a framework featuring a mixture of multiplication primitives.

̸

Mixture of Multiplication Primitives. Motivated by the sensitivity analysis in Sec. 4.1, we consider two experts Mult. and Shift for processing important yet sensitive tokens and insensitive tokens, respectively. As illustrated in Fig. 1 (c), we apply the MoE framework to compensate for the accuracy drop. Each input token representation x will be passed to one expert according to the gate values p i = G ( x ) in routers, which are normalized via a softmax distribution p i := e p i / ∑ n j e p j and will be jointly trained with model weights. The output can be formulated as y = ∑ n i G ( x ) · E i ( x ) , where gate function G ( x ) = p i ( x ) · ✶ { p i ( x ) ≥ p j ( x ) , ∀ j = i } , n and E i denote the number of experts and i -th expert, respectively. Specifically, there are two obstacles when implementing such MoE in TVM:

Dynamic Input Allocation . The input allocation is determined dynamically during runtime, and the trainable router within the MoE layers automatically learns this dispatch assignment as the gradients decay. We know the allocation only when the model is executed and we receive the router outputs. Therefore, the shape of expert input and corresponding indexes are dynamically changed. This can be handled by PyTorch with dynamic graphs while TVM expects static input shape. To this end, we follow and leverage the compiler support for dynamism as proposed in Nimble [49] on top of the Apache TVM to handle the dynamic input allocation.

Parallel Computing . Different experts are expected to run in parallel, this can be supported by several customized distributed training frameworks integrated with PyTorch, e.g., FasterMoE [24], and DeepSpeed [44]. However, it remains nontrivial to support this using TVM. One option to simulate is to perform modularized optimization to mimic parallel computing, in particular, we optimize each expert separately and measure their latency, the maximum latency among all experts will be recorded and regarded as the latency of this MoE layer, and the aggregated total of the time spent for each layer is the final reported modularized model latency. To avoid any potential confusion between real-measured wall-clock time, i.e., no parallelism assumed, and simulated modularized latency, i.e., ideal parallelism assumed, we reported both for models containing MoE layers in Sec. 5 to offer more insights into the algorithm's feasibility and cost implications.

Latency-aware Load-balancing Loss. The key to our MoE framework is the design of a routing function to balance all experts towards higher accuracy and lower latency. Previous solutions [20, 23] use homogeneous experts and treat them equally. While in our MoE framework, the divergence and heterogeneity between powerful yet slow Mult. and fast yet less powerful Shift experts incur one unique question: How to orchestrate the workload of each expert to reduce the synchronization time? Our answer is a latency-aware load-balancing loss, which ensures (1) all experts receive the expected weighted sums of gate values; and (2) all experts are assigned the expected number of input tokens. Those two conditions are enforced by adopting importance and load balancing loss as defined below:

̸

where SCV denotes the squared coefficient of variation of given distributions over experts (also used in [46, 48]); α i refers to the latency-aware coefficient of i -th expert and is defined by ( Lat i ) / ( ∑ j Lat j ) because that the expected assignments are inversely proportional to runtime latency of i -th expert, Lat i ; q i ( x ) is the probability that the gate value of i -th expert outweighs the others, i.e., top1 expert, and is given by P ( p i ( x ) + ϵ ≥ p j ( x ) , ∀ j = i ) . Note that this probability is discrete, we use a noise proxy ϵ following [48, 23] to make it differentiable. The above latency-aware importance loss and load-balanced loss help to balance gate values and workload assignments, respectively, and can be integrated with classification loss L CLS as the total loss function L ( X ) = L CLS ( X ) + λ · ( L IMP ( X ) + L LOAD ( X )) for training ViTs and gates simultaneously. λ is set as 0.01 for all experiments.

5 Experiments

5.1 Experiment Settings

Models and Datasets. Tasks and Datasets. We consider two representative 2D and 3D Transformerbased vision tasks to demonstrate the superiority of the proposed ShiftAddViT, including 2D image classification on ImageNet dataset [15] with 1.2 million training and 50K validation images and 3D novel view synthesis (NVS) task on Local Light Field Fusion (LLFF) dataset [40] with eight scenes. Models. For the classification task, we consider PVTv1 [60], PVTv2 [61], and DeiT [55]. For the NVS task, we consider Transformer-based GNT [53] with View- and Ray-Transformer models.

Training and Inference Details. For the classification task, we follow Ecoformer [34] to initialize the pre-trained ViTs with Multihead Self-Attention (MSA) weights, based on which we apply our reparameterization a two-stage finetuning: (1) convert MSA to linear attention [73] and reparameterize all MatMuls with add layers with 100 epoch finetuning, and (2) reparameterize MLPs or linear layers with shift or MoE layers after finetuning another 100 epoch. Note that we follow PVTv2 [61] and Ecoformer to keep the last stage as MSA for fair comparisons. For the NVS task, we still follow the two-stage finetuning but do not convert MSA weights to linear attention to maintain the accuracy. All experiments are run on a server with eight RTX A5000 GPUs with each having 24GB GPU memory.

Baselines and Evaluation Metrics. Baselines. For the classification task, we reparameterize and compare our ShiftAddViT with PVTv1 [60], PVTv2 [61], Ecoformer [34], and their MSA variants. For the NVS task, we reparameterize on top of GNT [53] and compare with both vanilla NeRF [41]

and GNT [53]. Evaluation Metrics. For the classification task, we evaluate the ShiftAddViT and baselines in terms of accuracy, GPU latency and throughput measured on an RTX 3090 GPU. For the NVS task, we evaluate all models in terms of PSNR, SSIM [63], and LPIPS [76]. We also measure and report the energy consumption of all the above models based on an Eyeriss-like hardware accelerator [12, 77], which calculates not only computational but also data movement energy.

5.2 ShiftAddViT over SOTA Baselines on 2D Tasks

To evaluate the effectiveness of our proposed techniques, we apply the ShiftAddViT idea to five commonly used ViT models, including DeiT [55] and various variants of PVTv1 [60] and PVTv2 [61]. We compare their performance with baselines on the image classification task. Tab. 3 highlights the overall comparison with the most competitive baseline, Ecoformer [34], from which we see that ShiftAddViT consistently outperforms all baselines in terms of accuracy-efficiency tradeoffs, achieving 1.74 × ∼ 5.18 × latency reduction on GPUs and 19.4 % ∼ 42.9 %energy savings measured

on the Eyeriss accelerator [12] with comparable or even better accuracy ( ↑ 0.04 % ∼↑ 0.20 %). Note that we apply MoE to both linear and MLP layers. Moreover, we further provide comparisons of a variety of ShiftAddViT variants and PVT with or without multi-head self-attention (MSA) and MoE in Tab. 4 and 6. We observe that our ShiftAddViT can be seamlessly integrated with linear attention [73, 65], simple or advanced Q / K quantization [34, 27], and the proposed MoE framework, achieving up to 3.06 × / 4.14 × / 8.25 × and 2.47 × / 1.10 × / 2.09 × latency reductions and throughput improvements after the TVM optimization on an RTX 3090 as compared to MSA, PVT, and PVT+ MoE (note: two Mult. experts) baselines, respectively, under comparable accuracies, i.e., ± 0.5%.

Training Wall-clock Time. To offer more insights into the training cost, we collect the training wall-clock time, which is 21% ∼ 25% less than training from scratch as the original Transformer model and more than 50 × less than the previous ShiftAddNet [70] did on GPUs. For example, training the PVTv1-Tiny model from scratch takes 62 hours while our finetuning only needs 46 hours.

Linear Attention vs. MSA. We ensure batch size (BS) is 1 for all latency measurements and find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA. The reasons are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWConv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits. To validate both points, we measure the latency of PVTv2-B0 with various BS and input resolutions as shown in the Appendix F, the results indicate that linear attention's benefits show up under larger BS or input resolutions.

5.3 ShiftAddViT over SOTA Baselines on 3D Tasks

We also extend our methods to 3D NVS tasks and built ShiftAddViT on top of Transformer-based GNT models [53]. We test and compare the PSNR, SSIM, LPIPS, latency, and energy of both ShiftAddViT and baselines, i.e., vanilla NeRF [41] and GNT [53], on the LLFF dataset across eight scenes. As shown in Tab. 5, our ShiftAddViT consistently leads to better accuracy-efficiency tradeoffs, achieving 22.3 %/ 50.4 %latency reductions and 20.8 %/ 54.3 %energy savings under comparable or even better generation quality ( ↑ 0.55/ ↓ 0.19 averaged PSNR across eight scenes), as compared to NeRF and GNT baselines, respectively. Note that GNT costs more than NeRF because of an increased number of layers. In particular, ShiftAddViT even achieves better generation quality (up to ↑ 0.88 PSNR) than GNT on some representation scenes, e.g., Orchid and Flower, as highlighted in Tab. 5, where the first , second, and third ranking performance are noted with corresponding colors. The full results of eight scenes are supplied in the Appendix C. Note that we only finetune ShiftAddViT for 140K steps on top of pre-trained GNT models instead of training up to 840K steps from scratch.

5.4 Ablation Studies of ShiftAddViT

Performance Breakdown Analysis. To investigate how each of the proposed techniques contributes to the final performance, we conduct ablation studies among various kinds of ShiftAddViT variants on PVTv1-T, PVTv2-B0/B1/B2 models to gradually break down and show the advantage of each component. As shown in Tab. 4 and 6, we have three general observations: (1) ShiftAddViT is robust to the binarization method of Q / K , e.g., it is compatible with either KSH [34] or vanilla binarization [27], achieving on average 3.03 × latency reductions than original PVT under comparable accuracies ( ± 0 . 5% ), and both 3.29 × / 1.32 × latency/energy reductions and 0.04% ∼ 0.20% accuracy improvements over Ecoformer [34]; (2) vanilla binarization methods work better than KSH in our ShiftAddViT framework in terms of both accuracy ( ↓ 0.05% ∼↑ 0.21%) and efficiency (on average ↓ 5.9% latency reductions and ↑ 5.8% throughput boosts). Moreover, KSH requires that Q and K are identical while vanilla binarization [27] does not have such a limitation. That explains why applying vanilla quantization to the linear attention layers results in an accuracy improvement of on average 0.15% as compared to using KSH. Also note that our adopting linear attention following [73] works better ( ↑ 0.29%) than PVT [60, 61]; (3) Replacing linear layers in attention modules or MLPs with Shift layers lead to an accuracy drop while our proposed MoE can help to compensate for it. Specifically, adopting Shift layers leads to on average 1.67% accuracy drop while MoE instead improves on average 1.37% accuracy. However, MoE hurts the efficiency of both the PVT baseline

[B, K, M, N] =

[1, 64, 3136, 512]

[B, K, M, N] =

[1, 128, 784, 1024]

[B, K, M, N] =

[1, 512, 3136, 64]

[B, K, M, N] =

[1, 1024, 784, 128]

0.00

0.03

0.05

0.08

0.10

0.12

0.15

Latency (ms)

Pytorch FakeShifts

TVM FakeShifts

TVM MatMuls

TVM

MatShifts

(Ours)

[B, H, K, M, N] =

[1, 5, 196, 32, 16]

[B, H, K, M, N] =

[1, 2, 784, 32, 16]

[B, H, K, M, N] =

[1, 2, 784, 64, 64]

0.00

0.01

0.02

0.03

0.04

0.05

Latency (ms)

TVM MatMuls

TVM

MatAdds

(Ours)

and our ShiftAddViT without customized system acceleration, e.g., PVT+MoE increases 94% latency and reduces 51% throughput than PVT on GPUs, due to limited parallelism supports, especially for TVM. We report the modularized latency by separately optimizing each expert to demonstrate the potential of MoE's double-winning accuracy ( ↑ 0.94% ∼↑ 2.02%) and efficiency ( ↓ 15.4% ∼↑ 42.7%).

In addition, we want to clarify that the seemingly minor latency improvements of adopting shifts in Tab. 4 and 6 are due to full optimization of the compiled model as a whole (e.g., 6.34ms → 1ms for PVTv2-B0) on GPUs with sufficient chip area. Most gains are concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1 and the Appendix F. Under the same chip areas, latency savings of adopting shifts are more pertinent, e.g., PVTv2-B0 with shift or MoE achieve 3.9 ∼ 5.7 × or 1.6 ∼ 1.8 × speedups.

Apart from 2D tasks, we also report the performance breakdown on 3D NVS tasks, as shown in Tab. 5. The above three observations still hold except for the Shift layer, as we find that ShiftAddViT with all linear layers or MLPs replaced by Shift layers achieve even better PSNR ( ↑ 0.13 ∼ ↑ 0.20), SSIM ( ↑ 0.005 ∼ ↑ 0.007), and LPIPS ( ↓ 0.007 ∼ ↓ 0.009) than other variants. In addition, we profile on an Eyeriss

0

20

40

60

80

100

Energy Percentage (%)

Ours

GNT

Ours

DeiT

42.9% Saving

40.9% Saving

MatMuls

Adds

Linears

Shifts

Others

accelerator to show the energy breakdown of both ShiftAddViT and baselines. As shown in Fig. 3, our ShiftAddViT reduced 42.9 % and 40.9 % energy on top of DeiT-T and GNT, respectively. Among them applying Add layers lead to 93.8 %and 63.8 %energy reductions than original MatMuls , Shift layers help to reduce 28.5 %and 37.5 %energy than previous linear/MLP layers. This set of experiments validates the effectiveness of each component in our proposed ShiftAddViT framework.

Speedups of Shifts and Adds. Apart from the overall comparison and energy reduction profiling, we also test the GPU speedups of our customized Shift and Add kernels, as shown in Fig. 4 and 5, respectively. We can see that both customized kernels achieve faster speeds than PyTorch and TVM baselines. For example, our MatAdds achieve on average 7.54 × / 1.51 × speedups than PyTorch and TVM MatMuls , respectively. Our MatShifts achieve on average 2.35 × / 3.07 × / 1.16 × speedups than PyTorch FakeShifts [17], TVM FakeShifts , and TVM MatMuls , respectively. More comparisons and speedup analysis under larger batch sizes are supplied in the Appendix A. Note that in those comparisons, we use the default einsum operator for the PyTorch MatMuls , floatingpoint multiplication with power-of-twos for FakeShift, and implement the MatAdds and MatShifts using TVM [10] ourselves. We compare those customized operators with their multiplication counterparts in the same setting for fair comparisons. The speedup of MatAdds is because we replace multiplication-and-accumulation with accumulation only. The speedup of MatShifts is mainly attributed to the bit reductions ( INT32 and INT8 for inputs and shift signs/weights) and thus data movement reductions instead of computations which are almost fully hidden behind data movements.

MoE's Uniqueness in ShiftAddViT. Our contributions are the multiplication-reduced ViTs and a new load-balanced MoE framework. They are linked in one organic whole. Unlike previous MoE works where all experts are of the same capacity and there is no distinction between tokens, our MoE layers combine unbalanced experts, i.e., multiplication and shift, with a hypothesis to divide important object tokens and background tokens. Such a new MoE design offers a softer version of

Image

Token Gate

Image

Token Gate

Image

Token Gate

Image

Token Gate

SWA

ShiftAddViTs, i.e., instead of aggressively replacing all multiplications with cheaper shifts and adds, we keep the powerful multiplication option to handle importance tokens for maintaining accuracy while leaving all remaining unimportant tokens being processed by cheaper bitwise shifts, winning a better accuracy and efficiency tradeoff.

Ablation studies of Latency-aware Load-balancing Loss (LL-Loss). To demonstrate the effectiveness of the proposed LL-Loss in our ShiftAddViT MoE framework, we conduct ablation studies on two PVT models as shown in Tab. 7. We find that ShiftAddViT w/ LLLoss achieves better accuracy-efficiency tradeoffs, e.g., 14.6 %latency reductions while maintaining comparable accuracies ( ± 0.1%) when considering Mult. and Shift

experts. The latency reduction could be larger if we consider more unbalanced experts with distinct runtimes. This set of experiments justifies the effectiveness of our LL-Loss in the MoE system.

Visualization of Token Dispatches in MoE. To validate our hypothesis that important yet sensitive tokens require powerful experts while other insensitive tokens can be represented with much cheaper Shift experts, we visualize the token dispatches in MoE routers/gates of the first MLP layer in our ShiftAddViT PVTv2-B0 model as shown in Fig. 6. We see that our designed router successfully identifies the important tokens that contain object information, which are then dispatched to more powerful Multiplication experts, leaving unimportant tokens, e.g., background, to cheaper Shift tokens. This set of visualization further validates the hypothesis and explains why our proposed MoE framework can effectively boost our ShiftAddViT towards better accuracy-efficiency trade-offs.

5.5 Limitation and Societal Impact Discussion

We made a firm step to show the practical usage of the proposed hardware-inspired ShiftAddViT. While this relies on dedicated TVM optimization, the full potential can be unraveled with customized hardware accelerators toward naturally hardware-efficient ViTs. Also, the unbalanced MoE framework shows its generalizability but highly demands system support with ideal parallelism.

6 Conclusion

In this paper, we for the first time propose a hardware-inspired multiplication-reduced ViT model dubbed ShiftAddViT. It reparameterizes both attention and MLP layers in pre-trained ViTs with a mixture of multiplication primitives, e.g., bitwise shifts and adds, towards end-to-end speedups on GPUs and dedicated hardware accelerators without the need for training from scratch. Moreover, a novel mixture of unbalanced experts framework equipped with a new latency-aware load-balancing loss is proposed in pursuit of double-winning accuracy and hardware efficiency. We use multiplication or its primitives as experts in our ShiftAddViT cases. Extensive experiments on both 2D and 3D Transformer-based vision tasks consistently validate the superiority of our proposed ShiftAddViT as compared to multiple ViT baselines. We believe that this paper opens up a new perspective on designing energy-efficient ViT inference based on widely available pre-trained ViT models.

References

Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.

Moab Arar, Ariel Shamir, and Amit H Bermano. Learned queries for efficient local attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10841-10852, 2022.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. arXiv preprint arXiv:2209.07484, 2022.

Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution lowcomputation visual recognition. arXiv preprint arXiv:2205.14756, 2022.

Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413-17426, 2021.

Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.14899, 2021.

Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1468-1477, 2020.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12270-12280, 2021.

Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv:1802.04799, 2018.

Xinghao Chen, Chang Xu, Minjing Dong, Chunjing Xu, and Yunhe Wang. An empirical study of adder neural networks for object detection. Advances in Neural Information Processing Systems, 34:6894-6905, 2021.

Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE journal of solid-state circuits, 52(1):127-138, 2016.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.

Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. Deepshift: Towards multiplication-less neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2359-2368, 2021.

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824-6835, 2021.

Hongxiang Fan, Thomas Chau, Stylianos I Venieris, Royson Lee, Alexandros Kouris, Wayne Luk, Nicholas D Lane, and Mohamed S Abdelfattah. Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 599-615. IEEE, 2022.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-5270, 2022.

Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01136, 2021.

Bah-Hwee Gwee, Joseph S Chang, Yiqiong Shi, Chien-Chung Chua, and Kwen-Siong Chong. A lowvoltage micropower asynchronous multiplier with shift-add multiplication approach. IEEE Transactions on Circuits and Systems I: Regular Papers, 56(7):1349-1359, 2008.

Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-ofexpert training system. arXiv preprint arXiv:2103.13262, 2021.

Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 120-134, 2022.

Yefei He, Zhenyu Lou, Luoming Zhang, Weijia Wu, Bohan Zhuang, and Hong Zhou. Bivit: Extremely compressed binary vision transformer. arXiv preprint arXiv:2211.07091, 2022.

Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In International Conference on Computer Vision (ICCV), 2021.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-6898, 2017.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerarithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.

Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local binary convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 19-28, 2017.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.

Wenshuo Li, Xinghao Chen, Jinyu Bai, Xuefei Ning, and Yunhe Wang. Searching for energy-efficient hybrid adder-convolution neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1943-1952, 2022.

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 48044814, 2022.

Xiaocong Lian, Zhenyu Liu, Zhourui Song, Jiwu Dai, Wei Zhou, and Xiangyang Ji. High-performance fpga-based cnn accelerator with block-floating-point arithmetic. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 27(8):1874-1885, 2019.

Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bohan Zhuang. Ecoformer: Energy-saving attention with linear complexity. In NeurIPS, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.

Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer. arXiv preprint arXiv:2205.13016, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.

Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297-21309, 2021.

Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.

Zheng Qu, Liu Liu, Fengbin Tu, Zhaodong Chen, Yufei Ding, and Yuan Xie. Dota: detect and omit weak attentions for scalable transformer acceleration. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 14-26, 2022.

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International Conference on Machine Learning, pages 18332-18346. PMLR, 2022.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937-13949, 2021.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.

Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. Nimble: Efficiently compiling dynamic neural networks for model inference. Proceedings of Machine Learning and Systems, 3:208-222, 2021.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.

Han Shu, Jiahao Wang, Hanting Chen, Lin Li, Yujiu Yang, and Yunhe Wang. Adder attention for vision transformer. Advances in Neural Information Processing Systems, 34:19899-19909, 2021.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.

Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations, 2023.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021.

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 459-479. Springer, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

Guangting Wang, Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2423-2430, 2022.

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 568-578, October 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.

Yunhe Wang, Mingqiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2101.10015, 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.

Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9127-9135, 2018.

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138-14148, 2021.

Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing XU, and Yunhe Wang. Kernel based progressive distillation for adder neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 12322-12333. Curran Associates, Inc., 2020.

Ping Xue and Bede Liu. Adaptive equalizer using finite-bit power-of-two quantizer. IEEE transactions on acoustics, speech, and signal processing, 34(6):1603-1611, 1986.

Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809-10818, 2022.

Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662, 2019.

Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. Advances in Neural Information Processing Systems, 33:2771-2783, 2020.

Haoran You, Baopu Li, Shi Huihong, Yonggan Fu, and Yingyan Lin. Shiftaddnas: Hardware-inspired search for more accurate and efficient neural networks. In International Conference on Machine Learning, pages 25566-25580. PMLR, 2022.

Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator codesign. In The 29th IEEE International Symposium on High-Performance Computer Architecture (HPCA 2023), 2023.

Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Lin. Castling-vit: Compressing self-attention via switching towards linear-angular attention during vision transformer inference. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023.

Zhongzhi Yu, Yonggan Fu, Sicheng Li, Chaojian Li, and Yingyan Lin. Mia-former: efficient and robust vision transformers via multi-grained input-adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8962-8970, 2022.

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.

Yang Zhao, Chaojian Li, Yue Wang, Pengfei Xu, Yongan Zhang, and Yingyan Lin. Dnn-chip predictor: An analytical performance predictor for dnn accelerators with various dataflows and hardware architectures. In International Conference on Acoustics, Speech, and Signal Processing, pages 1593-1597, 05 2020.

Zhe Zhou, Junlin Liu, Zhenyu Gu, and Guangyu Sun. Energon: Toward efficient acceleration of transformers using dynamic sparse attention. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(1):136-149, 2022.

[B, K, M, N] =

[32, 64, 3136, 512]

[B, K, M, N] =

[32, 128, 784, 1024]

[B, K, M, N] =

[32, 512, 3136, 64]

[B, K, M, N] =

[32, 1024, 784, 128]

0.0

0.5

1.0

1.5

2.0

2.5

Latency (ms)

TVM FakeShift

TVM MatMul

TVM

MatShift

(Ours)

[B, H, K, M, N] =

[32, 5, 196, 32, 16]

[B, H, K, M, N] =

[32, 2, 784, 32, 16]

[B, H, K, M, N] =

[32, 1, 3136, 32, 16]

0.00

0.03

0.05

0.08

0.10

0.12

Latency (ms)

PyTorch MatMul

TVM MatMul

TVM

MatAdd

(Ours)

A More Speedup Comparisons of Shift and Add

Apart from the speedup analysis when the batch size is one, we also conducted GPU speedup tests on our customized Shift and Add kernels when using larger batch sizes, such as 32, as depicted in Fig. 7 and 8, respectively. The results demonstrate that both our customized kernels consistently outperform the PyTorch and TVM baselines in terms of speed. For instance, our MatAdds achieve an average speedup of 2.80 × / 1.65 × compared to PyTorch and TVM MatMuls , respectively. Similarly, our MatShifts achieve an average speedup of 3.38 × / 1.21 × compared to TVM FakeShifts [17] and TVM MatMuls , respectively. It is important to note that in these comparisons, we ensured fair conditions by using the default einsum operator for PyTorch MatMuls , employing floating-point multiplication with power-of-twos for FakeShift, and implementing the MatAdds and MatShifts ourselves using TVM [10]. We compared these customized operators with their multiplication counterparts in the same setup to ensure a fair evaluation. The observed speedup of MatAdds is due to the replacement of multiplication-and-accumulation with accumulation only. On the other hand, the speedup of MatShifts is primarily attributed to the reduction in bit precision ( INT32 and INT8 for inputs and shift signs/weights, respectively), resulting in reduced data movement instead of computations. These computational improvements are almost entirely hidden behind data movements.

B More Visualization of ShiftAddViT MoE Framework

In order to validate our hypothesis that important yet sensitive tokens require powerful experts, and other insensitive tokens can be represented by cheaper Shift experts, we have conducted additional token dispatch visualizations in the MoE routers/gates of the first MLP layer in our ShiftAddViT PVTv2-B0 model, as illustrated in Fig. 9. These visualizations provide further evidence that our designed routers successfully identify the crucial tokens that contain object information. Consequently, these tokens are dispatched to more powerful Mult. experts, while less important tokens, such as background tokens, are assigned to cheaper Shift experts. This collection of visualizations further reinforces our hypothesis and elucidates why our proposed MoE framework effectively enhances the accuracy-efficiency trade-offs of our ShiftAddViT model.

C More Results on 3D Tasks

We have extended our methods to 3D NVS tasks and have built ShiftAddViT on top of Transformerbased GNT models [53]. In the main paper, we tested and reported the averaged PSNR, SSIM, LPIPS, as well as latency and energy of both ShiftAddViT and baselines, i.e., the vanilla NeRF [41] and GNT [53], on the LLFF dataset across eight scenes. Here, we further supply the full results of eight scenes. As shown in Tab. 8, 9, 10, our ShiftAddViT consistently leads to better accuracyefficiency tradeoffs in terms of the three metrics mentioned above, achieving comparable or even better generation quality ( ↑ 0.55/ ↓ 0.19 averaged PSNR across eight scenes), as compared to NeRF and GNT baselines, respectively. In particular, ShiftAddViT achieves better generation quality (up to ↑ 0.88 PSNR) than GNT on some representation scenes, e.g., Orchid and Flower. In all tables, the first , second, and third ranking performances are noted with corresponding colors.

In addition, we supply more qualitative rendering visualization on three scenes, Horns, Orchids, and Flowers, as shown in Fig. 10. We see that our ShiftAddViT generates more clear images than vanilla

Image

Token Gate

Image

Token Gate

Image

Token Gate

Image

Token Gate

(b) GNT

(c) Ours

(a) NeRF

Horns

Flowers

Orchids

NeRF [41] and comparable or even better details than GNT [53]. Quantitative and qualitative results consistently show that our ShiftAddViT does not hurt rendering quality when improving rendering speeds and energy efficiency.

D More Results on NLP Tasks

We also test our proposed optimization of MHSA and MLPs to NLP tasks. In particular, we apply our methods to Transformer models and Long Range Arena (LRA) benchmarks consisting of sequences ranging from 1K to 16K tokens [54]. The results are shown in Tab. 11. We see that our proposed ShiftAdd Transformers consistently show superior performance in terms of both model accuracy (+0.45% ∼ +5.79%) and efficiency (1.5 × ∼ 11.5 × latency reductions and 2.2 × ∼ 16.4 × energy reductions on an Eyeriss-like accelerator) as compared to original Transformer and other linear

attention baselines, which means that our shift and add reparameterization and load-balanced MoE ideas are generally applicable to Transformer models and are agnostic to domains and tasks.

E More Experiment Settings

For the classification task, we first replace the original attention layers in PVTv1 [60] and PVTv2 [61] with the standard MSA, and then finetune the MSA-based variants on top of the ImageNet1 kpretrained weights for 100 epochs with a learning rate (lr) of 5 × 10 -5 following Ecoformer [34]. After that, we reparameterize the MSA-based models through a two-stage finetuning process: (1) convert MSA to linear attention [73] and reparameterize MatMuls with add layers via 100 epochs of finetuning with a base lr of 1 × 10 -5 , and (2) reparameterize MLPs or linear layers with shift or MoE layers through another 100 epochs of finetuning with a base lr of 1 × 10 -5 . All models are trained using eight RTX A5000 GPUs with a total batch size of 256 , and we use the AdamW optimizer [37] with a cosine decay lr scheduler. All other hyperparameters are the same as those in Ecoformer [34] and PVTv2 [61]. For the NVS task, we apply reparameterization on top of the pretrained models from GNT [53] using a similar two-stage fine-tuning process, except that we do not convert MSA to linear attention for maintaining accuracy. Specifically, we first (1) reparameterize MatMuls with add layers and then (2) reparameterize MLPs or linear layers with shift or MoE layers. Both stages are finetuned for 140 K steps with a base lr of 5 × 10 -4 , and we sample 2048 rays with 192 coarse points sampled per ray in each iteration. All other hyperparameters are the same as those in GNT [53], including the use of the Adam optimizer with an exponential decay lr scheduler.

Scaling Factor Clarification. For shift layers, we reparameterize shifts following DeepShift-PS [17] and do not use a scaling factor. A straight-through estimator (STE) [69] is adopted to make the shift layer differentiable. For ShiftAddViT with KSH [34], there is no scaling factor needed as a set of

hash functions is applied to convert Q/K to binary codes. For ShiftAddViT with Quant. [27], we leverage layer-wise Quant. for both Q & K, the scaling factor can be multiplied after add operations. It can be efficiently implemented following [28].

F More Ablation Studies

Linear Attention vs. MSA under Various Batch Sizes and Input Resolutions. We find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA in the Tab. 4 and 6. To investigate the hidden reasons, we further measure the latency of PVTv2-B0 with various BS and input resolutions, the results are summarized in the Tab. 12. From that, we observe that linear attention's benefits show up under larger BS or input resolutions. The hidden reasons for this phenomenon are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWConv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits.

The Advantage of Using Shift. Replacing Multiplication-based MLPs with shift layers significantly reduces latency, energy, and chip area costs. Our shift kernel offers an average speedup of 2.35 × /3.07 × /1.16 × compared to PyTorch FakeShift, TVM FakeShift, and TVM MatMuls, respectively, as shown in Fig. 3. The seemingly minor latency improvements in Tab. 4 and 6 are due to full optimization of the compiled model as a whole (e.g., 6.34ms → 1ms for PVTv2-B0) on GPUs with sufficient chip area. Most gains are concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1. Under the same chip areas, latency savings are more pertinent, e.g., PVTv2-B0 w/ shift or MoE achieve 3.9 ∼ 5.7 × and 1.6 ∼ 1.8 × speedups, respectively, as summarized in Tab. 13.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient accelerating framework for vision transformer, ShiftAddViT, which reparameterizes pre-trained ViTs with a mixture of complementary multiplication primitives and MoE designs. Specifically, All MatMuls in self-attention modules are reparameterized by additive kernels, and the remaining linear layers and MLPs are reparameterized by shift kernels. Besides, they develop a new MoE system for maintaining accuracy after reparameterization, and use a latency-aware load-balancing loss term to assign a dynamic amount of input tokens to each expert. Extensive experiments on various 2D/3D Transformer-based vision models demonstrate their superiority and efficiency.\n\nSTRENGTHS:\nThis paper is well-written and easy to follow. Converting multiplication operations to additive and shift is a promising technique in model compression and accleration, this paper provides a systematic design solutions for vision transformer architectures, which is interesting.\nExperiments in this paper are sufficient, including 2D and 3D task., and they provides detailed latency comparison for different methods, which much helps prove their effectiveness.\n\nWEAKNESSES:\nShiftAddViT is a complex system design solution, where the acceleration comes from the replacing of additive and shift operations with multiplication, the maintaining accuracy relies on the MoE design. And ShiftAddViT utils TVM to implement and optimize their customized kernels for practical hardware deployment on GPUs. In this paper, the contributions of multiplation less and MoE are much independent, this paper  seems not much suitable for algorithm-based conferences like NeurIPS/CVPR/ICML.\n\nQUESTIONS:\n1. What is the most important contributions in this paper？Multiplacation-less solution and MoE seems indepedent, It is a common sence that MoE could boost the performance on various tasks, improving accuracy with MoE seems unnecessary in this paper. \n\n2. Re-parameterizing usually refers to one kind of mathematical equivalence transformations, such as RepVGG [1], AC-Net [2], what is the re-parameterizing in this paper?\n\n3. ShiftAddNet [3] provides the first scheme for the replacing of shift and additive opration with multiplication, so what is the weakness for comination of ShiftAddNet  and ViT architectures, could you show the comparision with ShiftAddViT.\n\n4. When quantized to 4/8-bit, original ViT models could also achieve high performance and better acceleration, could you compare the quantized ViT with ShiftAddViT? \n\n5. More complex task is better, such as detection and segementation.\n\n[1] Ding X, Zhang X, Ma N, et al. Repvgg: Making vgg-style convnets great again[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 13733-13742.\n\n[2] Ding X, Guo Y, Ding G, et al. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1911-1920.\n\n[3]You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783.\n\nLIMITATIONS:\nThis paper provides the the limitation and societal impact discussion for their proposed technique.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose re-parameterizing ViTs to speed up inference without a full retraining. To do this, they introduce a new operation ShiftAddViT, which works well when applied to attention. For the MLP in a transformer, the authors use a mixture of experts. This operation reduces latency and energy usage, while maintaining accuracy.\n\nSTRENGTHS:\n- The premise of leveraging power-of-two multiplications (a shift) is clever and a well-stated intuition (justified by Table 1 and also in L150-156). The additional energy analysis and chip-area statistics build ethos very well.\n- The paper presents a very thorough set of experiments on ImageNet classification and novel view synthesis. The latency analysis is thorough and well-documented / reasoned.\n\nWEAKNESSES:\n- My only qualm is that (it appears) this relies on the existence of an effective binary quantization algorithm in L176. The results convince me that this binary quantization is acceptable, but this seems aggressive (as the authors mention later, when replacing the MLP).\n- From the introduction, I initially thought this re-parameterization would be arithmetically identical to the original. After reading the methods, it appears this isn't the case? In which case, wouldn't you need some fine-tuning? If so, how costly is the fine-tuning? If I'm wrong, this may be because I don't quite understand Figure 2 -- how is this related to the original matmul?\n\nQUESTIONS:\n- I would've thought that the ShiftAddViT technique can really be applied to any operation (Clearly, that's not the case, as you added a modification for MLPs.) Why doesn't this work out of the box for any matrix multiplication?\n- What is \"ideal parallelism\" in the footnote for Table 4? Are these latencies not actually measured end-to-end? If this is the case, do the baseline latency measurements also use the same assumption of ideal parallelism? How fast/slow is the MoE kernel when plugged into the network and *entire network is measured end-to-end?\n\nLIMITATIONS:\nSee above.\n\nThe evaluation is thorough, and the method is clearly presented. Despite the questions and objections I had above, I feel this is a cogently stated idea and clearly a thoroughly-investigated problem. I do still have some questions about the method and would like to know the answers, but the idea alone is definitely worth publishing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new type of multiplication-reduced model ShiftAddViT, which use the additive kernels to reparameterize the batched GEMM in the attention block and uses the shift kernels to reparameterize other MLPs or linear layers. In this way, it can reduce energy-intensive multiplications. The authors utilize TVM to implement and optimize those kernels and achieve a speedup on GPUs, and also energy savings.\n\nSTRENGTHS:\n- The proposed ShiftAddViT achieves significant acceleration across different models on different tasks.\n- Experiments show the real speedup on GPUs to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n- Many implementation details of the proposed method are not clear enough.\n- It is unfair for the comparison with DeiT. The proposed method first modified the model architecture, and then introduced the shift and add kernel to improve the model efficiency. When compared with DeiT, the architectures of those two models are not the same. The comparison should be made under the condition that the model architectures are kept the same.\n- Based on Table 3 and Table 4, it appears that the “Shift” was not finally used in the classification task. And in Table 4, compared with the method that only enables “Add”, the final result (highlighted with red background) does not have any advantages in both model accuracy and latency. This seems to demonstrate that the “Shift” and “MoE” methods are not effective on classification tasks.\n- Table 4 and Table 5 lack the result of method “Quant.”+”Shift”+”MoE”. It would be better if these results could be provided to see the impact on model accuracy and latency when all the proposed methods are adopted.\n\nQUESTIONS:\nAbout shift and add:\n- I think the Shift method and the power-of-two quantization should be similar. But there is no scaling factor in the Shift operation (equation 3). In this way, how to convert weights in pre-trained models to the shift layers format? Is the nearest neighbor method directly adopted? Wouldn't the reconstruction error be too large without a scaling factor?\n- How many bits of the P is used in the shift layers? And during finetuning, is the STE method used to solve the gradient backpropagation for s and P?\n- Both two binary methods KSH[32] and Quant[26] use the scaling factors, but those scaling factors are not shown in Figure 1. During inference, where to multiply those scaling factors? Just after the add operation?\n- How many bits are used for the input activations of the ShiftAddViT attention? Since only the fixed-point activations can be shifted, the input activation of shift layers needs to be quantized to an integer first. However, it seems that the article does not mention anything related to activation quantization. And there is no step related to the activation quantization in the two-stage finetuning process. In addition, if the activations are quantized, under the shift&add paradigm, where to multiply the scaling factor from activation quantization? \n- How to quantize the input of the last Shift in ShiftAddViT Attention to integer values? Those input activations should also be fixed-point numbers, not floating-point.\n\nAbout the MoE framework:\n- In line 223, how to get the gate value $p_i$ based on the input token $x$? $p_i(x):= e^{p_i}/\\Sigma_j^ne^{p_j}$ ? Is this a typo?\n\nAbout the experiments:\n- In Table 4, why the latency of PVT is larger than MHA? Is it because of the different batch sizes? For a fair comparison, it is better to use the same batch size here, since the model with batch_size=1 always gets worse throughput on GPUs. And it is better to show the speedup from the Linear Attention under the fair comparison.\n- What’s the difference between line 5 and line 10 in Table 4? They have the same configs, but different results.\n- Based on Table 4 and Table 6, “Shift” may cause an accuracy decrease, and it is necessary to introduce the “MoE” method to compensate for the model accuracy, which leads to an increase in latency. So overall, there is no advantage to using the “shift” method in terms of the model accuracy and latency. Does the “shift” method only for energy reduction?\n- Experiments use the Eyeriss accelerator to get energy consumption. What are the configs of this accelerator? What data type is used for this accelerator? When counting energy, is the energy of memory access counted?\n\nLIMITATIONS:\nThe authors discussed the limitations of the implementation of their work. It is necessary to use a customized hardware accelerator to fully leverage the benefits of “Shift” and “Add”.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel reparameterization method for efficient Vision Transformers (ViT). The method replaces the heavy multiplication operations in ViT with a combination of shift and add operations. By mapping queries and keys to binary codes in Hamming space and reparameterizing multi-layer perceptrons (MLPs) or linear layers, the multiplication operations in the model are effectively reduced. Experimental results demonstrate that ShiftAddViT achieves efficient performance on various 2D/3D Transformer visual tasks while achieving latency reduction and energy savings.\n\nSTRENGTHS:\nThis paper presents a clear motivation for each proposed compound. For example, the reparameterization of multiplications is inspired by hardware design practices and the concept of multiple experts (MOE) for dynamically routing tokens into multiplication or shift groups. The three questions posed by the authors clearly indicate the specific problems they are addressing. The writing and demonstration are clear and concise. In terms of results, this work delivers some impressive findings in regard to both efficiency and accuracy. Extensive ablation experiments are conducted to validate the proposed approach further.\n\nWEAKNESSES:\n1. A primary concern is the limited validation of the method proposed by the authors, which is solely performed on small-scale models, with the largest model encompassing 30M and 4G FLOPs. I am particularly interested in the performance of larger-scale models. Naturally, I acknowledge the constraints imposed by computational resources. However, considering the hardware specifications disclosed by the authors, it is apparent that training models of ViT-Base size or similar are feasible.\n\n2. Secondly, my concern lies in the stability of the training process and the associated wall-clock time. Obtaining information regarding the overall training cost and wall-clock time from the authors would offer valuable insights into the algorithm's feasibility and cost implications.\n\nQUESTIONS:\nOverall, I am satisfied with the results and motivation.   I am particularly concerned about whether this method can improve inference performance for larger models. The bottleneck for small models is not particularly high, to begin with, and compared to larger models, the performance of small models will undoubtedly be lower. Therefore, if the authors' method can enable larger models to achieve a significant increase in inference speed, it would greatly enhance the significance of the approach.\n\nLIMITATIONS:\nThe author discussed the limitation in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes ShiftAddViT, which is an efficient ViT reparameterization with a mixture of complementary multiplication primitives, such as bitwise shifts and adds. \nThe alternative parameterization (quantization) is carefully examined to allocate to different components (MHSA, MLP) in ViT. \nIn addition, the authors propose a mixture of experts (MoE) framework to classify input tokens and assign different primitives to best preserve accuracy. \nThe MoE framework is guided by latency-aware load-balancing loss. \nMultiple empirical results are demonstrated including 2D ViT in image classification task, as well as GNT for NVS task.\n\nSTRENGTHS:\n1. The paper is clearly written and easy to follow. \n2. Though shift and add reparameterization is not completely new (binary, ternary or power-of-2 quantization), this is a very pioneer work to systematically investigate layer sensitivity and accuracy impact, allocation strategy, and hardware implementations. The hardware benchmarks with considerable latency and energy savings are impressive. \n3. The experimental analysis is strong. In addition to conventional classification task, the authors also provide results on NVS task.\n\nWEAKNESSES:\n1. Since this work proposes a collection of analysis and optimizations on MHSA and MLP, reducing the computation complexity and energy consumption, I wonder if this work is portable to NLP tasks, especially for LLMs with more tokens? \n2. I cannot fully follow the token dispatching method mentioned in Section 4.2 and Figure 6. The authors did not discuss the methodology for input allocation and parallel computing issues in detail (line#227). Intuitively, if the important and sensitive tokens (yellow) in Figure 6 are fed into powerful experts (MULT.), while the rest are into SHIFT experts, are they still visible to each other (i.e. global receptive field)?\n\nQUESTIONS:\nPlease refer to concerns raised in weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}