SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[72] Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator code design. In The 29th IEEE International Symposium on High-Performance Computer Architecture (HPCA 2023), 2023.
[73] Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Lin. Castling-vit: Compressing self-attention via switching towards linear-angular attention during vision transformer inference. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023.
[74] Zhongzhi Yu, Yonggan Fu, Sicheng Li, Chaojian Li, and Yingyan Lin. Mia-former: efficient and robust vision transformers via multi-grained input-adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8962-8970, 2022.
[75] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch onImagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[76] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[77] Yang Zhao, Chaojian Li, Yue Wang, Pengfei Xu, Yongan Zhang, and Yingyan Lin. Dnn-chip predictor: An analytical performance predictor for dnn accelerators with various dataflows and hardware architectures. In International Conference on Acoustics, Speech, and Signal Processing, pages 1593-1597, 05 2020.
[78] Zhe Zhou, Junlin Liu, Zhenyu Gu, and Guangyu Sun. Energon: Toward efficient acceleration of transformers using dynamic sparse attention. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(1):136-149, 2022.
Figure 7: MLP/Linear latency speedups using shifts, where inputs are of shape  $B \times K \times M$ , weights are of shape  $K \times N$ . All dimensions are set w.r.t. the real dimensions in PVTs [60].
Figure 8: MatMuls latency speedups using adds, where inputs are of shape  $B \times H \times K \times M$ , weights are of shape  $B \times H \times K \times N$ .
A More Speedup Comparisons of Shift and Add
Apart from the speedup analysis when the batch size is one, we also conducted GPU speedup tests on our customized Shift and Add kernels when using larger batch sizes, such as 32, as depicted in Fig. 7 and 8, respectively. The results demonstrate that both our customized kernels consistently outperform the PyTorch and TVM baselines in terms of speed. For instance, our MatAdds achieve an average speedup of  $2.80 \times /1.65 \times$  compared to PyTorch and TVM MatMuls, respectively. Similarly, our MatShifts achieve an average speedup of  $3.38 \times /1.21 \times$  compared to TVM FakeShifts [17] and TVM MatMuls, respectively. It is important to note that in these comparisons, we ensured fair conditions by using the default einsum operator for PyTorch MatMuls, employing floating-point multiplication with power-of-twos for FakeShift, and implementing the MatAdds and MatShifts ourselves using TVM [10]. We compared these customized operators with their multiplication counterparts in the same setup to ensure a fair evaluation. The observed speedup of MatAdding is due to the replacement of multiplication-and-accumulation with accumulation only. On the other hand, the speedup of MatShifts is primarily attributed to the reduction in bit precision (INT32 and INT8 for inputs and shift signs/weights, respectively), resulting in reduced data movement instead of computations. These computational improvements are almost entirely hidden behind data movements.
B More Visualization of ShiftAddViT MoE Framework
In order to validate our hypothesis that important yet sensitive tokens require powerful experts, and other insensitive tokens can be represented by cheaper Shift experts, we have conducted additional token dispatch visualizations in the MoE routers/gates of the first MLP layer in our ShiftAddViT PVTv2-B0 model, as illustrated in Fig. 9. These visualizations provide further evidence that our designed routers successfully identify the crucial tokens that contain object information. Consequently, these tokens are dispatched to more powerful Mult. experts, while less important tokens, such as background tokens, are assigned to cheaper Shift experts. This collection of visualizations further reinforces our hypothesis and elucidates why our proposed MoE framework effectively enhances the accuracy-efficiency trade-offs of our ShiftAddViT model.
C More Results on 3D Tasks
We have extended our methods to 3D NVS tasks and have built ShiftAddViT on top of Transformer-based GNT models [53]. In the main paper, we tested and reported the averaged PSNR, SSIM, LPIPS, as well as latency and energy of both ShiftAddViT and baselines, i.e., the vanilla NeRF [41] and GNT [53], on the LLFF dataset across eight scenes. Here, we further supply the full results of eight scenes. As shown in Tab. 8, 9, 10, our ShiftAddViT consistently leads to better accuracy-efficiency tradeoffs in terms of the three metrics mentioned above, achieving comparable or even better generation quality ( $\uparrow 0.55 / \downarrow 0.19$  averaged PSNR across eight scenes), as compared to NeRF and GNT baselines, respectively. In particular, ShiftAddViT achieves better generation quality (up to  $\uparrow 0.88$  PSNR) than GNT on some representation scenes, e.g., Orchid and Flower. In all tables, the first, second, and third ranking performances are noted with corresponding colors.
In addition, we supply more qualitative rendering visualization on three scenes, Horns, Orchids, and Flowers, as shown in Fig. 10. We see that our ShiftAddViT generates more clear images than vanilla
Figure 9: Visualization of the token dispatches in MoE routers/gates given images from the validation set, where yellow color denotes Multiplication experts and blue color refers to Shift experts.
(b) GNT
Figure 10: Qualitative comparisons between (a) NeRF [41], (b) GNT [53], (c) ShiftAddViT (Ours), and (d) ground truth when tested on the LLFF dataset for single-scene rendering.
(d) Ground Truth
NeRF [41] and comparable or even better details than GNT [53]. Quantitative and qualitative results consistently show that our ShiftAddViT does not hurt rendering quality when improving rendering speeds and energy efficiency.
D More Results on NLP Tasks
We also test our proposed optimization of MHSA and MLPs to NLP tasks. In particular, we apply our methods to Transformer models and Long Range Arena (LRA) benchmarks consisting of sequences ranging from 1K to 16K tokens [54]. The results are shown in Tab. 11. We see that our proposed ShiftAdd Transformers consistently show superior performance in terms of both model accuracy  $(+0.45\% \sim +5.79\%)$  and efficiency  $(1.5\times \sim 11.5\times$  latency reductions and  $2.2\times \sim 16.4\times$  energy reductions on an Eyeriss-like accelerator) as compared to original Transformer and other linear
Table 8: Comparisons between ShiftAddViT and baselines. We apply our Shift&Add techniques on top of the SOTA Transformer-based NeRF model, GNT [53], and report averaged PSNR  $(\uparrow)$  on the LLFF dataset [40] across eight scenes.
Methods	Add	Shift	MoE	PSNR (↑) on LLFF Dataset
Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns
NeRF [41]	-	-	-	32.70	25.17	20.92	31.16	20.36	27.40	26.80	27.45
GNT [53]	-	-	-	32.96	24.31	22.57	32.28	20.67	27.32	28.15	29.62
ShiftAddViT	✓	✗	✗	31.68	24.63	22.34	31.79	20.74	28.02	26.99	28.62
✓	✓(Both)	✗	31.64	24.63	22.34	31.73	20.78	28.05	27.03	28.61
✓	✓(Attn)	✓(MLPs)	31.85	24.65	22.37	31.85	20.73	28.20	27.08	28.62
✗	✓(Both)	✗	32.12	24.35	22.42	32.15	20.84	28.14	27.32	29.09
Table 9: Comparisons between ShiftAddViT and baselines. We apply our Shift&Add techniques on top of the SOTA Transformer-based NeRF model, GNT [53], and report averaged SSIM (↑) on the LLFF dataset [40] across eight scenes.
Methods	Add	Shift	MoE	SSIM (↑) on LLFF Dataset
Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns
NeRF [41]	-	-	-	0.948	0.792	0.690	0.881	0.641	0.827	0.880	0.828
GNT [53]	-	-	-	0.963	0.846	0.852	0.934	0.752	0.893	0.936	0.935
ShiftAddViT	✓	✗	✗	0.949	0.839	0.838	0.922	0.730	0.891	0.916	0.911
✓	✓(Both)	✗	0.948	0.839	0.839	0.923	0.730	0.892	0.917	0.911
✓	✓(Attn)	✓(MLPs)	0.950	0.840	0.841	0.923	0.731	0.894	0.917	0.912
✗	✓(Both)	✗	0.955	0.836	0.841	0.930	0.746	0.896	0.923	0.922
Table 10: Comparisons between ShiftAddViT and baselines. We apply our Shift&Add techniques on top of the SOTA Transformer-based NeRF model, GNT [53], and report averaged LPIPS  $(\downarrow)$  on the LLFF dataset [40] across eight scenes.
Methods	Add	Shift	MoE	LPIPS (↓) on LLFF Dataset
Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns
NeRF [41]	-	-	-	0.178	0.280	0.316	0.171	0.321	0.219	0.249	0.268
GNT [53]	-	-	-	0.060	0.116	0.109	0.061	0.153	0.092	0.080	0.076
ShiftAddViT	✓	✗	✗	0.087	0.141	0.129	0.080	0.182	0.089	0.107	0.110
✓	✓(Both)	✗	0.088	0.142	0.128	0.080	0.182	0.088	0.106	0.110
✓	✓(Attn)	✓(MLPs)	0.086	0.140	0.127	0.079	0.180	0.087	0.104	0.109
✗	✓(Both)	✗	0.076	0.138	0.125	0.070	0.169	0.083	0.097	0.097
attention baselines, which means that our shift and add reparameterization and load-balanced MoE ideas are generally applicable to Transformer models and are agnostic to domains and tasks.
E More Experiment Settings
For the classification task, we first replace the original attention layers in PVTv1 [60] and PVTv2 [61] with the standard MSA, and then finetune the MSA-based variants on top of the ImageNet-1k-pretrained weights for 100 epochs with a learning rate (lr) of  $5 \times 10^{-5}$  following Ecoformer [34]. After that, we reparameterize the MSA-based models through a two-stage finetuning process: (1) convert MSA to linear attention [73] and reparameterize MatMu1s with add layers via 100 epochs of finetuning with a base lr of  $1 \times 10^{-5}$ , and (2) reparameterize MLPs or linear layers with shift or MoE layers through another 100 epochs of finetuning with a base lr of  $1 \times 10^{-5}$ . All models are trained using eight RTX A5000 GPUs with a total batch size of 256, and we use the AdamW optimizer [37] with a cosine decay lr scheduler. All other hyperparameters are the same as those in Ecoformer [34] and PVTv2 [61]. For the NVS task, we apply reparameterization on top of the pretrained models from GNT [53] using a similar two-stage fine-tuning process, except that we do not convert MSA to linear attention for maintaining accuracy. Specifically, we first (1) reparameterize MatMu1s with add layers and then (2) reparameterize MLPs or linear layers with shift or MoE layers. Both stages are finetuned for 140K steps with a base lr of  $5 \times 10^{-4}$ , and we sample 2048 rays with 192 coarse points sampled per ray in each iteration. All other hyperparameters are the same as those in GNT [53], including the use of the Adam optimizer with an exponential decay lr scheduler.
Scaling Factor Clarification. For shift layers, we reparameterize shifts following DeepShift-PS [17] and do not use a scaling factor. A straight-through estimator (STE) [69] is adopted to make the shift layer differentiable. For ShiftAddViT with KSH [34], there is no scaling factor needed as a set of
Table 11: Comparisons with our proposed ShiftAdd Transformer with baselines on Long Range Arena (LRA) benchmarks. Latency and energy are measured on an Eyeriss-like accelerator with a batch size of 1.
Models	Accuracy (%)	Average Latency (ms)	Average Energy (mJ)
Text (4K)	Listops (2K)	Retrieval (4K)	Image (1K)	Average
Transformer	65.02	37.10	79.35	38.20	54.92	84.54	139.83
Reformer	64.88	19.05	78.64	43.29	51.47	11.19	19.04
Linformer	55.91	37.25	79.37	37.84	52.59	12.13	19.68
Performer	63.81	18.80	78.62	37.07	49.58	11.93	18.74
ShiftAdd-Transformer	66.69	37.15	82.02	35.62	55.37	7.38	8.53
Table 12: Latency comparisons of PVTv2-B0 with different attention types when measured under various batch sizes (BS) and input resolutions on an RTX 3090 GPU. "Linear SRA" donates the linear spatial-reduction attention in PVTv2.
Attention Types	Pytorch Latency (ms)
Input Resolution = (224, 224)	Input Resolution = (448, 448)
BS = 1	BS = 2	BS = 4	BS = 8	BS = 16	BS = 32	BS = 64	BS = 1	BS = 2	BS = 4	BS = 8
MSA	4.62	4.90	5.09	8.48	16.61	32.49	64.31	16.20	32.10	63.96	126.81
PVTv2 (Linear SRA)	6.25	6.33	6.73	6.75	6.79	11.68	22.22	6.36	6.40	6.84	12.96
Linear Attention	6.34	6.73	6.73	6.75	7.19	13.56	25.91	6.56	6.58	7.75	14.02
hash functions is applied to convert Q/K to binary codes. For ShiftAddViT with Quant. [27], we leverage layer-wise Quant. for both Q & K, the scaling factor can be multiplied after add operations. It can be efficiently implemented following [28].
F More Ablation Studies
Linear Attention vs. MSA under Various Batch Sizes and Input Resolutions. We find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA in the Tab. 4 and 6. To investigate the hidden reasons, we further measure the latency of PVTv2-B0 with various BS and input resolutions, the results are summarized in the Tab. 12. From that, we observe that linear attention's benefits show up under larger BS or input resolutions. The hidden reasons for this phenomenon are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWConv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits.
The Advantage of Using Shift. Replacing Multiplication-based MLPs with shift layers significantly reduces latency, energy, and chip area costs. Our shift kernel offers an average speedup of  $2.35 \times 3.07 \times 1.16 \times$  compared to PyTorch FakeShift, TVM FakeShift, and TVM MatMuls, respectively, as shown in Fig. 3. The seemingly minor latency improvements in Tab. 4 and 6 are due to full optimization of the compiled model as a whole (e.g.,  $6.34\mathrm{ms} \rightarrow 1\mathrm{ms}$  for PVTv2-B0) on GPUs with sufficient chip area. Most gains are concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1. Under the same chip areas, latency savings are more pertinent, e.g., PVTv2-B0 w/ shift or MoE achieve  $3.9 \sim 5.7 \times$  and  $1.6 \sim 1.8 \times$  speedups, respectively, as summarized in Tab. 13.
Table 13: Comparisons of our ShiftAddViT with baselines in terms of task accuracy, latency measured with a batch size of 1 on an RTX 3090 GPU, and latency profiled on the Eyeriss accelerator under the same hardware area constraint.
Model	Linear Attention + Add	Shift	MoE	Accuracy (%)	GPU Latency (ms)	Eyeriss Latency (ms) (Under the Same Chip Area)
PVTv2-B0	✘	✘	✘	70.77	4.62 (Pytorch)	60.50
ShiftAddViT (PVTv2-B0)	✓	✘	✘	71.04	1.00	15.87
✓	✓(Attn & MLP)	✘	68.57	0.97	2.77
✓	✘	✓(Attn & MLP)	70.59	1.51 / 1.12*	9.77
PVTv2-B1	✘	✘	✘	78.83	4.77 (PyTorch)	147.77
ShiftAddViT (PVTv2-B1)	✓	✘	✘	78.70	1.57	58.64
✓	✓(Attn & MLP)	✘	77.55	1.54	12.42
✓	✘	✓(Attn & MLP)	78.49	2.49 / 1.35*	33.71