SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Dynamic Input Allocation. The input allocation is determined dynamically during runtime, and the trainable router within the MoE layers automatically learns this dispatch assignment as the gradients decay. We know the allocation only when the model is executed and we receive the router outputs. Therefore, the shape of expert input and corresponding indexes are dynamically changed. This can be handled by PyTorch with dynamic graphs while TVM expects static input shape. To this end, we follow and leverage the compiler support for dynamism as proposed in Nimble [49] on top of the Apache TVM to handle the dynamic input allocation.
- Parallel Computing. Different experts are expected to run in parallel, this can be supported by several customized distributed training frameworks integrated with PyTorch, e.g., FasterMoE [24], and DeepSpeed [44]. However, it remains nontrivial to support this using TVM. One option to simulate is to perform modularized optimization to mimic parallel computing, in particular, we optimize each expert separately and measure their latency, the maximum latency among all experts will be recorded and regarded as the latency of this MoE layer, and the aggregated total of the time spent for each layer is the final reported modularized model latency. To avoid any potential confusion between real-measured wall-clock time, i.e., no parallelism assumed, and simulated modularized latency, i.e., ideal parallelism assumed, we reported both for models containing MoE layers in Sec. 5 to offer more insights into the algorithm's feasibility and cost implications.
Latency-aware Load-balancing Loss. The key to our MoE framework is the design of a routing function to balance all experts towards higher accuracy and lower latency. Previous solutions [20, 23] use homogeneous experts and treat them equally. While in our MoE framework, the divergence and heterogeneity between powerful yet slow Mult. and fast yet less powerful Shift experts incur one unique question: How to orchestrate the workload of each expert to reduce the synchronization time? Our answer is a latency-aware load-balancing loss, which ensures (1) all experts receive the expected weighted sums of gate values; and (2) all experts are assigned the expected number of input tokens. Those two conditions are enforced by adopting importance and load balancing loss as defined below:
$$
\mathcal {L} _ {\mathrm {I M P}} = \operatorname {S C V} \left(\left\{\alpha_ {i} \cdot \sum_ {\mathbf {x} \in \mathbf {X}} p _ {i} (\mathbf {x}) \right\} _ {i = 1} ^ {n}\right), \quad \mathcal {L} _ {\mathrm {L O A D}} = \operatorname {S C V} \left(\left\{\alpha_ {i} \cdot \sum_ {\mathbf {x} \in \mathbf {X}} q _ {i} (\mathbf {x}) \right\} _ {i = 1} ^ {n}\right), \tag {4}
$$
where SCV denotes the squared coefficient of variation of given distributions over experts (also used in [46, 48]);  $\alpha_{i}$  refers to the latency-aware coefficient of  $i$ -th expert and is defined by  $\frac{(\mathrm{Lat}_i)}{(\sum_j\mathrm{Lat}_j)}$  because that the expected assignments are inversely proportional to runtime latency of  $i$ -th expert,  $\mathrm{Lat}_i$ ;  $q_{i}(\mathbf{x})$  is the probability that the gate value of  $i$ -th expert outweighs the others, i.e., top1 expert, and is given by  $\mathbf{P}(p_i(\mathbf{x}) + \epsilon \geq p_j(\mathbf{x}), \forall j \neq i)$ . Note that this probability is discrete, we use a noise proxy  $\epsilon$  following [48, 23] to make it differentiable. The above latency-aware importance loss and load-balanced loss help to balance gate values and workload assignments, respectively, and can be integrated with classification loss  $\mathcal{L}_{\mathrm{CLS}}$  as the total loss function  $\mathcal{L}(\mathbf{X}) = \mathcal{L}_{\mathrm{CLS}}(\mathbf{X}) + \lambda \cdot (\mathcal{L}_{\mathrm{IMP}}(\mathbf{X}) + \mathcal{L}_{\mathrm{LOAD}}(\mathbf{X}))$  for training ViTs and gates simultaneously.  $\lambda$  is set as 0.01 for all experiments.
Models and Datasets. Tasks and Datasets. We consider two representative 2D and 3D Transformer-based vision tasks to demonstrate the superiority of the proposed ShiftAddViT, including 2D image classification on ImageNet dataset [15] with 1.2 million training and 50K validation images and 3D novel view synthesis (NVS) task on Local Light Field Fusion (LLFF) dataset [40] with eight scenes. Models. For the classification task, we consider PVTv1 [60], PVTv2 [61], and DeiT [55]. For the NVS task, we consider Transformer-based GNT [53] with View- and Ray-Transformer models.
Training and Inference Details. For the classification task, we follow Ecoformer [34] to initialize the pre-trained ViTs with Multihead Self-Attention (MSA) weights, based on which we apply our reparameterization a two-stage finetuning: (1) convert MSA to linear attention [73] and reparameterize all MatMUs with add layers with 100 epoch finetuning, and (2) reparameterize MLPs or linear layers with shift or MoE layers after finetuning another 100 epoch. Note that we follow PVTv2 [61] and Ecoformer to keep the last stage as MSA for fair comparisons. For the NVS task, we still follow the two-stage finetuning but do not convert MSA weights to linear attention to maintain the accuracy. All experiments are run on a server with eight RTX A5000 GPUs with each having 24GB GPU memory.
Baselines and Evaluation Metrics. Baselines. For the classification task, we reparameterize and compare our ShiftAddViT with PVTv1 [60], PVTv2 [61], Ecoformer [34], and their MSA variants. For the NVs task, we reparameterize on top of GNT [53] and compare with both vanilla NeRF [41]
Table 4: Comparisons between ShiftAddViT and baselines. We show the breakdown analysis of ShiftAddViT with two kinds of  $\mathbf{Q} / \mathbf{K}$  quantization. The throughput or latency is measured with a batch size of 32 or 1, where  $\dagger$  denotes that the numbers are measured and reported after optimizing ShiftAddViT using TVM. We report both real-device and modularized latency for models with MoE.
Methods	Linear Attn	Add	Shift	MoE	PVTv2-B0 [61]	PVTv1-T [60]
KSH	Quant.	Acc. (%)	Lat. (ms)	T. (img./s)	Acc. (%)	Lat. (ms)	T. (img./s)
MSA	X	X	X	X	X	70.77	4.62	989	76.21	4.73	903
PVT [61]	✓	X	X	X	X	70.50	6.25	2227	75.10	5.78	1839
PVT+MoE	✓	X	X	X	✓ (MLPs)	70.82	12.46	1171	75.27	10.91	834
Ecoformer [34]	✓	✓	X	X	X	70.44	7.82	1348	NaN	7.43	1021
ShiftAddViT (with KSH [34] or Quant. [27] to binarize Q/K)	✓	X	X	X	X	71.19	6.13	2066	75.50	5.78	1640
✓	✓	X	X	X	70.95	\( 1.07^† \)	\( 2530^† \)	75.20	\( 1.42^† \)	\( 1683^† \)
✓	✓	X	✓ (Attn)	X	70.53	\( 1.04^† \)	\( 2447^† \)	74.77	\( 1.39^† \)	\( 1647^† \)
✓	✓	X	✓ (Attn)	✓ (MLPs)	70.16	\( 1.39^†/1.11^* \)	N/A	74.44	\( 1.91^†/1.21^* \)	N/A
✓	✓	X	X	✓ (Both)	70.38	\( 1.59^†/1.20^* \)	N/A	74.73	\( 2.12^†/1.21^* \)	N/A
✓	X	X	X	X	71.36	6.34	2014	75.64	5.48	1714
✓	X	✓	X	X	71.04	\( 1.00^† \)	\( 2613^† \)	75.18	\( 1.20^† \)	\( 1907^† \)
✓	X	✓	✓ (Both)	X	68.57	\( 0.97^† \)	\( 2736^† \)	73.47	\( 1.18^† \)	\( 1820^† \)
✓	X	✓	X	✓ (Both)	70.59	\( 1.51^†/1.12^* \)	N/A	74.93	\( 1.97^†/1.02^* \)	N/A
\* denotes the modularized latency simulated by separately optimizing each expert/rodter with ideal parallelism.
Table 5: Comparisons between ShiftAddViT and baselines. We apply our Shift&Add techniques on top of SOTA Transformer-based NeRF model, GNT [53], and report averaged PSNR  $(\uparrow)$ , SSIM  $(\uparrow)$  and LPIPS  $(\downarrow)$  on the LLFF dataset [40] across eight scenes. In addition, we also show the results on two representative scenes, Orchids, and Flower, and the rendering latency and energy measured on an Eyeriss-like accelerator. More results for other scenes are available in the Appendix C.
Methods	Add	Shift	MoE	LLFF Averaged	Orchids	Flower	Lat. (s)	Energy (J)
PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
NeRF [41]	-	-	-	26.50	0.811	0.250	20.36	0.641	0.321	27.40	0.827	0.219	683.6	1065
GNT [53]	-	-	-	27.24	0.889	0.093	20.67	0.752	0.153	27.32	0.893	0.092	1071	1849
ShiftAddViT	✓	✗	✗	26.85	0.874	0.116	20.74	0.730	0.182	28.02	0.891	0.089	1108	1697
✓	✓(Both)	✗	26.85	0.875	0.116	20.78	0.730	0.182	28.05	0.892	0.088	568.5	844.0
✓	✓(Attn)	✓(MLPs)	26.92	0.876	0.114	20.73	0.731	0.180	28.20	0.894	0.087	746.6	1093
✗	✓(Both)	✗	27.05	0.881	0.107	20.84	0.746	0.169	28.14	0.896	0.083	531.2	995.6
and GNT [53]. Evaluation Metrics. For the classification task, we evaluate the ShiftAddViT and baselines in terms of accuracy, GPU latency and throughput measured on an RTX 3090 GPU. For the NVS task, we evaluate all models in terms of PSNR, SSIM [63], and LPIPS [76]. We also measure and report the energy consumption of all the above models based on an Eyeriss-like hardware accelerator [12, 77], which calculates not only computational but also data movement energy.
5.2 ShiftAddViT over SOTA Baselines on 2D Tasks
To evaluate the effectiveness of our proposed techniques, we apply the ShiftAddViT idea to five commonly used ViT models, including DeiT [55] and various variants of PVTv1 [60] and PVTv2 [61]. We compare their performance with baselines on the image classification task. Tab. 3 highlights the overall comparison with the most competitive baseline, Ecoformer [34], from which we see that ShiftAddViT consistently outperforms all baselines in terms of accuracy-efficiency tradeoffs, achieving  $1.74 \times \sim 5.18 \times$  latency reduction on GPUs and  $19.4\% \sim 42.9\%$  energy savings measured
on the Eyeriss accelerator [12] with comparable or even better accuracy ( $\uparrow 0.04\% \sim \uparrow 0.20\%$ ). Note that we apply MoE to both linear and MLP layers. Moreover, we further provide comparisons of a variety of ShiftAddViT variants and PVT with or without multi-head self-attention (MSA) and MoE in Tab. 4 and 6. We observe that our ShiftAddViT can be seamlessly integrated with linear attention [73, 65], simple or advanced Q/K quantization [34, 27], and the proposed MoE framework, achieving up to  $3.06 \times / 4.14 \times / 8.25 \times$  and  $2.47 \times / 1.10 \times / 2.09 \times$  latency reductions and throughput improvements after the TVM optimization on an RTX 3090 as compared to MSA, PVT, and PVT+ MoE (note: two Mult. experts) baselines, respectively, under comparable accuracies, i.e.,  $\pm 0.5\%$ .
Table 6: Comparisons between ShiftAddViT and baselines on PVTv2-B1 and PVTv2-B2. The throughput or latency is measured with a batch size of 32 or 1 on an RTX 3090 GPU.
Methods	Linear Attn	Add	Shift	MoE	PVTv2-B1 [61]	PVTv2-B2 [61]
KSH	Quant.	Acc. (%)	Lat. (ms)	T. (img./s)	Acc. (%)	Lat. (ms)	T. (img./s)
MSA	X	X	X	X	X	78.83	4.77	821	81.82	9.08	508
PVT [61]	✓	X	X	X	X	78.70	6.75	1512	82.00	12.20	921
Ecoformer [34]	✓	✓	X	X	X	78.38	8.02	874	81.28	15.43	483
ShiftAddViT	✓	X	X	X	X	78.80	6.36	1373	81.40	9.21	796
✓	✓	X	X	X	78.70	\( 1.70^† \)	\( 923^† \)	81.31	\( 3.18^† \)	\( 518^† \)
✓	✓	X	✓ (Attn)	✓ (MLPs)	78.20	\( 2.46^†/1.46^* \)	N/A	81.06	\( 4.17^†/3.22^* \)	N/A
(with KSH [34]	✓	✓	X	X	✓ (Both)	78.38	\( 2.64^†/1.48^* \)	N/A	81.32	\( 4.83^†/3.35^* \)	N/A
or Quant. [27]	✓	X	X	X	X	78.93	6.49	1327	81.55	11.82	782
to binarize Q/K)	✓	X	✓	X	X	78.70	\( 1.57^† \)	\( 942^† \)	81.33	\( 2.82^† \)	\( 553^† \)
✓	X	✓	✓ (Both)	X	77.55	\( 1.54^† \)	\( 1021^† \)	79.97	\( 2.93^† \)	\( 600^† \)
✓	X	✓	X	✓ (Both)	78.49	\( 2.49^†/1.35^* \)	N/A	81.01	\( 4.55^†/3.16^* \)	N/A
Training Wall-clock Time. To offer more insights into the training cost, we collect the training wall-clock time, which is  $21\% \sim 25\%$  less than training from scratch as the original Transformer model and more than  $50\times$  less than the previous ShiftAddNet [70] did on GPUs. For example, training the PVTv1-Tiny model from scratch takes 62 hours while our finetuning only needs 46 hours.
Linear Attention vs. MSA. We ensure batch size (BS) is 1 for all latency measurements and find the counterintuitive phenomenon that PVT with linear attention is slower than PVT with MSA. The reasons are two folds: (1) linear attention introduces extra operations, e.g., normalization, DWConv, or spatial reduction block that cost more time under small BS; and (2) the linear complexity is w.r.t. the number of tokens while the input resolution of 224 is not sufficiently large, resulting in limited benefits. To validate both points, we measure the latency of PVTv2-B0 with various BS and input resolutions as shown in the Appendix F, the results indicate that linear attention's benefits show up under larger BS or input resolutions.
5.3 ShiftAddViT over SOTA Baselines on 3D Tasks
We also extend our methods to 3D NVs tasks and built ShiftAddViT on top of Transformer-based GNT models [53]. We test and compare the PSNR, SSIM, LPIPS, latency, and energy of both ShiftAddViT and baselines, i.e., vanilla NeRF [41] and GNT [53], on the LLFF dataset across eight scenes. As shown in Tab. 5, our ShiftAddViT consistently leads to better accuracy-efficiency tradeoffs, achieving  $22.3\% / 50.4\%$  latency reductions and  $20.8\% / 54.3\%$  energy savings under comparable or even better generation quality ( $\uparrow 0.55 / \downarrow 0.19$  averaged PSNR across eight scenes), as compared to NeRF and GNT baselines, respectively. Note that GNT costs more than NeRF because of an increased number of layers. In particular, ShiftAddViT even achieves better generation quality (up to  $\uparrow 0.88$  PSNR) than GNT on some representation scenes, e.g., Orchid and Flower, as highlighted in Tab. 5, where the first, second, and third ranking performance are noted with corresponding colors. The full results of eight scenes are supplied in the Appendix C. Note that we only finetune ShiftAddViT for 140K steps on top of pre-trained GNT models instead of training up to 840K steps from scratch.
5.4 Ablation Studies of ShiftAddViT
Performance Breakdown Analysis. To investigate how each of the proposed techniques contributes to the final performance, we conduct ablation studies among various kinds of ShiftAddViT variants on PVTv1-T, PVTv2-B0/B1/B2 models to gradually break down and show the advantage of each component. As shown in Tab. 4 and 6, we have three general observations: (1) ShiftAddViT is robust to the binarization method of  $\mathbf{Q} / \mathbf{K}$ , e.g., it is compatible with either KSH [34] or vanilla binarization [27], achieving on average  $3.03 \times$  latency reductions than original PVT under comparable accuracies  $(\pm 0.5\%)$ , and both  $3.29 \times 1.32 \times$  latency/energy reductions and  $0.04\% \sim 0.20\%$  accuracy improvements over Ecoformer [34]; (2) vanilla binarization methods work better than KSH in our ShiftAddViT framework in terms of both accuracy  $(\downarrow 0.05\% \sim \uparrow 0.21\%)$  and efficiency (on average  $\downarrow 5.9\%$  latency reductions and  $\uparrow 5.8\%$  throughput boosts). Moreover, KSH requires that  $\mathbf{Q}$  and  $\mathbf{K}$  are identical while vanilla binarization [27] does not have such a limitation. That explains why applying vanilla quantization to the linear attention layers results in an accuracy improvement of on average  $0.15\%$  as compared to using KSH. Also note that our adopting linear attention following [73] works better  $(\uparrow 0.29\%)$  than PVT [60, 61]; (3) Replacing linear layers in attention modules or MLPs with Shift layers lead to an accuracy drop while our proposed MoE can help to compensate for it. Specifically, adopting Shift layers leads to on average  $1.67\%$  accuracy drop while MoE instead improves on average  $1.37\%$  accuracy. However, MoE hurts the efficiency of both the PVT baseline
Figure 4: MLP/Linear latency speedups using shifts, where inputs are of shape  $B \times K \times M$ , weights are of shape  $K \times N$ . All dimensions are set w.r.t. the real dimensions in PVTs [60].
Figure 5: MatMuls latency speedups using adds, where inputs are of shape  $B \times H \times K \times M$ , weights are of shape  $B \times H \times K \times N$ .
and our ShiftAddViT without customized system acceleration, e.g., PVT+MoE increases  $94\%$  latency and reduces  $51\%$  throughput than PVT on GPUs, due to limited parallelism supports, especially for TVM. We report the modularized latency by separately optimizing each expert to demonstrate the potential of MoE's double-winning accuracy ( $\uparrow 0.94\% \sim \uparrow 2.02\%$ ) and efficiency ( $\downarrow 15.4\% \sim \uparrow 42.7\%$ ).
In addition, we want to clarify that the seemingly minor latency improvements of adopting shifts in Tab. 4 and 6 are due to full optimization of the compiled model as a whole (e.g.,  $6.34\mathrm{ms} \rightarrow 1\mathrm{ms}$  for PVTv2-B0) on GPUs with sufficient chip area. Most gains are concealed by data movements and system-level schedules. Adopting shift layers substantially lowers energy and chip area usage as validated in Tab. 1 and the Appendix F. Under the same chip areas, latency savings of adopting shifts are more pertinent, e.g., PVTv2-B0 with shift or MoE achieve  $3.9 \sim 5.7\times$  or  $1.6 \sim 1.8\times$  speedups.
Apart from 2D tasks, we also report the performance breakdown on 3D NVs tasks, as shown in Tab. 5. The above three observations still hold except for the Shift layer, as we find that ShiftAddViT with all linear layers or MLPs replaced by Shift layers achieve even better PSNR  $(\uparrow 0.13 \sim \uparrow 0.20)$ , SSIM  $(\uparrow 0.005 \sim \uparrow 0.007)$ , and LPIPS  $(\downarrow 0.007 \sim \downarrow 0.009)$  than other variants. In addition, we profile on an Eyeriss
Figure 3: Energy breakdown on an Eyeriss accelerator.
accelerator to show the energy breakdown of both ShiftAddViT and baselines. As shown in Fig. 3, our ShiftAddViT reduced  $42.9\%$  and  $40.9\%$  energy on top of DeiT and GNT, respectively. Among them applying Add layers lead to  $93.8\%$  and  $63.8\%$  energy reductions than original MatMuls, Shift layers help to reduce  $28.5\%$  and  $37.5\%$  energy than previous linear/MLP layers. This set of experiments validates the effectiveness of each component in our proposed ShiftAddViT framework.
Speedups of Shifts and Adds. Apart from the overall comparison and energy reduction profiling, we also test the GPU speedups of our customized Shift and Add kernels, as shown in Fig. 4 and 5, respectively. We can see that both customized kernels achieve faster speeds than PyTorch and TVM baselines. For example, our MatAdds achieve on average  $7.54 \times 1.51 \times$  speedups than PyTorch and TVM MatMuls, respectively. Our MatShifts achieve on average  $2.35 \times 3.07 \times 1.16 \times$  speedups than PyTorch FakeShifts [17], TVM FakeShifts, and TVM MatMuls, respectively. More comparisons and speedup analysis under larger batch sizes are supplied in the Appendix A. Note that in those comparisons, we use the default einsum operator for the PyTorch MatMuls, floating-point multiplication with power-of-twos for FakeShift, and implement the MatAdding and MatShifts using TVM [10] ourselves. We compare those customized operators with their multiplication counterparts in the same setting for fair comparisons. The speedup of MatAdding is because we replace multiplication-and-accumulation with accumulation only. The speedup of MatShifts is mainly attributed to the bit reductions (INT32 and INT8 for inputs and shift signs/weights) and thus data movement reductions instead of computations which are almost fully hidden behind data movements.
MoE's Uniqueness in ShiftAddViT. Our contributions are the multiplication-reduced ViTs and a new load-balanced MoE framework. They are linked in one organic whole. Unlike previous MoE works where all experts are of the same capacity and there is no distinction between tokens, our MoE layers combine unbalanced experts, i.e., multiplication and shift, with a hypothesis to divide important object tokens and background tokens. Such a new MoE design offers a softer version of
Token Gate
Image
Figure 6: Visualization of the token dispatches in MoE routers/gates given images from the validation set, where yellow color denotes Mult. experts while blue color refers to Shift experts.
ShiftAddViTs, i.e., instead of aggressively replacing all multiplications with cheaper shifts and adds, we keep the powerful multiplication option to handle importance tokens for maintaining accuracy while leaving all remaining unimportant tokens being processed by cheaper bitwise shifts, winning a better accuracy and efficiency tradeoff.
Ablation studies of Latency-aware Load-balancing Loss (LL-Loss). To demonstrate the effectiveness of the proposed LL-Loss in our ShiftAddViT MoE framework, we conduct ablation studies on two PVT models as shown in Tab. 7. We find that ShiftAddViT w/ LL-Loss achieves better accuracy-efficiency tradeoffs, e.g.,  $14.6\%$  latency reductions while maintaining comparable accuracies  $(\pm 0.1\%)$  when considering Mult. and Shift
Table 7: Ablation studies of the ShiftAddViT w/ or w/o LL-Loss on PVT models.
Models	Methods	Acc. (%)	Norm. Latency
PVTv2-B0	w/o LL-Loss	70.38	100%
w/ LL-Loss	70.37	85.4%
PVTv1-T	w/o LL-Loss	74.73	100%
w/ LL-Loss	74.66	85.5%
experts. The latency reduction could be larger if we consider more unbalanced experts with distinct runtimes. This set of experiments justifies the effectiveness of our LL-Loss in the MoE system.
Visualization of Token Dispatches in MoE. To validate our hypothesis that important yet sensitive tokens require powerful experts while other insensitive tokens can be represented with much cheaper Shift experts, we visualize the token dispatches in MoE routers/gates of the first MLP layer in our ShiftAddViT PVTv2-B0 model as shown in Fig. 6. We see that our designed router successfully identifies the important tokens that contain object information, which are then dispatched to more powerful Multiplication experts, leaving unimportant tokens, e.g., background, to cheaper Shift tokens. This set of visualization further validates the hypothesis and explains why our proposed MoE framework can effectively boost our ShiftAddViT towards better accuracy-efficiency trade-offs.
5.5 Limitation and Societal Impact Discussion
We made a firm step to show the practical usage of the proposed hardware-inspired ShiftAddViT. While this relies on dedicated TVM optimization, the full potential can be unraveled with customized hardware accelerators toward naturally hardware-efficient ViTs. Also, the unbalanced MoE framework shows its generalizability but highly demands system support with ideal parallelism.
In this paper, we for the first time propose a hardware-inspired multiplication-reduced ViT model dubbed ShiftAddViT. It reparameterizes both attention and MLP layers in pre-trained ViTs with a mixture of multiplication primitives, e.g., bitwise shifts and adds, towards end-to-end speedups on GPUs and dedicated hardware accelerators without the need for training from scratch. Moreover, a novel mixture of unbalanced experts framework equipped with a new latency-aware load-balancing loss is proposed in pursuit of double-winning accuracy and hardware efficiency. We use multiplication or its primitives as experts in our ShiftAddViT cases. Extensive experiments on both 2D and 3D Transformer-based vision tasks consistently validate the superiority of our proposed ShiftAddViT as compared to multiple ViT baselines. We believe that this paper opens up a new perspective on designing energy-efficient ViT inference based on widely available pre-trained ViT models.
The work is supported in part by the National Science Foundation (NSF) through the RTML program (Award number: 1937592) and CoCoSys, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.