SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with LoRA-based baselines: Our LBP-WHT method consistently surpasses the LoRA-based method across all eight datasets in both partial and full training modes. For instance, when only training the final stage of EfficientFormer-L1, LBP-WHT using  $\mathrm{LP}_{L_1} - 2$  base selection requires 8.9 MFLOPs fewer computations than LoRA, yet achieves  $10\%$  greater accuracy on the CIFAR100 dataset. When the entire model is trained, the accuracy difference is smaller, but LBP-WHT still outperforms the LoRA-based method. For instance, in comparison to LoRA-all, LBP-WHT using  $\mathrm{LP}_{L_1} - 7$  base selection requires less computation (66.68 MFLOPs), but still improves accuracy by  $2\%$  on CIFAR100 when training the EfficientFormerV2-S0 model.
Comparison with traditional full-rank BP: With  $\mathrm{LP}_{L_1}$ -8 base selection, our LBP-WHT method either matches or surpasses the accuracy of full-rank BP while only requiring about  $80\%$  of the total
Computation [MFLOPs]
Figure 3: Accuracy and computation for training the last stage of different models with different ranks on CIFAR100 dataset. Our method consistently outperforms the baseline LoRA-all.
Partial Training: Training Last Stage + Decoder	Full Training
Method	R	MFLOPs	City	VOC12A	Method	R	MFLOPs	City	VOC12A
Full BP	-	10052.00	62.85	69.30	Full BP	-	16700.26	67.37	70.84
LoRA	8	5854.61	51.43	58.18	LoRA	8	11976.46	62.57	58.18
LoRA-all	8	6262.01	58.07	66.26	LoRA-all	8	11971.13	65.74	67.82
LPL1-2★	3	1481.94	58.95	67.93	LPL1-2	3	5746.54	61.57	67.93
LPL1-4★	10	2725.39	60.97	68.85	LPL1-4★	10	7295.52	64.72	68.85
LPL1-8	36	7308.45	62.68	68.95	LPL1-8	36	13086.06	66.17	68.95
Table 3: Experimental results for semantic segmentation. Results are highlighted as in Table2.
computation. When using smaller ranks, LBP-WHT significantly reduces the cost with only minor accuracy costs. For example, when training the final stage of EfficientFormer-L1 using  $\mathrm{LP}_{L_1}$ -4 base selection, LBP-WHT achieves a  $3.5 \times$  speedup with just a  $1\%$  loss in accuracy on CIFAR100. With  $\mathrm{LP}_{L_1}$ -8 base selection, LBP-WHT achieves even higher accuracy  $(79.34\%)$  with a  $1.2 \times$  speedup.
These results underscore the merits of our method. As shown in Table 2, our method achieves computational savings by systematically reducing the computational cost for all operations during backpropagation, including the gradient computation for both input and weight. Specifically, when we apply a similar rank for LoRA-all and LBP-WHT, we anticipate that both methods will have similar computational costs for computing the weight gradient. However, as LoRA-all cannot speed up the gradient computation for the input while LBP-WHT can, our LBP-WHT method requires only half the total computation of LoRA-all. Consequently, for a similar computational budget, LBP-WHT can employ a higher rank for low-rank projection, thus leading to a higher accuracy. For example, when training the entire EfficientFormerV2-S0 model, LBP-WHT with  $\mathrm{LP}_{L_1} - 4$  (rank 10) only requires 1187 MFLOPs, which is  $62\%$  of the computational cost for LoRA-all. Thus, for a similar budget, LBP-WHT can use a rank 28 projection  $(\mathrm{LP}_{L_1} - 7)$  and achieve a higher accuracy.
4.3 Semantic Segmentation
Table 3 presents the experimental results for adapting the ADE20K-pretrained Segformer model on Cityscapes and augmented Pascal VOC 2012 dataset. Our LBP-WHT has better results in most cases. For instance, when partially training on the Cityscapes dataset, our approach using  $\mathrm{LP}_{L_1} - 4$  base selection achieves a mIoU score approximately  $0.9\%$  higher than that of LoRA-all. Moreover, it only requires 1481.9 MFLOPs, which is  $4.2\times$  faster. These findings not only further validate the efficacy of our method, but also demonstrate its broad applicability across key computer vision tasks.
4.4 Exploration 1: Different Ranks for Low-rank Projection in LBP-WHT
Figure 3 shows the accuracy achieved when adapting ImageNet-pretrained ViTs for CIFAR100, with varying ranks for low-rank model adaptation. Our observations from this figure are as follows:
Figure 5: The WHT spectrum for gradient w.r.t. layer output  $(g_y)$  collected from the last attention block of EfficientFormer-L1. The brightness for each pixel  $(i,j)$  in the spectrum represents the energy preserved by the WHT base  $B_{i,j}$  during projection. A brighter pixel means a larger energy. As shown in Figure 2c, the WHT base with smaller indices corresponds to a lower frequency component.
1. Our LBP-WHT method consistently outperforms the LoRA-all method, i.e., for a similar level of computation, LBP-WHT yields higher accuracy.
2. By altering the rank, LBP-WHT provides a broader range of cost options than the baseline method.
3. LBP-WHT's accuracy monotonically improves as more ranks are employed for projection.
4. For all models with our LBP-WHT method, a generally concave accuracy-computation curve is observed. This indicates strong diminishing returns in using larger ranks.
5. LBP-WHT with  $\mathrm{LP}_{L_1} - 6$  base selection achieves an accuracy very close to that of full BP.
Our first observation further confirms the superior performance of our method. The second observation indicates the broad applicability of our method. For instance, for edge devices with limited computational budgets, like Raspberry Pi, we can employ LBP-WHT with a lower rank to reduce computational cost. On the other hand, for more powerful devices, such as personal desktops equipped with GPUs, a larger rank can be chosen to enhance accuracy. This ensures that users with various computational resources and constraints can benefit from our method.
The last three observations offer guidelines for rank selection with our LBP-WHT method.
With strict computational constraints: Given our third observation above, if there is a hard limit on the maximum number of FLOPs allocated for training, selecting the rank for LBP-WHT is straightforward: we simply opt for the largest possible number of ranks, which in most cases yields the highest accuracy.
Without strict computational constraints: Our final two observations suggest that training efficiency can be characterized by the marginal accuracy, or the slope of the accuracy-computation curve. As shown in Figure 4, before  $\mathrm{LP}_{L_1} - 4$ , the marginal accuracy is significantly greater than zero. However, after  $\mathrm{LP}_{L_1} - 6$ , the marginal accuracy is very close to zero. This
Figure 4: Marginal accuracy: the slope of the accuracy-computation curve in Figure 3.
implies that choosing fewer ranks than  $\mathrm{LP}_{L_1}$ -4 or more than  $\mathrm{LP}_{L_1}$ -6 may not be advantageous, as it could either forgo the opportunity for good performance with a small amount of computation or waste computation with little benefit. Thus, a good selection empirically lies between  $\mathrm{LP}_{L_1}$ -4 and 6.
4.5 Exploration 2: Different Bases Selection Method
Figure 5 shows the WHT spectrum for the gradient w.r.t. layer output  $(g_y)$  collected from the last attention block in EfficientFormer-L1. We observe that most energy concentrates in the low-frequency area, i.e., the top-left corner, which supports claim in Section 3.2 that natural images have strong spatial locality and strong low-frequency components. Furthermore, Figure 5 demonstrates that by choosing WHT
bases with low-frequency responses - that is, using the selection methods  $\mathrm{LP}_{L_1}$  and  $\mathrm{LP}_{L_{\infty}}$  - we can preserve most of the energy and minimize error during the low-rank projection. As indicated in Table 4, both of these low-pass base selection methods yield accuracy levels very similar to those achieved with the low-heuristic-error (LHE) method. The LHE method profiles the WHT spectrum
Method	Rank	CF100	CF10
LPL1	10	78.27	94.60
LPL∞	9	77.64	94.30
LHE	10	78.06	94.60
Table 4: Experimental results for adapting EfficientFormer-L1 on CIFAR100 and CIFAR10 with different base selection methods. Accuracy is in percentages  $(\%)$
and selects the WHT bases with the strongest response. Given that the  $\mathrm{LP}_{L_1}$  base selection method eliminates the need for profiling (unlike LHE) and offers a more favorable balance between accuracy and cost compared to  $\mathrm{LP}_{L_{\infty}}$ , we have selected  $\mathrm{LP}_{L_1}$  as the standard method for LBP-WHT.
4.6 Limitation and Broader Impacts
Full training with a small number of ranks: As shown in Table 2, we find that the accuracy degradation is not negligible when using a small number of ranks for LBP-WHT full training. We consider this is the issue of accumulating error introduced by low-rank projection during BP. We expect that an improved approximation method can perform even better. Of note, even with accuracy degradation, our method still consistently outperforms the baselines, i.e., LoRA-based methods.
Broader Impact: Our method greatly reduced the barrier for training ViTs. As a positive feature, our method may push the development of privacy-centric on-device training methods like federated learning; our method may also enable more researchers to test their ideas with powerful ViTs. On the other hand, our method may lower the barrier for irresponsible customization and use of ViT.
Low-rank Model Adaptation: [14] proposes to speed up transformer training by attaching and training only low-rank branches to the linear layer. More precisely, consider a linear layer with equation  $y = x \cdot w^{T}$ , where  $x$  is the input,  $y$  is the output, and  $w$  is the weight. LoRA adds a branch that contains low-rank weights  $w_{A}$  and  $w_{B}$ , forming  $y_{\mathrm{LoRA}} = x \cdot w^{T} + x \cdot (w_{A} \cdot w_{B})^{T}$ . The original weight  $w$  is kept frozen, while the appended weights  $w_{A}$  and  $w_{B}$  are trained. Since the ranks of  $w_{A}$  and  $w_{B}$  are much smaller than that of  $w$ , the computation needed to calculate the gradients with respect to  $w_{A}$  and  $w_{B}$  is significantly reduced. However, this method does not decrease the computation for calculating the gradient w.r.t.  $x$ . This is because it still needs to propagate the gradient through the weights  $w$  to  $x$ , which considerably limits the performance of LoRA-based methods. As demonstrated in Figure 3, our LBP-WHT, requires much less computation while having better accuracy than LoRA-based methods; this is because our method reduces the computation for all procedures in BP, including the gradient calculation for both input and weights.
Other Orthogonal Methods for On-device Training: Previous research on efficient on-device model adaptation falls into two main categories. The first category [33, 38-43] suggests reducing the computational cost of arithmetic operations (addition and multiplication) in BP through quantization. The second category [20, 44] proposes to append a smaller neural network to the original model and accelerate adaptation by only training the attachment. To the best of our knowledge, our paper is the first to apply low-rank BP for ViT model adaptation. Therefore, our method, LBP-WHT, is distinct from previous research and can be combined with those methods for enhanced performance.
In this paper, we have addressed the problem of efficient model adaptation for ViT. We have proposed a novel low-rank BP technique designed to reduce the computational load associated with the propagation of gradients through the linear layers of ViT, which is a significant bottleneck when fine-tuning ViT. In Section 3, we introduced the LBP-WHT method as a solution to accelerate model adaptation. More specifically, LBP-WHT operates by projecting the gradient w.r.t. the output  $(g_{y})$  into a low-rank space, performing matrix multiplications within this low-rank space, and then projecting the results back into the original space. Since all matrix multiplications occur in a low-rank space, the computational cost is significantly reduced. Additionally, thanks to the properties of the Walsh-Hadamard Transform (WHT), the overhead for these projections is minimal (as discussed in Section 3.3). Through extensive experiments in Section 4, we have demonstrated the efficiency and broad applicability of our method. Our LBP-WHT approach consistently outperforms existing methods with a significant speedup and higher accuracy.
This work was supported in part by the US National Science Foundation (NSF) grant CNS-2007284.
[41] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 2017.
[42] Kang Zhao, Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Distribution Adaptive INT8 Quantization for Training CNNs. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[43] Ziyang Hong and C Patrick Yue. Efficient-grad: Efficient training deep convolutional neural networks on edge devices with grad ent optimizations. ACM Transactions on Embedded Computing Systems, 2022.
[44] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder Side-tuning for Parameter and Memory Efficient Transfer Learning. In Advances in Neural Information Processing Systems, 2022.
[45] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40 (6):1452-1464, 2017.