Title: Efficient Low-rank Backpropagation for Vision Transformer Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Low-rank backpropagation

model adaptation

transfer learning

vision transformer

Edge AI
Abstract: The increasing scale of vision transformers (ViT) has made the efficient fine-tuning of these large models for specific needs a significant challenge in various applications. This issue originates from the computationally demanding matrix multiplications required during the backpropagation process through linear layers in ViT.
In this paper, we tackle this problem by proposing a new Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method. Intuitively, LBP-WHT projects the gradient into a low-rank space and carries out backpropagation. This approach substantially reduces the computation needed for adapting ViT, as matrix multiplication in the low-rank space is far less resource-intensive. We conduct extensive experiments with different models (ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the effectiveness of our method. For instance, when adapting an EfficientFormer-L1 model on CIFAR100, our LBP-WHT achieves 10.4\% higher accuracy than the state-of-the-art baseline, while requiring 9 MFLOPs less computation.
As the first work to accelerate ViT adaptation with low-rank backpropagation, our LBP-WHT method is complementary to many prior efforts and can be combined with them for better performance.

Efficient Low-rank Backpropagation for Vision Transformer Adaptation

Abstract

The increasing scale of vision transformers (ViT) has made the efficient finetuning of these large models for specific needs a significant challenge in various applications. This issue originates from the computationally demanding matrix multiplications required during the backpropagation process through linear layers in ViT. In this paper, we tackle this problem by proposing a new Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method. Intuitively, LBP-WHT projects the gradient into a low-rank space and carries out backpropagation. This approach substantially reduces the computation needed for adapting ViT, as matrix multiplication in the low-rank space is far less resource-intensive. We conduct extensive experiments with different models (ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the effectiveness of our method. For instance, when adapting an EfficientFormer-L1 model on CIFAR100, our LBPWHT achieves 10.4% higher accuracy than the state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the first work to accelerate ViT adaptation with low-rank backpropagation, our LBP-WHT method is complementary to many prior efforts and can be combined with them for better performance. Code: https://github.com/SLDGroup/LBP-WHT

1 Introduction

Vision transformers (ViT) have emerged as the latest state-of-the-art tool in numerous general computer vision tasks [1-7]. However, tailoring these models to meet specific needs ( e.g. , new dataset with different distribution) can be challenging. Indeed, adapting ViT models via finetuning demands considerable computational resources and is often impractical for most edge applications. For instance, to maintain privacy, in federated learning [8-10], model adaptation is limited to users' personal edge devices ( e.g. , smartphones), where computational power is tightly restricted [11, 12].

The primary computational bottleneck arises from gradient propagation through the dense layers of ViT. Specifically, calculating gradients for layer weights and inputs requires two computationallyintensive matrix multiplications, given the gradient for output [13]. To tackle this issue, [14] tries to simplify matrix multiplications using low-rank reparametrization. However, this method only reduces the gradient computation for weights and not for inputs, thus limiting the overall speedup. This observation raises the following question:

How can we decrease the computational cost for all operations, including gradient computations for weights and inputs, involved in backpropagation (BP) through any linear layer in the ViT model?

To answer this question, we introduce a new L ow-rank B ack P ropagation via W alshH adamard T ransformation (LBP-WHT) method. As shown in Figure 1, our method intuitively performs BP for gradients w.r.t. inputs and weights in a low-rank space. To achieve this, we project the gradient w.r.t.

the output into a low-rank space using WHT [15], then perform low-rank matrix multiplications, and finally project the results back. This way, all matrix multiplications occur in a low-rank space, hence the computational cost is significantly reduced. In summary, our contributions are as follows:

We propose LBP-WHT, a new approach which greatly reduces the computational cost for adapting ViT while maintaining accuracy; our method lowers the computational barrier and enables adapting large ViT models on resource constrained edge devices.

LBP-WHT is the first work accelerating ViT training by low-rank BP; thus, LBP-WHT is orthogonal to prior works and can be combined with them for a better performance. Additionally, LBP-WHT offers abundant flexibility that can provide a good tradeoff between accuracy and cost.

Extensive experiments on multiple datasets demonstrate the effectiveness of our method. Indeed, LBP-WHT consistently outperforms the baseline methods both in accuracy and speed. For instance, LBP-WHT achieves 10.4% higher accuracy, while requiring 9 MFLOPs less computation than [14] for training EfficientFormer-L1 on CIFAR100 dataset.

The paper is organized as follows. Section 2 formulates the problem associated with BP for linear layers. Section 3 presents our method LBP-WHT in detail. Experimental results are presented in Section 4. Section 5 reviews relevant work. Finally, Section 6 summarizes our main contributions.

2 Problem Formulation

Naming conventions: In this paper, we treat all feature maps as matrices composed of real numbers, with dimensions R C × L , where C represents the number of rows and L denotes the number of columns. Each row in the matrix is regarded as a 'channel' consisting of L elements, and there are a total of C channels in the feature map. We use subscripts to identify specific variables, such as C x for the number of channels associated with variable x . Gradients with respect to x are denoted by g x , with the subscript indicating the target variable x .

Backpropagation for linear layers: We focus on the BP process for linear layers, a crucial building block for vision transformers. Given an input x ∈ R C x × L and weights w ∈ R C y × C x , the forward propagation to compute the output y ∈ R C y × L can be expressed as:

Therefore, as shown in Figure 2a, given the gradient with respect to the output y , i.e. , g y ∈ R C y × L , the back-propagation for computing the gradient with respect to the weights w , g w ∈ R C y × C x , and the gradient with respect to the input x , g x ∈ R C x × L , can be represented as two matrix multiplications:

The gradient w.r.t. the weight ( g w ) is utilized for updating the weights w , while the gradient w.r.t. the input ( g x ) is employed for propagating the gradient to other layers. During the BP process, each matrix multiplication incurs a computational cost of 2 C x C y L FLOPs, which amounts to 4 C x C y L FLOPs, in total. Given that in ViT models, the number of channels ( C x and C y ) and the length of the input feature map ( L ) are substantial [1-7], the computational cost for BP becomes significant.

Low-rank backpropagation: As shown in Figure 1 and 2b, we propose reducing the computational cost for both matrix multiplications by employing low-rank approximations. Specifically, we first project variables into a low-rank space as follows:

Here, ˆ g y ∈ R C y × R and ˆ x ∈ R C x × R represent the low-rank space projections ( R << L ) for the gradient with respect to the output ( g y ) and input x , respectively. The projection function p ( · ) will be introduced in the next section.

Next, we execute the BP through the linear layer in the low-rank spaces as follows:

WHT Reverse

Projection

WHT Low-rank

Projection

Gradient w.r.t.

output

Gradient w.r.t.

input & weights

Low-rank Back Propagation via WHT

𝑝𝑝

-1

(

⋅

)

𝑝𝑝

(

⋅

)

𝑝𝑝

(

⋅

)

𝑥𝑥

𝑤𝑤

𝑔𝑔

𝑦𝑦

𝑤𝑤

𝑥𝑥

𝑔𝑔

𝑦𝑦

𝑔𝑔

𝑤𝑤

𝑔𝑔

𝑥𝑥

�

𝑔𝑔

𝑤𝑤

�

𝑔𝑔

𝑥𝑥

Low-rank

Projection

Reverse

Projection

Low-rank

Matrix Mul

Full-rank

Matrix Mul

(a) Vanilla BP

(b) LBP-WHT

×

×

×

×

�

𝑔𝑔

𝑤𝑤

�

𝑔𝑔

𝑥𝑥

𝑥𝑥 �

�

𝑔𝑔

𝑦𝑦

0

0

1

2

3

1

2

3

(c)

Finally, we project the low-rank gradient with respect to the input ( ˆ g x ) back into its original space. The reverse projection for ˆ g w can be omitted as it already exists in the same space R C y × C x as the target g w . For ˆ g x , the reverse projection is accomplished using the function p -1 ( · ) , the details of which will be presented later:

Here, ˜ g w and ˜ g x represent the resulting gradients for weights and input. As these gradients are generated through an approximated back-propagation process rather than the standard BP, we denote these variables with tildes.

3 LBP-WHT: Low-rank BackPropagation via WHT

As shown in Figure 2b, intuitively, we reduce the computational cost by performing back-propagation in a low-rank space, as described in Equation 4. For instance, using a rank R approximation, each matrix multiplication requires 2 C x C y R FLOPs, which can be substantially smaller than 2 C x C y L when R < < L . Nevertheless, this approach necessitates two additional steps, projection and reverse projection (as illustrated in Equation 3 and 5), which introduce some computational overhead. Furthermore, the low-rank projection may add noise and potentially diminish the quality of training. To address these concerns, our method incorporates a low-overhead projection function based on the WHTand tackles the second issue by selecting an appropriate set of WHT bases.

WHT is a generalized Fourier transformation. Figure 2c displays the transformation basis for an order-4 WHT. For an ordern 2D WHT, there are n × n bases B i,j , with each basis being an n × n matrix containing only +1 and -1 . Of note, in the context of ViT, 2D feature maps are flattened into 1D maps, so we utilize a flattened WHT base-a vector with a length of n 2 , i.e. , B i,j ∈ Z n 2 × 1 , 0 ≤ i, j < n . WHTpossesses four properties that make it advantageous for us:

The transformation bases are complete.

The transformation bases are orthogonal.

The transformation bases contain only +1 and -1 .

The transformation cost can be reduced via fast WHT algorithm with O ( n log n ) complexity.

The first property (completeness) allows WHT to perform transformations ranging from lossy (when few bases are activated) to lossless (when all bases are activated). This grants flexibility in exploring the trade-off between efficiency and accuracy. The second property ensures that any variable has precisely one projection result, obtainable via matrix multiplication. For instance, the projection

function for g y (Equation 3) with basis B i,j can be expressed as p ( g y ) = g y · B i,j . Likewise, the reverse projection can also be implemented using a simple matrix multiplication. The third and final properties demonstrate the efficiency of WHT implementation, requiring only O ( n log n ) additions/subtractions and no multiplications [16].

3.1 Low-rank Back-Propagation with WHT

Indeed, these four properties demonstrate that WHT is an ideal fit for our needs, offering both low overhead and high flexibility for selecting an appropriate set of bases. Therefore, we employ WHT as the projection function p ( · ) and reverse projection function p -1 ( · ) in Equations 3 and 5. More specifically, for an ordern WHT with a set of R bases chosen by an index set I , the projection function can be written as:

where I = { ( i k , j k ) | 1 ≤ i k , j k ≤ n, 1 ≤ k ≤ R } indicates which bases are activated. Similarly, the reverse projection function can be expressed as:

For simplicity, both Equations 6 and 7 are presented using the vanilla WHT algorithm with computational complexity O ( n 2 ) , rather than the fast WHT algorithm with complexity O ( n log n ) . Consequently, our LBP-WHT algorithm can be summarized as Algorithm 1 also shown in Figure 2b.

Algorithm 1 Backpropagation through a linear layer with LBP-WHT.

Input: Input x , weight w , gradient w.r.t. output g y , Selected WHT base indices I Output: Approximated gradient w.r.t. input ˜ g x , approximated gradient w.r.t. weight ˜ g w

ˆ x ← p ( x ) = WHT ( x ; I ) ▷ Projection to a low-rank space with WHT (Equation 3) ˆ g y ← p ( g y ) = WHT ( g y ; I )

ˆ g w ← ˆ g T y · ˆ x ▷ Efficient matrix multiplication in a low-rank space (Equation 4) ˆ g x ← ˆ g y · w

˜

g

x

p

-

1

(ˆ

g

x

) =

WHT

-

1

(ˆ

g

x

;

I

)

▷

Reverse projection to a full-rank space (Equation 5)

˜

g

w

ˆ

g

w

▷

Skipped reverse projection since

ˆ

g

w

is already in a full-rank space

Given input for BP, we first project x and g y into low-rank space (Equation 3), then we performs matrix multiplication (Equation 4) and lastly we project the results back (Equation 5).

3.2 WHTBases Selection

Here we explore two types of basis selection strategies: low-pass and low-heuristic-error.

Low-pass (LP) Base Selection: Natural images have strong spatial locality, i.e. , pronounced lowfrequency components [17, 18]. We take advantage of this feature and choose bases with stronger low-frequency responses, which have smaller indices as illustrated in Figure 2c. More specifically, we consider both L 1 -based and L ∞ -based low-pass basis selection strategies (LP L 1 and LP L ∞ ):

I L 1 and I L ∞ are the index sets for selecting WHT bases, as described in Section 3.1. For example, with LP L 1 -2 base selection, three bases are chosen, i.e. , I L 1 = { (0 , 0) , (0 , 1) , (1 , 0) } , and the rank for projection, namely R , is three.

Low-heuristic-error (LHE) Base Selection: According to Parseval's Theorem [19], WHT preserves the signal energy, so by selecting the WHT bases with the topr strongest responses, we can preserve most energy during low-rank projection and minimize the error. Since profiling the energy for all WHT bases on all training steps is expensive, we profile the energy for all WHT bases only for a small number of training steps and select the bases with the topR energy.

Considering that the L 1 -based low-pass basis selection has a much lower profiling overhead than the low-heuristic-error basis selection and provides finer granularity in balancing accuracy and efficiency, we primarily focus on the LP L 1 selection method and explore the other two in Section 4.5.

3.3 Overhead Analysis

Since the computational cost for the fast WHT algorithm depends on the basis selection, we simplify the analysis in this section by considering the matrix multiplication-based vanilla WHT algorithm, as shown in Equations 6 and 7. Table 1 presents the computation requirements for a linear layer with input and output channels C x and C y , feature map size L , and the rank for low-rank WHT approximation r . Our LBP-WHT achieves a L R times speedup with an overhead of (2 C x + C y ) LR FLOPs, which is only (2 C x + C y ) LR 4 C x C y L or ( 1 C x + 1 2 C y ) R 2 of the total computation required by vanilla BP. Given that ViT typically has a large number of channels, the overhead is very small.

Table 1: Computation required by vanilla BP and components in our LBP-

WHT. We consider the projection and reverse projection as overhead. 'MM' is short for 'Matrix Multiplication'.

For instance, the final linear layer in SwinV2-small [1] consists of 3072 input channels, 768 output channels, and a feature map size of 49, which means C x = 3072 , C y = 768 , and L = 49 . As per Table 1, conventional backpropagation (BP) requires 462.3 MFLOPs. In contrast, our Low-Rank Backpropagation with WHT (LBP-WHT) method, assuming a rank of 8 ( R = 8 ), needs only 78.2 MFLOPs, which is roughly 16.9% of the computation required by vanilla BP.

Breaking down the 78.2 MFLOPs for LBP-WHT, we see that 1.5 MFLOPs are needed for the low-rank projection, 75.5 MFLOPs for BP in the low-rank space, and 1.2 MFLOPs for the reverse projection. The combined overhead is 2.7 MFLOPs, accounting for just 0.6% of vanilla BP's computation and 3.5% of LBP-WHT's computation. This demonstrates that with WHT, we can significantly reduce the computation for BP while incurring negligible overhead for low-rank projection.

4 Experimental Results

In this section, we first present our experimental results on image classification and semantic segmentation tasks. Then, we explore the impact of different ranks for low-rank projection and different base selection strategies. Lastly, we present our preliminary results for deploying our methods on real edge devices in the supplementary material.

4.1 Experimental Setup

Environment: We setup our environment with PyTorch 1.13, MMClassification v0.25 and MMSegmentation v0.30. Models are trained with an NVIDIA-A6000 GPU.

Classification: We conduct experiments for image classification following [20]. We use ImageNet [21]-pretrained ViTs and finetune them on six different datasets, namely, CIFAR100 [22] (CF100), CIFAR10 [22] (CF10), Cars [23], Flowers [24], Food [25], and Pets [26]. We standardize the image resolution across all datasets to 224 × 224. Each model is finetuned for 50 epochs using the AdamW [27] optimizer and a batch size of 64. The learning rate is adjusted for each dataset based on the performance of EfficientFormer-L1 [28] with vanilla BP.

Semantic Segmentation: We use the ADE20K [29]-pretrained Segformer-mit-b0 [30] model and finetune it on two datasets, Cityscapes [31] (City) and the enhanced Pascal-VOC 2012 [32] (VOC12A). The images are downscaled and cropped to a size of 512 × 512 pixels for training. Models are finetuned for 20,000 steps using the AdamW optimizer and a batch size of 8.

Partial Training: Weprimarily report on the results of training the final stage of the ViT using various methods, a common approach in transfer learning to reduce the computational cost [18, 33-36]. More results for full training are included in the supplementary material.

Baselines Comparisons: We compare our results against three baseline methods: Full BP, 'LoRA', and 'LoRA-all'. Full BP refers to training the model with standard full-rank backpropagation. 'LoRA' and 'LoRA-all' are methods derived from [14]. 'LoRA' strictly follows [14], which uses low-rank reparametrization solely in the ViT's attention modules, while 'LoRA-all' applies this method to all linear layers. For hybrid CNN-ViT models, where the attention modules are usually only in the final stage, we use 'LoRA-all' for full training.

Computation Measurements and Preliminary Deployment Results: To determine the computational requirements of different models and methods, we run model training on an Intel 11900K CPU and measure the exact FLOPs using the embedded performance tools 'perf' in the Linux kernel v5.15.87. For preliminary deployment results, we test our method on the last two linear layers of EfficientFormer-L1, using OpenBLAS and CuBLAS for CPU and GPU testing respectively on an NVIDIA Jetson Nano. The results for deployment are reported in the supplementary material.

4.2 Image Classification Results

Table 2 demonstrates the effectiveness of our LBP-WHT method in adapting ViT for image classification tasks. Here are some more specific observations:

Comparison with LoRA-based baselines: Our LBP-WHT method consistently surpasses the LoRAbased method across all eight datasets in both partial and full training modes. For instance, when only training the final stage of EfficientFormer-L1, LBP-WHT using LP L 1 -2 base selection requires 8.9 MFLOPs fewer computations than LoRA, yet achieves 10% greater accuracy on the CIFAR100 dataset. When the entire model is trained, the accuracy difference is smaller, but LBP-WHT still outperforms the LoRA-based method. For instance, in comparison to LoRA-all, LBP-WHT using LP L 1 -7 base selection requires less computation (66.68 MFLOPs), but still improves accuracy by 2% on CIFAR100 when training the EfficientFormerV2-S0 model.

Comparison with traditional full-rank BP: With LP L 1 -8 base selection, our LBP-WHT method either matches or surpasses the accuracy of full-rank BP while only requiring about 80% of the total

200

400

600

800

1000

1200

1400

1600

72

74

76

78

LP

L

1

-2

LP

L

1

-4

LP

L

1

-6

LP

L

1

-8

4

6

8

16

32

Full

BP

EfficientFormer-L1

2000

4000

6000

8000

10000

84.0

84.5

85.0

85.5

86.0

86.5

LP

L

1

-2

LP

L

1

-4

LP

L

1

-6

LP

L

1

-8

6

8

16

32

Full

BP

EfficientFormer-L7

100

150

200

250

300

350

400

450

50

60

70

LP

L

1

-2

LP

L

1

-4

LP

L

1

-6

LP

L

1

-8

4

6

8

16

32

Full

BP

EfficientFormerV2-S0

Ours

LoRA-all

Full BP

500

1000

1500

2000

2500

3000

3500

70.0

72.5

75.0

77.5

80.0

82.5

LP

L

1

-2

LP

L

1

-4

LP

L

1

-6

LP

L

1

-8

6

8

16

32

Full

BP

EfficientFormerV2-L

Accuracy [%]

Computation [MFLOPs]

computation. When using smaller ranks, LBP-WHT significantly reduces the cost with only minor accuracy costs. For example, when training the final stage of EfficientFormer-L1 using LP L 1 -4 base selection, LBP-WHT achieves a 3 . 5 × speedup with just a 1% loss in accuracy on CIFAR100. With LP L 1 -8 base selection, LBP-WHT achieves even higher accuracy (79.34%) with a 1 . 2 × speedup .

These results underscore the merits of our method. As shown in Table 2, our method achieves computational savings by systematically reducing the computational cost for all operations during backpropagation, including the gradient computation for both input and weight. Specifically, when we apply a similar rank for LoRA-all and LBP-WHT, we anticipate that both methods will have similar computational costs for computing the weight gradient. However, as LoRA-all cannot speed up the gradient computation for the input while LBP-WHT can, our LBP-WHT method requires only half the total computation of LoRA-all. Consequently, for a similar computational budget, LBP-WHT can employ a higher rank for low-rank projection, thus leading to a higher accuracy. For example, when training the entire EfficientFormerV2-S0 model, LBP-WHT with LP L 1 -4 (rank 10) only requires 1187 MFLOPs, which is 62% of the computational cost for LoRA-all. Thus, for a similar budget, LBP-WHT can use a rank 28 projection (LP L 1 -7) and achieve a higher accuracy.

4.3 Semantic Segmentation

Table 3 presents the experimental results for adapting the ADE20K-pretrained Segformer model on Cityscapes and augmented Pascal VOC 2012 dataset. Our LBP-WHT has better results in most cases. For instance, when partially training on the Cityscapes dataset, our approach using LP L 1 -4 base selection achieves a mIoU score approximately 0.9% higher than that of LoRA-all. Moreover, it only requires 1481.9 MFLOPs, which is 4.2 × faster. These findings not only further validate the efficacy of our method, but also demonstrate its broad applicability across key computer vision tasks.

4.4 Exploration 1: Different Ranks for Low-rank Projection in LBP-WHT

Figure 3 shows the accuracy achieved when adapting ImageNet-pretrained ViTs for CIFAR100, with varying ranks for low-rank model adaptation. Our observations from this figure are as follows:

0

2

4

6

0

2

4

6

WHT Base Index for

Bi

,

j

- i

Multihead Attention-QKV

0

2

4

6

0

2

4

6

Multihead Attention-Projection

0

2

4

6

0

2

4

6

Feed Forward 1

0

2

4

6

0

2

4

6

Feed Forward 2

7.5

7.0

6.5

6.0

9.0

8.5

8.0

7.5

7.0

9.5

9.0

8.5

8.0

7.5

10

9

8

7

Unit: Decibel

WHT Base Index for

Bi

,

j

- j

Our LBP-WHT method consistently outperforms the LoRA-all method, i.e. , for a similar level of computation, LBP-WHT yields higher accuracy.

By altering the rank, LBP-WHT provides a broader range of cost options than the baseline method.

LBP-WHT's accuracy monotonically improves as more ranks are employed for projection.

For all models with our LBP-WHT method, a generally concave accuracy-computation curve is observed. This indicates strong diminishing returns in using larger ranks.

LBP-WHT with LP L 1 -6 base selection achieves an accuracy very close to that of full BP.

Our first observation further confirms the superior performance of our method. The second observation indicates the broad applicability of our method. For instance, for edge devices with limited computational budgets, like Raspberry Pi, we can employ LBP-WHT with a lower rank to reduce computational cost. On the other hand, for more powerful devices, such as personal desktops equipped with GPUs, a larger rank can be chosen to enhance accuracy. This ensures that users with various computational resources and constraints can benefit from our method.

The last three observations offer guidelines for rank selection with our LBP-WHT method.

With strict computational constraints: Given our third observation above, if there is a hard limit on the maximum number of FLOPs allocated for training, selecting the rank for LBP-WHT is straightforward: we simply opt for the largest possible number of ranks, which in most cases yields the highest accuracy.

Without strict computational constraints: Our final two observations suggest that training efficiency can be characterized by the marginal accuracy, or the slope of the accuracycomputation curve. As shown in Figure 4, before LP L 1 -4, the marginal accuracy is significantly greater than zero. However, after LP L 1 -6, the marginal accuracy is very close to zero. This

2000

4000

6000

8000

10000

0

1

LP

L

1

-4

LP

L

1

-6

EfficientFormer-L1

1000

1500

2000

2500

Computation [MFLOPs]

2.5

5.0

7.5

LP

L

1

-4

LP

L

1

-6

EfficientFormerV2-S0

Marginal Accuracy [%/GFLOPs]

implies that choosing fewer ranks than LP L 1 -4 or more than LP L 1 -6 may not be advantageous, as it could either forgo the opportunity for good performance with a small amount of computation or waste computation with little benefit. Thus, a good selection empirically lies between LP L 1 -4 and 6.

4.5 Exploration 2: Different Bases Selection Method

Figure 5 shows the WHT spectrum for the gradient w.r.t. layer output ( g y ) collected from the last attention block in EfficientFormer-L1. We observe that most energy concentrates in the low-frequency area, i.e. , the top-left corner, which supports claim in Section 3.2 that natural images have strong spatial locality and strong low-frequency components. Furthermore, Figure 5 demonstrates that by choosing WHT

Table 4: Experimental results for adapting EfficientFormer-L1 on CIFAR100 and CIFAR10 with different base selection methods. Accuracy is in percentages (%)

bases with low-frequency responses - that is, using the selection methods LP L 1 and LP L ∞ - we can preserve most of the energy and minimize error during the low-rank projection. As indicated in Table 4, both of these low-pass base selection methods yield accuracy levels very similar to those achieved with the low-heuristic-error (LHE) method. The LHE method profiles the WHT spectrum

and selects the WHT bases with the strongest response. Given that the LP L 1 base selection method eliminates the need for profiling (unlike LHE) and offers a more favorable balance between accuracy and cost compared to LP L ∞ , we have selected LP L 1 as the standard method for LBP-WHT.

4.6 Limitation and Broader Impacts

Full training with a small number of ranks: As shown in Table 2, we find that the accuracy degradation is not negligible when using a small number of ranks for LBP-WHT full training. We consider this is the issue of accumulating error introduced by low-rank projection during BP. We expect that an improved approximation method can perform even better. Of note, even with accuracy degradation, our method still consistently outperforms the baselines, i.e. , LoRA-based methods.

Broader Impact: Our method greatly reduced the barrier for training ViTs. As a positive feature, our method may push the development of privacy-centric on-device training methods like federated learning; our method may also enable more researchers to test their ideas with powerful ViTs. On the other hand, our method may lower the barrier for irresponsible customization and use of ViT.

5 Related Work

Low-rank Model Adaptation: [14] proposes to speed up transformer training by attaching and training only low-rank branches to the linear layer. More precisely, consider a linear layer with equation y = x · w T , where x is the input, y is the output, and w is the weight. LoRA adds a branch that contains low-rank weights w A and w B , forming y LoRA = x · w T + x · ( w A · w B ) T . The original weight w is kept frozen, while the appended weights w A and w B are trained. Since the ranks of w A and w B are much smaller than that of w , the computation needed to calculate the gradients with respect to w A and w B is significantly reduced. However, this method does not decrease the computation for calculating the gradient w.r.t. x . This is because it still needs to propagate the gradient through the weights w to x , which considerably limits the performance of LoRA-based methods. As demonstrated in Figure 3, our LBP-WHT, requires much less computation while having better accuracy than LoRA-based methods; this is because our method reduces the computation for all procedures in BP, including the gradient calculation for both input and weights.

Other Orthogonal Methods for On-device Training: Previous research on efficient on-device model adaptation falls into two main categories. The first category [33, 38-43] suggests reducing the computational cost of arithmetic operations (addition and multiplication) in BP through quantization. The second category [20, 44] proposes to append a smaller neural network to the original model and accelerate adaptation by only training the attachment. To the best of our knowledge, our paper is the first to apply low-rank BP for ViT model adaptation. Therefore, our method, LBP-WHT, is distinct from previous research and can be combined with those methods for enhanced performance.

6 Conclusion

In this paper, we have addressed the problem of efficient model adaptation for ViT. We have proposed a novel low-rank BP technique designed to reduce the computational load associated with the propagation of gradients through the linear layers of ViT, which is a significant bottleneck when fine-tuning ViT. In Section 3, we introduced the LBP-WHT method as a solution to accelerate model adaptation. More specifically, LBP-WHT operates by projecting the gradient w.r.t. the output ( g y ) into a low-rank space, performing matrix multiplications within this low-rank space, and then projecting the results back into the original space. Since all matrix multiplications occur in a low-rank space, the computational cost is significantly reduced. Additionally, thanks to the properties of the Walsh-Hadamard Transform (WHT), the overhead for these projections is minimal (as discussed in Section 3.3). Through extensive experiments in Section 4, we have demonstrated the efficiency and broad applicability of our method. Our LBP-WHT approach consistently outperforms existing methods with a significant speedup and higher accuracy.

Acknowledgments and Disclosure of Funding

This work was supported in part by the US National Science Foundation (NSF) grant CNS-2007284.

References

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin Transformer v2: Scaling up Capacity and Resolution. In IEEE conference on computer vision and pattern recognition , 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , 2021.

Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Stefano Mattoccia. Monovit: Self-supervised monocular depth estimation with a vision transformer. In International Conference on 3D Vision , 2022.

Ashutosh Agarwal and Chetan Arora. Depthformer: Multiscale vision transformer for monocular depth estimation with local global information fusion. arXiv preprint arXiv:2207.04535 , 2022.

Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from a single input image. In IEEE Winter Conference on Applications of Computer Vision , pages 806-815, 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Artificial intelligence and statistics , 2017.

Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A Survey on Federated Learning. Knowledge-Based Systems , 2021.

Wei Chen, Kartikeya Bhardwaj, and Radu Marculescu. Fedmax: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020 , 2021.

Yuedong Yang, Zihui Xue, and Radu Marculescu. Anytime Depth Estimation with Limited Sensing and Computation Capabilities on Mobile Devices. In Conference on Robot Learning , 2022.

Guihong Li, Sumit K Mandal, Umit Y Ogras, and Radu Marculescu. FLASH: Fast Neural Architecture Search with Hardware Optimization. ACM Transactions on Embedded Computing Systems , 2021.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Herbert John Ryser. Combinatorial Mathematics . Carus Mathematical Monographs. Mathematical Association of America, 1963. doi: 10.5948/UPO9781614440147.

J.L. Shanks. Computation of the fast walsh-fourier transform. IEEE Transactions on Computers , C-18(5): 457-459, 1969. doi: 10.1109/T-C.1969.222685.

Rafael C. Gonzales and Paul Wintz. Digital Image Processing (2nd Ed.) . Addison-Wesley Longman Publishing Co., Inc., USA, 1987. ISBN 0201110261.

Yuedong Yang, Guihong Li, and Radu Marculescu. Efficient On-device Training via Gradient Filtering. In IEEE conference on computer vision and pattern recognition , 2023.

Marc-Antoine Parseval. Mémoire sur les séries et sur l'intégration complète d'une équation aux différences partielles linéaires du second ordre, à coefficients constants. Mém. prés. par divers savants, Acad. des Sciences, Paris,(1) , 1:638-648, 1806.

Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce Memory, not Parameters for Efficient On-device Learning. In Advances in Neural Information Processing Systems , 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision , 2015.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine-grained Categorization. In Proceedings of the IEEE international conference on computer vision workshops , 2013.

Maria-Elena Nilsback and Andrew Zisserman. A Visual Vocabulary for Flower Classification. In IEEE conference on computer vision and pattern recognition , 2006.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - Mining Discriminative Components with Random Forests. In European Conference on Computer Vision , 2014.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and Dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv preprint arXiv:1711.05101 , 2017.

Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed, 2022.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In IEEE conference on computer vision and pattern recognition , pages 633-641, 2017.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS) , 2021.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device Training under 256kb Memory. In Advances in Neural Information Processing Systems , 2022.

Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: Transfer Learning through Adaptive Fine-tuning. In IEEE conference on computer vision and pattern recognition , 2019.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning Transferable Features with Deep Adaptation Networks. In International conference on machine learning , 2015.

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How Transferable are Features in Deep Neural Networks? In Advances in neural information processing systems , 2014.

Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking Vision Transformers for MobileNet Size and Speed. arXiv preprint arXiv:2212.08059 , 2022.

Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A Statistical Framework for Low-bitwidth Training of Deep Neural Networks. In Advances in Neural Information Processing Systems , 2020.

Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in neural information processing systems , 2018.

Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-Low Precision 4-bit Training of Deep Neural Networks. In Advances in Neural Information Processing Systems , 2020.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research , 2017.

Kang Zhao, Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Distribution Adaptive INT8 Quantization for Training CNNs. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Ziyang Hong and C Patrick Yue. Efficient-grad: Efficient training deep convolutional neural networks on edge devices with grad ient optimizations. ACM Transactions on Embedded Computing Systems , 2022.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder Side-tuning for Parameter and Memory Efficient Transfer Learning. In Advances in Neural Information Processing Systems , 2022.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40 (6):1452-1464, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an efficient training method, called Low-rank Back Propagation via Walsh-Hadamard Transformation (LBP-WHT). Specifically, unlike previous works that target low-rank adaptation for model parameters, LBP-WHT aims to reduce the computational cost during backpropagation by projecting the gradients w.r.t inputs into a low-rank space, thus the overall computation can be done efficiently. Extensive experiments on classification and semantic segmentation have demonstrated that LBP-WHT significantly reduces MFLOPs during partially training.\n\nSTRENGTHS:\n1. To my knowledge, adopting low-rank backpropagation for ViT adaptation is novel.\n2. The method is easy to follow. The visualization of LBP-WHT in Figure 2 clearly presents the proposed framework.\n3. The results are impressive. Based on partially training (Table 2), LBP-WHT achieves great efficiency-accuracy trade-off.\n\nWEAKNESSES:\n1. Theoretical complexity reduction cannot reflect the real speedup for training. It would be better for the authors to report training time (GPU hours) or training throughput (images/s) in main experiments, especially for Table 2. Besides, as shown in Table 5 of the supplementary material, the authors have compared latency across different hardwares. However, the real speedup is only around 1-2x, which is very different from the reduction ratio of MFLOPs (6-7x).\n2. The experiments do not report memory cost. Recent works (especially for LLMs) usually care more about memory cost during adaptation since fully finetuning or low-rank finetuning requires intensive memory. Moreover, on-device training also requires low-memory consumption, see [33]. However , the effect of the propsoed method on memory cost is also unknown.\n3. As this paper aims to reduce the computational cost during backpropagation, it does not have advantages in inference. Actually, it even introduces overhead on model parameters as it is not parameter-efficient on downstream tasks.\n4. Lack ablation study of combining LoRA with LBP-WHT. Since authors have claimed that the proposed technique is complementary to other works, it should have at least one experiment to prove this claim and show the advantage of their method.\n\nQUESTIONS:\n1. In Table 3, why LoRA and LBP-WHT have much more MFLOPs than Full BP on partial training? \n2. Can authors also explain why the proposed method performs better on partial training?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to efficiently finetune Vision Transformers (ViTs) for specific tasks such as image classification and semantic segmentation. The method, Low-rank Backpropagation via Walsh-Hadamard Transformation (LBP-WHT), reduces the computational cost of backpropagation in ViTs by projecting the gradients (dy/dx and dy/dw) into a low-rank space using WHT basis where the bases are made up of -1's and 1's. Experiments are only shown with EfficientFormer (V1, V2), SwinTransformerV2 and SegFormer models. Proposed method outperforms LoRA in terms of both accuracy and efficiency. The paper also shows actual speed improvements on hardware (GPU/CPU) which is promising.\n\nSTRENGTHS:\n### Simple method with significant speedups and possible efficient implementation\n* The proposed method is simple in design and easy to implement *efficiently* on any target hardware. The authors have already shown speed-ups on CPU and GPU with a naive implementation which I believe can be sped up for commercial deployment\n\n### Good results compared to LoRA\n* The experiments demonstrate strong results on EfficientFormer (V1,V2), SwinTransformer and SegFormer models as compared to LoRA. The proposed method achieves better speed-up than LoRA while also not losing as much accuracy.\n\n### Can be applied to both dy/dw and dy/dx\n* LoRA cannot speed-up the gradient computation for dy/dx but the proposed method can LBP-WHT.\n\nWEAKNESSES:\n### Why only ViTs? That too only smaller scale ViT models?\n* The proposed method is an efficient variant of backpropagation that should work with any architecture that contains linear layers. So, why not evaluate the method with MLP-Mixer models (Tolstikhin, Ilya O., et al. \"Mlp-mixer: An all-mlp architecture for vision.\" NeurIPS 2021) which are full of linear layers? \n* Also, many CNN models (such as all MobileNets, EfficientNets) use pointwise convolution layers and pointwise convolutions are basically linear layers. The authors should show results of the proposed method with such architectures too, not just ViTs.\n* How does it compare with LoRA on Large Language Models? The LoRA paper showed exceptional results with LLMs, but in this paper, the proposed method has been compared with LoRA only on small-scale ViTs.\n\n### Comparison with Gradient Compression\n* The main contribution of this paper is a method to reduce the cost of computing gradients, which they do so by computing low-rank approximations. There is a popular method called Gradient Compression (Lin, Yujun, et al. \"Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training.\" ICLR 2018) which compresses gradients by 270x to 600x without losing accuracy. Comparison with such a method should be shown in the results.\n\nQUESTIONS:\n* Do you use your method in all linear layers or only the attention modules (like \"LoRA baseline\")? My guess is all linear layers, but I don't see this explicitly mentioned.\n\nLIMITATIONS:\nYes, limitations have been sufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper presents to compute the low-rank approximation for the matrices used in back-propagation, and thus it can save the computation in multiplication. The proposed approach is more efficient than LoRA because no need to backpropagate to the original weights, and experiments also support the claims.

STRENGTHS:
1. The proposed approach is simple and well-motivated.
2. The experimental results show that the approach is better than LoRA.

WEAKNESSES:
1. Although the paper claims that the approach is complementary to other efficient approaches (such as quantization), there is no experimental support for it.

QUESTIONS:
1. In Table 1, what is the difference between the r in "LPL1 -r” and R? I thought they both represent rank.
2. Is the WHT differentiable? How is the model trained with this operation.
3. I thought WHT^{-1} was not a square matrix, was it? If not, how to compute WHT^{-1}?
4. How is the "Speed up" in table 1 measured?

---
**Post-rebuttal**
Thank you for the authors' response. I have read it and it addressed my questions.

LIMITATIONS:
1. There is no experiment to support whether the approach can work well with other efficient approaches.
2. The experimental results are only for vision tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author discusses the issue of computational complexity during matrix multiplication in the backpropagation of linear layers in the vision transformer architecture. To address this issue, the author proposes a method called Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT), which projects the gradient into a low-rank space and performs BackPropagation computations. Finally, the author evaluates the proposed method using different models such as ViT and a hybrid convolution-ViT model on multiple datasets. The author claims that LBP-WHT achieves 10.4% higher accuracy than the state-of-the-art baseline, providing SOTA performance on the CIFAR100 dataset with the EfficientFormer-L1 model.\n\nSTRENGTHS:\n1. This paper is well-written and easy to understand.\n\n2. The Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method in the gradient domain is new and effective.\n\n3. The novelty of LBP-WHT as the first work to accelerate ViT training using low-rank BP (Backpropagation). It also demonstrates the ability to lower the computational barrier and enable the adaptation of large ViT models on resource-constrained edge devices.\n\n4. The paper addresses an essential problem in transformer architecture and proposes a novel solution that achieves promising results.\n\n5. The author provides more quantitative results on various tasks, including classification and semantic segmentation. Additionally, the comparison with LoRA, one of the state-of-the-art (SOTA) models, adds value to the evaluation.\n\nWEAKNESSES:\nWeaknesses:\n\n1. The literature survey section in this paper is a bit weak. The author should compare related/similar transformer models, particularly QLoRA, and similar work on \"Hadamard product for low-rank bilinear pooling.\" A more thorough literature review is necessary in this regard.\n\n2. It would be beneficial to add a section briefly discussing LoRA, QLoRA, and the proposed method, as well as comparing the results with the proposed method.\n\n3. The author should provide results on larger datasets like the standard ImageNet-1k and evaluate other large datasets such as JFT-Imagenet-21K.\n\n4. Instead of Swin-Transformer, the author should evaluate the proposed method on the SOTA model like WaveViT or SpectFormer to validate the SOTA performance.\n\n5. The paper misses one of the core aspects of machine learning practice: readability and reproducibility of results. It would be more transparent and beneficial if the author could provide source code to reproduce the results, similar to what was done with LoRA.\n\n6. It seems like a minor typo: \"finally project them black\" should be \"finally project them back.\"\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

comments: SUMMARY:
The paper proposes a cost-efficient back-propagation method using a low-rank trick for vision transformers. When transferring a huge-size vision transformer to the downstream task, the training cost would be high. The paper tackles the problem using low-rank approximation with Walsh-Hadamard Transformation (WHT). The experiments demonstrate the training efficiency of the proposed low-rank back-propagation.

STRENGTHS:
- The paper presents their motivation and method well. 
- Experiments show promising performance. 
- Ablation studies are informative.

WEAKNESSES:
- The targeting scenario of the paper seems somewhat narrow. The paper focuses on vision transformers and its adaptation (transfer learning scenario). It would be nice to discuss the possibility of the proposed method for the efficient pre-training scenario. Also, a discussion about general transformer (e.g., language modeling transformer) adaptation beyond the vision domain would make the paper stronger.

QUESTIONS:
- The analysis of computation overhead is in section 3.3., but I could not find the cost of computing the low-rank projection matrix (p). It is neglectable, right? 
- Could you provide more experiments on vanilla vision transformers?

LIMITATIONS:
The authors addressed limitation in the manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26