Title: Activation Map Compression through Tensor Decomposition for Deep Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Learning

Computer Vision

Compression
Abstract: Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.

Activation Map Compression through Tensor Decomposition for Deep Learning

Abstract

Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint. 1

1 Introduction

Recent advances in Deep Learning have enabled Deep Neural networks to be used as an efficient solution for a wide variety of use cases, including computer vision [21, 39, 30], speech recognition [8, 31] and natural language processing [45, 42]. Much of this performance improvement is linked to the exponential increase in the number of parameters in the neural architectures. According to Sevilla et al. , the release of AlphaGo [47] in late 2015 marks the advent of a new era, which they call the 'Large Scale Era', in reference to the computational cost of training doubling every 8 to 17 months [38]. At the root of exponential growth in neural network size is the improvement in hardware capabilities, particularly those designed for large-scale parallel computing, such as GPUs and TPUs [1]. While this trend demonstrates the strength of neural networks as a powerful generalization tool in many fields, it goes in the opposite direction when it comes to environmental concerns [41], making the deployment of newer architectures increasingly difficult. This is particularly true for edge devices such as mobile phones and embedded sys-

Forward

Backward

Memory

a

Memory

a

Efficient

reconstruction

Compress

and store

tems, which cannot afford the high computing or memory costs. To address these challenges, three interconnected factors must be taken into account: power consumption, memory usage, and latency.

When considering larger neural networks with more layers and nodes, reducing their storage and computational cost becomes essential, especially for certain real-time applications such as edge computing. In addition, recent years have seen significant advances in the fields of intelligent edge and embedded intelligence, creating unprecedented opportunities for researchers to address the fundamental challenges of deploying deep learning systems on edge devices with limited resources (e.g. memory, CPU, power, bandwidth). Efficient deep learning methods can have a significant impact on distributed systems and embedded devices for artificial intelligence. Since training is supposed to take place in the cloud, most research on model compression and acceleration is specifically focused on inference [7]. There is, however, an emerging area of research concerning on-device training, which represents a decisive advance in the field of artificial intelligence, with considerable implications for a variety of practical situations [17]. Models trained offline on a dataset built at one point in time tend to fall victim to data drift when deployed 'in the wild' [36]. Its combination with online learning strategies has the potential to enable continuous model improvement after deployment [14], thus adapting the model predictions to observed evolutions in the data distribution. We can illustrate this with the example of sensors in autonomous vehicles, where deep learning models must correctly classify vehicles with new designs never seen in the training set. Other advantages of on-device learning include security and privacy. By processing data locally, sensitive information remains more secure and less susceptible to data breach, a major concern in applications such as healthcare.

The main challenge limiting the feasibility of on-device learning lies in the computational demands of the backward pass, as gradient computation and parameter updates are significantly more resourceintensive than the forward pass (Appendix A.1). On embedded devices, memory and computation limitations act as strict budgets that must not be exceeded. Some approaches address the memory constraints by exploring alternatives to traditional backpropagation, including unsupervised learning for image segmentation [51], the Forward-Forward algorithm [16], and PEPITA [34]. While promising, these methods typically fall short of backpropagation-based techniques in terms of performance. A pioneering effort by Lin et al. , demonstrated that fine-tuning a deep neural network within a 256 kB of memory is feasible by selectively updating a sub-network, achieving a good results [25]. In a complementary approach, Yang et al. proposed reducing the number of unique elements in the gradient map through patch-based compression of the input and gradients of a given layer with respect to the output, thereby lowering memory costs and speeding up the learning process [53].

Inspired by tensor decomposition methods, we propose a method that compresses activation maps, reducing the memory demands for backpropagation while maintaining the deep neural network's generalization capability(Fig. 1). Our approach adaptively captures the majority of tensor variance, offering guaranteed accuracy in the gradient estimation. The key contributions of our work are as follows:

We propose to exploit powerful low-rank approximation algorithms to compress activation maps, enabling efficient on-device learning with controlled information loss (Sec. 3.2 and Sec. 3.3).

We provide a theoretical foundation for our method, along with an error analysis demonstrating that high compression ratios are achievable with limited performance degradation (Sec. 3.4).

We extensively explore a diverse experimental landscape, demonstrating the generalization capacity of our proposed algorithm (Sec. 4).

2 Related Works

Tensor Decomposition. Model compression and acceleration is a well-developed and ongoing area of research in the deep learning community. It can be divided into two major areas, namely hardware and algorithm design. In the case of algorithmic compression, the five main components that stand out are efficient and compact model design, data quantization, network sparsification, knowledge distillation, and tensor decomposition [4, 7]. All these approaches aim to reduce the space occupied by network parameters. Also known as a low-rank approximation, tensor decomposition has emerged as a robust solution due to its combination of grounded theoretical findings and its

practicality in terms of hardware implementation [49]. Originating in the field of systems theory and signal processing [28], low-rank approximation has attracted growing interest in the deep learning community. Early examples were limited to the application of singular value decomposition (SVD) used to compress fully connected layers [50]. More recent work includes the application of generalized Kronecker product decomposition (GKPD) to both linear and convolutional layers [13], the introduction of semi-tensor product (STP) to improve compression ratios with reduced loss [56] or even more recently the acceleration of inference and backpropagation in vision transformers [52]. Evolutions in this domain can be summed up as improvements in compression ratios, latency, and power consumption, with limited performance loss and diversification of architectures considered.

Activation Map Compression. The term 'activations' here refers to the outputs of each layer after the non-linearity has been applied, which are then fed to the next layer. While model compression generally aims at reducing the storage space occupied by parameters (which typically translates into network acceleration during inference), a key observation is that activations occupy much more space than parameters in memory during backward pass, as they are required to compute weight derivatives [2]. This state of facts motivates a new line of research, whose main objective is to compress activation maps using methods inspired by the literature on weight compression. For example, we mention the use of quantization [9], sparsification [22], a combination of both with the addition of entropy encoding [12], or even the application of wavelet transform in combination with quantization [10]. With the exception of Eliassen et al. 's work which accelerates training runtime, most of these works focus on accelerating inference, in a similar way to traditional model compression.

On-device Learning. Today's typical pipeline for on-device AI consists of designing resourceefficient deep neural networks [24], training them offline on target data, compressing them, and deploying them for inference on the resource-constrained environment. Alongside the rapid commercial expansion of the IoT field and the ubiquitous presence of embedded devices in our daily lives, AI at the edge has naturally attracted a great deal of interest. However, recent research has shown that real-world data often deviated from training data, resulting in poor predictive performance [40]. In such a context, continual learning proved to be a strong candidate for ensuring ongoing adaptation after deployment. Similar to the human learning process, it provides models with the ability to adapt to new incoming tasks, ideally without falling into the trap of catastrophic forgetting (i.e., degrading performance on older tasks in favor of new ones) [11, 33, 19]. In this respect, the very low memory and computing resources of extreme edge devices are an obstacle to the high cost of backpropagation. Lin et al. have shown that this challenge can be overcome by selectively fine-tuning a sub-graph of pre-trained networks on a general dataset [25]. Their work started a line of research showing the possibility of improvement through careful selection of channels to be updated at training time [23, 35]. However, none of these works addresses the computational cost of training a neuron, which is related to both the cost of loading the necessary weights and activations into RAM.

Gradient Filtering. In their work, Yang et al. demonstrate that it is possible to significantly reduce the memory and computational cost of full layer learning using a method called gradient filtering, which reduces the number of unique elements in the gradient map [53]. In such a frame and with the same specific objective, we propose to compress the activation maps by using tensor decomposition to minimize the memory cost. We perform this while providing a strong guarantee on the signal loss: unlike gradient filtering, we adaptively size the decomposed tensor to guarantee minimal information loss. In the next section, we will illustrate our proposed approach.

3 Method

In this section, we motivate our compression proposal by exposing the major memory bottleneck of backpropagation (Sec. 3.1). We then present our contribution which is the compression of activation maps through two well-known low-rank decomposition strategies, namely Singular Values Decomposition (SVD) and its tensor counterpart Higher Order SVD (HOSVD) (Sec. 3.2). In Sec. 3.3 we explore the induced changes to backpropagation calculation and in Sec. 3.4 we study the resulting memory and computational complexity, alongside an evaluation of the error introduced. Our ultimate goal here is to reduce the memory footprint of backpropagation (BP).

3.1 The Memory Bottleneck of Backpropagation

Following the formalism introduced in [2], we consider a simple convolutional neural network (CNN) represented as a sequence of n convolutional layers (excluding the bias for simplicity):

where W i ∈ R C ′ × C × D × D represents the filter parameters of the i th layer, with a kernel size of D × D . This layer receives a C -channel input and produces an output with C ′ channels. For this layer, we denote the input and output activation tensors as A i ∈ R B × C × H × W and A i +1 ∈ R B × C ′ × H ′ × W ′ , respectively. Note that H and W denote the height and width of each element of the input, while H ′ and W ′ denote the height and width of each element of the output, with B as the minibatch size. The loss L is computed at the output of the network and backpropagated until the i th layer as ∂ L ∂ A i +1 . At this stage, the filter parameters are updated thanks to the computation of ∂ L ∂ W i and the loss is propagated to the previous layer as ∂ L ∂ A i . The computation of these terms follows the chain rule:

where conv( · ) is the traditional convolution operation, convolving the kernel ∂ L ∂ A i +1 with the input A i ; while conv full ( · ) is the convolutional operation that naturally maps the input ∂ L ∂ A i +1 to an output with the same dimensions as A i by using the 180 ◦ rotated kernel W i .

From (2), it is clear that computing the weight derivatives requires to load input activation A i and (3) shows that the weights W i must be loaded into memory to calculate activation derivatives. We obtain the same conclusions for linear layers and provide the demonstration in Appendix A.2.

To save memory, two possibilities naturally emerge: either compressing the weights or compressing the activation. Weight compression is an extensively explored matter for network acceleration and we do not intend to further this area of research in this work. This leaves us with activation compression, which is still a new domain of exploration, but shows great potential for prospectively enabling on-device backpropagation. In such a regard, thanks to its strong theoretical grounding, tensor decomposition stands as a promising approach.

3.2 Tensor Decomposition

We will present here first the general Singular Value Decomposition (SVD) approach, which is then extended to a multidimensional variant (HOSVD), instrumental for our purposes.

SVD. Given a matrix A ∈ R M × N , applying SVD to A consists in a factorization of the form:

where Σ is a rectangular diagonal matrix composed of r singular values s i ∈ [1 ,r ] with r the rank of A . From this, we can deduce the amount of overall variance σ 2 i explained by the i th pair of SVD vectors as σ 2 i = s 2 i / ∑ j s 2 j .

Let us assume the singular values in Σ are ordered in descending order, s i ≥ s j , ∀ i ≤ j . Given a desired threshold of cumulated explained variance ε ∈ [0 , 1] , it becomes easy to find the 'truncation threshold' which is the minimal K ∈ [1 , r ] such that ∑ K i =1 σ 2 i ≥ ε . We can then approximate A by only selecting the K first columns of U and V and the K first singular values from Σ , according to:

Historically, SVD was the first example of tensor decomposition applied to neural network compression and acceleration [55] but applied to the model's parameters [49]. We take the SVD decomposition to the activation maps as one possible baseline to compare with more complex low-rank compression solutions.

In the case of convolutional layer i , since SVD is designed for matrix decomposition, we will simply reshape the activation A i as matrix A i of dimensions B × ( CHW ) . Given a desired level of explained variance ε , we obtain the decomposition described in (5). We then store in memory the terms

U ( K ) × Σ ( K ) and V T ( K ) , meaning that instead of storing Θ space ( BCHW ) unique elements in memory, we are only storing Θ space [ K ( B + CHW )] unique elements. Regarding linear layers, activations are M × N matrices, and applying SVD is much more straightforward, leading to the storage cost of Θ space [ K ( M + N )] instead of Θ space ( MN ) elements. The larger the explained variance, the closer ˜ A will be to A , intuitively allowing for better estimation when performing backpropagation. However, this also means a larger K which results in a larger memory occupation. The goal is then to find an efficient trade-off between the desired explained variance and compression rates.

HOSVD. By construction, SVD is designed for matrix decomposition. It was demonstrated that the reshaping operation on tensors introduced structure information distortion, leading to sub-optimal performance [49]. Other methods more suited for tensor decomposition such as Canonical-Polyadic (CP) decomposition [20] or Tucker decomposition [43] were naturally introduced to tackle this issue. Given a n th -order tensor T ∈ R M 1 × M 2 ×···× M n , its Tucker decomposition corresponds to:

where S ∈ R L 1 × L 2 ×···× L n is the core tensor which can be viewed as a compressed version of T , U ( j ) ∈ R M j × L j are the factor matrices and their columns correspond to the principal components over the j th mode. The i -mode product ' × i ' of a n th -order tensor G ∈ R P 1 × P 2 ×···× P n and a matrix B ∈ R Q × P i is a n th -order tensor R∈ R P 1 ×···× P i -1 × Q × P i +1 ×···× P n which can be expressed as:

In such a setup, Higher-Order SVD is a specific case of Tucker decomposition where the factor matrices U ( j ) are orthogonal [44]. As a follow-up to SVD, we propose to compress activation tensors through HOSVD, with the intuition that each dimension encodes a different variance, potentially providing enhanced compression rates for equivalent performance and vice-versa. Similarly to traditional SVD, we can truncate S and each U ( j ) along each mode given a desired level of explained variance resulting in:

where U ( j ) ( K j ) ∈ R M j × K j corresponds to the K j first columns of U ( j ) and ˆ S ∈ R K 1 ×···× K n is the truncated version of S . In the following section, we focus on HOSVD decomposition, more precisely on the alterations induced to the backpropagation graph, the resulting compression and speedup ratio, and the error bound. An equivalent analysis can be made for SVD.

3.3 Backpropagation with Compressed Activations

Prior works [13, 18] already demonstrated the possibility of performing backpropagation operations in the decomposed space without relying on the recomposition of compressed tensors. In their work, Kim et al. compress a 4 th -order kernel tensor W ∈ R C ′ × C × D × D through Tucker decomposition limited to mode 1 and 2 :

Then, it is shown that the output is calculated through

Our problem is similar although with some fundamental differences linked to the need to reconstruct gradients . A convolution operator is still required to compute the weight derivatives as introduced in (2): we apply a specific case of Tucker to one of the two components, namely the activation A i . Following the same reasoning, we derive that the computation of ∂ L ∂ W i in (2) becomes:

where conv ∗ is a 2D convolution with a specific kernel size as defined in (23), and U ( j ) ( K j ) is the vertically padded version of U ( j ) ( K j ) as defined in (19). Demonstration details are available in

5

10

K

1 =

K

2 =

K

3 =

K

4

10

0

10

1

10

2

10

3

10

4

10

5

RC

B = C = C' = H = W = H' = W' = 16

B = C = C' = H = W = H' = W' = 32

B = C = C' = H = W = H' = W' = 64

B = C = C' = H = W = H' = W' = 128

(a)

5

10

K

1 =

K

2 =

K

3 =

K

4

10

0

10

1

10

2

10

3

RS

B = C = C' = H = W = H' = W' = 16

B = C = C' = H = W = H' = W' = 32

B = C = C' = H = W = H' = W' = 64

B = C = C' = H = W = H' = W' = 128

(b)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ε

10

0

10

1

10

2

10

3

10

4

SNR

˜

I

(c)

Appendix A.3. This way, we can compute the approximated weight derivative without reconstructing the activation, through the successive computation of simpler convolutions.

Fig. 1 illustrates our method. During training, the forward pass proceeds as usual, with one key modification to memory management. Instead of keeping complete activation maps in memory, we store only their principal components, which are derived from the decomposition process. During the backward pass, the principal components are retrieved from memory and used for calculations as described in (11).

3.4 Complexity and Error Analyis

Computational Speedup and Space Complexity. For simplicity we assume here that A i +1 and A i have the same shape and that the truncation thresholds across all modes are equal for HOSVD, i.e., K 1 = K 2 = K 3 = K 4 . Considering different thresholding values, we can predict the relative improvement of HOSVD to vanilla training in both space complexity (30) and computational speedup (29), for a backward pass. As illustrated in Fig. 2a and Fig. 2b, our method is more effective for both space complexity and latency for the backward pass when the truncation threshold decreases and the activation size increases. More details are provided in Appendix A.4.

We hypothesize that the first components along each dimension are enough to encode most of the variance, implying that with relatively low values of K j , we can achieve good training performance. We later empirically confirm this assumption in a variety of experimental setups (Sec. 4.2).

Signal to Noise Ratio. For simplicity, we rewrite in this paragraph ∂ L ∂ W i as ∆ W , A i as I and ∂ L ∂ A i +1 as ∆ Y . Regarding the error introduced by truncating I in order to retain the components corresponding to a given level of explained variance ε ∈ [0 , 1] , we demonstrate that the energy contained in the resulting gradient is equal to the energy contained in ˜ I . We note I, ∆ W and ∆ Y as the input activation, weight derivative, and output activation derivative in the frequency domain; I [ u, v ] , the spectrum value at frequency ( u, v ) . Applying the discrete Fourier transformation to ˜ I gives us ˜ I = εI , which we use to compute the signal to noise ratio SNR ˜ I :

Similarly, in the frequency domain, as the convolutional operation becomes a regular multiplication, ∆ ˜ W becomes ∆ ˜ W = ˜ I ∆ Y = εI ∆ Y . As for (12), we obtain SNR ∆ ˜ W = (1 -ε ) -2 = SNR ˜ I . A visual representation is provided in Fig. 2c. Analytically, this confirms the idea that the closer ε is to 1 , the larger the energy is transferred from the compressed input to the weight derivatives (for ε = 0 . 8 we have SNR ∆ ˜ W = 25 ).

Additionally, in our setup since we are only compressing the activations and given (2) and (3), the introduced error only transfers to the weight derivative for each layer. The activation derivatives that are propagated from one layer to another are exact computations as they do not involve the

1

51

101

128

K

1

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

1

=24

1

st

dimension

1

201

401

576

K

2

0

.

4

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

2

=32

2

nd

dimension

activations. This means that the error introduced when compressing the activations at each layer does not accumulate through the network.

4 Experiments

In this section, we describe the experiments conducted to support the claims presented in Sec. 3. First, we introduce the setups used for our experiments (Sec. 4.1); then, we analyze the energy distribution in the different dimensions of HOSVD, providing an overview of the typical values of K (Sec. 4.2); finally, we test our algorithm in different setups, state-of-the-art architectures and datasets to evaluate the tradeoff between accuracy and memory footprint (Sec. 4.3). Experiments were performed using a NVIDIA RTX 3090Ti and the source code uses PyTorch 1.13.1.

4.1 Experimental setup

To validate our approach, we perform a variety of computer vision experiments split across two tasks, classification and segmentation.

Classification. We explore two types of fine-tuning policies:

Full fine-tuning (referred to as 'setup A'): Following conventional fine-tuning trends, we load models pre-trained on ImageNet [21] and we fine-tune them on a variety of downstream datasets (CIFAR-10, CIFAR-100, CUB [46], Flowers [32] and Pets [54]).

Half fine-tuning (referred to as 'setup B'): Following Yang et al. approach, each classification dataset (ImageNet, CIFAR-10/100) is split into two non-i.i.d. partitions of equal size using the FedAvg [29] method. The partitions are then split as follows: 80% for training and 20% for validation. The first partition is used for pretraining, and the second partition is used for finetuning.

Semantic Segmentation. Similarly to the half fine-tuning method, we reproduce Yang et al. segmentation setup: we fine-tune models pretrained on Cityscapes [6] by MMSegmentation [5]. Here there is only one downstream dataset which is Pascal-VOC12 [3]. Experimental details for both tasks (hyper-parameters, policy, etc.) are provided in Appendix B.2.

Memory Logging. For HOSVD and SVD, instead of focusing on compressing the tensor based on rank, we control compression through the desired amount of retained information. As a consequence, we cannot explicitly control the memory usage of the principal components. In the results presented below, we will always include two columns displaying peak and average memory, along with their standard deviations.

4.2 Explained Variance Evolution

In this section, we conduct experiments to fine-tune the last four convolutional layers of MCUNet [24] following Setup A, and CIFAR-10 as the downstream dataset. Using HOSVD with ε set to 0 . 8 , Fig. 3 shows the explained variance retained across dimension j as a function of K j , where j = { 1 , 2 } corresponds to the two largest dimensions of the activation map. We define K ( ε = x ) j as the number of principal components required to retain at least a fraction x of the explained variance.

10

1

10

2

10

3

10

4

10

5

10

6

Peak Activation Memory (kB)

50

55

60

65

70

75

80

85

90

Top1 Validation Accuracy (%)

18.87

×

smaller

in memory

19.11% higher

in accuracy

HOSVD

ε

= 0

.

8

SVD

ε

= 0

.

8

Gradient Filter R7

Gradient Filter R4

Gradient Filter R2

Vanilla training

We observe that along the largest dimensions (batch size and number of channels), less than 20% of the components capture more than 80% of the explained variance, confirming the assumption proposed in Sec. 3.4. Additionally, the curves observed present a logarithmic behavior, hinting at the possibility of reaching high explained variance with little loss regarding the compression ratio. This is especially important as the SNR transmitted to the weight derivatives increases quadratically to the explained variance (Fig. 2c). The results for the other layers are presented in Appendix B.3.

Fig. 4 illustrates the results of performing HOSVD with different explained variance thresholds ε . The results are averaged over three different random seeds. For ε smaller than 0 . 8 , we observe that as it increases, we achieve sig-

0

.

70

0

.

74

0

.

78

0

.

82

0

.

86

0

.

90

0

.

94

0

.

98

ε

70

.

0

72

.

5

75

.

0

77

.

5

80

.

0

82

.

5

Top1 Validation Accuracy (%)

Top1 Validation Accuracy (%)

Peak mem (MB)

10

-

1

10

0

10

1

Peak mem (MB)

nificant gains in accuracy, along with impressive compression ratios. However, when ε exceeds 0 . 8 , the accuracy growth starts slowing down. Above the 0 . 9 threshold, the exponential growth of peak memory results in a worsened tradeoff between accuracy and compression ratio. Therefore, in subsequent experiments presented in this paper, we will use ε values of 0 . 8 and 0 . 9 .

4.3 Main Results

MCUNet on classification with setup A. In this experiment, we fine-tune on CIFAR-10 an MCUNet pre-trained on ImageNet, with the number of finetuned layers increasing from 1 to 42 (all layers). We compare vanilla training, and gradient filtering with patch sizes of 2 , 4 , and 7 , SVD, and HOSVD with an explained variance threshold of 0 . 8 . For each method, we compare the effect of fine-tuning different model depths. Fig. 5 presents the performance curves for our experiments, with the X-axis representing activation memory in kiloBytes (kB) on a logarithmic scale and the Y-axis representing the highest validation accuracy. Each marker indicates the number of convolutional layers finetuned. For example, on the yellow curve representing the HOSVD method, the marker closest to the Y-axis shows the result when finetuning the last convolutional layer, the next marker represents finetuning the last two convolutional layers, and so on. The most effective method is the one whose performance curve trends towards the upper-left corner of the plot.

We observe that as the number of finetuned layers increases, the gradient filtering accuracy increases up to a certain point, and then deteriorates, whereas the accuracy for SVD, HOSVD, and vanilla training keeps improving with additional layers, along a similar trend. Intuitively, gradient filtering might propagate errors through the layers during training, while our HOSVD method keeps the error introduced on each individual layer confined to that specific layer as demonstrated in Sec. 3.4. Moreover, we observe that for identical depths, SVD accuracies consistently exceed HOSVD ones. We hypothesize that this is due to HOSVD performing SVD across all modes of the tensor: it potentially loses information in all modes, whereas SVD only loses information in one mode.

Compared to SVD, HOSVD significantly reduces memory, given equivalent accuracy levels (up to 18 . 87 times). Similarly, for equivalent memory usage, HOSVD yields greatly improved accuracy compared to SVD (up to 19 . 11% ). When compared to methods such as gradient filtering and vanilla training, given an equivalent memory budget, HOSVD yields significantly higher accuracies. Notably, fine-tuning all layers with HOSVD requires much less memory than fine-tuning only the last layer with vanilla training, which is consistent with the theoretical compression ratio shown in Fig. 2a. Additional experiments with setup A can be found in Appendix B.5.

ImageNet Classification with setup B. Table 1 presents the classification performance and memory consumption for MobileNetV2 [37], ResNet18, and ResNet34 [15] models using various methods, including vanilla training, gradient filtering, HOSVD, and SVD on the ImageNet dataset. We observe in most cases that, for the same depth, SVD and HOSVD are competitive with the gradient filtering method in terms of performance while reaching much lower activation memory with HOSVD.

Segmentation. Table 2 reports the segmentation performance and memory consumption for a variety of architectures as presented in Yang et al. . In general, we observe that increasing the level of explained variance from 0 . 8 to 0 . 9 substantially increases the performance with little trade-off on retained activation memory. This further confirms that most of the explained variance is contained in the first few components, allowing for efficient generalization with high compression rates.

5 Conclusion

In this work, we have addressed one of the main obstacles to on-device training. Inspired by traditional low-rank optimization approaches, we propose to compress activation maps by tensor decomposition, using HOSVD as a supporting example (Sec. 3.2). We demonstrate that the compression error introduced is bounded and confined to each individual layer considered, validating our approach

through extensive experimentations. This work paves the way for a new line of research combining the accumulated knowledge on tensor decomposition strategies and the recent field of on-device learning.

References

Toru Baji. Evolution of the gpu device widely used in ai and massive parallel processing. In 2018 IEEE 2nd Electron devices technology and manufacturing conference (EDTM) , pages 7-9. IEEE, 2018.

Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems , 33:1128511297, 2020.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 , 2017.

MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE , 108(4):485532, 2020.

Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 8599-8603. IEEE, 2013.

Sebastian Eliassen and Raghavendra Selvan. Activation compression of graph neural networks using block-wise quantization with improved variance minimization. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7430-7434, 2024.

Shahaf E. Finder, Yair Zohav, Maor Ashkenazi, and Eran Treister. Wavelet feature maps compression for image-to-image cnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 20592-20606. Curran Associates, Inc., 2022.

Robert M. French. Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented? In Proceedings of the 6th International Conference on Neural Information Processing Systems . Morgan Kaufmann Publishers Inc., 1993.

Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7085-7095, 2019.

Marawan Gamal Abdel Hameed, Marzieh S Tahaei, Ali Mosleh, and Vahid Partovi Nia. Convolutional neural network compression through generalized kronecker product decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 771-779, 2022.

Tyler L Hayes and Christopher Kanan. Online continual learning for embedded devices. arXiv preprint arXiv:2203.10681 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345 , 2022.

Ozlem Durmaz Incel and Sevda Ozge Bursa. On-device deep learning for mobile and wearable sensing applications: A review. IEEE Sensors Journal , 2023.

Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 , 2015.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114, 2017.

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review , 51(3):455-500, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5533-5543. PMLR, 13-18 Jul 2020.

Young D Kwon, Rui Li, Stylianos I Venieris, Jagmohan Chauhan, Nicholas D Lane, and Cecilia Mascolo. Tinytrain: Deep neural network training at the extreme edge. arXiv preprint arXiv:2307.09988 , 2023.

Ji Lin, Wei-Ming Chen, Yujun Lin, john cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 11711-11722. Curran Associates, Inc., 2020.

Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems , 35, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Ivan Markovsky. Structured low-rank approximation and its applications. Automatica , 44(4):891909, 2008.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(7):3523-3542, 2021.

Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE access , 7:19143-19165, 2019.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , 2008.

German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Netw. , 113, 2019.

Danilo Pietro Pau and Fabrizio Maria Aymone. Suitability of forward-forward and pepita learning to mlcommons-tiny benchmarks. In 2023 IEEE International Conference on Omnilayer Intelligent Systems (COINS) , 2023.

Aël Quélennec, Enzo Tartaglione, Pavlo Mozharovskyi, and Van-Tam Nguyen. Towards ondevice learning on the edge: Ways to select neurons to update under a budget constraint. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 685-694, 2024.

Berkman Sahiner, Weijie Chen, Ravi K Samala, and Nicholas Petrick. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology , 96(1150):20220878, 2023.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Gabriele Spadaro, Riccardo Renzulli, Andrea Bragagnolo, Jhony H Giraldo, Attilio Fiandrotti, Marco Grangetto, and Enzo Tartaglione. Shannon strikes again! entropy-based pruning in deep neural networks for transfer learning under extreme memory and computation budgets. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243 , 2019.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950 , 2019.

Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika , 31(3):279-311, 1966.

MAlex O Vasilescu and Demetri Terzopoulos. Multilinear subspace analysis of image ensembles. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. , volume 2, pages II-93. IEEE, 2003.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset, 2011.

Fei-Yue Wang, Jun Jason Zhang, Xinhu Zheng, Xiao Wang, Yong Yuan, Xiaoxiao Dai, Jie Zhang, and Liuqing Yang. Where does alphago go: From church-turing thesis to alphago thesis and beyond. IEEE/CAA Journal of Automatica Sinica , 3(2):113-120, 2016.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV) , pages 418-434, 2018.

Ou Xinwei, Chen Zhangxin, Zhu Ce, and Liu Yipeng. Low rank optimization for efficient deep learning: Making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics , 2023.

Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech , pages 2365-2369, 2013.

Junhuan Yang, Yi Sheng, Yuzhou Zhang, Weiwen Jiang, and Lei Yang. On-device unsupervised image segmentation. arXiv preprint arXiv:2303.12753 , 2023.

Yuedong Yang, Hung-Yueh Chiang, Guihong Li, Diana Marculescu, and Radu Marculescu. Efficient low-rank backpropagation for vision transformer adaptation. In Proceedings of the 37th International Conference on Neural Information Processing Systems , pages 14725-14736, 2023.

Yuedong Yang, Guihong Li, and Radu Marculescu. Efficient on-device training via gradient filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3811-3820, 2023.

Hui Zhang, Shenglong Zhou, Geoffrey Ye Li, and Naihua Xiu. 0/1 deep neural networks via block coordinate descent. arXiv preprint arXiv:2206.09379 , 2022.

Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence , 38(10):1943-1955, 2015.

Hengling Zhao, Yipeng Liu, Xiaolin Huang, and Ce Zhu. Semi-tensor product-based tensordecomposition for neural network compression. arXiv preprint arXiv:2109.15200 , 2021.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

A Additional Theoretical Details

A.1 Forward pass Vs. Backward pass

We consider the same notations introduced in Sec. 3.1. Regarding the training of a convolutional layer:

There is one convolution operation in each forward pass with a computational complexity of Θ FLOPS ( D 2 CC ′ BH ′ W ′ )

In each backward pass, there are two convolution operations, corresponding to formulas (2) and (3), and one weight update operation, with computational complexities of Θ FLOPS ( D 2 CC ′ BH ′ W ′ ) , Θ FLOPS ( D 2 CC ′ BHW ) and Θ FLOPS ( D 2 CC ′ ) , respectively.

From this, we can calculate the ratio of computational complexity between forward and backward pass R FLOPs as follows:

It is evident that R FLOPs is always smaller than one, indicating that the backward pass is always more computationally intensive than the forward pass.

A.2 Backpropagation Derivatives in Linear Layers

With the same notations introduced in Sec. 3.1 but in the case of a linear neural network represented as a sequence of n linear layers:

where W i ∈ R O × I corresponds to parameters matrix of the i th layer. This layer receives an I -feature input A i ∈ R B × I with minibatch size B to produce an O -feature output A i +1 ∈ R B × O . The derivatives in that setup become:

The same conclusions as for convolutional layers can naturally be derived from these equations for linear layers, namely that activations and weights occupy most of the space in memory when performing backpropagation.

A.3 Details of Backpropagation with Decomposed Activation Tensors

Since the convolution operation involves other variables such as stride, dilation, and groups, we provide a more precise description by rewriting in (2), ∂ L ∂ W i as ∆ W ∈ R N out × C × D × D , A i as I ∈ R B × C × H × W and ∂ L ∂ A i +1 as ∆ Y ∈ R B × C ′ × H ′ × W ′ which gives us:

In our case, before being saved in memory, I is decomposed and compressed through truncated HOSVD as presented in (8):

Therefore, in the backward pass, its padded version can be restored using the following formula:

where, U (3) ( K 3 ) and U (4) ( K 4 ) are U (3) ( K 3 ) and U (4) ( K 4 ) with vertical padding (top and bottom edges only), respectively. Then, we substitute (19) into (17) which gives us, after reordering and grouping:

Computing (20), (21), (22) and (24) corresponds to performing 1 × 1 convolutions while (23) is a H ′ × W ′ convolution, resulting in (11).

A.4 Details of Overhead, Computational Speedup and Space Complexity

Following the notations introduced in the main paper, we propose in this section to analitically study the overhead introduced in the forward pass when performing HOSVD compression, the speedup of efficient weight derivative computation and the required space to store the compressed activations.

Overhead. During the forward pass, the overhead of activation compression is the computational complexity of performing the tensor decomposition, which can be calculated as follows:

The computational complexity of SVD for a matrix of size m × n with m ≥ n is Θ FLOPS ( m 2 n ) . HOSVD essentially performs SVD on each mode of the tensor. During the forward pass, for each convolution layer, given an activation map of size B × C × H × W , HOSVD involves performing SVD on four matrices of sizes B × CHW , C × BHW , H × BCW , and W × BCH . Therefore, the computational complexity for decomposition in the forward pass is:

1

16

32

64

128

B

=

C

=

H

=

W

=

C

′

10

3

10

6

10

9

10

12

10

15

Forward pass FLOPs

HOSVD

Vanilla training

Additionally, the amount of FLOPs that vanilla training requires to perform the forward pass is:

From this, the total FLOPs that HOSVD requires is:

Based on these calculations, we represent in Fig. 6 the evolution of FLOPs in the forward pass for both vanilla training and HOSVD, showing that our method results in an increased latency in the forward pass.

Speedup. The key difference between HOSVD and Vanilla BP lies in computing ∆ W . For Vanilla BP, the cost of computing ∆ W is Θ FLOPS ( D 2 CC ′ BH ′ W ′ ) . Whereas our method involves:

Computing Z (1) : Θ FLOPS ( K 1 C ′ H ′ W ′ B ) , Computing Z (2) : Θ FLOPS ( K 1 K 2 HK 4 K 3 ) , Computing Z (3) : Θ FLOPS ( K 1 K 2 HK 4 K 3 + K 1 K 2 HWK 4 ) , Computing Z (4) : Θ FLOPS ( K 1 C ′ H ′ W ′ B + K 1 K 2 HK 4 K 3 + K 1 K 2 HWK 4 + C ′ K 2 D 2 H ′ W ′ K 1 ) .

Therefore, the complexity for computing ∆ W with our efficient reconstruction is:

We thus deduce the speedup ratio R S for the backward pass, between vanilla training and HOSVD (results represented in Fig. 2b):

Space Complexity. For vanilla training, storing A i correspond to the storage of Θ space ( BCHW ) elements.

For HOSVD, instead of storing the entire tensor, we store its truncated principal components: ˜ S , U (1) ( K 1 ) , U (2) ( K 2 ) , U (3) ( K 3 ) , and U (4) ( K 4 ) , which corresponds to a total of Θ space ( K 1 K 2 K 3 K 4 + BK 1 + CK 2 + HK 3 + WK 4 ) elements.

Thus, we obtain the storage complexity ratio R C as follows (results represented in Fig. 2a):

B Additional Experimental Details

B.1 Variance of Different Runs

We conducted classification experiments using MCUNet with setup A, fine-tuning on CIFAR-10, and segmentation experiments using the DeepLabV3 checkpoint provided by [53], fine-tuning on the augmented Pascal-VOC12 dataset. Both experiments employed three different random seeds ( 233 , 234 , and 235 ). The results, presented in Table 3 and Table 4, indicate no significant variation in performance across the different random seeds. Therefore, for subsequent experiments, we used a single random seed, 233 .

±

±

±

±

±

±

B.2 Detailed Experimental Setup

ImageNet Classification. In this setup, we use a similar finetuning policy to [53]. Specifically, we finetune the checkpoints for 90 epochs with L2 gradient clipping with a threshold of 2 . 0 . We use SGD with a weight decay of 1 × 10 -4 and a momentum of 0 . 9 . The data is randomly resized, randomly flipped, normalized, and divided into batches of 64 elements. We use cross-entropy as the loss function. The difference from their setup lies in the initial learning rate value after the warm-up epochs: in our case, the learning rate increases linearly over 4 warm-up epochs up to 0 . 005 (while in [53] it is 0 . 05 ). After this, similarly to their approach, the learning rate decays according to the cosine annealing method in the following epochs.

Other dataset finetuning. We use the same set of hyperparameters as described in [53] for both setup A and setup B. For training, we use cross-entropy loss with the SGD optimizer. The learning rate starts at 0 . 05 and decays according to the cosine annealing method. Momentum is set to 0 , and weight decay is set to 1 × 10 -4 . We apply L2 gradient clipping with a threshold of 2 . 0 .

For setup B, batch normalization layers are fused with convolutional layers before training, a common technique for accelerating inference. We set the batch size to 128 and normalized the data before feeding it to the model.

Semantic Segmentation. Following the policy outlined in [53], we utilized the pretrained and calibrated checkpoints provided by the authors for our experiments. The models were pretrained on Cityscapes using MMSegmentation, then we finetuned them on the augmented Pascal-VOC12 dataset. The optimizer's learning rate starts at 0 . 01 and decays according to the cosine annealing schedule during training. Additionally, we set the weight decay to 5 × 10 -4 and momentum to 0 . 9 . The batch size is set to 8 . For data augmentation, we randomly crop, flip, apply photometric distortions, and normalize the images. Cross-entropy is used as the loss function. Models are finetuned for 20 , 000 steps.

±

±

B.3 Additional Explained Variance Evolution Results

1

51

101

128

K

1

0

.

94

0

.

96

0

.

98

1

.

00

Explained Varianced

K

(

ε

=0

.

8)

1

=1

K

(

ε

=0

.

9)

1

=1

1

st

dimension

1

201

384

K

2

0

.

90

0

.

95

1

.

00

Explained Varianced

K

(

ε

=0

.

8)

2

=1

K

(

ε

=0

.

9)

2

=2

2

nd

dimension

1

3

5

7

K

3

0

.

96

0

.

98

1

.

00

Explained Varianced

K

(

ε

=0

.

8)

3

=1

K

(

ε

=0

.

9)

3

=1

3

rd

dimension

1

3

5

7

K

4

0

.

98

1

.

00

Explained Varianced

K

(

ε

=0

.

8)

4

=1

K

(

ε

=0

.

9)

4

=1

4

th

dimension

(a) Last Layer.

1

51

101

128

K

1

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

1

=36

K

(

ε

=0

.

9)

1

=67

1

st

dimension

1

51

96

K

2

0

.

4

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

2

=21

K

(

ε

=0

.

9)

2

=39

2

nd

dimension

1

3

5

7

K

3

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

3

=3

K

(

ε

=0

.

9)

3

=5

3

rd

dimension

1

3

5

7

K

4

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

4

=3

K

(

ε

=0

.

9)

4

=4

4

th

dimension

(b)

2

nd

1

51

101

128

K

1

0

.

7

0

.

8

0

.

9

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

1

=22

K

(

ε

=0

.

9)

1

=57

1

st

dimension

1

201

401

576

K

2

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

2

=17

K

(

ε

=0

.

9)

2

=50

2

nd

dimension

1

3

5

7

K

3

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

3

=3

K

(

ε

=0

.

9)

3

=4

3

rd

dimension

1

3

5

7

K

4

0

.

7

0

.

8

0

.

9

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

4

=2

K

(

ε

=0

.

9)

4

=4

4

th

dimension

(c)

3

rd

1

51

101

128

K

1

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

1

=24

K

(

ε

=0

.

9)

1

=58

1

st

dimension

1

201

401

576

K

2

0

.

4

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

2

=32

K

(

ε

=0

.

9)

2

=83

2

nd

dimension

1

3

5

7

K

3

0

.

6

0

.

8

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

3

=3

K

(

ε

=0

.

9)

3

=4

3

rd

dimension

1

3

5

7

K

4

0

.

8

0

.

9

1

.

0

Explained Varianced

K

(

ε

=0

.

8)

4

=2

K

(

ε

=0

.

9)

4

=3

4

th

dimension

(d)

4

th

As anticipated in Sec. 4.2, Fig. 7 presents the explained variance as a function of K j at the j th dimension of the four fine-tuned layers of MCUNet. We observe that in the first two dimensions, which are also the largest, more than 80% of the tensor's energy is concentrated within the first 30% of the principal components.

With the same experiment, Fig. 8 shows that K j tends to gradually increase after each training epoch, then peaks and begins to decline. Intuitively, at the beginning, the activation map energy is mainly concentrated in the few first components. As training progresses, this energy starts to spread out to other components, leading to an increase in the value of K j . Over time, the model learns how to efficiently condense the energy, leading to a gradual decrease in K j . This instability in K j throughout the training process is why we had to log memory as described in Section 4.1.

1

21

41

50

Epoch

0

.

95

1

.

00

1

.

05

K

1

1

st

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

1

.

0

1

.

5

2

.

0

K

2

2

nd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

0

.

95

1

.

00

1

.

05

K

3

3

rd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

0

.

95

1

.

00

1

.

05

K

4

4

th

dimension

ε

= 0

.

8

ε

= 0

.

9

(a) Last Layer.

1

21

41

50

Epoch

0

20

40

60

K

1

1

st

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

20

40

K

2

2

nd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

2

4

K

3

3

rd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

1

2

3

4

K

4

4

th

dimension

ε

= 0

.

8

ε

= 0

.

9

(b)

2

nd

1

21

41

50

Epoch

20

40

60

K

1

1

st

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

20

40

60

K

2

2

nd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

2

3

4

K

3

3

rd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

2

3

4

K

4

4

th

dimension

ε

= 0

.

8

ε

= 0

.

9

(c)

3

rd

1

21

41

50

Epoch

0

20

40

60

K

1

1

st

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

0

50

100

K

2

2

nd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

2

4

K

3

3

rd

dimension

ε

= 0

.

8

ε

= 0

.

9

1

21

41

50

Epoch

1

2

3

K

4

4

th

dimension

ε

= 0

.

8

ε

= 0

.

9

(d)

4

th

B.4 Processing Time Results

This section presents latency comparisons between vanilla training and HOSVD. We conducted the experiments using MCUNet on a single batch of CIFAR-10 dataset with setup A and for one epoch only.

Fig. 9 compares the execution time of the forward, backward, and total training processes between vanilla training and HOSVD. It can be observed that the backward processing time of HOSVD is tens of times lower than vanilla training, but the forward time is up to a thousand times higher. As a result, when considering the total training time (the sum of forward and backward), HOSVD is on average 4 . 29 times slower than vanilla training. This outcome is entirely expected, as the decomposition process introduces an overhead, as predicted and described in Fig. 6 and Appendix A.4.

1

2

3

4

5

6

7

8

9

10

#Layers

10

-

3

10

-

2

10

-

1

10

0

Forward Time (s)

Vanilla training

HOSVD

(a)

1

2

3

4

5

6

7

8

9

10

#Layers

10

-

2

10

-

1

10

0

Vanilla training

HOSVD

(b)

1

2

3

4

5

6

7

8

9

10

#Layers

10

-

1

10

0

Total Time (s)

×

13.33

×

2.48

×

4.29 in

average

Vanilla training

HOSVD

(c)

B.5 Additional Classification Results

In addition to the main experiments, we also performed many classification experiments with setup A on various datasets, the results are shown in Table 5. Especially, when fine-tuning SwinT on CUB200, Pets, and CIFAR-100, applying HOSVD to the last four layers (maintaining the rest of the architecture frozen) consumes less memory than fully fine-tuning the last two layers. This happens because the variance of the activation maps is concentrated in few components: for the same ε , K will be smaller.

C Limitation

While HOSVD is effective in compressing the forward signal in DNNs for gradient estimation during backpropagation, it is just one among possible choices when dealing with tensor decomposition. Indeed, more variants of tensor decomposition methods such as GKPD or STP could have been used and evaluated and could have led to larger memory and computational gains. Our work opens the door to further explorations in such direction.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We develop the theoretical aspects of tensor decomposition applied to activations in Sec. 3 (construction, practical implementation, complexity, and error bounding). In Sec. 4 we provide experimental results on a wide variety of popular tasks, architectures, and datasets, while also exploring the combined effect of various hyper-parameters.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of the work are discussed in the dedicated section of the paper(Sec. C).

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Sec. 3.1 we develop the formulas necessary to perform backpropagation in a convolutional network using the chain rule. In Sec. 3.2, we provide explicit definition of the tools used in this papers. The complete proof for Sec. 3.3 is provided in the supplemental materials(Sec. A.3). In Sec. 3.4 we evaluate the resulting memory required, the computational complexity of our method, and the error introduced in the gradient computation.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Algorithmic details are developped throughout 3. Code to reproduce results is included in the supplemental materials, alongside instructions on how to use it. The experimental setup is described briefly in Sec. 4 and more exhaustive explicit details are provided in the supplemental materials B.2.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In footnote 1 on the first page of the paper, the exact code used to collect the experimental results is provided in an open GitHub repository, along with instructions on how to run the code.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Such details are discussed in Sec. 4 and developed further in the corresponding section of the supplementary material (Sec. B.2).

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Given the computational cost of the experiments, we have performed all our experiments (even reproducing those from the literature) under the same conditions, but we did not provide confidence intervals for all of them. In the supplementary material we show some results providing confidence intervals, and observing a very small variance between the results obtained from different seeds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We detail the material used in Sec. 4 and the time of execution in the supplementary materials.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Research conducted in this paper is conform with NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our method is meant to enable on-device learning and we discuss in the introduction about possible impacts of such a solution in our lives (increased privacy and security, reduced energy consumption and network overload, ...) (Sec. 1). Nevertheless, a specific section is not devoted given that this work is very general and not specific to some application.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The proposed approach is a generalist compression algorithm tailored for on-device learning, thus, it does not fall in that category.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the existing datasets, models, and code used are presented in Sec. 4, and credit is given to the respective creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All the details about the new assets introduced are provided in the paper. At submission time the source code is not provided, but we commit to release it (alongside a detailed documentation) upon acceptance of the paper.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of on-device training for deep learning models, particularly focusing on the memory bottleneck caused by the storage of activation maps during backpropagation. The authors propose a method to compress activation maps using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and its tensor variant, High-Order Singular Value Decomposition (HOSVD).\n\nSTRENGTHS:\n1. Using tensor decomposition for activation map compression to address a bottleneck in on-device training.\n2. The paper provides a solid theoretical foundation, including error analysis and guarantees for convergence.\n\nWEAKNESSES:\n1. Using SVD or tensor decomposition to compress activation maps incurs significant computational overhead. This approach of trading computational complexity for space complexity is questionable, especially considering that the computational capacity of embedded devices is usually also limited. Moreover, this compression may introduce additional information loss, potentially affecting the model's performance.\n2. For SGD, recalculating activations instead of storing them is a viable method to save memory. Given that both methods trade computation time for storage space, why use low-rank compression which may potentially affect model performance?\n3. With limited computational resources, why not search for a more appropriate model architecture using NAS?\n\nQUESTIONS:\n1. LOMO[1] fused gradient computation and parameter update in one step to minimize the size of gradient tensors. Compared to this approach, what are the advantages of your method?\n2. Would using quantization methods instead of compression methods produce better results?\n\n\n[1] Lv, Kai, et al. \"Full parameter fine-tuning for large language models with limited resources.\" arXiv preprint arXiv:2306.09782 (2023).\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to compress activation maps in deep neural networks using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and Higher Order SVD (HOSVD). The goal is to reduce memory requirements during backpropagation, enabling on-device learning for resource-constrained environments. The authors provide theoretical analysis of their method's impact on memory usage, computational complexity, and error bounds. They demonstrate the effectiveness of their approach through experiments on various tasks, architectures, and datasets.\n\nSTRENGTHS:\n- Novelty: The use of tensor decomposition for compressing activation maps is a novel approach that addresses a significant bottleneck in neural network training.\n- Efficiency: The method significantly reduces memory usage during backpropagation, which is crucial for deploying deep learning models on resource-constrained edge devices.\n- Theoretical Support: The paper provides  theoretical background for the proposed method, including error analysis and guarantees of minimal information loss.\n\nWEAKNESSES:\n- Severe performance degradation relative to memory reduction\n- Complexity of Implementation: Implementing tensor decomposition techniques like HOSVD can be complex and may require specialized knowledge, potentially limiting its adoption.\n- Lack of Detailed Rank Selection: The paper does not provide sufficient explanation on how the appropriate rank for decomposition is selected, which is crucial for balancing memory reduction and accuracy.\n- Inconsistent and contradictory results: SVD frequently outperforms HOSVD, contradicting the authors' hypothesis and lacking adequate explanation.\n- Limited comparison with other activation map compression techniques.\n\nQUESTIONS:\n- How does the proposed method perform on more diverse and complex neural network architectures, such as transformers?\n- What are the practical challenges and considerations when deploying this method on real-world edge devices with strict latency and power constraints?\n- How is the appropriate rank for decomposition selected, and what guidelines can be provided for tuning this parameter effectively?\n\nLIMITATIONS:\n- Specific to Certain Architectures: The experiments focus mainly on specific neural network architectures and tasks. It is unclear how well the method generalizes to other types of models and applications.\n- Dependence on Hyperparameters: The effectiveness of the compression is highly dependent on the chosen hyperparameters for the decomposition, which may require extensive tuning.\n- Resource Requirements for Decomposition: While the method reduces memory usage during training, the initial decomposition itself can be computationally intensive and may require powerful hardware.\n- Practical Usability Concerns: Despite significant memory reduction, the method results in considerable accuracy drops, raising concerns about its practical usability. Vanilla backpropagation, while using more memory, generally maintains higher performance and might be a more realistic approach for actual deployment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Editing to reflect my score increase from 6 to 7 after the author discussion phase.]\n\nThe authors tackle the problem of memory consumption due to needing to keep realized activation tensors available between their use in the forwards pass and backwards propagation in training neural networks.  They propose the use of tensor decomposition, specifically SVD and higher-order SVD, to compress activation maps and show that the compressed activations can be used in backpropagation without first decompressing.  An error analysis is performed, and experimental results confirm the hypothesis that significant compression can be applied while maintaining significant explained variance.  To validate the method's success on real workloads, the authors apply it to image classification and semantic segmentation tasks, fine-tuning pre-trained models for both tasks, as well as training from scratch for image classification.\n\nSTRENGTHS:\n**Originality**\nI believe this is a novel application of tensor decomposition, a well-known technique previously applied to weight compression.  Adaptively sizing the decomposed tensor to maximize retained information in the activation maps is an advancement over past work.\n\n**Quality**\nThe authors have performed a reasonable set of experiments to show the technique works as intended: model quality when activations are subject to tensor decomposition for backpropagation remain competitive for image classification tasks and semantic segmentation tasks.  Further, the authors have reported the memory consumption, proving that it is reduced significantly enough to satisfy constraints of edge devices in many cases.  I greatly appreciate the time spent in the background, as well as the seemingly offhand note that any error induced by these decompositions will be limited to that layer's weight gradients and does not accumulate deeper into the network.\n\n**Clarity**\nI generally found the manuscript easy to follow: the overall organization was great, and the method itself was explained adequately.\n\n**Significance**\nThe intended use case aside, I think even datacenter users may be interested in this technique, as many of today's latest networks are simply huge.  Reducing the storage costs for activations between forward and backwards passes could reduce the parallelism needed, and in turn the overhead of communications in distributed training.  I'd encourage the authors to consider this angle and apply the technique to, for example, large language models.\n\nWEAKNESSES:\n**Originality**\nOnly one piece of past work comes to mind that was omitted: Rhu et al., \"Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks\" (HPCA 2018) compresses activation maps, using their existing sparsity, to reduce the cost of offloading in order to accelerate training.  While novel hardware was designed to keep up with compute throughput, if compression is the only goal, then this could simply be performed in software (like the submission's tensor decomposition).\n\n**Quality**\nA missing piece is the cost of compression: what impact does this have on the training speed?  This is related to the claim on line 59 that \"gradient computation and parameter updating are considerably more expensive than the forward pass,\" which isn't obvious to me.  At a high level, each of the three calculations (forward propagation, gradient calculation, weight update calculation) are roughly equivalent in cost, and applying the weight update is very simple.  Increasing the cost of any one of the phases would seem to be as painful as any of the others.  Another claim that stood out, given the stated goal of \"reducing the memory required for backpropagation\" (line 72) is that the \"main challenge limiting the feasibility of on-device learning lies in the computational cost of backpropagation\" (line 58).  I'd also question the claim that \"we can assume that the networks considered are already [weight] compressed\" (line 159) - while it's extensively studied, it's not a solved problem.  It's fine to not tackle this particular problem, but stating that it's assumed to not be an issue is misleading.\n\nSome of the results deserve some extra discussion.  Do the authors have a hypothesis about why SVD is superior to HOSVD (line 288)?  Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?\n\nI have not seen CIFAR treated as a \"downstream task\" of ImageNet-1k before; it's typically just a simpler data set used, recently, as a proving ground before testing on larger data sets.  Is it common to fine-tune a network trained on ImageNet-1k on CIFAR10/100?\n\nGiven the significant memory savings of the technique, I'd have liked to see how it performs when fine-tuning all layers, not just a subset.  Is there a reason this result wasn't gathered?\n\n**Clarity**\nI've had to make an assumption about the behavior of the method: during forward propagation, I assume the *uncompressed* activations are used, and only compressed when writing to memory to be used later in the backwards pass.  (If this is not the case, then errors *will* compound.)\n\nFigure 2 shows speedups and space gains, but without indicating what, exactly, is speeding up or seeing a reduction in memory space.  I assume, but am not sure, that it is for a single weight gradient calculation (of a size noted in the figure's legend and axes).\n\nThe wording of Figure 3's caption is confusing: I do not understand the meaning behind \"Mbv2 fine-tuned on Cifar10 at initialization.\"  \n\nNone of the networks used in the experimental results have references attached.  For example, MCUNet is prominently used in Figure 4, but I couldn't find either a description of this network or a reference to it in the text.  It looks likely that it is in reference [24], but this only has a passing citation in the related work section.\n\nThere are some typos:\n\n- Line 223: \"derives\" -> \"derivatives\"\n\n- Line 234: the range of explained variance uses two open brackets.\n\n- Reference [48] is listed with the incorrect year: 2024 should be 2023.\n\n\n**Significance**\nThere are several ways that the results would be more compelling:\n\n- Evaluation on more (difficult) tasks, like language modeling or object detection.\n\n- Application to more recent or larger networks.\n\n- More closely matching the densely trained or fine-tuned baselines.  As this is initial work in this area, the results shouldn't be dismissed, but there's still a non-trivial gap between the tensor-decomposed results and what the networks are capable of learning.\n\nQUESTIONS:\n1. Is there a reason that results for fine-tuning all layers were omitted?\n2. Why is SVD superior to HOSVD in some results (line 288)?\n3. Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?\n4. What impact does the compression process have on the training speed?\n\nLIMITATIONS:\nThe authors have addressed the limitations (in the appendix).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTensor low-rank decomposition is used to compress the backpropagation process of full connection and convolution in neural networks. The main process is to decompose input activation tensors into Tucker structures using HOSVD algorithm and truncate subtensors at each mode. Experimental result find an efficient trade-off between the desired explained variance and compression rates.\n\nSTRENGTHS:\n(1) Compute the approximated weight derivatives without reconstructing the activations, through the successive computation of simpler convolutions.\n\n(2) The experiments on trade-off between the desired explained variance and compression rates are sufficient.\n\n(3) Experimental results obtained on mainstream architectures and tasks show that it has a Pareto advantage over other comparison schemes in terms of the trade-off between accuracy and memory footprint.\n\nWEAKNESSES:\n(1) There are a lot of confusing expressions on the formula.\n\n(2) In formula 8, How to distinguish $U^{(k_j)}$s when the size retained after truncation on two modes is the same?\n\n(3) The symbol $]$ in formula 11 is in the wrong position, and $\\frac{\\partial A_{i+1}}{\\partial W_i}$ should be $\\frac{\\partial L}{\\partial A_{i+1}}$?\n\n(4) In Sec3.4 $\\frac{\\partial L}{\\partial A_{i+1}}=\\Delta Y$, but in appendix A.2 it expressed as \\Delta O.\n\nQUESTIONS:\n(1) In formula 12, how to use the discrete Fourier transform to get $\\tilde{I} =\\varepsilon I$? What does $uI$in the denominator mean?\n\n(2) In formula 16, please give the definitions of $d, h’,w’$. \n\n(3) The methods mentioned in the related work are not compared with the proposed methods.\n\nLIMITATIONS:\nIf the contraction between matrix and tensor is replaced by 1*1 convolution, then the proposed method should also be compared with CP decomposition, tensor train decomposition and other related methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-23"}