Activation Map Compression through Tensor Decomposition for Deep Learning

Authors: le-trung  nguyen
aël  quélennec
enzo  tartaglione
samuel  tardieu
van-tam  nguyen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:37:28

Abstract: Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.

Main paper: 
 introduction: 
 Recent advances in Deep Learning have enabled Deep Neural networks to be used as an efficient solution for a wide variety of use cases, including computer vision [21, 39, 30] , speech recognition [8, 31] and natural language processing [45, 42] . Much of this performance improvement is linked to the exponential increase in the number of parameters in the neural architectures. According to Sevilla et al., the release of AlphaGo [47] in late 2015 marks the advent of a new era, which they call the "Large Scale Era", in reference to the computational cost of training doubling every 8 to 17 months [38] . At the root of exponential growth in neural network size is the improvement in hardware capabilities, particularly those designed for large-scale parallel computing, such as GPUs and TPUs [1] . While this trend demonstrates the strength of neural networks as a powerful generalization tool in many fields, it goes in the opposite direction when it comes to environmental concerns [41] , making the deployment of newer architectures increasingly difficult. This is particularly true for edge devices such as mobile phones and embedded sys-tems, which cannot afford the high computing or memory costs. To address these challenges, three interconnected factors must be taken into account: power consumption, memory usage and latency.When considering larger neural networks with more layers and nodes, reducing their storage and computational cost becomes essential, especially for certain real-time applications such as edge computing. In addition, recent years have seen significant advances in the fields of intelligent edge and embedded intelligence, creating unprecedented opportunities for researchers to address the fundamental challenges of deploying deep learning systems on edge devices with limited resources (e.g. memory, CPU, power, bandwidth). Efficient deep learning methods can have a significant impact on distributed systems and embedded devices for artificial intelligence. Since training is supposed to take place in the cloud, most research on model compression and acceleration is specifically focused on inference [7] . There is, however, an emerging area of research concerning on-device training, which represents a decisive advance in the field of artificial intelligence, with considerable implications for a variety of practical situations [17] . Models trained offline on a dataset built at one point in time tend to fall victim to data drift when deployed "in the wild" [36] . Its combination with online learning strategies has the potential to enable continuous model improvement after deployment [14] , thus adapting the model predictions to observed evolutions in the data distribution. We can illustrate this with the example of sensors in autonomous vehicles, where deep learning models must correctly classify vehicles with new designs never seen in the training set. Other advantages of on-device learning include security and privacy. By processing data locally, sensitive information remains more secure and less susceptible to data breach, a major concern in applications such as healthcare.The main challenge limiting the feasibility of on-device learning lies in the computational demands of the backward pass, as gradient computation and parameter updates are significantly more resourceintensive than the forward pass (Appendix A.1). On embedded devices, memory and computation limitations act as strict budgets that must not be exceeded. Some approaches address the memory constraints by exploring alternatives to traditional backpropagation, including unsupervised learning for image segmentation [51] , the Forward-Forward algorithm [16] , and PEPITA [34] . While promising, these methods typically fall short of backpropagation-based techniques in terms of performances. A pioneering effort by Lin et al., demonstrated that fine-tuning a deep neural network within a 256 kB of memory is feasible by selectively updating a sub-network, achieving a good results [25] . In a complementary approach, Yang et al. proposed reducing the number of unique elements in the gradient map through patch-based compression of the input and gradients of a given layer with respect to the output, thereby lowering memory costs and speeding up the learning process [53] .Inspired by tensor decomposition methods, we propose a method that compresses activation maps, reducing the memory demands for backpropagation while maintaining the deep neural network's generalization capability(Fig. 1 ). Our approach adaptively captures the majority of tensor variance, offering guaranteed accuracy in the gradient estimation. The key contributions of our work are as follows:• We propose to exploit powerful low-rank approximation algorithms to compress activation maps, enabling efficient on-device learning with controlled information loss (Sec. 3.2 and Sec. 3.3). • We provide theoretical foundation for our method, along with an error analysis demonstrating that high compression ratios are achievable with limited performance degradation (Sec. 3.4). • We extensively explore a diverse experimental landscape, demonstrating the generalization capacity of our proposed algorithm (Sec. 4). 
related works: 
 Tensor Decomposition. Model compression and acceleration is a well-developed and ongoing area of research in the deep learning community. It can be divided into two major areas, namely hardware and algorithm design. In the case of algorithmic compression, the five main components that stand out are efficient and compact model design, data quantization, network sparsification, knowledge distillation, and tensor decomposition [4, 7] . All these approaches aim to reduce the space occupied by network parameters. Also known as a low-rank approximation, tensor decomposition has emerged as a robust solution due to its combination of grounded theoretical findings and its practicality in terms of hardware implementation [49] . Originating in the field of systems theory and signal processing [28] , low-rank approximation has attracted growing interest in the deep learning community. Early examples were limited to the application of singular value decomposition (SVD) used to compress fully connected layers [50] . More recent work includes the application of generalized Kronecker product decomposition (GKPD) to both linear and convolutional layers [13] , the introduction of semi-tensor product (STP) to improve compression ratios with reduced loss [56] or even more recently the acceleration of inference and backpropagation in vision transformers [52] .Evolutions in this domain can be summed up as improvements in compression ratios, latency, and power consumption, with limited performance loss and diversification of architectures considered.Activation Map Compression. The term "activations" here refers to the outputs of each layer after the non-linearity has been applied, which are then fed to the next layer. While model compression generally aims at reducing the storage space occupied by parameters (which typically translates into network acceleration during inference), a key observation is that activations occupy much more space than parameters in memory during backward pass, as they are required to compute weight derivatives [2] . This state of facts motivates a new line of research, whose main objective is to compress activation maps using methods inspired by the literature on weight compression. For example, we mention the use of quantization [9] , sparsification [22] , a combination of both with the addition of entropy encoding [12] , or even the application of wavelet transform in combination with quantization [10] . With the exception of Eliassen et al.'s work which accelerates training runtime, most of these works focus on accelerating inference, in a similar way to traditional model compression.On-device Learning. Today's typical pipeline for on-device AI consists of designing resourceefficient deep neural networks [24] , training them offline on target data, compressing them, and deploying them for inference on the resource-constrained environment. Alongside the rapid commercial expansion of the IoT field and the ubiquitous presence of embedded devices in our daily lives, AI at the edge has naturally attracted a great deal of interest. However, recent research has shown that real-world data often deviated from training data, resulting in poor predictive performance [40] . In such a context, continual learning proved to be a strong candidate for ensuring ongoing adaptation after deployment. Similar to the human learning process, it provides models with the ability to adapt to new incoming tasks, ideally without falling into the trap of catastrophic forgetting (i.e., degrading performance on older tasks in favor of new ones) [11, 33, 19] . In this respect, the very low memory and computing resources of extreme edge devices are an obstacle to the high cost of backpropagation. Lin et al. have shown that this challenge can be overcome by selectively fine-tuning a sub-graph of pre-trained networks on a general dataset [25] . Their work started a line of research showing the possibility of improvement through careful selection of channels to be updated at training time [23, 35] . However, none of these works addresses the computational cost of training a neuron, which is related to both the cost of loading the necessary weights and activations into RAM.Gradient Filtering. In their work, Yang et al. demonstrate that it is possible to significantly reduce the memory and computational cost of full layer learning using a method called gradient filtering, which reduces the number of unique elements in the gradient map [53] . In such a frame and with the same specific objective, we propose to compress the activation maps by using tensor decomposition in order to minimize the memory cost. We perform this while providing a strong guarantee on the signal loss: unlike gradient filtering, we adaptively size the decomposed tensor to guarantee minimal information loss. In the next section, we will illustrate our proposed approach. 
method: 
 In this section, we motivate our compression proposal by exposing the major memory bottleneck of backpropagation (Sec. 3.1). We then present our contribution which is the compression of activation maps through two well-known low-rank decomposition strategies, namely Singular Values Decomposition (SVD) and its tensor counterpart Higher Order SVD (HOSVD) (Sec. 3.2). In Sec. 3.3 we explore the induced changes to backpropagation calculation and in Sec. 3.4 we study the resulting memory and computational complexity, alongside an evaluation of the error introduced. Our ultimate goal here is to reduce the memory footprint of backpropagation (BP).Following the formalism introduced in [2] , we consider a simple convolutional neural network (CNN) represented as a sequence of n convolutional layers (excluding the bias for simplicity):EQUATIONwhere W i ∈ R C ′ ×C×D×D represents the filter parameters of the i th layer, with a kernel size of D × D. This layer receives a C-channel input and produces an output with C ′ channels. For this layer, we denote the input and output activation tensors as A i ∈ R B×C×H×W and A i+1 ∈ R B×C ′ ×H ′ ×W ′ , respectively. Note that H and W denote the height and width of each element of the input, while H ′ and W ′ denote the height and width of each element of the output, with B as the minibatch size. The loss L is computed at the output of the network and backpropagated until the i th layer as ∂L ∂Ai+1 . At this stage, the filter parameters are updated thanks to the computation of ∂L ∂Wi and the loss is propagated to the previous layer as ∂L ∂Ai . The computation of these terms follows the chain rule:EQUATIONEQUATIONwhere conv(•) is the traditional convolution operation, convolving the kernel ∂L ∂Ai+1 with the input A i ; while conv full (•) is the convolutional operation that naturally maps the input ∂L ∂Ai+1 to an output with the same dimensions as A i by using the 180 • rotated kernel W i . From (2), it is clear that computing the weight derivatives requires to load input activation A i and (3) shows that the weights W i must be loaded into memory to calculate activation derivatives. We obtain the same conclusions for linear layers and provide the demonstration in Appendix A.2.To save memory, two possibilities naturally emerge: either compressing the weights or compressing the activation. Weight compression is an extensively explored matter for network acceleration and we do not intend to further this area of research in this work. This leaves us with activation compression, which is still a new domain of exploration, but shows great potential for prospectively enabling on-device backpropagation. In such a regard, thanks to its strong theoretical grounding, tensor decomposition stands as a promising approach.We will present here first the general Singular Value Decomposition (SVD) approach, which is then extended to a multidimensional variant (HOSVD), instrumental for our purposes.SVD. Given a matrix A ∈ R M ×N , applying SVD to A consists in a factorization of the form:EQUATIONwhere Σ is a rectangular diagonal matrix composed of r singular values s i∈ [1,r] with r the rank of A.From this, we can deduce the amount of overall variance σ 2 i explained by the i th pair of SVD vectors as σ 2 i = s 2 i / j s 2 j . Let us assume the singular values in Σ are ordered in descending order, s i ≥ s j , ∀i ≤ j. Given a desired threshold of cumulated explained variance ε ∈ [0, 1], it becomes easy to find the "truncation threshold" which is the minimal K ∈ [1, r] such that K i=1 σ 2 i ≥ ε. We can then approximate A by only selecting the K first columns of U and V and the K first singular values from Σ, according to:EQUATIONHistorically, SVD was the first example of tensor decomposition applied to neural network compression and acceleration [55] , but applied to the model's parameters [49] . We take the SVD decomposition to the activation maps as one possible baseline to compare with more complex lowrank compression solutions.In the case of convolutional layer i, since SVD is designed for matrix decomposition, we will simply reshape the activation A i as matrix A i of dimensions B × (CHW ). Given a desired level of explained variance ε, we obtain the decomposition described in (5) . We then store in memory the termsU (K) × Σ (K) and V T (K), meaning that instead of storing Θ space (BCHW ) unique elements in memory, we are only storing Θ space [K(B + CHW )] unique elements. Regarding linear layers, activations are M × N matrices, and applying SVD is much more straightforward, leading to the storage cost of Θ space [K(M + N )] instead of Θ space (M N ) elements. The larger the explained variance, the closer Ã will be to A, intuitively allowing for better estimation when performing backpropagation. However, this also means a larger K which results in a larger memory occupation. The goal is then to find an efficient trade-off between the desired explained variance and compression rates.HOSVD. By construction, SVD is designed for matrix decomposition. It was demonstrated that the reshaping operation on tensors introduced structure information distortion, leading to sub-optimal performance [49] . Other methods more suited for tensor decomposition such as Canonical-Polyadic (CP) decomposition [20] or Tucker decomposition [43] were naturally introduced to tackle this issue. Given a n th -order tensor T ∈ R M1×M2×•••×Mn , its Tucker decomposition corresponds to:EQUATIONwhere S ∈ R L1×L2×•••×Ln is the core tensor which can be viewed as a compressed version of T , U (j) ∈ R Mj ×Lj are the factor matrices and their columns correspond to the principal components over the j th mode. The i-mode product "× i " of a n th -order tensorG ∈ R P1×P2×•••×Pn and a matrix B ∈ R Q×Pi is a n th -order tensor R ∈ R P1×•••×Pi-1×Q×Pi+1×•••×Pn which can be expressed as:EQUATIONIn such a setup, Higher-Order SVD is a specific case of Tucker decomposition where the factor matrices U (j) are orthogonal [44] . As a follow-up to SVD, we propose to compress activation tensors through HOSVD, with the intuition that each dimension encodes a different variance, potentially providing enhanced compression rates for equivalent performance and viceversa. Similarly to traditional SVD, we can truncate S and each U (j) along each mode given a desired level of explained variance resulting in:EQUATIONEQUATIONwhere U(j) (Kj ) ∈ R Mj ×Kj corresponds to the K j first columns of U (j) and Ŝ ∈ R K1×•••×Kn is the truncated version of S. In the following section, we focus on HOSVD decomposition, more precisely on the alterations induced to the backpropagation graph, the resulting compression and speedup ratio, and the error bounding. An equivalent analysis can be made for SVD.Prior works [13, 18] already demonstrated the possibility to perform backpropagation operations in the decomposed space without relying on the recomposotion of compressed tensors. In their work, Kim et al. compress a 4 th -order kernel tensor W ∈ R C ′ ×C×D×D through Tucker decomposition limited to mode 1 and 2:W = W ′ × 1 U (1) × 2 U (2) , W ′ ∈ R L1×L2×D×D , U (1) ∈ R C ′ ×L1 , U (2) ∈ R C×L2 . (9)Then, it is shown that the output is calculated throughEQUATION)Our problem is similar although with some fundamental differences linked to the need to reconstruct gradients. A convolution operator is still required to compute the weight derivatives as introduced in (2): we apply a specific case of Tucker to one of the two components, namely the activation A i . Following the same reasoning, we derive that the computation of ∂L ∂Wi in (2) becomes:EQUATIONEQUATIONEQUATION, UEQUATIONwhere conv * is a 2D convolution with a specific kernel size as defined in (23) , and U (j) (Kj ) is the vertically padded version of U (j) (Kj ) as defined in (19) . Demonstration details are available in5 10 K 1 = K 2 = K 3 = K 4 10 0 10 1 10 2 10 3 10 4 10 5 R C B = C = C' = H = W = H' = W' = 16 B = C = C' = H = W = H' = W' = 32 B = C = C' = H = W = H' = W' = 64 B = C = C' = H = W = H' = W' = 128(a) 5 10 Appendix A.3. This way, we can compute the approximated weight derivative without reconstructing the activation, through the successive computation of simpler convolutions.EQUATIONFig. 1 illustrates our method. During training, the forward pass proceeds as usual, with one key modification to memory management. Instead of keeping complete activation maps in memory, we store only their principal components, which are derived from the decomposition process. During the backward pass, the principal components are retrieved from memory and used for calculations as described in (11) .Computational Speedup and Space Complexity. For simplicity we assume here that A i+1 and A i have the same shape and that the truncation thresholds across all modes are equal for HOSVD, i.e.,K 1 = K 2 = K 3 = K 4 .Considering different thresholding values, we can predict the relative improvement of HOSVD to vanilla training in both space complexity (30) and computational speedup (29) , for a backward pass. As illustrated in Fig. 2a and Fig. 2b , our method is more effective for both space complexity and latency for the backward pass when the truncation threshold decreases and the activation size increases. More details are provided in Appendix A. 4 . We hypothesize that the first components along each dimension are enough to encode most of the variance, implying that with relatively low values of K j , we can achieve good training performance. We later empirically confirm this assumption in a variety of experimental setups (Sec. 4.2).Signal to Noise Ratio. For simplicity, we rewrite in this paragraph ∂L ∂Wi as ∆W, A i as I and ∂L ∂Ai+1 as ∆Y. Regarding the error introduced by truncating I in order to retain the components corresponding to a given level of explained variance ε ∈ [0, 1], we demonstrate that the energy contained in the resulting gradient is equal to the energy contained in Ĩ. We note I, ∆W and ∆Y as the input activation, weight derivative, and output activation derivative in the frequency domain;I[u, v],the spectrum value at frequency (u, v). Applying the discrete Fourier transformation to Ĩgives us Ĩ = εI, which we use to compute the signal to noise ratio SN R Ĩ :EQUATION)Similarly, in the frequency domain, as the convolutional operation becomes a regular multiplication, ∆ W becomes ∆ W = Ĩ∆Y = εI∆Y . As for (12) , we obtain SN R ∆ W = (1ε) -2 = SN R Ĩ . A visual representation is provided in Fig. 2c . Analytically, this confirms the idea that the closer ε is to 1, the larger the energy is transferred from the compressed input to the weight derivatives (for ε = 0.8 we have SN R ∆ W = 25). Additionally, in our setup since we are only compressing the activations and given ( 2) and ( 3), the introduced error only transfers to the weight derivative for each layer. The activation derivatives that are propagated from one layer to another are exact computations as they do not involve the activations. This means that the error introduced when compressing the activations at each layer does not accumulate through the network. 
experiments: 
 In this section, we describe the experiments conducted to support the claims presented in Sec. 3. First, we introduce the setups used for our experiments (Sec. 4.1); then, we analyze the energy distribution in the different dimensions of HOSVD, providing an overview of the typical values of K (Sec. 4.2); finally, we test our algorithm in different setups, state-of-the-art architectures and datasets to evaluate the tradeoff between accuracy and memory footprint (Sec. 4.3). Experiments were performed using a NVIDIA RTX 3090Ti and the source code uses PyTorch 1.13.1.To validate our approach, we perform a variety of computer vision experiments split across two tasks, classification and segmentation.Classification. We explore two types of fine-tuning policies:• Full fine-tuning (referred to as "setup A"): Following conventional fine-tuning trends, we load models pre-trained on ImageNet [21] and we fine-tune them on a variety of downstream datasets (CIFAR-10, CIFAR-100, CUB [46] , Flowers [32] and Pets [54] ). • Half fine-tuning (referred to as "setup B"): Following Yang et al. approach, each classification dataset (ImageNet, CIFAR-10/100) is split into two non-i.i.d. partitions of equal size using the FedAvg [29] method. The partitions are then split as follows: 80% for training and 20% for validation. The first partition is used for pretraining, and the second partition is used for finetuning.Semantic Segmentation. Similarly to the half fine-tuning method, we reproduce Yang et al. segmentation setup: we fine-tune models pretrained on Cityscapes [6] by MMSegmentation [5] . Here there is only one downstream dataset which is Pascal-VOC12 [3] . Experimental details for both tasks (hyper-parameters, policy, etc.) are provided in Appendix B.2.Memory Logging. For HOSVD and SVD, instead of focusing on compressing the tensor based on rank, we control compression through the desired amount of retained information. As a consequence, we cannot explicitly control the memory usage of the principal components. In the results presented below, we will always include two columns displaying peak and average memory, along with their standard deviations.In this section, we conduct experiments to fine-tune the last four convolutional layers of MCUNet [24] following Setup A, and CIFAR-10 as the downstream dataset. Using HOSVD with ε set to 0.8, Fig. 3 shows the explained variance retained across dimension j as a function of K j , where j = {1, 2} corresponds to the two largest dimensions of the activation map. We define K (ε=x) jas the number of principal components required to retain at least a fraction x of the explained variance. We observe that along the largest dimensions (batch size and number of channels), less than 20% of the components capture more than 80% of the explained variance, confirming the assumption proposed in Sec. 3.4. Additionally, the curves observed present a logarithmic behavior, hinting at the possibility to reach high explained variance with little loss regarding the compression ratio. This is especially important as the SN R transmitted to the weight derivatives increases quadratically with respect to explained variance (Fig. 2c ). The results for the other layers are presented in Appendix B.3. Fig. 4 illustrates the results of performing HOSVD technique with different explained variance thresholds ε. The results are averaged over three different random seeds. For ε smaller than 0.8, we observe that as it increases, we achieve significant gains in accuracy, along with impressive compression ratios. However, when ε exceeds 0.8, the accuracy growth starts slowing down. Above the 0.9 threshold, the exponential growth of peak memory results in a worsened tradeoff between accuracy and compression ratio. Therefore, in subsequent experiments presented in this paper, we will use ε values of 0.8 and 0.9.MCUNet on classification with setup A. In this experiment, we fine-tune on CIFAR-10 an MCUNet pretrained on ImageNet, with the number of finetuned layers increasing from 1 to 42 (all layers). We compare vanilla training, gradient filtering with patch sizes of 2, 4, and 7, SVD, and HOSVD with an explained variance threshold of 0.8. For each method, we compare the effect of fine-tuning different model depths. Fig. 5 presents the performance curves for our experiments, with the X-axis representing activation memory in kiloBytes (kB) on a logarithmic scale and the Y-axis representing the highest validation accuracy. Each marker indicates the number of convolutional layers finetuned. For example, on the yellow curve representing the HOSVD method, the marker closest to the Y-axis shows the result when finetuning the last convolutional layer, the next marker represents finetuning the last two convolutional layers, and so on. The most effective method is the one whose performance curve trends towards the upper-left corner of the plot. We observe that as the number of finetuned layers increases, the gradient filtering accuracy increases up to a certain point, and then deteriorates, whereas the accuracy for SVD, HOSVD, and vanilla training keeps improving with additional layers, along a similar trend. Intuitively, gradient filtering might propagate errors through the layers during training, while our HOSVD method keeps the error introduced on each individual layer confined to that specific layer as demonstrated in Sec. 3.4. Moreover, we observe that for identical depths, SVD accuracies consistently exceed HOSVD ones. We hypothesise that this is due to HOSVD performing SVD across all modes of the tensor: it potentially loses information in all modes, whereas SVD only loses information in one mode. Compared to SVD, HOSVD significantly reduces memory, given equivalent accuracy levels (up to 18.87 times). Similarly for equivalent memory usage, HOSVD yields greatly improved accuracy compared to SVD (up to 19.11%). When compared to methods such as gradient filtering and vanilla training, given equivalent memory budget, HOSVD yield significantly higher accuracies. Notably, fine-tuning all layers with HOSVD requires much less memory than fine-tuning only the last layer with vanilla training, which is consistent with the theoretical compression ratio shown in Fig. 2a . Additional experiments with setup A can be found in Appendix B.5.ImageNet Classification with setup B. Table 1 presents the classification performance and memory consumption for MobileNetV2 [37] , ResNet18, and ResNet34 [15] models using various methods, including vanilla training, gradient filtering, HOSVD, and SVD on the ImageNet dataset. We observe in most cases that, for the same depth, SVD and HOSVD are competitive with the gradient filtering method in terms of performance while reaching much lower activation memory with HOSVD.Segmentation. Table 2 reports the segmentation performance and memory consumption for a variety of architectures as presented in Yang et al.. In general, we observe that increasing the level of explained variance from 0.8 to 0.9 substantially increases the performance with little trade-off on retained activation memory. This further confirms that most of the explained variance is contained in the first few components, allowing for efficient generalization with high compression rates. 
conclusion: 
 In this work, we have addressed one of the main obstacles to on-device training. Inspired by traditional low-rank optimization approaches, we propose to compress activation maps by tensor decomposition, using HOSVD as a supporting example (Sec. 3.2). We demonstrate that the compression error introduced is bounded and confined to each individual layer considered, validating our approach  
 
 Appendix: A.1 Forward pass Vs. Backward passWe consider the same notations introduced in Sec. 3.1. Regarding the training of a convolutional layer:• There is one convolution operation in each forward pass with a computational complexity ofΘ FLOPS (D 2 CC ′ BH ′ W ′ )• In each backward pass, there are two convolution operations, corresponding to formulas ( 2) and ( 3), and one weight update operation, with computational complexities ofΘ FLOPS (D 2 CC ′ BH ′ W ′ ), Θ FLOPS (D 2 CC ′ BHW ) and Θ FLOPS (D 2 CC ′ ), respectively.From this, we can calculate the ratio of computational complexity between forward and backward pass R FLOPs as follows:EQUATION)It is evident that R FLOPs is always smaller than one, indicating that the backward pass is always more computationally intensive than the forward pass.With the same notations introduced in Sec. 3.1 but in the case of a linear neural network represented as a sequence of n linear layers:EQUATIONwhere W i ∈ R O×I corresponds to parameters matrix of the i th layer. This layer receives an I-feature input A i ∈ R B×I with minibatch size B to produce an O-feature output A i+1 ∈ R B×O . The derivatives in that setup become:EQUATION)EQUATION)The same conclusions as for convolutional layers can naturally be derived from these equations for linear layers, namely that activations and weights occupy most of the space in memory when performing backpropagation.Since the convolution operation involves other variables such as stride, dilation, and groups, we provide a more precise description by rewriting in (2), ∂L ∂Wi as ∆W ∈ R Nout×C×D×D , A i as I ∈ R B×C×H×W and ∂L ∂Ai+1 as ∆Y ∈ R B×C ′ ×H ′ ×W ′ which gives us:Following the notations introduced in the main paper, we propose in this section to analitically study the overhead introduced in the forward pass when performing HOSVD compression, the speedup of efficient weight derivative computation and the required space to store the compressed activations. Overhead. During the forward pass, the overhead of activation compression is the computational complexity of performing the tensor decomposition, which can be calculated as follows:The computational complexity of SVD for a matrix of size m × n with m ≥ n is Θ FLOPS (m 2 n).HOSVD essentially performs SVD on each mode of the tensor. During the forward pass, for each convolution layer, given an activation map of size B × C × H × W , HOSVD involves performing SVD on four matrices of sizes B × CHW , C × BHW , H × BCW , and W × BCH. Therefore, the computational complexity for decomposition in the forward pass is:EQUATIONAdditionally, the amount of FLOPs that vanilla training requires to perform the forward pass is:EQUATIONFrom this, the total FLOPs that HOSVD requires is:EQUATIONBased on these calculations, we represent in Fig. 6 the evolution of FLOPs in the forward pass for both vanilla training and HOSVD, showing that our method results in an increased latency in the forward pass. Speedup. The key difference between HOSVD and Vanilla BP lies in computing ∆W. For Vanilla BP, the cost of computing ∆W is Θ FLOPS (D 2 CC ′ BH ′ W ′ ). Whereas our method involves:Computing Z (1) : Θ FLOPS (K 1 C ′ H ′ W ′ B), Computing Z (2) : Θ FLOPS (K 1 K 2 HK 4 K 3 ), Computing Z (3) : Θ FLOPS (K 1 K 2 HK 4 K 3 + K 1 K 2 HW K 4 ), Computing Z (4) : Θ FLOPS (K 1 C ′ H ′ W ′ B + K 1 K 2 HK 4 K 3 + K 1 K 2 HW K 4 + C ′ K 2 D 2 H ′ W ′ K 1 ).Therefore, the complexity for computing ∆W with our efficient reconstruction is:Θ FLOPS (K 1 C ′ H ′ W ′ B+K 1 K 2 HK 4 K 3 +K 1 K 2 HW K 4 +C ′ K 2 D 2 H ′ W ′ K 1 +C ′ CD 2 K 2 ). (28)We thus deduce the speedup ratio R S for the backward pass, between vanilla training and HOSVD (results represented in Fig. 2b ):EQUATION)Space Complexity. For vanilla training, storing A i correspond to the storage of Θ space (BCHW ) elements.For HOSVD, instead of storing the entire tensor, we store its truncated principal components: S, UEQUATION(K3) , and U (4) (K4) , which corresponds to a total of Θ space (K 1 K 2 K 3 K 4 + BK 1 + CK 2 + HK 3 + W K 4 ) elements. Thus, we obtain the storage complexity ratio R C as follows (results represented in Fig. 2a ):EQUATIONWe conducted classification experiments using MCUNet with setup A, fine-tuning on CIFAR-10, and segmentation experiments using the DeepLabV3 checkpoint provided by [53] , fine-tuning on the augmented Pascal-VOC12 dataset. Both experiments employed three different random seeds (233, 234, and 235). The results, presented in Table 3 and Table 4 , indicate no significant variation in performance across the different random seeds. Therefore, for subsequent experiments, we used a single random seed, 233. ImageNet Classification. In this setup, we use a similar finetuning policy to [53] . Specifically, we finetune the checkpoints for 90 epochs with L2 gradient clipping with a threshold of 2.0. We use SGD with a weight decay of 1 × 10 -4 and a momentum of 0.9. The data is randomly resized, randomly flipped, normalized, and divided into batches of 64 elements. We use cross-entropy as the loss function. The difference from their setup lies in the initial learning rate value after the warm-up epochs: in our case, the learning rate increases linearly over 4 warm-up epochs up to 0.005 (while in [53] it is 0.05). After this, similarly to their approach, the learning rate decays according to the cosine annealing method in the following epochs.Other dataset finetuning. We use the same set of hyperparameters as described in [53] for both setup A and setup B. For training, we use cross-entropy loss with the SGD optimizer. The learning rate starts at 0.05 and decays according to the cosine annealing method. Momentum is set to 0, and weight decay is set to 1 × 10 -4 . We apply L2 gradient clipping with a threshold of 2.0.For setup B, batch normalization layers are fused with convolutional layers before training, a common technique for accelerating inference. We set the batch size to 128 and normalized the data before feeding it to the model.Semantic Segmentation. Following the policy outlined in [53] , we utilized the pretrained and calibrated checkpoints provided by the authors for our experiments. The models were pretrained on Cityscapes using MMSegmentation, then we finetuned them on the augmented Pascal-VOC12 dataset. The optimizer's learning rate starts at 0.01 and decays according to the cosine annealing schedule during training. Additionally, we set the weight decay to 5 × 10 -4 and momentum to 0.9. The batch size is set to 8. For data augmentation, we randomly crop, flip, apply photometric distortions, and normalize the images. Cross-entropy is used as the loss function. Models are finetuned for 20, 000 steps. As anticipated in Sec. 4.2, Fig. 7 presents the explained variance as a function of K j at the j th dimension of the four fine-tuned layers of MCUNet. We observe that in the first two dimensions, which are also the largest, more than 80% of the tensor's energy is concentrated within the first 30% of the principal components.With the same experiment, Fig. 8 shows that K j tends to gradually increase after each training epoch, then peaks and begins to decline. Intuitively, at the beginning, the activation map energy is mainly concentrated in the few first components. As training progresses, this energy starts to spread out to other components, leading to an increase in the value of K j . Over time, the model learns how to efficiently condense the energy, leading to a gradual decrease in K j . This instability in K j throughout the training process is why we had to log memory as described in Section 4.1. This section presents latency comparisons between vanilla training and HOSVD. We conducted the experiments using MCUNet on a single batch of CIFAR-10 dataset with setup A and for one epoch only.Fig. 9 compares the execution time of the forward, backward, and total training processes between vanilla training and HOSVD. It can be observed that the backward processing time of HOSVD is tens of times lower than vanilla training, but the forward time is up to a thousand times higher. As a result, when considering the total training time (the sum of forward and backward), HOSVD is on average 4.29 times slower than vanilla training. This outcome is entirely expected, as the decomposition process introduces an overhead, as predicted and described in Fig. 6 and Appendix A.4. In addition to the main experiments, we also performed many classification experiments with setup A on various datasets, the results are shown in Table 5 . Especially, when fine-tuning SwinT on CUB200, Pets, and CIFAR-100, applying HOSVD to the last four layers (maintaining the rest of the architecture frozen) consumes less memory than fully fine-tuning the last two layers. This happens because the variance of the activation maps is concentrated in few components: for the same ε, K will be smaller.While HOSVD is effective in compressing the forward signal in DNNs for gradient estimation during backpropagation, it is just one among possible choices when dealing with tensor decomposition. Indeed, more variants of tensor decomposition methods such as GKPD or STP could have been used and evaluated and could have led to larger memory and computational gains. Our work opens the door to further explorations in such direction. Code: https://github.com/Le-TrungNguyen/NeurIPS2024-ActivationCompression.git 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: The authors submitted rebuttals, and all the reviewers here acknowledged them and had discussions with the authors.
I requested clarifications about their comments, and some of them responded.
Variance of the ratings was somewhat high as one of the reviewers (Reviewer HEKd) raised their score to 7, following multiple discussions with the authors. As they suggested, this is an interesting tensor decomposition research with great clarity and writing in general. They also value originality and its potential impact and interest of data center users.

I should clarify that I decided to put my focus on Reviewers HEKd, WgAz, and dqZf as I found none of the weaknesses claimed by Reviewer f1cQ's concrete, but more like questions. The reviewer argued that they had found authors' rebuttal satisfactory, but didn't provide any justifications (including their initial review) to support the rating.

Reviewer WgAz gave the lowest rating among the four reviewers, and key weaknesses they claimed in the initial review and follow-up are 
1.  "why not search for a more appropriate model architecture using NAS?"
2.  "Rebuttal was not convincing because the experiments are on small datasets with small models. [1] is more efficient and doesn't affect model performance ". 

For 1, I saw that the reviewer pushed NAS through interactions, but NAS is a resource-intensive approach. This work discusses on-device learning, and I believe that not using NAS cannot be a weakness.

For 2, I agree with the reviewer to the point in general, but again this work is about on-device learning. I also checked [1], but it doesn't discuss model performance (accuracy) but FLOPs, latency, and memory consumption.

Reviewer dqZf pointed out multiple weaknesses of this work, and the authors rebuttal addressed their concerns. But the reviewer didn't explain rationale of the rating 4. I requested a follow-up from the reviewer, but the reviewer did not answer my question.
Having reviewed the authors rebuttal, I found the rebuttal satisfactory. E.g. (1), "Severe performance degradation relative to memory reduction" should be observed as part of trade-off discussions, which also shows better trade-off by the proposed method than those of the baselines. E.g. (2), "Limited comparison with other activation map compression techniques" <- The authors provided additional experimental results using Swin Transformer model (a significantly different architecture) in the supplementary material, which seems like a reasonable choice to me. Overall, it seems to me that the reviewer is looking for reasons to reject.

Considering all these points, I found my position literally on on fence, but towards acceptance. My recommendation is based on the fact that the review by Reviewer HEKd shared very concrete suggestions and relevant concerns, and the authors successfully addressed the concerns through the discussions, thus I would like to recommend acceptance for this work.


[Suggestions]
While my recommendation is acceptance, I still highly recommend that the authors take into account all review comments and address their concerns in the camera-ready (e.g., clarifying motivation of this work). 
- Replace "AI model" with "deep learning model". AI is not the right term in this scientific paper, but just a distraction.
- My recommendation is partly conditioned on the authors' code release.

Reviews: review: paper_summary: paper_summary: Tensor low-rank decomposition is used to compress the backpropagation process of full connection and convolution in neural networks. The main process is to decompose input activation tensors into Tucker structures using HOSVD algorithm and truncate subtensors at each mode. Experimental result find an efficient trade-off between the desired explained variance and compression rates.
strength_weakness: strength_weakness: (1) There are a lot of confusing expressions on the formula.

(2) In formula 8, How to distinguish $U^{(k_j)}$s when the size retained after truncation on two modes is the same?

(3) The symbol $]$ in formula 11 is in the wrong position, and $\frac{\partial A_{i+1}}{\partial W_i}$ should be $\frac{\partial L}{\partial A_{i+1}}$?

(4) In Sec3.4 $\frac{\partial L}{\partial A_{i+1}}=\Delta Y$, but in appendix A.2 it expressed as \Delta O.
questions: questions: (1) In formula 12, how to use the discrete Fourier transform to get $\tilde{I} =\varepsilon I$? What does $uI$in the denominator mean?

(2) In formula 16, please give the definitions of $d, h’,w’$. 

(3) The methods mentioned in the related work are not compared with the proposed methods.
limitations: limitations: If the contraction between matrix and tensor is replaced by 1*1 convolution, then the proposed method should also be compared with CP decomposition, tensor train decomposition and other related methods.
paper_summary: paper_summary: [Editing to reflect my score increase from 6 to 7 after the author discussion phase.]

The authors tackle the problem of memory consumption due to needing to keep realized activation tensors available between their use in the forwards pass and backwards propagation in training neural networks.  They propose the use of tensor decomposition, specifically SVD and higher-order SVD, to compress activation maps and show that the compressed activations can be used in backpropagation without first decompressing.  An error analysis is performed, and experimental results confirm the hypothesis that significant compression can be applied while maintaining significant explained variance.  To validate the method's success on real workloads, the authors apply it to image classification and semantic segmentation tasks, fine-tuning pre-trained models for both tasks, as well as training from scratch for image classification.
strength_weakness: strength_weakness: **Originality**
Only one piece of past work comes to mind that was omitted: Rhu et al., "Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks" (HPCA 2018) compresses activation maps, using their existing sparsity, to reduce the cost of offloading in order to accelerate training.  While novel hardware was designed to keep up with compute throughput, if compression is the only goal, then this could simply be performed in software (like the submission's tensor decomposition).

**Quality**
A missing piece is the cost of compression: what impact does this have on the training speed?  This is related to the claim on line 59 that "gradient computation and parameter updating are considerably more expensive than the forward pass," which isn't obvious to me.  At a high level, each of the three calculations (forward propagation, gradient calculation, weight update calculation) are roughly equivalent in cost, and applying the weight update is very simple.  Increasing the cost of any one of the phases would seem to be as painful as any of the others.  Another claim that stood out, given the stated goal of "reducing the memory required for backpropagation" (line 72) is that the "main challenge limiting the feasibility of on-device learning lies in the computational cost of backpropagation" (line 58).  I'd also question the claim that "we can assume that the networks considered are already [weight] compressed" (line 159) - while it's extensively studied, it's not a solved problem.  It's fine to not tackle this particular problem, but stating that it's assumed to not be an issue is misleading.

Some of the results deserve some extra discussion.  Do the authors have a hypothesis about why SVD is superior to HOSVD (line 288)?  Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?

I have not seen CIFAR treated as a "downstream task" of ImageNet-1k before; it's typically just a simpler data set used, recently, as a proving ground before testing on larger data sets.  Is it common to fine-tune a network trained on ImageNet-1k on CIFAR10/100?

Given the significant memory savings of the technique, I'd have liked to see how it performs when fine-tuning all layers, not just a subset.  Is there a reason this result wasn't gathered?

**Clarity**
I've had to make an assumption about the behavior of the method: during forward propagation, I assume the *uncompressed* activations are used, and only compressed when writing to memory to be used later in the backwards pass.  (If this is not the case, then errors *will* compound.)

Figure 2 shows speedups and space gains, but without indicating what, exactly, is speeding up or seeing a reduction in memory space.  I assume, but am not sure, that it is for a single weight gradient calculation (of a size noted in the figure's legend and axes).

The wording of Figure 3's caption is confusing: I do not understand the meaning behind "Mbv2 fine-tuned on Cifar10 at initialization."  

None of the networks used in the experimental results have references attached.  For example, MCUNet is prominently used in Figure 4, but I couldn't find either a description of this network or a reference to it in the text.  It looks likely that it is in reference [24], but this only has a passing citation in the related work section.

There are some typos:

- Line 223: "derives" -> "derivatives"

- Line 234: the range of explained variance uses two open brackets.

- Reference [48] is listed with the incorrect year: 2024 should be 2023.


**Significance**
There are several ways that the results would be more compelling:

- Evaluation on more (difficult) tasks, like language modeling or object detection.

- Application to more recent or larger networks.

- More closely matching the densely trained or fine-tuned baselines.  As this is initial work in this area, the results shouldn't be dismissed, but there's still a non-trivial gap between the tensor-decomposed results and what the networks are capable of learning.
questions: questions: 1. Is there a reason that results for fine-tuning all layers were omitted?
2. Why is SVD superior to HOSVD in some results (line 288)?
3. Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?
4. What impact does the compression process have on the training speed?
limitations: limitations: The authors have addressed the limitations (in the appendix).
paper_summary: paper_summary: This paper proposes a method to compress activation maps in deep neural networks using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and Higher Order SVD (HOSVD). The goal is to reduce memory requirements during backpropagation, enabling on-device learning for resource-constrained environments. The authors provide theoretical analysis of their method's impact on memory usage, computational complexity, and error bounds. They demonstrate the effectiveness of their approach through experiments on various tasks, architectures, and datasets.
strength_weakness: strength_weakness: - Severe performance degradation relative to memory reduction
- Complexity of Implementation: Implementing tensor decomposition techniques like HOSVD can be complex and may require specialized knowledge, potentially limiting its adoption.
- Lack of Detailed Rank Selection: The paper does not provide sufficient explanation on how the appropriate rank for decomposition is selected, which is crucial for balancing memory reduction and accuracy.
- Inconsistent and contradictory results: SVD frequently outperforms HOSVD, contradicting the authors' hypothesis and lacking adequate explanation.
- Limited comparison with other activation map compression techniques.
questions: questions: - How does the proposed method perform on more diverse and complex neural network architectures, such as transformers?
- What are the practical challenges and considerations when deploying this method on real-world edge devices with strict latency and power constraints?
- How is the appropriate rank for decomposition selected, and what guidelines can be provided for tuning this parameter effectively?
limitations: limitations: - Specific to Certain Architectures: The experiments focus mainly on specific neural network architectures and tasks. It is unclear how well the method generalizes to other types of models and applications.
- Dependence on Hyperparameters: The effectiveness of the compression is highly dependent on the chosen hyperparameters for the decomposition, which may require extensive tuning.
- Resource Requirements for Decomposition: While the method reduces memory usage during training, the initial decomposition itself can be computationally intensive and may require powerful hardware.
- Practical Usability Concerns: Despite significant memory reduction, the method results in considerable accuracy drops, raising concerns about its practical usability. Vanilla backpropagation, while using more memory, generally maintains higher performance and might be a more realistic approach for actual deployment.
paper_summary: paper_summary: The paper addresses the challenge of on-device training for deep learning models, particularly focusing on the memory bottleneck caused by the storage of activation maps during backpropagation. The authors propose a method to compress activation maps using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and its tensor variant, High-Order Singular Value Decomposition (HOSVD).
strength_weakness: strength_weakness: 1. Using SVD or tensor decomposition to compress activation maps incurs significant computational overhead. This approach of trading computational complexity for space complexity is questionable, especially considering that the computational capacity of embedded devices is usually also limited. Moreover, this compression may introduce additional information loss, potentially affecting the model's performance.
2. For SGD, recalculating activations instead of storing them is a viable method to save memory. Given that both methods trade computation time for storage space, why use low-rank compression which may potentially affect model performance?
3. With limited computational resources, why not search for a more appropriate model architecture using NAS?
questions: questions: 1. LOMO[1] fused gradient computation and parameter update in one step to minimize the size of gradient tensors. Compared to this approach, what are the advantages of your method?
2. Would using quantization methods instead of compression methods produce better results?


[1] Lv, Kai, et al. "Full parameter fine-tuning for large language models with limited resources." arXiv preprint arXiv:2306.09782 (2023).
limitations: limitations: None

Comments: title: Answer to Reviewer WgAz concerns
Quality and clarity greatly improved
Reviewer-author discussions will end in about 30 hours
Thank you, answer to the extra two questions
Comments appreciated, two remaining questions
Start reviewer-author discussions right now
comment: Thank you for your response. Please find the answers to your concerns below.

[**What I want to know is, under limited memory conditions, which method has the least negative impact on model performance or even improves it?**]
HOSVD is the method that has the least negative impact on model performance for the same memory budget (and in the very low memory budget regime under exam).
Specifically, in Figure 1 in the Rebuttal file, the horizontal axis represents activation memory, and the vertical axis represents Top-1 validation accuracy. The first data point of a curve corresponds to fine-tuning the last layer, the second point corresponds to fine-tuning the last two layers, and so on until all layers are fine-tuned. The best method is the one with a Pareto curve that leans towards the **top left** corner of the graph (indicating a low memory usage and a high Top-1 validation accuracy). Among the methods compared, the one with the Pareto curve closest to this position is HOSVD.

[**If NAS has already identified a network that meets resource constraints, why would you still opt for compressing activations, a method that can potentially degrade performance?**]
Because under the same memory constraint, NAS requires a *significant offline computational cost* to find an optimal solution, while our method can easily address this issue directly online (on-device).
We remark that the potential loss is also tunable through a hyperparameter ($\varepsilon$).
Finally, under the same memory constraints, combining our method with NAS obviously can expand the search space by relaxing memory constraints hundreds of times, leading to even more optimal models. Our approach can be integrated with NAS approaches.

[**Additionally, I didn’t see a comparison between this method and using checkpointing.**]
In any case, the checkpointing would be worse than HOSVD.
In terms of memory, the best scenario for activation checkpointing is exactly the case when fine-tuning only the last layer with vanilla BP (>$10^4$kB in Fig. 1 of the rebuttal file). That is also one of the least memory-expensive layers, so we can expect that checkpointing would consume considerably more memory than HOSVD, making it in general worse. We agree that including this curve can be an interesting comparison, and *we commit to doing it in the final version of the paper*.

[**Additionally, why not choose to reduce memory consumption by swapping storage instead?**]
Because we expect offloading data to other storage units will *significantly* increase latency. For this reason, we want to design a technique that can directly fit the model in memory and we compare with similar approaches not relying on external memory sources. However, we agree that this is an interesting baseline, and *we will include the time overheads of the memory transfers on real devices in the final version of the paper*.
Thank you, authors, for your responses.  I find my concerns regarding the quality and clarity of the submission largely addressed, and significance is the main remaining weakness.  I'll therefore raise my score to 7 - Accept.

One note: the FLOPs overhead figure may be more impactful if the baseline computation amount is also shown, or if the overhead is expressed as a percentage of the baseline computation.  Without quickly seeing how many FLOPs are required for the un-modified  forward pass, the FLOPs required for the decomposition aren't as informative as they could be.
Thank you for providing a detailed rebuttal. What I want to know is, under limited memory conditions, which method has the least negative impact on model performance or even improves it? If NAS has already identified a network that meets resource constraints, why would you still opt for compressing activations, a method that can potentially degrade performance? Additionally, I didn’t see a comparison between this method and using checkpointing. Additionally, why not choose to reduce memory consumption by swapping storage instead? Thank you for your response.
Dear reviewers,

This is the final reminder for the reviewer-author discussions.
It will end on August 13 11:59 AoE, and then we will start AC-reviewer discussions.

If you have already concluded the discussions with the authors, thank you so much!

If you have not responded to the author rebuttal yet, please do so immediately. We have been waiting for your response.

In case you missed it, the general author rebuttal includes a PDF file.

Best,  
Your AC
Dear Reviewer dqZf,

Thank you for responding to the author rebuttal.

>  However, I believe my initial evaluation remains valid.

Can you elaborate this point? From the above comment, it is not clear to me which parts of your initial evaluation are addressed/not addressed.

Please clarify that and explain why you think so. That will help the authors improve their work if it's the case.

Best,  
Your AC
Many thanks for your prompt response. Here below please find answers for the embedded questions.

[**Units of overhead in Fig. 3(c)**]

The computation overhead is here measured in FLOPs. Specifically, the x-axis represents the assumed size of the tensor that needs to be decomposed, while the y-axis shows the overhead of the forward pass, which is the computational complexity of HOSVD for decomposing that tensor (as we discussed in General answer #3).

[**Rebuttal Fig. 1 different from paper’s Fig. 4**]

Many apologies for the confusion around this point. That's because we have updated the plot in the Rebuttal Fig. 1 (we will replace Fig. 4 in the paper with this one). Specifically:
- For HOSVD/SVD, we used to record the activation memory in the first epoch, but we changed it for the peak activation memory throughout the training process for a more accurate comparison as this is a key aspect in memory-constrained environments. The change in memory occupation is possible given that the number of components is not fixed but depends on the variance $\varepsilon$, causing a shift to the right of the curves. Please note as well that we have added more points for both the curves, and that the x axis covers a larger range of values (because of the addiction of the Vanilla BP). We remark that all the other memories presented in the paper are calculated on the *peak occupation throughout the full training*.
- The Gradient Filter curves remained unchanged.
- The curve on Vanilla BP has been added.
Thank you to the authors for the clarifications. These efforts have significantly enhanced the quality of the research. However, I believe my initial evaluation remains valid. Therefore, my score remains unchanged.
W1: A reference-by-reference can be acceptable in cases where the later reference supersedes the former.  In light of the different target domains (training vs. inference) and your claim that "With the exception of Eliassen et al.’s work which accelerates training runtime, most of these works focus on accelerating inference, in a similar way to traditional model compression," then I think an explicit reference and inclusion is warranted, so thank you for the update.  

W2.1: What are the units of overhead in Figure 3(c)?

W2.2: I understand, now - your "gradient computation and parameter updating" includes all the operations in backprop.  Thank you for the clarification.

W2.3: Sounds like a good revision, thanks.

W3.1: I see - for a given memory budget, HOSVD is superior to SVD.  It might make this argument more clear if the SVD and HOSVD results in Table 2 resulted in either the same accuracy or the same memory consumption (or if HOSVD were superior in both metrics).  Further, if there's an explanation for something the authors find "Suprising" (line 288), then including the explanation for this phenomenon will help the reader understand the behavior.

Figure 4 in the submission looks like it has different data than Figure 1 in the rebuttal PDF.  For example, HOSVD (red curve) at 10^2 kB peak memory has an accuracy of ~84% in Figure 4, but only ~66% in Figure 1.  Is this a different experiment, or has something else changed?  The Gradient Filter curves appear unchanged.

W3.2: I see, thank you for the explanation and update.  Please note in tables and figures where the data presented is from another source or your own implementation.

W3.3: I cannot see in [49] where a model trained with the ImageNet1K is applied to a CIFAR data set; I see different models trained for either data set, but perhaps this detail is omitted since it's so common?  Thank you for pointing out the transfer learning application of ImageNet1K->CIFAR10/100 in [2] and [25].

W3.4: Thank you for these additions.

W4.1: Thank you for the confirmation, and the proposed revision looks great.

W4.2: Thank you for the confirmation.

W4.3: I see, so this reflects one input sample from CIFAR10 (on a network pre-trained with ImageNet1K?).  

Embedded above are two remaining questions:
1) What are the units of overhead in Figure 3(c)?
2) Why does rebuttal Figure 1 differ so much from submission Figure 4?
Dear reviewers,

Authors submitted rebuttals, which should be visible to you now.

Please read the rebuttals carefully and start discussions with the authors now. 

The reviewer-author discussion period will end on August 13, 2024. Since authors usually need time to prepare for their responses, your quickest response would be very appreciated.

In case you had requested additional experiments / analysis and the authors provided in the rebuttal, please pay extra attention to the results.

Thank you,  
Your AC