Compressing the Activation Maps in Deep Convolutional Neural Networks and Its Regularizing Effect
Authors: Minh Hoang Vu
Anders Garpebring
Tufve Nyholm
Tommy Löfstedt
Venue: Accepted by TMLR ()

Abstract: Deep learning has dramatically improved performance in various image analysis applications in the last few years. However, recent deep learning architectures can be very large, with up to hundreds of layers and millions or even billions of model parameters that are impossible to fit into commodity graphics processing units. We propose a novel approach for compressing high-dimensional activation maps, the most memory-consuming part when training modern deep learning architectures. The proposed method can be used to compress the feature
maps of a single layer, multiple layers, or the entire network according to specific needs. To this end, we also evaluated three different methods to compress the activation maps: Wavelet Transform, Discrete Cosine Transform, and Simple Thresholding. We performed experiments in two classification tasks for natural images and two semantic segmentation tasks for medical images. Using the proposed method, we could reduce the memory usage for activation maps by up to 95%. Additionally, we show that the proposed method induces a regularization effect that acts on the layer weight gradients.

Published in Transactions on Machine Learning Research (03/2024)
Compressing the Activation Maps in Deep Neural Networks and Its Regularizing Effect
Minh H. Vu Deportment of Radiation Sciences
minh.on@@umu.
Umed Universily
Anders Garpebring
anders.garrperiingghhmm.mms
Department of Radiation Sciences
Umed University
Tufve Nyholm
Iufoe.nggholm@@mm..s
Department of Radiation Sciences
Umed University
Tommy Löfstedt
tommmy.ofstted@@uuu.ss
Deportment of Computing Science
Umeé University
Reviewed on OpenReview: https: // openreview. net/ forum? id=sIgh12FReM
Abstract
Deep learning has dramatically improved performance in various image analysis applications in the last few years. However, recent deep learning architectures can be very large, with up to hundreds of layers and millions or even billions of model parameters that are impossible to fit into commodity graphics processing units. We propose novel approach for compressing high-dimensional activation maps, the most memory-consuming part when training modern deep lcarning architectures. The proposed method can be used to compress the fcature maps of a single layer, multiple layers, or the entire network according to specific needs. To this end, we also evaluated three different methods to compress the activation maps: Wavelet Transform, Discrete Cosine Transform, and Simple Thresholding. We performed experiments in two classification tasks for natural images and semantic segmentation lasks for medical images. Using the proposed method, could reduce the memory usage for activation maps by up to 95%. Additionally,
show that the proposed method induces regularization effect that acts on the layer weight gradients. Code is available at https://a htpps:///ittan //tgtttt......// http://uhoaaggmi///ooeeeeeee theeengttthheeeeeeeeeeee Mase -a-.-
1 Introduction
In recent years, deep learning (DL) has emerged as a very powerful machine learning (ML) technique. The rapid development of deep neural networks (DNNs) partly due the availability of large-scale public data and increase in computing power. DNNs have led to breakthroughs in wide range of research areas such computer vision (CV), natural language processing (NLP), speech recognition, and autonomous vehicles, to mention just a few. Thanks to the evolution of new learning methodologies and architectures, DNNs often surpass other machine learning methods and sometimes even exceed human performance on an increasing number of tasks (Goodfellow et al., 2016). However, recent DL architectures can be very large, with up to hundreds of layers and millions or even billions of model parameters that are impossible fit into commodity graphics processing units (GPUs), that often today have between 11 and 24 GB of memory. For example, Brown et al. (2020) introduced auto-regressive language model, Generative Pre-trained Transformer 3

---

Published in Transactions on Machine Learning Research (03/2024)
(GPT-3), that has 175 billion parameters and requires 700 GB of GPU memory in total with the purpose of producing realistic text outputs. In recent work, Du et al. (2022) presented the Generalist Language Model (GLaM), the largest of which has about 1.2 trillion parameters, occupying around 4.9 TB of memory. These are the extremes, but standard convolutional neural network (CNN) models also require large amounts of memory for commodity GPUs. For instance, the ResNet152 by He el al. (2016) and the VGG-16 by Simonyan & Zisserman (2015) have about 60 and 138 million parameters, respectively, and require 24 and GB of memory (with a mini-batch size of 64 and an input resolution of 256 x 256). Another example is the recent Vision Transformer (ViT) model, trained by Zhai et al. (2022), that has two billion parameters and occupies 8 GB of memory (only for the parameters); it achieved a new top-1 accuracy of 90.45% on the ImageNet dataset (Krizhevsky et al., 2012). A popular and powerful architecture for semantic image segmentation, the two-dimensiinall (2D) U-Net, proposed by Ronneberger et, al. (2015) comprises about 8 million parameters and requires around 18 GB of GPU memory (with a mini- batch size of 64, an input resolution of 256 x 256, and 32 base filters).
A drawback of DNNs is that they are oflen weer-parameleriee,, making them expensive, both compulationally and memory-wise, but also making them prone to overfitting. Hence, modern DNNs are often expensive resources-wise which becomes a problem on platforms with limited hardware, especially on embedded systems but also on regular laptop and desktop computers. Over the years, much effort has been made to reduce the number of floating point operations (FI.OPs) and the overall network sizes. For instance, Tan & Le (2019a) presented more efficient network architectures by scaling and identifying the balance between network depth, width, and resolution to obtain better performance. Another direction has been to compress DNNs, often to compress the model weights (Hubara et al., 2017). Most works in model compression aim to reduce the model's weight size, but this is not the most memory-consuminn part (Mishra et al., 2020). The most memory-consunnin
parts of neural network are the intermediate activation maps (Rhu et al., 2016). This becomes especially apparent in medical imaging, when processing three-dimensionall (3D) volume images (Milletari et al., 2016; Kamnitsas et al., 2017). For example, with 32-bit doating-poin representations, a single 256 x 256 x 128 image (128 slices, each with resolution 256 x 256) occupy about 34 MB, a single 32> x 256 x 256 x 128 activation map (32 filters for 128 slices of resolution 256 x 256) occupy about GB, while their corresponding 32 x 3 x weight tensor only occupy about 3 kB.
When training a DNN, there are five categories of memory allocated on the GPU (Gao et al., 2020): model parameters (weights), feature maps, gradient maps (for back-propagation), ephemeral tensor (NVIDIA CUDA Deep Neural Network library (cuDNN) workspace and temporary tensors) and resident buffer (CUDA context, internal tensor fragmentation, and allocator reservation). Rhu et al. (2016) showed that the more layers a network has, the larger the fraction of memory allocated for the feature maps on a GPU. For instance, the percentage is always over 50% for a large model such as e.g., VGG-16 (Simonyan & Zisserman, 2015). It is even higher when the data is very high-dimensional, for example, when training on 3D medical images (Rhu et al., 2016). A potential solution to this problem would be to compress the intermediate feature maps in the network layers. It should be noted that the three terms "activation maps", "feature maps" and "intermediate feature maps" are used interchangeably in the rest of the paper.
In this study, we systematically investigated three approaches to compress the intermediate feature maps of a DNN to decrease the memory footprint on a GPU. The study was performed on natural and medical images for classification and segmentation, respectively. Unlike much previous work, the proposed method generally applies to any DNN model. Our method is uniquely versatile and capable of handling any combination of layers within the model. This adaptability allows us to optimize the compression process based the specific characteristics and interactions of the layers. Whether it is single layer, a complex combination of multiple layers, or whole network, our method consistently delivers efficient and effective compression. This demonstrates the potential of our approach for enhancing the performance and scalability of models across a wide range of applications.
Our main contributions can be summarized as follows:
We propose a novel way to compress the intermodiate feature maps of DNN. Our method stands out in its ability to handle any combination of layers within the model.
2

---

Published in Transactions on Machine Learning Research (03/2024)
2. We prove that the proposed method works regularizer that acts the layer weight gradients and, by that, can help reduce overfitting.
3. We demonstrate the utility of the proposed method by comparing the GPU memory usage and performance of bascline models to that of models using the proposed compression method. The comparisons were made with respect to graphical memory use, training time, prediction time, and relevant model evaluation metrics. The experiments were performed on multiple datasets and network architectures.
2 Related Work
Many lightweight CNN architectures have emerged in recent years that try to find compromise between computational complexity and model accuracy (Howard et al., 2017; Iandola et al., 2016; Deng et al., 2020). For instance, Redmon et al. (2016) proposed the You Only Look Once model, that includes e.g.; convolutions and depttwise--cparabbl convolutions. Standard compression methods include for instance to prune weights (Yu et al., 2018; Hagiwara, 1993; Lee et al., 2018) or quantize the weights or activa- tions (Dally, 2015; Rastegari et al., 2016; Gysel et al., 2018). Other common approaches include adjusting pre-trained networks by substituting the convolutional kernels with low-rank factorizations of them and grouped convolutions (Xue et al., 2013; 2014). In particular, in these works, a pre-trained weight matrix is lactorized into a product of two smaller matrices, serving the same purpose as the original weight matrix but consuming less memory.
Another approach is knowledge distillation, where a smaller network learns to replicate the function of a more extensive and more complicated network Buciluã et al. (2006). Buciluă et al. (2006) investigated the possibility of shrinking model sizes by constructing an ensemble of models and then using those models to teach a student network. They trained student network utilizing supervision from the pscudo labels supplicd by the teacher network. This phase was done by employing a leacher net work to label a significant amount of previously unlabeled data. They found that the performance was comparable to the first ensemble despite the network being one thousand times smaller. Hinton et al. (2015) introduced a neural network knowledge distillation strategy in which a single model was trained to distil the knowledge of group of other models.
Different forms of activation regularization have been presented in the literature and include common methods such as dropout (Srivastava et al., 2014), batch normalization (loffe & Szegedy, 2015), layer normalization (Ba et al., 2016), and & regularization (Merity et al., 2017). Additionally, the impact of sparse activations has been investigated, for instance, in sparse autoencoders (Hu et al., 2018) and in CNNs (Glorot et al., 2011) where rectified linear unit (RcLU) activations give rise to sparsity in the activation maps. Glorot et al. (2011) also briefly explored the usage of &, regularization to enhance the regularization effect. On the other hand, the usefulness of the regularizer has not been well investigated. Wang & Cheng (2016) presented a low-rank and group sparse tensor decomposition technique to speed up the CNN test phase. In their approach, the kernel tensor was decomposed for each convolutional layer into the sum of limited number of low-rank tensors. Back-propagation was then used to fine-tune the whole network to meet its ultimate classification target by replacing all of the original kernels with the estimated tensors.
There are a limited number of works that address activation map compression. Dong et al. (2017) aimed to forecast which output activations are zero to avoid calculating them, hence decreasing the number of multiply-accumulate operations executed. Their strategy also altered the network architecture but did not highlight the sparsity of the activation maps. Gudovskly et al. (2018) compressed the feature maps by first projecting them to binary vectors and then using a nonlinear dimensionality reduction method. It performed marginally better than quantizing the activation maps. Dhillon et al. (2018) proposed stochastic activation pruning as an adversarial defense. This method prunes à random subset of the activations and rescales the remaining ones to compensate for the pruned activations. However, their proposed sampling method yields the highest performance when 100% of the samples are selected, resulting no change in sparsity. In other works, Alwani et al. (2016) and Chen et al. (2016) reduce the network's memory consumption by recomputing the activation maps rather than saving them, which is effective but incurs a heavy computational cost.
3

---

Published in Transactions on Machine Learning Research (03/2024)
Rhu et al. (2018) explored three techniques for lossless activation map compression: run-length encod- ing (Robinson & Cherry, 1967), zero-value compression, and Zlib compression (Deutsch & Gailly, 1996). Because of high computational costs, Zlib cannot be used in practice. The other two are hardware-friendlfy and achieve competitive compression when sparsity is high. Huffman coding (Huffman, 1952) and arithmetic coding (Rissanen & Langdon, 1979) are two other methods proposed the research community for accom- plishing lossless weight compression. Finder et al. (2022) proposed Wavelet Compressed Convolution (WCC), compression method based on the Wavelet Transform (WT) for high-resolution feature maps. The WCC was integrated with point-wise convolutions and light quantization (1-4 bits).
We have developed a novel compression method that acts on the activation maps (primarily in CNNs). The activation maps are decompressed in the backward pass and used instead of the original activation maps when computing the gradient of the loss with respect to trainable parameters in that layer. We also show that this activation compression induces a regularization on the compressed layer's weight gradients, similar to gradient penalty (Gulrajani al., 2017).
3 Proposed Methods
We propose to compress the intermediate activation maps in the layers of a neural network to reduce the GPU memory use. The compressed feature maps are kept in GPU memory instead of the original feature maps until they are decompressed (or reconstructed) when needed in the back-propagation pass when computing the gradient of the loss function with respect to the network parameters in that layer.
We evaluate three different compression techniques for the activation maps: WT (Haar, 1909; Zweig, 1976; Mallat, 1989), Discrete Cosine Transform (DCT) (Ahmed et al., 1974), and what we denote Simple Thresholding (ST). The DCT is used in JPEG* and MPEG-C*, while the WT is employed in JPEG 20003, In the WT method, the feature maps are first WT transformed, and the WT coefficients are then thresholded. In the DCT method, we apply DCT to image blocks, using standard DCT 16 x 16 blocks. Next, each block is thresholded. The thresholded feature maps are reconstructed by computing the inverse transforms in both the WT and DCT methods. We further evaluate the ST, in which thresholding is performed directly on the feature maps. The thresholded feature maps are converted to sparse tensors that are stored on the GPU, In the following section, describe these compression techniques in more detail.
3.1 Compression Methods
3.1.1 The Wavelet Transform
This work employs the WT (Haar, 1909; Zweig, 1976; Mallat, 1989) to compress 2D and 3D feature maps at multiple resolutions. Let X E R22mx2m be real array, and let g and h be low-pass and high-pass filters related to each other by what's called a mother wavelet.
The decomposition of X along the x-direction in the first level results in,
y = (X * g) 2 (X * h) 2
(1)
y(1)
(2)
high
where (1))
high mm222m are wavelet coefficients with respect to low and high pass filters along x-direction. The | is a subsampling operation. Equation 1 and 2 can be extended along y-direction to
mmx22 and (1)
http:///ww...pec.....ooo/.////// Picturec Piituure_xxpertt__._e_eeee Grroup
4

---

Published in Transactions on Machine Learning Research (03/2024)
accomplish 2D wavelet decomposition at one level,
y())
(V
low ! )) 2,
(3)
low,low Jow
y(1)
(Y
(1)
) ) 2,
(4)
how,
low
(
high 5 T) 2, (Y h hT) 2
(y
(5)
lowh,low
y
(3) high,high high
(6)
high, E Rm denote wavelet, coefficients in the first level of the wavelet decomposition. The process is repeated recursively at higher levels using the low-pass filtered outputs. Figure 1 illustrates a two-level wavelet decomposition. Thresholding is applied to all components, i.e. (2)
where y
low, low: Now.high- low and (
(
(2)
low, high how? high, Xoow high Ynigh, hiwh,low!
(() (1)
(2)
high, and low
1-
y-1tt higs..ov
( low, Jow
high,Jow
high,low
y- ligh.atss
X
(-I)) high
y high
(-) low, low,high high
y high,high
a) Tensor
b) Level 1
c) Level 2
Figure 1: Illutratioon of two-level WT docomposition. After dccomposition, the top left component Y-
low. low,low low
in (b) contains the low-frequency component of the image in (a), and similarly, the lop-left corner in (c) contain the low-frequency components of the top-left down-scaled part in (b). Thresholding is then applied to all components, such that the smallest wavelet coefficients set to zero, giving minimal losses when the image is reconstructed later on.
Assuming we employ only one-level WT decomposition, using Equation 1-6, the X can be reconstructed as,
X low
(1)
g T g + (X-(1)
(1) T l To *
high
(Yrl))
(1)
T g h + high T h h x h,
(1)
low
X Xlow,low low
X low Xlow,iig high hligh,low + X high,
where T denotes a transposed convolution. For higher decomposition levels, X still has the same form as Equation 7. It should be noted that we used a three-levels WT decomposition for all experiments in this work.
The Wavelet transform can also be expressed more compactly as,
= WX,
(8)
where W is an orthogonal matrix that performs the WT.
3.1.2 The Discrete Cosine Transform
In the DCT method, we first split the matrix into 16 blocks. We then employ the DCT of type II (DCT-II) proposed in Ahmed al. (1974) to compress each block. We used an invertible DCT formulation in this work.
5

---

Published in Transactions on Machine Learning Research (03/2024)
compress
reconstruct
X
Y
Y
II
II
Y
X
T
TR
Sparse
GPU
Dense
T-1
Forward propagation
Backward propagation
Figure 2: The proposed compression method. The transform T and T denote decomposition and reconstruction operation, respectively. The TR denotes the thresholding step, while Sparse and Dense stand for sparse encoding and dense decoding, respectively. First, the imput X is: (i) decomposed into multi-level coefficients Y if the WT is used, or (ii) split in 16 x 16 blocks and applied DCT for each block, if the DCT is used, or (iii) unchanged, if the ST is used (see Section 3.1). Second, each element in Y is thresholded to produce a thresholded tensor Y (Section 3.2). Third, the Y is encoded in a sparse format, denoted II, to decrease the GPU memory usage, and these sparse lensors are then stored on the GPU instead of storing X there (Section 3.3). Fourth, Y' can be retrieved by reconstructing the sparse tensor II. Last, the thresholded version of X, the X is used compute the gradient instead of X in the backward propagation (Section 3.1). In summary, propose to GPU memory by storing II instead of X in the forward propagation. In the backward propagation, X decompressed version of X, is used when computing the gradients.
Let X R22x2m be real array, which split block. In this case, m equivalent to 8. The DCT of X is defined as,
2m 1a--1 2m
(2u + 1)xi
(2v + 1)xj
Vs.j m
Σ Σ
A,A5 cos
cos
-uuv
4m
4m
==0
with Tu,o E X and Yij E Y. The and denote the row index of X and Y, respectively, The and j denote the column index of X and Y, respectively. The A is
k
V2 A
1.
otherwise.
The corresponding inverse 2D DCT is defined,
1
2m-1
(2u + 1)mi
(2v + 1)mj.
Lu,
A,A
Vigi
m
4m
4m
i=
with Fu.o X and Vigj E Y.
The DCT is also linear function, and hence, similar to the WT, we can also express it more compactly as,
== BX,
(9)
with B the (orthogonal) DCT.
3.1.3
The Simple Thresholding
With the ST method, the activation maps are not transformed but are dircctly thresholded. The thresholding method employed described below.
3.2
Thresholding
The second step of the compression is threshold the (transformed, in the case of WT and DCT) activation maps. The core idea is determine threshold, where values below the threshold set to and thus

---

Published in Transactions on Machine Learning Research (03/2024)
removed. The threshold is found using "top-k approach" that aims to retain a certain number of elements. It is detailed below.
Let Y be a real array of arbitrary dimension, and let
y vec(Y) = (V1. U2
UN)
be vectorization of Y, where N is the number of elements in y. Without loss of generality, we assume the elements sorted in monotonically decreasing order of their absolute values.
The index, k of a threshold element, yk, of is determined as,
1 N
k
where 1 S < 00 is the compression ratio and |-] is the floor function. The compression ratio is defined the ralio between the uncompressed size (e.g., an array has N elements) and compressed size (e.g., compressed array has k elements). The impact of X on three arrays with different characteristics is illustrated in Figure 3. The first array has a substantial number of low values, the second has a normal-like distribution, and the third has a substantial number of high values.
We construct binary arrays, M and M that, when vectorized, have elements
mf =
- 0. otherwise,
1, i k
and
m m 1,
if k,
0, otherwise,
respectively (large and small), giving masks for large and small values as,
M' = vec (mm" and M = vec (mm).
Hence, we determine the & largest absolute values of the given array. We denote the special case of no compression when À = 1 as the baseline (BL).
We then select the large and small values as,
y = M' ④ Y and Y -0 OY=
(10)
respectively, and decompose the Y by
T Md) @ M
(11)
where compression is achieved by removing the smallest (in absolute value) N - elements of Y, i.e. setting Y to zero.
3.3 Sparse Tensor
In the third step, the thresholded coefficients are stored a sparse tensor format (the coordinate (COO) format) to actually reduce the memory consumption after thresholding. Hence, the sparse tensor what is actually stored on the GPU. The sparse representation is computed and denoted as,
II Sparse(Y
(12)
and when retrieved in the back pro-ppropagttion phase, the reconstruction of the dense tensor is denoted,
Y DDeneeeII (II).
(13)
See Figure for the entire process of compressing a tensor (transforming and thresholding it), storing it in sparse format GPU, and retrieving and reconstrueting it.
7

---

Published in Transactions on Machine Learning Research (03/2024)
1.0
22-
0.8
stmmpees
0.4
0.0
intensity
A=111
1=1.33
2 2.00
A 4.00
A 10.00
(a) Low
ooter
0.8
ssamples
0.6
0.4
0.2
5
6
intensity
1-333
2.0.0
-
10.00
(b) Normal
isamples
0.6
0.4
0.0
intensity
2.00
(c) High
Figure 3: Effects of compression ratio, X, on three arrays with: (a) substantial number of low values, (b) normal-like distribution and (c) substantial number of high values.

---

Published in Transactions on Machine Learning Research (03/2024)
4 Analysis of the Regularizing Effect
We show in this section that the proposed compression methods, using any of the proposed compression methods, induce a regularization effect that acts on a compressed layer's weight gradients. We show it in the general case and then demonstrate also for three common layers: convolution, ReLU, and fully connected layers.
4.1 Forward and Backward Propagation
Suppose that we have a feod-forward neural network (NN) with n layers,
(2)) V = ((U;0) = - ( (()
(14)
where U and V are real imput and put arrays the network, and (wo()),... (x)) denotes all trainable parameters for all layers. The f()) the first layer of the network, f(2) the second layer, and so on.
When apply the network to an input, U, the data flow forward through the network. The input, U, provides the initial information, which is then propagated to all hidden units before producing output prediction, V. Together with a ground-truth target, V, we want to find network parameters that minimize a scalar loss function, L(V, V). When using first-order optimization algorithms to update the network's parameters, the back-propagation algorithm [Rumelhart et al., 1986) is the most common way to determine the gradient of the loss, VoL(V, V), with respect to the model parameters, O.
Consider the m-th layer of the network in Equation 14. Let the input and output of the m-th layer be X and Z, i.e.,
Z= f(m) (X),
(15)
where the f(m) would typically either be an alfine function or element-wise non-linear activation function. We denote by I. and VzL the gradients of the loss, T.VV, V), with respect to X and Z, respectively, and Vo L denote the gradient of the loss with respect to the m-th layer's trainable parameters, ((()) We further denote by V Ww the gradient of the loss, L, with respect to the layer's weights, w(m), when the input X is replaced by the reconstructed X
The claims below are made under the following assumptions.
Assumption 1. The elements of the binory array, M', in Equation 10, independently follow a Bernoulli distribution with probability }
Assumption The input X to a compressed layer is in one of the following forms, either
x DD @ (Cxx)) or x (M' (XC.))
with the C, and Dt, for E {r,0}, being arbitrary real (or complex) arrays, and with M binary array as in Equation 10.
The following lemma allows us to determine a regularization effect on the layer weight gradient when the layer is compressed, i.e., when layer's input, X, is replaced in the backward pass of back- pack-propagation by its (possibly transformed) thresholded (and possibly back-transformee) version, X
Lemma 1. Under Assumption 1 and Assumption 2, if V Wwimm L either has the form,
Wimm T g( ((Zz)).
(16)
the form,
V =
(17)
where is a linear function, then
EMi [[ (m)) -1bi] S 1 ||c,
|DT 9(V2 LI.. =t=1,
[PiglVZZL) 12 ft=r,
9

---

Published in Transactions on Machine Learning Research (03/2024)
where 1 1 > so is the compression ratio, E {r,4} denotes the left or right versions of the compression in Assumplion 2, and |I-||p denotes the Frobenius norm. The expectation is taken over all possible binary matrices, M
Proof. To simplify the notation, let
A = A=g(ZZ),
and let
C
and expand Equation 16, we see that,
E M
V
E M
(X') A
MM
[I[MM'
④ E)
D. Al
I EM
[||
E
||D) A
Σ (m)e:)
||D Al
Σ EE ((m
ee
D| A
ΣΣ K of |DD A
GEE
Σ c |DT A
GEE
EE
|ID
Al
|| ID A
where E A denotes element (the i-th element in any order) from arbitrary shaped array A. The inequality is the Cauchy-Schwarz, and we use that the expectation of a squared Bernoulli random variable has the same expectation as the Bernoulli random variable itself.
A Analogously, we expand Equation 17 and have that
EMI
[[[
= EM [[I @ AA
< E
[[M C (XC,)
[ AD
|XC, |DA
where we followed the same steps as in the expansion of Equation 16 above.
Lemma tells us that by adjusting the compression ratio, X, we control a bound on EM [[[ w(m) L1 When X is set 1, then there is no compression, and the layer is thus unaffected. When X is increased towards xc (L.e. decreases), the 4 (m)) L is shrunk towards zero (its expected squared Frobenius norm goes zero).
We now use Lemma 1 to show that all the proposed compression methods induce gradient regularization the layer's weight gradients.
Theorem 1. Under Assumption 1 and Assumption 2, if V (m) L either has the form,
V (wim)
X g( V ZL);
(18)
or the form,
T= (VVgv(ll I.).
(19)
10

---

Published in Transactions on Machine Learning Research (03/2024)
where is lincar function, then
| |
for ST, g L) for WT, | 9(V2 P.. for DCT,
E
V (m) LIli!
x
[ |
(20)
||X
where W is the WT and B is the (orthogonal) DCT.
Proof. We consider three cases, corresponding to the three proposed compression methods.
Case 1: The ST. We let C, = D: be identity matrices, for either value of E [1,r}, then invoke Lemma
Case 2: The WT. The WT an orthogonal lincar transform, and we can express the layer's inputs in cither of the two forms,
x = Wr' (M' @ (WiX)) (M' @ (X,
(21)
and invoke Lemma 1.
Case The DCT. For simplicity, we only consider the orthogonal discrete cosine transform, but the following also holds for a non-orthogonal DCT, just with different forward and backward transform matrices. We write the inputs to the layer as,
B @ (BX))
where B is the discrete cosine transform, and then we invoke Lemma 1.
Before showing that this also applies to the result of a convolution, we first introduce matrix representation of a convolution (see, e.g., Goodfellow et al., 2016). Consider two real arrays, A and B, of arbitrary dimension, and their discrete convolution, which we write as
A B vee (vec(A) B,)
vec -1 (A, vec( BB),
where * denote the (discrete) convolution operator, and A, and B, are the matrices corresponding to the convolution (they are Toeplitz matrices) such that the conversion from convolution multiplication operation is valid.
We note that,
A B |/ee (vec( (A). B
|lve(A)) B,
S (ve(A))l
|BP
|||l ||2
(23)
Lemma 1 can also be immediately applied when the gradient, V Voiim L, the result of a convolution. Corollary 1. Under Assumption 1 and Assumption 2, if Weitn has the form,
A,
(24)
where denotes a discrete convolution and g( V z LZ) with g a linear function, then
1 ||
for ST, I -TII 1 for WT,
EM [
w L S
W 1
BX 2
B 1-
I
1A, for DCT,
where W is the WT and B is the (orthogonal) DCT, The A, matria representation of the convolution with A.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Proof. We again consider three cases, corresponding to the three proposed compression methods.
Case 1: The ST. We express
X A= = vec -1 ((vec()). ,A),,
then
E M
V
wim) L1b
EM
[|| Al?
EM'
[[|vec - X []]
< EM
||| | 1r
EM
[[]
x [ ||
Hence, following Lemma 1, we obtain,
[ wmm) T S
| 11 ||
Case 2:
The WT. We express X' as in Equation 21, follow the same steps in the expansion in the ST, apply the Cauchy-Schwarz inequality, and use the same approach as in Lemma 1.
Case 3:
The DCT. We express X' as in Equation 22, follow the same steps as in the expansion in the ST, apply the Cauchy-Schwarz inequality, and use the same approach as in Lemma 1.
4.2 Convolution layer
Consider the m-th layer of a neural network with input X and output Z as per Equation 15. Suppose that the m-th layer in Equation is a convolution layer with weights Q and biases b, such that,
Z, ((m) (m)) (x) X * S2 + by, Vj
(25)
where Sy denotes the j-th convolution filter and by is the j-th bias
Take the derivatives the loss with respect to X, bj, and S2 in Equation 25, using the multivariate chain rule (e.g., Rumelhart et al., 1986), we express the gradients as (Zhang, 2016),
OZ, =2,L= V2
T
VxL.
ax
8Z
Va
V2, 1= X V2
(26)
an,
aZg
Vb 1=
Vz, I, E(V2,
ab;
with Jacobians az, az, and az; 8X on, 86, and where Sz have all dimensions reversed (flipped) compared to Zz
We thus see that neither Va, L. nor V L depend on X (and thus neither on X'), but that Vn, L does. The following corollary to Corollary is an immediate consequence.
Corollary 2. Under Assumption 1 and Assumption the expected squared norm of the gradient, Vn, L, of the loss function with respect to the j-th convolution filter, S2j, in convolution layer is bounded as,
X || ||A,1
for ST, | for WT, T 1
E EM'
mm
L
1
S
1
W X 1 I
BX
1 |B-T13
||
for DCT,
where A = A= and A, is a motriz representation of the convolution with A. The W the WT and B is the (orthogonal) DCT.
Proof. Follows from Equation 26 and Corollary 1.
12

---

Published in Transactions on Machine Learning Research (03/2024)
4.3 ReLU layer
Consider the m-th layer of the network in Equation 15 with input, X, and output, Z. Suppose that the m-th layer in Equation 15 is ReLU layer, such that
0,
2, S0
Tiz
otherwise
with Z and 2, X.
Taking the derivative of the loss L with respect to 24, and using the chain rule, we get
0L
0,
2; 0
Dz,
aL 881
otherwise
with OL Sms
E VxL and OL dss
E VzL.
For the ST, we let
X' M O
then
aL
0,
mfa; 0
Oz.
8L
herwwss
82,
which we collect as
X M ② VXL.
(27)
A corollary to Lemma now follows.
Corollary 3. Under Assumplion 1 and Assumption 2, when ST compression is used, the expected gradient. of the loss function with respect to the input X of a ReLU layer is bounded as,
1 X LV
EMP [IV
1 S
Proof. We use Equation 27 and have
EMi [1 [| = EM |/| V x [
-EM [||M' ( V T
From here, we follow Lemma 1, and obtain
Em L S
L
which concludes the proof.
4.4 Fully-connected layer
Suppose that the m-th layer of the network in Equation 15 is a fully-connected layer with weights Ω Rexm biases b E IRm inputs X guxp, and outputs 12axm such that,
Zj f(m) (X) = Xw, + bb.
where wy denotes the j-th column of Ω the by is the j-th bias, and zj is the j-th column of the output, Z.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Taking the derivatives of the loss with respect to X, Wj, and by, and using the chain rule, obtain,
dzj
Vx x
2, L - (w) 2 V)
OX
dzj
V ww L
1= X V
(28)
dw,
0L
Ozj
bbg ab, V2,L= V2 =v,
with the Kronecker product and I the identity matrix with dimensionality implied by context.
We see that neither Vx I. nor Al is affected by the replacement of X by X since they do not contain X, 36, but that Vu,Ld does. corollary Theorem then follows.
Corollary 4. Under Assumption 1 and Assumption the expected squared norm of the gradient, Ve,, L, of the loss function with respect to the j-th weight vector, Wj, of a fully-counecled layer is bounded as,
X P.
V 2, LIl
for ST,
E
S A
[
X
|
Vz, L1
for WT, for DCT.
|BX P
B L|::
where W is the WT and B is the (orthogonal) DCT.
Proof. Follows immediately from Equation 28 and Theorem 1.
4.5 Connection to Robustness
To date, several studies have investigated regularizing the Jacobian matrix of a DNN's output to improve its robustness. For instance, Jakubovitz & Giryes (2018) used the Frobenius norm of the Jacobian matrix of the network output. This regularization was applied after standard training was completed. Their empirical findings indicated that their proposed approach resulted in a more robust model, while minimally affecting the accuracy of the original network. Sokolić et al. (2017) demonstrated that to achieve good generalization with DNN with arbitrary depth and width, it imperative that the network's Jacobian matrix has limited spectral norm within the vicinity of the training data. Tn another work, Hoffman et al. (2019) improved training stability by Jacobian regularization, and also showed that it increased the classification margins. They reported that with Jacobian regularization, the model became notably more robust, when evaluated against both random and deliberate input perturbations, while simultaneously maintaining the model's gencralization on clean data.
Consider an arbitrary DNN. The full Jacobian of the network can be writlen as,
J = A
where is the gradient of the network output, C is the Jacobian of all layer after layer B, A is the Jacobian of all layers before layer B, and B denotes the Jacobian in layer B.
The norm of the full Jacobian is then,
J - = T S I B ICP.
and hence, if we apply the proposed compression in layer B, bounding the expected norm of the Jacobian in layer B, then full Jacobian is also bounded. Thus, constraining a network's layers should result in a more robust model. This is an interesting connection that will be evaluated in future work.
5 Experiments
This section presents the experiments that we have conducted and the datasets and evaluation metrics used. The implementation and training details are detailed in the Supplementary Material.
14

---

Published in Transactions on Machine Learning Research (03/2024)
5.1 Datasets
To evaluate the utility of the proposed method, we conducted tests on four distinct datasets. All datasets used in this work are accessible to the public. The evaluated datasets, data augmentation, and experimental setup in this study are summarized in Table
MNIST The Modified National Institute of Standards and Technology database (MNIST)4 (Deng, 2012) is an extensive library of handwritten digits containing training set of 60 000 and a test set of 10000 grayscale 28 28 images. It is subset of more extensive NIST-provided collection. We used the MNIST for classification.
CIFAR-100 The Canadian Institute for Advanced Research, 100 classes (CIFAR-100) (Krizhevsky, 2009) dataset is for classification. It includes 60 000 color images with a resolution of 32 x 32 pixels. The images are grouped into one hundred categories, with 600 images in each category. There are 50 000 training and 10000 lest images.
BRATS19 The Brain Tumors in Multimodal Magnetic Resonance Imaging Challenge 2019 (BRATS19) (Menze et al., 2014; Bakas et al., 2017; 2018) released in conjunction with the International Conference Medical Image Computing and Computer Assisted Intervention (MICCAT) conference in 2019. It provides 3D multiple pre-operative magnetic resonance imaging (MRJ) sequences for 285 patients. For each patient, T1-weighted (Tlw), post-contrast T1-weighted (Tlc), T2-weighted (T2w), and T2 Fluid Attenuated Inversion Recovery (FLAIR) scans were obtained using various procedures and scanners al a magnetic field strength of T.
SPLEEN The Spleen Segmentation Decathlon (SPLEEN) data part of the Medical Segmentation De- cathlon", an open-source collection of datasets encompassing various anatomies and segmentation tasks (Simp- son et al., 2019). It comprises = 61 3D computed tomography (CT) scans in which the spleen was semi-automatically annotated and was originally part of a research project on splenic volume change after chemotherapy in patients with liver metastases.
5.2 Experiments
We carried out the experiments on four different datasets (see Section 5.1) and three neural work architectures (see Section 5.4 in the Supplementary Material). These networks cover several network sizes and architectural families. We conducted experiments using the three different compression methods on numerous choices of layers or blocks to compress. Further, we varied the compression ratios between and 1000. We investigated the impact of the proposed activation compression method with respect to the following factors.
Impact on Memory Footprint (Section 6.1). In this experiment, we analyzed the detailed footprint in each step in the proposed method by altering compression methods and compression ratios on two arrays with different sizes.
Impact Reconstructed Image (Section 6.2). This experiment examined the impact of the compression methods at different compression ratios. Two images were selected from the datasets: one natural image from the CIFAR-100 dataset, and a 2D slice from the BRATS19 dataset. Further, we compared the mean squared error (MSE), peak signal-to-noiss noise ratio (PSNR), and size in MB stored in GPU memory betweon the original and the reconstructed images. Additionally, we computed the percentage from the fraction of the compressed tensors' memory consumption over the original images' memory consumption.
Impact Performance (Section 6.3). We evaluated the performance of the proposed method compared to the baseline by varying compression methods, compression ratios, and layers or blocks to compress on the four datasets. In this experiment, we compressed only batch normalization layers.
'http:/ lecun wwtp:/ewww.n....//eeeee..o.. comm/xxb/nnist/ inttps:/kww.co. toronto. adu/-krri/cifar html hnttpe://www. med. www.neee..dddd.emmedmm http:/c/bbbbbbbee1////// 7 "http://nedicall http://eddaaaeeccaaaaaaa
15

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: The datasets, augmentation, and experimental setup in this study. depth of D means that the volume shape had a varied depth.
MNIST
CIFAR-100
BRATS19
SPLEEN Segmentation
Task
Classification
Classification
Segmentation
Network
LeNet
ResNet-18
U-Net
U-Net
Resolution
28 x 28
32 x 32
240 240 X 155
512 x 512 x D
Type
grayscale
color
MRI
CT
Moddalitics/ # Channels
3
4
Classes/ Maasks
10
100
3
#Samples
60 000
60 000
335
41
Train
42 000
42 000
235
29
Validation
12 000
12 000
67
8
Test
6 0000
6.000
33
4
Augmentation
Flip left-right
X
Rotation
X
X
Scale
X
Shift
Training
#Epochs
50
70
70
70
Optimizer
Adam
Adam
Adam
Adam
Learning rate
1 10-4
1- 10-4
1 10-4
1 10-4
Batch size
128
128
2
Impact on Regularization (Section 6.4). To invostigate the rogularization effect of the proposed mothod, we performed experiments using the WT method with two filters: "Haar" and "Daubechies 3" (Lee el al., 2019) on the four different datasets.
Impact Training Speed (Section 6.5). We investigated the training speed (per-batch training time) when using different compression methods and compression ratios. The experiments were performed all datasets, and only batch normalization layers were compressed.
Impact on Allocated Memory (Section 6.6). The allocated memory on GPU was evaluated on two architectures, ResNet18 and U-Net, using the WT (with "Haar" filters), the DCT and the ST methods. We generated real-time graphs for GPU memory usage by tracking memory allocations each step during forward and backward propagation (see Figure 9a and Figure 9b in the Supplementary Material). To further evaluate the efficient usage of memory with respect to only the activation maps (see Equation 34) of the proposed method, conducted supplemental analysis with numerous settings (different networks, input shapes and batch sizes) and compared the maximum memory occupied on the GPU between methods. In addition to ResNet18 and U-Net, we included 15sNet152, WideResNet101. and ViT 2D/3D. We also compared our method to those of Finder et al. (2022) and Chen et al. (2016), and a 16-bit quantization technique regarding the efficient usage of memory using another of network architectures: EfficientNet (Tan & Le, 2019b), DeepLabV3Phr (Chen et al., 2018) with MobileNet V2 as backbone, and EDSR (Lim et al., 2017) in different. settings.
16

---

Published in Transactions on Machine Learning Research (03/2024)
5.3 Evaluation
We used the MSE and PSNR to evaluate the image quality after reconstruction,
MSE(X, x
x1)² c
(29)
t=1
max X
PSNR(X, X =20 1lg10
(30)
MSE(X, X!)
where logio is the baso-10 logarithm and max(X) the maximum gray level in an image, X. The X is a compressed version of X. The 2, is the i-th element of X (in any order), and N is the number of pixels voxels in the images.
We employed the soft Sørensen-Dice coelficient (DSC) loss in the segmentation tasks (Milletari el al., 2016; Vu et al., 2020a;b; 2021b;a), defined as,
-2E -2E u. +E EN + Ei=1 +c
DDsc(U,
(31)
where u, is the i-th element of a softmax output of the network, v; is the i-th element of onc-hot encoding of the ground truth, and 1 10-5 is small constant added to avoid division by sero. The is present in both the numerator and the denominator to make true zero predicted masks have a score of 1.
Further, we used the categorical cross-entropy (CE) loss for the classification tasks,
N K
CCE(U.V V) = Σ
uf
log(u?)
(32)
where of and of are the i-th elements for class j-th of network output, U, and groundtruth, V, respectively. The K is the number of classes.
In this study, we used accuracy to evaluate the classification tasks and the negative DSC to evaluate the segmentation tasks,
D(U, V) (U, V)..
(33)
We also introduce n, measure of the effective usage of memory with respect to the activation maps,
M MBL x 100%,
1) MBL. -Mo
(34)
where M, MBL. and Mo respectively represent the GPU memory that a method, the baseline, and other categories (model parameters, gradient maps, ephemeral tensor, and resident buffer; Gao et al., 2020) occupy after the forward pass. A smaller y) (in negative value) indicates a more efficient method. In particular, 0% means that no GPU memory in terms of activation maps is saved, and -100% means that the compressed activation maps consume no GPU memory.
5.4 Implementation Details and Training
Our method was implemented in PyTTorch 1.10 The experiments were run on NVIDIA Tesla K80 GPUs. Depending on the model, convergence speed, and dataset size, a single experiment was finished within few minutes or up to several days.
We used the Adam optimizer (Kingma & Ba, 2015) all experiments, with various initial learning rates (see Table in the Supplementary Material). The number of epochs for each experiment was set to 100 for
hnttps://pytorchh.org
17

---

Published in Transactions on Machine Learning Research (03/2024)
the classification task and 80 for the segmentation task. We used small custom version of LeNet (LeCun et al., 1989) and ResNet-18 18 (He et al., 2016) for the MNIST and CTFAR-100 datasets, respectively. Each dataset was split into 70 % for training, 20 % for validation, and 10 % for final testing.
We used the U-Net (Ronneberger et al., 2015) for the BRATS19 and the SPLEEN datasets. Batch normal- ization was used after all convolutional layers the transition block and the main network. We used the ReLU activation function. A transition block of the U-Net consists of a batch normalization layer, a ReLU layer, and a convolution layer in order to adjust (i.e. downsample upsample) the channel size and the feature map size. For the ResNet18, convolutional block included the first convolutional, the first batch normalization, a RcLU, the second convolutional, and the second batch normalization layers. In contrast, a convolutional block in the evaluated U-Net comprised a convolutional, batch normalization, and a ReLU layer. Because there are many layers in the ResNet18 and U-Net, we conducted experiments by compressing only the batch normalization layers in a convolutional block.
Except for the SPLEEN, all datasets were standardized zero-mean and unit variance. Before normalization, the N4ITK bias field correction (Tustison et al., 2010) was applied to the BRATS19 datasets. Samples from the SPLEEN dataset were rescaled to [-1,
To artificially enlarge the SPLEEN and BRATS19 dataset sizes and to increase the variability in the data, we applied on-the-fly data augmentation. Those were: horizontal flipping, rotation within a range of -1 to degrees, rescaling with a factor of 0.9 to 1.1, and shifting the images by -5 percent in both directions.
We employed publicly available PyTorch package, torchvision", for the ResNet18, ResNet152, WideRes- Ne1101, and EfficientNet models. We used a popular implementation of the VIT architectures. For the other architectures, we made use of their official impleementations11122
6 Results and Discussion
6.1 Impact on Memory Footprint
Figure 4 presents the detailed footprint in each step in the proposed method (see Figure 2) using different settings in small and large arrays with input shape of 128 x 16 16 x and 16 x 128 128 128 x 1, respectively.
WT +. db3 (.1)
WT + db3 (A=1.1) (C-11)
DCT (2=111
0.6
db3 10.00.0)
WT + db3 (A=10.0)
Memory (MB)
DCT (A=10.0)
Memory (MB)
250
(A-1000
0.4
200
150
0.2
100
50
x
n
X
X
(a) Tensor size 128 x 16 x 16 x
(b) Tensor size 16 x 128 x 128 128 x
Figure 4: Illusrationn of footprint in each step in the proposed mothod (see Figure 2) over different compression
ratios, X, on two proposed methods: WT with "Daubechies 3" filter (WT + db3) and DCT.
Looking Figure 4, it apparent that the sparse representation in COO format (see II) incurs an overhead that becomes noticeable when the compression ratio is small, i.e. 1.1, in both evaluated arrays. On the
18

---

Published in Transactions on Machine Learning Research (03/2024)
contrary, significant drops in allocated GPU memory be seen from the same figure when À 10. More detailed analyses in terms of À can be found in Section 6.2 and Section 6.6.
The differences between WT and DCT in terms of memory footprint are highlighted in Figure 4. Specifically, the allocated memory on GPU of X, X Y and Y of the DCT method are the same except for II. By way of contrast, X; Y, and of the WT method reserve different amounts memory. These results are related to the shapes of the transformed matrices, Y, after the WT (with padding) and DCT (without padding) are applied.
6.2 Impact on Reconstructed Image
Figure (top) shows the original image and its compressed, thresholded, and decompressed version at different compression ratios using four methods: WT with "Haar" filter (WT + "haar"), WT with "Daubechies 3 filter (WT + "db3"), DCT, and ST. Further, Figure (bottom) compares the MSE, PSNR, size in megabytes (MB) memory, and the percent of memory used by the compressed image over the original image.
WT "dh3"
WT "haor"
DCT
TH
2 1.0
A 1.1
2. 1.4
A 2.5
A 10.0
100.0
2. 1000.0
PSNR (DB)
120
80
MSE
40
Size (Mgabbte)
Percentage
50
25-
2.5
10.0
100.0
1000.0
10
2.5
100
100.0
1000.0
Compression ratio A
Compression ratio A
WT haar
WT + db3
DCT
Figure 5: Quality of reconstructed images over different compression ratios, X, on four proposed methods: WT with "Haar" filter (WT + "haar"), WT with "Daubechies 3" filter (WT + "db3"), DCT, and ST. The = represents the original image (or bascline, i.e. without compression). Top: Illustration of original and uncompressed (or reconstructed) images. Bottom: Comparison of the MSE, PSNR, size in MB stored in the memory, and the percentage computed fraction of the memory consumption of the compressed image over the original image.
19

---

Published in Transactions on Machine Learning Research (03/2024)
From Figure 5, we see that when the compression ratio is smaller than 100, the DCT method outperforms the others in terms of the MSE and PSNR. Looking at Figure 5 (bottom), it is apparent that the size in MB in memory is comparable for all methods at all compression ratios. This result can be explained by the fact that the thresholding technique used in this work is top-k, meaning that a certain amount of non-zero elements are expected to be kept when the same compression ratio is used in all methods.
Another observation emerges from Figure 5 (top) is that the reconstructed images using the WT compression method become mostly black, but not blurry as seen in other works (e.g., Haar, 1909; Zweig, 1976; Mallat, 1989) when the compression ratio increases. There are two possible explanations for that behavior: (1) the thresholding technique used in this work is top-k and (2) we threshold all sub-bands, including the low-frequency component Y low, (see Figure 1).
It can also be seen from Figure 5 (bottom) that when the compression ratio is smaller than 10, the reconstructed image sizes of all methods are larger than the size of the original image. The behavior related to the size can be explained by the fact that the sparse version of the thresholded tensor, in the COO format, is ineffective in terms of memory allocation. A simple solution for the proposed method would be to compare the sparse tensor's s size to the original tensor's size and simply store and use the smallest one. This would climinate the dense and inverse transform steps when there is no benefit of converting to the sparse format and use the reconstructed tensor directly when computing the gradients. This solution saves some computations but does not save any memory.
6.3 Impact on Performance
Figure 6 illustrates the accuracies and their standard errors on the MNIST and CIFAR-100 data and the DSC scores and their standard errors on the SPLEEN and BRATS19 data, for different compression ratios. The title of each sub-figure for the MNIST dataset (Figure 6a) indicates which layer was compressed. For example, "convl" means that only the first convolutional layer was compressed, while "convl-relul" means that the first convolutional and the first ReLU layers were compressod. For the other three datascts, we used "blockO" to indicate that all batch normalization layers in the first convolutional layer were compressed. Similarly, we used blockl" to specify that all batch normalization layers in the first and the second convolutional layers were compressed, and so on.
It is apparent from Figure 6a that the performance of the DCT method on the MNIST dataset is comparable to the baseline's when more than one layer was compressed. Another striking observation to emerge from Figure 6a is that the accuracies on the MNIST dataset go down gradually starting from the compression ratio of 10.
It can be seen in Figure 6b that the accuracies of the DCT and WT methods are on a level with the baseline's 's when the batch normalization layers are compressed in block zero, one, and two. However, the curves seem to decline slowly from block three to block four and drop dramatically when more than three blocks are compressed, specifically from about À = 2.5. There are several possible explanations for these results. First, the resolutions of fcatures maps in the third block and last block of ResNet18 are 256 8 x and 512 x x 4, respectively. Therefore, when À is larger than 2.5, very few details are left causing the network to fail to learn (see "Block3" and "Block4" in Figure 6b). Second, when all blocks are compressed, the information, that is carried from prior blocks ("Block0" to "Block2") might be insufficient to make the last two compressed blocks ("Block3" and "Block4") extract relevant features. In other words, this may be explained by collective information loss.
What is interesting in Figure 6c and Figure 6d that the DSC scores of the DCT methods are on par with the baseline's when the compression ratio is less than or equal to 100. It seems that X 100 is a delicate threshold to obtain satisfactory performance for the DCT method.
20

---

Published in Transactions on Machine Learning Research (03/2024)
conv
conv2
mp
convi-relu1
66
relu1-conv2
conw2-rebl--oomz
conv1-18U1-conv22
2002
WT + haar
WT
db3
OCT
ST.ST
MNIST
Blocko
block]
block2
back3
block4
75.2
blocko-1
blocki-2
block0-2
block--3
blocko-4
75.0
Add
hear
db3
DCT
CIFA
2.99
3.00
block1-2
block2-3
block0-2
block1-3
blocko-3
2.9
2.00
haar
db3
+ DCI S1
(c)
SPLEEN
blockU
block1
block2
bickk
blocko-1
2.05
3.75
block1-2
block2-3
block0-2
block1-J
blocko-3
5.5
6.75
2.55
18.
WT haar WT db3 DCT
(d) BRATS19
Figure 6: Performance over different compression ratios on four different datasets using four proposed methods: WT with "Haar" filter (WT "haar"), WT with "Daubechies 3" filter (WT "db3"), DCT, and ST. The z-axis shows the compression ratio, X. The "mp" stands for max pooling layer in the MNIST. In the MNIST, "convl", "relu1", and so on, show which layer is compressed. For the others, "blockO", "block1", and so on indicates which convolutional block the batch normalization layers are compressed.
21

---

Published in Transactions on Machine Learning Research (03/2024)
6.4 Impact as Regularization
Figure ralustrates the regularization effect of the proposed method and shows the accuracies and DSC scores and their standard errors. The title of each sub-figure in Figure 7 denotes the compressed layer. The "convl" is the first convolutional layer of the network, the "conv2" is the second convolutional layer, and so on.
conv1
relu1
conv2
conv1
bn1
relul
Accuracy
100
Accuracy,
25.0
on
62.5
100.0 1600
300.0
10.0
100.0
1010
100
1000
100
1000
1000.0
25
10.0
100.
18000
WT + haar
WT + db3
WT haar
WT + db3
(a) MNIST
(b) CIFAR-100
conv1
bnl
conv2
conv1
bn1
conv2
1.00
DSC
DSC
0.95
0.00
1009
100
100.0
10.0
100.0
0.85
10.0
10.0
100.0
1802.0
WT + haar -. WT db3
WT haar WT db3
(c) SPLEEN
(d) BRATS19
Figure 7: Illustration of regularizing effect. Performance over different compression ratios on four different datasets using two proposed methods: WT with "Haar" filter (WT + "haar") and WT with "Daubechies 3 filter (WT + "db3".. The x-axis shows the compression ratio, A.
From Figure 7, we see that: (1) the accuracy on the CIFAR-100 and the DSC on the BRATS19 remain unchanged when the compression ratios increase on all evaluated compressed layers, and (2) the accuracy on the MNIST and the DSC on the SPLEEN fluctuate when the compression ratios increase.
6.5 Impact on Training Speed
Figure 8 illustrates the training time per batch (in seconds) on the four datasets for different compression ratios in the ResNet18 and U-Net. In all cases, we can again see that a compression ratio of 10 might be a good value for the WT methods, because the batch durations did not increase much compared to the baseline.
Figure 8 illustrates a slight decrease in the batch duration in all cases when the compression ratio increases. Additionally, it is clear that the DCT method is computationally more expensive than the other methods. This is because the DCT method splits an image into a large number of 16 x 16 blocks, and processes each block separately, making the computation inefficient. An alternative could be to perform the DCT on the whole image instead of splitting the image into blocks and applying the DCT on them.
As can be seen in Figure Sa, there is a considerable variance in batch duration when different layers are compressed on the MNIST dataset when the WT and ST methods are used. Specifically, the batch durations when we compress the first convolutional ("convl") or max pooling layers ("mp") are on the same level the baseline. The batch durations jump dramatically for the other cases. For the CIFAR-100 100 dataset that can be seen in Figure 8b, the batch duration increases slightly for the WT and ST methods when block zero ("block0") is compressed.
As shown in Figure Sc and Figure 8d, there is only a small difference in the batch durations when comparing the bascline the WT and ST methods at different compression ratios. We can also see from Figure &c and Figure 8d that the batch duration when batch normalization layers are compressed in the first convolutional block ("block0") is 14% longer than that in the third convolutional block ("block2").
22

---

Published in Transactions on Machine Learning Research (03/2024)
conv2
mp
convi-telu1
Duration
0.3
-
relu1-conv2-mp
cow--neluu-ccnn22-
Duration
0.0
WT hear
WT db3
DCT
ST
(a) MNIST, LeNet,
BS 128, Input shape 28 28
blocko
block1
block2
block3
block4
+
blockD-3
block3-2
block0-2
block0.3
block0-4
Duration
WT hear WT
DCT
(b) CIFAR- R-100, ResNet18, BS 128,
Input shape
32 x 32 X
BlockO
block1
blozk2
block3
block0-1
4.5
bbck1-2
block2-3
block-2
bocki-3
blck0-3
And aun
(c) BRATS19, U-Net, BS 2, Input shape 96 x 96 x 96 x 1
Blocko
block1
block2
block3
block0-1
bboc112
Bock2-3
block--2
block-3
Duration.o
2.2
Jana
WT haar WT db3
OCT
ST
(d) SPLEEN, U-Net, BS Input shape x 96 96 x
Figure Batch duration over different compression ratios on four different datasets using four proposed methods: WT with "Haar" filter (WT "haar"), WT with "Daubechies 3" filter (WT + "db3"), DCT, and ST. The a-axis shows the compression ratio, X. "BS" stands for batch size. The "mp" stands for max pooling layer in the MNIST. In the MNIST, "convl", "relul", and on, show which layer is compressed. For the others, "blockO", "block1", and so on indicates which convolutional block the batch normalization layers are compressed.
23

---

Published in Transactions on Machine Learning Research (03/2024)
ST
ST
ST
1000
1000
1005
Memory (KB)
Memory (MBB)
1-11
2-22
520
Memory
520
500
1-33923
2003
1333000
Inguu
Ourput
Inpuc
Innput
Oulput
Input
Inout
Outout
Input
DCT
DCT
DCT
Momeny (110)
1-10
Meeocry (MR)
Meeoor (MB)
330
--233
Imput
Dulpat
Inpat
+ haar
+ haar
Memory (MB)
1000
1-10
Memory (MS)
1000
Memory (MB)
1000
3.00
300
25
300
2300
3933
1-1832.
Inpul
Ourput
Input
Irpput
Dulput
Input
Cutput
Input
(a) ResNet18
ST
ST
2000
Memory (MD)
A-100
Menory (MS)
Memory (MBS
A-1.
1000
2000
A-252
2004
Imput
Ourput
Input
Input
OuipuE
Input
Cutput
Input
DCT
DCT
DCT
20:00
20:0
2000
Nemory (MB)
Mumory (MB)
Meroery (Mbb)
10:00
10:00
1200
2206
1-18913
1999
39900
Input
Output
Inputt
Vriput
Dulppl
Input
Input
Culput
Input
WT + haar
WT haar
WT + haar
2000
2000
Memory (NB)
A-10
Mermary (MB)
Memorry (MB)
3,000
1-33000
Irppt
Input
Inguut
Input
Input
(b) U-Net
Figure 9: Illustration of the memory allocation at each step (I-axis) during the forward and backward pass on (a) ResNet18 and (b) U-Net at the compression ratio of 10. The r-axis shows the compression ratio, A. The input shape and batch size of ResNet18 are 32 x 32 3 and 128, respectively. The input shape and batch size of U-Net are 96 x 96 x 96 x 3 and 1, respectively. The vertical dashed line denotes where the forward-propagatioo ends and the back-propagation begins. Left: batch normalization layers in the first convolutional block are compressed. Middle: batch normalization layers in the first two convolutional blocks are compressed. Right: convolutional layers, batch normalization layers, and ReLU layers in the first two convolutional blocks are compressed.
6.6 Impact on Allocated Memory
Figure 9a and Figure 9b illustrate the memory allocation in MB at each step during the forward followed by the backward pass on ResNet18 and U-Net, respectively, with 10 using the DCT, the WT with "Haar" filter and the ST methods. In the first two columns in both figures, the "first convolutional block (BN)" and "first two convolutional blocks (BN)" show at which block the batch normalization layers compressed. For the last column, "first two convolutional blocks (Conv. + BN + ReLU)" indicates that three layers, including convolutional, batch normalization, and ReLU, are compressed in the first two convolutional blocks. The vertical dashed line shows where the forward-propagation ends and where the back-propagation begins.
As shown in both Figure 9a and Figure 9b, the GPU memory usage is higher than the bascline when the compression ratio is smaller than 10 for the WT and ST methods and for both the ResNet18 and U-Net. An explanation for this is that very little information is removed when the compression ratio is small, for
24

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Illustration of memory allocated (in MB) on the GPU when DCT is used with compression ratio 100. "Comp. layers" denote which layers or modules were compressed. The "block0" denotes that activation maps were compressed at the first convolutional block, the "block0-1" denotes activation maps compressed at the first two convolutional blocks, and so on. The "BN", "Conv." and "ReLU" denote batch normalization, convolutional, and ReLU layers, respectively. The " "Attention" and "Feedforward" are modules introduced in ViT. "Others" indicates the GPU memory occupied by all categories except for the activation maps. The "BL." shows the GPU memory occupied by all categories (including the activation maps) on the baseline (without layers compressed). The % values denote the memory efficiency, 13, in negative percentage with respect to only the activation maps (a lower negative value better). ResNet152 and 1esNet152 denote two versions of ResNet152 that use BasicBlock (as in ResNet18) and BottleneckBlock, respectively.
Memory (In MB)
Network
Comp. layers
Others (Mo)
Beseline (MaL)
DCT + A= 100 (M)
block0
block0
block0-2
block0-3
blodk0--4 -4
ResNet18 Batch size 128 Res. 32 x 32 x 3
BN
586 (-6%)
462 (-32%)
400 (-15%)
369 (-51%)
353 (-55%)
Couv, + BN
133
617
586 (-6%%)
197 (-25%)
451 (-31%)
431 (-38%)
418 (-11%)
Conv. BN ReL.U
595 (-5%)
425 (-40%)
360 (-53%)
327 (-00%)
311 (-63%)
ResNet18 Batch size 512
BN
2213 (-6%)
1726 (-32%)
1482 (-14%)
1380 (-51%)
1298 (-5-1%)
Couv, + BN Conv. BN ReLU
402
2338
2213 (-65%)
1867 (-2198)
1698 (-33%)
1610 (-38%)
1563 (-10%)
Res. 32 x 32 x 3
2252 (-4%)
1575 (-10%)
1321 (-53%)
1196 (-50%)
1133 (-62%)
ResNet18 Batch size 128
BN
2214 (-6%)
1727 (-32%)
1483 (-14%)
1361 (-50%)
1299 (-5-1%)
Conv. + BN
402
2338
2214 (-6%)
1869 (-24%)
1701 (-33%)
1615 (-37%)
1565 (-40%)
Res. 64 x 64 x
Conv. BN ReL.U
2253 (-45)
1576 (-30%)
1323 (-52%)
1190 (-50%)
1136 (-62%)
U-Net Batch size
1206 (-36%)
1102 (-45%)
1076 (-47%)
1089 (-48%)
0..
Conv. + BN
458
1630
1004 (-53%)
636 (-08%)
793 (-71%)
782 (-72%)
0..0
Res. 96 96 96
Couv. ReLU
(-71%6
(-89%)
527 (-04%)
514 (-95%)
U-Net Batch size Res. 96 96 96
2321 (-36%)
2112 (-459)
2060 (-47%)
2046 (-48%)
Conv. BN
827
3158
1916 (-5394)
1579 (-6896)
1491 (-71%)
(-72%)
Couv. ReLU
1509 (-71%)
1072 (-80%)
9622 (-04%)
934 (-95%)
U-Net Batch size Res. 128 x 128 128
2732 (-36%)
2482 (-459)
(-47%)
2407 (-48%)
Couv. BN
962
3724
2250 (-5:54)
1853 (-6896)
1753 (-71%)
(-72%)
Couv. ReLU
1771 (-7196
1251 (-9096
1120 (-04%)
1088 (-95%)
0.a
155
2382 (-1%)
2196 (-10%)
1947 (-22%)
1389 (-49%%
1385 (-50%)
Batch size 128
Couv. BN
312
2113
2382 (-154)
2201 (-1096)
1961 (-22%)
(-47%)
1398 (-18%)
x x 3
Couv. ReLU
2383 (-156)
2016 (-199)
1544 (-41%)
452 (-90%)
410 (-95%)
ResNes152 Batch 128
6056 (0%))
6399 (-095)
5609 (-22%)
3909 (-49%)
3827 (-51%)
Conv. BN
738
6287
6956 (U94)
6414 (-996)
5650 (-21%)
(-18%)
3926 (-19%)
Res. 32% x x 3
Conv. BN- ReLU
6957 (0%)
5986 (-10%)
4492 (-10%)
1179 (-93%)
1036 (-96%)
WideRosNNe100 Batch size 128
4880 (-33%)
4522 (-40%)
4204 (-43%)
3811 (-52%)
3809 (-51%)
Conv. BN
1000
6824
4880 (-3354)
4542 (-3996)
4334 (-12%)
3899 (-50%)
3980 (-19%)
Res, 32 x x 3
Conv. BN ReLU
4880 (-38%)
3526 (-67%)
3192 (-62%)
1314 (-93%)
1276 (-96%)
VII Batch 256 Res, 256 250
Asteution + Feedforward
467
6299
5331 (-17%)
4378 (-33%)
3427 (-19%)
2479 (-68%)
1582 (-82%)
VII Batch size 32
Asteation + Feedforward
631
6333
5973 (-6%)
5601 (-18%)
4235 (-37%)
2867 (-61%)
1498 (-85%)
Res, 16 128 128
instance, when À = 1.1. Thus, the overhead of the sparse representation (in COO format) begins to show in this case, as the tensors are not actually particularly sparse.
From Figure 9a, we see that when batch normalization layers are compressed in only the first block ("block0")) (left. column of Figure 9a), the GPU usage for the activation maps decreases by only 5%. There is a substantial drop (about 25%) in GPU memory usage at a compression ratio of 10 when batch normalization layers are compressod in the first two convolutional blocks (middle column of Figure 9a). The GPU memory usage plummets by 35% at a compression ratio of 10 when three layers, including convolutional, batch normalization, and ReLU, are compressed in the first two convolutional blocks (right column of Figure 9a).
25

---

Published in Transactions on Machine Learning Research (03/2024)
It can be seen from Figure 9b that about 30% of the GPU memory usage is saved when batch normalization layers are compressed in the first convolutional block (left column) and the first two convolutional blocks (right column) in U-Net with the compression ratio of 10. Again, the GPU memory use falls significantly (about 80%) at a compression ratio of 10 when three layers, inchuding convolutional, batch normalization, and ReLU, are compressed in the first two convolutional blocks (right column).
Table presents the memory allocated (in MB) on the GPU using DCT at compression ratio of 100 when different layers ("Comp. layers") are compressed at different convolutional blocks. The "Block0" represents evaluated layers' activation maps that are compressed at the first convolutional block, the "Block0-1" represents evaluated layers' activation maps that are compressed at the first two convolutional blocks, and so on. The "BN", "Conv.", ", and "ReLU" denote batch normalization, convolutional, and ReLU layers, respectively. The "Other" indicates the GPU memory occupied by all categories except for the activation maps. The "BL" shows the GPU memory occupied by all categories (including the activation maps) on the baseline (without layers compressed). The percent values denote the memory efficiency, 7). in pereentage with respect to only the activation maps (see Equation 34).
It can be seen from Table 2 that there are remarkable rises in GPU memory efficiency when layers' activation maps (for all cases) are compressed in the first convolutional ("Block0") compared to in the first two convolutional blocks ("Block0-1"), from 5% to around 40% in ResNet18 and from 71% to 89% in U-Net. The jumps are insignificant when layers' activation maps are compressed from the third convolutional block forward ("BlockO 2", "BlockO 3", and "BlockO 4"). Compared to 1e and U-Net, Table 2 reveals that there have been significant rises in GPU memory efficiency when layers' activation maps are compressed from the third convolutional block forward ("BlockO-3") in ResNet152* (from 41% to 93%), 5esNet152 (from 40% to 93%) and WideResNet101 (from 62% to 95%). In contrast, the figures are proportional to the number of compressed blocks in ViT architectures.
From Table 2, we can see that, with respect to the GPU memory occupied by only activation maps, the proposed method using DCT saves about 40% when convolutional, batch normalization, and ReLU layers are compressed in the first two convolutional blocks in the ResNet18. The figures then rise to 62 63% when these layers are compressed in all convolutional blocks in all evaluated settings of ResNet18. The figures for the U-Net are even higher, going from 36 to 48%, from 53 to 72%, and from 71 to 95% when "BN", "Conv. + BN" and "Conv. + BN + ReLU" layers are compressed, respectively. ResNet152* 1esNet152, and WideResNNe101 exhibit the same patterns as U-Net. However, when the "BlockO" is compressed, only 0-1% of the GPU memory occupied by activation maps is saved by two ResNet152 variants. This result may be attributed to the significantly smaller number of layers in the first block of these architectures compared to the whole network.
As shown in Table 2, the proposed compression method using DCT is more memory-efficient in the higher dimensional activation maps when comparing 1) in evaluated architectures. For instance, the proposed method saves GPU memory usage by about 5-6% in ResNet18, 36-71% in U-Net, around 1% in two ResNet variants, and 33% in WideResNet101 when evaluating layers of the first convolutional block. The figures increase from 38 to 60% in ResNet18, from 48 to 95% in U-Net, and from about 52% to 94% in larger models (ResNet152 and WideResNNet01) when evaluating layers of the first four convolutional blocks.
What is surprising from Table 2 is that compressing more layers results in more GPU memory saving for the case of U-Net, but not for the ResNet. family (ResNet ResNet1 15s and WeleResNee101). In particular, compressing convolutional and batch normalization layers ("Conv. + BN") consumes more memory than compressing only batch normalization layers ("BN"), but compressing three layers ("Conv. + BN + ReLU") reduces the memory consumption. The reason for this unexpected behavior is likely due to the presence of addition operation in ResNet block that prevents the original tensor from being released from the GPU f.e. PyTorch stores both original and compressed tensors on the GPU.
Table 3 compares the memory allocated on the GPU using different methods with different parameters. Instead of employing 32-bit in feature maps and network parameters as in other methods, we analyze the quantization technique with 16-bit. In the "Checkpoint" method (Chen et al., 2016), stands for the nunber of segments. In WCC, 4-bit weights and 8-bit activations are selected, and w denotes the compression rate. In this table, we experiment with our method with different compression ratios, A.
26

---

Published in Transactions on Machine Learning Research (03/2024)
Table 3: Illustration of memory allocated (in MB) on the GPU when different methods are used. The Mo indicates the GPU memory occupied by all categories except for the activation maps. The MBL shows the GPU memory occupied by all categories (including the activation maps) on the baseline (without layers compressed). The M shows the GPU memory that a method occupy. The % values denote the memory efficiency, 7, in negative percentage with respect only the activation maps (a lower negative value is better). In "Checkpoint" " method, stands for the number of segments. The w denoles the compression rale, while 4-bit weights and 8-bit activations are selected in WCC.
Effecienteet B0
EfficientNet B4
EfficientNet B8
Batch size 4
Batch size 2
Batch size 1
Contiguration
Res. 640 640 3
Res. 6410 x 640 x
Ros. 6400 640 x
.Mo
.MB..
M(n)
Moo
.Mar.
M((1)
.Mo
MBL
M(n)
Quantization
16 bit
20
1017
25
939
40
888
1196 (-42%)
1130 (-41%)
1098 (-40%)
2
310 (-86%)
322 (-85%)
345 (-84%)
Checkpoint (Chen et al., 2016]
2023
(-90%)
1871
185 (-93%)
79
1781
216 (-92%)
140 (-95%)
151 (-94%)
177 (-94%)
5
117 (-96%)
130 (-96%)
161 (-95%)
None
7120 (257%)
6580 (258%)
6179 (258%)
50%
5629 (182%)
5121 (178%)
4843 (180%)
WCC (Finder et al., 2022)
25%
39
2023
4876 (1:4%%)
49
1871
4418 (1-10%)
79
1781
4193 (142%)
12.5%
4493 (124%)
4051 (120%)
3834 (121%)
w 6.25%
4306 (115%)
3862 (1099))
3645 (110%)
1.1
7427 (272%)
6805 (271%)
6387 (271%)
14
5786 (190%)
5314 (189%)
5013 (190%)
2.5
3327 (66%)
3075 (66%)
2973 (70%)
Ours (DCT)
39
2023
49
1871
79
1781
A = 10
872 (-58%)
806 (-58%)
786 (-58%)
A 100
121 (-96%)
124 (-96%)
149 (-96%)
A = 1000
48 (-100%)
57 (-100%)
87 (-100%)
DrepLabV3PPss + MobileNe+V2
EDSR Batch size
Batch size
Configuration
Res. 128 128 3
Res. 48 x 48
Mo
MR..
M(n)
Mo
Mar..
M(x))
Quantization
16-
160
1878
7..
487
n.a
2683
(-2%)
941 (0%)
Checkpoint (Chen et al., 2016)
2
7L.
644 (-38%)
3
2722
n.
166
943
536 (-52%)
4
491 (-58%)
5
71.
455 (-63%)
None
8295 (232%)
3288 (302%)
50%
6512 (158%)
3280 (301%)
WCC (Finder et al,, 2022)
25%
318
2722
5612 (120%)
166
943
3277 (300%)
w 12.5%
5165 (102%)
3275 (300%))
6.25%
4929 (92%)
3274 (300%)
1.1
6296 (149%)
2781 (237%)
X = 1.4
5307 (108%)
2400 (188%)
2.5
3209 (20%)
1443 (64%)
Ours (DCT)
318
2722
166
943
X 10
1053 (-69%)
488 (-59%)
> 100
390 (-97%)
199 (-96%)
= 1000
325 (-100%)
170 (-99%)
27

---

Published in Transactions on Machine Learning Research (03/2024)
From Table 3, we see the following. First, the 16-bit quantization technique reduces the allocated memory by half of that as in the default configuration i.e. 32-bit. For example, MBL. is 2023 and 1017 in the EfficientNet BO when 32-bit and 16-bit are used, respectively. Second, the "Checkpoint" " method decreases the GPU usage by large margins in all models except for the combination of DeepLabV3Plus and MobileNet where the sole choice for the number of segments, 8, is 1. A possible explanation for this might partly be explained by the fact that the official PyTorch implementation for this method considers MobileNet V2, which based on inverted residual blocks, as a single module. Third, the WCC consumes more GPU memory than the baseline in all cases. These results can be explained by the fact that WCC trades GPU memory for computational cost (Finder et al., 2022). Last, in our method, the figures are positive when À are smaller than 2.5, but become negative afterward. A possible explanation was mentioned earlier that very little is saved when the compression ratio is small, because of the overhead in the sparse representation.
The "Checkpoint" method optimizes GPU memory usage and works well on most evaluated architectures. However, it has a major weakness with architectures based on skip connections (e.g., U-Net) or residual blocks (e.g., ResNet family). To be precise, the "Checkpoint" method fails easily if the feature maps that are checkpointed lie in the middle of residual blocks. In contrast, the proposed method tackles this challenge comfortably and naturally.
7
Conclusion
We have presented a novel method for intermediate feature map compression in DNNs that reduces the memory footprint on a GPU, The study was performed on classification and semantic segmentation tasks. We evaluated compression based on the WT, the DCT, and ST method. We demonstrated that the proposed method reduces the activation map memory usage by large margins (by up to 95%) while maintaining the performance compared to the baseline, and especially so when using DCT compression. In addition, we showed in theory that the proposed activation compression method induces a gradient penalty-a constraint on the expected squared Frobenius norm of the compressed layer's 's weight gradients. More experiments are needed for in-depth characterization of the regularization effect.
The evidence reviewed here suggests that top-k is not an adequate approach to threshold sub-bands decomposed using the WT since its performances are less stable and lower compared to when using the DCT. Hence, further work could determine more effective thresholding techniques for compression with the WT. Future work could also include determining the thresholds or the transforms adaptively, f.e. automatically finding optimal ones for a particular dataset.
Though the proposed compression method is able to save much GPU memory terms of activation maps, the sparse tensor technique appears to have a large overhead. Future research could include other compression methods, such as Fourier transform and Huffman coding, with the purpose of not only saving even more GPU memory but also speeding up the training. Another research question that could be asked includes comparing different sparse representations.
References
Nasir Ahmed, T. Natarajan, and Kamisetty Rao. Discrete Cosine Transform. IEEE Trunsactions on Computers, C-23(1):90 93, 1974.
Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder. Fused-layer CNN accelerators. In 2016 49th Annual IEEE/ACE AAC International Symposium Microarchätecture (MICRO), pp. 1 12. IEEE, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NewrIPS), volume 29. Curran Associates, Inc., 2016.
Spyridon Bakas, Hamed Akbari, Aristcidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data, 4:1701 17, 2017.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfter, Alessandro Crimi, Rus- sell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge. Preprint, arXiv:1811.0622) 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzi. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541. ACM, 2006.
Liang-Chieh Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 801 818, 2018.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. Preprinl, Xix:1604.06111 2016.
William Dally, High-performance hardware for machine learning. NIPS Tutorial, 2015.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey. Proceedings of the IEEE, 108(4):485-533 2020.
Li Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine, 29(6):141 142, 2012.
Peter Deutsch and Jcan-Loup Gailly. ZLIB Compressed Data Format Specification version 3.3. Technical report, 1996.
Guncet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy D Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic Activation Pruning for Robust Adversarial Defense. In International Conference on Learning Representations, 2018.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated notwork with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 5840 5848, 2017.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient Scaling of Language Models with ointurre-f-Exxeerts. In International Conference Machine Learning, pp. 5547 5569. PMLR, 2022.
Shahaf E Finder, Yair Zohav, Maor Ashkenazi, and Eran Treister. Wavelet Feature Maps Compression for Image-to-Imape CNNs. Advances in Neural Information Processing Systems, 35, 2022.
Yanjie Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang, Estimating gpu memory consumption of deep learning models. In Proceedings of the 28th ACM Joint Meeling on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1342-1352, 2020.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. In 14th International Conference on Artificial Intelligence and Statistics, volume 15, pp. 315-323, 2011.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Denis Gudovskiy, Alec Hodgkinson, and Luca Rigazio. DNN Feature Map Compression Using Learned Representation over GF (2). In Furopenn Conference on Computer Vision, pp. 502-516. Springer, 2018.
29

---

Published in Transactions on Machine Learning Research (03/2024)
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5769 5779, 2017.
Philipp Gysel, Jon Pimentel, Mohammad Motamedi, and Soheil Ghiasi. Ristretto: A framework for empirical study of resource-efficiet inference in Convolutional Neural Networks. IEEE transactions on newral networks and learning systems, (9(1))55784 5789, 2018
Alfred Haar. On the Theory of Orthogonal Function Systems. Pundamental Papers in Wavelet Theory, pp- 155 188, 1909.
Masafumi Hagiwara. Removal of hidden units and weights for back propagation networks. In Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan), volume 1, pp. -344. IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
GeofTrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. Preprint, arXiv:1503. 02531 [s]at.ML], 2015.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint Xr:::19088.021 02729, 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adamn MobileNets: Efficient. Convolutional Neural Networks for Mobile Vision Applications, 2017.
Yiming Hu, Siyang Sun, Jianquan Li, Xingang Wang, and Qingyi Gu. A novel channel pruning method for deep neural network compression. Preprint, arXiv: 10:1805.11 1.05.1139/ 11394, 2018.
Itay Hubara, Matthicu Courbariaux, Daniel Soudry. Ran EI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-668) -8988, 2017.
David Huffman. A Method for the Construction of Minimun-Reduudaancy Codes. Proceedings of the IRE, 9 (40):1098-1101, 1952.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. Preprint, arXiw:1602.07333 2016.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
Daniel Jakubovitz and Raja Giryes. Improving DNN Robustness to Adversarial Attacks Using Jacobian Regularization, In Computer Vision-ECCV ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XII 15, pp. 525-541. Springer, 2018.
Konstantinos Kamnitsas, Christian Ledig, Virginia FJ Newcombe, Joanna P. Simpson, Andrew D Kane, David K Menon, Daniel Rueckert, and Ben Glocker. Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation. Medical Image Analysis, 36:61-78 78, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 7th. International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
30

---

Published in Transactions on Machine Learning Research (03/2024)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1097-1106 2012.
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropa- gation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541-551, 12 1989.
Gregory R. Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron Leary. Py Waweeeess: Python package for wavelet analysis. Journal of Open Source Software, 4(36):1237, 2019.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. In International Conference on Learning Representations, 2018.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec. Enhanced Deep Residual Networks for Single Image Super-Resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132-1140. IEEE Computer Society, 2017.
Stephane G Mallat. theory for multiresolution signal decomposition: The wavelet representation. IEFE Trans. Pattern Annalysis and Machine Intell, 11(7):647 693, 1989.
Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10):1993-2000 2014.
Stephen Merity, Bryan McCann, and Richard Socher. Revisiting Activation Regularization for Language RNNs. Preprint, arxiv: 708. 01009, 2017.
F. Milletari, N. Navab, and S. Ahmadi. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In Fourth International Conference on 3D Vision (3DV), pp. 565-571, 2016.
Rahul Mishra, Hari Prabhat Gupta, and Tanima Dutta. A Survey Deep Neural Network Compression: Challenges, Overview, and Solutions. Preprint, or #rXx:201000095/, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-nct: Imagenct classification using binary convolutional neural networks. In European Conference on Compuler Vision, pp. 525 542. Springer, 2016.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779 788, 2016.
Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Fffficiet Neural Network Design. In 2016 49th Annual IEEE/ACM International Symposium on Microurchitecture (MICRO), pp. 13. IEEE, 2016.
Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Kecklor. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 78 91. IEEE, 2018.
Jorma Rissanen and Glen G Langdon. Arithmetic Coding, IBM Journal of Research and Development, 23(2): 162, 1979.
A Harry Robinson and Colin Cherry. Results of a prototype television bandwidth compression scheme. Proceedings of the IEEE, 55(3):356-336 1967.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomodical Image Segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 234-241. Springer, 2015.
31

---

Published in Transactions on Machine Learning Research (03/2024)
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back- mack-propagating errors. Nature, 323(6088):533 536, 1986.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Largo-Scale Image Recognition. Tn International Conference on Learning Representations, 2015.
Amber I. Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. Preprint, arXiv:192..000 09063, 2019.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Tronsactions on Signal Processing, 65(1)):4265 12422266--800,, 2017.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhuldinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning, 2019a.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks 11946, arXiv preprint arxiv:100s 1195,1661 2019b.
Nicholas J Tustison, Brian B Avants, Philip A Cook, Yuanjie Zheng, Alexander Egan, Paul A Yushkevich, and James C Gee. N4ITK: Improved N3 Bias Correction. IEEE Transactions on Medical Imaging, 29(6): 1310, 2010.
Minh H Vu, Guus Grimbergen, Tufve Nyholm, and Tommy Löfstedt. Evaluation of Multi-Slice Inputs Convolutional Neural Networks for Medical Image Segmentation. Medical Physics, 47:6216-622 2020a.
Minh H Vu, Tufve Nyholm, and Tommy Löfstedt. TuNet: End-to-end Hierarchical Brain Tumor Segmentation using Cascaded Networks. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Troumatic Brain Injuries, volume 11992, pp. 174 186. Springer, Cham, 2020b.
Minh H Vu, Gabriella Norman, Tufve Nyholm, and Tommy Löfstedt. A Data-Adaptive Loss Function for Incomplete Data and Incremental Learning in Semantic Image Segmentation. IEEE Transuctions on Medicol Imaging, 41(6):1320 1330, 2021a.
Minh H Vu, Tulve Nyholm, and Tommy Löfstedt. Multi-Decoder Networks with Multi-Denoising Inputs for Tumor Segmentation. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Broin Injuries, volume 12658, pp. 412 423. Springer, Cham, 2021b.
Peisong Wang and Jian Cheng, Accelerating Convolutional Neural Networks for Mobile Applications. In Proceedings of the 24th ACM International Conference on Multimedia, pp. 541 545, 2016.
Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition. In Interspeech, pp. 2365-2369, 2013.
Jian Xue, Jinyu Li, Dong Yu, Mike Seltzer, and Yifan Gong. Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network. In 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6359 6363, 2014.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Lin, and Larry S Davis. NISP: Pruning Networks using Neuron Importance Score Propagation. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 9194-9203, 2018.
Xiaohua Zhai, Alexander Kolesníkov, Neil Houlsby, and Lucas Beyer Scaling Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
32

---

Published in Transactions on Machine Learning Research (03/2024)
Zhifei Zhang. Derivation of Backpropagation in Convolutional Neural Network (CNN). University of 33
Tennessee, Knorville, TN, 22:23, 2016.
George Zweig. Basilar Membrane Motion. In Cold Spring Harbor Symposia on Quantitative Biology, volume 40, pp. 619 119-633. Cold Spring Harbor Laboratory Press, 1976.


Reviews:
[{"review_id": "Hs7TlI88rd", "reviewer": "Reviewer_gCkT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+The proposed method for compressing high-dimensional activation maps addresses a significant challenge in deep learning - the large memory consumption of contemporary deep learning architectures, particularly in GPUs​​.\n\n+The paper not only introduces a method for compression but also demonstrates that this method induces a regularization effect on the layer weight gradients. This dual effect (compression and regularization) adds significant value to the approach​​.\n\n+The regularizing effect of the proposed method helps in reducing overfitting, which is a common challenge in deep learning models, especially those with large numbers of parameters​​.\n\n+ The paper gives a good overview of related work in the field, providing context for the significance and novelty of its contribution​​.\n\nWeaknesses\n-While the paper outlines the method's theoretical basis, the practical challenges of implementing this approach in diverse real-world scenarios might be underrepresented. The complexity of the method, especially in terms of the mathematical formulations and assumptions involved, could pose implementation challenges​​. Can the authors comment on how their infrastructure is pytorch or tensorflow compatible? Does it add any overhead to the training?\n\n- While the paper discusses the regularization effect, it could explore more deeply how the compression affects the overall performance of the neural network, especially in terms of accuracy and loss, beyond just the reduction in memory usage.\n\n- The evaluation, though comprehensive, seems to focus more on memory reduction and regularizing effects. An expanded evaluation including a wider range of neural network architectures and applications could provide a more rounded assessment of the method's versatility and limitations.\n\n- The effectiveness of the method in various hardware configurations is not deeply explored. Given that the activation map compression is particularly relevant for GPU memory optimization, its performance across different GPU architectures and configurations would be valuable.\n\n-A more detailed comparative analysis with existing methods, particularly in terms of trade-offs between compression, performance, and training efficiency, would provide a clearer understanding of the method's relative advantages and disadvantages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents anadvancement in addressing the memory consumption issues in deep learning, particularly for GPU-based architectures. Its strengths lie in its innovative approach and comprehensive testing, but there are areas where further research and elaboration could strengthen its contributions, particularly regarding practical implementation and performance implications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Look at weaknesses", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nGMQS3aEzu", "reviewer": "Reviewer_MXoD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper effectively underscores the importance of the problem, providing quantitative data on the memory requirements of different deep neural network models.\n• It highlights the necessity for compressing activation maps in deep neural networks to address their overparameterization.\n• The paper thoroughly discusses memory requirements at different stages of the training process.\n• The work is grounded in solid theoretical foundations, particularly Lemma 1 and Theorem 1.\n• The experimental results are generally satisfactory, and the authors conduct a comprehensive ablation study to assess the impact of their compression method on various aspects, including training accuracy, image reconstruction, GPU memory allocation, training speed, and to some extent, regularization.\n\nWeaknesses:\n\nThe paper should clarify the distinction and purpose of Corollary 5, 6, and 7 compared to Corollary 2, 3, and 4.\n• It should provide a more comprehensive discussion of how their method differs from and outperforms existing lossless compression techniques mentioned in related work.\n• The handling of the parameter λ needs more explanation, especially in cases with a significant number of high or low values in the array. The paper could provide examples to illustrate this issue.\n• The use of thresholding techniques and the relevance of Assumption 1 should be more convincingly justified.\n• The paper lacks insights into the applicability of the proposed method to complex architectures beyond ResNet-18 and U-NET. It should explore the potential impact on model performance and GPU memory allocation for various modern architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of dealing with large-scale model parameters that cannot fit into standard GPUs.\n• The authors propose an innovative approach to compress high-dimensional activation maps, which are the most memory-intensive components in modern deep learning model training.\n• The proposed method involves saving GPU memory by storing compressed activation maps during forward propagation and using their corresponding decompressed versions during backward propagation for gradient computation.\n• Three compression techniques, namely Wavelet Transform, Discrete Cosine Transform, and Simple Thresholding, are employed.\n• The authors conducted experiments on two classification tasks and two semantic segmentation tasks, primarily using ResNet-18 and U-NET architectures across various compression ratios and model layers, achieving up to a 95% reduction in memory usage.\n• The authors also claim that their approach induces a regularization effect on layer weight gradients.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper should explain why the compressed and thresholded variables are not used in back-propagation and what challenges this may pose.\n• Instead of invoking Lemma 1, it might be clearer to specify that the same approach used in Lemma 1 should be employed for computing expectations in the proof of Corollary 1.\n• Thepapershouldinvestigateandexplainwhytheaccuracycurvedecreases slowly from block 3 to 4 for λ = 2.5 and why Figure 4b shows a significant drop in model accuracy.\n• The authors need to examine and elucidate the unexpected behavior mentioned in Table 2, where compressing more layers results in more GPU memory savings for U-NET but not for ResNet-18.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gqu7fySAJW", "reviewer": "Reviewer_X2os", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed pipeline has been well employed in practice. However, the proposed analyses are interesting.\n\nBriefly, there are a few major weaknesses:\n\n1. The novelty of the proposed pipeline is not clear. Several implementation and algorithmic details are also missing.\n\n2. The theoretical analyses should be improved considering all of the proposed methods, e.g. including sparsification.\n\n3. Experimental analyses should be improved using additional models and with ablations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach for compressing high-dimensional activation maps obtained using deep learning architectures. The proposed approach was compared with Wavelet Transform, Discrete Cosine Transform, and Simple Thresholding in two classification tasks for natural images and two semantic segmentation tasks for medical images. In the experimental analyses, the proposed method enables reduction of the memory usage for activation maps.  In the mathematical analyses, regularization property of the proposed approach was explored.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some of the major/minor comments:\n\n- Section 3 needs to be revised. More precisely, the novelty of the proposed approach is not well defined. The section starts by reviewing different transformation methods. Then, thresholding and sparsification/reconstruction methods are employed. Since this pipeline (decomposition, pruning and sparsification) is popularly utilized for data compression, the novelty of the proposed approach should be explained more clearly.\n\n- Could you please explain the sparse encoding eqn. (12) and dense decoding eqn. (13) methods?\n\n- In the theoretical analyses (e.g. Theorems), sparsification/reconstruction was not considered. How do they affect the theoretical results?\n\n- How do you employ W of WT in (8) and B of DCT in (9)? Do you further optimize them or use fixed random matrices?\n\n- Sparse computations on GPU can be computationally complex indeed since they are not supported by many libraries and their implementation can be challenging. Could you please provide implementation details?\n\n- Could you please provide a detailed footprint analysis of each step in the pipeline?\n\n- How does reconstruction increase the footprint?\n\n- In the motivation, larger DNNs such as transformers have been considered. Could you please extend the analyses with larger ResNets (e.g with layers >50), WideResnets and Transformers?\n\n- Ablations implementing different pruning and sparsification methods should be provided as well.", "broader_impact_concerns": "No concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE sides with acceptance. This work provides an experimental study of wavelet, DCT, and thresholding transforms for compressing features during training across a variety of deep network architectures for vision on MNIST, CIFAR-100, BRATS19, and SPLEEN. While these are known transforms, the results here are carefully measured and informative, and several categories of related work is credited (though please see the concluding note for a suggestion). The review and revision process has resulted in clearer text and expanded results to cover a greater diversity of model architectures.

The AE thanks the authors and reviewers for engaging in the TMLR process to deliver an improved paper with agreement between its claims and evidence and a clear audience.

Feedback to authors:

- The AE suggests citing a survey on model compression in the related work due to the popularity of the topic and the notes by reviewers on the existence of prior methods. For instance, please consider the survey [Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey](https://ieeexplore.ieee.org/abstract/document/9043731) by Deng et al. in IEEE'20. A pointer to this is a positive, while a highlighting of related methods on transforming activations is better still.
- The AE suggests reformatting to avoid the need to rotate Table 3 on Page 26 for readability, by for example by distributing the architecture columns across multiple rows.
Decision: Accept (poster)