Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition
Authors: Ismail Alkhouri
Xitong Zhang
Rongrong Wang
Venue: Accepted by TMLR ()

Abstract: Deep Neural Networks (DNNs) have achieved remarkable success in addressing many previously unsolvable tasks. However, the storage and computational requirements associated with DNNs pose a challenge for deploying these trained models on resource-limited devices. Therefore, a plethora of compression and pruning techniques have been proposed in recent years. Low-rank decomposition techniques are among the approaches most utilized to address this problem. Compared to post-training compression, compression-promoted training is still under-explored. In this paper, we present a theoretically-justified technique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. This is achieved without the need to change the structure at inference time or require constrained and/or additional optimization, other than the standard weight decay regularization. Moreover, LoRITa eliminates the need to (i) initialize with pre-trained models, (ii) specify rank selection prior to training, and (iii) compute SVD in each iteration. Our experimental results (i) demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/100 and ImageNet on Convolutional Neural Networks, and (ii) illustrate that we achieve either competitive or state-of-the-art results when compared to leading structured pruning and low-rank training methods in terms of FLOPs and parameters drop. Our code is available at \url{https://github.com/XitongSystem/LoRITa/tree/main}.

Published in Transactions on Machine Learning Research (11/2024)
Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition
Ismail R. Alkhouri*
alkhour3@msa, edu; ismaiaa@eumich..ee
Deportment of Computational Mathematics, Science & Engineering Michiggan State University
Department of Electrical Engineering & Computer Science University of Michigan Ann Arbor
Xitong Zhang
zhangrit&@msu.c edu
Deportment of Computational Mathematics, Science & Engineering Michigan Stale University
Rongrong Wang
mangron6&@msu.eet
Department of Computational Mathematics, Science & Engineering Department of Mathematics Michigan State University
Reviewed on OpenReview: https: // openreview. net/
net/forum? forum? id= 1KCrVMJoJJ
Abstract
Deep Neural Networks (DNNs) have achieved remarkable success in addressing many pre- viously unsolvable tasks. However, the storage and computational requirements associated with DNNs pose a challenge for deploying these trained models on resource-limitee devices. Therefore, a plethora of compression and pruning techniques have been proposed in re- cent years. Low-rank decomposition techniques are among the approaches most utilized to address this problem. Compared to post-training compression, training is still under-explored. In this paper, we present theoreticalll-jussifeee tech- nique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. This is achieved without the need to change the structure at inference time or require con- strained and/or additional optimization, other than the standard weight decay regulariza- tion. Moreover, LoRITa eliminates the need to (f) initialize with pre-trained models, (ii) specify rank selection prior to training, and (Hi) compute SVD each iteration. Our experimental results (i) demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/ 100 and Im- ageNet on Convolutional Neural Networks, and (fi) illustrate that we achieve either com- petitive state-of-the-art results when compared to leading structured pruning and low- rank training methods in terms of FLOPs and parameters drop. Our code is available at https://githns http:////.ooggg///////aaaa/aaa//
conppession-prrmoooe
1 Introduction
In recent years, the rapid progress of machine learning has sparked substantial interest, particularly in the realm of Deep Neural Networks (DNNs), utilizing architectures such as Fully Connected Networks (FCNs) Rumeehart et al.; 1986), Convolutional Neural Networks (CNN) [LCunn et al. 1998). and Transformers
Vaswani et al.) 2017). DNNs have demonstrated remarkable performance across diverse tasks, including
classification (Han et al. 2022 image reconstruction (Ravishankar et al. 2019). object recognition (Minaee
"The first. authors contributed equally.

---

Published in Transactions on Machine Learning Research (11/2024)
et al. 2021) natural language processing (Vaswani al. 2017), and image generation (Yang et al. 2028 However, the extensive storage and computational requirements of these models hinder their application in resource-limites devices. Addressing these challenges has become a focal point in recent research efforts within the context of a variety of model compression algorithms [Li et al. 2023 Marinó et al. 2023
Model compression techniques, based on their core strategies, can be classified into parameter quantiza- tion [Krishnamoorthi, 2018), knowledge distillation (Hinton et al., 2015), lightweight model design (Zhang et al. 2018), model pruning (He et al., 2019), and low-rank decomposition (Lin et al.) 2018). Post-training
low-rank decomposition algorithms for network compression necessitate pre-trained weights (Marinó et al,
2023), whereas existing low-rank training methods require structural alterations to the network architecture
[Howard 2017], the specification of a pre-defined rank during training (Kwon et al. 2023), and/or addi-
tional constraints or iterations in the training process (Hawkins et al., 2022). This paper explores an effective technique for low-rank training: Over-parameterizing weight matrices to achieve low-rank model directly through training from random initialization.
Over-pen parameterizing weight matrices has been employed in neural networks for various purposes beyond compression, as discussed in more detail in Appendix [ In this paper, we revisit the concept of over- parameterization, providing a detailed exploration of it from the perspective of compression. For clarity, we introduce the term Low-Rank Induced Training (ToRITa) to describe the specific version of this tech- nique of interest, which employs over paramttrrizationn of the weight matrices by a linear composition of dense matrices. We show both numerically and theoretically how this technique promotes low-rankness of weight matrices. In addition, we extend this technique to convolutional neural networks, Unlike the well-known Low-Rank Adaptation (LoRA) method (Hu et al. 2021, which focuses on low-rank fine-tuning, the LoRITa approach utilizes dense full-rank factors during training. This method, when paired with post- training compression techniques such as Singular Value Thresholding, eliminates the need to determine or know the (sub)optimal ranks of the weights during training. A key finding of this paper is that the laye--compostioon-basd overparametrization should be combined with weight decay-a commonly used rog- ularizer to effectively promote low-rankness models. Next, we summarize our contributions.
We demonstrate that the simple technique of weight matrix over-parameeterizaion through linear layer composition, when combined with weight-decay, elfectively encourages low-rankness during training.
We provide justification for this observation and analyze its theoretical properties.
We show through extensive experiments that LoRITa applics to a wide range of network architectures and can be used in combination with various post-training compression methods. The experiments include DNN-based image classification tasks across different FCNs, CNNs, and ViTs architectures, using MNIST, CIFAR10, CIFARI00, and ImageNet datasets.
We note that LoRITa is a prrseviu technique, meaning it does not alter the network architec- ture, such as reducing the number of filters in CNNs or pruning redundant nodes. To modify the network structure, LoRITa should be combined with other pruning techniques. In this regard, LoRITa functions as a regularization method, akin to weight decay or dropout, which are commonly used in combination with other regularizations.
Additionally, while LoRITa promotes low rankness during training, actual truncation compression is performed post-training using Singular Value Truncation (SVT) or other more advanced methods.
Since this paper focuses solely on LoRITa, the reported numerical results are based on LoRITa regular- ization alone during training, in combination with the simplest SVT for compression. Nevertheless, we obscrve comparable performance to more sophisticated algorithms, highlighting the benefits of the over- parameterization approach. We leave studying the combinations of LoRITa with other pruning methods (or using more advanced post-training compression approaches) to future studies.
2

---

Published in Transactions on Machine Learning Research (11/2024)
2 Related Work
In the past decade, numerous mothods have been proposed in the realm of DNN compression, employing techniques such as low-rank decomposition or model pruning. Recent survey papers, such as (Li al. 2023 Marinó et al.) 2023 He & Xiao) 2023). offer comprehensive overviews. This section aims to survey recent works closely related to our method. Specifically, we explore papers focused on (i) post-training low-rank compression methods, (fi) low-rank training approaches, and (iit) structured pruning methods.
2.1 Post-training Low-Rank Compression Methods
As an important compression stralegy, low-rank compression seeks to utilize low-rank decomposition tech- niques for factorizing the original trained full-rank DNN model into smaller matrices or tensors. This process results in notable storage and computational savings (refer to Section 4 in (Li et al. 2023
In the work by Yu el al. 2017 pruning methods were combined with SVD, requiring feature reconstruction during both the training and testing phases. Another approach, presented in (Lin et al. 2018). treated the convolution kernel as a 3D tensor, considering the fully connected layer as either a 2D matrix or a 3D tensor. Low-rank filters were employed to expedite convolutional operations. For instance, using tensor products, a high-dimensional discrete cosine transform (DCT) and wavelet systems were constructed from 1D DCT transforms and ID wavelets, respectively.
The authors in (Liebenwein et al., 2021) introduced the Automatic Layer-wise Decomposition Selector (ALDS) method. ALDS uses layer-wise error bounds to formulate an optimization problem with the objective of minimizing the maximum compression error across layers.
While our method and the aforementioned approaches utilize low-rank decomposition and maintain the network structure in inference, a notable distinction lies our approach being a training method that promotes low-rankness through the composition of lincar layers. It important to emphasize that any low- rank-based post-training compression technique can be applied to DNN trained with LoRITa. This will be demonstrated in our experimental results, particularly with the utilization of three singular-value truncation methods.
2.2 Low-Rank Promoting Methods
Here, review methods designed to promote low-rankness in DNN training. These approaches typically involve employing one a combination of various techniques, such int rnduuing structural modifications (most commonly ander-parameterriattte implementing custom and computationally--nteessve regularization techniques such as the use of Bayosian estimator (Hawkins et al., 2022), iterative SVD (Gural el al. 2020), or the implementation of partially low-rank training (Waleffe & Rekatsinas 2020).
encoding constraints within the training optimization process, or
The study presented in (won et al. 2023 introduces approach leveraging SVD compression for over- parameterized DNNs through low-dimensional learning dynamics inspired by theoretical study deep linear networks. The authors identify a low-dimensional structure within weight matrices across diverse architectures. Notably, the post-training compression exclusively targets the linear layers appended to the FCN, necessitating a specialized initialization. More importantly, a pre-specified rank is required, posing a challenge as finding the optimal combination of ranks for all layers is a difficult problem. Since different layers in the model may have different importance to the performance and should be compressed differently, it requires the rank to be layer-specific. For example, the necessity of using layer-specifie rank in LoRA is dis- cussed in (Zhang et al. 2024p. However, finding the ranks of all layers can be computationally expensive. In LoRITa, this requirement is not needed, as we thcoretically show that standard weight decay encourages low rankness without specifying the rank during training. In comparison, our work shares similarities as it em- ploys a composition of multiple matrices. However, our approach encompasses all weight matrices, attention layer weights, and convolutional layers, providing a more comprehensive treatment of DNN architectures.
The study conducted by Tai et al. 2015) introduced an algorithm aimed at computing the low-rank tensor decomposition to eliminate redundancy in convolution kernels. Additionally, the authors proposed mothod
3

---

Published in Transactions on Machine Learning Research (11/2024)
for training low-rank-constrainee CNNs from scratch. This involved parameterizing each convolutional layer as composition of two convolutional layers, resulting in a CNN with more layers than the original. The training algorithm for low-rank constrained CNNs required enforcing orthogonality- based regularization and additional updates. Notably, their approach did not extend to attention layers in ViTs and fully connected weight matrices. Moreover, during lesting, unlike our method, their approach necessitates the presence of additional CNN layers.
In recent contribution by (Sui et al. 2024), low-rank training approach was proposed to achieve high accuracy, high compactness, and low-rank CNN models. The method introduces structural change in the convolutional layer, employing under-paree parameterization. Besides altering the structure, this method requires low-rank initialization and imposes orthogonality constraints during training.
The authors in [Idelbayev & Carreira-Perpine 2020) proposed the low-rank compression of neural networks (LCNN) method, which uses ADMM to optimize the weight matrices constrained with pro-sclected ranks. Training using this method is computationally expensive, as in each iteration, the SVD of each matrix is computed. It is noteworthy that LoRITa, in contrast, not only preserves the existing structure at inference time but also eliminates the need for optimization with complex constraints, relying on the standard training with weight decay.
2.3 Structured Pruning Methods
When contrasted with weight quantization and unstructured pruning approaches, structured pruning tech- niques emerge as more practical (He & Xiao, 2023). This preference stems from the dual benefits of struc- tured pruning: not only does it reduce storage requirements, but it also lowers computational demands. As delincated in a recent survey (He & Xiao, 2023). there exists a plcthora of structured pruning methods, particularly tailored for CNNs. As our proposed approach offers both storage and computational reductions, the following methods will serve our baselines in the experimental results.
These methods belong to six categories. Firstly, we consider regularization- methods, such as Scientific Control for reliable DNN Pruming (SCOP) (Tang et al. 2020), Adding Before Pruning (ABP) (Tian et al. 202I), and the only train once (OTO) [Chen et al. 2021) methods which introduce extra parameters for regularization. Secondly, we consider mothods based on joint pruning and compression, exemplified by the Hinge technique (Li et al. 2020). and the Efficient Decomposition and Pruning (EDP) method (Ruan) et al. 2021). Thirdly, we consider activation-based methods such Graph Convolutional Neural Pruning (GCNP) (Jiang et al. 2022) (where the graph DNN is utilized to promote further pruning the CNN of interest), Channel Independenee-basee Pruning (CHIP) (Sui et al. 2021), Filter Pruning via Deep Learning with Feature Discrimination in Deep Neural Networks with Receptive Field Criterion (RFC) (DLRFC) (He filter importance criterion, and Provable Filter Pruning (PFP) (Liebenwein et al.) )019)) Next, we consider weightt-dependent methods, such as the adaptive Exemplar filters method (EPruner) (Lin et al.] 2021]. , Cross-layer Ranking & K-reciprocal Nearest Filters method (CLR-RNF) (Lin et al. 2022), and Filter pruning using high-rank feature map (HRank) [Lin et al., 2020). Next, we consider a method proposed in (Liu et al. 2022) based on automatically scarching for the optimal kernel shape (SOKS) and conducting stripe-wise pruning. Lastly, we consider Reinforcement Learning based methods such as Deep compression with reinforcement learning (DECORE) (Alwani et al.
ct al. 2022 which utilize a feature discrimination-basse
2022h.
3 Preliminaries
Given a matrix A E Rm"", factorizes it into three matrices: A = UΣVT, where U E mm and V E Rax= are orthogonal matrices, and Σ E Rmxx is a diagonal matrix with non-negative real numbers known as singular values, st. For a matrix A, its Schatten p-norm, ||A]|p- is the Ep norm on the singular values, i.c., || == (E, M/. The muclcar (resp. Frobenius) norm, denoted as || (resp. [| corresponds to the Schatten p-norm with p (resp. p = 2). For any positive integer N. [N] := NI. We use af)) and Max()) to denote the ReLU and softmax activation functions, respectively.
Notation:
4

---

Published in Transactions on Machine Learning Research (11/2024)
3.1 Fully Connected, Convolutional, & Attention Layers
Consider a DNN-based model with L fully-connected layers for which input and output y related as:
y(O,x) W
(1)
where e - (W;, Vi [L]] is the set of all parameters. For the sake of simplicity in notation, we will exclude the consideration of the bias, as it should not affect our conclusion.
For CNNs, the imput is third-order tensor for multi-channel image, with dimensions H x W x D, where H is the height, W is the width, and D is the depth (the number of channels). The convolutional kernel is fourth-order tensor denoted by K, with dimensions Fn x Fw x FD x M. The output of the convolution operation is a third-order tensor O given as
Fg Fw
OZ
(2)
k=
Here, x and y are the pixel indices and m is the channel index in image O. The total number of output channels is equal to the number of filters M. The summation iterates over the spatial dimensions of the filter (1,5) and the input depth k.
ViTs consist of multi-head attention and fully connected layers. For one-head attention layer (Vaswani et al. 2017), we have Y = SoftMax XW (WW.WWW k) XWv, where X and Y are the input and output matrices, respectively. We consider the trainable weights We, WK, and Wv. Here, corresponds to the queries and keys dimension (Vaswani et al. 2017).
The goal is to compress the trained weights to minimize storage and computational requirements without compromising test accuracy significantly.
3.2 Singular Value Thresholding
For a given matrix W and its SVD W UEVT E Rmxx, its best rank-r approximation can be represented as W ~ W, U,E. V where U, E mmxr contains the first columns of U, Σ, Rrxr is a diagonal matrix that include the largest singular values, and V E Rexa contains the first rows of VT. These are called the r-terin truncation of U, E, and V.
4 Compression with LoRITa
In this section, we start by presenting three simple ways to apply SVT neural networks. Then, we present the details of the LoRITa technique, followed by discussing the theoretical foundation of our method.
4.1 Singular Value Truncation of Trained Weights
Local Singular Value Truncation (LSVT) of Trained Weights: We simply apply SVT to each weight matrix independently with a fixed rank r.
Global Singular Value Truncation (GSVT) of Trained Weights: For DNNs, not every layer holds the same level of importance the output. Therefore, using the local SVT might not lead the best compression method. To tackle this problem, This involves normalizing the singular values of each matrix, i.c., dividing each weight matrix by its largest singular value, and then sorting the singular values of the normalized matrices globally. Subsequently, we decide which singular values to drop based on this global ranking. This approach offers an automated hed for identifying the principal components of the entire network.
apply global singular value truncation strategy.
Iterative Singular Value Truncation (ISVT) of Trained Weights: An alternative strategy to global ranking is performancce-preservin truncation. In each iteration, fix the number of parameters to truncate,
5

---

Published in Transactions on Machine Learning Research (11/2024)
Activation
Activation
W
W2
W
W
W
W
Post Training (Standard Pipeline):
Post Training (Our Pipeline):
Activation
Activation
(V),
(W,)
SW;
SVD W
Figure Illustrative example showcasing the standard (left) and our (right) pipelines. Here, factorization of is applied to each weight matrix in simple 2-layered neural network. Post-training, conventional methods employ singular value truncation on trained weights, whereas our approach, singular value truncation conducted on the product of two trained matrices for each weight matrix.
then we decide which layer we will use remove these parameters by examining the one that induces the lowest increase in the training loss after the truncation. More details are given in Appendix A
With the r-term truncation, for LSVT, GSVT, and ISVT, the storage requirement reduces from mn to (m + n)r. We use SVD, E Rи to denote the SVD r-truncated matrix of W E Rmxx
In LSVT, GSVT, and ISVT, our aim is to minimize the accuracy drop resulting from truncation by ensuring that each weight matrix, W, exhibits high compressibility. This compressibility characterized by rapid decay in the singular values of W. Such a decay pattern enables us to discard substantial number of singular values without significantly sacrificing accuracy. Consequently, this property enhances the effectiveness of the compression process. Based on this discussion, notable insight emerges:
The faster the singular values of W docay, the more favorable the compression outcome using SVD. This observation leads to the desire for W to be of low rank, as low-rank matrices exhibit faster decay in singular values. The question that naturally arises is: How to enforce low-rande now-rankness in W during training while preserving the DNN structure at the inference phase?
Next, propose our method of low-rank promoted training through the composition of linear layers before activation.
4.2 Model Re-parmeterization with LoRITa
We employ the following simple compression pipeline:
During training, we employ LoRITa to achieve maximal decay of the singular values of the weight matrices in global manner without sacrificing the model's capacity.
Post training, the LSVT, GSVT ISVT on the trained weights.
More specifically, we perform the following steps:
We first express each trainable weight matrix W E mmx as a composition of N > matrices: W w'w?... W Here, W E mx and subsequent matrices W2, W wa...., W. E Rax=, For the model given Equation T the input/output relation becomes:
x)) = II w o
II KE[M]
Who II wfx)),
(3)
kE[N]
kE[M]
where (W), A4 E [N], Vi E [T]]-

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm Compression with LVRI
Input: L trainable weights W,, Vi E [L], factorization parameter N > 1, and singular value truncation
parameter r.
Output: Compressed and trained Weights.
1: For each i [T]]
2: Replace W, by W...., W.
3: Train W Vi E [4]. Vk E [M] using Adam and weight decay.
For each i E [4]
5: Use SVD. W instead of W.
During training, we minimize the objective
min J Σ (y)(O.)).,))+x JEIN] (E[T]
+1 + Ewil
where À is the weight decay parameter, and J the number of data points (with (xx, in the training set. We note that the proposed factorization works for arbitrary m and n.
Throughout the training process, optimize the weight matrices W for all E [4], and & E [N].
After the training is finished, we compute the product W; IIEETNN W [Z], and assign them the trained weights of the original model.
In summary, LoRITa utilizes the over-parameterrieed weights during training and reverts to the original weights dimensions after training. The reversion ensures that LoRITa is a structure-preserinng technique.
The underlying idea that a model trained in this manner exhibits enhanced compressibility compared to the original model described by Equation [ The resulting weight matrices W, demonstrate faster decay in singular values. This strategic approach to training and approximation aims achieve a more compact and efficient representation of the neural network trained weights, and preserve structure during inference. 1 presents example of our proposed method.
The procedure outlined in Algorithm I Figure
The matrices that correspond to the weights of attention layers are treated in a similar fashion as fully- connected layers.
For convolutional layers, the fourth-order tensor K, with dimensions Fμ x Fw x FD x M, is reshaped into a matrix. This achieved through a mode-4 unfolding, resulting in matrix (4)) of size Fw FD M. ((4) is then expressed as composite of N matrices, denoted as (4) K2K K. Throughout the training phase, each K' is updated as weight matrix. Post-training, SVD with r-truncation is applied to K(4) = K-K2... .. KN to obtain low-rank factors L and R, each with columns, ensuring that ( ~ LR T. During the inference phase, we first compute the convolutions of the input with the filters in L reshaped back to tensors, then apply R T to the output of the convolutions. More specifically, we reshape L back to a fourth-order tensor of size FH x Fw x FD X r, denoted by L, then we conduct its convolution with the input
Fg F11--1FF -1r- Fb
Ó(z,y,m²) = Σ Σ
Σ
1=0
j=0
k=0
with m' E (r]. The final output is obtained as
O (a
Ō y, m m')mmm,m)))
Compared to the convolution in Equation 2 the computational cost for a single convolution operation is roduced from Fw to Fw Similarly, the storage requirements are also docreased by comparable magnitude. Figure 2 iilussrates the operations performed with CNNs. Next, We will explain the rationale behind our proposed approach.

---

Published in Transactions on Machine Learning Research (11/2024)
M
M
M
M
R
Reshape
Compression
Reshaps
FFFFFo
FgEwFF D
FD
Ff x Fw
k())
SVT with
L
K
Ph X Fw
Reshape
FD
M
Training
K2
K
Fy x Fw
*
*
*
K²
M
M
M
FD.
Pi x Fw
Inference:
*
*
R
FD
Figure 2: Illustrative block diagram showcasing the compression (lop). training (middle), and inference (bottom) for every convolution layer in CNNs. Here, 'x' denotes the convolution operator.
Remark 4.1. The benefit of nuc-ppamaetteviaationn in network training has been observed previously in KKoodak et al.) 2021 Arora et al, 2018 Guo et al. 2020;; Huh et al. (2021) for various reasons. (Arora et al. 2018) proved that over-paramelteriattoon can accelerate the convergence of gradient descent deep Unear networks by analyzing the trajectory of gradient flow. (Guo et al.} (020) found that combining over- parameterization with width exponsion can enhance the overall performance compact networks. (Huh el al. 2027) noted through experiments that pereparrameeeiiatiionn slightly improves the performance of the trained network, and deeper networks appear to have better rank regularization. These previous works focus on performance and acceleration. Our work complements these findings, providing numerical and theoretical evidence to justify the benefits of over pver-porameterized weight factorization in network compression due to the weight deeny behaving as a rank regularizer. Further discussions for these methods are provided Appendir C
4.3 Theoretical Underpinnings of LoRITa
In this subsection, we provide theoretical insight into why LoRITa has to be used with weight-decay.
4.3.1 Weight Decay Regularization & Low-Rankness in LoRITa
Here, we start by stating the following proposition, which says that minimizing the Frobenius norm of the over-parameterized weight matricos oquivalent to minimizing the Schatton p-norm (0 < p < 1) of the original weight matrices and the latter well-known to promote the low-rankness. The proof of this proposition is given in Appendix B.I
Proposition 4.2. Let A RRxx be an arbitrary matriz and S min(m, n} be ils rank. For a fixed integer N L4, A can be expressed as the product of matrices R; mmxx i.e., A II R, (m++1 n, 2 r, i= 1,.... N-1, Mi = m, ny = n), in infinitely many ways. For any e (0, and p. > 0, Vi E [N], satisfying Σ EVENN| p' it holds
A 1p min R.JE[N]
W SE[M Di
(R.IP
1/p
8.l.
RR = A. (E[N]
(4)
Proposition 4.2 slightly extends Theorems 4 and from (Shang al. 2016). its requirement on the dimensions of the factors R, Furthermore, we do assume any prior knowledge of the rank of A, which is conventionally set to its upper bound min
8

---

Published in Transactions on Machine Learning Research (11/2024)
Proposition 4.2 indicates that the Schatten-p norm of any matrix is equivalent to the minimization of the weighted sum of Schatten-p: norm of each factor matrix by which the weights of these terms are 2 for all Pi
E [.].
For 1, N = 2, and P1 P2 = 2, Equation
reduces to
|||| min R R2
R
[R211E] s.1.
R, A.
(5)
For some integer 9>> 1, let p y/ N 2g, and p1 = P2 -... P2g = 2. Then, Equation
becomes:
||/|/// = min [,,
1 Σ 24
s.t. II R A. VE[2g]
(6)
kE|2g]
P (0, II, minimizing the Schatten p-norm encourages low-rankness. A smaller p sttrenthhens the promotion of sparsity by the Schatten p-norm (Nie al. 2012). Think of the A matrix in Equation 5 and Equation (6)
as weight matrices in FCNs, matricized convolutional layers in CNNs, matrices representing query, key, and value in attention layers. These identities (Equation (5) and Equation (6) imply that by re-aarameterizin
the weight matrix A as a product of 2q other matrices R., where E [2g], and using R3 (instead of A) as the variable in gradient descent (or Adam), the weight decay on the new variable corresponds to the right- hand side of Equation (6) Additionally, Equation (6) suggests that the more R; we use to represent A, the lower rank we obtain for A. This explains why our proposed model in Equation (3) can achieve a lower rank for the weights compared to the traditional formulation in Equation
Remark 4.3. For any weight matrix of size by n, our proposed training method requires training nmN parameters instead of mn parameters where N is the factorization parameter. Yet in practice, it is more efficient than those SVD-based low-rankness promoling techniiuues since explicil SVD is needed. Although Proposition 4.2 suggests that increasing N leads to enhancement in the low-rankness of the appended trained weights, in practice, this leads to prolonged training durations and more nonlinear landscape of the op- timizalion problem. experiments, we observe that = 3 usually sufficient to achieve near-oplimal performance, under affordable computational cost. Moreover, prioritizing shorter test times for large models is particularly crucial. This consideration is significant, especially given that testing inference happens continuously, unlike training, which occurs only once infrequently. This aspect significantly affects user experience, especially resource-limite devices where compressed models are often deployed.
Remark 4.4. In contrust to previous works, Proposition 4.2 the foundation of LoRITa, reveals that. we do not assume that the weights are strictly low-rank nor require the knowledge of the weight matrix's (ap- prozimate) rank during the training phase. Consequently, our approach encourages low-rakneess without compromising the network's capacity.
4.3.2 Sufficiency of Single Weight Decay Parameter in LoRITa
Proposition 4.2 is used for compressing single matrix. In the case of nonlincar networks, we have to apply it layer by layer. For each layer, a Schalten-p norm regularization of its weight matrix is introduced to the objective function to penalize the rank, together we have L penalty terms,
min 1 Σ (()().)). + Σ aul Wil LE(L)
1 I LE|T
where (W,l E [4]], and on > 0 for all E [L] are the strengths of the penalties. Let us set p = A with some even integer K (the smaller the p is set, the stronger encourages low-rankness). Then for each weight matrix, apply Proposition 4.2 to re-parameterize the weight matrix into a depth-K deep matrix factorization. This turns the above optimization into the following equivalent form that avoids the computation of SVD on-the-fly.
K + ou
min e
Σ J
((y(0,x))) v:)
W 1
(7)
2
1=1
9

---

Published in Transactions on Machine Learning Research (11/2024)
Model
Dataset
N
N
Model
Dataset
==1
N =
=
CNN VGG13
CIFART0
0.919
0,922
FCN-6
MNIST
0.983
0,982
0.977
CNN ResNetL8
CIFAR1O
0.928
0.927
FCN-8 FCN- 10
MNIST
0.983
0.982
0.977
CNN VGG13
CIFAR100
0.678
0.686
MNIST
0.983
0.981
0.977
CNN ResNet18
CIFAR100
0.708
0.714
VIT-L&-HI
CIFARIO
0.717
0.729
0.727
VII-L4-H8
CIFARIO
0.861
0.802
VIT-L&-IIA
CIFARIO
0.710
0.718
0.701
VIT-LS-II8
CIFARIO
0.965
0.857
VIT-L8 H8
CIFARI0
0.705
0.719
0.714
VIT-L16 H8
CIFAR10
0.856
0.867
(a) Without data augmentatiin.
(b) With data augmentation.
Table 1: 'est Accuracy results of standard (NV 1) and LoRITa (N > 1) models, Here, represents the number of composed matrices for each layer. When N = 1, it reduces to the original non perapaaamtteee model. When N 2, there actual over- pee--arameterization and can be called LoRITa (our proposed technique).
where θ = ( [L], E [K]].
However, Equation 7 still has too many tuning parameters on, E [L]. Fortunately, for ReLU networks, we can reduce the number of eyper-parameters to as shown in the following Proposition where the proof is deferred to Appendix B.2
Proposition 4.5. With ReLU aclivalion, the oplimizalion problem in Equalion
7 is equivalent to the
following single-hpperppraameee problem
K ΣΣ wil
min Σ ((x(0,x)).))
(8)
with some proper choice of X, in the sense that they share the same mindmizers.
Remark 4.6. Proposition 4.5 supports the praclice of using a single weight decay parameter during network truining, is commonly done in the literature.
5
Experimental Results
5.1 LoRITa Evaluation on FCNs, CNNs, & ViTs
To rigorously evaluate the effectiveness of the LoRITa method in achieving rank reduction, we first consider FCNs, CNNs, and VITs on simple datasets (motivated by the composition of
their attention layers, which are essentially built from fully connected layers). The evalua- tion criterion is the reduction in accuracy w.r.t. the percentage of Retained Singular Values.
The latter is defined as the number of retained singular values divided by the total number of singular values per ma- trix, averaged across all trained matri-
Gelayer PCN (LSVT)
8-Jayer FCN
10-laver (LSVT)
Racunacyytnnn (43
ACC.rSSy Drop (5)
Accuracy Groe 196)
8
ces. As a result, the drop in accuracy is computed by subtracting the testing accuracy with SVT from the testing ac- curacy without applying SVT.
15
No.
N-3
40
15
20
10
25
Retained Singulrr Values (9)
Retained Singuar Values (96
Retaired Singular Valias (%6
6-layer
FCN FCN
(GSVT)
8-layer ECN (GSVT)
10-layer ECN (GSVT)
Accurscy Drep (8))
A variety of models, datasets, and over- parameterization scenarios are consid- ered, as outlined in Table For the VIT
Fecurecy Drop
20
Aocurety Drop
models, the number following 'L' and 'H' represents the number of layers and the number of heads, respectively. For comparison with baselines, we consider the models with N (models with- out over-parameteriiation using LoRITa
Retained Varues
Figure 3: Results of the FCNs in Table 1a The top (resp. bottom) row corresponds to applying the Local (resp. Global) SVT. N = 1 results represent the baseline, whereas N results are for the LoRITTattriine models.
10

---

100 Published in Transactions on Machine Learning Research (11/2024)
during training). Further experimental setup details can be found in the caption of Table 1 We use PyTorch to conduct our experiments, and our code will be released at later date.
In each experiment, our imitial phase consists of training the baseline model with optimal weight decay settings to achieve the highest achievable test accuracy. Subsequently, we apply the LoRITa method to re-parameterize the original model. This process involves initializing from a random state and tuning the weight decay to ensure that the final test accuracies remain comparable to those of the initial models. See the test accuracies in Table 1
(GG)) CIFARIO
Re (I CIFAR1O CIFARI
VGGI3 (L.VV) CIFARIOO
ResNet (VT) CIFAR 100
Accunicy Drep (90)
Accumccy Grop (96)
Accuraccy Drop (%6
Crea (16)
-20
-20
-20
Accurscy Crea
-20
-40
-40
60-
1=1
-00
N=1
N=1
N-1
-60
N-2
20
40
60
80
100
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
100
Rotained Singular Values
(50)
Recained Singuar Values (%)
Retzined Singular Values 1Sc
Reteined Singuler Values
(%)
YGGI3 CIFARIO
8 8 (GSVT) CIFAR10
VGGI3 (GSVT) CIFARIOO
ResNet (GSVT) CIFARI00
Accuracy Drea (40
Accuracy grop 196
Accuracy Drop (N)
Accuracy CeeurscyDrer Drep (N)
-20
-20
-40
40-
-40
-60
N-1
1-1
N-1
N-1
N
N-
-60
N=2
40
60
30
100
20
40
60
80
0
20
40
60
80
100
20
40
60
80
100
Reteined Singular Values 1%)
Retained Singuaar Volues (9)
Retzined Singular Values 156)
Rotained Singulor Valluss (%))
Figure 4: Low-rank compression results of the considered CNNs using the CIFAR10 (left) and the CIFAR100 (right)
datasets for the settings in Table Ib N denotes the baseline, whereas N 2 represents our method.
Subsequently, we implement SVT on the weight matrices to compress the model, examining the impact test accuracy as smaller singular values are eliminated. We delve into the LSVT and GSVT strategies, discussed in Section| to determine which singular values to eliminate.
When employing GSVT on ViTs, singular values
4- layer ViT (LSVT)
8- 8-layer VIT (LSVT)
16-layer VIT (LSVT)
within the attention and fully connected layers are independently managed.
Accnney Keop (8)
4000000 Dool Pi 8
t0
Dep
8 %
Similarly, for CNNs, compression is applicd dis- tinetly to both the convolutional and fully connected layers, treating each type separately.
Betained Singuter Uellaen
Detaired Singster Valuees
Retwreed Singuler Valum (S)
4-layer ViT (GSVT)
8-layer VIT (GSVT)
16-layer ViT (GSVT)
First, we evaluate our proposed method on fully connected neural networks, varying the number of layers, utilizing the Adam optimizer with a learn- ing rate set to 1 x 10-2, and employing a constant layer dimension of 96 (other than the last). Over- parameterization is applied across all layers in the model. To ensure a fair comparison, we begin by tuning the baseline model (N = 1) across a range of weight decay parameters (5 x 10 1x 10 10-5,2 x 10-1,5 x 10-5,1 x 10-1,2 x 10-1). Subsequently, we extend our exploration of weight decay within the same parameter range for models with N >>> 1. As depicted in Table La] setting N to values larger than one results in closely matched final test accu- racies. The results for FCNs on the MNIST dataset are illustrated in Figure g In these plots, LSVT (resp. GSVT) employed for the top (resp. bot- tom) three plots, providing insights into the effectiveness of the proposed technique. As observed, in almost all the considered cases, models trained with N > 1 achieve better compression results (w.r.t. the drop in
Accuraty Drep rgi
Drap MR)
Accurscy Crop (N)
-20
-20
50
N-1
-60
N-11
N-2
4-2
5,00
20.0
10.00 20.0
Retained Singuar Values (hl
Retained
Retaired Singulor Values Fh)
1-head ViT (LSVT)
4-head ViT (LSVT)
8-head ViT (LSVT)
Mctursyy Doop mM)
Droe
Km]
Nw
No-2
Km
N=3
30 Retained Singular Volues (h6)
15
Reta med Sinqular Virues 1%)
Retained Singulor Values Mil
Figure 5: Low-rank compression of ViT models with data augmentation with varied attention heads and layers. The (top) and (bottom) plots correspond to the settings of Ta- ble 1b and Table Ia] respectively. N = denotes the baseline, while
and represent our method.
11

---

Published in Transactions on Machine Learning Research (11/2024)
test accuracy) when compared to the N = 1 models. For the example of the 10-layer FCN model (with GSVT), using LoRITa with N 3 achieves no drop in test accuracy by keeping only 15% of the singular values. When LoRITa is not employed (N = 1), the model is required to retain approximately 22% of the singular values in order to achieve the zero drop in test accuracy.
It is important to highlight that the strength of the parameter À associated with the sparsity-promoting regularizer (in this instance, weight decay) is not selected to maximize the compression rate. Instead, the choice of À is geared towards maximizing the test performance of the network, as is conventionally done. Remarkably, adopting this approach still yields commendable compression rate. In Appendix D we empirically demonstrate the faster decay of the singular values of the trained weights.
Next, we evaluate the performance of our proposed method on the well-known VGG13 and ResNet18 ar- chitectures using the CIFAR1O and CIFAR100 datasets. The learning rate applied in this evaluation is set to 10-4, The weight decay was scarched over {1 10-2, 5, x 10-3, 1 x 10-33 for CIFAR10 and [1 {1x 10010-5,5 x 10-5, , x 10-2) for CIPAR100. The results are depicted in Figure 4 Consistent with the FCN results, we observe improved compression outcomes for models trained with LoRITa. The findings from the CNN results suggest that, despite using straightforward matricization of the convolutional filter (Figure 2 enhanced compression results are still evident. For the instance of VGG13, GSVT, and CIFAR100, retaining 20% of the singular values, not using LoRITa results in 60% drop in accuracy, whereas our LoRITaa-rainne model only results in nearly 5% drop in test accuracy. Furthermore, we observe that the test accuracy of the compressed model when LoRITa is applied is slightly higher than the test accuracy of the uncompressed model. This is reported at approximately 65% retained singular values when N = 2 in Figure 4 left).
To validate the efficacy of our comprossion technique, we evaluate it using ViTs (Beyer et al. 2022 which are noted for their capability to deliver leading-edge image classification outcomes with adequate pre-training. To comprehensively show that our training approach can yield a model of lower rank under various conditions, we first test our method on the CIFAR10 image classification task without data augmentation, employing ViTs with different head counts. Following this, we evaluate our approach on larger ViTs, incorporating data augmentation.
All the considered VIT models underwent optimization via the Adam optimizer with learning rate of 3x The hidden dimension is 256 for all ViTs. To facilitate compression, we apply perrpaarameterization across all attention modules within the model. For fair comparison, we initially fine-tuned the baseline model (N = 1) across the following weight decay parameters (5 x 10-5, x 10-4,2 x 10-4, 5 x 10-4, x 10-3, 3 2,2xx x 10 10-3). Subsequently, we explore weight decay within the same range for our models. As evidenced in Table Ta and Table 1b selting N to different values results closely matched final test accuracies.
The compression effects on VIT models without data augmentation are depicted in Figure 5 top). We note that with a higher N value, it is possible to achieve nearly lower model rank. The baseline model's test accuracy begins to decline at an 85% compression rate (retaining 15% of the singular values), whereas the case of N = 3 leads to stable test accuracy even at a 95% compression rate (retaining 5% of the singular values). We extend our evaluation to include ViTs with different layers and data augmentation (Figure 5) bollom)) The use of data augmentation displayed a considerable improvement in testing accuracy (before compression) as shown when comparing the ViT results of Table Ib and Table La consistently resulted models of a lower rank, as illustrated in Figure The outcomes consistently indicate that our training approach results in models with lower rank compared to employing only weight decay, across variety of training configurations and network architectures.
Nevertheless, employing our method
5.2 Comparison with Structured Pruning & Low-Rank Training Baselines on CNNs
Here, we compare LORITa+ISVT with structured pruning and low-rank training techniques delineated in Scction 2 The metrics employed for this comparison are the required FLOPs and the number of parameters dropped after compression/pruuiii computed using the 'puflops' Python package !
Table 2 and Table 3 present the results for the ResNet20 and VGG-16 architectures, respectively, on the CIFAR10 dataset. Table and Table 5 present the results on ImageNet using ResNet18 and ResNet34
12

---

Published in Transactions on Machine Learning Research (11/2024)
Method
Acc.(%) (1)
Pruned Acc.(%)
Acc. Drop (%) (4)
Pruned
FLOPs Drop (%)
Pruned Model Param.(M)
Param. Drop(())
FLOPs (M) (4)
LoRIT=+IVVT (N 3)
91.63
-0.63
18.47
54.7
0.11
61
LoRITs+ISVT (N
91.63
90.51
-1.09
15.41
62.3
0.081
68.8
LoRIT++IVVT (N 3)
91.63
90.17
-1.46
15.06
63.2
0.078
71
LoRITs+ISVT (N
92.33
91.64
-0.69
19.00
53,22
0.12
55.8
LoRITa (IVV (N -2)
92.33
90.27
-2.06
19.09
63.61
0.086
67.9
Baseline (N
92.34
90.26
-2.08
17.52
57.2
0.096
64.2
Baseline (N
92.34
90.01
-2.33
16.03
60.09
0.091
61.9
SOKS
os
2022
92.05
90.78
-1.27
15.40
62.04
0.14
48.14
SCOP
Lang a
2120
92.22
90.75
-1.44
18.08
55.7
0.12
56.3
Hinge
A121
92.54
91.81
-0.7
18.57
54.5
0.12
55.45
GCNP
Bang ct 0
21122
92.25
91.58
-0.67
20.18
50.54
0.17
38.51
ABP
2121
92.15
91.03
-1.12
21.31
47.7
0.15
45.1
LCNN [deebayee & Carrelza-Prppioe Perpinin 2020
91.25
90.13
-0.12
13.56
66.78
0.003
65.39
Table 2: Evaluation of LoRITa models compared to SOTA structured pruming/low-nng training methods using ResNet28 on CIFAR1O. The results the structured pruming methods are reported according to Table 5 in (He & Xiao
2023 (most recent survey paper) and ranked according to the FLOPs drop percentage. The last row results
are reported from Table in (Niao et al. 2023). The ResNet20 FT.OPs (resp. parameters) is 40.81M (resp. 0.27M).
Method
Avc. (1)
Prunod
Acc.
Pruned FLOPs (M) (1)
FLOPs
Pruned Model Param,.(M) (4)
Param.
Aec.(%) (1)
Drop (%) (4)
Drop (5) (1)
Drop(%) (f)
LoRITs+ISVT
-0.55
17.73
0.79
91.6
LoRITsIISVT (N =3
90.62
92.58
-1.04
42.59
86.43
0.66
96.5
LoRITs+ISVT
-1.43
0.465
LoRIIa+I8V I' (N
03.62
91.21
-2.41
30.1
90.41
D.465
97.35 58
LoRITs+ISVT LoRIT&IIVV (N =2)
50.21
64
91.13
92.8
-1.33
10.09
87 23
0.661
85, 48
LoRITs+1SVT (A=22) 222)
-2.13
36.72
Baseline (N 1)
93.78
92.09
-1.69
57.4
81.8
1.16
92.
Baseliae
1)
-2.59
0.985
DECORE (Setting 1)
Alwani et ml.
2022
90.96
91.68
-2.28
36.95
88.25
0.26
98.20 26
DECORE (Settins 2)
Alwan
-1.22
PFP
[Liebenwenn dt
92.89
92.39
-0.5
47.09
85.00
0.84
04.52
OTO
51.26
ABP
DMI mt m
3121
83.96
92.65
-1.31
52.81
83.21
1.5
89 71
EDP
-0.08
CHIP Dtd db 41
0721
83.96
95.18
-0.78
67.32
78.6
1.87
87.8
DIRFC
93.25
76.96
HRank
Lin el
20/21
83.95
91.23
-2.73
73,9
76.51
1.75
88 12
EPruner
74.42
76.34
1.65
CLR-RNE
Lin et. m
21223
93.02
93.332
0.3
81.48
74.1
0.74
95
LCNN (dlllaye
Perpiiik
20000
45.71
1.45
Table 3: Evaluation of LoRITa models as compared to SOTA structured pruning/low-ren training methods using
VGG16 on CIFAR10. The results of the structured pruming methods are reported according to Table in (He & Xiao 2023) (most recent survey paper) and ranked according to the FLOPs drop percentage. The last row results are reported from Table (Xiao et al. 2023). The VGG16 FLOPs (resp. parameters) is 314.59M (resp. 14.78M). Settings 1 and 2 for DECORE correspond to using different hyper-parameters including the penalty incorrect
predictions (Alwani al. 2022
architectures, respectively. The columns denote: 1) Test accuracy before pruning (%); 2) Test accuracy after pruning/ http///mmmmeoiooon (%); 3) Accuracy Drop (%); 4) FLOPs after pruning/comppesse ccompression (M); 5) FLOPs drop (%); 6) Pruned model parameters (M); 7) Parameters drop (%).
For CIFAR1O (resp. ImageNet), the results of the structure pruning baselines are as reported in Table 3 and Table (resp. Table 18 and Table 19) of (He & Xiao 2023), with arrows indicating preferable results. The results of the low-rank training baseline, LCNN Idelbayev & Carreira-Perpinán 2020)) are reported from Table in (Xiao et al., )023).
For the basclinc results, we remark that there could be different reasons for not using the same compression ratio: (i) Different goals: Reducing the number of parameters and reducing the number of FLOPs are two distinct goals, albeit related. Some papers may focus on one of these objectives, while others aim to achieve balance between both; (#) Compression is not continuous: In the low-rank case, each time a singular value is climinated, we remove m + parameters where m and n are the lengths of the left and right singular vectors of that layer, respectively. Other methods may try to remove filters in CNN, which also has jumps
13

---

Published in Transactions on Machine Learning Research (11/2024)
Pruned
Acc.
Pruned
FLOPs
Pruned Model
Param,
Method
Acc.(%) (t)
Acc.(%) (f)
Drop (%) (4)
FLOPs (M) (4)
Drop (%) (4)
Param.(M) (L)
Drop(%) (1)
LSVI (N = 3)
67.82
66.47
-1.35
1.01
44.2
6.17
47.22
LoRIIa+ISVT (N = 2)
69.22
68.14
-1.08
1.08
40.33
6.67
42.91
SOKS
Liu et al.
2022
70.42
69.16
-1.26
0.817
54.95
6.27
16.36
SCOP
lang et ml
2020
69.76
69.18
-0.58
1.11
38.8
7.1
39.30
SCOP
Tang 05 ml
69.76
68.62
-1.14
0.997
45
43.5
ABP
2021
70.29
67.83
-2446
1.021
43.7
6.31
16
Table 4: Evaluation of LoRITa models compared to SOTA structured pruming/low-rank training methods using ResNet18 on ImageNet. The results of the structured pruning methods are reported according to Table 5 in (He &
Xiao 2023) (most recent survey paper) and ranked according to the FLOOPs drop percentage. The ResNet18 FT.OPs
(resp. parameters) is 1.81G (resp. 11.69M).
Pruned
Acc.
Pruned
FLOPs Drop (%) (4)
Pruned Model (M)am.(M) (L)
Param,
Method
Acc.(%) (f)
Aoc.(%) (1)
Drop (%6 (4)
FLOPs (M) (4)
Drop(%) (t)
LoRIT&+ISVT (N
72.05
71.91
-0.14
1.92
17.51
11.31
48.11
SOKS
Liu et al.
2022
74.01
73.52
-0.49
1.612
55.98
11.27
48.3
SCOP
Fang tl ml
2020
73.31
72.62
-0.99
2.021
44.8
11.86
45.6
SCOP
Rang ec ml
2020
73.31
72.93
-0.38
2.231
39.1
13.15
39.7
ABP
Iau at.
2021
73.86
72.15
-1.71
1.923
47.5
10.75
50.7
Table 5: Evaluation of LoRITa models compared to SOTA structured pruning/ gruning/low-rait /low-rank training methods using ResNet34 on ImageNet. The results of the structured pruning methods are reported according to Table 3 in (He &
Xiao
2023) (most recent survey paper) and ranked according the FLOPs drop percentage. The ResNet34 FLOPs
(resp. parameters) is 3.66G (resp. 21.8M).
in the ratios; (ii) Some compression methods may suddenly break when the compression ratio exceeds certain threshold, and this threshold varies.
In the tables, N > 1 refers to the use of over--arameete over-parameeerriation i.e. LoRITa, and N = refers to the classical training of the non-overparameterizel network, i.c., the baseline. For both LoRITa+ISVT and the baseline, we apply GSVT, followed by implementing the ISVT approach for which only 120 training data points were used to compute the loss. Apart from using N 2 and correspond to applying ISVT with different desired compression rates (see Appendix A for more details). Here,
3, the different results of LoRITa+ISVT
SGD with learning rate 10-2 Furthermore, it's noteworthy that following all the considered baselines, the reported results for LoRITa+IVVT in Tables 2 and 5 are posi one round of fine-tuning whereas most of the considered baselines employ multiple rounds of fine-tuning of the compressed model. As long as the top-1 accuracy isn't compromisod by much (approximatcly 2%, as per (He & Xiao 2023 larger FLOPs drop signifies better compression methods. It's also imperative to note that the last colunı, Parameters drop, is vital as it signifies the amount of memory reduction.
For VGG16, our results boast the best FLOPs drop and demonstrate competitive parameter drop. In the case of ResNet20, our results achieve the best parameter drop and FLOPs drop compared to structured pruning methods. When compared to the low-rank training baseline (L.CNN), our method outperforms LCNNN in VGG16 while slightly underperforming in ResNet20 in terms of FLOPs drop. However, it is important to mention that LCNN is significantly more expensive to run, as it requires computing the SVD for every matrix at each iteration during training. In addition to the expensive training, the strong penalty term in LCNN prevents it from reaching the same test accuracy as other methods before compression, resulting in non- competitive test accuracy after pruning even with a small compression rate. Moreover, we observe that for both the considered architectures, with any fixed level of accuracy drop, our N = model, on average, outperforms our N 2 model in terms of both FLOPs drop and parameter drop. Furthermore, LoRITa models achieve improved performance when compared to applying ISVT on the baseline model (N = 1).
LoRITa combined with ISVT compression achieves either the best or nearly the best results across the two CNN CFIAR10 architectures. Most other methods achieve competitive results on one architecture. To highlight this point, we include comparison results from the baselines that considered both architectures in Table
14

---

Published in Transactions on Machine Learning Research (11/2024)
Method
FLOPs Drop(%) ResNet20 (f) Param, Drop(%) ResiNe20 (1)
FLOPs Drop(%) VGG16 (1)
Param. Drop(%) VCG16 (f)
LoRITh ISVT
ISVT (N -3)
61.2
71
90.41
97.58
SOKS
62.04
48. 14
72.2
78.33
Hinge
GCNP
50.54
38.51
73.07
93.06
ABP
PFP
45.46
62.67
85.03
94.32
LONN
06.78
66.38
85.47
91.14
Table 6: Evaluation of LoRITa compared to SOTA structured pruning/lowe lowwrank training methods that considered ResNet20 and VGG16. The results the structured pruning methods are reported according to Table (He &
Xiao 2023 (most recent survey paper) and ranked according to the FLOPs drop percentage. The last row results are reported from Table in (Xiao et al. 2023).
For ImageNet results on ResNet18, we achieve the best parameters drop while reporting slightly lower FLOPs drop when compared to the second best results. For ResNet34, LoRIT&+ISVT reports the second best FLOPs drop and lightly lower parameters drop when compared to the second best results with only -0. 14% of pruned test accuracy drop.
As mentioned in the introduction section, our numerical experiments focus on examining the effect of LoRITa, the reported results are based LoRITa regularization alone with the simplest SVT post-training pression. Across different architectures and datasets, our performance is competitive with other more so- phisticated methods. It is possible to further boost the performance when LoRITa is used in combination with hthe pruning methods and/or with more advanced post-training compression methods.
6 Conclusion & Future Work
In this study, we studied a compression technique, Low-Rank Induced Training (LoRITa). This thoorctically-justiti technique promotes low-rankness through the composition of lincar layers and achieves compression by employing simple singular value truncation. Notably, LoRITa accomplishes this without ne- cessitating changes to the model structure at inference time, and it avoids the need for constrained additional optimization steps. Furthermore, LoRITa eliminates the requirement to initialize with full-rank pre-trained models or specify rank selection before training. Our experimental validation, conducted on a diverse range of architectures and datasets, attests to the effectiveness of the proposed approach. Through rigorous testing, we have demonstrated that LoRI'a combined with an iterative singular value truncation yields compelling results in terms of model compression and resource efficiency, offering a promising av- enue for addressing the challenges associated with deploying deep neural networks on resource-constrained platforms.
In future works, we plan to explore more efficient fine-tuning methods for LoRlTe-compressed models for which larger models such as LLMs are considered.
Acknowledgements
The work was supported by NSF CCF-2212065 and NSF BCS-2215155. The authors would like to thank Avrajit Ghosh (Michigan State University) for insightful discussions.
References
Manoj Alwani, Yang Wang, and Vashisht Madhavan. Decore: Deep compression with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 12349 12359, 2022.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by oxerparameterization. In International Conference Machine Learning, pp. 244 253. PMT.R, 2018.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit basclines for imagenet-1k: arXiv preprinl ar 2rrx:22205.0555 2022.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. Advances in Neural Information Processing Systems, 34:19637 14:19637-119511, 2021.
Shuxuan Guo, Jose Alvarcz, and Mathicu Salzmann. Expandncts: Linear over-paramcterization to train compact convolutional nelworks. Advances in Neural Information Processing Systems, 33: 13:1298 1310, 2020.
Albert Gural, Phillip Nadeau, Mehul Tikekar, and Boris Murmann. Low-rank training of deep neural networks for emerging memory technology. arXiv preprint ar Xiv:2000 03887, 2020.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-115 110, 2022.
Cole Hawkins, Xing Liu, and Zheng Zhang. Towards compact neural networks via end-to-end training: A bayesian tensor approach with automatic rank determination. SIAM Journal on Mathematics of Data Science, 4(1):46-71, 2022.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Transactions on Pattern Analysis and Mochine Intelligence, 2023.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4340-4349, 2019.
Zhiqiang He, Yaguan Qian, Yuqi Wang, Bin Wang, Xiaohui Guan, Zhaoquan Gu, Xiang Ling, Shaoning Zeng, Haijiang Wang, and Wujie Zhou. Filter pruning via feature discrimination in deep neural networks. In European Conference on Computer Vision, pp. 245-261. Springer, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint trXiv:1704.04880 2017.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103, 10427, 2021.
Yerlan Idelbayev and Miguel A Carreira-Perpinee Low-rank compression of neural nets: Learning the rank each layer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8049 8059, 2020.
Di Jiang, Yuan Cao, and Qiang Yang. On the channel pruning using graph convolution network for convo- lutional neural network acccleration. In IJCAI, pp. 3107-3113, 2022.
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. arXiv preprinl arX:::2105001029, 2021.
Raghuraman Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint Xix:1806 0834,, 2018.
Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, and Qing Qu. Efficient compression of overpa- rameterized deep models through low-dimensional learning dynamics. arXio preprint orXiv:2311.05061 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEF, 86(11):2278 (2(2))2278--222,, 1998.
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVP conference computer vision and pattern recognition, pp. 8018 8027, 2020.
Zhuo Li, Hengyi Li, and Lin Meng- Model compression for deep neural networks: A survey. Computers, (3):60, 2023.
Lucas Licbenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Danicla Rus. Provable filter pruning for efficient neural networks. arXiv preprint 1xXi::9111.04112 2019.
Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Danicla Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Newral Information Processing Systems, 34:5328 5344, 2021.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1538, 2020.
Mingbao Lin, Rongrong Ji, Shaojie Li, Yan Wang, Yongjian Wu, Feiyue Huang, and Qixiang Ye. Network pruning using adaptive exemplar filters. IEEE Transactions on Neural Networks and Learning Systems, 33(12))7357-7333 7366, 2021.
Mingbao Lin, Liujuan Cao, Yuxin Zhang, Ling Shao, Chia-Wer Lin, and Rongrong Ji. Pruning networks with cross-layer ranking & k-reciprocal nearest filters. IEFF Transactions on neural networks and learning systems, 2022.
Shaohui Lin, Rongrong Ji, Chao Chen, Dacheng Tao, and Jiebo Luo. Holistic cmn compression via low-rank decomposition with knowledge transfer. IEEE transactions pattern analysis and machine intelligence, 41(12):2889 2905, 2018.
Guangzhe Liu, Ke Zhang, and Meibo Lv. Soks: Automatic searching of the optimal kernel shapes for stripe-wise network pruming. IEEE Transactions Neurol Networks and Learning Systems, 2022.
Giosué Cataldo Marinó, Alessandro Petrini, Dario Malchiodi, and Marco Frasca. Deep neural networks compression: A comparative survey and choice recommendations. Neurocomputing, 520:152-177 170, 2023.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehlarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523 3542, 2021.
Feiping Nie, Heng Huang, and Chris Ding. Low-rank matrix recovery via efficient schatten p-norm min- imization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pp. 655 661, 2012.
Saiprasad Ravishankar, Jong Chul Ye, and Jeffrey A Fessler. Image reconstruction: From sparsity to data- adaptive methods and machine learning. Proceedings of the IEEE, 108(1))86-100) 2019.
Xiaofeng Ruan, Yufan Liu, Chunfeng Yuan, Bing Li, Weiming Hu, Yangxi Li, and Stephen Maybank. Edp: An efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Transactions on Neural Networks and Learning Systems, 32(10):449945b)) 2021. doi: 10.1109/TNNLS. 2020.3018177.
D. E. Rumelhart, J. L. McClelland, and the PDP Research Group. Parallel Distributed Processing: rulions in the Microstruclure of Cogrilion (Vol. 2). MIT Press, 1986.
Explo-
Fanhua Shang, Yuanyuan Liu, and James Cheng. Unified scalable equivalent formulations for schatten quasi-norms. Xiu preprint 1rXio:1606 006600668, 2016.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks. Advances in Neural Informalion Processing Systems, 34: 24604 24616, 2021.
Yang Sui, Miao Yin, Yu Gong, Jinqi Xiao, Huy Phan, and Bo Yuan. Eirt: Efficient low-rank training for compact convolutional neural networks. arXiv preprint arXi::2401.10341, 2024.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:511106607,, 2015.
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 33:10936- 10947, 2020.
Guanzhong Tian, Yiran Sun, Yuang Liu, Xianfang Zeng, Mengmeng Wang, Yong Liu, Jiangning Zhang, and Jun Chen. Adding before pruning: Sparse filter fusion for deep convolutional neural networks via auxiliary attention. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural information processing systems, 30, 2017.
Roger Waleffe and Theodoros Rekatsinas. Principal component networks: Parameter reduction early in training. Xiviv preprint or 2rXiv:2006.133/ 13347, 2020.
Jinqi Xiao, Chengming Zhang, Yu Gong, Miao Yin, Yang Sui, Lizhi Xiang, Dingwen Tao, and Bo Yuan. Haloc: hardware-aware automatic low-rank compression for compact neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10464-10472, 2023.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-30 2023.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7370-7379, 2017.
Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xic. Autolora: Automatically tuning matrix ranks in low-rank adaptation based meta learning. arXiv preprint urXin:2403.09113, 2024.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenct: An extrenely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. (848-6850) 2018.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
In this Appendix, we first provide a detailed description of the ISVT approach used (Appendix A Then, present the proofs in Appendix B In Appendix C
we review recent works that linear composition for non-compression methods, followed by demonstrating the faster decay of singular values in LoRlTa-trained networks (Appendix D
A Detailed Description of the Iterative SVT Approach
Here, we provide the exact implementation of the ISVT approach we employ in our paper. In each iteration, we fix the number of parameters to truncate. Then, we decide which layer for which these paramcters are to be removed by examining which induces the lowest increase in the training loss after the truncation. particular, for each iteration, begin by fixing the number of parameters truncate, setting this number 500. Next, we select layer, denoted as I, and remove the smallest singular values from this layer until have moved 500 parameters. After this truncation, E()). This process is repeated for all layers to identify the layer 1 that results in the smallest. training loss E()). Once this layer is identified, we truncate 500 parameters from it. We then proceed to the next iteration, truncating another 500 parameters. This iterative process continues until the desired compression rate reached. To alleviate the computational cost, we use only 120 randomly subsampled training data to compute EII).
compute the training loss and store this loss
B Proofs
B.1 Proof of Proposition 4.2
Proof of Proposition 4.2 By Theorem of (Shang al.) 2016), we have
1/2
|
min [A
p
Σ
R
Ip: P/p/Pi/
KE[N]
(9)
s.t. II R A
[E[N]
provided that
(R.) ET:= [T : T,
- 1, T1 TRaT, TN E [R"X))
with r being the rank of A E mp We want to show that this result still holds if we replace the set with
T= ((T) T, m,, n. m. 2 r, and m441 na for i 1, N 1, n, mi m).
In other words, we want to show that
min ET, II R.=A
R. Bp/p.)) Σ SE[N]
1/8
min (RA Hh R.=A
Σ [E[N]
R. pi (p/pp)))
IB/DD:
1/p
where the only distinction between the optimizations on the left and right hand sides of this equality lies in substituting the set T under the min operator with T', so that we allow a more flexible choice of dimensions of the factors. We will prove this equality by separately proving
1/p
|R /p.) 1/P
min ET, IL R
Σ |R VEINI
min ET, IL R.
Σ SE[N]
RR
(10)
19

---

Published in Transactions on Machine Learning Research (11/2024)
and
1/r
R PRIppp// (ppppp)) 1/8 (11)
min E-T [eN R,
R pp/pp/))
min (R R- H.M R,=A
Σ KE[N]
(E[N]
To prove Equation (10). let {Ri E T be a global minimizer of the optimization on the left hand side of Equation 10). We denote by R the augmented matrices obtained by padding/appendingg all-zero rows and columns to cach R. until dimension grows from r m, n4. Then, { is a point in T' In addition, since the 0-padding does not change the product nor the Schatten p-norm of the matrices, plugging {R] 1 into the left hand side optimization of Equation (10) and {R²) y into the right hand side one yields to:
1/r pp/pp)))
R lp:/P:) 1/r =
1R:11p/P))) 1/P
min E-T
Σ kE[N]
R p.
Σ :E[N]
NEIN]
RR (R/1B//1))) 1/P
min ET.
Σ [E[N]
11 11 R,=A
which proves this direction.
To prove Equation [11,, with little abuse of notation, suppose now {R T' represents global minimizer of the right hand side of Equation (1]). Let U E Rmxr be the matrix containing the top left singular vectors of A (A is of rank-r by assumption), then
... R
Next, we want to construct from (Ri E T' new sequence of matrices of different dimensions (R; [R E T which yiclds equal or a smaller objective value than {R; The construction procoods like the following, first define R == UU TURR, V1, where V1 is the matrix containing the first. right eigenvectors of UU R This definition ensures that Ri grxr and || [Ril For i = 2, ... N 1, define R; = VI,,, R;V; e defined in the previous iteration and V, is matrix holding the top-r right singular
Rexr, where V.- vectors of V,-RR as columns. For = N, define R = V V Ry grx=, With this definition, we can verify the following properties for R
(R ET
|R < |R for any i = I,... N, since V, have orthonormal columns
NEN R A, which is due to
A = UUTA A SUTRR .......... R ...
=(UU (V VV-
.R. .
20

---

Published in Transactions on Machine Learning Research (11/2024)
Plugging (R() T' and {R)) E T into the right and left hand sides of Equation 11 respectively yields,
1/2
min I II HEM R,SA
Σ
JR,
SE[N]
|R |RIIP//D 1/P
ic[N]
|
(E[N]
1/8
min R
R. pp: pppp
ET, EENI R.=A
NE[N]
where the first incquality ariscs because R E T may not represent a minimizer, and therefore, its corre- sponding objective function value would be greater or equal to the minimum. The second inequality is due to the second bullet point above, whereas the final equality due the assumption that (R;) T represents minimizer of the right hand side of Equation [1]. This completes the proof of the second direction. ]
B.2 Proof of Proposition 4.5
Proof of Proposition 4.5 The proposition is derived from the scaling ambiguity inherent in the ReLU acti- vation. The the scaling ambiguity allows for the output of the network to remain unchanged when scalar is multiplied by the weight matrix of one layer and the same scalar is divided by the weight matrix of another layer.
Let (W), i E [K], E [Z]], be the minimizer of Equation
Then by taking
1/L
I.
A
p
ai
(12)
one can verify that the rescaled weights,
oup
(B (K), [4]} with B = V 2A
(13)
become the minimizer of Eqquation and the corresponding minimum coincides with the minimum of Equation
Let us first prove that the objective of Equation S evaluated at the rescaled weights & Wi coincides with the objective of Equation (7) evaluated at the original weights W Indeed, since Πe) & = (due to the definition of Br and A), and the scaling ambiguity, the two networks corresponding to the rescaled weights and the original weights have identical output. Consequently, when insert the original and rescaled weights to Equation 77) and Equation (8) respectively, the first terms of the objectives are identical. Additionally, with the chosen value of 31, direct calculations confirm that the second terms in these objectives are also the same. This indicates that rescaling each weight Wi by & maintains the consistency of the objective values.
Next, we show that [& E [K], E [L]} is minimizer (may not be unique) of Equation 8 by contradiction. If it is not one of the minimizers, then there must exist another set of weights ( E [K], E [L]} that achieve lower objective values for Equation (8) This in turn implies that the reversely rescaled weights { Wi, [K], E [L]] by 8, must achieve the same low value for the original objective
7 This contradicts the assumption that (W), [K] [L]] is minimizer of Equation 77)- Thus, the proof is concluded.
function Equation
21

---

Published in Transactions on Machine Learning Research (11/2024)
C Discussion on Linear Layers Composition Methods For Non-Compression Tasks
Here, we review recent studics that utilize lincar composition for non-compression purposes. This means the methods that involve substituting each weight matrix with a sequence of consecutive layers without including any activation functions. The study presented in &Guo et al. 2020) introduced ExpandNet, where the primary objective of the composition is to enhance generalization and training optimization. The authors empirically show that this expansion also mitigates the problem of gradient confusion.
The research conducted in (Khodak et al., 2021 explores spectral initialization and Frobenius decay in DNNs with linear layer composition to enhance training performance. The focus is the tasks of training low-memory residual networks and knowledge distillation. In contrast to our method, this approach employs under-pante paraneterization, where the factorized matrices are smaller than the original matrix. Additionally, the introduced Frobenius decay regularizes the product of matrices in a factorized layer, rather than the individual terms. This choice adds complexity to the training optimization compared to the standard weight decay used in our approach.
The study conducted in (Huh al. 2021 provides empirical evidence demonstrating how linear overparam- eterization of DNN models can improve the generalization performance by inducing a low-rank bias. Unlike our work, they did not consider the role of weight decay in enforcing low-rankness.
D Demonstrating the Faster Decay of Singular Values in LoRiTa-trained Models
In Figure G we empirically demonstrate the faster decay of singular values in LoRTTa-trained models. In particular, Fig Figure (left) (resp. Figure 0 (right)) show the singular values of the first (resp. second) weight matrix of the standard model (N = 1) and LoRITa-trained models (N = 2 and N = 3) for the FCN8 architecture of Table Ia As observed, models trained with LoRITa exhibit faster decay, and increasing promotes faster docay.
Bascling Rodel Ne1 Ludklte N-12
Dasel Model N--1 LoAiTb
Demrmm Sinquler Valuan
LodiTa
LoAiTa N-3
Dormataen coed Strquserr
9.6
0.5
0.6
0.4
0.2
20
SV Indox
5V Index
Figure 6: Empirically showing the faster decay of singular values of the first two weight matrices (layer O (left) and layer 1 (right)) of the standard model (N = 1) vs. LoRlTa-trained models (N = 2 and N = 3) using the FCN8 architecture of Table Ia
22


Reviews:
[{"review_id": "gHBREAMOWd", "reviewer": "Reviewer_wD5i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The method is straightforward, and the manuscript is easy to read.\n- The mathematical formulation in Section 4 is clear.\n- Despite N being limited to 3, Table 1 and Figures 3, 4, and 5 effectively present the motivation for the proposed work.\n\n[Weaknesses]\n- The manuscript does not address the computational overhead for training, particularly when N becomes large. Comparisons may be unfair if the baseline for structured pruning starts from a relatively smaller model, whereas the proposed method might require additional computations during training.\n- The manuscript only uses very small datasets. The authors are strongly encouraged to include experiments using the ImageNet dataset. Without this, it is difficult to assess whether the proposed method is effective beyond small datasets, as CIFAR-10/100 and MNIST are too small compared to ImageNet.\n- Previous works on structured pruning or low-rank training methods include experimental results using ImageNet, which the authors do not provide.\n- For instance, from Tables 2 and 3, it appears that the proposed method is only effective for small datasets or highly redundant models. Conducting experiments with a combination of Vision Transformer and ImageNet would significantly strengthen the claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this manuscript, the authors propose a novel training scheme called LoRITa, which leverages the composition of linear layers and compression via singular value truncation. Using datasets such as MNIST and CIFAR-10/100, the proposed method demonstrates competitive accuracy for Vision Transformers and CNN models. The experimental results obtained with the proposed scheme are compared against established structured pruning and low-rank training methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to weakness part", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DSv9C7atgl", "reviewer": "Reviewer_JhGQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is generally well-written and well-motivated.\n- Improving model serving speed and efficiency is a good research direction.\n- The proposed LoRITa approach generally makes sense.\n\n**Weaknesses:**\n- **[Major]** Similar approaches have been proposed previously, which makes the contribution of this paper a bit limited, but they have not been cited or compared in this paper, e.g., [1-3].\n- **[Major]** The conducted experiments are all on a small scale. It is generally understood that convolutional neural networks trained on small image classification tasks, e.g., MNIST and CIFAR, are easy to compress.\n- **[Major]** Methods like LoRITa essentially manipulate model architectures. In this sense, how does it compare to MobileNet and/or EfficientNet [4-5]?\n- FLOPs do not seem to be a good measurement of inference speed, especially when the data cannot be batchified. What about the latency/throughput of inference?\n- Though it is proposed as future work, I strongly believe compressing LLMs is of more interest and practical need compared to compressing CNNs and ViTs.\n\n[1] https://proceedings.mlsys.org/paper_files/paper/2021/file/94cb28874a503f34b3c4a41bddcea2bd-Paper.pdf  \n[2] https://proceedings.mlsys.org/paper_files/paper/2023/file/c2db3ef0b1ad4c5ec7c3a0a0c6f6c832-Paper-mlsys2023.pdf  \n[3] https://arxiv.org/abs/2202.00834  \n[4] https://arxiv.org/abs/1704.04861  \n[5] https://arxiv.org/abs/1905.11946", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a post-training neural network compression approach, i.e., Low-Rank Induced Training (LoRITa). The major aim of LoRITa is efficient inference. LoRITa works by composing linear layers and compressing them using singular value truncation from randomly initialized model weights. Both theoretical justification and experimental results are provided to demonstrate the effectiveness of the proposed LoRITa method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see \"Weaknesses\" above for requested changes. I summarize the items below:\n- Compare the proposed LoRITa method with those proposed in [1-3].\n- Add MobileNet and EfficientNet experiments.\n- Report inference throughput and latency numbers.\n- Try LoRITa on larger-scale experiments, e.g., ImageNet and GPT pretraining.", "broader_impact_concerns": "I don't find any concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "srbb3yhF1Y", "reviewer": "Reviewer_BQ1P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n1. The authors provide a lot of technical details on the proposed method LoRITa. I think this has helped me a lot in understanding the paper.  \n2. The abstract and introduction provides a clear summary of the paper, with a detailed list of contributions.\n\n\nWeaknesses:  \n1. I feel the way the authors use to reference LoRITa in the experiments is somewhat confusing. Why using N=2 and N=3 for the proposed approach but N=1 for referencing the dense models.  \n\n\n2. The experimental results are not entirely convincing. From Table 2 and Table 3, the improvement over other baseline approaches seem small. Regarding this part, another question is that why the compression ratio (i.e. parameter drop) is different across methods. This makes it hard to compare between LoRITa and baseline approaches.  \n\n\n3. The authors experimented with ViTs on CIFAR-10, but Vision Transformers are typically used on large-scale datasets as they scale with large amounts of data and compute. I think if the authors want to demonstrate the effectiveness of the proposed approach LoRITa on ViTs, it would be better to evaluate ViTs pre-trained on ImageNet.  \n\n\n4. Rather than using singular value decompositions (SVD) on the weights only, have the authors think of more advanced methods, like reduced rank regression? You can try to optimize a low-rank weight matrix that minimizes the reconstruction error of each linear layer. In principle, this should work better than minimizing the l2 distance of the weights.\n\n\n5. “Moreover, LoRITa eliminates the need to (i) initialize with pre-trained models, (ii) specify rank selection prior to training”. I don’t understand this part. Could the authors elaborate on this or provide me a reference into a part in the paper that explains this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced a method to find compressed neural networks via low-rank decomposition of the weight matrices. The proposed method, LoRITa, promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. The authors demonstrate the effectiveness of the proposed approaches compared to many baseline approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses. I would like to see more experimental results to support the proposed approach.", "broader_impact_concerns": "I do not have concerns regarding the broader impacts of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the major comments raised by the reviewers was the lack of experiments on relatively large datasets and neural networks. In response, the authors conducted additional experiments using the ImageNet dataset. All reviewers recommended acceptance after the authors’ response. I concur with the reviewers: the idea of exploiting the low-rank induced training method for model compression is interesting, and the results demonstrate its potential for model compression, though it does not achieve the best results on ImageNet.
Decision: Accept (poster)