ELRT: Towards Efficient Low-Rank Training for Compact Neural Networks
Authors: Yang Sui
Miao Yin
Wanzhao Yang
Yu Gong
Jinqi Xiao
Huy Phan
Ding Ding
Xiaozhong Xu
Shan Liu
Zhenzhong Chen
Bo Yuan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well studied in the literature. On the other hand, low-rank training, as an alternative way to train low-rank CNNs from scratch, is little exploited yet. Unlike low-rank compression, low-rank training does not need pre-trained full-rank models and the entire training phase is always performed on the low-rank structure, bringing attractive benefits for practical applications. However, the existing low-rank training solutions are still very limited and do not demonstrate their effectiveness for training modern low-rank CNN models in the large-scale dataset from scratch. In this paper, we perform a systematic investigation on low-rank CNN training. By identifying the proper low-rank format and performance-improving strategy, we propose ELRT, an efficient low-rank training solution for high-accuracy high-compactness low-rank CNN models. Our extensive evaluation results for training various CNNs on different datasets demonstrate the effectiveness of ELRT.

Under review as a conference paper at ICLR 2023
ELRT: EFFICIENT LOW-RANK TRAINING FOR COM- PACT CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well studied in the literature. On the other hand, low-rank training, as alternative way to train low-rank CNNs from scratch, little exploited yet. Unlike low-rank compression, low-rank training does not need pre-trained full-rank models and the entire training phase is always performed on the low-rank structure, bringing at- tractive benefits for practical applications. However, the existing low-rank training solutions are still facing several challenges, such as considerable accuracy drop and/or still needing update full-size models during the training. In this paper, we perform a systematic investigation on low-rank CNN training. By identify- ing the proper low-rank format and performance-improving strategy, we propose high-compactness
ELRT, an efficient low-rank training solution for low-rank CNN models. Our extensive evaluation results for training various CNNs on different datasets demonstrate the effectiveness of ELRT.
INTRODUCTION
Convolutional neural networks (CNNs) have obtained widespread adoption in numerous real-world computer vision applications, such as image classification, video recognition and object detection. However, modern CNN models are typically storage- inteeniv and computation-intenstt tially hindering their efficient deployment in many resource-constrainee scenarios, especially the edge and embedded computing platforms. To address this challenge, many prior efforts have been proposed and conducted to produce low-cost compact CNN models. Among them, low-rank com- pression is popular model compression solution. By leveraging matrix or tensor decomposition techniques, low-rank compression aims to explore the potential low-rankness exhibited in the full- rank CNN models, enabling simultaneous reductions in both memory footprint and computational cost. To date, numerous low-rank CNN compression solutions have been reported in the literature (Phan et al. (2020): Kossaifi al. (2020) Liet al. (2021b): Liebenwein al. (2021)).
poten-
Low-rank Training: A Promising Alternative Towards Low-rank CNNs. From the perspec- tive of model production, performing low-rank compression the full-rank networks is not the only approach to obtaining low-rank CNNs. In principle, we can also adopt low-rank training strat- egy directly train a low-rank model from scratch. As illustrated in Fig. [ low-rank training starts from a low-rank initialization and keeps the desired low-rank structure in the entire training phase. Compared with low-rank compression that is built on two-stage pipeline ("pre-training-lie -thnn compressing"), "), the single-stage low-rank training enjoys two attractive benefits: relaxed opera- tional requirement and reduced training cost. More specifically, first, the underlying training from- scratch scheme, by its nature, completely eliminates the need for pre-trained full-rank high-accuracy models, thereby lowering the barrier to obtaining low-rank CNNs. In other words, producing low- rank networks becomes more feasible and accessible. Second, the overall computational cost for the entire low-rank CNN production pipeline is significantly reduced. This is because: 1) the removal of the pre-training phase completely saves the incurred computations that were originally needed for pre-training the full-rank models; and 2) directly training on the compact low-rank CNNs naturally consumes much fewer floating point operations (FLOPs) than full-rank pre- pre-training.
Existing Works and Limitations. Despite the above analyzed benefits, low-rank training is cur- rently little exploited in the literature. Unlike the prosperity of studying low-rank compression, to

---

Under review as a conference paper at ICLR 2023
Pre-Train
Decompose
B
Fine-tune
B
A
Initial Full Rank Model
Pre-trained Full Rank Model
Low-rank
Low-rank Model
Structure
Low-rank Compression
Train from scratch
g
T
U
Keep Low-rank Structure
Initial Low-rank Model
Low-rank Model
Low-rank Training
Figure 1: Different paths towards producing low-rank CNN models.
date very few research efforts have been conducted towards efficient low-rank training. (loannou jet al. (2015): Tai et al. (2013) are the pioneering works in this research direction; however the ob-
tained low-rank models suffer considerable accuracy drop. In addition, the corresponding training
methods are not evaluated on the modern CNNs such as ResNet. Recently, (Gural et al. (2020);
Hawkins et al. (2022) propose emerging memory-aware and Bayesian estimation-based low-rank training. respectively: however, these two method are either built on costly repeated SVD operations
(Gural et al. (2200) or Bayesian estimation (Hawkins et al. (2022), ). which are very computation
intensive and potentially not scalable for practical deployment. (Hayashi et al. (2019); Khodak et al.)
(2020) Su et al. (2022) propose to learn the suitable low-rank format and/or apply spectral initial- ization during the training. However, the trained models with the new format have considerable accuracy drop, even on CIFAR-I0 dataset. (Waleffe & Rekatsinas (2020); Wang et al.] (202T) per- form several epochs of full-size training to mitigate this issue. However, with the cost of increasing memory and computational overhead, this racy drop. And it is essentially not training low-rank model from scratch. Therefore, the satisfactory answer to the following fundamental question is still missing:
strategy still brings a considcrable accu-
Fundamental Question for Low-rank Training: What is the proper segtninn-fromsccreeee solution that can produce modern low-rank CNN models with high accuracy on the large-scale dataset, even outperforming the state-of-the -hte-rrr-tth-rtt--e low-rank compression methods?
Technical Preview and Contributions. To answer this question and put the low-rank training tech- nique into practice, in this paper, we perform systematic investigation training low-rank CNN models from scratch. By identifying the proper low-rank format and performance-improvte egy, propose ELRT, an efficient low-rank training solution for high-accuracy high-compactness CNN models. Compared with the ttate-of-the-as low-rank compression approaches, ELRT demon- strates superior performance for various CNNs on different datasets, demonstrating the promising potential of low-rank training in practical applications. Overall, the contributions of this paper are summarized as follows:
strat-
We systematically investigate the important design knobs of low-rank CNN training from scratch, such as the suitable low-rank format and the potential performance-improving strategy, understand the key factors for building high-performance low-rank CNN training framework.
Based on the study and analysis these design knobs, we develop ELRT, an orthogonality- aware low-rank training approach that can train high-accuracy high-compactness low- tensor-rank CNN models from scratch. By enforcing and imposing the desired orthogo- nality the low-rank model during the training process, significant performance improve- ment with low computational overhead can be obtained.
We conduct empirical evaluations for various CNN models to demonstrate the effective- ness of ELRT. On CIFAR-10 dataset, ELRT can train low-rank ResNet-20, ResNet-56 and MobileNerV2 from scratch with providing 1,98x, 2.05 and 1.71x FLOPs reduction, re-
spectively; and meanwhile the trained compact models enjoy 0.48%. 0.70% and 0.29% accuracy increase over the baseline. On ImageNet dataset, compared with the state-of- the-art approaches that generate compact ResNet-50 models, ELRT achieves 0.49% higher
2

---

Under review as a conference paper at ICLR 2023
accuracy with the same even higher inference and training FLOPs reduction, respec- tively.
2
RELATED WORKS
Low-rank Compression. As an important type of model compression strategy, low-rank compres- sion aims to leverage low-rank decomposition techniques to factorize the original full-rank neural network model into a set of small matrices or tensors, leading to storage and computational sav- ings. Based on the adopted factorization methods, the existing low-rank CNN compression works can categorized into 2-D matrix decomposition based (Tai et al. (2015); Li & Shil (2018); Xu eet al. (2020); [delbayev & Carreira-Perpinee (2020); Yang al. (2020); Liebenwein
(202
and high-order tensor decomposition based (Denton et al. (2014); ; Kim et al. (2015); Novikov et al.
(2015); Yang et al. (2017); Wang et al. (2018); Kossaifi et al. (2019) Phan et al. (020); Kossaifi
et al.
2020);
Li et al. (2021a) Lin in et al. (020bb: | Yu et al. (2021).
Low-rank Training. Similar low-rank compression, the goal of low-rank training is also to pro- duce compact neural network models with low-rankness; while the key difference is that low-rank training initializes and updates the low-rank CNNs during the entire training process. In other words, the pre-trained full-rank models are not required in this scenario, and the CNN models being updated always kept in the low-rank format. To date efficient low-rank training approaches are still little exploited. More specifically. the existing works either have considerable accuracy loss (Ioannou al.] (2015); Tai et al. (2015) Hayyssi al. (2019); Khodak et al. (2020): [Su et al. (2022) or suf- fer high computational overhead because of the use of costly SVD operations (Gural et al. (2020)
Bayesian estimation (Hawkins al. (2022)) or only performing partially low-rank training (Waleffe
& Rekatsinas 2020); Wang et al. (2021)) limiting their effectiveness in the practical scenarios.
Unstructured & Structured Sparse Training. Low-rank training is essentially a type of compression-awarr training solutions, which include another related strategy as sparse training. Sparse training can be performed the unstructured (Lee et al. (2018); Wang et al. (2019); Evci et al. (2020); Mostafa & Wang (2019); et al. (2020); Mocanu al. (2018) Bellec et al. (2018) and structured (Yuan et al. (2021); Zhou et al. (202I)) ways, corresponding to training/obtaiming unstructured and structured sparse CNN models, respectively. From the perspective of model de- ployment, structured sparse training is more practical and important than its unstructured counterpart
because can produce structured sparse CNN models that exhibit considerable speedup on the off- the-shelf GPU/CPU platforms.
Orthogonality in CNN Training. Another line of works that are related to this paper is training orthogonal full-size full-rank CNNs (Rodriguez et al. (201)); Huang et al. (2018); Miyato et al. Bansal et al. (2018); [Wang al. (2020)). Based on the observation that orthogonality in CNN weights can stabilize the distribution of activations and bring efficient optimizations, these prior efforts explore different methods to enforce orthogonality on convolutional layer in both ini- tialization and training phases. Compared with them, ELRT has two key differences. First, the ex- isting orthogonal CNN training works are performed on full-rank full-size models, and they cannot bring any memory FLOPs reduction. Instead, ELRT aims to train compact low-rank CNN mod- els from scratch, naturally bringing reduced inference and training costs. Second, orthogonal CNN training directly enforces orthogonality on the entire convolutional layer. The underlying motivation is mainly based on experimental observation (e.g., stabilizing the distribution of activations). On the other hand, ELRT focuses exploring the orthogonality of the decomposed components (e.g., factor matrices) of the weight tensors. Such strategy is naturally consistent with the requirement of low-rank theory the decomposed matrix/tensor should exhibit self-orthogonality after SVD or tensor decomposition.
Bansal
3 PRELIMINARIES
Notation. Throughout this paper the d-order tensor, matrix and vector are represented by boldface calligraphic script letter X E [R"1xngx"" Xm2xxxx4 boldface capital letters X R xg2, and boldface lower-case letters x E R. respeectiil. Also, X i and X,, denote the entry of tensor X' and matrix X, respectively.
3

---

Under review as a conference paper at ICLR 2023
Tucker-2 Format for Convolutional Layer. As will be analyzed in Section 4 in this paper choose to use low-tensor-rank format (e.g., Tucker-2) to form efficient low-rank training approach. In general. given convolutional layer W R nh x XCot xk it can be represented in Tucker-2 format as follows:
W P.g.i
Σ 11-1
9 Gran
U U Fu 12
where g is the 4-D core tensor of size 1 22 K K, and U E R and U(2) B are the factor matrices. In addition, +1 and 2 are the tensor ranks that determine the complexity of compact convolutional layer.
Computation on the Tucker-2 Convolutional Layer. Given the above described Tucker-2 format representation, the corresponding execution on this compact convolutional layer can performed via consecutive computations as:
Cin
K
K
+
U X rr.p p.b,ws
The Tr
9 Gri.zs. r1.y
Fi.
N,
p=1
j=1m==
4/2
V y, se!
T r2.9
12=1
where E C and y E Cou X xW" layer, respectively. For indices of T h1 = stride (h' - 1) + (w' - 1) + padding. In addition, T and T are the incurred intermediate results.
are the input and output tensor of the convolutional
padding, Wj = stride x
PROPOSED LOW-RANK TRAINING SOLUTION
As outlined in Section to date the efficient training for high-accuracy low-rank CNN models from scratch is still largely under-explored. In this section we propose to systematically explore several important design knobs and factors when training low-rank CNN models from scratch. Based on the outcomes from these analytic and empirical studies, we will then further develop efficient solution for low-rank CNN training from scratch.
Questions to be Answered. To be specific, in order to obtain better understanding of low-rank training and improve performance, we explore the answers to the following three questions.
Which type of low-rank format is more suitable for efficient training from scratch, 2-D matrix or high-order tensor?
Questionl:
Analysis. In general, when training compact CNN model from scratch, there are two types of low-rank formats that can be con- sidered. The 4-D weight tensor of the trained convolutional layer can be either in the format of low- matrix-rank 2-D matrix or a high- order Now-tensor-rank tensor. As il- lustrated in Fig. [2] the low-matrix- rankness means the flattened and ma- tricized 4-D weight tensor exhibits low-rankness; while the low-tensor- rankness means the trained convo- lutional layer can be directly rep- resented and constructed via mul- tiple small-size factorized matri- ces/tensors without any flattening op- erations.
(Gn Coue KK)
Cin +)
(1b Cour KK
Reshape
Matrix
R
A
(Cn, Cour X xXx))
Decomposition:
Tucker
(#, Cnn) (0, x @ x X)) (4, X Cova)
Weight
yta
Decomposition
Factor
Core Tensor
Factor Matrix
Figure 2: A low-rank CONV layer can either exhibit low-
matrix-rankness (Top) or now-tensor-ranknee (Bottom).
Our Proposal. Currently most of the existing low-rank training works Yang al. 22000; loan- nou et al. (2015); Tai et al. (2015)) conduct and keep 4-D convolutional layer trainig in the for- mat of low-rank 2-1) matrix. Instead, propose perform low-rank CNN training directly
4

---

Under review as a conference paper at ICLR 2023
in the high-order tensor format. In other words, each convolutional layer always stays in the low-rank tensor decomposition format, e.g., Tucker (Tucker (1966) or CP HHtchhoock (1927)
during the entire training phase. Our rationale for this proposal is that, un- like low-rank matrix format that may lose the important spatial weight cor- relation incurred by the inevitable flattening operation; the low-rank tensor format is more natural way to represent 4-D weight tensor of con- volutional layer; and therefore it can better extract and preserve the weight information and correlation existed in the 4-D space (Liu et al. (2012) For instance, as illustrated in Fig. when representing the same weight tensors of layers in ResNet-20 model, the low-rank Tucker format enjoys smaller approximation error than the low-rank matrix format with the same number of weight parameters. En- couraged by this better representation
6.0
Approximation Error
8.5
Approximmatin coroxiittiinn Keroe
5.0
4.5
4.5
4.0
3.5
Tucker-D
Tucker-D
Compression Ratio
Compression Ratio
(a) The 15-th convolutional layer.
(b) The 16-th convolutional layer.
Figure 3: Approximation error (Mean Square Error (MSE) of low-matrix-rank and low-tensor-rank methods for ap- proximating ResNet-20 layers. Notice that MSE measure- ment is our analysis and exploration to identify the suitable low-rank format. It is not actually executed during training.
capability for the high-order weight tensors, we choose to train low-rank tensor format CNN from scratch.
Question2: Consider low-rankness implies potentially low model capacity, what is proper stralegy to improve the performance of low-rank training?
Analysis. As analyzed in Section T a key challenge of the existing low-rank training works TTa et al. (0155; Ioannou et al. (2015)) is their inferior model accuracy as compared to model compression approaches. We hypothesize that this phenomenon is caused by two reasons: 1) low-rank training starts from a low-accuracy random initialization with low-rank constraints; while model compres- sion is built on pre-trale treitreied high-accuracy model; and 2) the entire procedure of low-rank training constrained the low-rank space, thereby limiting the growing space for model capability, No- tice that possible solution is that we can reconstruct and train full-rank model in some epochs, and then decompose the model to low-rank format again during the training procedure. However, such "train treai-ddccompss-ttraae scheme very costly since needs perform computation-inten -intenssiveeeieiiie tensor decomposition many times.
Our Proposal. To improve the performance of low-rank training with preserving low computational and memory costs, we propose to perform orthogonality aware aware low-tensor-rante training. Our key idea impose and enforce the orthogonality on the factor matrices (1)) and ( during the entire training process, and such training philosophy lies in the following rationale: It is well known that in principle using orthogonal- format basis can maximize the capacity of information representation.
For instance, when we aim to ap- proximate a full-rank matrix with low-rank SVD factorization, the de- composed U and V always exhibit self self-orthogonality (as unitary matrix) with the smallest approximation er- A similar phenomenon also ists for tensor decomposition, where the factor matrices U(!) and U(2) are also unitary matrices after per- forming Tucker decomposition on the original full-rank tensor. Inspired by above observations, when we aim to approach a high-capacity full-rank model using the low-rank format, the orthogonality-baa based representation is
1.1
59
Test A.sumazy
with S0
07
witbout 90
so so 100
120 140
160 100 200
Epochs
Epochs
(a) Training loss.
(b) Test accuracy.
Figure 4: Training loss (left) and test accuracy (right) for low-tensor-rank ResNet-20 CIFAR-10 with/without SO regularization. Same ranks are used for different experi- ments. Ranks are selected to provide 2x FLOPs reduction.
5

---

Under review as a conference paper at ICLR 2023
R
Regularization scope
(U
Residual Matrix
2
so
minimize
minimize
(2 of
W
F
Σ
2
R₁
C
of)
(U
DSO
(
|
minimize
2
+
2
minimize
yra)
SRIP
minimize
R
minimize
max (+i
(UiRlTT
MC
U
minimize
R
minimize
K
B
LSESR
0
Figure 5:
The mechanism of different approaches to impose orthogonality on U From top to bottom: (a) Soft Orthogonal Regularization, (b) Double Soft Orthogonal Regularization, (c) Spectral Restricted Isometry Property Regularization, (d) Mutual Coherence Regularization.
very suitable solution. However, such orthogonality cannot be automatically obtained and en- sured during the low-rank training. Therefore, an efficient mechanism of enforcing and ensuring orthogonality is very desired.
To verify the potential effectiveness of the proposed orthogonality-wwrr idea, we perform pre- liminary ablation study of low-tensor-rank training with and without enforcing orthogonality. Here a ResNet-20 model trained from scratch on CIFAR-10 dataset, and simple soft orthogonal reg- ularization term (Xie et al. 2017); Bansal et al. (0018) used to impose orthogonality on U(!) and u during the training. As shown Fig. enforcing orthogonality during the training can significantly improve training and testing performance.
Question3: How should we property impose the orthogonality during the low-tensor-rank training?
Analysis. The preliminary experiment in Fig. shows the encouraging benefits of orthogonality- aware low-tensor-ee rank training. To fully unlock its promising potentials, efficient scheme for impos- ing the desired orthogonality should be properly explored. To be specific, there four commonly used regularization approaches (Bansal et al. (2018)) that can introduce the orthogonality on the target factor matrices:
Soft Orthogonal (SO) Regularization:
R. (A)
P $2
A
1
(3)
where || || is the Frobenius norm, p is regularization strength, A is the matrix to be enforced with orthogonality, and t is the rank A. Notice the experiment in Fig. 4 addopt this approach.
Double Soft Orthogonal (DSO) Regularization:
(A)
62 (IIA
1 + AAT -112).
(4)
Here compared with SO regularization, this DSO scheme can always impose the proper orthogonal- ity on A no matter it is over-complete or under-complete.
Mutual Coherence (MC) Regularization:
Rme (A) P(IIATA Illoc).
(5)

---

Under review as a conference paper at ICLR 2023
where || b/ is the matrix norm induced by the lo norm.
Spectral Restricted Isometry Property (SRIP) Regularization:
Rap( = p (A I),,
(6)
where (R) "
R= is the spectral norm of A I (Yohhida & Miyato 2017
based RIP condition (Candes & Tao (2005)).
Our Proposal. We propose to develop DSO regularized scheme to impose the desired orthogonality on the factor matrices. Here main rationale is that DSO regularization can better provide the desired orthogonality. To be specific, as illustrated in Fig. S the orthogonality of the factor matrix
((2) can be measured by its corresponding residual matrix R U U I. Here when R. has lower energy, it is more likely that U will exhibit more self-orthogonnaliyy Therefore, the essential mechanism of SRIP and MC is to minimize the maximum singular value of R and the maximum row sum norm of R. respectively, to push R close all-zero matrix. Evidently, such constraint posed by SRIP and MC schemes inherently targets the local components of R. thereby de- grading the overall effect of orthog- onality. On the other hand, DSO and SO schemes aim to push all the singular values of R. to zero in a global way. thereby in principle bringing stronger orthogonality for U(2), In addition, since U(1) and U may not be square matrix in prac- tice. DSO scheme, as approach that simultaneously considers the po- tential completnnss and over- completeness of matrix, is more
Algorithm 1: Overall ELRT training procedure
Input: Dataset D. pre-set ranks (Ø). Tucker-2-forma weights in each layer
U( RCxxC g E R x U(2) E R #2xCuue,
orthogonal parameters p. Xa, training epochs T.
Output: Trained (U()) U g).
Initialize: xavier uni uniform((U)) U .g).
for = to ''' do
X, y sample.batch(
y forward(X (U(), U g}) via Fq. 2
Orthogonal regularization
Ra(U(2) 2 ([[U(t)
o7 (1100)0@000) -0 000
Ra(U(z) + $1 =
loss + (())3) +a(aRa(( + R.()()))
update ((U0)) ( 9), loss);
end
general and better solution than SO scheme. To verify this hypothesis, we also perform ablation study for low-tensor-rank training using different orthogonal regularization schemes. As reported the Appendix B DSO scheme shows consistently better performance than other schemes, and hence it is adopted in our proposed low-rank training procedure.
Overall Training Procedure. Based on the above analysis and proposals, we summarize them and develop the corresponding efficient low-rank training (ELRT) algorithm to train high-performance high-compactness low-tensor rank CNN model from scratch. Algorithm describes the details.
EXPERIMENTS
Datasets and Baselines. We evaluate our approach on CIFAR-10 and ImageNet datasets with dif- ferent inference and training FLOPs reduction ratios (e.g., 2x FLOPs reduction ratios refers to 50% FLOPs reduction). For experiments on CIFAR-I0 dataset, the performance of ELRT for VGG-16, ResNet-20, ResNet-56 and MobileNetV2 are evaluated and compared with the results using com- pression and structured sparse training methods. For experiments on ImageNet dataset, we evaluate approach for training ResNet-50.
Calculation of Inference and Training FLOPs Reduction. A very important benefit provided by compression-awant training, e.g., low-rank training and sparse training, is simultaneously achieving both the "Inference FLOPs Reduction" and "Training FLOPs Reduction". The details of calculation mechanism for these two metrics described in the Appendix
Hy Hyperparameter. We use SGD optimizer for training with batch size, momentum and weight decay 128, 0.9 and 0.0001, respectively. The learning rates are 0. on CIFAR-10 dataset and 0.05
7

---

Under review as a conference paper at ICLR 2023
on ImageNet dataset, respectively, with the cosine scheduler. All experiments performed via using PyTorch 1.12 and following the PyTorch official training strategy. The detailed configurations for the tensor rank values in each layer are reported in the Appendix
Table I: Results for VGG-16, ResNet-20, ResNet-56 and MobileNetV2 on CIFAR- 10 dataset. "*" denotes compression ratio since the corresponding work does not report FLOPs reduction.
Always Train Direct Train Post-Train Top-1 Inference Training
Method
Compact Model (No Pre-train)
Model
(%) FLOPs FLOPs
VGG-16
Original VGG-16
Dense
1.00
1x
LREI. (delhayev & Carreira Perpinán
2020
X
Low-rank 92.72
6.88x
<Ix
ALDS Lichenwein ef al. 2021
X
Low-rank 92.67
7.26x
<1x
TETD Yin ct 2021
X
X
Low-rank 93.11
x.37x
<1x
GrowBfhccent Yuan et al
2021
V
Sparse 92.50
7.35x
1.22x
BackSparse (Zhou cf
V
Sparse 92.50
11.5x
8.69x
HRank (Lin
X
Sparse 92.34
2.24x
<1x
CHIP (Su
X
Sparse 93.18
4.67x
<1x
ELRI(UUrS)
Low-rank 93.30
9.16x
9,16x
ELRT (Ours)
Low-rank 92,99
12.4x
12.4x
ResNet-20
Original ResNet-20
X
V
Dense 91.25
1.00x
1x
SVDT Yang al 2020
X
Low-rank 90.39
2.94x
<1x
PSTRN:S LL al 2021a
X
Low-rank 90.80
2.25x*
<1x
ALDS (Lichenwein of al.
2021
x
Low-rank 90.92
3.11x
<Ix
(delbayev & Careira-Perminán Perpinan 2020
Low-rank 90.20
3.00x
<1x
GrowEtheient Yuan
Sparse 90.91
2.00x
1.13x
BackSparse (Zhou al. 2021
V
Sparse 90.93
2.77x
2.09x
ELRT (Ours)
Low-rank 91.73
1.98x
1.98x
ELKT (Ours)
Low-rank 91.26
2.51x
2.51x
ELRT (Ours)
Low-rank
90.95
3.02x
3.02x
ELRT (Ours)
Low-rank 6.01
3.87
ResNet-56
Original ResNet-56
/
Dense 93.26
1.00x
1x
SVDT Yang ul. 2020
X
Low-rank 93.17
3.75x
<1x
TRP Xu
X
Low-rank 92.63
2.43x
<1x
ENC.Inl (Km
2019
X
Low-rank 93.00
2.00x
<Ix
CaP (Miniehan & Savakis 2019
X
X
Low-rank 93.22
2.00x
<1x
CC (Li ct [2021b]
X
Low-rank 93.64
2.08x
<1x
ELRT (Ours)
Low-rank 93.96
2.05x
2.05x
ELRT (Ours)
Low-rank 94.01
2.52x
2,52x
EIRT (Ours)
Low-rank 93.67
3.01x
3.01x
MobileNetV2
Original MobileNetV2
Dense 94.48
1.00x
1x
GAL (Lin al. 2019
Sparse 93.07
<Ix
DCP
Zhuang
2018
X
Sparse 94.25
1.36x
<Ix
SCOP Tang et al.
X220
X
Sparse 94.24
1.67x
<1x
ELRT (Ours)
Low-rank
94.87
1.47x
1.47x
ELRT (Ours)
Low-rank
94.77
1.71x
1.71x
VGG-16, ResNet-20, ResNet-56, MobileNetV2 on CIFAR-10. As shown in Table for ResNet- 20 model, the proposed ELRT can bring 0.48% accuracy increase over baseline model with 1.98x
inference and training FLOPs reduction. For ResNet-56 model, ELRT method can bring 0.75% accuracy increase over baseline model with 2.52x inference and training FLOPs reduction. We also compare ELRT with several compression approaches on MobileNetV2 model. ELRT brings 1.71 inference and training FLOPs reduction with 0.29% accuracy increase over the baseline. Notice that similar to TETD, HRank and CHIP, we add BN layers in VGG-16 to stable the training process.
Re on ImageNet. Table summaries the performance of different methods for generating the compact ResNet-50 model on ImageNet. Compared with the state-of-the-ar approaches, ELRT achieves least 0.49% accuracy increase with 2. inference and training FLOPS reduction.
Benefits of ELRT on Training FLOPs Reduction. From Table I and it is seen that ELRT enjoys two benefits on low training cost. First, unlike low-rank compression or pruning, ELRT can reduce overall training cost because it eliminates the need of pre-training phase. Second, unlike structured sparse training, e.g., GrowEfficien and BackSp ELRT has the same FLOPss
Remark:
8

---

Under review as a conference paper at ICLR 2023
reduction in both inference and training. This is because structured sparse training consumes tra computation to calculate the channel mask during backward propagation, which is not needed in low-rank training. More details of calculation of Inference and Training FLOPs reduction for different methods are reported the Appendix
Table Results for ResNet-50 on ImageNet dataset.
Always Train Direct Train Post-Train Top-1 Top-SInferenc Training
Method
Compact (No Pre-train)
Model
(%)
(%)
FLOPs FLOPs
ResNet-50
Original ResNet-50
Dense
76.15 92.87
1.00x
AutoP (Luo & Wu (2020)
Sparse
74.76 92.15
1.95x
<1x
PruneTrain (Lym al. 2019
Sparse
74.62
N/A
2.13x
1.67x
HRank [Lin al. 20
Sparse
74.98 92.33
1.78x
<1x
SCOP Tang ct 2124
Sparse
75.26 92.53
2.21x
<Ix
CHIP Sui et al. [2021
Sparse
75.26 92.53
2.68x
<Ix
ISP (Ganddanesh al.
2022
Sparse
75.97 92.74
2.31x
<1x
SVDT Yang el al.
2020
Low-rank
N/A 91.81
1.79x
<1x
TRPNu XXI ot al. 20201
Low-rank
74.06 92.07
1.80x
<Ix
Stable [Phan el al.
2021
Low-rank
74.68 92.16
2.64x
<Ix
Hinge et et al. 2020)
Low-rank
74,70 N/A
2.15x
<Ix
CC
(Li ct al.
2021b
Low-rank
75.59 92.64
2.12x
<Ix
CC
al.
2021b
Low-rank 74.54 92.25
2.68x
<Ix
GrowEllicres Yuan et al.
2021
Sparse 75.20 N/A
1.99x
1.10x
BackSparse
(Zhou et al. )
Sparse 76.00 N/A
2.13x
1.60x
ELRT (Ours)
Low-rank
76.49 93.28
2.19x
2.19%
ELRT (Ours)
Low-rank 76.11 92.88
2.49x
2.49
ELRT (Ours)
Low-rane 75.32 92.57
2.91
2.91
Comparison with Existing Low-rank Training and Directly Training Small Model. Because the existing low-rank training works (Tai al. Ioannou et al. (2015): Hawkins et al. (2022)
evaluated on small dataset (MNIST) and/or non-popularly used models, we report their comparison
with ELRT in the Appendíx A.3 Also, we compare ELRT with directly training a small dense
models with the same target model size, and the results are reported in the Appendix A.44
Ablation Study. We perform ablation study on the effect of imposing orthogonality and different orthogonal regulation schemes, the details are reported in the Appendix
Practical Speedup of Low-Tensor-Rant CNNs. To demonstrate the practical effectiveness of low- tensor-rank models, we measure the inference time of the models trained by using ELRT. Here we evaluate the speedup on four hardware platforms: Nvidia V100 (desktop GPU), Nvidia Jetson TX2 (embedded GPU), Xilinx PYNQZI (FPGA) and Eyeriss (ASIC). The evaluated models are low- rank ResNet-50 for ImageNet with three target FLOPs reduction settings. As summarized in Table 3. the low-rank CNNs obtained from ELRT enjoy considerable speedup across different hardware platforms.
Table 3: Runtime (per image) for the low-rank ResNet-50 trained via our proposed ELRT.
FLOPs Reduction
ASIC Eyeriss 65nm
FPGA Xilinx PYNQZI
Desktop GPU
Embedded GPU
Nvidia V100
Nvidia Jetson TX2
Original ResNet-50
1x
38. 10ms (1x)
172.0ms (1x) 0.8145ms (1x)
29. 29.46ms (1x)
ELRT (Ours)
2.19x
19,24ms (1.98x) 85.62ms (2.01 x) 0.6934ms (1.17x)
21.47ms (1.37x)
ELRT (Ours)
2.49
15.96ms (2.39x) 71.08ms (2.42x) 0.6241ms (1.31x)
19.24ms (1.53x)
EIRT (Ours)
2.91
13.85ms (2.75x) 61.02ms (2.82x) 0.5963ms
(1.37x)
18.29ms (1.61x)
CONCLUSION
This paper proposes ELRT, efficient low-rank training approach for training CNN models from scratch. ELRT is essentially an orthogonality training solution that can impose and ensure the important orthogonality the low-rank models during the training process. Evaluation results demonstrate the effectiveness of ELRT as compared to state-of- of-the-art model compression and other compression-awant training approaches.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regu- larizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
Atilim Gunes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. Journal of Marchine Learning Research, 18:1-43, 2018.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018.
Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions information theory, 51(12):4203-422 2005.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Informa- tion Processing Systems, pp. 1269-1277, 2014.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952.
PMLR, 2020.
Alireza Ganjdanesh, Shangqian Gao, and Heng Huang. Interpretations steered network pruning via amortized inferred saliency maps. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Albert Gural, Phillip Nadeau, Mehul Tikekar, and Boris Murmann, Low-rank training of deep neural networks for emerging memory technology. arXiv preprint arXiv:2009.03887, 2020 2020.
Cole Hawkins, Xing Liu, and Zheng Zhang. Towards compact neural networks via end-to-en end training: A bayesian tensor approach with automatic rank determination. SIAM Journal on Math- ematics of Data Science, 4(1):46-71, 2022.
Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, and Shin-ichi Maeda. Exploring unexplored ten- sor network decompositions for convolutional neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. Proceedings of the Twenty-Sene -Sventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 2234 2240, 2018.
Yang He. Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Frank L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics, 6(1-4):164-189 1927.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Yerlan Idelbayev and Miguel A Carreira-Perpinat Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8049-8059, 2020.
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv: 1.077444, 2015.
Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regular- ization of factorized neural layers. In International Conference on Learning Representations, 2020.
10

---

Under review as a conference paper at ICLR 2023
Hyeji Kim, Muhammad Umar Karim Khan, and Chong- Min Kyung. Efficient neural network com- the IEEE/CVF conference on computer vision and pattern recognition,
pression. In Proceedings the pp. 12569-12577, 2019.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression deep convolutional neural networks for fast and low power mobile applications. arXiv preprint 1xXiv:1511.065300 2015.
Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. T-net: Parametrizing fully convolutional nets with single high-order tensor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. pp. 7822-7831, 2019.
Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M Hospedales, and Maja Pantic. Factorized higher-order cnns with an application to spatio-temporal emotion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6060-6069, 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Leurning Representations, 2018.
Chong Li and CJ Shi. Constrained optimization based low-rank approximation of deep neural net- works. In Proceedings of the European Conference Computer Vision (ECCV), pp. 732-747, 2018.
Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems, pp. 1-15, 2021a.
Yawei Li, Shuhang Gu Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8018-8027, 2020.
Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6438-6448 6484,, 2021b.
Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL /
wtt.:ewwwwweeeetouu.nc.ooo JkwMhy Inm.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint 1rXi::1112.4400, 2013.
Minghao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529-1538, 2020a.
Rui Lin, Ching- Yun Ko, Zhuolun He, Cong Chen, Yuan Cheng, Hao Yu, Graziano Chesi, and Ngai Wong. Hotcake: Higher order tucker articulated kernels for deeper enn compression. 2020 IEEE 15th International Conference on Solid-State & Integrated Circuit Technology (ICSICT),
pp. 1-4. IEEE, 2020b.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adver- sarial learning. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 2790-2799, 2019.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arxiv preprint arXiv:2006.0723. 2020c.

---

Under review as a conference paper at ICLR 2023
Ji Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. Tensor completion for estimating missing values in visual data. IEEE transactions on pattern analysis and machine intelligence, 35(1):208-23) 220, 2012.
Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So. Dynamic sparse train- ing: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005. 06870, 2020.
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:107461, 2020.
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceed- ings of the International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-13, 2019.
Breton Minnehan and Andreas Savakis. Cascaded projection: End-to-end network compression and acceleration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10715-10724, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 11): 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646-4655. PMLR, 2019.
Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In The European Conference on Computer Vision (ECCV), 2020.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in Neural Information Processing Systems, pp. 442-450, 2015.
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavský, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor de- composition for compression of convolutional neural network. In European Conference on Com- puter Vision, pp. 522-539. Springer, 2020.
Pau Rodriguez, Jordi Gonzalez Sabate, Guillem Cucurull, Josep Maria Gonfaus Sitjes, Francesc Xavier, and Roca Marvà. Regularizing enns with locally constrained decorrelations. Interna- tional Conference on Learning Representations (ICLR), 2017.
Jiahao Su, Jingling Li, Xiaoyu Liu, Teresa Ranadive, Christopher Coley, Tai-Ching Tuan, and Furong Huang. Compact neural architecture designs by tensor representations. Frontiers in arti- ficial intelligence, 5, 2022.
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks. In Advances in Neural Information Processing Sys- tems, 2021.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low- rank regularization. arXiv preprint arXiv:1511. 06067, 2015.
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing XU, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. In Advances in Neural Information Pro- cessing Systems, pp. 10936 10947, 2020. URL ht https: http://::///pppeeeeeeeeee www.innnipp... cc/
12

---

Under review as a conference paper at ICLR 2023
Ledyard R Tucker. Some mathematical notes mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
Roger Waleffe and Theodoros Rekatsinas, Principal component networks: Parameter reduction
early in training. arXiv preprint arXiv:2006.133: 13347, 2020.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: cetimmumication-ffcicee models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 11505-1 11515, 2020.
Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compres- sion: Tensor ring nets. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 9329-9338. 2018.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176- 6185, 2017.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trp: Trained rank pruning for efficient deep neural networks. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 977- 983, 2020.
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and sin- gular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition workshops, pp. 678-677 669,, 2020.
Yinchong Yang, Denis Krompass, and Volker Tresp. Tensor-train recurrent neural networks for video classification. In International Conference on Machine Learning, pp. 3891-3900, 2017.
Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decompositiin--bsse dnn model compression with optimization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10674-10683, 2021.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 7005. 10941, 2017.
Deli Yu, Peipei Yang, and Cheng-Lin Liu. Learning-based tensor decomposition with adaptive rank penalty for cnns compression. In 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 42-48. IEEE, 2021.
Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In Internatioonll Conference on Learning Representationss 2021. URL http:ss:/ppeeeeeieerttt
www.: www.rruummiiwwwwwwww..b.. wd-ww3300bbbbB
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural net- work training via forward and backward propagation sparsification. Advances in Neural Informa- tion Processing Systems 34, 2021.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awan channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 875-886, 2018
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
The entire Appendix consists of four sections.
Section A lists the additional experiments results with more model types and more com- parisons.
Section reports the ablation study for orthogonality-mposiing strategy.
Section presents the calculation schemes for two important performance metrics: "infer- ence FLOPs reduction"" and "training FLOPs reduction".
Section shows the layer-wise rank distribution in the experiments, thereby demonstrating the convenience of rank setting.
A ADDITIONAL EXPERIMENTS
A.1 RESNET-32 AND WRN-28-8 ON CIFAR-10
4 the evaluation results of low-rank training for ResNet-32 and WideResNett28-8 model on CIFAR-10 dataset. Ror ResNet-32 mode, it is seen that the proposed ELRT method can bring 0.23% accuracy increase tion. Compared with the existing methods, the proposed low-rank training from scratch approach shows better performance with respect to FLOPs reduction and accuracy.
the baseline model with 2.31 inference and training FLOPs reduc-
Table 4: Results for ResNet-32 and WRN-28-8 on CIFAR-10 dataset. "*" denotes compression ratio since the corresponding work does not report FLOPs reduction.
Always Train
Direct Train
Post-Train Model
Top-1
Inference Training
Method
Compact Model
(No Pre-train)
(%)
FLOPs J. FLOPS. +
ResNet-32
Original ResNett-3
Dense
92.49
1.00x
1x
FPGM al. (019
Sparse
91.93
2.12x
<1x
SCOP lang
Sparse
92.13
2.27x
<1x
SVDT Yang
Low-rank
90.55
3.93x
N/A
PSTRN-S 202Ta)
Low-rank
91.44
2.60x
<Ix
ELRT (Ours)
Low-rank
92.67
2.05x
2.05x
ELRT (Ours)
Low-rank
92.72
2.31x
2.31x
ELRT (Ours)
Low-rank
92.03
3.01
3.01x
ELRT (Ours)
Low-rank
91.21
x
3,34x
WRN-28-8
Original WRN 28-8
X
Dense
95.60
1.0x
1x
DST Liu et al.
2020
Sparse
94,80
10.0x
N/A
SFP He et al. (0188)
Sparse
94.22
5.00x
<1x
DPF
Lin et al.
2020c
Spurse
95.15
5.00x
N/A
ELRT
(Ours)
Low-rank 95.65
5.36x
3.14x
ELRT (Ours)
Low-ram
95.22
6.44x*
3.32x
ELRT (Ours)
Low-re oow-kank
94.87
11.8x
5.76x
A.2 RESNET-18 ON IMAGENET
Table shows the evaluation results of low-rank training for ResNet-18 18 ImageNet dataset. It secn that the proposed ELRT method can increase 0.01% accuracy over the baseline model with x.40x inference and training FLOPS reduction.
14

---

Under review as a conference paper at ICLR 2023
Table Results for ResNet-18 on ImageNet dataset.
Always Train
Direct Train
Post-Train Top-1
Top-5 Inference Training
Method
Compact Model
(No Pre-train)
Model
(%)
(%)
FLOPs & FLOPs
ResNet-18
Original ResNet-
18
x
Dense
69.79
89.08
1,00x
1x
FPGM He
2019
Sparse
68.34
88.53
1.72x
<1x
DSA Ning et al. 2020
Sparse
68.61
88.35
1.67x
<1x
SCOP Tang ef a. 2020
X
Sparse
68.62
88.45
188x
<1x
SVDT Yang ct al. 2020
X
Low-rank
N/A
87.26
2.03x
<1x
TRP Xu et al. 20)20)
x
X
Low-rank
65.46
86.48
1.81
<1x
ELRT
(Ours)
Low-rank
69.80 89.29
1.40x
1.40x
ELRT (Ours)
Low-rank
68.75
88.61
1.84x
1.84x
ELRT (Ours)
Low-rank
68.22 88.18
2.17x
2.17x
A.3 COMPARISON WITH EXISTING LOW-RANK TRAINING
We also compare the performance of ELRT with the existing low-rank training methods TTi et al. 2013); Joannou et al. )2015) Hawkins et al. (2022)). Table 0 shows the experimental results of training NIN model (Lin al. ) from scratch on CIFAR-10 dataset. seen that compared with Taii et al. (2015); Joannou al. (2015). ELRT can achieve higher FLOPs reduction with providing better accuracy performance.
addition, we also compare our proposed method with another Tucker-forma work (Hawkins al, (2022) ) for low-rank training from scratch. As shown in Table 7 ELRT can achieve higher accuracy and FLOPs reduction than (Hawkins et al. (2022) ) In addition, consider ELRT does not require iomputation-intensst Bayesian estimation that used in (Hawkins et al. 2022)) ELRT is more attractive for practical applications.
Table 6: Comparison with existing low-rank training works for NIN on CIFAR- 10 dataset.
Always Train
Direct Train Post-Train Top-1 Inference Training
Method
Compact Model Pre-train)
Model (%) FLOPs FLOPs
NIN
Original NIN
Dense 91.88
1.00x
1x
LRR
(Tai et al. 2015
Low-rank 93.02
1.50x
N/A
LRF
Toannou et at. 2015))
V
Low-rank 91.78
1,86x
N/A
ELRT
(Ours)
Low-rank 93.16
1.62x
1.62x
ELRT (Ours)
Low-rank 92.23
2.03x
2.03x
ELRT (Ours)
Low-rank 91.86
2.52x
2.52x
Table 7: Results for low-rank training works on MNIST dataset. The model used here can be referred (Hawkins et (2022))
Always Train Direct Train Post-Train Top-1
Inference Training
Method
Compact Model (No Pre-train)
Model (%)
FLOPs FLOPs
Customized Model from Hawkins
(022
Original Customized Model
X
Dense 98.09
1.00x
1x
ARD-LU (Hawkins et al.
2022
Low-rank 98.30
4.00x
N/A
ARD-HC [Hawknns et al.
2022
Low-rank 98.30
4.50x
N/A
ELRT (Ours)
Low-rank 98.41
4.74x
4.74x
A.4 COMPARISON WITH OTHER COMPACT MODELS TRAINED FROM SCRATCH
We also compare the performance of ELRT with directly training small dense models from scratch with the similar target model sizes. Table g shows the experimental results of training ResNet- 20 model from scratch CIFAR-10 dataset. It is that compared with models by uniformly removing 25% and 37.5% filters, ELRT can achieve better FLOPs reduction with providing higher accuracy.
15

---

Under review as a conference paper at ICLR 2023
Table 8: Comparison with directly training small dense models with the similar target model sizes on CIFAR-10 dataset.
Always Train
Direct Train Post-Train Top-1 Inference Training
Method
Compact Model (No Pre-train)
Model (%) FLOPs FLOPs
ResNet-20
Original ResNet-20
X
Dense 91.25
1.00x
1x
Uniform-0.75
Dense
91.08
1.70x
1.70x
ELRT (Ours)
V
V
Low-rank
91.73
1.98x
1.98x
Uniform-O.66 0.625
V
V
Dense
90.11
2.49x
2.49
ELRT (Ours)
V
Low-rank 91.26
2.51x
2.51x
A.5 COMPARISON WITH OTHER RELATED WORKS
We also compare the performance of ELRT with
(1) the works that employ tensor networks straight-
forwardly (Hayashi et al. (2019); Su et al.
2022 ); (2) the works that are trained densely, and com-
pressed with low-rank tensors (Lin al.
2020b
Yu et al. (202T): (3) other advanced low-rank
schemes (Koodak et al. (2020); Wang et al. (2021) Waleffe & Rekatsinas (2020).
(Hayashi al. 2019) explores train low-rank CNNs with new discovered decomposition for- mats via using architecture search. A key drawback is its very high training cost. As reported in CIFAR- 10 dataset, limiting the practicality of this approach, especially on large-scale dataset. Instead, ELRT provides an efficient low-rank training solution with much fewer costs and higher model performance. As shown in the
(Hayashi et al. a (2019). ) needs 829 GPU days to train ResNet-50
table below, ELRT shows higher 0.5% higher accuracy than (Hayashi et al. (2019)) with times fewer model parameters for training low-rank ResNet-50 on CIFAR-TO dataset.
Table 9; Comparison with Hayashi et al. 2019 on CIFAR-10 dataset.
Remaining Top-1 Parameters (%)
Method
ResNet-50
Hayashi et al. 2019))
22.2M
92.2
ELRT (Ours)
10.6M
92.73
Similar to (Hayashi et al. (2019)) SSu et al. (2022) also aims to train low-rank CNNs using new
decomposition formats. However, the model obtained via using the method proposed in [R2] has limited performance. As shown in the following Table, for training low-rank ResNet-32 on CIFAR- 10 dataset, ELRT shows much higher accuracy (at least 3%) than (Su et al. (202 compression ratio.
with even higher
Table 10: Comparison with Su al. (202)) on CIFAR-10 dataset.
Compression Top-1 Ratio (%)
Method
ResNet-32
mCP (Su et al. 2022
10.0x
82.93
mTK
Su et al. 2022
10.0x
65.75
mTT
Su et al. (2022)
10.0x
83.08
10.2x
87.89
ELRT (Ours)
12.7x
86.60
Lin et al. (020b)) is built on generalized higher-order Tucker articulated kernel scheme. A key difference between (Lin et al. (2020b)) and ELRT is that (Lin al. (2020b)) aims for low-rank com-
16

---

Under review as a conference paper at ICLR 2023
pression that requires the existence of pre- pre-trained model; while ELRT perform low-rank training from scratch without consuming any pre-traine training cost, significantly reducing training complexity. shown in the following table, ELRT shows better model performance (more than I % accuracy increase) than (Lin et al. (2220b)) for obtaining low-rank AlexNet on CIFAR-10 dataset. Here the ranks for Conv2, Conv3, Conv4 and Conv5 are [32, 64), [72, 108], [64, 32] and [40, 40], respectively.
More importantly,
Table 1I: Comparison with Lin et al. (2020b) CIFAR-I0 dataset.
Compression Top-1 Ratio
Method
(%)
AlexNet
Hotcake (Lin et al. (020bb)
9.4x
83.17
ELRT (Ours)
9.4x
84.45
Yu et al. (202T) is also a low-rank compression work that requiring pre-training phase. As shown in the following table, even without using any pre-trained high sighaccuracy model, ELRT still achieves higher accuracy than the pre-trained model-required YYu et al. (2021)) for obtaining low-
rank ResNet-56 on CIFAR-10 and ResNet-50 on ImageNet.
Table 12: Comparison with Yu et al.
2021
Top-1 Reduction | (%)
FLOPs
Method
Re
eu al.
2021
2.8x
93.52
ELRI (Our))
3.0x
93.67
ResNet-50
et (2021)
2.5x
75.58
ELRT (Ours)
2.5x
76.11
Khodak et al. (2020) applies spectral initialization (SI) and Frobenius decay (FD) for low-rank trainin. As shown in the following table, ELRT outperforms different variants of [R5] when training VGG on CIFAR- 10 dataset with higher compression ratio and at least 1.4% accuracy increase.
Table 13: Comparison with Khodak et al. 2020 on CIFAR-10 dataset.
Compression Top-1 Ratio
Method
(%)
VGGNet
Low-rank (Khodak et al. (2020)
10.0x
90.71
Fow-rank+Fl Khodak et af. (2020)
10.0x
90.99
Low-rank+Fl KKodak et al. (0200)
10.0x
91.57
Low-rank+SI+FID (Khodak al. (2020))
10.0x
91.58
ELRI
12.4x
92.99
Wang et al. 2222 needs warm-up phase train full-rank model in the first epochs, so it does not reduce the overall memory requirement, which measured by the peak memory usage. Instead, ELRT trains a low-rank model from scratch and always keeps the low-rank format during the training. As shown in the following table, ELRT brings higher model accuracy with fewer FLOPs than (Wang al. (202T) for training ResNet-50 ImageNet.
17

---

Under review as a conference paper at ICLR 2023
Table 14: Comparison with Wang et al. (2021) on ImageNet dataset.
FLOPs Reduction 4 (%)
Top-1
Method
ResNet-50
FP32
Wang et al.
2021
1.14x
76.43
AMP
Wang et al.
[2021
1.14x
76.35
ELRI (Ours)
2.2x
76.49
Similar to (Wang et al. (00TT), (Waleffe & Rekatsinas (2000) is initialized with wide and full- rank model and then decompose it low-rank format after few epochs. Instead, ELRT performs low-rank training from scratch and always keeps the model stay in the low-rank format during the entire training procedure. As shown in the following table, ELRT shows 0.51 % accuracy increase over (Waleffe Rekatsinas (2020)) with 2 times model size reduction for training ResNet-50 on ImageNet dataset.
Table 15: Comparison with Waleffe & Rekatsinas (2020) on ImageNet dataset.
Remaining Top-1 Parameters (%)
Method
ResNet-50
Waleffe & Rekatsinas
2020
25.5M
75.6
(ORT()
11.4M
76.11
B ABLATION STUDIES
Effect of Imposing Orthogonality. We conduct experiments to study the effect of imposing the orthogonality on the factor matrices via using DSO regularization. As shown in Fig. 6 our pro- posed orthogonality the standard low-rank training with the same FLOPs reduction and same low-rank format, thereby demonstrating the importance of enforcing orthogonality on the components of the low-rank model.
low-rank training shows very significant performance improvement than
ResNet-20 w/ DSO
ResNet-20 woo DSO
93
ResNet-32 w/ DSO
Accuracy (%)
ResNet-32 w/o DSO
92
ResNet-56 w/ DSO
ResNet-56 w/o DSO
91
90
2.00
2.25
2.50
2.75
3.00
3.25
3.50
3.75
4.00
FLOPs Reduction
Figure 6: Performance low-tensor-rank training with and without using DSO for 26//25-20/32256 on CIFAR-10 dataset.
Different Orthogonal Regularization Schemes. We also conduct the ablation study to explore the best-suited scheme to impose the orthogonality. As shown in Table 16 DSO regularization demon-
18

---

Under review as a conference paper at ICLR 2023
strates consistently better performance than the other schemes. Such empirical results also coincide with our analysis Question 3. Therefore we adopt the DSO scheme for all our experiments.
Table 16: Performance of using different orthogonal regularization schemes for training low-rank ResNet-20, ResNet-32 and ResNet-56 on CIFAR-10 10 dataset.
FLOPs
SRIP
DSO
ResNet-20
2x
91.43
91.32
91.36
91.73
2.5x
91.05
90.92
90.91
91.26
3x
90.72
90,59
90,43
90.95
ResNet-32
92.75
92.38
92.67
91.79
91.72
91.76
92.03
ResNet-56
2x
93.52
93.44
93.45
93.96
3x
93.35
93.26
93.28
93.67
Effect of Different Orthogonality Penalty Parameters d2e As shown in Table 17
18 and Figure
we evaluate effect of different orthogonality penalty parameters Ad.
Table 17: ResNet-20 with different orthogonality penalty parameters (Aạ).
Aa
FLOPs | Accuracy (%)
FLOPs
Accuracy (%)
FLOPs 1
Accuracy (%)
le-6
91.48
91.11
90.68
5c-4
91.64
91.16
90.80
le-3
91.73
91.26
90.95
5e-3
1.98x
91.61
91.29
90.96
2.51x
3.02x
le-2
91.66
91.22
90.88
5e-2
91.55
91.37
91.28
91.78
91.12
91.09
Table 18: ResNet-56 with different orthogonality penalty parameters (Ag).
FLOPs 1 Accuracy (%)
FLOPSs
Accuracy (%)
FLOPs 1
Accuracy (%)
Te-6
92.82
93.24
93.28
5c-4
93.62
93.86
93.51
le-3
93.96
94.01
93.67
5c-3
2.05x
93.95
93.87
93.64
2.52x
3.01
le-2
93.87
93.86
93.59
5e-2
94.03
93.71
93.55
93.87
93.75
93.64
19

---

Under review as a conference paper at ICLR 2023
94
93
Accuracy (%)
ResNet-20 2x
ResNet-20 2.5x
ResNet-20 3x
ResNet--6 2x
89
ResNet-56 2.5x
ResNet-56 3x
88
le-6
5e-4
le-3
5e-3
le-2
5e-2
Orthogonal Penalty
Figure 7: Performance of ELRT for ResNet-20/56
CIFAR-10 dataset with different orthogonal
penalty parameters Ad-
C CALCULATION SCHEMES FOR INFERENCE AND TRAINING FLOPS REDUCTION
C.I CALCULATION INFERENCE FLOPs REDUCTION
To calculate the inference FL.OPs reduction of one convolution layer after using Tucker-2 decompo- sition, we adopt the scheme used in (Kim et al. (2015)):
H'W SR.''W' + D R3 H'W +R,,'''
where D is the kernel height and width, S is the number of input channels, T is the number of output channels, H', W' are output height and width, and R1, Rg are the ranks the Tucker decomposi- tion.
En kam With the above notation, the inference FLOPs reduction can be calculated as Ealtiges B
where A
W' represents the number of mulippiictiion-additoo operations of layer in the oiginal model, and B = SR, H'W' +D2 R2 H''' "''''+'''
+T +TR, H'W' represents the number of multiplication- addition operations of layer in the Tucker decomposed low-rank model.
C.2 CALCULATION OF TRAINING FLOPs REDUCTION
As defined in (Zhou al. (202 Evei al. (020 "training FLOPs reduction", also noted as "training-cost saving", the ratio of the average FLOPs of the dense network over that of the compact network. Here the total FLOPs training a network consists of the part forward pass and backward pass. As indicated in (Evci al. (202); Zhou al. (221); Baydin al. (2018), the FLOPs of backward propagation can be roughly counted as about times that is consumed in the forward propagation. Next denote the FLOPs of the dense network, the sparse network, the low-rank network during forward propagation as fD, Iss and ft., respectively.
C.2.1 DENSE NETWORK.
The FLOPs of the forward propagation is fp. The FLOPs of the backward propagation is 2fp. The training FLOPPs reduction is /D+2/p 1, which means there is training FL.OPs reduction.
20

---

Under review as a conference paper at ICLR 2023
C.2.2 PRUNING.
Assume T epochs are needed to obtain a pre-trained model. After pruning, another K epochs are needed to re-train the model for fine-tuning. In the pre-training phase, the total computation cost of the forward propagation and backward propagation are fo * T and 2fD * ', respectively. In the re-training phase, the total computation cost of the forward propagation and backward propagation fs K and 2fs K, respectively. So overall the training FLOPs reduction is 3/D+T which means there no training FLOPs reduction.
3f < 1,
C.2.3 LOW-RANK COMPRESSION.
Assume T epochs needed to obtain pre-trained model. After low-rank decomposition, other K epochs are needed to re-train the model for fine-tuning. In the pre- pre-training phase, the total computation cost of the forward propagation and backward propagation are fn T and 2/n T, respectively. In the re-training phase, the total computation cost of the forward propagation and backward propagation are fL K and 2 K, respectively. So overall the training FLOPs reduc- 3fo=T
tion is 33fok 1, which means there is training FLOPs reduction.
C.2.4 STRUCTURED SPARSE TRAINING.
GrowEfficient (Yuan ef al. (202I)). For training a sparse model via GrowEfficient, the FLOPs of the forward propagation is fs. Since the channel masks and scores, which de- termine to keep or remove the corresponding channels/filters, are updated by the Straight- Through gradient Estimation (STE), the backward propagation has go through all the channels/filters, leading to dense computation. Therefore, the FLOPs of the backward propagation is 2f.. For training dense model, the computation cost of the forward prop- agation and backward propagation are fo and 2 respectively. Therefore, the training 2fD FLOPs reduction is fs+2fo Js+2ID
3fp
BacrseBackward (Zhou et al. (2021). For training sparse model via SparseBackward, the FL.OPs of the forward propagation is ʃs- The backward propagation keeps sparse with 2ʃs computation cost. Unlike updating masks/scores via dense backward propagation in GrowEfficient, SparseBackward updates them via Variance Reduced Policy Gradient Esti- mator (VR-PGES which only requires extra one-time forward propagation with training cost fs. For training a dense model, the computation cost the forward propagation and backward propagation fo and 2fD, respectively. Therefore, the training FLOPs reduc- tion is fD+2fp 3fp
2fs+2fs 4Js
C.2.5 ELRT (OURS).
Recall that ELRT is built on Tucker-2 dccomposition that converts one convolutional layer into two factor matrices, which can be viewed as 1 x convolutional layers, and one core tensor that is viewed as 3 x 3 convolutional layer. Therefore, our low-rank model can be viewed as a compact dense network with more convolutional layers but fewer FLOPs and parameters. Therefore, assume the FLOPs of the forward propagation is ft.- The FLOPs of the backward propagation is 2/r.- The training FLOPs reduction is calculated as fD+2fp fo which is identical to the inference FLOPPs fr.+2fr. reduction as calculated in Eq. 7 In other words, the inference and training FLOPs reductions brought by ELRT are the same. Notice that the FLOPSs of calculating the orthogonality loss term is ignored here since it only occurs per batch, while fl and fo are consumed per data. Considering the batch size is typically large (e.g., 128 in our experiment), the FLOPS contribution of calculating the orthogonality loss term is negligible.
Fr.
D RANK SETTINGS
Table 19 20 and 21 list the layer-wise rank settings of ResNet-20 model on CIFAR- 10 dataset under
1.98 x, 3.02x x FLOPs and 6.01 x parameters reduction, respectively. Table 23 list the layer-wiss wise
rank settings of ResNet-56 model CIFAR-10 dataset under x.05x and 2.52x FLOPs reduction.
21

---

Under review as a conference paper at ICLR 2023
Table 24 lists the layer-wise rank settings of ResNet-50 model ImageNet dataset under 2.49 FLOPs reduction.
From these tables, it is seen that many adjacent layers share the same rank value, and the entire model only needs a few rank values to be assigned. For instance, for ResNet-56 model under 2.05x x FLOPs reduction on CIFAR-I0 dataset, only three numbers: 12, 18, 26, are selected as the ranks to be assigned to all layers, where the first/sccond/last eighteen layers share the rank value 12/18/26, respectively. Such rank-sharing phenomenon significantly simplifies the rank selection process.
Table 19: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 1.98x FLOPs reduction.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FLOPs Reduction = 1.98
goyerl.0.covv
(14, 14)
layer2.0.conv.
(14, 14)
layer3.0.comv
(28, 28)
Jayer1.0.comg
(12,
layer2.0.conv2
(14, 14)
layer3.0.com
(28, 28)
L.yerl.L.onvl
(12, 12)
layer2.1.com
(14, 14)
layer3.1
(28, 28)
layerI. 1conv2
(12, 12)
layer2.1.com's
(14, 14)
1ayer3.1.comv
(28, 28)
layer1. 12ccovl
(12, 12)
layer2.2.com's
(16, 16)
1ayer3.2.conv
(28, 28)
layer1.2.conv2
(12, 12)
layer2.2.conv2
(16, 16)
layer3.2.conv
(28, 28)
Table 20: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 3.02x FLOPs reduction.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FLOPPs Reduction
3.02
layerI.0.com 1.covv
(10, 10)
layer2.0.conv
(12, 12)
layer3.0.com
(20, 20)
2.0.ov2
(10, 10)
laycr2.0.conv2
(12, 12)
layer3.0.comv
(20, 20)
layer1.1.comvi 1
(10, 10)
layer2.1.conv/
(12, 12)
1ayer3.1.comv
(20, 20)
layer1.1.comv/
(8,
layer2.1.conv2
(14, 14)
layer3.1.com
(22, 22)
layer1.2.comv 1
(8, 8)
layer2.2.convl
(14, 14)
layer3.2.com convi
(22, 22)
1.yor1.2.conv2
(8, 8)
layer2.2.conv2
(14, 14)
layer3.2.conv
(22, 22)
Table 21: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 6.01 parameters reduction.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FLOPs Reduction = 6.01 X
layer I.ooov
(9, 9)
layer2.0.conv
(12, 12)
layer3.0.com
(16, 16)
layerl.0.conv2
(9, layer2.0.conv2
(12, 12)
layer3.0.convl
(16, 16)
(9, 9)
layer2.
(12, 12)
1ayer3.1.covv
(16, 16)
(9, 9)
Jayer2.1.com
(12, 12)
layer3. I.oovv...convl
(16, 16)
1ayer1.2.conv
(9, 99
1ayer2.2.conv
(12, 12)
layer3. l
(16, 16)
(9, 9)
layer2.2.conv2
(12, 12)
layer3.2.conv
(16, 16)
22

---

Under review as a conference paper at ICLR 2023
Table 22: Layer-wise rank settings of the compressed ResNet-56 model on CIFAR-10 dataset with 2.05 FLOPs reduction.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FLOPs Reduction
2.05 x
layerI.0.com Jyyerl..convl
(12, 12)
layer2.0.conv
(18, 18)
layer
(26, 26)
layer1.0.com/ c0.conn2
(12, 12)
layor2.0.conv2
(18, 18)
layer3.0.com'
(26, 26)
1ccovl...oovv1
(12, 12)
layer2.1.com
(18, 18)
layer3.1.com
(26, 26)
1.co22.1.oom2
(12, 12)
layer2.1.coms
(18, 18)
layer3.1.com
(26, 26)
layer1.2.conv1
(12, 12)
layer2.2.conv1
(18, 18)
layer3.2.comvl
(26, 26)
layer1.2.comv2
(12, 12)
layer2.2.conv
(18, 18)
layer3.2.comv
(26, 26)
layer1_3.conv
(12, 12)
layer2.3.conv
(18, 18)
1ayer3.3.conv
(26, 26)
layer1.3.conv2
(12, 12)
layer2.3.conv2
(18, 18)
Iayer3,3.conv
(26, 26)
layerl.4.conv1
(12, 12)
layer2.4.conv1
(18, 18)
layer3.4.convl
(26. 26)
layer1.4.conv2
(12, 12)
layer2.4.conv2
(18, 18)
1ayor3.4.convl
(26. 26)
layor1.5.conv1
(12, 12)
layer2.5.conv1
(18, 18)
layer3.5.coms
(26, 26)
laycr1,5.conv2
(12, 12)
layor2.5.conv2
(18, 18)
laycr3.5.conv
(26, 26)
layer1.6.conv
(12, 12)
layer2.6.convl
(18, 18)
laycr3.6.comv
(26, 26)
layer1.6.conv2
(12, 12)
layer2.6.conv2
(18, 18)
layor3.6.comv
(26, 26)
layer1.7.conv1
(12, 12)
layer2.7.conv1
(18, 18)
layer3.7.com
(26, 26)
layer1.7.conv2
(12, 12)
layer2.7.conv2
(18, 18)
layer3.7.com/
(26, 26)
Mnyer1...cmm
(12, 12)
layer2.8.conv
(18, 18)
0yyer3..connv I
(26, 26)
layer1.8.com connv2
(12, 12)
layer2.8.conv2
(18, 18)
layer3.8.com
(26, 26)
Table 23: Layer-wise rank settings of the compressed ResNet-56 model on CIFAR-10 dataset with
2.52x FLOPs reduction.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FI.OPs Reduction
2.52
layerI.0.com 1.coovv
(10, 10)
layer2.0.convl
(15, 15)
layer3.0.com
(28, 28)
layer1.0.om 1.0.co
(10, 10)
layer2.0.conv2
(15, 15)
layer3.0.com
(28, 28)
layer1.1.com
(10, 10)
1ayor2.1.convl
(15, 15)
layer3.1.com
(28, 28)
layer1. 2ccov2
(10, 10)
layer2.1.comv/
(15, 15)
layer3.1.comv
(28, 28)
layer1.2.convl
(10. 10)
layer2.2.conv1
(15, 15)
layer3.2.comvl
(28, 28)
layer1.2.con22
(10, 10)
layer2.2.conv2
(15, 15)
layer3.2.conv
(28, 28)
layerI.3.conv1
(10, 10)
layer2.3.conv
(15, 15)
layer3.3.conv
(28, 28)
layer1.3.cov2
(10. 10)
layer2.3.conv2
(15, 15)
layer3.3.conv
(28, 28)
layer1.4.comv
(10, 10)
layer2.4.convl
(15, 15)
layor3.4.convl
(28, 28)
1ayor1.4.conv2
(10, 10)
layor2.4.conv2
(15, 15)
1ayor3.4.conv
(28, 28)
laycr1.5.conv
(10, 10)
layer2.5.convl
(15, 15)
layer3.5.con
(28, 28)
layer1.5.comv/
(10, 10)
layer2.5.conv2
(15, 15)
layer3.5.conv
(28, 28)
Anyor1.6.convl
(10, 10)
layer2.6.convl
(15, 15)
layer3.6.comvl
(28, 28)
2..oo1.6.com22
(10, 10)
layer2.6.comv2
(15, 15)
layer3.6.conv
(28, 28)
1ayorl.7.convl
(10, 10)
layer2.7.convl
(15, 15)
layer3.7.comv
(28, 28)
1ayor1.7.conv2
(10, 10)
layer2.7.conv2
(15, 15)
layer3.7.conv
(28, 28)
mnyer1.8.com
(10, 10)
layer2.8.conv1
(15, 15)
layor3.8.conv
(28, 28)
layer1.8.co .8.on22
(10, 10)
layer2.8.comv2
(15, 15)
layer3.8.com
(28, 28)
23

---

Under review as a conference paper at ICLR 2023
Table 24: Layer-wise rank settings of the compressed ResNet-50 model on ImageNet dataset with x.49 FLOPs reduction. "N/A" denotes we do not compress that layer.
Layer Name
Rank
Layer Name
Rank
Layer Name
Rank
FLOPs Reduction
2.49 x
layer1.0.com/
layer2.3.com
(48)
layer3.5.com
(64)
Jayer1.0.comy
(32, 32)
layor2.3.conv2
(48, 48)
layer3.5.com
(64, 64)
layerl.0.oe 1..covv3
(32)
layer2.3.conv3
(48)
layer3.5 3onvv
(72)
layerl.1.covv
N/A
layer3.0.convl
(64)
layer4.0.convl
(96)
layer1.1.conv2
(32, 32)
layer3.0.conv2
(64, 64)
layer4.0.conv2
(96, 96)
1.yor1.1.comv.
(32)
layer3.0.comv3
(72)
layer4.0.comv3
(96)
laycr1.2.convl
N/A
1ayer3.1.convl
(64)
1.yer4.1.comv
(96)
layer1.2.com/
(32, 32)
layer3.1.comv2
(64, 64)
layer4.1.comv2
(96. 96)
layer1.2.conv3
(32)
l.yor3.1.conv3
(72)
l.ymr3
(96)
layer2.0.conv
(48)
1ayor3.2.conv
(64)
layer4.2.conv
(96)
layer2.0.conv2
(48, 48)
layer3.2.conv2
(64, 64)
layer4.2.conv2
(96, 96)
laycr2.0.conv3
(48)
layor3.2.conv3
(72)
layor4.2.conv3
(96)
1ayor2.1.conv1
(48)
layer3.3.convl
(64)
1.yor2.1.conv2
(48, 48)
layer3.3.conv2
(64, 64)
1.com2.
(48)
layer3.3.comv3
(72)
layer2.2.conv1
(48)
layer3.4.convl
(64)
1ayer2.2.comv2
(48, 48)
layer3,4.conv2
(64, 64)
layer2.2.convv
(48)
layer3.4.conv3
(72)
24


Reviews:
[{"review_id": "kS01v2iBtm5", "reviewer": "Reviewer_yVpF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to train low-rank CNNs from scratch. To achieve this, the authors propose Tucker-2 decomposition with low-rankness, and improve the training with orthogonal weight regularizations.", "summary_of_the_review": "The authors provide sufficient analysis for their design choices and conduct intensive experiments with hardware measurements.\nHowever, as discussed above, my main concern is the core novelty of the method.", "strength_and_weaknesses": "Strength:\n1) The paper is well-written and easy to follow.\n2) Questions in section 4 is well-motivated.\n3) It is good to have intensive experiments and latency measured on embedded GPUs.\n\n\nWeakness:\n1. The analysis for the question1 is not convincing. 1) There could be other formulations. For example, what if one perform SVD on each filter, instead of on the reshaped 2D weight matrix? 2) It would be better to show the averaged approximation error across all layers, instead of picking some specific layers.\n2. The analysis for question 3 is not novel. The original DSO method is not developed in this work. The authors just reiterate orthogonal regularizations from Bansal et al. (2018).\nThe two bullet points make Algo. 1 is not novel.\n3. No ablation studies on the regularization scaling factor $\\lambda_d$.\n4. Paper writing: 1) I think the authors used too many underlines, making it hard to find out which part did the authors really want to emphasize. 2) Minor issues: on the 4th row on page five, it should be “lose”, not “loss”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kLQ7-VlRTCH", "reviewer": "Reviewer_rfLo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to achieve low-rank training for CNNs. Concretely, the authors consider the form of tucker-2 decomposition to build the convolutional kernel, while during training the orthogonality regularization is imposed on the non-core matrices. The authors experimentally demonstrate the effectiveness of the proposed method using several well-known CNN structures for image classifcation tasks.", "summary_of_the_review": "This is a solid paper, it could be impactful in both the application of tensor decomposition and network compression and acceleration communities. I vote to accept this paper if the authors could address the aforementioned weaknesses.", "strength_and_weaknesses": "Strength\n\n1. This paper provides a very neat and practical solution for low-ranking training with the tucker decomposition. Although the topic seems to overlap with former literature, to the best of my knowledge, the method still holds its novelty.\n\n2. The experiments show promising results, and the actual running times in edge devices convince the effectiveness of the proposed method.\n\nWeaknesses\n\n1. The authors should compare the proposed method with two more families of works if possible: i) the works that employ tensor networks straightforwardly, such as [1, 2]; ii) the works that are trained densely, and compressed with low-rank tensors, such as [3, 4].\n\n[1] Hayashi, K., Yamaguchi, T., Sugawara, Y., & Maeda, S. I. (2019). Exploring unexplored tensor network decompositions for convolutional neural networks. Advances in Neural Information Processing Systems, 32.\n\n[2] Su, Jiahao, et al. \"Compact Neural Architecture Designs by Tensor Representations.\" Frontiers in artificial intelligence 5 (2022).\n\n[3] Lin, Rui, et al. \"Hotcake: Higher order tucker articulated kernels for deeper CNN compression.\" 2020 IEEE 15th International Conference on Solid-State & Integrated Circuit Technology (ICSICT). IEEE, 2020.\n\n[4] Yu, Deli, Peipei Yang, and Cheng-Lin Liu. \"Learning-based Tensor Decomposition with Adaptive Rank Penalty for CNNs Compression.\" 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2021.\n\n2. The structures of CNNs are slightly out-of-date, the authors could try their method on search-based optimal network structures (eg. CoAtNet) or networks with unusual designs (eg. RepLKNet with 31x31 kernels) to enhance the effectiveness of the proposed method.\n\n3. The proposed method could result in matrix multiplications with non-optimal dimensions in practice. In Table 3, the theoretical reduction of FLOPs is somewhat larger than the reduction of running time. Hence the design of the network structure could be difficult and time-consuming.\n\n4. I am still not sure why orthogonality regularization can boost the performance by a large margin since it does not ensure the orthogonality on $\\mathbf{U}^{1}, \\mathbf{U}^{2}$. If this conclusion still holds under larger and wider CNNs?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gVwFb5kQ73", "reviewer": "Reviewer_DfQU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ELRT, a low-rank training method that enables training low-rank factorized neural networks from scratch to attain faster training speed and smaller models. Different from the previously proposed low-rank factorization schemes designed specifically for CNNs,  ELRT leverages tensor decomposition. ELRT also utilizes Soft Orthogonal Regularization to further improve the accuracy of the low-rank training. Thorough experimental results on vision tasks are provided to demonstrate the effectiveness of ELRT.", "summary_of_the_review": "The proposed ELRT method is novel in the sense that it uses tensor decomposition with specialized regularization for training. The method might lack of importance given its performance is very similar to the previously proposed low-rank training schemes (please refer to \"Strength And Weaknesses\" for more details). I thus urge the authors to include comparisons between ELRT and the aforementioned low-rank training schemes.", "strength_and_weaknesses": "Strength:\n- The paper is well-written. The research direction of speeding up large model training is important.\n- The experimental results are thorough and solid.\n\nWeakness:\nOverall, my major concern is that the statement \"the existing low-rank training solutions are still very limited and do not demonstrate their effectiveness for training modern low-rank CNN models in the large-scale dataset from scratch.\" is too strong. The papers of [1-3] (which are missing in the reference) have already demonstrated similar promising results. In such a sense, the authors are expected to demonstrate the advantages of ELRT against the low-rank schemes proposed in [1-3]. More weakness is summarized below:\n\n- Using tensor decomposition essentially factories each convolution layer into three \"thinner\" layers. However, it is well-known that deeper models are harder to train. Thus, does ELRT require any special (if not heavy) hyper-parameter tuning for training such deep low-rank neural networks?\n- Given that the transformer network is dominating many application domains currently. I wonder if ELRT also performs well for transformers. Though I understand that the factorization scheme can not be used directly for transformer networks.\n-  It was observed by [2] that factorizing the beginning layers can cause accuracy drops. Does ELRT encounter that?\n- It seems selecting the \"appropriate rank\" for different layers is of importance, otherwise, accuracy drops can be observed, e.g., in Table 2. Is there an approach to determine the ranks to select for the network layers for ELRT?\n\n\n[1] https://arxiv.org/abs/2105.01029\n[2] https://proceedings.mlsys.org/paper/2021/hash/84d9ee44e457ddef7f2c4f25dc8fa865-Abstract.html\n[3] https://arxiv.org/abs/2006.13347", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "WW9QZWyf_8b", "reviewer": "Reviewer_Afr4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While there have been many studies using Low-rank approximation (compression) method for pre-trained models, this paper proposes a new training method from scratch using tucker decomposed models w/o full pre-training steps. Low-rank training from scratch has been conventionally viewed as an insufficient method, which can lead to lower accuracy, but this paper shows comparable accuracy with 2x-3x reduced FLOPs (maybe by adopting SO regularizer). ", "summary_of_the_review": "In my opinion, this paper should strengthen the experimental results and answer some questions on tucker decomposition and regularizer. I understand this method can be applicable to limited networks and dataset, but to prove higher novelty, it should be extended.", "strength_and_weaknesses": "Strength\n- The proposed method, ELRT, can reduce training cost because the original networks are already decomposed.\n- Unlike conventional thinking, they show that low-rank training from scratch can lead to comparable accuracy for image classification models (ResNet/MobileNet).\n- They also show accelerated results on various H/W (V100 GPU, FPGA, ASIC, Jetson).\n\nWeakness\n- Most experiments are performed on the CIFAR-10 dataset (even including the VGG-16 model). It can show a method is working well, but it doesn’t mean that the method is novel or works well for many NNs for practical applications. Because the key to improved accuracy seems to be adopting a regularizer to the current training system, training redundant models may cause much controversy. To prove the effectiveness of a method, the CIFAR-10 results and ResNet-50(w/ ImageNet) results are not adequate because this model may be not fully regularized. \n- There is only one ImageNet result w/ ResNet-50. How about ResNet-18 or other compact networks on ImageNet? \n- I think acceleration results are a good way to prove our compression methods are effective and practical. But, how about smaller models including ResNet-18, MobileNet, EfficientNet and so on? Decomposed layers w/ small weights can be not accelerated compared to dense models in highly-parallelized computing systems. \n- About Question 1, I have a concern on Figure 3 with MSEs. According to the analysis on MSE, tucker decomposition is selected as a main approach of this paper. However, how can we judge which training system from scratch is better by measuring the MSE of compression? This method is performed w/ decomposed layers and there is no compression step.\n- I think there is a lack of explanations and reasons that the orthogonality of U^1 and U^2 can lead to better NNs. But, I’d like to follow other reviewer’s opinions in this aspect.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject