Decoupling Weight Regularization from Batch Size for Model Compression
Authors: Dongsoo Lee
Se Jung Kwon
Byeongwook Kim
Yongkweon Jeon
Baeseong Park
Jeongin Yun
Gu-Yeon Wei
Venue:  (2020)
Keywords: Model compression
Weight Regularization
Batch Size
Gradient Descent

Abstract: Conventionally, compression-aware training performs weight compression for every mini-batch to compute the impact of compression on the loss function. In this paper, in order to study when would be the right time to compress weights during optimization steps, we propose a new hyper-parameter called  Non-Regularization period or NR period during which weights are not updated for regularization. We first investigate the influence of NR period on regularization using weight decay and weight random noise insertion. Throughout various experiments, we show that stronger weight regularization demands longer NR period (regardless of batch size) to best utilize regularization effects. From our empirical evidence, we argue that weight regularization for every mini-batch allows small weight updates only and limited regularization effects such that there is a need to search for right NR period and weight regularization strength to enhance model accuracy. Consequently, NR period becomes especially crucial for model compression where large weight updates are necessary to increase compression ratio. Using various models, we show that simple weight updates to comply with compression formats along with long NR period is enough to achieve high compression ratio and model accuracy.

Under review as a conference paper at ICLR 2020
DECOUPLING WEIGHT REGULARIZATION FROM BATCH SIZE FOR MODEL COMPRESSION
Anonymous authors Paper under double-bling review
ABSTRACT
Batch size selection affects the generalization ability of gradient descent, and small batch size is usually preferred. Conventionally, weight regularization is performed for every mini-batch without considering when would be the right time to regularize weights during optimization steps. For model compression, which regularizes weights to follow compressed forms, compression- performs weight compression for every mini-batch to compute the impact of com- pression on the loss function. In this paper, we propose new hapeeeparameter called "Non-Red Regularization period" or NR period during which weights are not regularized (or compressed). We first investigate the influence of NR period on regularization using weight decay and weight random noise insertion. Throughout various experiments, we show that stronger weight regularization demands longer NR period (regardless of batch size) to best utilize regularization effects. From our empirical evidence, we argue that weight regularization for every mini-batch allows small weight decay/noise only and limited regularization effects such that there is need to search for right NR period and weight regularization strength to enhance model accuracy. Consequently, NR period becomes especially crucial for model compression where strong weight regularization is necessary to increase compression ratio. Using various models, we show that simple weight regulariza- tion to comply with compression formats along with long NR period is enough achieve high compression ratio and model accuracy.
training also
INTRODUCTION
For Deep Neural Networks (DNNs), a common training method updates weights by gradient descent and explicit weight manipulation through regularization. To overcome some practical issues of gra- dient descent on non-convex optimization problems, there have been several enhancements such as learning rate scheduling and adaptive update schemes using momentum and update history (Ruder) 2016). Optimizing batch size another way to yield efficient gradient descent. Note that large batch size has the advantage of enhancing parallelism of the training system in order to speed up training, critical for DNN research (Dean et al. 2012). Despite such advantages, small batch size is preferred because it improves generalization associated with flat minima search (Keskar et al. 2016) and other hyper-parameter explorations are more convenient (Masters & Luschi (2018). Small batch size also affects weight regularization if weight updates for gradient descent and weight regularization are supposed to happen for every mini-batch. For example, for weight decay conducted for every mini- batch, if batch size modified, then the weight decay factor should also be adjusted accordingly
Loshchilov & Hutter 2017).
Weight regularization a process that adds information the model as a way to avoid overfitting [Godffello et al.) 2016 van Laarhoven 2017). In this paper, we explore weight compression as a form of weight regularization as severely restricts the search space of weights (i.e., regularized by compression forms). Moreover, model compression shrinks the effective model size, which is an important regularization principle (Goodfellow et al. (2016) (note that improved model accuracy by model compression is reported (Frankle et al. 2019)). Weights are regularized in numerous ways by model compression. For example, each weight can be pruned (e.g., Han et al. (2015)) or quantized (e.g., [uoo et al. (2017)) to yield sparse model representation to reduce the number of bits to represent each weight. While weight regularization for model compression can simply be performed once after training, compression-wware training improve model accuracy by reflecting

---

Under review as a conference paper at ICLR 2020
300
Q
1-bit
100-
Q=2-bit
(ount(
Count(
Q-3-bit
R=1024
4=4bit
50-
100
01
-3
3
-3
Error (ate (W
Error rate(W"/WW
Figure I: Distribution weight noise e after low-rank approximation via SVD (Left) or quantization
(Right) when R is the rank and is the number of quantization bits.
the impact of model compression the loss function for every mini-batch update (Courbariaux ctt al. 2013 Zhu Gupta 2017 Zhu et al. 2017 Guo et al. 2017]. Now, our question in this paper the following: Should weight regularization follow batch size that is considered for gradient descent? If the answer is "No," " then we would need to consider a new hyper-parameter search space with the right weight regularization timing for existing weight regularization methods and model compression schemes.
In this paper, report the following observations: 1) Model compression tends to induce weight decay and random weight noise; 2) For weight decay and weight noise insertion, less frequent weight regularization allows stronger regularization best utilize regularization effects regardless of batch size; and 3) Similarly, if weight regularization for compression is performed less frequently, training for model compression permits stronger weight regularization with improved model accuracy, and thus, achieves higher compression ratios.
We confirm our above observations through various experiments and propose an occasional reg- ularization method to decouple weight regularization from batch size selection that is optimized for gradient descent. We verify that our simple model compression techniques (without modifying the underlying training procedures) based occasional weight regularization (with longer non- regularization period) can achieve higher compression ratio and higher model accuracy compared to previous techniques that demand substantial modifications to the training process.
NOISE MODEL ON WEIGHT COMPRESSION
We first study the relationship between model compression ratio and weight regularization strength using quantization and singular-value decomposition (SVD) model compression techniques. We assume a popular quantization method based on binary codes for which a weight vector w is ap- proximated to be Σi= o,,b for q-bit quantization, where a is scaling factor and b(= +13") a binary vector, and is the vector size. The quantization error / E, a,,; (12 is minimized by a method proposed by Xu et al. (2018) to compute a and For SVD, a weight matrix W E mxx approximated be W |R by minimizing W -W'I subject (an(WW") S R, where R is the target rank.
For our experiments, domly generated from the Gaussian distribution N(u- = 0, o² =1). Then, we are interested in the amount of change of each weight after quantization and SVD. Assuming that weight noise through compression is expressed as in the form of w' = w(1 + <). Figure shows the distribution of with various quantization bits or target ranks. From distributions skewed to be negative, it is clear that weights tend to decay more with higher compression ratio, along with wider range of random noise. Reasonable explanations of Figure [ would include: 1) weights generated from the Gaussian distribution are uncorrelated such that an approximation step (by compression) using mul- tiple weights would result in noise for each weight, 2) in the case of SVD, elements associated with small eigenvalues are eliminated, 3) averaging effects in quantization reduce the magnitude of large weights. For weight pruning, e becomes -1 or 0 (i.e., weight decay for selected weights). Cor- respondingly, we study weight decay and weight noise insertion in the next two sections as effort to gain a part of basic knowledge on improved training for model compression, even though actual model compression would demand much more complicated weight noise models.
synthetie (2048 2048) weight matrix where each element is ran-
2

---

Under review as a conference paper at ICLR 2020
Loss
Ron--Regulaiiztiionn Period=pNR
K......******
Step
(N+1) Batch
(N-pNR) Batch
Step 3
Step
Step
Step AL
Step
Step
AW
Step 5
Weight Regularization Step
Weight Regularization
Parameter
Step
Weight Updates wo Regularization Weight Regularization
Figure 2: Gradient descent and weight regularization when NR period is given as a multiple of batches. Depending the loss surface and/or strength of regularization, regularization would lead to step (escaping from a local minimum) or step 5 (returning to a local minimum).
3 NON-REGULARIZATION PERIOD
Since weight regularization cannot precede updates for gradient descent, in order to control the fre- quency of weight regularization, available option is skip few batches without regularization. In this paper, we propose a new to enable occasional regularization and to define the interval of two consecutive regularization events as shown in Figure 2
called -Non-Regularization period" or NR period,
NR period is expressed as a multiple of batches.
Strong weight regularization facilitates the chance of escaping a local minimum (depicted step 2 in Figure 2 or require longer NR period to return to local minimum (described as step 5 in Figure 2p. Let us estimate NR period (pNNR) for returning case. Given a parameter set w (that is assumed to be close enough to local minimum) and learning rate the loss function of a model [(wr) can be approximated as
((w) L(wo) (w -wo) (H (w)) /2) w))
using a local quadratic approximation where H the Hessian of L and wn is set of parameters at a local minimum. After regularization is performed at step I, w can be updated by gradient descent as follows:
Wi+1 wi Y dw w= wt ---- -
HH (wo)(w; Wo).
Thus, after pNR. we obtain Wi+pNN Wy + (I - (H (wo))).RR (w; wo). where I is an identity matrix. Suppose that H is positive semi-definite and all elements of I - H ((w)) are less than 1.0, WM+DNN can converge to wo with long pNR which should longer with larger (w, - wo) (i.e., stronger weight regularization) relationship between pNR and the strength of weight regularization.
smaller 7 (wr)). In the next sections, focus the
4 NR PERIOD STUDY ON WEIGHT DECAY AND WEIGHT NOISE INSERTION
Weight decay is one of the most well-known regularization techniques (Zhang et al. 2018 and different from Le regularization in sense that weight decay is separated from the loss function calculation (Loshchilov & Hutter 2017). Weight decay performed as
(1 70w,) V C(w).
where 0 is a constant weight decay factor. Weight noise insertion is another regularization tech- nique aiminig at reaching flat minima (Goodfellow et al.) 2016; Hochreiter & Schmidhuber, 1995). Suppose that random Gaussian noise is added to weights such that w' + when (O, ml).. Then, L(w') = [[fw+c(x) y/² where æ, f are input, target, and pre- diction function, respectively. Using Taylor-series expansion to second-order terms, we obtain fwte (x:) Re fx: + Vf(x) + c ( Correspondingly, the loss function can also be approximated as
L(w + (x) yi + rB (x) y)v² fu (x)) + 1 V lw (x))
(4)
where the second term disappears near local minimum and the third term induces flat minima. Random noise insertion with other distribution models can be explained in a similar fashion (Good.
fellow al. 2016).
3

---

Under review as a conference paper at ICLR 2020
6.0e-844
6.0e-044
1.0c-03-
53.0
1.0c-03-
-9.00
Weight Deray Factor (0)
2.0e-03-
2.0c-03-
5.0c-03
5.0c-03
1.0e-02
52.4
1.0n-02
92.4
2.0e-02
2.0e-02
6.0e-02-
-1.88
5.0e-02-
-9..8
1.0e-02
1.0ee-1
2.0c-01-
91.2
2.0c-01-
91.2
5.0c-01
5.0c-01
1.0-+00-
50.5
1.0e-00- 00
90.5
2.0c+000
2.0c-00 00
5.0e+000
5.0e-00
90
100
2000
5000
1000
2000
30.3
30
1001
2000
500
10re
2011
90.0
Nou- Regularizstiinn Period
Nou- Rou-Reguarization Period
Figure 3: Model accuracy of ResNet-32 on CIFAR-IO using various NR period and amount of weight decay or noise for regularization (original model accuracy without regularization is 92.6% (Left): Weight decay. (Right): Uniform weight noise insertion.
5.0e-05
1.0e-04
115
0.
1.06-04
116.8
Welght Decay Factor (0)
2.0r-04-
2.0e044
5.0e-04-
.12
5.0
1.0c-03
1.0c-0
116.0
2.0e-03-
2.0-00
5.0e-03
8.0
1.0e-02
1.0e-02
115.2
2.0e-02
.01
2.0
5.0c-02-
1.0e001
1.0e-11
1.5e-01
1.5c-
2.0c-01-
2.0c-01
2.5e-01
2.5e-01
113.6
3.0c-01
96
3.0c-01
100
1161
2533
2053
Non-Regularization Period
Non-Regularkaation Period
Figure 4: Perplexity of LSTM model on PTB dataset using various NR period and amounts of weight decay or noise. (Left): Weight decay. (Right): Uniform weight noise insertion.
We study the impact of NR period on weight decay and weight noise insertion using ResNet-32 on CIFAR-10 model (He et al. 2016) and a long short-term memory (LSTM) model on PTB dataset (Zaremba et al.) 2014). For LSTM model, we use 2 layers with 200 hidden units and the hyper- parameter set introduced by Zaremba et al. (2014). For weight noise model, we plug ~ u((-a, +a) (uniform distribution) into Eq. (3) to simplify the experiments. Figure shows model accuracy of ResNet-32 given different NR period and weight decay factors. For both weight decay and weight noise insertion, the choice of 0 (representing the amount of each weight regularization) has clear correlation with NR period (refer to Appendix for training and test accuracy graphs). If we wish to apply stronger weight regularization, then such weight regularization should be conducted less frequently (i.e., larger weight decay factor requires longer NR period) to maximize the regularization effect and achieve high model accuracy. Similar observations are discovered by LSTM model on PTB as shown in Figure 4 Lower perplexity (indicating better generalization) is obtained when NR period increases as weight regularization becomes stronger for each regularization event.
Note that compared with a conventional weight decay factor selection (i.e., 0 in Eq. (3) when pNR 1), weight decay factor can approximately 1,000 times larger with pNR Rs 1, 000 in Figure and Figure 4 If ewr is the mean absolute difference given as (= F| w'I). where w'' is a weight vector after regularization, then Figure and Figure 4 imply that optimal ew/pNR seems to be constant. Consequently, optimal en, depends on p.NR, and hence, we argue that a wide exploration of various NR period (for occasional regularization) and different weight regularization strength is necessary in order to best utilize regularization effects a DNN model while previous attempts employ pNR = 1 only. In addition, the observation that stronger weight regularization is enabled by longer pNR is our basic training principle for model compression.
4

---

Under review as a conference paper at ICLR 2020
100
1152
144
Rank
60
1120
20
111.2
40
100
154
580
2787
2252
3710
4181
4046
5110
83P5
8s64
Non- Regularization Period
(a) Perplexity when the weights compressed by SVD.
1185
of 1itts
117.0
1123
2322
2737
3253
3716
4181
4040
3110
5873
0039
0504
Non- Rew.lariaiiioonn Period
(b) Perplexity when the weights compressed quantization based binary codes.
Pruning Itate
0.70-
117
0.85
0.90
118
2023
2787
3252
3716
4181
404c
5110
4375
6109
G384
caca
Non-Regularian Period
(c) Perplexity when the weights are compressed by Ingnitude-based pruning.
Figure 5: Model accuracy of an LSTM model on PTB compressed by quantization, low-rank ap- proximation pruning. Original perplexity without model compression 14..6. For more details, refer to Figure
NR PERIOD FOR MODEL COMPRESSION
As discussed, weight compression incurs much more complicated weight regularization model than weight decay or uniform weight noise insertion because 1) as shown in Figure [ diversified noise models need to combined describe weight regularization after model compression and 2) compression-awan training methods would reduce the strength of weight regularization as training performed with more epochs and weights converge to a compressed form. Nonetheless, we can conjecture that the best training scheme for model compression may require the condition of pNR + that can be empirically justified.
We apply weight quantization, low-rank approximation (SVD), and pruning to LSTM model on PTB that we selected for the previous section. We do not modify underlying training principles and the following simple strategy:
1) Train the model during NR period (as model compression is not being considered.)
2) Then, perform weight compression in the form of un' h(w)).
3) With new full-precision weight w', repeat the above two steps.
h(w) can be magnitude-based pruning (i.e., 1(w)=w if |w| is larger than certain threshold, or h( otherwise), ab for quantization, SVD function, or even as-yet undiscovered functions.
Figure 5 shows model accuracy associated with number of different sets of pNR and model com- pression strength (i.c., target rank for low-rank approximation and the number of quantization bits).
Optimal pNR p.R for the best model accuracy is definitely much larger than 1. To explain how Figure aligned with Section 4, we investigate the relationship between model accuracy and the average of ew/ /NR (e., Ellw- w'l]. where w' is weight vector after weight decay, SVD, pruning) during entire training. We first determine optimal ew ke/pNR (= Copt) for the best model accuracy. As discussed in Section 4, copt is assumed to be constant regardless of compression ratio decay
5

---

Under review as a conference paper at ICLR 2020
105
104
Ernorfoaak-000 100)
PPL 1000
0.08
124
Eroclpraming male-
PVLjpruning rate:
115-
PPL (P
PPLipresing rate_..))
0.28
103
0.13
Brrut (Pro) (wwoa//oo NP Kaye))
Emwripreaing
Perplexity (PPL)
102-
0,10
Perploxity ( (PPL)
114
0.00
Hapllxiityp
122
-0.00
101
0.08
0.04
190
100
-0.04
99-
11.04
116-
0.02
0.00
111
0.00
0.00
noo-Rogglartaation Period
Nou-Rogular isation Period
Non-Rogularization Period
(a) Weight Decay (copt=0.0007)
(b) SVD (Copt=0.08)
(c) Pruming (0op:=0.035)
Figure 6: Relationship between model accuracy and error (defined as the difference between ew wu/pNN /pNR and Copt) using PTB LSTM model when weights are regularized by weight decay, SVD, pruning.
factor, and obtained by finding hyper-paramettr sets associated with maximum model accuracy in Ligureaand Figure and by taking the average corresponding ew values. When error is defined to be lew. /pNR Eaptl Figure 6 shows test perplexity and error of PTB LSTM model with dif- ferent pNR. Such defined error Is affected by pNR as shown in Figure 6 and indeed, when error approaches to optimal value gain improved model accuracy. Unlike weight decay where ew is directly computed by decay factors, for model compression techniques, e, is not directly related to compression-relatee ure 4 shows clear correlation between decay factors and pNR for best model accuracy, Figure [ suggests that compression ratio and pNR are weakly correlated. Hence, pNR is hyper-parameter to be determined empirically for model compression. Nonetheless, optimal pNR is definitely much larger than 1, as shown in Figure 6 and decoupled from batch size selection. That means weight regularization for model compression needs to be conducted much less frequently compared with gradient descent since batch size selection considers generalization ability of gradient descent, not regularization effects.
syper--arameters (such ranks and pruning rates). As result, while Fig-
Note that periodic compression has been introduced in the literature to gradually improve com- pression ratio or automate hyper-paramess search process. DropPruning repeats dropping weights randomly and retraining the model while some previously dropped weights are unpruned until prun- ing rate reaches target number (Jia et al. 2018). Weights are incrementally quantized to improve model accuracy (Zhou et al. 2017) or the number of quantization bits can be controlled differently for each layer by doop based on reinforcement learning (Elthakeb et al. 2018). Structured pruning and fine-tuning process can be iterated to increase pruning rate (Molchanov et al. 2016)] Liu et al. 2017). All of these previous works assume pNR 1 (i.e., performing compression for every mini batch) while the goal is increasing compression ratio slowly or finding a set of hyper-parameters through iterative fine-tuning stages. Our proposed compression technique can be combined with such periodic compression methods (incremental compression or automatic hyper-parameter selec- tion are also applicable to our proposed method). In the work by He et al. (2018), soft filter pruning is conducted with pNR = epoch without analysis of why such occasional pruning improves model accuracy.
COMPARISON WITH PREVIOUS MODEL COMPRESSION TECHNIQUES
In this section, we compare some of previous model compression techniques with our compression scheme that introduces pNR and obviates special training algorithm modifications.
6.1 FINE-GRAINED WEIGHT PRUNING
The initial attempt of pruning weights was to locate redundant weights by computing the Hessian calculate the sensitivity of weights to the loss function (LeCun et al. 1990). However, such

---

Under review as a conference paper at ICLR 2020
Table
Pruning rate comparison using LeNet-300-100 and LeNet-5 models on MNIST dataset. DC
(Deep Compression) and Sparse VD represent a magnitude-based technique (Han et al. 2016) and
variational dropout method (Molchanov et 2017). respectively.
Weight
Pruning Rate (%)
Model
Layer
Size
DC
DNS
Sparse VD
Ours
FC1
235.2K
92
98.2
98.9
98.9
FC2
30K
91
98.2
97.2
96.0
LeNet-300-100
FC3
1K
74
94.5
62.0
62.0
Total
266.2K
92
98.2
98.6
98.4
Conv1
0.5K
34
85.8
67
60.0
Conv2
25K
88
96.9
98
97.0
LeNet-5
FCI
400K
92
99.3
99.8
99.8
FC2
5K
81
95.7
95
95.0
Total
430.5K
92
99.1
99.6
99.5
technique has been considered be practical due significant computation overhead for computing the Hessian. Magnitude-basee pruning (Han et al. 2015) has become popular because one can quickly find redundant weights by simply measuring the magnitude of weights. Since then, numerous researchers have realized higher compression ratio largely by introducing Bayesian inference modeling weights accompanying supplementary hyper-parameters.
For example, dynamic network surgery (DNS) (Guo al. 2016) permits weight splicing when separately stored full-precision weight becomes larger than a certain threshold. Optimizing splicing threshold values, however, necessitates extensive search space exploration. and thus, longer training time. Variational dropout method (Molchanov et al. (017) introduces an explicit Bayesian inference model for prior distribution of weights, which also induces various hyper-parameteen and increased computational complexity.
We perform magnitude- based pruning at every pNR step. As result, even though weights pruned and replaced with zero at pNR steps, pruned weights are still updated in full precision during NR period. If the amount of updates of a pruned weight grows large enough between regularization steps, then the weight pruned at last pNR step may not be pruned at the next pNR step. Such feature (i.e.. pruning decisions are fixed) is also utilized for weight splicing DNS (Guo et al. 2016). Weight splicing in DNS relies on a hysteresis function (demanding sophisticated line- tuning process with associated (yper-parameters) through our scheme, on the other hand, are newly determined at every pNR step.
consecutive
switch pruning decisions. Pruning decisions
We present experimental results with LeNet-5 and LeNet-300- models on MNIST dataset which are also reported by Guo et al. (2016); Molchanov al. (2017). LeNet-5 consists of 2 convolutional layers and 2 fully connected layers while fully connected layers construct LeNet-300- 100. We train both models for 20000 steps using Adam optimizer where batch size is 50. All the layers pruned at the same time and the pruning rate increases gradually following the equation introduced in Zhu & Guptal 2017):
P: = = (p: - ps) (1 t-4
t
E
(5)
15
where a constant, Py is the target pruning rate, p4 is the initial pruming rate, is the current step, and the pruning starts at training step 4; and reaches py at training step Ly- After ly steps, pruning rate is maintained
be py. For LeNet-5 and -NNet3300-100 4, Pi, E. are 8000 (step), 25(%), and 7, respectively. ty is 12000 (step) for LeNet-5 and 13000 (step) for LeNet-300-100 100. Note that these choices are not highly sensitive to test accuracy as discussed in Zhu & Gupta (0017). We exclude dropout to improve the accuracy of 1eNet-300-100 and LeNet-5 since pruning already works regularizer (Han et al.) 2015] Wan et al. (013). We keep the original learning schedule and the total number of training steps (no additional training time for model compression).
Table presents the comparison on pruning rates (see Appendix for test accuracy). Despite the simplicity, our pruning scheme produces higher pruning rate compared with DNS and similar com-
7

---

Under review as a conference paper at ICLR 2020
pared with variational dropout technique which involves much higher computational complexity. we use pNR==10 for LeNet-5 and pNRR=5 for LeNet-300-100.
For Table I
6.2 LOW-RANK APPROXIMATION
We apply our proposed occasional regularization algorithm integrated with Tucker decomposition Tucker 1966) to convolutional neural network (CNN) models and demonstrate superiority of p.RR- based scheme over conventional training methods. In CNNs, the convolution operation requires a 4D kernel tensor K. = RxxxdxxxT where each kernel has d x d dimension, S is the input feature map size, and T is the output feature map size. Then, following the Tucker decomposition algorithm, K. is decomposed into three components as
R. Kij.s. ΣΣ
Cairenn PS PT tiret
(6)
where Ci.yra.rr is the reduced kernel tensor, R, is the rank for input feature map dimension, Rg the rank for output feature map dimension, and ps and p" are 2D filter matrices to map Cr.j.r..ra
to j.et Each component is obtained to minimize the Frobenius norm of (K. - K.J.s.)) As
result, convolution layer is divided into three convolution layers, specifically, [1x11 convolution for Ps_ (d x d) convolution for Cij.ra.re. and (1 x 1) convolution for p" (Kim et al. 2016
fine-tuning procedure after the model is restructured and fixed (Lebedev et al., 2015 Kim et al. (0016). On the other hand, our training algorithm is conducted for Tucker decomposition as follows:
In prior tensor decomposition schemes, model training is performed
Step 1: Perform normal training for pNR (batches) without considering Tucker decomposition
Step 2: Calculate C, pS, and pr using Tucker decomposition obtain I.:
Step 3: Replace with K
Step 4: Go to Step with updated K
After repeating number of the above steps towards convergence, the entire training process should stop at Step 2, and then the final decomposed structure is extracted for inference. Because the model not restructured except the last step, Steps 2 and can be regarded as special steps to encourage wide search space exploration so as to find a compression-friendee local minimum where weight noise by decomposition does not noticeably degrade the loss function.
Using the pre-trained ResNet-32 model with CIFAR- 10 dataset (He et al. 2016; Kossaifi et al. 2019), we compare two training methods for Tucker decomposition: 1) typical training with de- composed model and 2) pNNR-based training, which maintains the original model structure and occasionally injects weight noise through decomposition. Using an SGD optimizer, both training methods follow the same learning schedule: learning rate is 0.1 for the first 100 epochs, 0.01 for the next 50 epochs, and 0.001 for the last 50 epochs. Except for the first layer, which is much smaller than the other layers, all convolution layers are compressed by Tucker decomposition with rank It, and R2 selected to be S and T multiplied by a constant number Rg (0.3 < R. S 0.7 in this ex- periment). Then, the compression ratio of convolution layer is (TST/(SF + R,, R, + TR,) = dT (T52 R2 + d2 R²ST +T2 which can be approximated to be 1/R if S = 7 and d > Re- pNR is chosen to be 200.
Figure shows test accuracy after Tucker decompositior by two different training methods. Note that test accuracy results are evaluated only at Step 3 where the training process can stop generate a decomposed structure. In Figure 7 across a wide range of compression ratios (determined by Re), the proposed scheme yields higher model accuracy compared to typical training. Note that even higher model accuracy than that of the pre-trained model can be achieved by our method if the compression ratio is small enough. In fact. Figure X shows that our technique improves training loss and test accuracy throughout the entire training process. Initially, the gap of training loss and test accuracy between pre-regularization and post-regularivation is large. Such a gap, however, quickly reduced through training epochs. Overall, ResNet-32 converges successfully through the
hhtps///ittub.cco//arny// DNdu/Deccoppse-CCNN
8

---

Under review as a conference paper at ICLR 2020
92.92
92.62
92.43
Pre PreIrained (92.63
92.21
Ancuracy (%)
92.09
91.75
91.49
91
90.36
Pruposed Scheme (p. R=200)
90.43
Typical Scheme (pN R=1)
89.77
Compression Ratio
Figure Test accuracy comparison on ResNet-32 using CIFAR-10 trained by typical training method and the proposed training method with various compression ratios. For the proposed scheme, test accuracy is measured only at Step that allows to extract a decomposed structure, and pNR is 200.
Typical Scheme(p/N
Propased (p.V R=200, Pust-Reg.)
Proposse (p.N 2-200, Pre-Reg.)
90
Tist Accuracy (%)
TTraning Loss
0.4
85-
Typical Schmmu(p R-1)
Proposed (p 2000, Post-Reg.))
Proposed (p- R-00, Pro-Reg.)
0.0
50
100 Epoch
150
200
100
150
200
Epoch
Figure 8: Training loss and test accuracy of ResNet-32 using CIFAR-10. For the proposed scheme, training loss and test accuracy are only monitored right before or after weight regularization for compression ppre-regularization post-regularization) Compression ratio is 2.8 with R,=0.5.
entire training process with lower training loss and higher test accuracy compared with typical training method.
For ResNet-34 on ImageNet experiments and VGG19 on CIFAR-10 (including additional compres- sion techniques), refer to Appendix.
7 CONCLUSION
In this paper, show that weight regularization should be decoupled from batch size enhance regularization effects. To accomplish such decoupling, we introduce a new hyper-parameter called non-regularization period or NR period during which weights are updated only for gradient compu- tations. A larger amount of weight decay and weight noise insertion need to be supported by longer NR period to maximize the regularization effect. We find that model compression with higher com- pression ratio is also better trained by longer NR period. We demonstrate that various DNN models can be compressed by pruning, quantization, and low-rank approximation successfully with longer NR period while the underlying training algorithm do not need to be modified.
REFERENCES
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv: 1410::/4100759

---

Under review as a conference paper at ICLR 2020
2014.
Minsik Cho and Daniel Brand. MEC: memory-efficient convolution for deep neural network. In International Conference on Machine Learning (ICML), pp. 815-824, 2017.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, pp. 31 3123-3131, 2015.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoe V. Le, and Andrew Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, pp. 1223-
1231, 2012.
Ahmed Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Esmacilzadeh. Releq: A reinforcement learning approach for deep quantization of neural networks. arXiv preprint 18Xiv: 1.01704, 2018.
K. Fatahalian, Sugerman, and P. Hanrahan. Understanding the efficiency of GPU algorithms for matrix-matrix multiplication. In Proceedings of the ACM SIGGRAPH//UROGRRPPIIII Confer- ence on Graphics Hardware, pp. 133-137, 2004.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //www.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Advances in Neural Information Processing Systems, 2016.
Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: exploiting binary struc- ture in deep CNNs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4040 4048, 2017.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808v:8808.666 068666 2018
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems, pp. 529-536, 1995.
Haipeng Jia, Xueshuang Xiang, Da Fan, Meiyu Huang, Changhao Sun, Qingliang Meng. Yang He, and Chen Chen. Droppruning for model compression. arXiv preprint 1rXiv:8812.02035, 2018.
Norman Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Ba- jwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg. John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi
10

---

Under review as a conference paper at ICLR 2020
Narayanaswami, Ray Ni, Kathy Nix. Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps. Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2016.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In International Conference Learning Representations (ICLR), 2016.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1-6, 2019. URL httth://mmmml
org/ pttp://ppesss///s/ wttss//20/188// 8777727772 1177.hmlll
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up up convolutional neural networks using fine-tuned CP-decomposition. In International Conference on Learning Representations (ICLR), 2015.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-605, 1990.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Li,Zhiqiang
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017.
Mitchell Marcus, Mary Ann Marcinkiewiez, and Beatrice Santorini. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist., 19(2):313-330, 1993.
Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning (ICML), pp. 2498-2507, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruming convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06444 2016.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv:6609.04747, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV,, 115())211-255) 2015. doi: 10.1007/s1 1263-015-0816
L. R. Tucker. Some mathematical notes three-mode factor analysis. Psychometrika, 31:279-311, 1966.
Twan van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv: 053500 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using DropConnect. In International Conference on Machine Learning (ICML), 2013.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi- quantization for recurrent neural networks. In International Conference Learning Representations (ICLR), 2018.

---

Under review as a conference paper at ICLR 2020
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv:1409.2329, 2014.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization, 2018.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint arXiv: 1.xiv::702.030444 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. In International Conference on Learning Representations (ICLR), 2017.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. CoRR, abs/1710.01873 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 SUPPLEMENTARY EXPERIMENTS FOR WEIGHT DECAY AND WEIGHT NOISE
5.0e-04-92
921
92.6
01.3
01.3
01.2
01.2
11.22
U1.3
B0.0
30.0
5.0e-04-00
91.9
93.4
91.3
91.3
91.4
111
51.22
D1.4
91.23
91.41
1.0e-033
926
01.5
01.3
51
91.4
01.5
01.3
01.3
1.0c-03
02.
01.5
91.3
91.4
91.4
11.2
0
01.4
91.5
Weight Decay Factor (0)
2.0e-03
91.5
2.0e-03
91.6
91.
81..8
11.33
91.8
91.2
1.0e-U2
M3.8
929
92.5
94.3
91.6
123.0
18.2
91.3
91.3
1.Ve-12
B31
MA.1
92.8
91.8
11.4
17.5
91.3
91.22
91.4
2.0e-02
92.
850
93.1
00.0
99.0
01.7
601.4
00.5
01.55
91.4
2.0e-11
050
83.2
92.9
91.8
01.22
01.02
01.5
001.2
10.00
5.0e-02
5.0n-02
1.0e-11
57.7
92.4
30.2
1.0e-01
90.6
784
85.8
87.8
98.
99.9
15.00
00.
01
5.0e-01
855
86.8
99.3
05
97.7
00
91.7
91.7
1.e4-00
07/0
291
63.0
40.
92
25
05
02
00
1.e-4000
75.
85,0
57.4
10
92.
900
05.
03
02.5
900
92.0
2.0e+00
57.8
21.2
80.0
54.5
59.5
05.0
13.1
00.0
2.0e+-00
62.
25.9
94.5
57.
90.
920
03.0
5.0c+000
5.0c+-00
73.1
1000
2000
5
1000
2000
Ron-Reeulaiizztion Period
Non- Rew.lariaaiiiinnn Period
Figure 9: Model accuracy of ResNer-32 on CIFAR-10 using various NR period and amount of weight regularization (original model accuracy without regularization 92.6%). (Left): Weight decay. (Right): Uniform weight noise insertion.
Table 2:
Model accuracy of ResNet-32 CIFAR-10 10 and LSTM model on PTB with various weight decay factor and corresponding pNR..
Weight Decay Factor(F)
Model
0
le-4
5e-4
le-3
5e-3
le-2
5e-2
Accuracy(% (%)
92.6
93.3
93.2
93.2
93.3
93.2
92.9
ResNet-32
optimal pNR
N/A
2
5
20
100
200
1000
Perplexity
114.6
108.1
97.7
97.1
97.1
97.0
97.2
LSTM on PTB
optimal pNR
N/A
10
100
93.0
Weight. Decay Factor (0)
5e-04
92.5
92.7
93.0
92.4 4
91.8
91.6
90.5
89.9
1c-03
91.9
92.8
93.0
92.6
92.5
92.0
91.0
90.2
91.5
5e-03
88.1
91.0
92.2
93.0
93.2
92.9
92.5
92.2
90.0
1e-02
85.7
87.9
91.1
92.55
92.7
93.
92.8
92.6
88.5
5c-02
50.0
79.5
85.2
87.7
90.4
91.9
92.6
92.9
1e-01
51.4
60.2
80.9
84.2
88.1
90.5
91.9
92.6
-7.00
5e-01
34.9
39.1
48.3
60.5
79.7
83.9
87.5
86.2
85.5
32
128
256
512
1024
Batch Size
Figure 10: Model accuracy (%) of ResNet-32 for various weight decay factors and batch size when pNR=1. Large batch size demands larger weight decay factors that is also reported by Loshchilov & Hutter (007))
13

---

Under review as a conference paper at ICLR 2020
5.0e-05
11.2
12.
113.9
13
14.
114
114
144
14
14
114
1.0e-04
108.4
111
113.1
13.
13.
114.3
124.5
114
11
1166
11..5
Weight Decay Factor (0)
2.0e-04
19.0
1:2.8
113.5
14
114.6
114.3
5.0e-04
98.6
98.9
101.3
108.5
111.3
122
113.
112.7
133.7
1110
1240
111.3
114
111
1114
111.1
1.0e-03
118.5
106.1
08.8
561
103.2
105.5
107.5
108.6
112
12.5
113.
113.6
114.0
114.
1138
114.1
2.0e-03
138.9
118.8
102.9
98.4
99.2
101
103.1
101.8
108.6
110.2
1019
112
113.
113.3
113.9
110.1
5.0e-03
161.3
139.1
113.9
103.5
86.8
M8.
99.2
100.3
104.4
106.9
103.8
11122
112
112.7
113.1
113.7
1.0e-02
233.6
152.
123.8
108
100.7
98 6
1017
104.3
108.3
110.
1113
111.7
112.6
11333
2.0e-02
309.0
172.1
188.5
11944
106.3
101.6
99.7
98.9
80088
104.9
107.0
108.8
110.1
111.2
112.6
5.0e-02-
697.2
252.1
150.0
10:.5
105.6
102.0
100
66.3
99.2
102.7
105.4
107.0
108.
110.3
111
1.0e-01
050.2
5:9.
163:22
1:30:
119.1
10:99
106.5
103.33
103.2
104.8
106.6
110.6
1.5e-01
678.0
185.7
126,2
15
110.2
106.5
98
164.9
109
2.0e-01
070.5
006.
230,1
L53.2
131.5
22.55
1.3.2
109.3
10L.0
36.0
98.88
101.0
102.3
103.8
105.9
108.9
2.5e-01
976.0
68330
7.7.7
165.7
141.1
28
120
1149
104.
915
58..3
013.5
100
101
104.2
7
3.0e-01
377.5
bry
639.1
2015
149.5
335
120 2
220.9
100
102
96.5
98.5
94.3
100
102
106.3
100
154
200
464
580
774
1161
2323
Non- Non-Regularization Period
(a) Initial learning rate 1.0
5.0e-05
110.8
12:
114.1
114.
149
114.3
145
14.7
1
11
115
1
1149
11
1.0e-04
108.0
11..0
113.1
13.9
1...
14.7
114.3
15.0
1446
14.6
114.9
114
114.7
14.5
1146
114.7
Weight Decay Factor (0)
2.0e-04
103.5
107.8
111.9
13
113.8
113
114.6
111
114.5
1112
120.7
1116
11.3
115.1
115.3
111.5
5.0e-04
97.7
101.9
108.1
10.5
112.8
13.5
113.8
113.7
174.7
14.4
114.8
114.
114.5
114.9
114.4
114.6
1.0e-03
97.1
97.8
103.5
107.8
111.1
.127
113.0
113.0
1.3.5
1100
114.3
114.5
116.
I1LS
111.
11LS
2.0e-03
101.7
9r.1
98.7
103.1
108.0
110.4
111.1
111.6
133.7
114.1
114.4
114.2
114.5
114.4
114.4
114.3
5.0e-03
119.0
104.6
97.1
97.6
101.5
104-4
106.3
10S
110.9
122
113.8
114.
113.9
13
114.
114.2
1.0e-02
139.1
18.55
97.0
97.6
101.6
108.0
113.0
113.6
114.2
113.9
2.0e-02
167.0
8
113.
101
97.0
06.8
97.7
103.5
106.5
100.7
111
112.6
113.3
113.7
5.0e-02
478.1
210:9
(4.2
19:6
104.9
100.0
088.3
97.3
57.2
98.88
104.0
107.3
108.5
109.7
1
122.4
1.0e-01
G87.1
493.6
173.
141
121.0
10:44
105.8
10222
17.55
97.0
10.2
101
103.5
105.2
107.6
110.1
1.5e-01
352.1
6001
231.6
155.6
133.2
121
114.5
LOS.9
90.7
9V
97.5
99.1
100. 5
102.3
101.7
108.1
2.0e=01
685.5
686
331.8
1.8
143.7
29
121.9
16.0
T0SS
98
97.1
97
08.7
99.7
102.4
106.1
2.5e-01
883.7
686.1
$18.7
218.
152.0
.33.1
129.6
22
107 22
101
97.6
97.2
88.8
98 4
100.7
104.0
3.0c-01
587
701.8
253
161.1
46
136
1110
104
97.2
97.2
97.7
99.5
102
20
30
50
100
154
290
464
580
774
1161
2323
Non- Non-Regularization Period
(b) Initial learning rate 1.5
3.0e-05
111.0
13:0
114.6
14.6
115.1 115.
114.7
11455
155.0
15.5
155.
115.
15.22
115.0
15.
115.4
1.0e-04
-07.5
107.5
11.33
11..5
11
1...5
11
115.6
1199
125.
119
1552
115.
115.1
15.
111
15.0
Weight Decay Factor (0)
2.0e-04
97.4
101.0
107.8
11
113.1
13.4
114.1
11.55
134.6
15.2
115.0
115
114.9
115.
15.
155.2
5.0e-04
103.3
107.6
1...8
1126
122.8
113.
1.4.2
111.9
115.0
1119
15.3
115.1
1.0e-03
119.0
9:22
101.4
104.0
106.2
107.6
10133
12
115.0
114.5
114.9
115.1
2.0e-03
130.1
1186
102.6
97.5
87.2
0
1014
113.9
18
114.2
5.0e-03
204..5
139.8
114.1
102.3
97.6
96.9
97.3
98.5
103.00
105.9
109.7
111.
11L.8
112.8
113.5
114.5
1.0e-02
254.6
15277
123.8
108.1
90.8
97.7
56.9
96.9
100.0
103
107.4 4
100
110.6
112.0
112.0
113.6
2.0e-02
711.0
217
140.2
19:55
105.9
T01.
08.2
97
97.2
90.
1033.
107.1
108. 1
109.
10.88
112.4
5.0e-02
700.3
353
150.5
128
101.8
105.0
101.S
98.5
96.7
97 7
101.2
104.2
106. 1
107.6 6
108.
111.4
1.0e-01
501.9
711.
200,D
141.8
120.8
12.1
106.4
103.1
97.5
97.0
98.7
101.5
102.0 0
105.0
107.2
109.5
1.5e-01
8800
706 7
214.4
1.49
127.6
16.7
111.4
106.5
88.9
96..5
97.3
919.7
101.0 0
10335
105.8
108
2.0e-01
200
253.8
133.3
12L.6
114.8
110.6
96.9
101.3
1013
107.
2.5e-01
549.4
193.3
145.1
123.0
56.2
90.6
101
105.2
3.0e-01
072.6
600.
F11.5
251
1000
130.
24.7
100.3
102.
58.0
97.3
00.8
103.
30
100
154
200
464
580
774
1161
2323
Non- Non-Regularization Period
(c) Initial learning rate
2.0
Figure 11: Perplexity of LSTM model on PTB dataset using various NR period and amounts of weight decay (original perplexity without regularization is 114.60).
14

---

Under review as a conference paper at ICLR 2020
5.0e-05
114.2
114.8
14
114.6
114.6
14.5
114.
114.7
114.7
114.6
114.8
1.0e-04
114.2
114.4
11:2
1144
114
114.5
105.1
114.6
113.9
114
114.5
114.1
2.0e-04
114.7
114.5
1139
1139
114.0
114.0
5.0e-04
111.
113.
114.7
115.0
114.5
1119
114.2
114.5
L10.3
11100
111.7
114.5
1. 0e-03
114.1
114.1
11366
11433
114.
114.7
114.2
11433
114.2
114.5
114.2
114.0
114.4
114.4
2. 0e-003
112.8
1111
114.4
1142
114.5
114.3
114.7
114.6
114.8
1119
114.7
114.5
111.3
1141
114.5
114.5
5.0e-03
139.4
11488
113.2
114.4
114.2
114
114.2
115.0
114.2
11488
114.5
114.5
114.3
114.1
114.
114.0
1.0e-02
404,9
153.9
114.2
13.22
113.7
13.9
114.3
1138
104.2
13.9
114.
113
114.4
114.7
114.6
114.8
2.0e-02
215.2
418
170.8
118.7
1:2.9
13.1
113.7
11.99
103.9
114.2
104.0
114.7
114.3
113.9
114.
114.0
5.0e-02-
200.7
2099
521.9
118.6
13.5
113.0
122
1240
13.9
1143
113
113.9
1112
111.9
114.6
e 1.0e-01
152.3
544.6
168.7
122.9
113.8
1142
114.8
1.5e-01
186.8
196.3
272.9
450.8
178.7
1:38.8
125.0
12.6
1146
114.0
113.5
2.0e-01
182.3
190.4
208.1
229.
501.9
192.
200.8
153.8
115.7
12.8
1.2.7
113.9
113.8
11000
113.9
111.
2.5e-01
1706
182.8
108.2
20355
79669
1967
474.9
524
142
19.3
122.7
113 2
13
18
11
1140
3.0e-01
182.9
181
150.6
199.0
212.6 2
230
229.0
430
2626
43
115.9
112
113
122
9
113.3
1136
100
154
290
464
580
774
1161
2323
Non- Non-Regullrization Period
(a) Initial learning rate 1.0
5.0e-05
-15.2
114.
114.6
1143
114.5
114.7
114.7
114
14.2
114.9
114.8
115.0
1.0e-04
114.3
114.6
114.4
114.9
114.5
114.5
114.6
114.6
134.6
115.0
114.8
114.4
114.6
114.8
114.
114.9
2.0e-04
114.6
111.3
114.9
114.3
115.0
1113
114.3
115.1
114.9
1118
114.6
111.1
111.9
114.5
114.5
11..5
5.0e-04
115.1
114.7
114.6
115.0
114.2
114.9
114.5
114.6
114.7
114.8
114.6
114.5
114.7
114.3
114.7
114.6
1.0e-03-0
1148
1119
114.8
114.8
114.4
111.7
114.5
114.7
114.7
111.6
115.1
115.0
111.8
111.7
114.
115.0
2.0e-03
114.5
114
115.2
114.
114.8
114.5
114.5
114.5
114.5
114.8
1148
114.6
110.7
114.3
114.9
114.7
5. 5.0e-03-H
114.2
2142
1443
114.5
114.6
114.8
114.5
114
114.9
114.7
115.0
114.6
114.9
114.6
114.8
114.9
1.0e-02
119.2
11..8
1142
114.5
115.0
114.4
114.7
134.6
114.4
114.6
114G
114.6
2.0e-02-
540.9
2
115.6
11377
113.8
11:.
114.8
114.7
114.8
114.9
114.8
114.9
114.3
5.0e-02
217.8
216.7
455.4
277.0
126.4
16.6
114.5
113
13:.88
13377
114.5
114.9
114.8
114.4
114.5
114.5
1.0e-01
212.5 202.9
219.8
225
5988.8
5:10
268877
174.9
118.9
11.66
11:333
118.0
11..
114.0
1140
114.5
1.5e-01
200 192.0
211.5
220
222.7
26133
326.2
50L.
2.0.5
131.9
115.2
113.5
113.8
113.6
113.9
114.2
2.0e-01
180.8
192.6
2077
216
224.0
221
19
258
6503
428
132.
117
114.5
113
11
113.9
2.5e-01
180.7
190.
199.3
212
217.0
220.
223.2
222
352.8
005.
233.0
132.5
122.2
16.5
11000
113.5
3.0c-01
194.0
189
194.3
2019
219.2
224
221.3
223
237
394
213
149
126
113.5
2
20
30
50
100
154
290
464
580
774
1161
2323
Non- Ron-Regularizatinn Period
(b) Initial learning rate
1.5
5.0e-05
115.5
115.3
115.5
1148
1961
114.9
115.2
114
B85
114.
195.6
115.2
115.2
115.6
115.3
115.4
1.0e-04
-01-04-115.3
115.3
115.1
115.1
11177
1LLS
1119
115.1
115.1
115.1
1118
115.2
115.8
112.
115.1
115.1
115.5
2.0e-04
115.3
11:66
115.4
115.5
114.8
115.5
114.7
115.3
115.0
115.5
115.1
114.6
115.3
115.1
115.1
115.8
5.0e-04
114.9
115.2
115.1
115.6
116.0
115.1
115.0
115.9
115.2
115.3
115.5
115.3
111.7
116.1
115.3
115.2
1.0e=03
1148
115.1
115.0
113.8
115.3
115.1
115.3
115.3
115.5
115.3
115.2
115.1
114.5
2.0e-03
156.7
1
114.7
115.2
114.8
115.4
115.3
115.3
115.4
115.3
115.6
115.0
115.5
115.1
115.3
5.0e-03
256.0
350.4
141.2
118.0
115.0
114.7
114.5
115.0
115.2
115.1
115.4
114.8
115.0
115.1
115.1
115.9
1.0e-02-
221.7
216.8
125.2
147
119.1
16.0
114.3
115.4
114.7
114.6
114.6
114.9
1114
115.3
115.5
114.8
2.0e-02-
224.4
229 4
241.0
336.6
212.8
143.5
126.9
120.8
135.0
114.6
135.0
114.8
115.0
114.8
115.0
115.4
5.0e-02
218.8
221 8
226.8
244.
364.7
2016.
2083.0
160.7
1201.2
115.
115.0
115.2
114.9
115.0
114.5
115.7
@ 1.0e-01
200.3
217
221.8
229.1
233.7
296.7
377.8
302.6
159.6
128.
L.6.7
114.5
1118
115.0
11.88
11LS
1.5e-01
214.5
212
224.6 278.6
72677
241
758
312.7
277.0
169
127.5
116.
114.9
115.3
115.1
114.9
2.0e-01
200.8
230
230.2
236.
138.3
115.7
114.
115.3
2.5e-01
198.0
212.6
255.9
1664
114.2
3.0e-01
197.6 L98
208.
7
218
228.6
225
231
23
266
201 8
166.
121
115
100
200
464
580
774
1161
2323
Non- Non-Regullaization Period
(c) Initial learning rate
2.0
Figure 12: Perplexity of LSTM model on PTB dataset using various NR period and amounts of uniform noise (original perplexity without regularization is 11.660).
15

---

Under review as a conference paper at ICLR 2020
Weight Decay (WD)
Noise (X ~ U(-a,a))
85
Accuracy (%)
90
(pNR, WD)
(pNRR. a)
(2, 0.01)
0.01)
85
(2, 0.01)
85
train (100, 0.5)
train (100,
80
test (100, 0.5)
80-
teet: (100, 1.0)
train (2000, 5.0)
train (2000, 5.0)
test
(2000, 5.0]
test (2000, 5.0)
73
75
50
100
160
200
100
150
200
Epoch
Epoch
Training and test accuracy of ResNet-32 on CIFAR-10 for variuos NR period and the amount of weight regularization. (Left): Weight decay. (Right): Uniform weight noise.
Figure 13:
A.2 PARAMETER PRUNING
Table Model accuracy comparison using LeNet-300-I1 and LeNet-5 models on MNIST dataset. DC (Deep Compression) and Sparse VD represent magnitude-basee technique (Han et al. 2016) and variational dropout method (Molchanov et al. 2017). respectively.
Accuracy (%)
Model
DC
DNS
Sparse VD
Ours
LeNet-300-100
98.4
98.0
98.1
98.1
LeNet-5
99.2
99.1
99.2
99.1
We apply pNR--based pruning an RNN model to verify the effectiveness of pNR. We choose an LSTM model (Zaremba et al., 2014) on the PTB dataset (Marcus et al. 1993). Following the model structure given in Zaremba et al. (2014), and a softmax layer. The number LSTM units in a layer can be 200, 650, or 1500, depending on the model configurations (referred as small, medium, and large model, respectively). The accuracy is measured by Perplexity Per Word (PPW), denoted simply by perplexity in this paper. We apply gradual pruning with E' large) to the pre-trained PTB models. pNNt-bbsed pruning for the PTB models is performed using pNR 100 and the initial learning rate is 2.0 for the medium model (1.0 for pre-training) and 1.0 for the large model (1.0 for pre-training) while the learning policy remains to be the same as in Zaremba et al. 22014).
model consists of an embedding layer, 2 LSTM layers,
t; = 0, Pt 0, ły 3rd epoch (for medium) or 5th epoch (for
Table Comparison on perplexity using various pruning rates. py the target pruning rates for the embedded layer, L.STM layer, and softmax layer.
Perplexity
Model Size
Pruning Method
Py=
0%
80%
85%
90%
95%
97.5%
Medium
Zhu & Gupta 2017)
83.37
83.87
85.17
87.86
96.30
113.6
(19.8M)
Proposed Scheme
83.78
81.54
82.62
84.64
93.39
110.4
Large
Zhu & Gupta 2017)
78.45
77.52
78.31
80.24
87.83
103.20
(66M)
Proposed Scheme
78.07
77.39
77.73
78.28
84.69
99.69
For all of the pruning rates selected, 4sows that our compression scheme improves perplexity better than the technique Zhu & Gupta (017)) which is based on ana et al. (2015). The superiority of A.R-based pruning is partly supported by the observation that non-zero weights successfully avoid to be small through retraining while the conventional pruning still keeps near near-zero zero (unmasked) weights depicted in Figure 15
16

---

conference Under review as a conference paper at ICLR 2020
176.2
134.4
119.6
1122
132.0
111.2
100.0
118.5
12.0
135.6
114.0
122.6
118
117
116
115
114
113
11
1133
1
1.S.
117.4
120.7
119.1
139. 4
115.2
114.0
115.1
115 5
118.5
1
1.
2
114.8
114.
114.2
1146
117.6
127
116.5
117.6
117.7
135.1
4504
2
114.3
114.2
112. 9
114.3
1174
MAA
117.4
117.6
116.4
134.1
114.8
1141
114.0
1142
1164
06775
117.7
119.0
116.8
133.2
6078
114.9
114.
1112
114.3
11688
5110
EE
$110
116.9
117.5
115.8
128.0
6110
Figure 14: Relationship between test perplexity and pNR using PTB LSTM model.
114.4
1L4.
114.5
114.7
117.1
4644
4046
117.0
117.8
116.0
126.2
4640
114.3
115.6
133.
114.2
116.2
1181
S
9718
116.2
116.8
115.6
123.6 6
4181
114.3
115.4
1133
113.3
115.7
32272
116.5
116.2
114.5
120.3
3710
114.4
113.9
113.4
113.0
114.3
3201
1224
2737
116.4
115.6
114.3
1194
32552
113.7
113.
113.5
113
114.3
3787
122
1122
2022
1161
Non- Mon-Rogularization Poriod
[Quantization]
116.2
116.6
115.3
116.0
115.0
113.8
115.3
116.6
2787
Non- Non-Regut Regularigas aariaaattin tnn Period
114.2
114.7 114.5
113.5
113.7
112.9
11266
1132
113.0
113.3
114.0
2333
1161
Non- Regularization Period
[Pruning]
[SVD]
2
2
114.1
113.9
112.7
113.4
774
115,7
115.6
112.8
113.1
1161
EEL THE 1ELL
22
115.3
114.5
112.6 6
112.7
774
114.9
113.9
114.2
113.1
113.
600
1882
464
114.0
114.0
1L4.0
113.1
1134
104
114.7
114.3
111.6
113.0
580
1::11
200
115.0
114.0
1L4.7
114.2
114.0
2000
114.4
113.8
111.7
113.2
464
122
1::11
114.7
115.5
114.3
114.7
113.8
154
114.0
113.6
112.5
115.9
290
11:.
100
115.6
113.
113.
114.9
114.7
1134
113.1
113.2
118.3 3
&
115.5
116.1
113.5
113.4
113.2
50
113.5
112.0
114.1
120.3
100
116.1
116.1
1162
11566
116.0
40
LA
LA
30
112.1
11L.8
117.7
125.0 D
600
115.2
110.5
116.5
113.8
115.8
30
S
2
X
20
111.2
111.9
118.5
120.8
40
116.2
116.2
115.5
116.3
115.6
20
2
112.3
112.1
120.9
128,0 0
30
116.6
116.5
117.3
116.3
116.8
T
A
112.1
112.5
123. 1
1311
117.1
116.2
116.6
117 0
117.2
111.1
114.3
128.6 6
135.2
117.6
116.0
LLT.
117.3 3
118.4
117.9
118.0
118.3
1176
117.2
11
1180
133.1
138.
T2
10
200
0.70
0.75
0.50
0.85
0.90
Rank
# 0 Fill
17

---

Under review as a conference paper at ICLR 2020
1600
1600
1400
1400
1200
1200
1000
1000
Count
Count
800
800
600
600
400
400
200
200
0
0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
Weight Value
Weight Value
Figure 15: Weight distribution of LSTM layer of the medium PTB model after retraining with (Left) magnitude-based pruning and (Right) pNNR-based pruning with 90% pruning rate. Our compression scheme incurs sharp drop the count of near-zero weights.
A.3 TUCKER DECOMPOSITION WITH OCCASIONAL REGULARIZATION
To investigate the effect of NR period local minima exploration with ResNet-32 CIFAR-IO, Figure [ the changes of loss function and weight magnitude values incurred by occasional regularization. In Figure [6]left) AL/ C is given as the loss function increase AL (due weight regularization at pNR steps) divided by L. which is the loss function value right before weight regularization. In Figure 16 right), Aw is defined as || Il /N /N(w). where is the entire set of weights to be compressed, w is the set of weights regularized by Tucker decomposition, N (w) the number of elements of w. and || f is the Frobenius norm of X. Initially, fluctuates with large corresponding AC. Then, both AL and Aw decrease and Figure [6] shows that occasional regularization finds flatter local minima (in the view of Tucker decomposition) successfully. When the learning rate is reduced 100th and I 150th epochs, AL and Aw decrease significantly because of a lot reduced local minima exploration space. In other words, occasional regularization helps an optimizer to detect local minimum where Tucker decomposition does not alter the loss function value noticeably.
250
0.010
/ L(%)
A
200
0.008
150
0.006
100
0.004
Vol www.
0.002
-50
0.000
150
200
Epoch
Epoch
Figure 16: Difference of training loss function and average Frobenius norm weight values by Step 2 and Step 3 of Figure] 2 R. 0.5 and pNR 200 are used.
18

---

Under review as a conference paper at ICLR 2020
A.4 2-DIMENSIONAL SVD ENABLED BY OCCASIONAL REGULARIZATION
In this subsection, we discuss why 2D SVD needs to be investigated for CNNs and how occasional regularization enables a training process for 2D SVD.
A.4.1 ISSUES OF 2D SVD ON CONVOLUTION LAYERS
Convolution can be performed by matrix multiplication if an input matrix is transformed into Toeplitz matrix with redundancy and a weight kernel is reshaped into a T x (S x d x d) matrix (i.e..& a lowered matrix) Chetlur et al. (014)) Then, commodity computing systems (such CPUs and GPUs) can use libraries such as Basic Linear Algebra Subroutines (BLAS) without dedicated hardware resources for convolution Cho & Brand (2017). Some recently developed DNN accelera- tors, such as Google's Tensor Processing Unit (TPU) et al. (2017). are also focused on matrix multiplication acceleration (usually with reduced precision).
For BLAS- CNN inference, reshaping a 4D tensor I. and performing SVD is preferred for low-rank approximation rather than relatively inefficient Tucker decomposition followed by a low- ering technique. However, critical problem with SVD (with lowered matrix) for convolution layers is that two decomposed matrices by SVD do not present corresponding (decomposed) con- volution layers, because of intermediate lowering steps. As result, fine-tuning methods requiring a structurally modified model for training are not available for convolution layers to be compressed by SVD. On the other hand, occasional regularization does not alter the model structure for training. For occasional regularization, SVD can be performed as a way to feed noise into a weight kernel IC for every regularization step. Once training stops at regularization step, the final weight values can be decomposed by SVD and used for inference with reduced memory footprint and computations. In other words, occasional regularization enables SVD SVD-aware training for CNNs.
10000
No SVD SVD, (128 128) Gling
Skewed Matrix
Tiled Skewed Matrix
8000
SVD, (16 16) tiling
mj. neem
Connt
0000
SVD, (1 x 1) tiling
2000
(nxx (). (r (rxm) m)
(In r), (r mv44) x
0.0
0.5
Value
1.5
2.0
3.0
Figure 17: Skewed matrix and a tiling technique are illustrated on the left side, while the right side presents distributions of weights after SVD with different tiling schemes (only positive weights are included).
A.4.2 TILING-BASED SVD FOR SKEWED WEIGHT MATRICES
A reshaped kernel matrix K E (xSxxxd is usually skewed matrix where row-wise dimension (n) is smaller than column-wise dimension (m) as shown in Figure (i.e., n < m). A range of available rank r for SVD, then, is constrained by small n and the compression ratio is approximated to be n/r. If such a skewed matrix is divided into four tiles as shown in Figure [7] and the four tiles do not share much common chateracteristics, then tiling-based SVD can be a better approximator and rank can be further reduced without increasing approximation error. Moreover, fast matrix multiplication is usually implemented by tiling technique in hardware to improve the weight reuse rate Fatahalian et al. (2004). Hence, tiling could be natural choice not only for high-quality SVD but also for high-performance hardware operations.
To investigate the impact of tiling on weight distributions after SVD, we tested a (1024 x 1024) random weight matrix in which elements follow a Gaussian distribution. A weight matrix is divided by (1 x 1), (16 x 16), or (128 128) tiles (then, each tile is a submatrix of (1024 x 1024), (64 64), or (8 x 8) size). Each tile is compressed by SVD to achieve the same overall compression ratio of 4x for all of the three cases. As described in Figure 17 (on the right side), increasing the number of tiles tends to increase the count of near zero and large weights (i.c., variance of weight values increases). Figure can be explained by sampling theory where decreasing the number of random samples (of
19

---

Under review as a conference paper at ICLR 2020
small tile size) increases the variance of sample mean. In short, tiling affects the variance of weights after SVD (while the impact of such variance on model accuracy should empirically studied).
Table 5: Test accuracy(%) of ResNet-32 model using CIFAR-10 dataset while the 9 largest con- volution layers (T=S=64, d=3) are compressed by SVD using different tiling configurations. For each tile size, rank is selected to achieve compression ratio of 2x 4x. pNR==00 is used for occasional regularization.
Compression Ratio
Size Each Tile
Pre-Trained
64x64
32x32
16x 16
8x8
2x
93.34 (r=16)
93.11 (r=8)
93.01 (r=4)
93.23 (r=2)
92.63
4x
92.94 (r=8)
92.97 (r=4)
93.00 (r=2)
92.81 (r=1)
We applied the tiling technique and SVD to the 9 largest convolution layers of ResNet-32 using the CIFAR-10 dataset. Weights of selected layers reshaped into 64 (64 3 3) matrices with the tiling configurations described in Table We perform training with the same learning schedule and pNNR(=000) used in Section Compared the test accuracy of the pre-trained model (=92.63%) all of the compressed models in Table S achieves higher model accuracy due to the regularization effect of our compression scheme. Note that for each target compression ratio, the relationship between tile size and model accuracy is not clear. Hence, various configurations of tile size need to be explored to enhance model accuracy, even though variation of model accuracy for different tile size is small.
93
93.03(1
93.11
93
92.95
02.96(r=
92.92
92.78(r=8)
92.78(r=3)
02.78
92.78(
Pro-Trained (92.633
92.62
Accuracy (%)
92
922.43
92.20
-92.07
92
91.96 (r=4)
Typical Scheme (pN R=1)
991.00
91
Tucker Decomp- R=20n)
Tiled SVD with (16.52.52)(pNN
#91.41
Tiled SVD with (16,166166(pNN1122000)
Tiled SVD with (B.8.8))pNN
91
SVD(pNR=200)
-00.84
1
2
3
4
Compression Ratio
Figure 18: Test accuracy of ResNet-32 model using CIFAR-10 with various target compression ratio and decomposition methods. Except the first small convolution layer, all layers are compressed by the same compression ratio. Convolution layers (=16, 32, or 64). For tiled SVD, three groups (of different S) are tiled in (k, x k1), (h2 x k2), or (ka k3) tile size. (ki, k2, ka) configuration is described in legends.
be grouped according to 3 different 5 values
A.5 EXPERIMENTAL RESULTS ON LOW-RANK APPROXIMATION FOR CNNS
In this subsection, we apply low-rank approximation trained by occasional regularization to various CNN models.
Figure 18 summarizes the test accuracy values of ResNet-32 (with CIFAR-10 dataset) compressed by various low-rank approximation techniques. Note that tiled SVD and normal SVD enabled
20

---

Under review as a conference paper at ICLR 2020
only by occasional regularization, which obviates model structure modification during training. All configurations in Figure 18 use the same learning rate scheduling and the number of training epochs described in Section 3. Results show that tiled SVD yields the best test accuracy and test accu- racy is not highly sensitive to tile configuration. SVD presents competitive model accuracy for small compression ratios. As compression ratio increases, however, model accuracy using SVD signifi- cantly degrades. From Figure [8] tiled SVD associated with occasional regularization is clearly the best low-rank approximation scheme.
Table 6: Comparison on various low-rank approximation schemes of VGG19 (using CIFAR-10 dataset). To focus on convolution layers only, fully-connected layers are compressed by 8x and trained by occasional regularization. Then, fully-connected layers are frozen and convolution layers are compressed (except small layers of 128) by Tucker decomposition or tiled SVD.
Comp. Scheme
Parameter
Weight Size
FLOPs
Accuracy (%)
Pre-Trained
18,98M
647.87M
92.37
Tucker
R,=0066
9.14M (2.08x)
319.99M (2.02x)
91.97
Decomposition
R.=0.5
6.71M (2.83x)
235.74M
(2.75x)
91.79
(Typical
R.=0.45
5,49M (3.45x)
191.77M
(3.38x)
91.36
Scheme)
R.=0.4
4.61M (4.11x)
161.60M
x)
91.11
Tiled
64x64 (r=16)
9,49M (2.00x)
316.28M
(2.04x)
92.42
SVD
64x64 (r=11)
6.52M (2.91x)
214.25M
(3.02x)
92.33
(Occasional
64x64 (r=10)
5,93M (3.20x)
193.85M
(3.34x)
92.23
Regularization,
64x64 (r=9)
5.55M (3.41x)
173.44M (3.73x)
92.22
pNVR=300)
64x64 (r=8)
4.74M (4.00x)
153.04M (4.33x)
92.07
We compare Tucker decomposition trained by typical fine-tuning process and tiled SVD trained by occasional regularization using the VGG19 mode with CIFAR-10. Since this work mainly dis- cusses compression on convolution layers, fully-connected layers of VGG19 are compressed and fixed before compression of convolution layers (refer Appendix for details the structure of VGGI9). Except for small layers with S < 128 (that presents small compression ratio as well), all convolution layers are compressed with the same compression ratio. During 300 epochs to train convolution layers, learning rate is initially 0.01 and is then halved every 50 epochs. In the case of tiled SVD, pNR is 300 and tile size is fixed be x4 (recall that the choice of pNR and tile size do not affect model accuracy significantly). As described in Table [ while Tucker de- composition with conventional fine-tuning shows degraded model accuracy through various R., occasional-regullrizztii assisted tiled SVD presents noticeably higher model accuracy.
Proposed Scheme, Tiled SVD(64:04,r=10)
Typical Scheme, Ticker Decomp.
Test A ccuracy (Top-1, %)
70
2.5
Training Loss
2.0
1.5-
Proposed Schemne, Tiked SVD(64.64,-110
Typical Scheme, Tucker Decomp..
1.0
20
40
60
80
0
20
40
60
80
Epoch
Epoch
Figure 19: Comparison of two compression schemes on training loss and (top-1) test accuracy of ResNet-34 model using ImageNet. pNR2=500.
21

---

Under review as a conference paper at ICLR 2020
We also test proposed low-rank approximation training technique with the ResNet-34 mode He et al. (2016) using the ImageNet dataset Russakovsky et al. (2015). pre-trained ResNet-34 is fine-tuned for Tucker decomposition (with conventional training) or tiled SVD (with occasional reg- ularization) using the learning rate of 0.01 for the first 20 epochs, 0.001 for the next 30 epochs, and 0.0001 for the remaining 30 epochs. Similar our previous experiments, the same compression tio is applied to all layers except the layers with S < 128 (such exceptional layers consist of 1.4% of the entire model). In the case of Tucker decomposition, selected convolution layers compressed with R. 0.46 to achieve an overall compression of 3.1x. For tiled SVD, lowered matrices are tiled and each tile of (64x64 64) size is decomposed with r=10 to match an overall compression of tiled SVD yields better training loss and test accuracy compared Tucker decomposition with typical training. At the end of the training epoch in Figure T9 tiled SVD and Tucker decomposition achieves 73.00% and 72.31% % for top-1 test accuracy, and 91.12% and 90.73% for top-5 test accuracy, while the pre-trained model shows 73.26% (top-1) and 91.24% (top-5).
3.1x. As shown in Figure 19 occasional- rewarionnnlrggglllaaattoooooa-oo
A.6 LOWERING TECHNIQUE FOR CNNS
Figure 20 describes a kernel matrix reshaped from a 4D kernel tensor and an imput feature map ma- trix in the form of Toeplitz matrix. At the cost of redundant memory usage create Toeplitz matrix, lowering enables matrix multiplication which can be efficiently implemented by BLAS li- braries. A kernal matrix can be decomposed by 2D SVD.
Kernel
Input Feature Map
Reshaping
4D Tensor Conv. Layer (T...d.))
to 2D Tensor
Sxdx
"when stride is with no padding
(H+1-d) (W+1-d)
Input Feature Map (S, H, W)
300
230
4
LRA via SVD
121415
6
(rank-k)
k
Lowering (exim2col)
Sxdxd
21 21
Figure 20: An example of lowering technique using im2col.
A.7 MODEL DESCRIPTIONS FOR LOw-RANK APPROXIMATION EXPERIMENTS
In this section, we describe model structures and layers selected for low-rank approximation experi- ments. Small layers close to the input not compressed because both weight size and compression rate are too small.
22

---

Under review as a conference paper at ICLR 2020
Table 7: Convolution Layers of ResNet-32 for CIFAR-10
# of layers
T
d
Weight Size
Decomposed
16
3
0.4K ( 0.1%)
No
10
16
16
22.5K ( 5.0%)
Yes
1
32
16
3
4.5K ( 1.0%)
Yes
9
32
32
81.0K (18.0%)
Yes
1
64
32
3
18.0K ( 4.0%)
Yes
9
64
324.0K (71.9%)
Yes
Total
450.4K (100.0%)
Table 8: Convolution and Fully-connected (FC) Layers of VGG-19 for CIFAR-10
Type
# #f layers
S
d
Weight Size
Decomposed
64
3
0.002M
0.01%)
No
1
64
64
3
0.035M
0.18%)
No
1
128
64
3
0.070M 0.36%)
No
1
128
128
3
0.141M 0.72%)
Yes
Conv.
1
256
128
3
0.281M 1.44%)
Yes
3
256
256
3
1.688M 8.61%)
Yes
I
512
256
3
1.125M 5.74%)
Yes
7
512
512
3
15.75M
(80.37%)
Yes
2
512
512
0.500M
2.55%)
Yes
1
512
10
0.005M 0.02%
Yes
Total
19.597M (100.0%)
Table 9: Convolution Layers of ResNet-34 for ImageNet
of layers
d
Weight Size
Decomposed
64
7
0,01M ( 0,04%)
No
6
64
64
3
0.21M ( 1.05%)
No
I
128
64
3
0.07M 0.35%)
No
7
128
128
3
0.98M 4.90%)
Yes
I
256
128
3
0.28M 1.40%)
Yes
11
256
256
3
6.18M (30.77%)
Yes
1
512
256
3
1.13M 5.59%)
Yes
5
512
512
3
11,25M (55,94%)
Yes
Total
20.1 IM (100.0%)
23


Reviews:
[{"review_id": "rkeT1bLCFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think that the paper is very well written, I like it. The authors localized a phenomenon  and demonstrated how to exploit it. I trust the results because I performed exactly the same experiments for CIFAR-10 with longer non-regularization periods and found that there is no effect (this is also that the authors show in the paper)  but I didn't test on other datasets and obviously didn't think about potential benefits for compression. \nSince model compression is not my field, I would just make a general suggestion to include and compare to state-of-the-art techniques. \n\n---\nUpdate: \nThe authors updated the paper and replied to all reviewers in detail. I still think that the paper should be accepted. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx5P39pKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a model compression algorithm, where training consists in alternatingly updating a full precision weight vector using SGD for a number of steps N, and then compressing the weight vector to a quantized or low-rank representation. The authors present an empirical study of how the properties of this algorithm change as they change the number N.\n\nThe contribution is marginal, and the paper is hard to read.\nPeriodically compressing networks weights, rather than at every iteration, is not new. The paper is missing a \"related work\" section that embeds the proposed algorithm in the literature.\nThe experiments are small scale and not exhaustive and there is very little comparison to previous work and alternative methods.\n\nQuestions/comments for the authors:\n- When you say \"regularization\" you seem to usually mean \"compression\"? This was confusing to me.\n- What do you mean by \"asynchronous regularization\"? You seem to mean periodic compression?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xn2oQs5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates compression-aware training and introduces a new hyper-parameter called \"Non-regularization period\". Basically, the paper proposes to apply weight regularization and compression less frequently so that they can use stronger regularization/compression, hence achieving high compression ratio and model accuracy.\n\nOverall, the idea of asynchronous regularization and compression is interesting and worth further investigation. However, I vote for a reject for now because\n(1) The paper is hard to follow and the writing can be significantly improved especially for the abstract and introduction. I felt rather confused when I first read the abstract and introduction. The authors frequently use the word \"weight update\", however it is unclear whether it refers to gradient update or the update caused by regularization/compression. Besides, the title is also a little confusing. There's little discussion about the choice of batch size and I don't really understand how weight regularization is decoupled from batch size after reading the paper. I suggest the authors to think more about the title.\n(2) The reason why such a training scheme improves model compression is unclear and needs further investigations. In the paper, the authors first interpret model compression as a way of inducing weight decay (and random weight noise). Particularly, the model accuracy is roughly constant over different value of NR period (as shown in Figure 3 and Figure 4) when weight decay is used. However, the results of model compression are quite different (according to Figure 5). The best performance is only achieved with very large NR period in the case of SVD and pruning. I think the authors should give some explanations for that difference. Also, I notice that quantization behaves more similarly to weight decay and it requires further discussion.\n\nTo sum up, I like the idea of asynchronous regularization/compression, but I'm not quite satisfied with current version of paper. I encourage the authors to improve their writing and add more discussions (see my point (2)) to the paper. I'm willing to increase my score if the authors can address my concerns.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to apply regularizers such as weight decay or weight noise only periodically, rather than every epoch. It investigates how the "non-regularization period", or period between regularization steps, interacts with other hyperparameters. 

Overall, the writing feels somewhat scattered, and it is hard to identify a clear argument for why the NRP should help. Certainly one could save computation this way, but regularizers like weight decay or weight noise incur only a small computational cost anyway. One explicit claim from the paper is that a higher NRP allows larger regularization. There's a sense in which this is demonstrated, though not a very interesting sense: Figure 4 shows that the weight decay strength should be adjusted proportionally to the NRP. But varying the parameters in this way simply results in an unbiased (but noisier) estimate of gradients of exactly the same regularization penalty, so I don't think there's much surprising here.

Similarly, Section 3 argues that a higher NRP allows for larger stochastic perturbations, which makes it easier to escape local optima. But this isn't demonstrated experimentally, nor does it seem obvious that stochasticity will help find a better local optimum.

Overall, I think this paper needs substantial cleanup before it's ready to be published at a venue such as ICLR.

Decision: Reject