Low Rank Training of Deep Neural Networks for Emerging Memory Technology
Authors: Albert Gural
Phillip Nadeau
Mehul Tikekar
Boris Murmann
Venue:  (2020)
Keywords: low rank training
kronecker sum
emerging memory
non-volatile memory
rram
reram
federated learning

Abstract: The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables applications such as real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. In this work, we address four key challenges for training on edge devices with non-volatile memory: low weight update density, weight quantization, low auxiliary memory, and online learning. We present a low-rank training scheme that addresses these four challenges while maintaining computational efficiency. We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight updates.

Under review as a conference paper at ICLR 2020
LOW-RANK TRAINING OF DEEP NEURAL NETWORKS FOR EMERGING MEMORY TECHNOLOGY
Anonymous authors Paper under double-bling review
ABSTRACT
The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. this work, we address two key challenges for training edge devices with non-volatile memory: low write density and low auxiliary memory. We present a low-rank training scheme that addresses these challenges while maintaining computational efficiency. We then demonstrate the technique on representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and number of weight writes.
INTRODUCTION
Deep neural networks have shown remarkable performance on a variety of challenging inference tasks. As the energy efficiency of deep-learning inference accelerators improves, some models are now being deployed directly edge devices to take advantage of increased privacy, reduced network bandwidth, and lower inference latency. Despite edge deployment, training happens predominately in the cloud. This limits the privacy advantages running models on-device and results in static models that do not adapt to evolving data distributions in the field.
Efforts aimed at on-device training address some of these challenges. Federated learning aims to keep data on-device by training models in distributed fashion (Koneený al. 2016). On-device model customization has been achieved by techniques such as weight-imprinting (Qi et al. 2018). by retraining limited sets of layers. On-chip training has also been demonstrated for handling hardware imperfections (Zhang et al.) 2017 Gonugondla et al. 2018). Despite this progress with small models, on-chip training larger models is bottlenecked by the limited memory size and compute horsepower of edge processors.
Emerging non-volatile (NVM) memories such resistive random access memory (RRAM) have shown great promise for energy and area-efficient inference Yuu 2018). However, on-chip training requires large number of writes the memory, and RRAM writes cost significantly more energy than reads (e.g., 10.9 pJ/bit versus 1.76 pJ/bit (Wu et al. 2019)). Additionally, RRAM endurance is on the order of 10 writes (Grossi et al. 2019, shortening the lifetime of a device due memory writes for on-chip training.
In this paper, we present an online training scheme amenable to NVM memories to enable next generation edge devices. Our contributions are (1) an algorithm called Streaming Kronecker Sum Approximation (SKS), and its analysis, which addresses the two key challenges of low write density and low auxiliary memory; (2) two techniques "gradient max- max-norm" and "streaming batch norm" to help training specifically in the online setting; (3) a suite of adaptation experiments to demonstrate the advantages of our approach.

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Efficient training for resistive arrays. Several works have aimed at improving the efficiency of training algorithms on resistive arrays. Of the three weight-commutations required in training (forward, backprop, and weight update), weight updates are the hardest to parallelize using the array structure. Stochastic weight updates (Gokmen & Vlasov 2016) allow programming of all cells in a crossbar at once, as opposed to row/column-win 2015) can also be used to update all the weights at once. Several works have proposed new memory structures to improve the efficiency of training (Soudry et al. 2015) Ambrogio et al.. 2018). The number of writes has also been quantified in the context of chip-in-the-lopy training Yuu et al. (016).
updating. Online Manhattan rule updating (Zamanidoost et al.,
Distributed gradient descent. Distributed training in the data center is another problem that suffers from expensive weight updates. Here, the model is replicated onto many compute nodes and in each training iteration, the mini-batch split across the nodes to compute gradients. The distributed gradi- ents are then accumulated on a central node that computes the updated weights and broadcasts them. These systems can be limited by communication bandwidth, and compressed gradient techniques (Aji & Heafield 2017) have therefore been developed. In Lin al. (2017), the gradients are accumulated over multiple training iterations on each compute node and only gradients that exceed a threshold are communicated back to the central node. In the context of on-chip training with NVM, this method helps reduce the number of weight updates. However, the gradient accumulator requires as much memory as the weights themselves, which negates the density benefits of NVM.
Low-Rank Training, Our work draws heavily from previous low-rank training schemes that have largely been developed for use in recurrent neural networks to uncouple the training memory require- ments from the number of time steps inherent to the standard truncated backpropagation through time (TBPTT) training algorithm. Algorithms developed since then address the memory problem in- clude Real-Time Recurrent Learning RT (Williams & Zipser 1989), Unbiased Online Recurrent Optimization (UORO) (Tallec & Ollivier, 2017), Kronecker Factored RTRL (KF-RTRL) (Mujikal al.] 2018), and Optimal Kronecker Sums (OK) (Benzing al. 2019). These latter few techniques rely on the weight gradients
weight-vector product looking like sum of outer products (i.e., Kronecker sums) of input vectors with backpropagated errors. Instead of storing growing number of these sums, they can be approximated with a low-rank representation involving fewer sums.
TRAINING NON- NON- VOLATIEE MEMORY
The meat of most deep learning systems are many weight matrix activation vector products W... a. Fully- Fully-connected (dense) layers use them explicitly: ald = a (Wifaa + bld)) ) for layer E, where is non-linear activation function (more details discussed in detail in Appendix C.1 Recurrent neural networks use one or many matrix-vecto products per recurrent cell. Convolutional layers can also be interpreted in terms of matrix-vector products by unrolling the input feature map into strided convolution-kernel-ee
slices. Then, each matrix-vecte product takes one such input slice and maps it to all channels of the corresponding output pixel (more details are discussed in Appendix C.2)
The ubiquity of matrix-vector products allows us to adapt the techniques discussed in "Low-Rank Training" of Section 2 other network architectures. Instead of reducing the memory across time steps, we can reduce the memory across training samples in the case of a traditional feedforward neural network. However, in traditional training (e.g., on GPU), this technique does not confer advantages. Traditional training platforms often have ample memory to store a batch of activations and backpropagated gradients, and the weight updates AW can be applied directly to the weights W once they are computed, allowing temporary activation memory to be deleted. The benefits of low-rank training only become apparent when looking at the challenges of proposed NVM devices:
Low write density (LWD). In NVM, writing to weights at every sample is costly in energy, time, and endurance. These concerns are exacerbated in multilevel cells, which require several steps of an iterative write-verify cycle program the desired level. We therefore want to minimize the number of writes to NVM.
Low auxiliary memory (LAM). NVM is the densest form of memory. In 40nm technology. RRAM IT-IR bitcells @ 0.085 um? (Chou et al. 2018) are 2.8x smaller than 6T SRAM cells @ 0.242 um? TSMC 2019). Therefore, NVM should be used to store the memory-intensin weights. By the same
2

---

Under review as a conference paper at ICLR 2020
token, other on-chip memory should come close the size of the on-chip NVM. In particular, if our bitit NVM stores a weight matrix of size n, X n2 we should use at most r(n, + no auxiliary non-NVM memory, where is small constant. Despite these space limitations, the reason might opt to use auxiliary (large, high endurance, low energy) memory is because there are places where writes are frequent, violating LWD if we
use NVM.
In the traditional minibatch SGD setting with batch size B, an upper limit on the write density per cell per sample is easily seen: 1/B. However, store such a batch of updates without intermediate writes to NVM would require auxiliary memory proportional to B. Therefore, trade-off becomes apparent. If B reduced, LAM is satisfied at the cost of LWD. B raised, LWD satisfied the cost of LAM. Using low-rank training techniques, the auxiliary memory requirements are decoupled from the batch size, allowing us to increase while satisfying both LWD and LAME Additionally, because the low-rank representation uses so little memory, a larger bitwidth can be used, potentially allowing for gradient accumulation in a way that is not possible with low bitwidth NVM weights. In the next section, we elaborate on the low-rank training method.
4 LOW-RANK TRAINING METHOD
Let (00) ((0) = (a( + b be the standard affine transformation building block of some larger network, (1) (inot(2(1) and (t)) (z)) with prediction loss L(yp (s)
Di )))),, where (xx)), y))
e.g. Up
is the 1hh training sample pair. Then weight gradient V ( = (z (0) ()) = dz (1) @ a where
dz(0) V 22( C()) minibatch SGD) weight update accumulates this gradient over B samples:
AW- = - >E 1 dz (9) @ (3) for learning rate
For rank-r training scheme, approximate the sum Σe_ dz (1) @a(() by iteratively updating two rank- matrices L Rr,xr, R R= with each new outer product: LR + rankRReuee LR" (6) @ a ()). Therefore, at each sample, we convert the rank-q = r 1 system LR + dz @ ())
into the rank-r LR,. In the next sections, we discuss how to compute rank Reduce.
4.1 OPTIMAL KRONECKER SUM APPROXIMATION (OK)
One option for estimator, which is implemented by selecting the top r components of a singular value decompo- sition (SVD) of X. However, naive implementation is computationally infeasible and biased: E|rank (educe(X) + X. Benzing et al. (2019) solves these problems by proposing a minimum variance unbiased estimator for ronk which they call the OK algorithm
X) ) to convert from rank q + +1x 1X to rank is minimum error
The OK algorithm can be understood in two key steps: first, an efficient method of computing the SVD of Kronecker sum; second, a method of splitting the singular value matrix Σ into two rank-r matrices whose outer product is a minimum-variance unbiased estimate of Σ. Details can be found
in their paper, however we include high-level explanation in Sections 4.1.I and 4.1.2 to aid our discussions. Note that our variable notation differs from Benzing et al. 2019
4.1.1 EFFICIENT SVD OF KRONECKER SUMS
Let L = (1) and R [R, (1) so that LRT = LĀ. T + ( Recall that rank Reduce should turn rank-q LR into updated rank-r ËR
QR-facoorize L = Q Rr and R = QRRa where QL E Ro,xq QR E Raax are orthogonal so that LR = QR, QT. C
Me.x9,QRR R, E R989, Then we can find the SVD of C UV in ((q²) time (Cline & Dhillon 2006), making computationally feasible on small devices. Now we have:
'This can alternately be achieved by sub-sampling the training data by r/B where is the OK rank. The purpose of using low-rank estimate is that for the memory cost, it significantly more informational than the sub-sampled data, allowing for faster training convergence.
Their target application differs slightly in that they handle matrix vector Kronecker sums rather than vector vector Kronecker sums.
3

---

Under review as a conference paper at ICLR 2020
LR =QL(UC-V (QUU))))A
(1)
which gives the SVD of LR since QuUc and Qa are orthogonal and Σ is diagonal. This SVD computation has time complexity of (m, +") and space complexity of (m;
4.1.2 MINIMUM VARIANCE, UNBIASED ESTIMATE OF
In Benzing et al. 2019). it is shown that the problem of finding a rank-r minimum variance unbiased be reduced to the problem of finding rank-r minimum variance unbiased estimator of Σ and plugging it in to ①
estimator of LR
Further, it shown that such an optimal approximator for Σ diag(o .2..... ,2....,07))
Tg), where oi > 02 2 ... 2 0g will involve keeping the m largest singular values and mixing the smaller singular values mm. ......... og within their (k 1) (k + 1) submatrix with m, k defined below. Let:
m = min s.t. (g 1)0 S
J
h: = q m
==
amk
Jghi
81
81
1=m
Note that | = Let E (++ be orthogonal such that left nullspace is the span of
Tp. Then XX I Zz20 Now, lets f-1, 1] (++1)x be uniform random signs and define:
81
X, (s ,.. x..))
Z
X
k
Σ, = ER = diag (Voi..... mm-1+ Z)
(2)
where @ is an element-wise product. Then Σ Σ is minimum variance, unbiased rank-r approximation of Σ. Plugging Σ into 1.
LR (Q.U6)
(Q) -Q0
(3)
Thus, L = UU RR and Å QnVRER Raix gives us a minimum variance, unbiased, rank-r approximation LR....
4.2 STREAMING KRONECKER SUM APPROXIMATION (SKS)
Although the standalone OK algorithm presented by Benzing al. (019) has good asymptotic computational complexity, our vector-vector outer product sum use case permits further optimizations. In this section we present these optimizations, and we refer readers the explicit implementation called Streaming Kronecker Sum Approximation (SKS) in Algorithm of Appendix A
4.2.1 MAINTAIN ORTHOGONAL Q1, Qn
The main optimization a method of avoiding recomputing the QR factorization of and R at every step. Instead, keep track of orthogonal matrices QL. QR. and weightings c, such that
L Qz diag( Cx [ir) and R Qn diag( Cz [irl- Upon receiving new sample, single inner loop of the numerically-stanl used to update QL and QR. The orthogonal basis coefficients CL = Q dz and cR = a
modified
(MGS) algorithm (Björck) 1967) can be
computed during MGS can be used to find the new value of C = c₁C₁ + diag(cz).
"The fact that it is unbiased: Σ
Σ can be easily verified.
4

---

Under review as a conference paper at ICLR 2020
Bn=E After computing Σい En in 2). orthogonalize these matrices into Σい Σn = Qq.. Then from (3, have LR (Q. (+ With this formulation, can maintain orthogonality in QL QR R by setting:
Qu+QUUeQu
QR - QRVCQ=
C2 diag(R..
These matrix multiplies require (n + +n))9²) multiplications, this optimization does not improve asymptotic complexity bounds. This optimization may nonetheless be practically significant since matrix multiplies are easy parallelize and would typically not be the bottleneck of the computation compared to SchmidS The next section discusses how to orthogonalize Σg efficiently and why (R,, R is diagonal.
4.2.2 ORTHOGONALIZATION OF Σg
Orthogonalization of E, is relatively straightforward. From (2). the columns of Σn are orthogonal since Z is orthogonal. However, they do not have unit norm. We can therefore pull the norm into separate diagonal matrix R2 with diagonal elements VCz:
m 0 X. Qx
VC (Vo1,.... mm
$1/k
q
4.2.3 FINDING ORTHONORMAL BASIS X
We generated X by finding an orthonormal basis that was orthogonal to vector Eg that could have X X = 2o An efficient method of producing this basis is through Householder
matrices (ao, X) = I - vo" /||=|12 where v = 20p e and (Co. X) a + x A: + matrix
with first column 2i and remaining columns X (Householder 1958 user1551
2013
4.2.4 EFFICIENCY COMPARISONS TO STANDARD APPROACH
The OK/SKS methods require O[ + n, + 9)4²) operations per sample and O(,;,oq)) operations after collecting B samples, giving an amortized cost of ((m; + n, + 9)4² + n.n.q B) operations per sample. Meanwhile, a standard approach expands the Kronecker sum at each sample, costing (n,n,, operations per sample. If < B, n,, n. then the low rank method is superior minibatch SGD in both memory and computational cost.
5 CONVEX CONVERGENCE
SKS introduces variance into the gradient estimates, so here we analyze the implications for online convex convergence. We analyze the case of strongly convex loss landscapes f'(")) for flattened weight vector wt and online sample l. In Appendix we show that with inverse squareroot learning rate, when the loss landscape Hessians satisfy O cl (=)"(w") and under constraint (4) for the size of gradient errors s², where w" the optimal offline weight vector, the online regret (5) is sublinear in the number of online steps T. We can approximate ||=|| and show that convex convergence is likely when (6) is satisfied in the biased, zero-variance case (equivalent to raw SVD, ie., not applying Section 2). or when is satisfied in the unbiased, minimum-variance case.
T
||''|| ~ 2
Glleo w"
(4)
R(T)
Σ
f'(w')
Σ t=1
f (w")
(5)
B
2
2 5 1|
2
11
w0
(6)
Σ
0
(1,i)
1
w0 112
Equations 6 7 suggest conditions under which fast convergence may more or less likely and also point methods for improving convergence. We discuss these in more detail in Appendix B.3

---

Under review as a conference paper at ICLR 2020
5.1 CONVERGENCE EXPERIMENTS
We validate (4) with several linear regression experiments on static input batch X E R 1022x 100 and target Y, R256x100 In Figure 1 a). Gaussian noise different strengths (represented by different colors) is added to the true batch gradients at each update step. Notice that convergence slows significantly the right of the dashed lines, which the region where no longer holds"
In Figure [b), we validate Equations 4 7 by testing the SVD and SKS cases with rank r = 10. In these particular experiments, SKS adds too much variance, causing it to operate the right of the dashed lines. However, both SVD and SKS can be seen to reduce their variance as training progresses. In the case SVD, is able continue training tracks the right dashed line.
3:30
3936
M83 Bedd)
38
16-4
10"
16/9
18a
10:1
367
207
10"
10P
200
220
1013
18"
11'
23t
2pm
Gradient Error Variance
Gradient Error Variance
(a) True gradients with artificial noise
(b) SVD/SKS gradients over learning rates
Figure 1: In both plots, the solid line with markers plots the loss vs. gradient error variance (LHS of across 50 steps of SGD for several different setups. The left dashed line represents the RHS of and the right dashed line is the RHS with C instead of c.
6 IMPLEMENTATION DETAILS
Quantization. The NN is quantized in both the forward and backward directions with uniform power-of-2 quantization, where the clipping ranges are fixed at the start of training Weights are quantized to 8 bits between 11 and 1, biases to 16 bits between -8 and 8, activations to 8 bits between and 2, and gradients to 8 bits between -1 and 1. Both the weights W and weight updates A AW quantized to the same LSB so that weights cannot be used for accumulation beyond the fixed quantization dynamic range. This is in contrast to using high bitwidth (Zhou et al. 2016 Banner al. 2018) floating point accumulators. See Appendix D for more details on quantization.
Gradient Max-Norming, State-of-the-at methods in training, such as Adam (Kingma Bal 2014). auxiliary memory per parameter to normalize the gradients. Unfortunately, we lack the memory budget to support these additional variables, especially if they must be updated every sample Instead, propose dividing each gradient tensor by the maximum absolute value of its elements. This stabilizes the range of gradients across samples. See Appendix [ for more details on gradient max-norming. In the experiments, we refer to this method as "max-norm" (opposite "no-norm"),
Streaming Batch Normalization. Batch normalization (Ioffe & Szegedy 2015) is a powerful technique for improving training performance which has been suggested to work by smoothing the loss landscape (Santurkar et al.) 2018). We hypothesize that this may be especially helpful when parameters are quantized as in our case. However, the online setting, we receive samples one-at-a-tinn rather than in batches. We therefore propose streaming batch norm that uses moving average statistics rather than batch statistics as described in detail in Appendix F
4 discussed Appendix B.1 < r4, so we substitute in c/2l|w² - wi|l with the minimum non-zero Eigenvalue of the Hessian è when plotting the RHS of 4
$ Future work might look into how change these clipping ranges, but this beyond the scope of this paper. *SKS could potentially approximate Adam. SKS on a² dz² a, dz allows for a low-rank approximation of the variance of the gradients, however, this is unlikely to work well because of numerical stability (e.g., estimated variances might he negative).

---

Under review as a conference paper at ICLR 2020
7 EXPERIMENTS
7.1 ADAPTATION EXPERIMENTS
To test the effectiveness of SKS, experiments are performed on a representative CNN with four 3 x 3 convolution layers and two fully-connected layers. We generate "offline" and "online" datasets based on MNIST (see Appendix G including one in which the statistical distribution shifts every 10k images. We then optimize online SGD and rank SKS model for fair comparison (see Appendix HD. To see the importance of different training techniques, we run several ablations in Appendix Finally, we compare these different training schemes in different environments, meant to model real life. In these hypothetical scenarios, a model first trained on the offline training set, and is then deployed to number of devices at the edge that make supervised predictions (they make prediction, then are told what the correct prediction would have been).
We present results four hypothetical scenarios. First, control case where both exter- nal/environment and internal/NV VM drift statistics are exactly the same as during offline training. Second, a where the input image statistical distribution shifts every 10k samples, selecting from augmentations such as spatial transforms and background gradients (see Section G Third and fourth cases where the NVM drifts from the programmed values, roughly modeling NVM memory degradation. In the third case, Gaussian noise is applied to the weights as if each weight was single multi-level memory cell whose analog value drifted in Brownian way. In the fourth case. random bit flips are applied as if cach weight was represented by memory cells (see Appendix G for details). For each hypothetical scenario, we plot five different training schemes: pure quantized inference (no training), bias-only training, standard SGD training, SKS training, and SKS training with max-normed gradients. SGD training and for training biases, parameters are updated at every step in an online fashion. These are seen as different colored curves in Figure 2
3.0
EasCae
Bix
842
souts More
550
19
40
Samboe
Sumple
(a) Original Dataset
(b) Distribution Shifts
500 Sconeatto
500 Wodoomm
sccMi Nive
124
100
127
(c) Analog/ Gaussian Weight Drift
(d) Digital/ Bit-Flip Weight Drift
Figure 2: Adaptation of various training schemes over four different training environments (a) to (d). In each training environment, the top plot shows the exponential moving averages (0.999) the per-sample online accuracy of the five training schemes, while the bottom plot shows the maximum number updates applied to any given convolution or fully-connected kernel memory cell. For the distribution shifts in (b), the enabled augmentations at each contiguous 10k samples is shown (CD= class distribution, ST spatial transforms, BG = background gradients, WN white noise).
Inference does best in the control case, but does poorly in adaptation experiments. SGD doesn't 't improve significantly on bias-only training, likely because SGD cannot accumulate gradients less than weight LSB. SKS, on the other hand, shows significant improvement, especially after several
7

---

Under review as a conference paper at ICLR 2020
thousand samples in the weight drift cases. Additionally, SKS shows about three orders of magnitude improvement compared to SGD in the worst case number of weight updates. Much of this reduction is due the convolutions, where updates connected writes still important because of potential energy savings. SKS/max-norm performs best in terms of accuracy across all environments and has similar weight update cost to SKS/no-norm.
applied at each pixel. However, reduction in fully-
7.2 TRANSFER LEARNING AND ALGORITHM COMPARISONS
To test the broader applicability of low rank training techniques, run several experiments on ImageNet with ResNet-34 (Deng et al. 2009; He et al. 2016), potentially realistic target for dense NVM inference on-chip. For ImageNet-size images, updating the low-rank approximation at each pixel quickly becomes infeasible, both because of the ingle-threaded nature of the algorithm, and because of the increased variance of the estimate at larger batch sizes. Instead, focus training the final layer weights (1000 x 512). ResNet-34 weights are initialized to those from Paszke et al. (2017) and the convolution layers are used generate feature vectors for 10k ImageNet training images' which are quantized and fed to a one-layer quantized neural network. To speed up experiments, the layer weights are initialized the pretrain weights. modulated by random noise that causes inference top-1 accuracy to fall to 52.7% + 0.9%. In Table | we see that the unbiased SKS has the strongest recovery accuracies, although biased SVD also does quite well. The high-variance UORO and true SGD have weak non-existent recoveries.
Table Accuracy recovery beyond inference (%. mean with standard deviation from 5 random seeds) between different algorithms (all with max-norm; effective batch size B 100 applicable), tested different ranks (r), and learning rates (1)) Optimal learning rates are bolded.
7
0.003
0.010
0.030
0.100
0.300
Algorithm
r
SGD
+0.3 = 0.2
0.2
+0.3 0.2
+0.9 0.2
-3.9 0.8
UORO
1
+0.4 0.2
0.4
-..
1.6
-31.7
SVD
1
0.2
+.0
1.04
-.9.4
1.0.7
2
+.44
+.75
4.6
4.9
-39
4
+.4
+.7
+.8
-3.
0.8
8
+.3
+..
+.9
1.0
+298
SKS
I
0.2
+.2
0..7
-277
2.6
2
0.2
+.34
-0.1
+.9
1.1
0.2
+.2
+1.9
18.0
-.11
8
0.2
+1.1
+.7
+..8
-15.8
8 CONCLUSION
We demonstrated the potential for SKS to solve the major challenges facing online training on NVM-based edge devices: low write density and low auxiliary memory. SKS computationally- efficient, memory-light algorithm capable of decoupling batch size from auxiliary memory, allowing larger effective batch sizes, and consequently lower write densities. Additionally, we noted that SKS may allow for training under severe weight quantization constraints rudimentary gradient accumulations are handled by the L, R matrices, which can have high bitwidths (as opposed to SGD, which may squash small gradients 0). We found expressions for when SKS might have better convergence properties. Across a variety of online adaptation problems and large- scale transfer learning demonstration, SKS shown to match exceed the performance of SGD while using a small fraction of the number of updates. Finally, we suspect that these techniques could be applied to broader range of problems. Auxiliary memory minimization may be analogous to communication minimization in training strategies such as federated learning, where gradient compression important.
Thee decision to use training data is deliberate, however experiments on out-of-sample images, such as Recht ct al. (2019) show similar hehavior.
Quantization ranges chosen optimize accuracy and different from those Section
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alham Fikri Aji and Kenneth Hcafield. Sparse communication for distributed gradient descent. arXiv preprint XXX:11704.05022 2017.
Stefano Ambrogio, Pritish Narayanan, Hsinyu Tsai, Robert M. Shelby, Irem Boybat, Carmelo di Nolfo, Severin Sidler, Massimo Giordano, Martina Bodini, Nathan C. P. Farinha, Benjamin Killeen, Christina Cheng, Yassine Jaoudi, and Geoffrey W. Burr. Equivalent-accurage accelerated neural-network training using analogue memory. Nature, 558(7708):60-67 June 2018. ISSN 1476-4687. doi: 10. 10003/44186--11---11
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pp. 5145-5153, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. 2013.
Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal Kronecker-sum approximation real time recurrent learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Learning, volume 97 of Proceedings of Machine Learning Research, pp. 604 613, Long Beach, California, USA, 09-15 Jun 2019. PMLR. benzzng 9a. html
Machine
iggs ml press/v977
Åke Björck. Solving linear least squares problems by gram-schmidt orthogonalization. BIT Numerical Mathematics, 7(1):1-21, 1967.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
C. Chou, Z. Lin, P. Tseng, C. Li, C. Chang, W. Chen, Y. Chih, and T. J. Chang. An N40 256K 44 embedded RRAM macro with SL-precharge SA and low-voltage current limiter to improve read and write performance. In 2018 IEEE International Solid State Circuits Conference (ISSCC), pp. 478 480, February 2018. doi: 10.1 (109//SSCC.211888100333.3
Alan Kaylor Cline and Inderjit S Dhillon. Computation of the singular value decomposition, 2006.
Deng, W. Dong, R. Socher, L.-J. Li. Li, and L. Fei-Fei. ImageNet: Large-Scall Hierarchical Image Database. In CVPR09, 2009.
M. Ernestus. Elastic transformation of an image in python. httpp::/ggit..itttt com/
2016.
Tayfun Gokmen and Yurii Vlasov. Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices. Frontiers in Neuroscience, 10, July 2016. ISSN 1662-453X. doi: 10.3389/ fnins.2016.00333.
S. K. Gonugondla, M. Kang, and N. R. Shanbhag. A Pariation-Tolerant In-Memory Machine Learning Classifier via On-Chip Training. IEEE Journal of Solid-State Circuits, 53(11)33133-317, November 2018. doi: 10.1 0000///SS.2011.2877777
Priya Goyal, Piotr Dollár, Ross Girshick. Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: arXiv:/706.02677 2017.
A. Grossi, E. Vianello. M. M. Sabry, M. Barlas, L. Grenouillet, J. Coignus, Beigne. T. Wu, Q Le, M. K. Wootters, C. Zambelli, E. Nowak, and S. Mitra. Resistive ram endurance: Array-level characterization and correction techniques targeting deep learning applications. IEEE Transactions on Electron Devices, 66(3): (6(3)::28112 1883):28111288 12888, March 2019. doi: 10. 10.1 10.110997ED.2019222444
S Haykin. Adaptive filter theory, ser. always learning, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Alston S Househokder. Unitary triangularization of a nonsymmetric matrix. Journal of the ACM (JACM), 5(4):339-34) 1958.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: (rXiv:/502.0316, 2015.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXXiv:4412.999 2014.
Jakub Koneený, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. CoRR, abs/1610.02527, 2016. URL
http::/arxiv.
http:///xxivv.rrgo www..orggaab........ aat//abs/11100 02527
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278)
1998.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv: 172.0887,, 2017.
Fuyun Ling, Dimitris Manolakis, and John Proakis. A recursive modified gram-schmidt algorithm for least-squares estimation. IEEE transactions on acoustics, speech, and signal processing. 34(4): 829-836, 1986.
Asier Mujika, Florian Meier, and Angelika Steger. Approximating real timetimme recurrent learning with random kronecker factors. In Advances in Neural Information Processing Systems, pp. 6594-6603, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hang Qi, Matthew Brown, and David G. Lowe. Low-Shot Learning with Imprinted Weights. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5822-5830, Salt Lake City, UT, June 2018, IEEE. ISBN 978- 978-1-5386-6420-9 doi: 10.
201.00000..
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers 2019.
generalize to imagenet? arXiv preprint
Jimmy SJ Ren and Li Xu. On vectorization of deep convolutional neural networks for vision tasks. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances Neural Information Processing Systems, pp. 2483-2493, 2018.
Patrice Y. Simard, Dave Steinkraus, and John Platt. Best practices for convolutional neural networks applied to visual document analysis. Institute of Electrical and Elec- tronics Engineers, Inc., August 2003. URL httpp::/www.microootft.....
en
us/ o=z/es/tt///llaaaoo/////rreeerrp
D. Soudry, D. Di Castro, A. Gal, A. Kolodny, and S. Kvatinsky. Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training. IEEE Transactions on Neural Networks and Learning Systems, 26(10):2408-24) 2421, October 2015. doi: 20.1009/TNNISS2214.22803
Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv: 1oXivv/702.050000 05043, 2017.
10

---

Under review as a conference paper at ICLR 2020
TSMC.
40nm Technology
Taiwan Semiconductor Manufacturing Company Limited.
2019.
user1551. Rotation matrix in arbitrary dimension to align vector. Mathematics
Stack Exchange, 2013. URL https: stackexchange ctt.c///m///////////
(version: 2013-10-14).
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-20)) 1989.
T.F. Wu, B. Q. Le, R. Radway, A. Bartolo, W. Hwang. S. Jeong, H. Li, P. Tandon, E. Vianello, Vivet, E. Nowak, M. K. Wootters, H. P. Wong, M. M. S. Aly, E. Beigne, and S. Mitra. 14.3 43pj/cycle non-volatile microcontroller with 4.7s shutdown/wake-up integrating 2.3-bit/cell resistive ram and resilience techniques. In 2019 JEEE International Solid- State Circuits Conference (ISSCC), pp. 226-228, Feb 2019. doi: 10. 10000//CC....1..666222
S. Yu, Z. Li, P. Chen, H. Wu, B. Gao, D. Wang. W. Wu, and H. Qian. Binary neural network with 16 Mb RRAM macro chip for classification and online training. In 2016 IEEE International Electron Devices Meeting (IEDM), pp. 16.2.1-16.2.4 December 2016. doi:
2.17783842.
Shimeng Yu. Neuro-inspired computing with emerging nonvolatile memorys. Proceedings of the IEEE, 106(2):200-285, 2018.
E. Zamanidoost, F. M. Bayat, D. Strukov, and Kataeva. Manhattan rule training for memristive crossbar circuit pattern classifiers. In 2015 IEEE 9th International Symposium Intelligent Signal Processing ( (WISP) Proceedings, pp. 1-6, May 2015. doi: 10. (190/W1SP2201..1139777
J. Zhang, Z. Wang, and N. Verma. In In-Memory Computation of Machine Machine-Learning Classifier in Standard 6T SRAM Array. IEEE Journal of Solid-State Circuits, 52(4):915 April 2017. ISSN 0018-9200.
doi: 10.1 109/ISSC..201.264422
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: (00::1666.000 16066061660 06060, 2016.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 928-936. 2003.

---

Under review as a conference paper at ICLR 2020
A SKS ALGORITHM
Algorithm Streaming Kronecker Sum Approximation
State: Qr. E R",x0, 9; QR E 1 C2 E R
Input: dz (6) E [R=aa1; (1) E [Rax11 for i E [1,
for
do
[Modified Gram- Gram-Schmidt,
CL. CR + 00x1 for 1...r do
Cr,j +- Qr.j dz
dzl() t- dz (1) cr.j Qu.
CR.j t Qu, a a al)) (a -CL.j" Q1,j
end for
Cr, ||dz(1) Qna dzl1//c1.n
CR.y + ||a QR.a t (1) /CR,
[Generate C and find its SVD.)
C - R + diag(c.)
Uc diag 0) Vg SVD(C)
(Minimum-vante variance unbiased estimator for E.}
m min j (g- j)o S Ei=
$1 + Σ os k q -- m
mm
v - -I k/s1 mm: - (1)
(-1, (1+ [Ind. uniform random signs.}
X, < (I (s+ (v/u))) [2:] (Householder.)
(QR-factorization of Et.}
I
Qu 0
X Rexr
+ (o1...., Om 1. 81/h 81/h)
g-m+1 times
(Update the first columns of Qi, QR.}
Q. + Q2 Uc Qr
Q Riar| Qn Vc Qu
end for
(Compute final É, f where VwL i
L + (2 diag( (loggv/e))) /cx))
R < (Qn diag( (os((v/)) /cz))
B CONVEX CONVERGENCE
In this section will attempt bound the regret (defined below) of an SGD algorithm using noisy SKS estimates ÿ = + e in the convex setting, where g are the true gradients and e are the errors introduced by the low rank SKS approximation. Here, g vector of size N and can be thought of as flattened/concattenated version of the gradient tensors (e.g., N = n2))
Our proof follows the proof in Zinkevich (2003). We define as the convex feasible set (valid settings for our weight tensors) and assume that F is bounded with D = max well/ w v|| being the maximum distance between two elements of F. Further, assume batch of samples out of T total batches corresponds to loss landscape (w(") that is strongly convex in weight parameters
w'. so there are positive constants 2 0 such that cf / ('u't) 3 CI for (Boyd
& Vandenberghe 2004 Section 9.3). We define regret as R(T) = E f'((") ET= f'(w") where wt argmin Σ/= (w)) (i.e., is optimal offline minimizer of P!...... ,
12

---

Under review as a conference paper at ICLR 2020
The gradients during SGD are g' (w"" and assume they are bounded by G maX [V We also assume errors are bounded by & = ma, (i)]e( Therefore, max S max + || I G + by the triangle inequality.
Theorem 1. Assume SKS-based SGD is applied with learning rate 1/vi. Then, under the additional constraint g' (w' w²) - f lw' w 11 - g (w' w we have sublinear regret:
D² R(T)= S VT (G (
Proof. From strong convexity of 42 ft (w') 3 CI all
(ww + g¹ (v - w) + c 1110 w1l2 S (b)
for all v
(8)
In partieular, if we consider w" and rearrange,
C (w) - (ww S g' (wr w²) (ww) - f'(w") s g' (w² w
11
wn
2
Consider gradient update w Pr(w" - n1'') ) where Pp projects the update back to F. Then,
1/w ++ w=1 PPw"" 7:9"" w0113 - || w²' n: w w011
lwo w w011 112 2n ( w") g" + n 1"11
S lwo w vo"1l 112 2 w²) g" + m (G + E)²
g' (w' - w²) S
(I/w² w 11 11 e+1 w" (3) + 11t (G + E)
(10)
2m.
From p 10
(ww) f'(2")
1
(1/2² ww" w=113) + 1h8 (G 8)²
(II)
21p.
2
We now bound the regret:
R(T) = Σ
[f (wo) - fww")]
t=1
<
(11w* -0011
1
w01l3 +
(G + +E82
2mp
1=1
w01
||
w=113
+
1
1
w w01 +
(G +
E)2 T
Σ
7
2771
277r
71
T--1
(w
w0
T
(G + E)²
T
+
1|
w w0113 +
Σ
2m
2
7h-1
2
tm2
D
T
(G (G+ T
<
+
D2 D²+
+
Σ m
2m
M--1
2
1=2
t=1
D
(G
+
2)
(12)
2mT
1=1
13

---

Under review as a conference paper at ICLR 2020
If nt. 1/vł, then Et=1 1 S 2VVT
Zinkevich 2003), so from 12
D VT + (G+ (G+ (VT 112)
R(T) 1 2
(13)
This is sublinear regret and therefore, average regret R(T)/T / is bounded above by 0 in the limit as
- oc. To achieve this result, we constrained g' (w' w²) - $1(00* w011 I g' (w' - w²).
We now examine sufficient conditions for this inequality to be satisfied.
g6 (w' w²) C =/llo -w"12 l/2 - g" (w' ww)
e¹ e' w²) S |1 -w'--"1F 11
(14)
Since Ex (w" w²) s ||=*|| 1/w" will by Seuucy, it is sufficient for:
| 1|w0 w0ll 5 w w" w011
||E0|| $ w0' w"1l
(15)
B. CONSIDERATIONS FOR RANK DEFICIENT HESSIANS
In the preceding proof, assumed 0. However, it is common for this not hold. For example, in linear regression, where Amin (XX for sample input X E ()xx) (Haykin 2014, Chapter 4.3), if < n2 then c = O. We can modify (15) to handle this case. Let č be the minimum non-zero Eigenvalue of XX and let i R represent R in the Eigenbasis of
X Then 15 becomes:
||
c S
(16)
B.2 ESTIMATES FOR THE SKS ERROR
We can estimate the SKS error ||e']] both the biased and unbiased cases. For the biased, zero- variance case, we get rid of the lowest singular value (out of q singular values) see each sample. Thus, a given sample i, the error is o Q()) (x) We treat this as a per-element variance. If these smallest singular components are uncorrelated from sample to sample, then the variances add:
L.. figg and the average squared error (1/N)o, (1)2
B
i1 9
(17)
For the unbiased, minimum-variance case, Theorem A.4 from Benzing et al. (019)) states that the minimum variance is st /s + 82 where $1 = Σi-m d,, 82 Σx_ of and k. m are as defined in Section 4.122 Since m is chosen to minimize variance, m = r and therefore = 1, good approximation. Then, the average per-element variance added at sample i approximately (2/N) (. 0g
choosing + ag. and 82 = o² + o² Empirically, this tends to be
upper bound the variance
(() Assuming errors between samples are uncorrelated, this leads to a total variance:
B
2
(18)
N
14

---

Under review as a conference paper at ICLR 2020
For either case, ||e||² No For the (-th batch and i-th sample, we denote og (2,8) the q-th singular value. For simplicity, we focus on the biased, zero-variance case (the unbiased case is similar). From [5]. an approximately sufficient condition for sublinear-regret convergence is:
B
6² 111000 10
S 4
(19)
B.3 DISCUSSION ON CONVERGENCE
Equation 19p suggests that as w + w., the constraints for achieving sublinear-regret convergence become more difficult to maintain. However, in practice this may be highly problem-dependent as the Bg will also tend to decrease near optimal solutions. To get a better sense of the behavior of the left-hand side of (I9), suppose that:
B
B (o(a))) ~ [ S Σ (o, |
B
where G w f (W") E R (m, are the matrix weight W gradients batch and -|- is Frobenius norm. We therefore expect both the left (proportional to |G' and the right (proportional to ||a0" w 22) (19) to decrease during training as w² -> =r" This behavior is in fact what seen in Figure bb). If accievinng convergence is found to be difficult, 19 provides some insight for convergence improvement methods.
One solution is to reduce batch size B to satisfy the inequality as necessary. This minimizes the weight updates during more repetitive parts of training while allowing dense weight updates (possibly approaching standard SGD with small batch sizes) during more challenging parts of training.
Another solution to reduce gg. One way to do this is to increase the rank r so that the spectral energy of the updates spread across more singular components. There may be alternate approaches based on conditioning the inputs to shape the distribution of singular values beneficial way.
A third method is to focus c, the lower bound on curvature of the convex loss functions. Perhaps a technique such as weight regularization can increase c by adding constant curvature in all Eigen- directions of the loss function Hessian (although this may also increase the LHS of (19). Alternatively, perhaps low-curvature Figen-directions are less important for loss minimization, allowing us to raise about. This latter approach requires
the that effectively except the recognition that fast convergence may only be guaranteed for high-curvature directions. This is exemplified in Figure [bb, where we can see SVD track the curve for C more than c.
particular action our part,
Finally, we note that this analysis focuses solely on the errors introduced by a floating-point version of SKS. Quantization noise can add additional error into the g term. We expect this add constant offset to the LHS of [9). For weight LSB A, quantization noise has variance A /12, so we desire:
2
2
N
(t.))
I|w0 - w1
(20)
12
C KRONECKER SUMS IN NEURAL NETWORK LAYERS
C. DENSE LAYER
dense fully-connected layer transforms input E RR an intermediate W to an output y = azx) E g where a is a non-linear activation function. Gradients of the loss function with respect the weight parameters can be found as:
VwC= (V ④ V w
dz a
(21)
a
15

---

Under review as a conference paper at ICLR 2020
which is exactly the par-sample Kronecker update saw in linear regression. Thus, every training sample, we add (dz( (+))) to our low rank estimate with SKS.
C.2 CONVOLUTIONAL. LAYER
convolutional layer transforms an input feature map E IR Kuinn Xetn to an intermediate feature map Z Wkern A + b E hout X through 2D convolution * with weight kernel W Werm E Root X Ri, Xkw Kin. Then it computes an output feature map y = (22) where a a non-linear activation function.
Convolutions can be interpreted as matrix multiplications through the im2col operation which converts the input feature map A into a matrix Acol E Who kwtin) where the jt row is a flattened version of the sub-tensor of which is dotted with Wkern to produce the eth pixel of the output feature map (Ren & Xu 2015). We can multiply Acol by a flattened version of the kemnel, W R x (kg Con) perform the W kern A convolution operation with a matrix multiplication. Under the matrix multiplication interpretation, weight gradients can be represented as:
Bout Wott
Vw (Vzcol Zcol C)
(w
dZ A
(22)
dz
Acol
which is the same hout Wout Kronecker sum updates. Thus, at every output pixel j of every training sample i, we can add (dZ (+)
@ A Ant.j) to our low rank estimate with SKS.
eot.j
Note that while we already save an impressive factor of B/q in memory when computing gradients for the dense layer, save much larger factor of Blout Wout/q /q in memory when computing gradients for the convolution layers, making the low rank training technique even more crucial here.
However, some must be taken when considering activation memory for convolutions. For compute-constrained edge devices, image dimensions may be small and result minimal intermediate feature map memory requirements. However, if image dimensions grow substantially, activation memory could dominate compared to weight storage. Clever dataflow strategies may provide a way reduce intermediate activation storage even when performing backpropagation
D HARDWARE QUANTIZATION MODEL
In real device, operations are expected to be performed in fixed point arithmetic. Therefore, all of our training experiments are conducted with quantization in the loop. Our model for quantization is shown in Figure The green arrows describe the forward computation. Ignoring quantization for a moment, we would have a' ReLU (a'W* W * a + +b), where * can represent either a convolution or matrix multiply depending on the layer type and a is the closest power-of-2 to He initialization (He al. 2015). For quantization, we rely on four basic quantizers: Que Qb, Qu, Qy, which describe weight quantization, bias and intermediate accumulator quantization, activation quantization, and gradient quantization, respectively. All quantizers use fixed clipping ranges as depicted and quantize uniformly within those ranges to the specified bitwidths.
In the backward pass, follow the orange arrows from 8 Backpropagation follows standard back- propagation rules including using the ttraight-throug estimator (Bengio et al. 2013) for quantizer gradients. However, because we want to perform training on edge devices, these gradients must themselves be quantized. The first place this happens is after passing backward through the ReLU derivitive. The other two places are before feeding back into the network parameters W b¹, so that W bf cannot be used to accumulate values smaller than their LSB. Finally, instead of deriving AW from a backward pass through the * operator, the SKS method is used.
For example, one could compute just a sliding window of rows of every feature map, discarding earlier rows as later rows are computed, resulting in a square-roo root reduction of activation memory. To incorporate hackpropagation, compute the forward pass once fully, then compute the forward pass again, well as the backward pass using the sliding window approach in both directions.
16

---

Under review as a conference paper at ICLR 2020
6--1
8
f-
ReLU
a
W
@b
Qwas
SKS
-8
Qbie0
8
Figure 3: Signal flow graph for forward and backward quantized convolutional or dense layer.
SKS collects a--1, dz' for many samples before computing the approximate A Wt. It accumulates information in two low rank matrices L, R which are themselves quantized to 16 bits with clipping ranges determined dynamically by the max absolute value of elements in each matrix. While SKS accumulates for B samples, leading to a factor of B reduction the rate of updates to W of is updated every sample. This is feasible hardware because b is small enough be stored in more expensive forms of memory that have superior endurance and write power performance.
prevent- Because of the coarse weight LSB size, weight gradients may be consistently quantized ing them from accumulating. To combat this, we only apply an update if a minimum update density Pmin 0.01 would be achieved, otherwise we continue accumulating samples in L and R, which have much higher bitwidths. When an update does finally happen, the "effective batch size" will be multiple of B and we increase the learning rate correspondingly. In the literature, a linear scaling rule is suggested (see Goyal et al. (2017)), however empirically find square-root scaling works better (see Appendix
E GRADIENT MAX-NORMIN
fcl.weight
20
10
-10
-20
100
200
300
400
500
600
Figure 4:
Maximum magnitude of weight gradients training step for standard SGD on CNN
trained on MNIST.
Figure HigureAplos. plots the magnitude of gradients seen in weight tensor over training steps. One apparent property of these gradients is that they have a large dynamic range, making them difficult to quantize. Even when looking at just the spikes, they assume wide range of magnitudes. One potential method of dealing with this dynamic range is to scale tensors so that their max absolute element is (similar per-tensor AdaMax (Kingma & Ba 2014) Range Batch-Norm (Banner et al.) 2018) applied to gradients). Optimizers such as Adam, which normalize by gradient variance, provide justification for why this sort of scaling might work well, although they work at per-element
17

---

Under review as a conference paper at ICLR 2020
rather than per-tensor level. We choose max-norming rather than variance- based norming because the former is casier computational and potentially more ammenable to quantization. However, a problem with the approach of normalizing tensors independently at each sample is that noise might be magnified during regions of quiet as seen in the Figure. What we therefore propose is normalization by the maximum of both the current
element and moving average of the max element.
Explicitly, max-norm takes two parameters a decay factor / = 0.999 and gradient floor & = 10-4 and keeps state variables the number of evaluations k 0 and the current maximum moving average Tmv = e. Then for a given input 2, max-norm modifies its internal state and returns Trorm
= +1
Tmar max [a:) +
Tme = B Imn + (1 - (3) Tmmax
Fma
Tmm
3
Unorm
max (Immax Tomo)
F STREAMING BATCH NORMALIZATION
Standard batch normalization (Toffe& & Szegedy 2015) normalizes tensor X along some axes, then
applies a trainable affine transformation. For each slice X of X that is normalized independently:
X
Y Y=7-
+
+
where (b: o6 are mean and standard deviation statistics of a minibatch and 7, B are trainable affine transformation parameters.
In our case, we do not have the memory to hold a batch of samples at a time and must compute 14a db in online fashion. To see how this works, suppose we knew the statistics of each sample 144 a for = 1... B a batch of samples. For simplicity, assume the 2th sample is vector X,,; E IR" containing elements Xi.. Then:
B
1
Pb =
14
(23)
B
1=1
B B
1 B
1 B 1=1
0f
X2 1d
14
2
(o + 12)
1
+
2
(24)
t=1
In other words, the batch variance is not equal to the average of the sample variances. However, if we keep track of the sum-of-square values samples of + μ4; then we can compute oî in 24 We keep track of two state variables: /4,, 8qs which we update /4, μ4 + /4; and sq,- := sq + o + /
for each sample After B samples, we divide both state variables by and apply 23 24 to get the
desired batch statistics. Unfortunately, in online setting, all samples prior to the last one in a given batch will only see statistics generated from a portion of the batch, resulting in noisier estimates of pbb. db.
In streaming batch norm, we alter the above formula slightly. Notice that in online training, only the most recently viewed sample is used for training, so there is no reason weight different samples of a given batch equally. Therefore we can use an exponential moving average instead true average to track H,, sqs- Specifically, let:
:=:= 17 Me + (1 77) u4
sqs = 1) sq. (1 1)) (o? p
18

---

Under review as a conference paper at ICLR 2020
If we 7) = 1 1/BB, weighting of 1/B is seen on the current sample, just as in standard averages with a batch of size B, but now all samples receive similarly clean batch statistic estimates, not just the last few samples in batch.
G ONLINE DATASET
For our experiments, we construct dataset comprising an offline training. validation, and test set, as well as an online training set. Specifically, we start with the standard MNIST dataset of LeCun et al.) (1998) and split the 60k training images into partitions of size 9k, Ik, and 50k. Elastic transforms (Simard et al. 2003; Ernestus 2016) are used to augment each of these partitions to 50k offline training samples, 10k offline validation samples, and 100k online training samples, respectively. Elastic transforms are also applied to the 10k MNIST test images to generate the offline test samples.
The source images for the 100k online training samples are randomly drawn with replacement, so there is a certain amount of data leakage in that an online algorithm may be graded on an image that has been generated from the same image previous sample it has trained on has been generated from. This is intentional and is meant to mimic a real-life scenario where a deployed device is likely to see restrictive and repetitive set of training samples. Our experiments include comparisons standard SGD to show that SKS's improvement is not merely due to overfitting the source images.
From the online training set, we also generate a "distribution shift" dataset by applying unique additional augmentations to every contiguous 10k samples of the 100k online training samples. Four types of augmentations are explored. Class distribution clustering biases training samples belonging to similar classes to have similar indices. For example, the first thousand images may be primarily "O's and "3"s, whereas the next thousand might have many "5"s. Spatial transforms rotate, scale, and shift images by random amounts. Background gradients both scale the contrast of the images and apply black-white gradients across the image. Finally, white noise is random Gaussian noise added to each pixel. Figure Figure]Jshows showws some representative examples of what these augmentations look like. The augmentations are meant to mimic different external environments an edge devices might need to adapt to.
0
7
3
A
Y
G
6
6
6
6
L
S
S
S
S
4
9
9
9
4
9
Y
(a) Spatial Transforms
(b) Background Grads
(c) White Noisc
Figure 5: Samples of different types of distribution shift augmentations.
In addition to distribution shift for testing adaptation, we also look at internal statistical shift of weights in two ways analog and digital. For analog weight drift, we apply independent additive Gaussian noise to each weight every d = 10 steps with o 0o/ / /Md/d where da 10 and re-clip the weights between -1 and 1. This can be interpreted as each cell having a Gaussian cumulative error with o = do after 1M steps. For digital weight drift, we apply independent binary random flips to the weight matrix bits every d steps with probability interpreted as each cell flipping an average of Po times over 1M steps. Note that in real life, do. Po depend on a host of issues such as the environmental conditions of the device (temperature, humidity, etc), as well as the rate of seeing training samples.
Po/(1M/d where Pu = 10. This can be
19

---

Under review as a conference paper at ICLR 2020
H HYPERPARAMETER SELECTION
In order to compare standard SGD with the SKS approach, we sweep the learning rates of both to optimize accuracy. In Figure 6 compare accuracies across range of learning rates for four different cases: SGD or SKS with or without max-norming gradients. Optimal accuracies are found when learning rate is around 0.01 for all cases. For most experiments. 8b weights, activations, and gradients, and 16b biases are used. Experiments similar to those in Section I of the hyperparameters related the SKS method in particular. In most experiments, rank-4 SKS with batch sizes of 10 (for convolution layers) or 100 (for fully-connected layers) are used. Additional details can be found in the supplemental code.
are used to select some
SGD/No-Norm
SGD/Max-Norm
SKS/No-Nore Norm
SKS/Max-Nors Norm
1.0
1.0
10
1.2
5%
1,2
0h
F4
Log10 Learning Rate
0.4 99
0.4
Accuracy Baat 500 of 10k]
0.2 41
02
95
0.8
5%
0.6
24
0.8
0.8
-08
0.0
0.0
1.4
144
20
1.2
-3.2-24
29
-18
3.8 0%
38 50
784
79
6%
2.4
07%
17%
0.2
-4.4
-4
-5.8
-50
V9A
0170
16%
6.7
64%
3.0
11/
100
114
1.0
1.5
2.0
2.5
3.0
1.0
1.5
2.0
3.0
0.0
Log10 FC Batch Size
Log10 FC Batch Size
Figure 6: The left two heat maps used select the base / standard SGD learning rate. The right two heat maps are used to select the SKS learning rate using the optimal SGD learning rate for bias training from the previous sweeps. For the SKS sweeps, the learning rate is scaled proportional to the square-root of the batch size B. This results in an approximately constant optimal learning rate across batch size, especially for the max-norm case. Accuracy is reported averaged over the last 500 samples from 10k portion of the online training set, trained from scratch.
I ADDITIONAL STUDIES
In Figure 7 rank and weight bitwidth is swept for SKS with gradient max-norming. As expected, training accuracy improves with both higher SKS rank and bitwidth. In dense NVM applications, higher bitwidths may be achievable, allowing for corresponding reductions in the SKS rank and therefore, reductions in the auxiliary memory requirements.
In Table biased (zero-variance) and unbiased (low-variance) versions of SKS are compared. Accuracy improvements are generally seen moving from biased to unbiased SKS although the pattern differs between the no-norm and max-norm cases. In the no-norm case, a significant improvement is seen favoring unbiased SKS for fully-connected layers. In the max-norm case, the choice of biased or unbiased SKS has only minor impact on accuracy. It might be expected that as the number of accumulated samples for given pseduobatch increases. lower variance would be increasingly important at the expense of bias. For our network, this implies convolutions, which receive updates at every pixel of an output feature map, would preferentially have biased SKS, while the fully-connected layer would preferentially be unbiased. This hypothesis is supported by the no-norm experiments, experiments.
but not by the
In Table 3 several ablations are performed on SKS with max-norm. Most notably, weight training is found to be extremely important for accuracy as bias-only training shows R 15 30% accuracy hit depending on whether max-norming is used. Streaming batch norm also found to quite helpful, especially in the no-norm case.
Now, we explain the Ke ablation. In Section 4.1.I we found the SVD of a small matrix C and its singular values ...... Tg. This allows to easily find the condition number of C' as x(CC)= 01/0g- We suspect high condition numbers provide relatively useless update information akin to noise, especially in the presence of L, R quantization. Therefore, we prefer not update L, R on samples whose condition number exceeds threshold Kt We can avoid performing an actual SVD (saving computation) by noting that C is often nearly diagonal, leading the approximation
20

---

Under review as a conference paper at ICLR 2020
1.0
8
59%
67%
69% 72% 75% 74%
76% 80% 76%
7
7.56% 65%
68%
72% 74% 79% 75% 79% 79% 76%
Accuracy (Last 500 of 2k)
Weight Bitwidth
6
55%
64%
64%
69% 74% 74% 76% 74% 77% 77%
0.6
5
51%
58%
61% 62% 68% 69% 75% 76% 77% 78%
4
45% 50%
55% 59% 54% 63% 63% 64% 70% 70%
0.4
3
40%
42%
40%
44% 43% 45% 49% 52% 52% 49%
2
12% 9%
31%
26%
10% 12%
9% 9%
9%
10%
0.2
1 12% 12%
13%
13%
12% 12% 14% 13% 14%
12%
0.0
10
SKS/Max-Norm Rank
Figure 7: Accuracy across variety of SKS ranks and weight bitwidths, showing the expected trends of increasing accuracy with rank and bitwidth. Accuracy is calculated by averaging the accuracy on the last 500 samples from a 2k portion of the training data. For bitwidths of and 2, mid-rise quantization is used (e.g., 1 bit quantizes values to -0.5 and 0.5 instead of -1 and 0).
Table 2: Importance of unbiased SVD. Accuracy is calculated from the last 500 samples of 10k samples trained from scratch. Mean and unbiased standard deviation are calculated from five runs of different random seeds.
Cony SKS
FC SKS
Accuracy (no-norm)
Accuracy (max-norm)
Biased
Biased
79.7% 1.1%
82.7% 1.3%
Biased
Unbiased
83.0% + 0.9%
82.4% + 1.2%
Unbiased
Biased
77.7% + 1.5%
84.6% 2.0%
Unbiased
Unbiased
81.0% 0.9%
83.6% 2.5%
Empirically, this rough heuristic works well to reduce computation load while having minor impact on accuracy. In Table Kith 10 does not appear to ubiquitously improve on the default Kez. 100, despite being x 2x slower to compute.
(CC) = C1,1/Cq.q-
Table 3: Miscellaneous selected ablations. Accuracy is calculated from the last 500 samples of 10k samples trained from scratch. Mean and unbiased standard deviation are calculated from five runs of different random seeds.
Modified Condition
Accuracy (no-norm)
Accuracy (max-norm)
baseline (no modifications)
80.2% + 1.0%
13.0%11.1%
bias-only training
51.8% + 3.2%
68.6% + 1.4%
no streaming batch norm
68.2% + 1.9%
81.8% + 1.3%
no bias training
81.3% + 1.0%
83.0% + 1.4%
Kith 10 instead 100
79.8% + 1.4%
84.2% + 1.4%
21


Reviews:
[{"review_id": "r1liBXXTKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a new online training scheme which is amenable to non volatile memories and particularly applicable to smart edge devices. This deals with 4 challenges with smart edge learning paradigms - low weight update density, weight quantization, low auxiliary memory, online learning and show experiments to understand benefits. \n\nDisclaimer: I am far from an expert on this domain, so my review is not very well calibrated or informative. \n\nHow exactly are you implementing optLR in Section 4.2?\n\nMaybe the section 4.3 can instead go to prior work/related work rather than be described in this paper if it’s not being used very much. It seems important for the mixing of the lower SVD elements in the OK method but perhaps can be removed. \n\nI think for people less familiar with the area, a more intuitive explanation of section 4.4 would be helpful. \n\nGenerally how much does the algorithm suffer from using an SVD type approximation for learning?\n\nThe new contributions of this paper are to use a running estimate of Q_L, Q_R and weightings using modified Gram Schmidt. \n\nNot immediately clear how the scheme described related to the quantization point in the challenges. \n\nExperiments seem to show the proposed benefits but are done with artificial models/simulations. Would it be easy to implement this on chip and try it on actual hardware? Can we also compare to things like OK/UORO in the experiments?\n\nOverall seems like a novel and interesting way to do SGD on the edge, I think that a little bit simpler explanation would make the paper more readable and some additional comparisons would also benefit the paper. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Hyxzg_vAFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a low rank training method called the Streaming Kronecker Sum approximation (SKS algorithm) for training low precision models on edge devices. The authors compare their method to SGD for convolutional networks on MNIST and demonstrate improvements in terms of accuracy. The authors make use of the Optimal Kronecker-sum algorithm of Benzing et al and propose further improvements to it in the form of the SKS algorithm.\n\nThe main weakness of the paper seems to me limited experimental results - they mainly show improvements for CNNs on MNIST. The gains from their method would be made more  convincing by doing more large scale experiments on other larger datasets such as CIFAR-10/100, Imagenet and text datasets. \n\nOverall, I recommend acceptance based on the thoroughness of the work and the authors open-sourcing their code which would additionally help with reproducibility of their results.\n\n[Edit: Upon reading discussion among other reviewers, I have updated my score to reject.]", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Hkeyp1RFqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a low-rank training method targeting for edge devices. The main contribution is an algorithm called Streaming Kronecker-Sum Approximation. The authors claim that the proposed method addresses four key challenges of low weight update density, weight quantization, low auxiliary memory, and online learning.\n\nThe paper should be rejected because of the following reasons:\n(1) The paper is a little hard to follow and the writing can be significantly improved. In particular, the authors introduce four main challenges in section 3. However, I found they are not that accessible and hard to understand. In section 4.4.2, the objective is to get a minimum variance rank-r approximation to the diagonal matrix \\Sigma, but I think the authors mix \"m\" up with \"r\".\n(2) The novelty of the algorithm is limited. From section 4.1 to 4.5, most discussions are about previously proposed methods. The algorithm proposed by the author (i.e., SKS) only involves some basic manipulations of linear algebra. I don't think it's novel enough to be a new algorithm.\n(3) Experimental results are limited. The authors spent a lot of time discussing on-device computing, but all their experiments are just simulations on standard benchmarks. For such a paper concerning training on edge devices, I would expect to see some experiments on real edge devices.\n\nOverall, I think the paper needs further improvements to be qualified for being accepted.\n\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------\npost rebuttal:\n\nI've read the authors' responses and the updated paper. Though my concern on writing has been resolved to some extent, I'm still unsatisfied with the empirical experiments. I believe the authors need to do experiments on edge devices since they have emphasized a lot about on-device computing. That being said, I'm not an expert in hardware and have no idea how hard it is to conduct those experiments. I've increased the score to 3 but still vote for rejection.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlMoRsWqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While inference on edge devices is a popular and well-studied problem in recent days, training on these devices comes with many challenges. This paper proposes a low-rank training schema that helps mitigate some of the critical challenges that occur during training models on NVM memory-based edge devices. Additionally, two techniques, namely streaming batch norm and gradient max norm, are proposed to help training in an online setting. The proposed method is mainly based on approximating the Kronecker sum and is largely inspired by (Benzing et al, ICML 2019, Optimal Kronecker-sum approximation of real time recurrent learning). The proposed approach provides a few optimizations that improves this performance further, and outperforms SGD in terms of accuracy and the number of weights updates in a limited experimental setting.\n\n+ves:\n+ Training on an edge device is a relevant setting, where there is very little work so far, and this is a useful objective.\n+ Focusing on the Kronecker sum and speeding it seems like an interesting solution to solving this problem.\n+ The experimental settings considered (e.g. flipping bits and adding Gaussian noise to weights) is interesting, and perhaps of larger relevance to other work in the area.\n+ The ablation studies provided in the appendix are useful and appreciable.\n\nConcerns:\n- The key concern is that most of the proposed method is built upon Benzing et al’s work (ICML 2019), and the original contribution seems limited. While the paper introduces a few optimizations further, this seems to be incremental than originally novel.\n\n- The problem is formulated for linear regression with least-squares loss, and the experimentation is carried out on the MNIST dataset (classification setting). How is the methodology relevant to the cross-entropy loss, typically used for classification? The paper does not talk about this.\n\n- The results are shown for a network with four 3 x 3 convolution layers and two fully connected layers on the MNIST dataset. Results with different architectures on a few other datasets (at least CIFAR-10) would be necessary to assess the usefulness of this method. Some discussion on what would be the maximum depth of the network that can be trained using this training schema and hardware would be very useful. \n\n- Do all deep neural network architectures (and loss functions) admit a Kronecker sum representation? What class of models can benefit from this method? Factorizations such as Cholesky allow to interpolate between computational complexity and decomposition rank by tuning a rank hyperparameter. Why would such factorizations not be better for memory-constrained settings?\n\n- It would be interesting to see the results using the proposed algorithm on standard hardware, will it provide the same performance as SGD when the scale of the problem increases in terms of layers and other regularization techniques, etc.? This would help understand the performance of the algorithm in a standard-setting.\n\nMinor issues:\n- The paper could have been organized better. Most of the main paper is used to describe Benzing et al’s method, and a lot of the details of the original contributions are in the Appendix. While the equations discussed in Section 4 hold for Linear Regression, the same cannot be directly extended for other networks like CNNs. Considering all the experiments are on CNNs, Sec B.2 in the Appendix should have been in the main paper to help a reader follow the experiments. \n\n=====POST-REBUTTAL COMMENTS========\nI thank the authors for the rebuttal. The paper's organization is much better now. Unfortunately, there are many concerns which are still not convincingly answered: (i) Considering the focus of this work (as admitted by the authors in the rebuttal) is empirical and with limited technical novelty, more comprehensive results on many datasets are required. To some extent, transfer to ImageNet addresses this partially, but not convincingly enough; (ii) It is not clear from the authors' rebuttal why testing this on standard hardware is not possible/appropriate. The rebuttal says \"hard to quantify due to a variety of factors\" but that is vague without stating the factors.\n\nI am willing to increase the rating from 3 to 4, but am unfortunately still towards rejecting the current shape of this work. I believe the work will benefit from considering all reviewers' comments more comprehensively to have a more impactful paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the novelty of the work was very limited. This is not necessarily a deal-breaker for a largely applied contribution, but for an applied paper, the evaluation of the actual application on edge devices is not present. So if the main contribution is the application, and there is no evaluation of this application, then it does not seem like the paper is really complete. As such, I cannot recommend it for acceptance.
Decision: Reject