Accelerating DNN Training through Selective Localized Learning 
Authors: Sarada Krithivasan
Sanchari Sen
Swagath Venkataramani
Anand Raghunathan
Venue:  (2021)
Keywords: Efficient DNN Training

Abstract: Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. We proposeLoCal+SGD, a new algorithmic approach to accelerate DNN train-ing by selectively combining localized or Hebbian learning within a StochasticGradient Descent (SGD) based training framework. Back-propagation is a computationally expensive process that requires 2 Generalized Matrix Multiply (GEMM)operations to compute the error and weight gradients for each layer. We alleviate this by selectively updating some layers’ weights using localized learning rules that require only 1 GEMM operation per layer. Further, since the weight update is performed during the forward pass itself, the layer activations for the mini-batch do not need to be stored until the backward pass, resulting in a reduced memory footprint. Localized updates can substantially boost training speed, but need to be used selectively and judiciously in order to preserve accuracy and convergence. We address this challenge through the design of a Learning Mode Selection Algorithm, where all layers start with SGD, and as epochs progress, layers gradually transition to localized learning. Specifically, for each epoch, the algorithm identifies a Localized→SGDtransition layer, which delineates the network into two regions. Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-based updates. The trend in the weight updates made to the transition layer across epochs is used to determine how the boundary betweenSGD and localized updates is shifted in future epochs. We also propose a low-cost weak supervision mechanism by controlling the learning rate of localized updates based on the overall training loss. We applied LoCal+SGDto 8 image recognition CNNs (including ResNet50 and MobileNetV2) across 3 datasets (Cifar10, Cifar100and ImageNet). Our measurements on a Nvidia GTX 1080Ti GPU demonstrate upto 1.5×improvement in end-to-end training time with∼0.5% loss in Top-1classification accuracy.

Under review as a conference paper at ICLR 2021
ACCELERATING DNN TRAINING THROUGH SELEC- TIVE LOCALIZED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. We propose LoCal+SGD, new algorithmic approach to accelerate DNN train- ing by selectively combining localized or Hebbian learning within a Stochastic Gradient Descent (SGD) based training framework. Back- kack-propagation is compu- tationally expensive process that requires 2 Generalized Matrix Multiply (GEMM) operations compute the error and weight gradients for each layer. We alleviate this by selectively updating some layers' weights using localized learning rules that require only GEMM operation per layer. Further, since the weight update performed during the forward pass itself, the layer activations for the batch do not need be stored until the backward pass, resulting reduced memory footprint. Localized updates can substantially boost training speed, but need to be used selectively and judiciously in order to preserve accuracy and convergence. We address this challenge through the design of Learning Mode Selection Algorithm, where all layers start with SGD, and as epochs progress, layers gradually transi- tion to localized learning. Specifically, for each epoch, the algorithm identifies a Localized-SSGi transition layer, which delineates the network into two regions. Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-basee updates. The trend in the weight updates made the transition layer across epochs used to determine how the boundary between SGD and localized updates is shifted in future epochs. We also propose low-cost weak supervision mechanism by controlling the learning rate localized updates based on the overall training loss. We applied LoCal+SGD to 8 image recognition CNNs (including ResNet50 and MobileNetV2) across datasets (Cifarl0, Cifar 00 and ImageNet). Our measurements on a Nvidia GTX 1080Ti GPU demonstrate upto 1.5 x improvement in end-to-end training time with ~0.5% loss in Top-1 classification accuracy.
INTRODUCTION
Deep Neural Networks (DNNs) have achieved continued success in many application domains involving images (Krizhevsky et al.) 2017). videos (Ng al. 2015), text (Zhou et al.) (015) and natural language (Goldberg & Hirst 2017). However training state-of-the-art DNN models is computationally quite challenging, often requiring FXa-FLOPs of compute as the models are quite complex and need to trained using large datasets. Despite rapid improvements in the capabilities of GPUs and the advent of specialized accelerators, training large models using current platforms still quite expensive and often takes days to even weeks. In this work, we aim to reduce the computational complexity of DNN training through new algorithmic approach called LoCal+SGQ]) which alleviates the key performance bottlenecks in Stochastic Gradient Descent (SGD) through selective use of localized Hebbian learning.
Computational Bottlenecks in DNN Training. DNNs are trained in a supervised manner using gradient-descee based cost minimization techniques such as SGD (Bottou) 2010)) or Adam (Kingma Ba 2015). The training inputs (typically grouped into minibatches) are iteratively forward propagated P) and back propagated (BP) through the DNN layers to compute weight updates that push the network parameters in the direction that decreases the overall classification loss.
In addition to combining localized and SGD based learning, LoCal+SGD is Low-Calorie SGD SGD with reduced computational requirements

---

Under review as a conference paper at ICLR 2021
Back-propagation is computationally expensive, accounting for 65-75% of the total training time on GPUs. This is attributed to two key factors: (i) BP involves 2 Generalized Matrix Multiply (GEMM) operations, to propagate the error across layers and the other compute the weight gradients, and (ii) when training on distributed systems using data/model parallelism (Den et al. 201 26 Krizhevsky et al. 2012), aggregation of weight gradients/errors across devices incurs significant communication overhead. Further, BP through auxiliary ops such as batch normalization are also more expensive than FP.
Prior Efforts on Efficient DNN Training. Prior research efforts to improve DNN training time can be grouped into a few directions. One group of efforts enable larger scales of parallelism in DNN training through learning rate tuning (You et al.
2017a; et al. 2017] You et al. (007b)
and asynchronous weight updates (Dean et al. 2012a). Another class of eftorts employ importance- based sample selection during training. wherein 'easier' training samples are selectively discarded to
improve runtime (Jiang et al. 2019) 9, Zhang et al. 2019). Finally, model quantization (Sun et al. 2019)
and pruning (Lym et al., 2019) can lead to significant runtime benefits during training by enabling the
use of reduced-bitwidih processing elements.
LoCal+SGD: Combining SGD with Localized Learning. Complementary to the aforementioned efforts, we propose a new approach, LoGa,+ to alleviate the performance bottlenecks in DNN training, while preserving model accuracy. Our hybrid approach combines Hebbian localized learning (Hebb) with SGD by selectively applying it in specific layers and epochs. Localized learning rules (Hebb) Oja 1982 Zhong 2005) utilize single ferd-fo weight update to learn the feature representations, eschewing BP. Careful formulation of the localized learning rule can result in ~2x computation savings compared SGD and also significantly reduces memory footprint as activations from FP needed not be retained until BP. The reduction in memory footprint can in turn allow increasing the batch size during training, which leads to further runtime savings due to better compute utilization and reduced communication costs. It is worth noting that localized learning has been actively explored in the context of unsupervised learning (Chen et al.) 2020; van den Oord et al., 2018 Hénaff et al. (019). Further, there has been active research efforts on neuro-scientific learning rules (Lee et al. 2015; Nøkland 2016). Our work is orthogonal to such efforts and represents a new application of localized learning in a fully supervised context, wherein we selectively combine it within an SGD framework to achieve computational savings.
Preserving model accuracy and convergence with LoCal+SGD requires localized updates to be applied judiciously i.e., only to selected layers in certain epochs. We address this challenge through the design of learning mode selection algorithm. At the start training, the selection algorithm initializes the learning mode of all layers to SGD, and as training progresses determines the layers that transition to localized learning. Specifically, for each epoch, the algorithm identifies Localized->SS SGD transition layer, which delineates the network into two regions. Layers before the transition layer use localized updates, while subsequent layers use gradient- based updates. This allows BP stop at the transition layer, as layers before it have no use for the back proppparee errors. The algorithm takes advantage of the magnitude of the weight updates of the Serealized+SSC. transition layer deciding the new position of the boundary every epoch. Further, we provide weak supervision by tweaking the learning rate of locally updated layers based on overall training loss.
Contributions: To the best of our knowledge, LoCal+SGD is the first effort that combines local- ized learning (an unsupervised learning technique) within a supervised SGD context to reduced computational costs while maintaining classification accuracy. This favorable tradeoff is achieved by LoCal+SGD through a Learning Mode Selection Algorithm that applies localized learning to selected layers and epochs. Further improvement is achieved through the use of weak supervision by modulating the learning rate of locally updated layers based on the overall training loss. Across 8 image recognition CNNs (including ResNet50 and MobileNet) and datasets (Oi Cifar 100 and ImageNet), we demonstrate that LoC achieves up to 1.5x improvement in training time with ~0.5% Top-1 accuracy loss on a Nvidia GTX 1080Ti GPU.
2 LoCal+ COMBINING SGD WITH SELECTIVE LOCALIZED LEARNING
The key idea in LoCaI+SGD is to apply localized learning selected layers and epochs during DNN training to improve the overall execution time, without incurring loss in accuracy. The following components are critical the effectiveness of LoCal+SGD:
2

---

Under review as a conference paper at ICLR 2021
Stage
Localized Updates
5GD-Based Updates
Mi Weight Layer L ZL: Pre Activation o/p of Layer L
2,
GEMM (O)-1. WD
(2)
Aui Post-Activation o/p of Layer L
Update Stage
AW; (a.M( 21)
(3)
GEMM(A-11 WD
by Error Layer
AW;
AW1 = GEMM(u..1 6)
(6)
D: Learning Rate
Wi=
W,+nAW
NOTE: GEMM includes operations in convolutional and fully connected layers
Figure 1: Comparing Localized Updates and SGD- BP
Localized Learning Rule Formulation. We formulate a computationally efficient localized learning rule and highlight the clear runtime benefits when compared to SGD.
Learning Mode Selection Algorithm. We propose learning mode selection algorithm that chooses between localized learning and SGD-based learning for each layer in every epoch, based on the potential impact accuracy and computational benefits.
Weak Supervision. We propose a weak supervision technique, which comprises of low- cost supervision signal communicated to the localized learning layers in each epoch. The signal modulates the learning rates of these layers based on the rate of change of the overall classification loss.
In the following sub-sections, we describe the salient aspects of these components in greater detail.
2.1 EFFICIENT LOCALIZED LEARNING
Localized learning has been extensively explored in the context of unsupervised learning. demon- strating success on small (<= 3 layer) networks using relatively simpler datasets (e.g. MNIST, al. a)) with an accuracy gap that yet to bridged on larger datasets (e.g. ResNer50 or MobileNetV2 on ImageNet (Deng et al., 2009h). First proposed in (Hebb), the key intuition behind localized learning rules is to encourage correlations between neurons that have similar activation patterns. Equation T depicts the Hebbian weight update proposed in (Hebb), for a synapse with weight W, connecting pair of input and output neurons whose activation values represented by and y respectively, with the learning rate.
Cifar-10) (TeCun Cortes 2010 Krizhevsky et al.
x0
(1)
Considerable research has gone into evolving this equation over the years improve the perfor- mance of localized learning (Oja. 1982, Zhong 2005). However, many of the proposed rules are computationally complex, are difficult parallelize modern hardware platforms such GPUs and TPUs. Since our primarily goal is improving DNN training time, we adopt the computationally simple localized learning rule presented in Equation
The learning rule in Equation T Jassumes a distinct synapse between each input and output neuron pair. While its application to fully-connected (fc) layers straightforward, we need to consider the sharing of weights between neuron pairs in convolutional (conv) layers. For updating shared weight of a conv layer, we calculate the individual updates due to each pair of pre- and post-synaptic neurons sharing the weight and sum all such updates. This essentially reduces to a convolution operation between the input and output activations of the layer and can be expressed by Equation in Figure [ For further computational efficiency improvement, unlike Equation [ we consider the pre-activation-funcee values of the outputs i.e., z instead of their post activation value a, Further, normalize the localized update values as shown in Equation 4 of Figure I as it was observed to achieve better convergence in practice.
Overall, we utilize Equations and 4 from Figure I perform the weight updates in all layers that are earlier than the Localized->SCC transition layer during a certain epoch. All other layers continue to be updated using SGD-based BP, expressed by Equations 5-7 in Figure [ SGD updates are applied to noaaaaoormaiization layers present after the Localized-SGGI transition layer, and are otherwise skipped. Clearly, Equation 3 has the same computational complexity Equation 6 of baD-based BP for conv and fe layers. Thus, from Figure I we can directly infer that our localized learning rule will be considerable faster than SGD-be based B.P. In practice, measured this improvement to be more than 2x on a NVIDIA GTX 1080Ti GPU for the ImageNet-Resse ResNet50 benchmark, across all conv and fe layers. In addition to the computational complexity, the memory footprint of SGD-based
3

---

Under review as a conference paper at ICLR 2021
BP is also higher. This is because DNN software frameworks commonly store all activation values computed during FP to avoid recomputing at-1. the input activations to the layers, used in Equation 6 of SGD-based BP. In contrast, the localized update for layer can be performed FP through the layer is complete. The activation tensor o of layer can be discarded or over-written soon as FP proceeds to the next layer in the network, thereby freeing up significant portion of on-device memory during training. In turn, this can allow larger minibatch sizes to be accommodated on a given hardware platform, when the localized updates are applied on a sufficient number of layers.
soon as the
2.2 LEARNING MODE SELECTION ALGORITHM
The compute benefits of localized learning come the cost of potential loss in classification accuracy with respect to SGD training. Thus, we utilize a learning mode selection algorithm to judiciously choose when and where apply localized learning. The proposed algorithm identifies the learning mode of each layer at every epoch to maximize the runtime benefits, while incurring minimal losses in classification accuracy.
To design an efficient learning mode selection algorithm, we first study the effects of different spatio- temporal patterns of localized learning on the computational efficiency and classification accuracy of a network. We specifically investigate whether localized learning is more suitable for specific layers the network and specific phases the training process.
Impact on runtime efficiency: We first analyze the spatial trends, i.e., if locally updating specific layers in the network results in better runtime efficiency. layer L, updated with SGD precedes convolutional layer K, that is updated locally, calculating the SGD-based error gradients of Layer L, i.e. 6t, requires error propagation through the locally updated layer K. From compute efficiency perspective, the benefits of using localized-updater in layer K completely vanish. Thus, makes sense to partition the network into two regions prefix (set of initial layers) that updated using localized learning, followed by layers that are updated with SGD. SGD-based BP is stopped at the junction of the two regions. Naturally, the compute benefits increase when the number of locally updated layers are higher and thus the boundary i.e., the Localized-SGD transition layer is moved deeper into the network.
particular epoch, if a convolutional
The impact of different temporal patterns on runtime efficiency is quite straightforward, with higher number of locally updated epochs leading to higher benefits. Further, as the compute complexity of localized updates constant across different epochs, these benefits are agnostic of which particular epoch involves localized learning.
Impact on accuracy: To analyze the impact on accuracy, we first examine the nature of features learnt by different layers trained by SGD. It is commonly accepted that the initial layers of network (Agrawal et (014)) perform feature extraction, while later layers aid in the classification process. As localized learning demonstrates better performance for feature extraction, applying it more aggressively, iee for higher number of epochs, the initial layers has much smaller impact accuracy. However, for later layers in the network, the number of localized learning epochs should be progressively reduced to preserve accuracy.
Localized Updates
SGD Updates
update of Trans.
Norm of Wt.
Shift
Boundary!
Epochs
Epochs +>
Layer Index >
SGD-BP stops at this tronsition loyer
Transition layer
shifts right with epachs
Figure 2: Overview of the Learning Mode Selection Algorithm
Overall, based the impact of localized learning both runtime and accuracy, find that good learning mode selection algorithm should favor application of localized learning to a contiguous group of initial layers, while ensuring fewer no localized learning epochs in later layers. We further
4

---

Under review as a conference paper at ICLR 2021
impose additional constraint on top of this spatio-temporal pattern. Specifically, allow each layer to transition from one learning mode to another at most once during the entire training process. We empirically observe that utilizing SGD the initial learning mode allows the network to achieve a higher accuracy than utilizing localized learning as the initial mode. SGD essentially provides better initialization point for all layers, and the subsequent use of localized updates enables the training to converge with good accuracy.
In accordance with the above considerations, we propose a learning mode selection algorithm, described in Algorithm [] that identifies the po-
Algorithm Learning Mode Selection Algorithm
Input: TE (Index of the transition layer at epoch E), es (epochs since last transition). A W r: (Lg norm of the weight update of the transi- tion layer epoch E). K (minimum interval between transitions), Eshift (number of layers to shift boundary)
sition of the boundary or the Localized-sSCh transition layer every epoch. To that end, the algorithm analyzes the T.2 norm of the SGD weight updates made to the Localized->SS0 transition layer across epochs and determines whether the boundary can be shifted deeper into the network for the next epoch. In order to ensure stability in the training process, the al- most once in
Output: TE+1 (Index of the transition layer at epoch E+1)
1: WAng K -==E-1 ll A&ll
gorithm moves the boundary every K epochs. It calculates the running av- erage of the norm of the updates, W avgg. over the last K epochs (line 1). The boundary is shifted to the right only if the weight update
2: if A WE <= a WAeg and ex>=
3: TE+1 = TE + tshift
4: eq =0
5: else
in epoch E is within a fraction a of Wavg, and K epochs have transpired since the last transition (line 2). The rationale for this cri- terion that sustained high magnitudes of weight updates the transition layer indicate that they potentially critical to accuracy, in which case the transition layer must continue be- ing updated with SGD. If the criterion is not satisfied, the boundary remains stationary (line 5).
6:
TE+1 =Tg
7:
ek c +
The value of a is set by analyzing the trends in the weight update magnitudes across the training process for different networks. The
35
fs is set to the size of recurring block, such as the residual blocks in ResNets and MobileNetV2.. The hyper-parameter K is selected in manner that ensures that localized updates are never applied beyond some fraction of the initial network layers. We denote this fraction as Lymas and is
30
ResNett0
Restett4
Transitoon Layer Index
20
15
10
set to 0.75 in all our experiments. Equation 2 is used to compute for a network of L. layers and total training period of Emaz epochs.
21
Epochs
Emar
K Lmaz
L Laheys
Figure
3: Progression of
Localized SSD transition layer
In Figure 3 we plot the progression the transition layer across the ResNet-34 and -50 benchmarks trained on the ImageNet dataset using LoCal+SGD, Interestingly. the weight update norm metric automatically modulates the rate at which the boundary progresses, as the boundary traverses the deeper layers at slower rate.
2.3 WEAK SUPERVISION
To further bridge the accuracy gap between our approach and end-to-ent SGD training, we introduce weak supervision in the locally updated layers. Unlike the SGDD-updated layers, the locally updated layers in our approach cannot take advantage of the information provided by supervision, i.e., the classification error evaluated at the output. We utilize this supervised information through a low-cost weak supervision scheme that consists of a single signal sent to all layers updated locally in a particular epoch, and is derived from the classification loss observed over past few epochs. The weak supervision scheme is described in Algorithm 2
The key principle behind the weak supervision scheme is control the learning rates of the locally updated layers based on the rate at which the overall classification loss changes. For example, if the overall classification loss has increased across consecutive epochs, reverse the direction of the
5

---

Under review as a conference paper at ICLR 2021
updates (line 3) in the next epoch. In contrast, the update direction is maintained if the overall loss is decreasing (line 5). We find that this weak supervision provides better accuracy results than other learning rate modulation techniques for the locally updated layers such as Adam momentum-basse updates.
We would like to highlight that traditional SGD pro- vides fine-grained supervision and involves evalu- ating the error gradients for every neuron in the network. In contrast, the proposed weak supervi- sion scheme provides coarse gaazsee supervision by forcing all weights to re-use the same loss infor- mation. Overall, our weak supervision scheme is not developed with the intent to compete with SGD updates, but is rather a simple, approximate and low-cost technique that brings the final accuracy of LoCal+SGD at par with end-to-end SGD training performance.
Algorithm Weak Supervision Scheme
Input: L, (Overall classification loss at epoch
i), ir, (original learning rate of layer L)
Output: Wt. (Weight update of layer L)
1: A Wt. conv(art11 21)
2: if L,-1 L,
3: W. = WL -rr
AWL
|AWl
4: else
5: W, = W, lrr."
AW. TTW2
EXPERIMENTAL RESULTS
In this section, we present the results of our experiments highlighting the compute benefits achieved by LoCal+SGD. We evaluate the benefits across suite of 8 image-recognition DNNs across datasets. We consider the ResNetl 8 (He et al., 2015) and VGG13 (Simonyan & Zisserman, 2015) networks for the Cifar10 (Krizhevsky
al.) a) and Cifar100 (Krizhevsky et al. datasets; and
the ResNet34, ResNer50 (He et al.] 2015) and MobileNetV2 (Sandler et al., 2018 networks for ImageNet dataset (Deng al. 2009). All experiments are conducted on Nvidia GTX 1080Ti
GPUs with the batch size set to 64 per GPU, unless otherwise mentioned. Further experimental methodology details for the baseline and proposed approach are provided in the Appendix.
3.1 SINGLE GPU EXECUTION TIME BENEFITS
ImageNet: Table presents the performance of the baseline (end-to-end SGD training) and the proposed LoCal+ SGD algorithm on the ImageNet benchmarks in terms of the Top- classification error and runtime observed on single GPU. For all benchmarks listed here, LoCal+ applies localized updates for nearly 50-60% of the layers. As can be seen, LoCal+SGG achieves upto ~ 1..4x reduction runtime compared to the baseline, while sacrificing <0.5% loss in Top-1 accuracy.
Table I also compares the performance of LoCal+SGD against existing research efforts designed to improve training efficiency. We perform this analysis against two efforts, namely (i) Training with stochastic depth (Huang al. (2016) and (ii) Structured Pruning during Training (Lym et al. 2019). Training with stochastic depth, as the name suggests, stochastically bypasses residual blocks by propagating input activations/error gradients via identity downsampling transformations. resulting in improved training time. However, the approach is targeted towards extremely deep networks and
Table 1: ImageNet
Network
Training Strategy
Top-1 Error
Speed
Bascline SGD
26.6%
1x
LoC
27.04%
1.26x
ResNet34
Training with Stochastic Depth
27.89%
1.13x
Freezing layers during training
27.42%
1.38x
Bascline SGD)
24.02%%
LoCaI+SGD
24.41 %
1.36x
ResNet50
Training with Stochastic Depth
26.76%
1.08x
Pruming during training
24.89%
1.32x
Freezing layers during training
25.64%
1.44x
Baseline SGD
28.41%
LoCal+SGD
28.94%
1,31X
MobileNetV2
Training with Stochastic Depth
30.53%
1.17x
Freezing layers during training
29.49%
1.52x

---

Under review as a conference paper at ICLR 2021
seen in Table 1 it incurs noticeable accuracy loss networks such ResNet34, ResNet50 and MobileNetV2. Compared to training with stochastic depth, our proposal clearly achieves better accuracy as well as training runtime benefits. The key principle behind the pruning during training approach to reduce the size of the weight and activation tensors in a structured manner during training, thereby providing speed-ups on GPU/TPU platforms. However, on complex benchmarks such as ResNet50, such techniques achieve speed-ups at the cost of significant drop in accuracy (~ 1.5%). To further demonstrate the utility of localized updates in our approach, we consider a third technique, wherein layers selected to be updated locally for given epoch are instead frozen, i.e., the parameters are held fixed during that epoch. While this achieves better runtime savings, it incurs considerably higher loss (~I ()) in accuracy, further underscoring the benefits of LoCal+SGD.
C-FAR-10 and CIFAR-100: Table presents the accuracy and corresponding compute benefits of the baseline and the proposed technique, well as training with stochastic depth and layer freezing, for the CIFAR- 10 and CIFAR- 100 datasets. Stochastic depth is applicable only to residual blocks and is hence not considered for the VGG-13 network. Across benchmarks, improvement in training runtime. Compared to the ImageNet benchmarks, LoCal+SGD applies localized updates
observe upto 1.51
aggressively in the CIFAR-10 and CIFAR- 100 benchmarks i.e., for more layers are updated locally for a higher number of epochs. This leads to the superior compute benefits of the proposed scheme these benchmarks.
Table 2: Cifar10 and C0
(ettork(Dataset)
Training Strategy
Top-1 emr.
Speed-Up
Baseline SGD
6.06%
ResNet18
LoC
6.23 %
1.51x
(O)
Training with Stochastic Depth
6. 8.79%
1.35x
Freezing layers during training
6.51%
1.65x
Baseline SGD
7.16%
1x
VGG13
LoCal+SGD
7.25 %
1.31x
(Cifar10)
Freezing layers during training
7.43%
1.42x
Baseline SGD
23.39%
ResNet18
LoCal+SGD
23.63%
1.44x
(Cifar100)
Training with Stochastic Depth
23.97%
1.35x
Freezing layers during training
23.74%
1.62x
Baseline SGD
31.36%
1X
VGG13
LoCal+SGD
31.59%
1.32x
(Cifar100)
Freezing layers during training
31.94%
1.42x
3.2 EXECUTION TIME BENEFITS FOR MULTI-GPU TRAINING
We analyze the memory footprint of the ResNet50 net- work when trained with LoCal+ dataset. Training first commences with all layers updated with SGD, resulting in high memory footprint. Due to the 10 GB capacity of the chosen GPU, the mini-batch size is set to 64 per GPU. As the Localized-sSGD tran- sition layer progresses across the network, the memory footprint required also gradually reduces across epochs. We take advantage of this reduction in memory footprint in the context of distributed training using 4 GPUs with data parallelism. Specifically, we extract additional run- time benefits by increasing the batch size on each GPU, which reduces the frequency of gradient aggregation between devices and alleviates the communi- cation overhead. At epoch 33, the memory footprint per GPU reduces to less than 5 GB, allowing training with increased mini-batch size of 128 per GPU from epoch 33 onwards. The doubling of the batch-size provides an additional 6% runtime improvement, when measured across the entire training period. We note that other training techniques such as training with stochastic depth cannot exploit this feature, due to minimal reduction in memory footprint.
the ImageNet
192
Mem. Footppiitt
1
Batch Siee
Norm. Memory Footpeint
128
Botch Size
0.4
64
Rooterint
0.2
4.7 01 2000h
11
21
31
41
51
61
71
81
Epochs >
Figure 4: Analyzing memory footprint and batch-size variation
3.3 ABLATION ANALYSIS
As mentioned in Section 2 the of the boundary across the network. Different values of either parameter result in dif-
a, tshift and Lynar control the progression
7

---

Under review as a conference paper at ICLR 2021
Table 3: Analyzing Impact of Increasing Batch-Size ImageNet
Network
Training Strategy
Top-1 err.
Speed-Up
Baseline SGD (fixed batch-size)
24.06%
1x
ResNe150
LoCal+SGD (fixed batch-size)
24.48%
1.27x
LoCal+SGD (variable batch-size)
24.51 %
1.34x
ferent learning mode configurations during training, resulting in different points in the com- putational efficiency vs. accuracy trade-off space. To understand the trade-off space be- tween accuracy and runtime benefits, we now individually study the impact of each parameter.
Speed-Up
Impact of Figure depicts the best compute benefits achieved for differ- ent a, for accuracy losses ranging from 0.1%-1.5% for the ResNet50 and MobileNetV2 benchmarks on ImageNet. On the ResNe150 benchmark. even while limiting the loss in accuracy 0.1%, LoCal+SGD achieves 1.1x speedup over tradi-
1.4
1.5
1.3
Spend-Up
1.4
Speed-Up -Up
1.2
0
0.4
0.8
1.2
1.6
0.4
0.8
1.2
1.6
Loss in Acc(%)
Loss Acc\%)
(a)
(b)
Figure 5: Compute efficiency accuracy trade-off the ImageNet dataset for a) ResNet50 and b) MobileNetV2
tional SGD. The speedups increase to 1.38 x when around 1.5% loss in accuracy is tolerable.
Impact of tehffe Figure 6[a) depicts the accuracy achieved for different fshift values for the ResNet50 net- work trained using ImageNet. Here, we denote Lshift as a percentage of the total network depth. For each value of tehaft, we identify a and Lmaz that result in the best accuracy while pro- viding considerable runtime savings (at least greater than 10%). The graph indicates that accuracy is severely im- pacted for extremely small values of tshift that are less than 3%. The curacy is largely stable in the regime of tabift between 5-12%, and begins to experience small degradations again when lahift exceeds 12%. These trends be explained by analyzing the rate which the transition layer progresses, and the number of layers transitioning to localized updates in an epoch for different tshift values. Smaller values of fahife (<3%) give rise low values of k (~~1-2 epochs), the minimum number of epochs that must elapse before the transition layer can shift again. This results in fast progression of the transition layer across the network, leading to rapid changes in the learning mode at the boundary, thereby negatively impacting accuracy. In contrast, while larger tskife values (>12%) encourage slow progression of the boundary, larger number layers transition from SGD to localized updates in single epoch, thereby impacting performance. We note here that in both cases, while a and Tanax can be tuned to control the progression and mitigate the loss in accuracy, the runtime savings is vastly reduced (<10%). Furthermore, for fixed values of Limaz and a, tehift is largely insensitive to runtime benefits, as the average number of layers updated with localized updates remains similar. Hence, for best accuracy and runtime benefits set thk in the range 5-10% for all networks.
Accuracy
Speed
35
1.4
^.
28
1.6
Top- Enror
1.3
30
Errer
26
1.4
1.2
25
1.1
Runtine
24
1.2
20
1
1
10
13
30
70
Tunt. %
Lyn. 36
Figure 6: Impact of (a) tshift and (b) Tamaz accuracy and runtime savings on the ImageNet dataset for ResNet50
Impact Lmaz Figure 6 depicts the impact of Lmaz accuracy for the ResNet50 network. For each Limaz we identify the a and txhift that provide the best runtime benefits with minimal loss in accuracy (less than 0.5%) As with lhifts denote Temaz percentage of the total network
8

---

Under review as a conference paper at ICLR 2021
depth. As seen in the figure, the degradation in accuracy increases slowly for Lmer in the initial layers it is merely 0.1% at around Lmaz = 30%, and increases to 0.4-0.5% for Lmax = 60-70% However, the accuracy degradation sharply increases beyond 2% once Lmaz exceeds 90% of the network depth. Further, runtime benefits generally increase with higher values of Tmmax for fixed ft and Hence, for achieving a good accuracy versus runtime trade-off, usually set Temax to 75% for all networks.
RELATED WORK
This section discusses related research efforts to the proposed LoCal + SGD training technique. These efforts can be broadly categorized into two classes. The first class of efforts focus on compute efficient DNN training. All efforts belonging to this class utilize gradient-descent algorithms train the DNN model. These techniques are largely complementary to LoCal+SGD, as they can potentially be applied to the parts of the DNN model updated with SGD. Section 3 we demonstrated how LoCal+ achieves superior accuracy versus computational efficiency trade-off than some of these efforts. Further, the second class efforts involve neuro-seen scientific faithful learning rules, such as feedback alignment based efforts etc (Nokland 2016). Our work is orthogonal to such efforts, selectively combine localized learning rules with SGD for better computational efficiency.
We elucidate upon the different research efforts in both directions follows.
Hyper-parameter tuning: Many notable algorithmic efforts are directed towards achieving training efficiency by controlling the dyper-paramenn
involved in gradient-descent notably the learning rate. (You et al., 2017a| Akiba et al., 2017; ; Goyal et al. 2017 You et al. 2017b) propose learning rate tuning algorithms that achieve training less than hour with loss in accuracy, when distributed to over hundreds of CPU/GPU cores.
Model size reduction during training: Model size reduction via pruning and quantization is popular technique to reduce compute costs during inference. In many of these efforts, a dense or full precision model is re-trained or fine-tuned to obtain pruned or quantized model. Recently, several efforts have also investigated dynamically pruning (Lym et al. (019)) or quantizing (Sun et al. 2019) a model during training itself. The reduction in model size results in training speed-ups. Taking slightly different approach (Huang et al. 2016) proposes stochastically dropping residual blocks on extremely deep networks such as ResNet- 1202, not only for training runtime benefits but also better accuracies due to improved gradient strength.
Instance importance based training: Recent research efforts have discovered that not all training samples are required for improving loss minimization during SGD training (Jiang et al. 2019 Zhang ct al. 2019). That is, a sizable fraction of the samples can be skipped during several epochs, depending on their impact the classification loss evaluated during FP. This translates reduction in mini-batches, providing considerable runtime benefits.
Neuro-scientific learning rules: Back- pockkpropagatton algorithms utilized in DNN training are not biologically plausible, and do not explain how learning actually happens in the brain. To this end, there have been several efforts that develop biological faithful learning algorithms, and demonstrate considerable success on complex benchmarks including Cifarl0 and ImageNet. For example, unlike conventional DNN training, feedback alignmnent algorithms (Nokland) 2016) tackle the weight transport problem (Liao et al. 2015) by allowing for asymmetry in the weight values during forward and back propagation. Likewise, Barget-Propagattin (Lee al., (2055) encourages neural activity to reach desired target activations evaluated during forward propagation itself, instead of utilizing loss gradients.
5 CONCLUSION
In this paper, we introduce a new approach to improve the training efficiency of state-of-the-ard DNNs. Specifically, we take advantage of the computationally efficient nature of localized learning rules and selectively update some layers with these rules instead SGD. We design an intelligent learning mode selection algorithm that determines the update method for the convolutional layers of the network in every epoch while maintaining the accuracy level and extracting maximum benefits. Further, we also implement low-cost weak supervision scheme that brings the accuracy of the proposed scheme closer to traditional SGD training. Across a benchmark suite of DNNs, we achieve upto 1.5 reduction in training times, as measured modern GPU platform.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pulkit Agrawal, Ross B. Girshick, and Jitendra Malik. Analyzing the performance of multilayer neural networks for object recognition. CoRR, abs/1407.1610, 2014. URL http:: httpp://axxiy
org/abs/ 1407. 1610
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch SGD: training resnet-50 on imagenct in 15 minutes. CoRR, abs/1711.04325, 2017. URL
1711. 04325
Léon Bottou. 2010.
Large-scale machine learning with stochastic gradient descent. In in COMPSTAT,
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc' Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, NIPS' 12, pp. Curran Associates Inc.
Red Hook, NY, USA, 2012a.
Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoe V. Le, Mark Z. Mao, Mare' Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang. and Andrew Ng. Large scale distributed deep networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume NIPS' 12, pp. 1223-1231, Red Hook, NY. USA, 2012b. Curran Associates Inc.
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Yoav Goldberg and Graeme Hirst. Neural Network Methods in Natural Language Processing. Morgan Claypool Publishers, 2017. ISBN 1627052984.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in hour. CoRR, abs/l 706.02677, 2017.
org/ http:////aaxxi...oaaa/////////// 02677
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
D.O. Hebb. The organization of behavior: A neuropsychological theory.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. CoRR, abs/1603.09382,, 2016. URL fttp://arxiv/ http:///xxivv.rg///e//. haargg/bb/6600 09382
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, 2019.
Angela H. Jiang, Daniel L. K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, and Padmanabhan Pillai. Accelerating deep learning by focusing the biggest losers, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: 7/arxiv. org. abs 1412. 6980
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). a. URL URL //t:://wwww.. cs www.ttor.oooooooooooooooo...o foo.canto.ect/-kkii//iiiii....ee html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL htt.http::// //wwtp http:///www.....oooott.o.eee/e// htmll
10

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoftrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume I, NIPS' 12, pp. 1097-1105, Red Hook, NY, USA, 2012. Curran Associates Inc.
Alex Krizhevsky, Πya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun. ACM, 60(6):84-9,, May 2017. ISSN 0001-0782. doi: 10.1 10.1145/3065386.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:tttpyyttt
lecun com/ wwwb/mmmaaee /mmnst/////
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Proceedings of the 2015th European Conference on Machine Learning and Knowledge Discovery in Databases Volume Part I, ECMLPKDD' 15, pp. 498-515, Gewerbestrasse 11 CH-6330, Cham (ZG). CHE, 2015. Springer. ISBN 97833192352777 doi: 10. 10.1007/998-33319-235558 31. URL
100./ http://d..0rrr0111000000/1113333
Duo Li, Aojun Zhou, and Anbang Yao. Hbonet: Harmonious bottleneck on two orthogonal dimen- sions. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
Qianli Liao, Joel 7.. Leibo, and Tomaso A. Poggio. How important is weight symmetry in backpropa- gation? CORR, abs/1510.05067, 2015.
waaprr//bb////aa /1510. 05067
Sangkug Lym, Esha Choukse, Siavash Zangench, Wei Wen, Mattan Erez, and Sujay Shanghavi, Prunetrain: Gradual structured pruning from scratch for faster neural network training. CoRR, abs/1901.00299, 2019. URL http::http::tiiee wwwpaaawwww..aoaaaaaooaaaeooeaae abs/ /901. 09290
Joe Yue-Hei Ng. Matthew J. Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. CoRR,
2015. URL httt:: http:///axxxvv.../abbsess abs/ wtt.:oo///abbboooo.o
08909
Arild Nøkland, Direct feedback alignment provides learning in deep neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 1045-1053, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819,
Erkki Oja. Simplified neuron model principal component analyzer. Journal of Mathematical Biology, 15:267-273, 1982.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, dAlché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL /ttp/t/ttt///pppe...oorii.speee
library.
pdf
Mark Sandler. Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov. and Liang-Chieh Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. CoRR, abs/1801 04381, 2018. (RRl:/ttp///axx.. wwtp:xiwwwiorroooooooo.oe wttp://orgga/bbbabbbeeooooe
04381
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representationss 2015.
Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurIPS, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018.
11

---

Under review as a conference paper at ICLR 2021
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. CoRR, abs/ abs//770..0388, 2017a. /Rl.aatt://aa....oo//////// 03888
Yang You, Zhao Zhang, Cho-Jui Hsieh, and James Demmel. 100-epoch imagenet training with alexnet in 24 minutes. CoRR, abs/1779.05001 2017b.
05011
Jiong Zhang, Hsiang-Fu Yu, and Inderjit Dhillon. Autoassist: framework to accelerate training of deep neural networks. CoRR, abs/1905.03381, 2019. URL htt.t::://aa....o//////
1905. 03381
Shi Hui Zhong. Efficient online spherical k-means clustering. Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., 5:3180-3185 vol. 5, 2005.
Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. A C-LSTM neural network for text classification. CoRR, abs/151 .86600, 2015. URL http::/tttttt.ortrrrete/
08630
APPENDIX
6.1
EXPERIMENTAL SETUP
This subsection describes the experimental setup used for realizing the baseline and proposed LoCal+SGD training schemes, on the benchmarks specified in Section 3 of the main paper. We conduct our experiments the complete training and test datasets of each benchmark, using the PyTorch (Paszke et al. 2019) framework.
Baseline: We consider end-to-end SGD training the baseline in experiments. The hyper- parameters used in SGD training of each of the benchmarks are described below.
For experiments in Section 3.1 we utilize batch-size of 64 per GPU, for all benchmarks. For the ResNet50 and ResNet34 benchmarks ks the initial learning rate set to 0.025. The learning rate is decreased by 0.1 every 30 epochs, for total training duration of 90 epochs, and the weight decay is 4e 5. The MobileNetV2 benchmark utilizes an initial learning rate of 0.0125. We use cosine learning rate decay schedule, as (Li al. 2019) for 150 epochs. The weight decay is set 4e Both benchmarks use input size of 224*2....
ImageNet:
For the experiments in Section 3.2, the total batch-size epoch is 256 (64*4), with the initial learning rate set to 0.1 for the ResNet benchmarks and 0.05 for the MobileNetV2 benchmark. All other parameters remain the same.
C0far10 and 1if All Cifar10 and Cifar100 experiments utilize a batch-size of 64. The Cifar10 benchmarks are trained with an initial learning rate of 0.05 that decayed by 0. every 10 epochs, across 90 epochs. The initial learning rate of the C00 benchmarks is 0.025 and decayed by 0.5 every 20 epochs, for 150 epochs total. The weight decay set to - 4. Both benchmarks utilize an input size of 32*32*3.
LoCal+SGD: In the proposed LoCal+SGD training scheme, the layers updated with SGD are trained with the same hyper-pane syper-parameters used in the baseline implementation. Further, LoCal+SGD training conducted using the same number of epochs as baseline SGD training. When a layer is updated locally, the initial learning rate is 0.01 and is decayed by a factor of 2 and 10 every 30 epochs, for the Cifar and the ImageNet benchmarks respectively. In all experiments, the parameter is set to 0.95. We measure the accuracy and runtime of the proposed scheme for the same number of training epochs as the baseline implementations.
6.2
TUNING
To realize LGC, we introduce three syper-parameters. a, tahift and Lmaa- tahift controls the number of layers that switch to SGD-based updates every epoch, Lyaz is the maximum number of layers that can be updated with localized learning rules, and determines the position of the
12

---

Under review as a conference paper at ICLR 2021
transition layer every epoch by analyzing the gradient information the boundary between the localized and SGD updates.
To obtain optimized values for these syper-parameters, first perform simple grid search using single network for a particular dataset (for example, we choose the ResNet50 network for ImageNet). We transfer the same cyper-paramees values other networks for the same dataset. We justify our use of common results on other ImageNet benchmarks (ResNet34 and MobileNetV2) when hyper-parameter tuning is performed for each benchmark individually. As can be seen, the accuracy and runtime benefits are only marginally better than those obtained using a common set of hyper-parameter obtained by tuning on the ResNet50 benchmark. We thus utilize common values for a dataset, effectively rendering them constants. The time taken to obtaín these constants is thus one-time cost, and does not impact the speedups obtained by LoCal+SGD.
values by the following experiment. In Table below, we depict the
Table 4: Analyzing impact of a common set of hyper-parameters,
Network
Training Strategy
Top-1 err.
Speed-Up
LoCal+SGD (finc tuned)
26.93%
1.32x
ResNet34
LoCal+SGD (common constants)
27.04%
1.26x
LoCal+SGD (fine-tuned)
28.82% 29.94%
1,36X
MobileNerV2
LoCal+SGD (common constants)
1.31x
6.3
IMPACT WEAK SUPERVISION
In Table 5 we highlight the impact of the weak supervision technique on final classification accuracy. As can be seen, across all our benchmarks, the weak supervision technique clearly improves accuracy by nearly 1.06%-0.7%% bringing the final accuracy of LoCal + SGD closer to baseline SGD.
Table Impact of Weak Supervision on accuracy
Top-1 em: with weak
Top-1 err. without
Dataset
Network
supervision
weak supervision
ResNet34
27.04%
27.1%
ImageNet
ResNet50
24.41%
24.49%
MobileNetV2
28.94%
29.03%
VGG13
7.25%
7.39%
Ci
ResNet18
6.23%
6.41%
VGG13
31.59%
31.7%
Cifar100
ResNet18
23.63%
23.75%
6.4 ADDITIONAL COMPARATIVE ANALYSIS
In addition to the experiments performed in Section to compare the performance of Lo- Cal+SGD against existing techniques such as pruming during training (Lym et al. 2019) and training with stochastic depth (Huang et al. (2116). conduct additional experiments further solidify the superiority our approach. We clucidate upon these comparisons as follows.
6.4.1 COMPARING LoCal SGD AGAINST SGD ISO-ACCURACY
We compare the proposed LoCal+SGD training strategy against SGD baseline that is trained with fewer epochs, i.e., the number of epochs required to reach the highest accuracy obtained by LoCal + SGD across the total training periods listed in Section 6.1 For the ImageNet benchmarks, the runtime improvements are listed in Table below. Clearly, LoCal+SGD continues to achieve significant speed-ups (around 1. 1.25x) compared the SGD baseline, even for complex benchmarks such as ResNet50 and MobileNetV2.
6.4.2 COMPARING LoCal+SGD AGAINST FREEZING LAYERS DURING TRAINING
In Section 3
we compare LoCal+SGD against a technique, freezing layers during training, wherein instead of updating the layers using localized learning, the weights held fixed. In this section,
13

---

Under review as a conference paper at ICLR 2021
Table 6: Analyzing SGD speed-up at iso-accuracy
Network
Training Strategy
Top-1 err.
Speed-Up
SGD baseline (76 epochs)
24.40%
1x
ResNet50
LoCal+SGD (83 epochs)
24.41%
1,26x
SGD baseline (138 epochs)
28.92%
1x
MobileNetV2
LoCal+SGD (146 epochs)
28.94%
1.24x
we perform more thorough comparison of LoCal+SGD against freezing layers during training. Specifically, we perform this comparison at iso-runtime, and analyze the resulting accuracy of either approach. To elaborate, we first identify the LoCal+SGD configuration that can reach the best accuracy within 0.05%, 0.1%, 0.25%, %,5, 0.5% and 1% of the baseline SGD accuracy. Then, for the same runtimes taken by each LoCal++GD configuration, we identify the configuration that provides the best accuracy for the freezing layers approach. Our results for the Cifar 10 ResNet18 benchmark can be found in Table 7 LoCal+SGD performs superior to freezing layers during training out of the 5 configurations studied, i.e., is a superior technique when the loss compared to SGD is allowed exceed 0.1 %.
Table 7: Comparing LoCal+ against freezing layers during training
Loss accuracy for Lo-
Loss accuracy for freezing
Maximum loss in Top-1 Accuracy
Cal+SGD
layers during training
0.05%
0.08%
0.04%
0.1%
0.1%
0.09%
0.25%
0.21
0.28%
0.5%
0.41 %
0.56%
1%
0.85%
0.97%
6.5 ANALYSIS OF STATIC SCHEDULES FOR LEARNING MODE SELECTION
The current LoCal+SGD framework is realized with the help of an automatic learning mode selection algorithm, which determines the position of the transition layer every epoch. Instead of dynamic data-dependent algorithm, we investigate the benefits of using a static schedule that is, the position of the transition layer is determined using some pre-defined scheduling function. To this end, we have implemented a simple static schedule that favors aggressive application of the localized learning rule in initial layers, and gradually decreases the number of epochs localized learning applied the deeper layers. As shown in Equation 3 we opt for quadratic scheduling function, observe they perform better compared to the linear functions studied. Here N determines the position of the transition layer every epoch, Emar is the maximum number of training epochs, and с₁ and c2 are constants obtained using grid search.
empirically
N |max(0, - C2 (E Emeex )
(3)
We report the results using this static schedule in Table X for the ImageNet- ResNet50 and Mo- bileNetV2 benchmarks. Compared to the results reported in Table 1, we find that the static schedule achieves slightly higher runtime benefits, for marginally lower accuracies. However, static schedules suffer from some drawbacks - several static scheduling functions are feasible, e.g. exponential, quadratic, etc., and identifying the best scheduling function for each network requires extensive empirical analysis. The learning mode selection algorithm utilized in the paper helps alleviate this by automatically identifying the position of the transition layer every epoch, leveraging the gradient information at the boundary between localized updates and SGD.
Table 8: Analyzing Impact of Static Learning Mode Selection Schedule
Network
Training Strategy
Top-1 emr.
Speed-Up
ResNet50
LoCal+SGD (static schedule)
24.51%
1.41X
MobileNerV2
LoC (static schedule)
28.90%
1.33x
14


Reviews:
[{"review_id": "xd5iQX7dhT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: 2020-11-27: Updated my score. Further explanations at the end of this post.\n\nSummary: \nThe authors propose an algorithm that separates a network into two distinct regions where one is trained with SGD while the other is trained with a local Hebbian learning rule. A weak supervision rule is proposed that improves localized learning. The authors demonstrate close to baseline performance on CIFAR-100 and ImageNet for diverse networks while speeding up training.\n\nStrong points:\n- Very creative use of multiple learning rules during training.\n- People could never get localized learning to work well enough to compete with SGD. This is the first work that uses some localized learning and manages to come close. This is a big success.\n- These findings have broad applicability, from brain-like learning algorithms, efficient training on one GPU, and efficient learning algorithms for model parallelism for massive networks where communication is the bottleneck.\n\nWeak points:\n- Missing ablation for the weak supervision algorithm.\n- This work is very impactful in many different ways, but it only mentions the computational efficiency perspective. The related work needs to be expanded to make readers aware of the impact of this work.\n- The heuristic for learning mode selection is complicated. Initial experiments suggested that a simple learning-rate-schedule-like way to select the learning mode would be possible.\n- Some algorithmic details in the weak supervision algorithm not clear.\n\nRecommendation (short):\nThis is very important work with results that will significantly impact many fields (efficient training, parallelization, brain-like algorithms). It is a creative solution to a significant problem in localized learning. I strongly recommend accepting this work. If this work is rejected, I will no longer review for future ICLR conferences. I recommend this work to be accepted as an oral presentation. Selecting this work for a best paper award would be appropriate.\n\nRecommendation (long):\nThis work is impactful for multiple reasons:\n- Localized learning is difficult. There has not been a work that uses localized learning and makes it work close to SGD performance on large datasets/models.\n- The brain does not use SGD, and it is difficult to think about algorithms that work in the brain and yield good performance. It might be that initial learning in the brain is done differently until local learning rules are used. This view is mostly ignored, but this paper yields evidence that such learning might be possible and efficient.\n- Layers updated with localized rules can be updated independently of other layers (if the weakly supervised rule is not used). This enables fully asynchronous training for early layers. There will be a synchronization point at the SGD layers, but through pipeline parallelism, the communication overhead can be hidden in Hebbian layers. This enables the training of massive neural networks with trillions of parameters. With current parallelism tools becoming more and more limited, this is a crucial innovation since previous asynchronous parallelism procedures always decrease predictive performance. This is the first work that shows a way to do asynchronous parallel training without performance degradation.\n\nBeyond this, the paper also yields some speedups for tasks while decreasing predictive performance only slightly. This is also an impressive feat, but the overall broad insights this paper yields make it much more impactful than this result. As such, I do not view the experimental results as the main contribution, but overall, the papers' main contribution is that it shows a way to include (gradual) localized learning in a neural network while not impacting performance.\n\n\nComments for authors:\nThis is excellent work — well done! I think the main weakness is currently a missing ablation on the effect of the weak supervision rule. You note that improves performance but by how much would be an important detail. If you do not include these ablations, I would still accept the paper, but I might rescind my oral presentation recommendation.\n\nAnother issue is that your work is relevant in many different domains, but you keep it confined to the idea that your method is only useful for faster training. I think making the reader aware that local training has many advantages across many domains could be very valuable. You do not need to elaborate on this, but I would like to see some of these connections in the paper because not everyone has the background to see these connections. I think you can do this mostly by mentioning it in the conclusion since you already mention a little bit of work in parallelism, and you mention previous results about local learning that failed to obtain good performance. Another line of work that I would mention in the related work section is that of neuroscientific faithful learning rules. The most relevant line of research is the work on various forms of feedback alignment and other algorithms. For a summary of past research and results on large datasets, see Bartunov et al., 2018[1]. Beyond this, you might want to add \"sparse training\" to the related work on efficient deep learning. Sparse training differentiates from pruning during training by initializing the neural network sparsely during initialization (not densely and then prune to sparse). See work on a mixture of experts[2,3] and sparse dynamic training[4,5,6]. I do not require you to include these references, but they might improve the related work section. \n\nOn the algorithmic and experimental side, it seems that a simple learning-rule-like schedule might be sufficient for selecting the learning mode. While I do not require you to add these experiments, it would make the algorithm simpler and more appealing if you can show that a simple learning rule works a la \"warmup with SGD for 5 epochs, then shift by 1 layer (block) every 5 epochs\" etc.\n\n[1] Sartunov et al., 2018. Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures\n[2] Shazeer et al., 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer\n[3] Lepikhin et al., 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\n[4] Mostafa & Wang et al., 2019. Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization.\n[5] Dettmers & Zettlemoyer et al., 2019. Sparse Networks from Scratch: Faster Training without Losing Performance.\n[6] Evci et al., 2019. Rigging the Lottery: Making All Tickets Winners.\n\n\nUpdate:\n==================================\n\nComments for Area Chair and Reviewers: \n\nIf I view this work merely by the story conveyed in the paper, my assessment would be more in line with the other reviewers. I am not quite sure if this is the right way to evaluate this paper since I view it as having a broader impact that goes beyond the story in the paper, but other reviewers disagree with my view on its broader impacts. I see this as a sign that the paper is currently not in a good enough state to really convey its potential impact.\n\nComments for authors:\n\nI believe you still did good work here on the merits of the \"speedup training\" story that you convey in your paper. I believe that you have much more than this in your hands though. I think you could go two ways from here: (1) get this paper accepted in this form and work closely on the other angles that this work offers in a new paper, e.g. learning which is in line with biological or efficient parallelization of large networks. The second way (2) would be to rewrite this paper more in line with that view and resubmit. I think (1) might be better for you. I do not think many reviewers would understand a paper that comes from the process in (2). Good luck!", "extra_scores": {}, "extra_text": {}}, {"review_id": "xYbpldQCHMC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Accelerating DNN Training through Selective Localized Learning\n\nIn this paper, the authors proposed a new approach by the name of LoCal + SGD (Localized Updates) to replace the traditional Backpropagation method. The key idea is to selectively update some layers’ weights using localized learning rules. For these layers, the computation cost is reduced from two matrix multiply operations to one matrix multiply operation. The authors also proposed the Learning Mode Selection Algorithm to maintain the accuracy and convergence.\n\nThe authors provided some experimental results on common deep learning benchmarks such as ImageNet/ResNet and CIFAR/VGG. Overall, the authors reported that this approach can achieve around 1.36x speedup for a 0.4% loss in accuracy for ResNet-50. The authors also reported that they can achieve a higher speed than recent methods such as Structured Pruning and stochastic depth.\n\nThis work is a trade-off between computation and accuracy (details in Figure 5). I have some questions for the authors:\n\n(1) How stable is the proposed method (LoCal + SGD)? Does it still work for large-batch optimization and asynchronous training?\n\n(2) What is the overhead of hyper-parameter tuning?\n\n(3) Did the authors use the same number of epochs as the baseline to finish the training?\n\n(4) What is the absolute speed (e.g. in GFlops or TFlops)? Can the proposed method beat a well-optimized NVIDIA implementation?\n\n(5) What is the limit of the proposed method?\n\n(6) Can the Learning Mode Selection Algorithm work with other methods?\n\nSince this work fundamentally changes the way of learning, it is probably necessary to do a convergence analysis for the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZaXIt7ZDPhl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper try to leverage the benefit of Hebb learning to reduce CNN training time cost. In order to achieve this,  a learning mode selection algorithm is proposed to progressively increase  number of layers using Hebb learning. The writing of this paper is good and the idea is also interesting, however, the experimental part should be improved:\n\n1. The criterion used in learning mode selection algorithm is the model-update norm of current epoch. If the norm is small enough, the transition layer index will be increased. A small model-update norm also means that current layer is nearly convergent. Could you just fix these layers to accelerate training? Yes, freezing layer exps are tried but the comparison is not fair in my opinion. When Hebb-Learning-Layers are frozen, the final accuracy drops, but the training speedup is improved. So if you freeze less layers to make training speedups of freezing-layer-training and Hebb-Learning same, what will the accuracy relationship be? Does the proposed method still outperforms freezing strategy?\n\n2. A weak supervision scheme is proposed in this paper, but I did not find any experiments to evaluate its effect, could you add this part? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "NY4HU8Sy-tZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a combination of SGD with selective application of a non-backprop learning rule (Hebbian). The two learning rules are not applied together, but rather a boundary is determined where layers prior use SGD, and the ones after use the Hebbian approach. A selection algorithm dynamically adjusts the boundary over training. For accuracy reasons, they include weak supervision by using the overall classification loss to control the sign of the update. \n\nFrom computational efficiency perspective, the contributions reduce the need for a backprop calculation, and also leads to a smaller memory footprint, since the activation values need not be stored. On ImageNet benchmark models, they show <0.5% Top1 drop in exchange for ~1.3x runtime speed compared to vanilla SGD.\n\nStrengths\n+ Focus on run-time improvements brings practical significant to their proposed method\n+ Algorithm is relatively simple to implement\n+ Convergence results only show a small degradation from SOTA\n+ The tuning results for alpha (Figure 5) are useful for practitioners that need to balance accuracy and compute\n\nWeaknesses\n- The 'meta' boundary selection and weak supervision approaches add additional hyperparameter complexity to the tuning process. These are also empirically but not theoretically motivated, and unclear if they generalize to other domains. I understand that non-classification models are out of scope for this paper, but his paper's impact would be improved by some comment on transferability. For example, U-net models have long range skip connections that span the model. \n- While the focus on runtime is welcome, what is relevant to the practitioner is time-to-train to a particular accuracy target, similar to the metric adopted by MLPerf. Since this method does introduce an accuracy degradation (e.g. for RN-34, 27.04% versus 26.6%; Table 1), the more fair comparison would account for the fewer epochs the baseline SGD needs to hit 26.6%. To make a more convincing argument to practitioners, I would compare either the wall-clock time, or the total FLOPS needed, to hit the target accuracy.\n- Several techniques are introduced without ablations to measure their effectiveness and justify the added complexity. This is particularly important as these techniques add additional burden on the practitioner in terms of tuning the new hyperparameters. \n\nThe work combines existing learning rules (SGD, Hebbian) with some novelty in how they are employed, and with a weak supervisory signal, to achieve reasonable results. These contributions were not foundational improvements, so the paper's main merit is in the potential practical impacts of this method. The significance to practitioners however, is greatly reduced by the weaknesses described above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6YizFfZAPun", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work demonstrates that localized learning can improve DNN training efficiency by reducing computation and memory requirements. The effectiveness of this approach is shown by the experimental results that report a nice trade-off of minimal accuracy loss and good throughput improvement compared to baseline SGD and competing efficient training techniques. \nWhile the experimental results are indeed appealing, a major flaw is that the paper does not sufficiently explain why the underlying techniques (learning mode selection and weak supervision) work. Since these techniques are configured with a number of hyper-parameters, and I could not gain an intuition of how/why/whether they work in general. For example, $t\\_{shift}$ is set to recurring block size of residual nets, but what is not given is the justification for this choice, or how to set for non-residual nets. In other words, the hyperparameter settings for these techniques appear ad-hoc (I suspect that it is not), and so it is not clear to me how much exploration is required. What could have helped is to include a study of the incremental benefits of these techniques in the experiment section. In summary, while the results are good, the writing and presentation could be greatly improved to help readers learn and use the proposal.\n`\n**Pros**\n1. Tackles an important problem of reducing time and resource requirements of DNN training.\n2. The general approach of computing layers differently over the course of training is quite intuitive. \n3. Presents two techniques that appear to make localized learning practical and effective for DNN training.  \n\n**Cons**\n1. The proposed techniques are parameterized (e.g., $t\\_{shift}$, $\\alpha$, etc.), but how, and effort required to configure them is not clear. \n2.  The proposed techniques are not sufficiently explained to help build intuition. For example, the weak supervision suggests that reversing the weight update direction can effectively reverse increase in classification loss (i.e., divergence), but it is not clear why this is the case or where this observation applies to SGD and other optimizers.  \n3. Incremental benefits of the techniques is not provided in evaluation. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a new approach by the name of LoCal + SGD (Localized Updates) to replace the traditional Backpropagation method. The key idea is to selectively update some layers’ weights using localized learning rules, so as to reduce the computational complexity of training these layers so as to achieve a better tradeoff between overall speed and accuracy.
The paper received quite mixed reviewers. Some reviewers criticized the incremental nature of the proposed technology, while some other reviewers thought that this is one of the very early papers that demonstrates the practical effectiveness of localized learning. 

The reviewers have made several rounds of discussions, and as a result of that, we think while this direction (localized learning) is very important and promising, this particular paper might not have provided a sufficiently novel and good solution to it.  Specifically, in terms of localized learning, this paper has not proposed brand new concepts or methodologies, instead it adopts existing methods in selective layers. In this sense, it does not really resolve the accuracy issue of localized learning, rather, it achieves the tradeoff by only applying localized learning in some layers. In other words, the current results still heavily rely on BP and has not brought a real breakthrough to localized learning.

Decision: Reject