ActCooLR – High-Level Learning Rate Schedules using Activation Pattern Temperature
Authors: David Hartmann
Sebastian Brodehl
Michael Wand
Venue: NeurIPS 2021 Submitted (2021)
Keywords: deep learning
learning rate schedules
information theory
activation pattern temperature

Abstract: We consider the aspect of learning rate (LR-)scheduling in neural networks, which often significantly affects achievable training time and generalization performance. Although schedules such as 1-cycle offer substantial gains over base-line methods, the effect of LR-curves on the training process is not very well understood. In order to gain more insight into the training process, we combine information theoretic ideas and probabilistic optimization, namely simulated annealing. In more detail, we introduce the activation pattern temperature, which (i) captures changes in the non-linear behavior of ReLU networks and (ii) is free of hyperparameters and thus is more interpretable. Examining the training process, 1-cycle simply yields a linear decrease in temperature, reminiscent of successful cooling strategies in simulated annealing. In order to test a causal connection, we devise ActCooLR, an automatic LR-scheduler that produces declining temperature profiles. In experiments with various CNN architectures and different image classification data sets, we obtain results that perform favorably or exceed the performance of hand-tuned schedules.

ActCooLR High-Level Learning Rate Schedules using A ctiivation Pattern Temperature
Anonymous Author(s)
Affiliation
Address email
Abstract
We consider the aspect of learning rate (LR-)scheduling in neural networks, which often significantly affects achievable training time and generalization performance. Although schedules such as I-cycle offer substantial gains over base-line methods, the effect of LR-curves on the training process is not very well understood. In order gain more insight into the training process, combine information theoretic ideas and probabilistic optimization, namely simulated annealing. In more detail, introduce the activation pattern temperature, which (i) captures changes in the non-linear behavior of ReLU networks and (ii) is free of hyperparameters and thus is more interpretable. Examining the training process, I-cycle simply yields linear decrease in lemperature, reminiscent of successful cooling strategies in simulated annealing. order test causal connection, we devise ActCooLR, an automatic LR- Resschdd that produces declining temperature profiles. In experiments with various CNN architectures and different image classificatinn data sets, we obtain results that perform favorably or exceed the performance of hand-tune schedules.
2
4
6
8
10
11
12
13
14
Introduction
18
Despite the huge success of deep networks, their training dynamics is still ground for many discussions. Above all, the reason for the good performance of such simple algorithm as Stochastic Gradient Descent (SGD) remains an open question. As stated by Bengio []. the learning rate (LR) of SGD is "[t]he single most important hyperparameter and one should always make sure that has been tuned". It is considered to steer the amount of noise that regularizes the optimization [6] (2]. Research spans from practical recommendations. such best practice learning rate schedules of distinct forms
17
18
19
20
21
to theoretical models that unveil the implicit regularization of SGD that depends on the learning rate 2 46). For instance, many training procedures include warm up phase into the learning rate schedules to adapt training to numerical limitations as well as the distinct behavior of the initial training phase compared to the rest of training [14 [5] 36). Recent studies divide the whole training process into phases of distinct characteristics. Nevertheless, the number of regimes phases is still
23
24
25
27
under discussion, most commonly described as two or three phases (13) 29, 31 (2) 39).. ). A broad variety of work introduce sharpness based measures that give mathematical characterizations of the
29 loss landscape promising deeper understanding of the phases, trainability and generalization of deep networks (24] However, these typically either introduce hyperparameters themselves describe
31 only a subspace of optimization directions.
In this paper, we trying to understand the effect of varying learning rates on the training process 33 better. As central tool, we propose new measure of learning progress, activation pattern temper- ature (APT), The key idea is to focus the "hard" part of optimization, which is the fitting of 35 non-linear function. We therefore measure changes not by step-size in parameter space but counting changes in activation patterns, i.e., testing if the decomposition of feature maps into piecewise
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 linear regions changes. Due to its independence of changes to the linear mappings, the measure is, unlike the original LR and other simple differential measures in parameter space, more stable under reparametrization.
38
39
Through the lens of this measure we analyze the training of convolutional neural networks on image classification tasks using several LR schedules (Section 3.2) and find that the commonly used 1-cycle scheduler [45] has very simple behavior, namely an approximately linear decrease during training. It also provides some additional insights into the training dynamic, such as connections between temperature and generalization behavior, and a visualization of phase- bound-reusssiess for different learning rates.
41
42
43
44
45
46
Using the analysis, we present method that adapts the learning rate automatically match a user- specified target temperature profile throughout training. Effective profiles start at high temperature and decrease monotonically until the activation patterns do not change anymore and optimization becomes purely linear. Correspondingly, we name our method ActCooL.R. As our method matches the performance of previously hand-tuned learning rate schedules in our experiments, it could be considered as a candidate for an effective and efficient, hyper-parameter free automatic LR-scheduler. The computational overhead is moderate. with only one additional forward-pass.
47
48
49
50
51
52
53 In summary, our main contributions are (i) the introduction of the activation pattern temperature, which reveals more uniform view of the effect of LR-scheduling training and (ii), based on 55 this, an automatic learning rate scheduler that provides accuracy for short training times in a fully automatic way.
56
57
2
Related Work
58
Driven by the goal to better understand generalization, the training process of deep network training has been analyzed in large body of work. We would structure the background follows:
59
60
Training Phases:
One approach of understanding the training process is to describe it in different phases. An early variant of this idea is the work of Bengio et al. [5], who showed that increasing the intrinsic complexity of data during training can help to improve generalization performance. Several studies identify two training phases: the network trains low- uww-freueeccy features first, yielding features in a second training phase that is more susceptible overfitting 42 26)- Similar observations have been made in studies that also take the effect of the learning rate into account 29 31
61
62
64 low generalization error and continues to learn
66
32.. A more recent study stated
that their "experiments suggest that this [(two training p is not the complete picture" [3). However, there appears to be consensus in literature that the beginning of the training, the activations of network mainly perform a random walk 13 20 More practically, increasing the randomness has been shown to
68
Others show evidence of three instead of two training phases 13 31
69
70
71
even improve performance (see e.g. 44 54).. having dropout as a more prominent example [47]. The other widely accepted fact relates to the end of training, where momentum becomes increasingly important [29] as gradients directions simplify [16) and the loss landscape flattens (2]. Hoffer et al.
72
73
74
20 have shown that this comes from the loss landscape getting smoother the farther the weights travel from initialization. More recently, mode connectivity has been used as tool to check whether a modification in the training process leads to distinct optimization trajectories in the loss landscape
75
76
77
and its found minima [4] 23
78
Mcasures: There are several studies on the correlation between complexity- and norm-based mea- sures 24 In particular, generalization improvements from flatness of the loss landscape has been 80 discussed both affirmatively and negatively 11 466. Nevertheless, sharpness or curvature based
79
81 methods have been utilized to improve generalization in practice [00 [2]. Numerous work have
82 included additional regularization into the training process. The angle between the momentum vector
83 and the local gradient has been utilized to construct a statistical test to determine convergence [28).
84 The value and statistics of the loss have also been used for regularization during training, either by relaxing the softmax loss [37] or by adapting the gradients in order to make constant progress on the loss 43 Lastly. Raghu et al. 41 proposed method to measure the layer-wise complexity of a
85
86
87 network by computing the Singular Value Decomposition of the activations. This method similar in
88 spirit to ours, but it measures the intrinsic dimensions in a stationary fashion, excluding the training
89 process in the measure itself. contrast to competing measures, our measure is hyperparameter-I free and does not depend on the setting it is evaluated in. It avoids the complexity of rescaling due to

---

91
surrounding layers, which plague many continuous measures, by solely focusing on the discrete activations of ReLU network.
92
99
Hyperparameter choice has a strong effect on (generalization) perfor- mance and convergence speed. Economically, under fixed training budgets, this leads to trade-off
Hyperparameter Schedules:
94
95
8 49). In cases of small batch sizes, one crucial invariant control parameter has been shown to be the
96
ratio of learning rate and batch size [15 20 SGD has been shown have an implicit bias towards There is strong empirical evidence that large initial learning rates can help with generalization over over- peer-paaaatteizeed networks
97
flat regions theoretically that is reinforced by high learning rates 2 46
96
99
32]. However, large networks require "warm up" phase to prevent divergence of deeper layers While most schedules let the learning rate approach zero with training time, especially the course of the learning rate in the middle of the training process has not yet been analyzed extensively. Our model suggest here an analogy to annealing schemes in discrete stochastic optimization and provides holistic perspective on the whole training process. LR scheduling is considered to be 29
31
100
101
102
103
104
directly linked to generalization performance 20 22 24
For instance, a cyclic schedule enables to train networks with good "anytime performance" 34 and the implicit learning rate schedules that are built into adaptive optimizers such as Adam 27 are topic of current research
32)
106
I 36. However, specific research in learning rate schedules is sparse. Although, correctly tuned, fewer training iterations [45], "large models in NLP and vision use schedules which can be easily resumed" [30), such "clipped" cosine decay 77] or exponential decay [48]. Recently, research in automatic and adaptive hyperparameter tuning has become more prominent. For instance, automatic tuning of decay time for the exponential decay schedule, [28] 30. and meta-networks designed to predict the learning rate based on learned typical training courses [2]]. more specifically, [ 51) have been tested. Another approach includes the learning rate into the optimization process by deriving the loss w.r.t. the learning rate
108
1 l-cycle achieves the same accuracy using order-of-magnituee
110
112
113
114
115
as well (3]. Lastly, weight decay has been focus of discussion related generalization gap between SGD and adaptive optimizers [35] 49). Also, it has been shown to affect the learning rate scheduling directly when used in conjunction with batch norm: every weight decay step increases the effective
116
117
118
learning rate by multiplicative factor for constant learning rate schedule (33 52). The most similar
119
of the named methods to ours from an optimization perspective probably that of de Roos et al. [10, where successive training steps are used to estimate change of curvature of the loss function to
120
121
adapt the learning rate automatically; however it requires additional hyperparameters and continuous
122
re- re-evaluations of the batch-loss.
123
3
Activation Pattern Temperature (APT)
124
We base our approach on the view that non-linearity is what makes deep networks actually expressive. Throughout this paper, we restrict ourselves to the non-linear aspects of training and study (the popular) ReLU activation function [8 which switches binarily between two linear states. Non- linearity of whole networks is thus encoded in the way the network is switching between those discrete states in an orchestrated way. We call these binary patterns assigned to data "activation patterns". Our idea is very simple: We track the change of activation patterns, by comparing corresponding outputs of ReLU layers for the same input data, and use the leg-log-likelihood of these changes to quantify the "step-size" of the training progress.
125
126
127
128
130
132
3.1
Formal Definition of APT
Formally, we consider feed-forward ReLU network F+ with parameters 0, that contains L activation layers. Further, let (x) E Rd denote the output of such a layer 1 E (1,.... L}. for an input batch Bxdo of batch-size B.
133
134
135
136
We now define the activation pattern for (ReLU) layer I as
(x)x) == (sign (z)) (z) E 10,1)4
(1)
137
which can be seen as bit-vector of ReLU activated neurons.
138
Training: Training is performed discrete steps 0, 1, 2, ... At each step, an input batch x
139
is considered and the optimizer computes new parameters 0, On is determined by the network's initialization, and
140
0 = 0: + de V Lo
(2)
3

---

141 is the update by single optimization step for batch x under loss L (which, we assume, is informed of the ground-trum outputs y(z)).
142
143
In this context, the incremental updates to 0g from randomly drawn batches 22 make the temporal sequence (0o, 0,, ......) Markov chain; the same holds for the sequence of activation patterns
144
MA. (x) over time t. Additionally, we model the effect of one optimization step on the activation
146
patters as a Markov chain,
Network Evaluation
(MB) ...., MM,
(X, ees iz O, + A L
Optlmizer Step
M
M
Mbr
(T) TT)
147
where denotes the random variables that chooses examples from any distribution, describes its true underlying information that are interested in, and Tý the change in activation patterns. Each stochastic process, symbolized by an arrow, optionally adds uncorrelated noise to the mapping.
149
150
Definition: We now define the activation pattern temperature (APT) on layer I as the self-information of the event that an activation pattern has not changed,
151
(x)) == loga (Pr (Mhh (x) My. (z)))
(4)
152
where Pr denotes the probability distribution This estimates the probability of an activation pattern change over the batch r. We also define the average activation pattern temperature Tú as the average of T. over all (RcLU) layers of the network. This measure is parameter-free and specifies the amount of non-linear change in network. Its lower bound is O, stating that no non-linear change occurred and only the linear parts of the network could have been changed during that step. The temperature approaches infinity if all activations have changed during single optimization step.
153
154
155
156
157
158
To measure TV, we run the forward-pass of a network twice, recording the activation patterns in the first pass and comparing and accumulating changes in the second. In experiments, we observe that evaluation on single training batch already give good estimates. Thus, we use the single- batch estimate of APT in the rest of the paper, unless stated otherwise. This allows the measure be calculated at the cost of only one single additional forward pass of the same batch that has been already used for the previous optimization step.
Computation:
159
180
181
162
183
3.2
Training Methods & their Training Dynamics
185
In this section, we utilize the activation pattern temperature (APT) (Equation (4) to gain more insight into the non-linear training dynamics affected by the choice of learning rate schedules. As baseline experiment, we compare the training of ResNet-32 (CIFAR-Variant, (19] trained on CIFAR-10 10 with step decay learning rate schedule [2] We study the effect of three modifications to the ResNet- Architecture: ResNet with removed shortcut connections, FixUp (53) (no batch normalization) and Pyramid-Net [77 (linear growing number of filters). In Figure T we show the learning rate (top row), the corresponding activation temperature of the first layer f (middle row), and the Top-1 1 validation error (bottom row) for the baseline experiment. In the following, we analyze these plots regarding their temperature profiles, training time and generalization performance.
166
167
168
169
170
171
172
173
APT simplified learning rate description: During the training process, APT reflects changes of the LR (APT is reduced, whenever LR is reduced). APT starts strictly greater than zero, the actual the architecture well as the training hyperparameters, and, during the training
175
176
value depends process, APT is reduced and approaches zero. Most importantly, 1-cycle schedule (blue curves), shows approximately linear decrease of APT.
177
178
179
Temperature cools down bottom-to-top: Defined as per-layer temperature, we discuss the APT on per-layer basis in the APPENDIX. We observe that deeper layers have higher temperature (we call these hotter) in comparison to the first layers in a network (we call these cooler). Also, we
181
Note the random variable in the formula does not measure the probability for a neuron change, instead, counts the probability of an activation pattern change. All used hyperparameters are listed in the APPENDIX.

---

1-cycle
1-cycle (No-Shoricuts)
Step Decay
Step Decay (No-Shorteuts)
CyelieLR
FixUp
Pyramid
10a
10
Learning
10-1
Rate
10-2
10 2
10
Validation Setvvctittoon Patter
10"4
10-3
(Top-1) Temperature
10
0.6
0.4
0.2
10
10
10
0.3
10-9.4
10
10 0.7
10 10-0.7
10
10
10
20
30
40
50
0
25
50
75
100
125
150
175
200
Training Epoch
Training Epoch
(a) Cyclic Learning Rates
(b) ResNet Variants
Figure I: Learning Rates, Validation Accuracy & Temperatures of the first ReI.U-Layer in ResNet-32 (and its Variants) on CIFAR-10 for different LR schedules.
182
observe that in each network the activation pattern temperatures decreases (or "cools down"") from bottom to top, in accordance to recent work 41
APT possibly relates to generalization: On the other hand, observe that the validation error has similarities to the APT profile, despite having no direct relation between the AT and the loss. In this setting, APT is proportional the validation error. Architectural adaptions also affect performance, negatively (as ResNet in the absence of shortcut connections), and positively (FixUp and PyramidNet) have that characteristics.
185
186
187
188
189
The temperature can change (decrease) despite a constant learning rate. This can be observed, for example, in all experiments of fig, [b). especially in the beginning of training, but also with every drop of the learning rate. We conclude: for fixed setting, the course of APT is not steered by the magnitude of learning rate alone, and APT contains hidden variables responsible for its course.
APT is not explained by learning rate alone:
191
199
This is in comply with previous work that studied the early phase of training in more detail 13 We think that this view could shed some light on the initial phase of training and possibly explains the requirement of using a warm-up schedule common training schemes beyond numerical instabilities.
185
196
3.3
Learning Rate Range Analysis
198
In the following chapter, we want to study the connection between LR and APT. We do so by re- evaluating the exact same update step with fixed range of learning rates. This allows us observe the training from global LR-invariant perspective. In more detail, we carry out this experiment different data sets, using different LR schedules. First
200
for two different architectures train a ResNet-56 on ImageNet, and second a ResNet-50 on CIFAR- 10. Both networks are trained using step decay (90 rep. 200 epochs) and 1-cycle scheduler (20 resp. 50 epochs) each. For the initialization and every epoch in training, we freeze the network and re-evaluate the temperature for the exact same range of learning rates. The learning rates use are uniformly sampled
202
204
206 a log scale, ranging from 10-4 to 102. The results are shown in Figure 2 The top row shows the average activation pattern temperature T, for each network and for each LR schedule. The white lines indicate the learning rate, which actually has been used for training The theoretical temperature is shown in the background as color gradient from black (cold) yellow (hot). We observe, that after a short initialization phase the temperature has a rather homogeneous behavior
208
209
210
For additional plots measuring for the last layer, or another data set, CIFAR- 10, also for the hyperparameters used, please refer the APPENDIX.
5

---

Imagellet (step
ImageNet (2-cycle)
CIFAR-10
CIFAR-10 (1-ryde)
60.3
0.50
-03
0.83
Redel
-0.3
30
Esorss
Epedhs
Esachs
Fpeths
Figure 2: Learning Rate Range Analysis: For each point in training time of ResNet-50 on ImageNet (left two columns) and ResNet-56 on CIFAR-10 (right two columns) each with 1-cycle LR and step decay schedules, we measure the temperature for a theoretical scaling of the global LR ().
211 over the whole training process. After few initial epochs, we observe only minor changes to the temperature profile. Furthermore, the temperature profiles of ResNet-56 (ImageNet) show phase in which, from a global and LR-independent perspective, the networks cool down slowly, independently of the choice of learning rate. Most notably, 1 1-cycle schedule uses LRs in higher temperature regime (near the vertex) for a longer period of time during training. In contrast, step decay passes through into the colder temperature regime very quickly (ResNet-566/mageNe)) already stats in (ReSNee-50/CIFFR-10)
213
215
217
218
Model of the Activation Pattern Temperature
Typically, weight initialization is based 18 which specifies the initial distribution of the weights such a way that output after the ReLU-activation Gaussian distributed on all layers. As the activation itself zeros the output if the weighted sum of normal distributed variables is smaller than zero, we model the probability of change in activation patterns itself as the cumulative distribution function of normal distribution. Thus, we postulate the following closed form formula.
220
222
224
Hypothesis The probability of an activation pattern change depending on the used learning rate for single update step any layer during training is given by
Pr (Nbyee -Mb) = (1+erf erf (log X - )
(o-22
The real numbers p1, E R depend the training process, choice of architecture and layer.
show next that the model, given by Equation (5). actually fits practice, more detail, for fixed network state, we fit the model using non-linear least squares to the data shown in Figure 2 We evaluate the fit by visualizing the the evaluated model together with its reconstruction error against ground truth in the second and third
227
229
of Figure The estimated parameters and of Equation (5) are shown in the bottom row of Figure 2 From our experience its values depends on the layer the temperature has been measured in, the data set and architecture used and varies also with the random seed of the initialization of weights.
231
233
Most importantly, we could observe favorably only small drifts of the parameter a that defines the width the Gaussian distribution used the model. (But we also show an example of larger changes in the APPENDIX). In contrast, the course of seems to be especially affected by the learning rate used for training. We will use this observation and Equation (5) to define an automatic learning rate
235
237
schedule in Section

---

4
Activation Cooling based Learning Rate Scheduler
239
240
4.1
Optimization using ActCooLR
241
The goal of the scheduler is to determine learning rates that impose a specified temperature profile. This has to be done online, adapting the learning rate À to the current network state. As design decision, we have the option to measure temperatures at every layer, and using adaptive learning rates, even specify them layer-wise. For simplicity we leave fine-grained adaptation for future work and
243
244
245
generally operate with a global learning rate (similar to 1-cycle) and use the mean temperature over
248
all layers for control.
247 In order to determine the learning rate, we simply rearrange Equation (5) from Section 3.4 which describes the probability of a pattern change using only three parameters: 0, j and A:
Jr = 22 orf (2- - 1) log X,
(6)
\= exp (a 22 err (2 Px - 1) 11)
249
where Px denotes the probability given the used learning rate A.
250
Thus, derive the learning rate Ā required to measure the target temperature C, we first estimate j
251
using the measured probability PA using the learning rate X Equation (6). Assuming that the value of
252 μ and only changes slowly (see Section (3.3 we use Equation (7) to estimate the learning rate
253
that would produce the given temperature C. We call this learning rate adaption technique ActCooLR.
254
Too reduce computational costs, we limit measuring and readjustments to once every 10 optimization steps. In between two measurements, we just interpolate linearly between the new and old learning rate. The computational overhead is thus moderate, in particular as only forward pass is needed.
255
256
257 Note, this requires to estimate a in the beginning of training. The value of f can be estimated using the formulas given above. We show in the APPENDIX dynamic version of this technique, that removes the requirement of estimating a and adapting the learning rate dynamically at the cost of additional hyperparameters. A numerical problem arises from temperatures living on a logarithmic
259
260
261
scale (Section 3.4). Due to finite sampling. We might estimate a probability of (all patterns have changed) by chance. According Equation (5) this would correspond an infinite learning rate.
282
263
We remove the singularity by an ad-hoc regularizer: For empirical propabilities of 1, we assume that hallf an activation has changed", but was not measured.
264
4.2
Designing Target Temperature Curves (CIFAR-10)
286
Until now we moved the problem of choosing the learning rate curve with a more abstract problem; choosing the temperature curve. It is clear that want to cool the mean temperature to value of zero, specifying explicitly that the network shall converge. This temperature is trivially given by learning rate of O. We have seen that for sufficienlly many data points it becomes increasingly hard to let the network change all patterns with a single optimization step, in the limit this becomes impossible.
266
270
272
Many previous work has analyzed the positive effects of large initial learning rates (see Section 2 for a discussion). A large initial learning rate corresponds directly to a larger temperature. Thus, we assume that we want to start training with a high temperature, i.e. huge flow of information through the network with every optimization step, and end with very low temperature (mostly only linear regression to be optimized). Empirically, we observed in the Section 3.2 that I-cycle shows a linear decrease in temperature. Thus, we test next if linear decrease in temperature accelerates training by actively controlling the temperature with ActCooLR. As a simple baseline experiment, to test against, we use, again, ResNet-32 on CIFAR-10 and train it using ActCooLR and momentum SGD (see appendix the used Hyperparameters). To test our hypothesis, we define a family of temperature curves of the form
274
276
278
280
cliaaaa (i, hotal, Tsam) = Tstart (1 - (*/1mm=)?)
(8)
282
starting at Timm and cooling down under reduction factor 7 to O.
Figure 3 shows the effect of the selected target temperature curve Clinear on the learning rate (Figure
285
(middle row) and the validation error (Figure 3 (bottom row)). From our experience and also in
285
comply with the observations made in Section 3.2 training generally longer on higher temperatures
(y 1) achieves favorable performance compared to faster reduction of temperature (Y << 1). In
7

---

0.125
0.25
0.50
100
1.50
2.00
2.50
3,00
3.50
4.0
(Target: C-)
130
150
170
190
210
Epochss
Learning Raie Temperaturr Men Mean
Validation Error
10--4
10-2
10
40 50 10 30
50
70
50
110
130
150
170
100
230
Fpochs
Eipochs
(a)
(b)
Figure 3: Training ResNet-32 on CIFAR-10 using a linear target temperature curve and (a) using several values for and (b) for varying number of epochs using linear temperature curve each. We choose the same initial target temperature for all experiments to be 3.17 (i.e. 89% chance that activation has changed), all other hyperparameters, well as seeds, are kept the same in each experiment set. Top row: measured mean temperature, Middle Row: resulting learning rates, bottom row corresponding top-1 validation error.
287
case of a too big y, the amount time optimization takes place in a linear way only becomes short at the cost of worse performance.
289
We test in the following the same setting for varying number epochs to show the stability of our method for various training budgets. For simplicity, restrict the analysis linear temperature decay (7 1). In Figure (bb), we show the disadvantage of our schedule: the validation error remains high for the most number of epochs during training, converging only very late compared methods like cyclic learning rate or step decay (see Section 3.2) For instance, in contrast to a cyclic learning rate schedules, our method does not show good anytime performance. On the one hand, cyclic temperatures could also work, but need to be evaluated separately, thus leaving this as topic of future research. We also show the APPENDIX that our method is independent of the initial learning rate as long as the first few optimization steps do not lead to diverged network. We believe that finding temperature curves with theoretical bounds a promising direction for future work to better understand the internal effects. For the rest of our work, we use linear temperature cool down.
291
292
293
295
296
297
298
4.3
Comparisons with other Methods
Table shows practical results on three different architectures (simple 4-layer CNN, VGG-16 and 301
ResNet-50) on three different data sets MNNSS,, CIFAR-10, ImageNet). We compare to baselines and two automatic LR-schedulerr (ABEL and Autol.RS). The experiments confirm that reach, similar to observations for I-cycle, comparable generalization performance within restricted training budget (Note though that ABEL uses 200 epochs, unlike the other methods), More results are provided in the APPENDIX.
303
305
Discussion
307
The key hypothesis of this paper is that tracking the changes to the nonlinear behavior only can provide with the information needed for step-size control. Our experiments support this view (for image classification with feed-forward ReLU CNNs, which is the class of techniques we restrict our analysis to this point). Specifically, performing simple linear decrease activation pattern temperature already yields L.R I.R-scheduler with performance comparable to 1-cycle, and distorting
309
311
313 the temperature curve towards staying longer in the high-temperature regime at the beginning appears
to improve generalization performance slightly (at least for the training time scales examined). The

---

Table 1: Comparison with previous automatic LR-schedulers
Table 2:
Test errors
for Fig. 3.
Setup
Test error
Epochs
Test Error
Data set
Network
Epochs
Method
Top-1 Error
(Top-1)
Sahhoon-MNIST
4-layer ConvNet 500
200
constant LR
6.95%
10
37.88%
Fashion-MNIST
4ayer ConvNet 50
200
ActCooLR
7.29 %
30
18.33%
CIFAR-10
VGG-16
350
step decay
6,30% 225
50
11.87% %
CIFAR-10
VGG-16 44
350
AutoLRS
5.87% 25
70
10.59%
CIFAR-10
VGG-16 44
200
ABEL
7.1% [30
90
9.09%
CIFAR-I0
VGG-16 44
350
ActCooLR
6.82 %
110
8.09%
130
7.50%
ImageNet
ResNet-50 []
20
1-cycle
27.27%
150
7.28%
ImageNet
ResNet-50 []
20
ActCooLR
27.88%
170
7.18%
190
7.15%
210
6.41 %
315
simple Gaussian two-parameter model of Eq. is approximates the temperature well empirically and leads to simpler and more efficient automatic LR- ---scheduling algorithm than direct optimization of the learning rate. The most important result is probably on the conceptual side: We observe that the rather complex LR-curve of a cyclic (or 1-cycle) scheduler appears to just correspond to an annealing of the APT. This reminiscent of simulated annealing methods which use very similar strategy in order to solve combinatorial optimization problems. The logarithmic temperature measure has an Saai--Chani-ntttt (MCMC) optimizer used there. In this sense, paper reveals that a good LR-scheduler for SGD just performs on the discrete, nonlinear network components process very similar to simulated annealing. More concretely, we would like to point to the results in Fig. 2 which show smooth transition between a linear training regime, with low probability of nonlinear changes and a segh-temperrture, presumably chaotic, regime at the high LRs. By controlling the APT, we can steer training within the band of non-linear but not chaotic learning automatically, and only dive into the purely linear regime the end, thereby plausibly obtaining quicker convergence. The drop in temperature early training steps is handled automatically, and provides a plausible explanation the utility of initial LW-ramp-up in I-cycle. It is also interesting that the phase boundary drifts only slowly after this, with constant width, but depends more strongly on the LR-schedule (and data set) used.
318
317
319
320
321
analogous form to the temperature in the bollzmann-distribttin of a
323
325
327
328
329
331
Limitations and future work:
Our consequential findings are empirical; we do not have an analytical derivation why the training process has the observed properties, or why the proposed temperature several data sets
335
curves reach high performance levels. Our empirical observations are consistent and rather different CNN architectures. Nonetheless, a broader study on a large corpus of models and architectures, well as examining applications beyond image recognition and feed-forward CNNs, is an important next step for future work. Further, a predictive theoretical model of the statistical dynamies of activation patterns under parameter trajectories and exploring a closer connection of SGD and MCMC optimization would be interesting avenues for future work.
335
337
339
Broader Impact
340
341
From an application perspective, our paper aims at improvements in learning time scheduling, such that fast-training schedulers similar to l-cycle can be used with little or no tuning of hyperparameters. To the extend of this being successful, we believe that this has significant positive impact in saving time and (electrical) energy in the development and deployment of deep networks (aside from potential rebound effects applying to any improvement in efficiency). We advise the reader though to use caution in that our key findings are empirical in nature and there no proof of absence of negative effects in terms of costs, and/or a random or systematic loss of accuracy (the same applies to related methods, too). The paper should be received as a step forward towards a more efficient automatic training schedules, not as a proven solution ready for wide deployment. We would correspondingly emphasize the impact of the conceptual view of tracking nonlinear optimization and the newly introduced techniques for implementing this over the practical aspects.
342
343
3/4
345
346
347
348
350

---

References
352
[1] N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates. CoRR, abs/2002.11883, 2020. bttpsttt:://axxi wwwp/b///ggg...oo.////////// httll/gge//arr.....////////0...0 11803
354
355
[2] D. G. T. Barrett and B. Dherin. Implicit gradient regularization. CoRR, abs/2009. 2020. URL https://arxie: wttpaaawww....rr///raaaaraaeae/e
11162
357
[3] A. G. Baydin, R. Cornish, D. Martinez-Rubio, M. Schmidt, and F. Wood. Online learning rate adaptation with hypergradient descent. In 6th International Conference Learning Representations, ICLR 2018, Vancouver. BC, Canada. April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httttt:://pperrrrrrreer
358
380
wwt. seerruunndddsssseee (ddBBrrsAWWAb
361 [4] Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In G. Montavon, G. B. Orr, and K. Müller, editors, Neural Networks: Tricks of the Trade Seccond Edition, volume 7700 of Lecture Notes in Computer Science, pages 437-478. Springer, 2012. doi: 10.1007/988-3-4423333333 26.
362
363
364
URL AtRL/htpss:/// //tp:::://oi.../ org/ 10. 10w.10007997--8 3- 042--6222------- 8__26
385
[5] Y. Bengio, J. Louradour, Collobert, and J. Weston. Curriculum learning. In A. Danyluk, L.. Bottou, and M. L. Littman, editors, Proceedings of the 26th Anmual International Conference on Machine Learning. ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41-48, ACM, 2009, doi: 10. 14/11144//3337.555333333 URL httpp:://dooo org/
366
367
368
369
10. 1 1 1553374. 1553380
370
[6]
L. Bottou. Online learning and stochastic approximations. On-line learning in neural networks, 17(9);142, 1998.
372
[7] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam. G. Sastry,
373
A. Askell, S. Agarwal, A. Herbert- Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler. J. Wu, C. Winter, C. Hesse, M. Chen, F. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NeurIPS seurips.cc/papers /020/
375
S. McCandlish,
376
377
378
2020, December 6-12, 2020, virtual, 2020. Abstsact..rtt..cc... html
380
[8]
D. Choi, J. Shallue, Z. Nado, J. Lee, C. J. Maddison, and G. E. Dahl. On empirical comparisons of optimizers for deep learning. CoRR, abs/1910.05446, 2019. URL http://arxiv aas/ 1910
382
05446
383
[9] C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In D. Schuurmans and M. P. Wellman, editors, Proceedings of the Thirtieth AAA/ Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 1519-1525. AAAI Press, 2016. URL. http://wuu, h/tp:///www.u aaai. org/ oce/ seee/indx....
385
386
[10] F. de Roos, C. Jidling, A. Wills, T. B. Schön, and P. Hennig, A probabilistically motivated learning rate adaptation stochastic optimization. CoRR, abs/2102. 10880, 2021. https:////// http:aawwwwwar..ra/aaoo/oo /bs 2102. 10880
388
389
[11] L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp minima can generalize for deep nets. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1019-1028 PMLR, 2017. URL httpp::/proceed/i/
391
392
393
mlr. wwwppppeessseeee ht/////0//daa dinhh/b htmll
394
[12] P. Foret, A. Kleiner. H. Mobahi, and Neyshabur. Sharpness-aware minimization for efficiently improving generalization. CoRR, abs/2010.01412, 2020. UtLLattps://arxi org/ aba/ 2010. .1412
395
396
[13] J. Frankle, D. Schwab, and A. S. Morcos. The early phase of neural network training. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. http::
397
398
399 [14] A. Gotmare, S. Keskar, C. Xiong, and R. Socher. A closer look deep learning heuristics: Learning rate restarts, warmup and distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. forum?id=r14E0sa.
400
401
net.
403 [15]
P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: training imagenet in hour. CoRR, 0bs/17066002777 2017. URL http://arxiv. www.wwwwwwww...aaeeeeee watps//1000000 02677
404
10

---

408
[16] G. Gur-Ari, D. A. Roberts, and E. Dyer. Gradient descent happens in tiny subspace. CoRR. abs/1812.04755,, 2018. URL httt::://arrii..rg/ab/////////// 04754
408
[17] D. Han. J. Kim, and J. Kim, Deep pyramidal residual networks. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA. July 21-26, 2017, pages 6307-6311 1119//CVPP
410
IEEE Computer Society, 2017. doi: N0.1109/CVPR.2017..666 URL 2017.668
411
412
[18]
K. He, Zhang, Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-lewe performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
413
414
415
[19] K. Hc, X. Zhang, S. Ren, and Sun. Identity mappings in deep residual networks. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision ECCV 2016 - 14th Earopean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. volume 9908 of Lecture Notes in Computer Science, pages 630-645. Springer, 2016. doi: 10.1007/978-33311-4469944 bttps://doi. http:///oo...rg//// wtt../og///0..//000000 197810077777700 3-919-1-9-99 46493-0 0_38
416
417
38. URL
419
[20]
E. Hoffer, I. Hubara, and D. Soudry. Train longer, generalize better: closing the generalization gap in large batch training neural networks.
421
Guyon, U. von Luxburg. Bengio, H. M. Wallach, R. Fergus. S. V. N. Vishwamathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 1731 1731-1776 1741, 2017. URL htttps:/ htwp:/cceednnnnnn.nni..........e cwcurips.cc/paperre 2017//a2h7 a5e0ff62be0b08456fcccmmmm wwt.ee/eee888122ff33 Abstratt. html
422
423
[21] T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey. Meta-learning in neural networks: A survey. CoRR, abs/2004.05439, 2020. httpsttps::// //arxiv/////i/////a www.oog/g////////////oooo.eee/// /2044 05439
428
[22] S. Jastrzebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. J. Storkey Three factors influencing minima in SGD. CoRR, Abs/1711.0042. 2017. URI http:/::/aaww...r////////// wttp/orr///aaa/////eee//// 04623
429
430
[23] S. Jastrzebski, M. Szymczak, S. Fort, D. Arpit. J. Tabor, K. Cho, and K. Geras. The break-even point optimization trajectories of deep neural networks. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreviee net/forum?id-eee (it.orrrrdd--1
432
433
434
[24]
Y. Jiang, B. Neyshabur, H. Mobahi. D. Krishnan, and Bengio. Fantastic generalization measures and where to find them. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htLhhtts:////eeeeeeee net/forum? Id-SJgIPJBFVH
435
436
438
[25] Y. Jin, T. Zhou, I.. Zhao, Y. Zhu, C. Guo, M. Canini, and A. Krishnamurthy (Irs( Automatic learning-rate schedule by bayesian optimization the fly. In International Conference on Learning Representartonss 2021. URL htttp:://operreviez
440
__yju
[26] D. Kalimeris, G. Kaplun, P. Nakkiran, B. I.. Edelman, T. Yang. B. Barak, and H. Zhang. SGD on neural networks learns functions of increasing complexity. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. A'Achhé Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019. December 8-14, 2019, Vancouver, BC, Canada, pages 3491 -501. 2019. URL https:tts:: http:::/pooooedeeeee itw.oedddinnn.euuiiiippeee newrips new..pp.cccpppppppprrrrr.. /aper/ 2019/hash/ http///hhhhb333334433222 http:///22233445999797888 www.::eeeeecccc6cccec663e0ee
443
444
445
Abstract .htmlaaccttttll html
447
(27) D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:://apxxi org/ aba/ /11//1112.66 6980
[28] H. Lang, L. Xiao, and P. Zhang. Using statistics to automate stochastic optimization. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alhé--Bue, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9536 9546, 2019. URL htttps:/prroceddigg weapodiinnsnnuurrppee wwwprrss.ccpppppp/////e/ cd- Abstract.
452
453
454
456
[29] G. Leclerc and A. Madry. The two regimes of deep network training. CoRR, abs/2002.10376. 2020. URL https://arxiv. org/ abs/ 10376
458
[30]
A. Lewkowycz. How to decay your learning rate, CoRR, abs/2103. 2021. URL https:/ //axxiv org/ hee/abb//103... 12682
11

---

480
[31] A. Lewkowycz. Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. CoRR, abs/2003.02218, 2020. htttts::///aratt http///ooo/r//ooooooooo aba/2003 02218
462
463
[32] Y. Li, C. Wei, and T. Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue, F. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neurul Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 1 11669-1 1600, 2019, URL https: htttps://proceedinne wwwpeddinnnnnuuuuppppeeeee nww..neurrcccc...ooo... wwtp:ipp.cpppppe///00/ httpppppppr/////aaaaapa hash/ (tt.:/229911777771187777668888
464
465
486
467
Abstract.htmi
469
[33] 7. li and S. Arora. An expomemtial learning rate schedule for deep learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL BRRLLtttt///pppreerzzi http://wwwwiii.eeeooootet wwtp///oorrmmmddrggeeeeeee..
470
472
(34] 1. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpt:ttttp//eeeeeeeee
474
475
[35] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representatiins,, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL
477
[36] J. and D. Yarats, On the adoguuscyyyyyycuuuuu of 2019. URL http::/ttt//ii..////////a/////tt org/ abaaabaaaaaa 19100 04209
untunee untuned warmup for adaptive optimization. CoRR, ab//1900004009
479
480
[37] R. Müller, S. Kormblith, and G. E. Hinton. When does label smoothing help? In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. B. Fox, and R. Garnett, edi- tots, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 4696-4705, 2019. URL https:://pooceeding wweurip.cc//eee wwcc/paper/2000 ppappe///1///aaapee h0s9/hash/ hbstract..bbtttttt.ccc. html
482
485
494
485
486
[38]
V. Nair and G. E. Hinton. 3d object recognition with deep belief nets. In Y. Bengio, D. Schuur- mans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Pro- cessing Systems 22: 23rd Annual Conference Neural Information Processing Systems 2009. Pro- ceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada, pages 1339- 1347. Curran Associates. Inc., 2009. URL httppp//////occeingg... www.neuuuruc/eepoece hwwrrippscccppppe2000900 / 6e7 baa.eeeeaaaae00eeeeeee66e6633330
487
489
490
491
Abstract.
492
[39]
P. Nakkiran, G. Kaplun, Y. Bansal, Yang, B. Barak, and Sutskever. Deep double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URI. URI. httpt:://ppeereeiieee net/forumiid-B d- g5s44trr
435
434
496
[40] A. Neelakantan, L. Vilnis, Q. V. Le, I. Sutskever, L. Kaiser, K. Kurach, and J. Martens. Adding gradient noise improves learning for very deep networks. CoRR, abs/1511.06800 2015. URL httpttp:/ http://arxiv org/ ///aabb///1111 1atp//11..... 068807
497
499
[41] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. von Luxburg. S. Bengio. H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 6076 6085, 2017. httpltpss//rroceeeiggg. neurips cc/ paper/2011 2207://///////h//// dc6a7
500
502
503
504
7 e655d7 e555e7658505666677777 (555550006633eeeeee.
Baregg--bttratt.mmm html
505
[42]
N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and C. Courville. On the spectral bias of neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5301-5316 PMLR, 2019. URL http://proceeding mlr, press/v9i
506
507
509
vv7/s//v///// 2057//raaaaan880 19. html
510 [43]
M. Rolinek and G. Martius. L4: practical loss-based stepsize adaptation for deep learn- ing. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6434 6444, 2018 URL httpps:/proceeeding neur wwtpccppp/ppcc/./.. 300776688bb0b6768811888eaeeeee htt.://aaaaeeeeeeeaaearrrre....a Abstratt.ttttttt.tt..m..
512
513
514
12

---

516
[44] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Reprexentations, ICLR 2015.
518
San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httLl:t////aiii.oorr
519
abs/ abs/1409. 1009/1499.11 1556
520
[45] I.. N. Smith and N. Topin. Super-convergence: Very fast training of residual networks using large learning rates. CoRR, abs/ 70001708.011000 071.0,7720000 2017. Utt.tttp:://///t http::/ttttpii..//////// wwwp:/i....rg.aoooooooo /
521
622
[46] S. L. Smith, B. Dherin, D. G. T. Barrett, and S. De. On the origin of implicit regularization in stochastic
523
gradient descent. CoRR, abs/2101.12177 2021. URL https: org. wbgg/b////// 2101. 12176
524
[47] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting J. Mach. Learn. Res., 15(1):1929- 2014. URL. http://dl. acm. wwwg/citatioo... cfm22132670313
525
526
[48]
M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105 14, PMLR, 2019. URL /ttp:ttt://prrceeeiii mlr press/ v97/tan19a. 9hmm
528
529
530
532
[49] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In I. Guyon, U. Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, 201///21/////// /hashs/
534
535
536
CA. USA, pages 4148 4158, 2017. URL https://ppooceee wtw.eeeddddnnnn..nn.......
81888362254444755775444778 7tt.////777000077777777776343414 Abstract.html
538
[50] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, abss/700.0777 07747077777778 2017. URL Uttp:tp::// httpttttp://iiiittt/// wtw.v.iiivvvv.......aaa. httg//bb/17000 07747
[51] Z. Xu, A. M. Dai, J. Kemp, and L. Metz. Learning adaptive learning rate schedule. abs/1909.08 0w111999.097222 09719,999222209 2019. URL http:://ttx:xii/ wwwp:/wwwwwwwww//////////// 09711
CoRR,
541
542
[52] G. Zhang, C. Wang, B. Xu, and R. B. Grosse. Three mechanisms of weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL URL /tttps:///pennevveee wwtp://nnneee/eneeeeeeeeeee http://oouumnd-111--333--
543
544
545 [53] H. Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htt.tttttt/////ttttt htttps:://peerreviiet net. /et///orm//////// wor.oooduudueeiiier Hd-Higs230ccK
548
548
[54] Z. Zhu, J. Wu, B. Yu, L. Wu, and J. Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages PMLR, 2019. URL http::t/pproooedinggsss mlr, www.esppess//eee / v97 /tt//e.ulleeeeeeeeee html
550
551
552
Checklist
553
554
For all authors...
555
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
557
(b) Did you describe the limitations of your work? [Yes] This is discussed in Section 5
558
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This is discussed in
Section [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
561
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The assumptions stated with the results.
563
564
(b) Did you include complete proofs all thcoretical results? [Yes] Complete proofs can found in the APPENDIX
565
3. If you experiments...
13

---

587
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material provided under free license.
URL)? [Yes] A reference implementation
569
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)? [Yes] Training details are specified with the experiments.
572
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] Errors bars are reported for our own experiments, where applicable.
573
(d) Did you include the total amount of compute and the type resources used (e.g., type GPUs, internal cluster, or cloud provider)? [Yes] We specify the used compute in the APPENDIX.
575
4. If you using existing assets (e.g.. code, data, models) curating/relessing
assets...
577
(a) If your work uses existing assets, did you cite the creators? [Yes]
578
(h) Did you mention license of the assets? [No] only use freely available data sets,
579
(c) Did you include any new assets either the supplemental material or a URL? [No] No new assets data sets were used.
580
581
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [No] No additional data was used.
582
585
(c) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content? [No] For the data sets we use, this is already discussed in the community.
584
585
5e6
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicahle? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Not applicable.
392
14


Reviews:
[{"review_id": "hSj6Rq2Wlf", "reviewer": "Reviewer_yjMf", "summary": "This papers aims to explore the impact of learning rate schedules in neural network training, through   a newly proposed measure denoted the “activation pattern temperature” which examines the probability that ReLU activations will change between the 0 and linear regions. Thus this measure aims to examine the changes in non-linear behavior of the network. Several learning rate schedules are examined through this lens, and a learning rate schedule based on reducing this temperature is proposed and examined.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of defining and estimating notion of temperature and linking it to the learning rate is interesting, and the text provides several experiments that explore the development of the temperature throughout the training of several networks. I think there are several aspects of the text that would benefit from further insight in order to understand the potential utility of the temperature measure.\n\nHow the APT is computed is not clear from the “computation” paragraph, could the authors provide an algorithm?\n\nExcept the first few epochs, the APT seems to track the LR very closely (as seen in Fig. 1) and the APT distribution seems to change very little (Fig. 2). Are the changes in APT distributions in early training understood or can they be explored more, and can these changes give insights into the early training process? Beyond that early training, APT seems very closely related to LR, what additional insights about the training development can I get from APT over LR? How does the APT relate to the gradient norms? Or how does it affect parameter trajectories?\n\nWhat is the benefit of using an APT scheduler instead of a LR scheduler? It would seem to require the same amount of schedule design, but more computation to eventually determine the LR is needed with APT. How does designing temperature schedule, i.e. a schedule damping of non-linear network changes, make the training process easier, it seems as abstract as an LR schedule?\n\nHow do you know when you are in a chaotic vs non-chaotic non-linear regime, and how can a reader see that controlling APT better navigates these regimes?\n\nTable 1 - why are different method comparisons used for ever experiment? It makes it challenging to compare methods across experiments.\n\nThe authors state that only tracking the nonlinear behavior is enough to provide the information needed for step-size control. It was not clear to me how the conclusion was drawn. Are there toy experiments that could show this?\n\n------------------------------\nAfter Author response:\n\nI believe the authors have provided detailed answers to all of my questions, and I believe these answers are sufficient and provide more clarification for me. I believe that ensuring this information is clearly discussed in the paper would be an important update for the text. I think the paper would benefit from a deeper mathematical / theoretical exploration of the ideas presented, but the current empirical analysis is still of value. I have update my scores accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors clearly discuss limitations and broader impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KL_RWCtJa-", "reviewer": "Reviewer_bPGR", "summary": "The paper attempts to understand the change of activation patterns, meaning the post-ReLu sign flip before and after one parameter update, for the same batch of input data. A measure, activation pattern temperature (APT), is then defined on a layer as the self-information of the event that an activation pattern has not changed. The author uses APT to interpret different training methods and training dynamics. Finally, based on the observation that existing learning rate schedule adheres to a linear decrease of temperature, the author propose ActCooLR, a LR scheduler that was tested on a variety of training tasks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is elegantly written, very engaging throughout. One impression I get after the first read is that it is more about the temperature concept, aka, APT, than the learning rate scheduler ActCooLR. But the title seems to suggest otherwise. One suggestion would be to modify the title to truthfully reflect the content.\n\nPossibly one remark that would be raised by reviewers, and all readers in general, is that the eventually devised LR scheduler doesn't really work -- on all the experiments it didn't show a superiority over other contemporary methods. But I am a strong believer that the merit of a scientific work does not solely depend on the positivity of results. And I do believe the APT method, the relationship between LR and APT, and the empirical analysis provide great insight to the community that's actively trying to uncover the mystery of training dynamics.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I strongly suggest rewriting the title (and maybe the abstract and part of the introduction) to reflect that this paper is more about the notion of APT, and how it can be an investigatory tool to uncover unknown bits of model training, than it is about a brand new, game-changing LR scheduler (too bad it's not). And I encourage the author to further explore APT as a tool to study other aspects of learning, not just LRs.\n\nThe major limitation, as far as I can see, is that ActCooLR doesn't really perform better than other hand designed schedulers. I do not overly discredit the overall scientific value of this work because of that, but it remains a major drawback of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "G_REkDwD0o", "reviewer": "Reviewer_weBR", "summary": "The paper introduces a measure (Activation Pattern Temperature, APT) of the rate of change in the function modelled by a neural network (with ReLU non-linearities) as training progresses. \n\nThis measure is then used to set a learning-rate schedule by regulating the APT, an adaptive learning rate schedule denominated ActCooLR.\n\nThe paper compares the performance achieved by ActCooLR to a standard step-wise decaying learning rate and to a schedule called 1-cycle.  ActCooLR did not obtain better results than the alternatives.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea presented in the paper, measuring the rate of change to a different linear regime (per linear-ReLU layer) is original and interesting. However,  the paper does not yet provide enough theoretical insight or compelling practical methods to warrant  being presented to a wide audience. The method, as presented, is only applicable to networks with piece-wise-linear non-linearities, and the adaptive learning-rate introduced does not achieve better results than the alternatives.\n\nThe paper starts with a good introduction to related work and the problem of hyperparameter scheduling. But the quality of the exposition deteriorates later on in the paper.  For example:\n * The reasons why the APT measure is denominated a temperature is not clearly explained given that it is introduced as a log-probability in equation 4. \n* There are several hints in the paper that the authors will connect this measure to temperatures in simulated annealing but this connection is never explicitly presented.\n* Why is APT calculated per layer, and not for the whole network or per unit? This is not even discussed in the main paper.\n* The sequences of $\\theta$ and $M$ are mentioned to be Markov chains, but this is not necessary for the rest of the exposition. Same when in line 193 the authors mention there are hidden variables responsible for the course of APT, this is rather mysterious and not necessary for the exposition.\n* The figure captions are incomplete and it is necessary to read the main text to understand what is being presented. Sometimes the axis labels used are difficult to understand (e.g. Figure 1 for the validation errors, why use $10^{-0.1} - 10^{-1}$ instead of 0.8 - 0.1? \n* They introduce Hypothesis 1, and then they support it by saying it fits in practice. This gives a veneer of formality that is not really necessary. Simply saying they are fitting the APT vs learning rate function with a logistic would be clearer.\n\nFinally, ActCooLR does not obtain better final performance than the alternatives (which are simpler to implement and more general).\n\nI would like to end encouraging the authors to pursue the ideas further, as they are interesting but the paper is not yet conference-ready. Some of the ideas left for future work in the paper, like using a different learning-rates per layer, may push the final accuracy and make it a practical alternative for applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors haver addressed the limitations and potential negative societal impact adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FbtlUmb0h9r", "reviewer": "Reviewer_gCvn", "summary": "This work contributes a analysis of the dynamics of neural network training using a new quantity called the activation pattern temperature, that measures the amount of (ReLU) neuron activations that switch from 0 to positive, or positive to 0.\n\nThey measure the APT during training of typical CNNs using different learning rate schedules and show that using these schedulers neural networks typically undergo a transition from a high temperature regime to a small temperature one.\n\nThey then design a learning rate scheduler that adapts the learning rate so that the temperature of the optimization dynamics during training follows a predefined curve -- in this case it decreases linearly from 3.17 (why?) to 0.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed the originality and clarity of this paper. There are however a few points that can be improved, and I am not exactly convinced that introducing the APT gives a useful tool to analyze training dynamics, or in other word that this contribution is significant enough for a publication at NeurIPS.\n\n(Major) things that can be clarified\n\n1. What new insight can APT give that other quantities in the literature cannot?\n2. Additionally, how can we use the APT to design better neural networks?\n3. How did you come up with the formula in eq. 5? \n4. line 255, you write \"we interpolate between the new and old learning rate\" => Can you make this statement more precise? At first read it looks that you can anticipate on the future (i.e. the new) optimal values for mu?\n5. Are you using momentum in the experiments in figure 2? If so, how does it affect the actual step size in parameter space after applying momentum?\n6. In Figure 3., you are using momentum. It seems that in this case, the actual APT is not fully determined by the learning rate, but also by the additional momentum term in the update. Can you please discuss how momentum interacts with the APT?\n\n(Minor) questions relating to log scales in plots:\n 1. I question the choice of a log scale in Fig. 1 right for the APT plot. Aren't we looking at the log of a quantity that is already a log (the temperature)?\n 2. On the same figure, I find it a little bit weird to use a log scale for the validation error, whereas I have always seen linear scales in other works.\n 3. Most of the change in accuracy typically occurs during a few training epochs in the beginning, can you maybe reflect that by plotting epochs on a log scale in Figure 2 e.g.?\n 4. In Figure 3, can you use a log scale for the learning rate, as it typically ranges over several orders of magnitude?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper the authors study the rate of sign flips of the RELU units (or more precisely their inputs) during training, and design a learning rate schedule which tries to achieve a linear decrease in this rate over iterations (which was seen empirically for other well-performing LR schedules).

The reviewers found the main idea of studying sign flips for RELUs during optimization to be original and interesting. Their main complaints are that the theoretical explanations for the method are missing or vague, and the experimental results aren't very exciting. Since this method doesn't actually outperform the best LR schedules, its utility comes down to whether it truly offers a simplifying explanation to the problem of tuning LR schedules, and whether this allows it to generalize in non-trivial/surprising ways to new problems. Unfortunately, the reviewers didn't really seem convinced by the explanations, and the experiments were limited to very standard problems that were of the same sort as those used to help design the method in the first place.
Decision: Reject