ActCooLR – High-Level Learning Rate Schedules using Activation Pattern Temperature

We consider the aspect of learning rate ( LR- ) scheduling in neural networks , which1 often significantly affects achievable training time and generalization performance.2 Although schedules such as 1-cycle offer substantial gains over base-line methods,3 the effect of LR-curves on the training process is not very well understood . In order4 to gain more insight into the training process , we combine information theoretic5 ideas and probabilistic optimization , namely simulated annealing . In more detail,6 we introduce the activation pattern temperature , which ( i ) captures changes in the7 non-linear behavior of ReLU networks and ( ii ) is free of hyperparameters and thus8 is more interpretable . Examining the training process , 1-cycle simply yields a linear9 decrease in temperature , reminiscent of successful cooling strategies in simulated10 annealing . In order to test a causal connection , we devise ActCooLR , an automatic11 LR-scheduler that produces declining temperature profiles . In experiments with12 various CNN architectures and different image classification data sets , we obtain13 results that perform favorably or exceed the performance of hand-tuned schedules.14 1 Introduction15 Despite the huge success of deep networks , their training dynamics is still ground for many discussions.16 Above all , the reason for the good performance of such a simple algorithm as Stochastic Gradient17 Descent ( SGD ) remains an open question . As stated by Bengio [ 4 ] , the learning rate ( LR ) of SGD is18 “ [ t ] he single most important hyperparameter and one should always make sure that has been tuned ” .19 It is considered to steer the amount of noise that regularizes the optimization [ 6 ; 22 ] . Research spans20 from practical recommendations , such as best practice learning rate schedules of distinct forms [ 4 ] 21 to theoretical models that unveil the implicit regularization of SGD that depends on the learning22 rate [ 2 ; 46 ] . For instance , many training procedures include a warm up phase into the learning rate23 schedules to adapt training to numerical limitations as well as the distinct behavior of the initial24 training phase compared to the rest of training [ 14 ; 15 ; 36 ] . Recent studies divide the whole training25 process into phases of distinct characteristics . Nevertheless , the number of regimes or phases is still26 under discussion , most commonly described as two or three phases ( [ 13 ; 29 ; 31 ; 32 ; 39 ] ) . A broad27 variety of work introduce sharpness based measures that give mathematical characterizations of the28 loss landscape promising a deeper understanding of the phases , trainability and generalization of deep29 networks [ 24 ] . However , these typically either introduce hyperparameters themselves or describe30 only a subspace of optimization directions.31 In this paper , we are trying to understand the effect of varying learning rates on the training process32 better . As a central tool , we propose a new measure of learning progress , activation pattern temper-33 ature ( APT ) . The key idea is to focus on the “ hard ” part of optimization , which is the fitting of a34 non-linear function . We therefore measure changes not by step-size in parameter space but counting35 changes in activation patterns , i.e. , testing if the decomposition of feature maps into piecewise36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . linear regions changes . Due to its independence of changes to the linear mappings , the measure is,37 unlike the original LR and other simple differential measures in parameter space , more stable under38 reparametrization.39 Through the lens of this measure we analyze the training of convolutional neural networks on image40 classification tasks using several LR schedules ( Section 3.2 ) and find that the commonly used 1-cycle41 scheduler [ 45 ] has a very simple behavior , namely an approximately linear decrease during training.42 It also provides some additional insights into the training dynamic , such as connections between43 temperature and generalization behavior , and a visualization of phase-boundaries for different learning44 rates.45 Using the analysis , we present a method that adapts the learning rate automatically to match a user-46 specified target temperature profile throughout training . Effective profiles start at high temperature47 and decrease monotonically until the activation patterns do not change anymore and optimization48 becomes purely linear . Correspondingly , we name our method ActCooLR . As our method matches49 the performance of previously hand-tuned learning rate schedules in our experiments , it could be50 considered as a candidate for an effective and efficient , hyper-parameter free automatic LR-scheduler.51 The computational overhead is moderate , with only one additional forward-pass.52 In summary , our main contributions are ( i ) the introduction of the activation pattern temperature,53 which reveals a more uniform view of the effect of LR-scheduling on training and ( ii ) , based on54 this , an automatic learning rate scheduler that provides accuracy for short training times in a fully55 automatic way.56 2 Related Work57 Driven by the goal to better understand generalization , the training process of deep network training58 has been analyzed in a large body of work . We would structure the background as follows:59 Training Phases : One approach of understanding the training process is to describe it in different60 phases . An early variant of this idea is the work of Bengio et al . [ 5 ] , who showed that increasing61 the intrinsic complexity of data during training can help to improve generalization performance.62 Several studies identify two training phases : the network trains low-frequency features first , yielding63 low generalization error and continues to learn high-frequency features in a second training phase64 that is more susceptible to overfitting [ 42 ; 26 ] . Similar observations have been made in studies65 that also take the effect of the learning rate into account [ 29 ; 31 ; 32 ] . A more recent study stated66 that their “ experiments suggest that this [ ( two training phases ) ] is not the complete picture ” [ 39 ] .67 Others show evidence of three instead of two training phases [ 13 ; 31 ] . However , there appears to68 be consensus in literature that at the beginning of the training , the activations of a network mainly69 perform a random walk [ 13 ; 20 ] . More practically , increasing the randomness has been shown to70 even improve performance ( see e.g . [ 40 ; 54 ] ) , having dropout as a more prominent example [ 47 ] .71 The other widely accepted fact relates to the end of training , where momentum becomes increasingly72 important [ 29 ] as gradients directions simplify [ 16 ] and the loss landscape flattens [ 2 ] . Hoffer et al.73 [ 20 ] have shown that this comes from the loss landscape getting smoother the farther the weights74 travel from initialization . More recently , mode connectivity has been used as a tool to check whether75 a modification in the training process leads to distinct optimization trajectories in the loss landscape76 and its found minima [ 14 ; 23 ] .77 Measures : There are several studies on the correlation between complexity- and norm-based mea-78 sures [ 24 ] . In particular , generalization improvements from flatness of the loss landscape has been79 discussed both affirmatively and negatively [ 11 ; 46 ] . Nevertheless , sharpness or curvature based80 methods have been utilized to improve generalization in practice [ 10 ; 12 ] . Numerous work have81 included additional regularization into the training process . The angle between the momentum vector82 and the local gradient has been utilized to construct a statistical test to determine convergence [ 28 ] .83 The value and statistics of the loss have also been used for regularization during training , either by84 relaxing the softmax loss [ 37 ] or by adapting the gradients in order to make constant progress on the85 loss [ 43 ] . Lastly , Raghu et al . [ 41 ] proposed a method to measure the layer-wise complexity of a86 network by computing the Singular Value Decomposition of the activations . This method is similar in87 spirit to ours , but it measures the intrinsic dimensions in a stationary fashion , excluding the training88 process in the measure itself . In contrast to competing measures , our measure is hyperparameter-free89 and does not depend on the setting it is evaluated in . It avoids the complexity of rescaling due to90 surrounding layers , which plague many continuous measures , by solely focusing on the discrete91 activations of a ReLU network.92 Hyperparameter Schedules : Hyperparameter choice has a strong effect on ( generalization ) perfor-93 mance and convergence speed . Economically , under fixed training budgets , this leads to a trade-off94 [ 8 ; 49 ] . In cases of small batch sizes , one crucial invariant control parameter has been shown to be the95 ratio of learning rate and batch size [ 15 ; 20 ] . SGD has been shown to have an implicit bias towards96 flat regions theoretically that is reinforced by high learning rates [ 2 ; 46 ] . There is strong empirical97 evidence that large initial learning rates can help with generalization in over-parameterized networks98 [ 31 ; 32 ] . However , large networks require a “ warm up ” phase to prevent divergence of deeper layers99 [ 14 ] . While most schedules let the learning rate approach zero with training time , especially the100 course of the learning rate in the middle of the training process has not yet been analyzed extensively.101 Our model suggest here an analogy to annealing schemes in discrete stochastic optimization and102 provides a holistic perspective on the whole training process . LR scheduling is considered to be103 directly linked to generalization performance [ 20 ; 22 ; 24 ; 29 ; 32 ] . For instance , a cyclic schedule104 enables to train networks with a good “ anytime performance ” [ 34 ] and the implicit learning rate105 schedules that are built into adaptive optimizers such as Adam [ 27 ] are topic of current research106 [ 1 ; 36 ] . However , specific research in learning rate schedules is sparse . Although , correctly tuned,107 1-cycle achieves the same accuracy using order-of-magnitude fewer training iterations [ 45 ] , “ large108 models in NLP and vision use schedules which can be easily resumed ” [ 30 ] , such as “ clipped ” cosine109 decay [ 7 ] or exponential decay [ 48 ] . Recently , research in automatic and adaptive hyperparameter110 tuning has become more prominent . For instance , automatic tuning of decay time for the exponential111 decay schedule , [ 28 ; 30 ] , and meta-networks designed to predict the learning rate based on learned112 typical training courses ( [ 21 ] , or more specifically , [ 9 ; 51 ] ) have been tested . Another approach113 includes the learning rate into the optimization process by deriving the loss w.r.t . the learning rate114 as well [ 3 ] . Lastly , weight decay has been focus of discussion related generalization gap between115 SGD and adaptive optimizers [ 35 ; 49 ] . Also , it has been shown to affect the learning rate scheduling116 directly when used in conjunction with batch norm : every weight decay step increases the effective117 learning rate by a multiplicative factor for a constant learning rate schedule [ 33 ; 52 ] . The most similar118 of the named methods to ours from an optimization perspective is probably that of de Roos et al.119 [ 10 ] , where successive training steps are used to estimate change of curvature of the loss function to120 adapt the learning rate automatically ; however it requires additional hyperparameters and continuous121 re-evaluations of the batch-loss.122 3 Activation Pattern Temperature ( APT ) 123 We base our approach on the view that non-linearity is what makes deep networks actually expressive.124 Throughout this paper , we restrict ourselves to the non-linear aspects of training and study ( the125 popular ) ReLU activation function [ 38 ] , which switches binarily between two linear states . Non-126 linearity of whole networks is thus encoded in the way the network is switching between those discrete127 states in an orchestrated way . We call these binary patterns assigned to data “ activation patterns ” .128 Our idea is very simple : We track the change of activation patterns , by comparing corresponding129 outputs of ReLU layers for the same input data , and use the neg-log-likelihood of these changes to130 quantify the “ step-size ” of the training progress.131 3.1 Formal Definition of APT132 Formally , we consider a feed-forward ReLU network FLθ with parameters θ , that contains L activation133 layers . Further , let f l ( x ) ∈ Rdl denote the output of such a layer l ∈ { 1 , ... , L } , for an input batch134 x ∈ RB×d0 of batch-size B.135 We now define the activation pattern for ( ReLU ) layer l as136 M lθ ( x ) : = ( sign ( f l ( x ) ) ) ( x ) ∈ { 0 , 1 } dl , ( 1 ) which can be seen as a bit-vector of ReLU activated neurons.137 Training : Training is performed in discrete steps t = 0 , 1 , 2 , ... At each step , an input batch x138 is considered and the optimizer computes new parameters θt . θ0 is determined by the network ’ s139 initialization , and140 θt+1 = θt + λt∇θtL ( FLθt , xt ) ( 2 ) is the update by a single optimization step for batch x under loss L ( which , we assume , is informed141 of the ground-truth outputs y ( x ) ) .142 In this context , the incremental updates to θt from randomly drawn batches xt make the temporal143 sequence ( θ0 , θ1 , θ2 , . . . ) a Markov chain ; the same holds for the sequence of activation patterns144 M lθt ( x ) over time t. Additionally , we model the effect of one optimization step on the activation145 patters as a Markov chain,146 ( X , Y ) ( M1θt , . . . , M L θt ) ( M1θt+1 , . . . , M L θt+1 ) Network Evaluation Optimizer Step θt+1 : = θt + λ∇ θt Lθt ( X , Y ) ( T 1t , . . . , T L t ) ( 3 ) where X denotes the random variables that chooses examples from any distribution , Y describes its147 true underlying information that we are interested in , and T lt the change in activation patterns . Each148 stochastic process , symbolized by an arrow , optionally adds uncorrelated noise to the mapping.149 Definition : We now define the activation pattern temperature ( APT ) on layer l as the self-information150 of the event that an activation pattern has not changed,151 T lt ( x ) : = − log2 ( Pr ( M lθt+1 ( x ) =M l θt ( x ) ) ) , ( 4 ) where Pr denotes the probability distribution.1 This estimates the probability of an activation pattern152 change over the batch x . We also define the average activation pattern temperature Tt as the average153 of T lt over all ( ReLU ) layers of the network . This measure is parameter-free and specifies the amount154 of non-linear change in a network . Its lower bound is 0 , stating that no non-linear change occurred155 and only the linear parts of the network could have been changed during that step . The temperature156 approaches infinity if all activations have changed during a single optimization step.157 Computation : To measure T lt , we run the forward-pass of a network twice , recording the activation158 patterns in the first pass and comparing and accumulating changes in the second . In experiments,159 we observe that evaluation on a single training batch already give good estimates . Thus , we use the160 single-batch estimate of APT in the rest of the paper , unless stated otherwise . This allows the measure161 to be calculated at the cost of only one single additional forward pass of the same batch that has been162 already used for the previous optimization step.163 3.2 Training Methods & their Training Dynamics164 In this section , we utilize the activation pattern temperature ( APT ) ( Equation ( 4 ) ) to gain more insight165 into the non-linear training dynamics affected by the choice of learning rate schedules . As a baseline166 experiment , we compare the training of ResNet-32 ( CIFAR-Variant , [ 19 ] ) trained on CIFAR-10167 with step decay learning rate schedule.2 We study the effect of three modifications to the ResNet-168 Architecture : ResNet with removed shortcut connections , FixUp [ 53 ] ( no batch normalization ) and169 Pyramid-Net [ 17 ] ( linear growing number of filters ) . In Figure 1 , we show the learning rate ( top row ) ,170 the corresponding activation temperature of the first layer f1 ( middle row ) , and the Top-1 validation171 error ( bottom row ) for the baseline experiment . In the following , we analyze these plots regarding172 their temperature profiles , training time and generalization performance.173 APT simplified learning rate description : During the training process , APT reflects changes of174 the LR ( APT is reduced , whenever LR is reduced ) . APT starts strictly greater than zero , the actual175 value depends on the architecture as well as the training hyperparameters , and , during the training176 process , APT is reduced and approaches zero . Most importantly , 1-cycle schedule ( blue curves ) ,177 shows approximately a linear decrease of APT.178 Temperature cools down bottom-to-top : Defined as a per-layer temperature , we discuss the APT179 on a per-layer basis in the APPENDIX . We observe that deeper layers have a higher temperature180 ( we call these hotter ) in comparison to the first layers in a network ( we call these cooler ) . Also , we181 1Note the random variable in the formula does not measure the probability for a neuron change , instead , it counts the probability of an activation pattern change . 2All used hyperparameters are listed in the APPENDIX . observe that in each network the activation pattern temperatures decreases ( or “ cools down ” ) from182 bottom to top , in accordance to recent work [ 41 ] .183 APT possibly relates to generalization : On the other hand , we observe that the validation error has184 similarities to the APT profile , despite having no direct relation between the AT and the loss . In this185 setting , APT is proportional to the validation error . Architectural adaptions also affect performance,186 negatively ( as ResNet in the absence of shortcut connections ) , and positively ( FixUp and PyramidNet ) 187 have that characteristics.188 APT is not explained by learning rate alone : The temperature can change ( decrease ) despite a189 constant learning rate . This can be observed , for example , in all experiments of fig . 1 ( b ) , especially190 in the beginning of training , but also with every drop of the learning rate . We conclude : for a fixed191 setting , the course of APT is not steered by the magnitude of learning rate alone , and APT contains192 hidden variables responsible for its course.193 This is in comply with previous work that studied the early phase of training in more detail [ 13 ] . We194 think that this view could shed some light on the initial phase of training and possibly explains the195 requirement of using a warm-up schedule in common training schemes beyond numerical instabilities.196 3.3 Learning Rate Range Analysis197 In the following chapter , we want to study the connection between LR and APT . We do so by re-198 evaluating the exact same update step with a fixed range of learning rates . This allows us to observe199 the training from a global LR-invariant perspective . In more detail , we carry out this experiment200 for two different architectures on different data sets , using two different LR schedules . First we201 train a ResNet-56 on ImageNet , and second a ResNet-50 on CIFAR-10 . Both networks are trained202 using step decay ( 90 rep. 200 epochs ) and 1-cycle scheduler ( 20 resp . 50 epochs ) each . For the203 initialization and every epoch in training , we freeze the network and re-evaluate the temperature204 for the exact same range of learning rates . The learning rates we use are uniformly sampled on205 a log scale , ranging from 10−4 to 102 . The results are shown in Figure 2 : The top row shows206 the average activation pattern temperature Tt for each network and for each LR schedule . The207 white lines indicate the learning rate , which actually has been used for training.3 The theoretical208 temperature is shown in the background as color gradient from black ( cold ) to yellow ( hot ) . We209 observe , that after a short initialization phase the temperature has a rather homogeneous behavior210 3For additional plots measuring for the last layer , or another data set , CIFAR-10 , also for the hyperparameters used , please refer to the APPENDIX . over the whole training process . After few initial epochs , we observe only minor changes to the211 temperature profile . Furthermore , the temperature profiles of ResNet-56 ( ImageNet ) show a phase in212 which , from a global and LR-independent perspective , the networks cool down slowly , independently213 of the choice of learning rate . Most notably , 1-cycle schedule uses LRs in a higher temperature214 regime ( near the vertex ) for a longer period of time during training . In contrast , step decay passes215 through into the colder temperature regime very quickly ( ResNet-56/ImageNet ) or already stats in it216 ( ResNet-50/CIFAR-10 ) .217 3.4 Model of the Activation Pattern Temperature218 Typically , weight initialization is based on [ 18 ] , which specifies the initial distribution of the weights219 in such a way that the output after the ReLU-activation is Gaussian distributed on all layers . As the220 activation itself zeros the output if the weighted sum of normal distributed variables is smaller than221 zero , we model the probability of a change in activation patterns itself as the cumulative distribution222 function of a normal distribution . Thus , we postulate the following closed form formula.223 Hypothesis 1 . The probability of an activation pattern change depending on the used learning rate224 λ for a single update step on any layer during training is given by225 Pr ( M lθt+λ·∇L =M l θt ) = 1 2 · ( 1 + erf ( log λ− µ σ · √ 2 ) ) . ( 5 ) The real numbers µ , σ ∈ R depend on the training process , choice of architecture and layer.226 We show next that the model , given by Equation ( 5 ) , actually fits in practice . In more detail , for a227 fixed network state , we fit the model using non-linear least squares to the data shown in Figure 2.228 We evaluate the fit by visualizing the the evaluated model together with its reconstruction error229 against ground truth in the second and third row of Figure 2 . The estimated parameters µ and σ of230 Equation ( 5 ) are shown in the bottom row of Figure 2 . From our experience its values depends on the231 layer the temperature has been measured in , the data set and architecture used and varies also with232 the random seed of the initialization of weights.233 Most importantly , we could observe favorably only small drifts of the parameter σ that defines the234 width of the Gaussian distribution used in the model . ( But we also show an example of larger changes235 in the APPENDIX ) . In contrast , the course of µ seems to be especially affected by the learning rate236 used for training . We will use this observation and Equation ( 5 ) to define an automatic learning rate237 schedule in Section 4.238 4 Activation Cooling based Learning Rate Scheduler239 4.1 Optimization using ActCooLR240 The goal of the scheduler is to determine learning rates that impose a specified temperature profile.241 This has to be done online , adapting the learning rate λ to the current network state . As a design242 decision , we have the option to measure temperatures at every layer , and using adaptive learning rates,243 even to specify them layer-wise . For simplicity we leave fine-grained adaptation for future work and244 generally operate with a global learning rate ( similar to 1-cycle ) and use the mean temperature over245 all layers for control.246 In order to determine the learning rate , we simply rearrange Equation ( 5 ) from Section 3.4 , which247 describes the probability of a pattern change using only three parameters : σ , µ and λ:248 µ = σ · √ 2 · erf−1 ( 2 · Pλ − 1 ) − log λ , ( 6 ) λ = exp ( σ · √ 2 · erf−1 ( 2 · Pλ − 1 ) − µ ) , ( 7 ) where Pλ denotes the probability given the used learning rate λ.249 Thus , to derive the learning rate λ required to measure the target temperature C , we first estimate µ250 using the measured probability Pλ using the learning rate λ Equation ( 6 ) . Assuming that the value of251 µ and σ only changes slowly ( see Section 3.3 ) we can use Equation ( 7 ) to estimate the learning rate252 that would produce the given temperature C. We call this learning rate adaption technique ActCooLR.253 Too reduce computational costs , we limit measuring and readjustments to once every 10 optimization254 steps . In between two measurements , we just interpolate linearly between the new and old learning255 rate . The computational overhead is thus moderate , in particular as only a forward pass is needed.256 Note , this requires to estimate σ in the beginning of training . The value of µ can be estimated using257 the formulas given above . We show in the APPENDIX a dynamic version of this technique , that258 removes the requirement of estimating σ and adapting the learning rate dynamically at the cost of259 additional hyperparameters . A numerical problem arises from temperatures living on a logarithmic260 scale ( Section 3.4 ) . Due to finite sampling , We might estimate a probability of 1 ( all patterns have261 changed ) by chance . According to Equation ( 5 ) this would correspond to an infinite learning rate.262 We remove the singularity by an ad-hoc regularizer : For empirical propabilities of 1 , we assume that263 “ half an activation has changed ” , but was not measured.264 4.2 Designing Target Temperature Curves ( CIFAR-10 ) 265 Until now we moved the problem of choosing the learning rate curve with a more abstract problem ; 266 choosing the temperature curve . It is clear that we want to cool the mean temperature to a value267 of zero , specifying explicitly that the network shall converge . This temperature is trivially given268 by a learning rate of 0 . We have seen that for sufficiently many data points it becomes increasingly269 hard to let the network change all patterns with a single optimization step , in the limit this becomes270 impossible.271 Many previous work has analyzed the positive effects of large initial learning rates ( see Section 2272 for a discussion ) . A large initial learning rate corresponds directly to a larger temperature . Thus , we273 assume that we want to start training with a high temperature , i.e . a huge flow of information through274 the network with every optimization step , and end with a very low temperature ( mostly only linear275 regression to be optimized ) . Empirically , we observed in the Section 3.2 that 1-cycle shows a linear276 decrease in temperature . Thus , we test next if a linear decrease in temperature accelerates training277 by actively controlling the temperature with ActCooLR . As a simple baseline experiment , to test278 against , we use , again , a ResNet-32 on CIFAR-10 and train it using ActCooLR and momentum SGD279 ( see appendix the used Hyperparameters ) . To test our hypothesis , we define a family of temperature280 curves of the form281 C linearγ ( i , itotal , Tstart ) : = Tstart · ( 1− ( i/itotal ) γ ) , ( 8 ) starting at Tstart and cooling down under a reduction factor γ to 0.282 Figure 3 shows the effect of the selected target temperature curve C linearγ on the learning rate ( Figure 3283 ( middle row ) and the validation error ( Figure 3 ( bottom row ) ) . From our experience and also in284 comply with the observations made in Section 3.2 , training generally longer on higher temperatures285 ( γ > 1 ) achieves favorable performance compared to a faster reduction of temperature ( γ < 1 ) . In286 case of a too big γ , the amount of time optimization takes place in a linear way only becomes short at287 the cost of a worse performance.288 We test in the following the same setting for varying number of epochs to show the stability of our289 method for various training budgets . For simplicity , we restrict the analysis to a linear temperature290 decay ( γ = 1 ) . In Figure 3 ( b ) , we show the disadvantage of our schedule : the validation error remains291 high for the most number of epochs during training , converging only very late compared to methods292 like cyclic learning rate or step decay ( see Section 3.2 ) . For instance , in contrast to a cyclic learning293 rate schedules , our method does not show a good anytime performance . On the one hand , cyclic294 temperatures could also work , but need to be evaluated separately , thus leaving this as topic of future295 research . We also show in the APPENDIX that our method is independent of the initial learning296 rate as long as the first few optimization steps do not lead to a diverged network . We believe that297 finding temperature curves with theoretical bounds is a promising direction for future work to better298 understand the internal effects . For the rest of our work , we use a linear temperature cool down.299 4.3 Comparisons with other Methods300 Table 1 shows practical results on three different architectures ( simple 4-layer CNN , VGG-16 and301 ResNet-50 ) on three different data sets ( Fashion-MNIST , CIFAR-10 , ImageNet ) . We compare to302 baselines and two automatic LR-schedulers ( ABEL and AutoLRS ) . The experiments confirm that303 we reach , similar to our observations for 1-cycle , comparable generalization performance within a304 restricted training budget ( Note though that ABEL uses 200 epochs , unlike the other methods ) . More305 results are provided in the APPENDIX.306 5 Discussion307 The key hypothesis of this paper is that tracking the changes to the nonlinear behavior only can308 provide us with the information needed for step-size control . Our experiments support this view ( for309 image classification with feed-forward ReLU CNNs , which is the class of techniques we restrict310 our analysis to at this point ) . Specifically , performing a simple linear decrease in activation pattern311 temperature already yields a LR-scheduler with performance comparable to 1-cycle , and distorting312 the temperature curve towards staying longer in the high-temperature regime at the beginning appears313 to improve generalization performance slightly ( at least for the training time scales examined ) .The314 Table 2 : Test errors for Fig . 3 . Epochs Test Error ( Top-1 ) 10 37.88 % 30 18.33 % 50 11.87 % 70 10.59 % 90 9.09 % 110 8.09 % 130 7.50 % 150 7.28 % 170 7.18 % 190 7.15 % 210 6.41 % simple Gaussian two-parameter model of Eq . 5 is approximates the temperature well empirically and315 leads to a simpler and more efficient automatic LR-scheduling algorithm than a direct optimization of316 the learning rate . The most important result is probably on the conceptual side : We observe that the317 rather complex LR-curve of a cyclic ( or 1-cycle ) scheduler appears to just correspond to an annealing318 of the APT . This is reminiscent of simulated annealing methods which use a very similar strategy in319 order to solve combinatorial optimization problems . The logarithmic temperature measure has an320 analogous form to the temperature in the Boltzmann-distribution of a Markov-Chain-Monte-Carlo321 ( MCMC ) optimizer used there . In this sense , our paper reveals that a good LR-scheduler for SGD just322 performs on the discrete , nonlinear network components a process very similar to simulated annealing.323 More concretely , we would like to point to the results in Fig . 2 which show a smooth transition324 between a linear training regime , with low probability of nonlinear changes and a high-temperature,325 presumably chaotic , regime at the high LRs . By controlling the APT , we can steer training within326 the band of non-linear , but not chaotic learning automatically , and only dive into the purely linear327 regime at the end , thereby plausibly obtaining a quicker convergence . The drop in temperature at328 early training steps is handled automatically , and provides a plausible explanation the utility of initial329 LW-ramp-up in 1-cycle . It is also interesting that the phase boundary drifts only slowly after this,330 with constant width , but depends more strongly on the LR-schedule ( and data set ) used.331 Limitations and future work : Our consequential findings are empirical ; we do not have an analytical332 derivation of why the training process has the observed properties , or why the proposed temperature333 curves reach high performance levels . Our empirical observations are consistent over several data sets334 and rather different CNN architectures . Nonetheless , a broader study on a large corpus of models and335 architectures , as well as examining applications beyond image recognition and feed-forward CNNs,336 is an important next step for future work . Further , a predictive theoretical model of the statistical337 dynamics of activation patterns under parameter trajectories and exploring a closer connection of338 SGD and MCMC optimization would be interesting avenues for future work.339 Broader Impact340 From an application perspective , our paper aims at improvements in learning time scheduling , such341 that fast-training schedulers similar to 1-cycle can be used with little or no tuning of hyperparameters.342 To the extend of this being successful , we believe that this has a significant positive impact in343 saving time and ( electrical ) energy in the development and deployment of deep networks ( aside344 from potential rebound effects applying to any improvement in efficiency ) . We advise the reader345 though to use caution in that our key findings are empirical in nature and there is no proof of absence346 of negative effects in terms of costs , and/or a random or systematic loss of accuracy ( the same347 applies to related methods , too ) . The paper should be received as a step forward towards a more348 efficient automatic training schedules , not as a proven solution ready for wide deployment . We would349 correspondingly emphasize the impact of the conceptual view of tracking nonlinear optimization and350 the newly introduced techniques for implementing this over the practical aspects.351 References352 [ 1 ] N. Agarwal , R. Anil , E. Hazan , T. Koren , and C. Zhang . Disentangling adaptive gradient methods from353 learning rates . CoRR , abs/2002.11803 , 2020 . URL https : //arxiv.org/abs/2002.11803.354 [ 2 ] D. G. T. Barrett and B. Dherin . Implicit gradient regularization . CoRR , abs/2009.11162 , 2020 . URL355 https : //arxiv.org/abs/2009.11162.356 [ 3 ] A. G. Baydin , R. Cornish , D. Martínez-Rubio , M. Schmidt , and F. Wood . Online learning rate adaptation357 with hypergradient descent . In 6th International Conference on Learning Representations , ICLR 2018,358 Vancouver , BC , Canada , April 30 - May 3 , 2018 , Conference Track Proceedings . OpenReview.net , 2018.359 URL https : //openreview.net/forum ? id=BkrsAzWAb.360 [ 4 ] Y. Bengio . Practical recommendations for gradient-based training of deep architectures . In G. Montavon,361 G. B. Orr , and K. Müller , editors , Neural Networks : Tricks of the Trade - Second Edition , volume 7700 of362 Lecture Notes in Computer Science , pages 437–478 . Springer , 2012. doi : 10.1007/978-3-642-35289-8\_26.363 URL https : //doi.org/10.1007/978-3-642-35289-8_26.364 [ 5 ] Y. Bengio , J. Louradour , R. Collobert , and J. Weston . Curriculum learning . In A. P. Danyluk , L. Bottou,365 and M. L. Littman , editors , Proceedings of the 26th Annual International Conference on Machine Learning,366 ICML 2009 , Montreal , Quebec , Canada , June 14-18 , 2009 , volume 382 of ACM International Conference367 Proceeding Series , pages 41–48 . ACM , 2009. doi : 10.1145/1553374.1553380 . URL https : //doi.org/368 10.1145/1553374.1553380.369 [ 6 ] L. Bottou . Online learning and stochastic approximations . On-line learning in neural networks , 17 ( 9 ) :142,370 1998.371 [ 7 ] T. B . Brown , B. Mann , N. Ryder , M. Subbiah , J. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry,372 A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger , T. Henighan , R. Child , A. Ramesh , D. M. Ziegler,373 J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler , M. Litwin , S. Gray , B . Chess , J. Clark , C. Berner,374 S. McCandlish , A. Radford , I. Sutskever , and D. Amodei . Language models are few-shot learners . In375 H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , and H. Lin , editors , Advances in Neural Information376 Processing Systems 33 : Annual Conference on Neural Information Processing Systems 2020 , NeurIPS377 2020 , December 6-12 , 2020 , virtual , 2020 . URL https : //proceedings.neurips.cc/paper/2020/378 hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.379 [ 8 ] D. Choi , C. J. Shallue , Z. Nado , J. Lee , C. J. Maddison , and G. E. Dahl . On empirical comparisons380 of optimizers for deep learning . CoRR , abs/1910.05446 , 2019 . URL http : //arxiv.org/abs/1910.381 05446.382 [ 9 ] C. Daniel , J. Taylor , and S. Nowozin . Learning step size controllers for robust neural network training . In383 D. Schuurmans and M. P. Wellman , editors , Proceedings of the Thirtieth AAAI Conference on Artificial384 Intelligence , February 12-17 , 2016 , Phoenix , Arizona , USA , pages 1519–1525 . AAAI Press , 2016 . URL385 http : //www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/11763.386 [ 10 ] F. de Roos , C. Jidling , A. Wills , T. B. Schön , and P. Hennig . A probabilistically motivated learning rate387 adaptation for stochastic optimization . CoRR , abs/2102.10880 , 2021 . URL https : //arxiv.org/abs/388 2102.10880.389 [ 11 ] L. Dinh , R. Pascanu , S. Bengio , and Y. Bengio . Sharp minima can generalize for deep nets . In D. Precup390 and Y. W. Teh , editors , Proceedings of the 34th International Conference on Machine Learning , ICML391 2017 , Sydney , NSW , Australia , 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research,392 pages 1019–1028 . PMLR , 2017 . URL http : //proceedings.mlr.press/v70/dinh17b.html.393 [ 12 ] P. Foret , A. Kleiner , H. Mobahi , and B. Neyshabur . Sharpness-aware minimization for efficiently improving394 generalization . CoRR , abs/2010.01412 , 2020 . URL https : //arxiv.org/abs/2010.01412.395 [ 13 ] J. Frankle , D. J. Schwab , and A. S. Morcos . The early phase of neural network training . In 8th Interna-396 tional Conference on Learning Representations , ICLR 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020.397 OpenReview.net , 2020 . URL https : //openreview.net/forum ? id=Hkl1iRNFwS.398 [ 14 ] A. Gotmare , N. S. Keskar , C. Xiong , and R. Socher . A closer look at deep learning heuristics : Learning399 rate restarts , warmup and distillation . In 7th International Conference on Learning Representations , ICLR400 2019 , New Orleans , LA , USA , May 6-9 , 2019 . OpenReview.net , 2019 . URL https : //openreview.net/401 forum ? id=r14EOsCqKX.402 [ 15 ] P. Goyal , P. Dollár , R. B. Girshick , P. Noordhuis , L. Wesolowski , A. Kyrola , A. Tulloch , Y. Jia , and403 K. He . Accurate , large minibatch SGD : training imagenet in 1 hour . CoRR , abs/1706.02677 , 2017 . URL404 http : //arxiv.org/abs/1706.02677.405 [ 16 ] G. Gur-Ari , D. A. Roberts , and E. Dyer . Gradient descent happens in a tiny subspace . CoRR,406 abs/1812.04754 , 2018 . URL http : //arxiv.org/abs/1812.04754.407 [ 17 ] D. Han , J. Kim , and J. Kim . Deep pyramidal residual networks . In 2017 IEEE Conference on Computer408 Vision and Pattern Recognition , CVPR 2017 , Honolulu , HI , USA , July 21-26 , 2017 , pages 6307–6315.409 IEEE Computer Society , 2017. doi : 10.1109/CVPR.2017.668 . URL https : //doi.org/10.1109/CVPR.410 2017.668.411 [ 18 ] K. He , X. Zhang , S. Ren , and J . Sun . Delving deep into rectifiers : Surpassing human-level performance on412 imagenet classification . In Proceedings of the IEEE international conference on computer vision , pages413 1026–1034 , 2015.414 [ 19 ] K. He , X. Zhang , S. Ren , and J . Sun . Identity mappings in deep residual networks . In B. Leibe,415 J. Matas , N. Sebe , and M. Welling , editors , Computer Vision - ECCV 2016 - 14th European Conference,416 Amsterdam , The Netherlands , October 11-14 , 2016 , Proceedings , Part IV , volume 9908 of Lecture Notes417 in Computer Science , pages 630–645 . Springer , 2016. doi : 10.1007/978-3-319-46493-0\_38 . URL418 https : //doi.org/10.1007/978-3-319-46493-0_38.419 [ 20 ] E. Hoffer , I. Hubara , and D. Soudry . Train longer , generalize better : closing the generalization gap in large420 batch training of neural networks . In I. Guyon , U. von Luxburg , S. Bengio , H. M. Wallach , R. Fergus,421 S. V. N. Vishwanathan , and R. Garnett , editors , Advances in Neural Information Processing Systems 30:422 Annual Conference on Neural Information Processing Systems 2017 , December 4-9 , 2017 , Long Beach,423 CA , USA , pages 1731–1741 , 2017 . URL https : //proceedings.neurips.cc/paper/2017/hash/424 a5e0ff62be0b08456fc7f1e88812af3d-Abstract.html.425 [ 21 ] T. M. Hospedales , A. Antoniou , P. Micaelli , and A. J. Storkey . Meta-learning in neural networks : A survey.426 CoRR , abs/2004.05439 , 2020 . URL https : //arxiv.org/abs/2004.05439.427 [ 22 ] S. Jastrzebski , Z. Kenton , D. Arpit , N. Ballas , A. Fischer , Y. Bengio , and A. J. Storkey . Three factors428 influencing minima in SGD . CoRR , abs/1711.04623 , 2017 . URL http : //arxiv.org/abs/1711.04623.429 [ 23 ] S. Jastrzebski , M. Szymczak , S. Fort , D. Arpit , J. Tabor , K. Cho , and K. Geras . The break-even point430 on optimization trajectories of deep neural networks . In 8th International Conference on Learning431 Representations , ICLR 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020 . OpenReview.net , 2020 . URL432 https : //openreview.net/forum ? id=r1g87C4KwB.433 [ 24 ] Y. Jiang , B. Neyshabur , H. Mobahi , D. Krishnan , and S. Bengio . Fantastic generalization measures and434 where to find them . In 8th International Conference on Learning Representations , ICLR 2020 , Addis435 Ababa , Ethiopia , April 26-30 , 2020 . OpenReview.net , 2020 . URL https : //openreview.net/forum ? 436 id=SJgIPJBFvH.437 [ 25 ] Y. Jin , T. Zhou , L. Zhao , Y. Zhu , C. Guo , M. Canini , and A. Krishnamurthy . Auto { lrs } : Automatic438 learning-rate schedule by bayesian optimization on the fly . In International Conference on Learning439 Representations , 2021 . URL https : //openreview.net/forum ? id=SlrqM9_lyju.440 [ 26 ] D. Kalimeris , G. Kaplun , P. Nakkiran , B. L. Edelman , T. Yang , B. Barak , and H. Zhang . SGD on neural441 networks learns functions of increasing complexity . In H. M. Wallach , H. Larochelle , A. Beygelzimer,442 F. d ’ Alché-Buc , E. B . Fox , and R. Garnett , editors , Advances in Neural Information Processing Systems 32:443 Annual Conference on Neural Information Processing Systems 2019 , NeurIPS 2019 , December 8-14 , 2019,444 Vancouver , BC , Canada , pages 3491–3501 , 2019 . URL https : //proceedings.neurips.cc/paper/445 2019/hash/b432f34c5a997c8e7c806a895ecc5e25-Abstract.html.446 [ 27 ] D. P. Kingma and J. Ba . Adam : A method for stochastic optimization . In Y. Bengio and Y. LeCun , editors,447 3rd International Conference on Learning Representations , ICLR 2015 , San Diego , CA , USA , May 7-9,448 2015 , Conference Track Proceedings , 2015 . URL http : //arxiv.org/abs/1412.6980.449 [ 28 ] H. Lang , L. Xiao , and P. Zhang . Using statistics to automate stochastic optimization . In450 H. M. Wallach , H. Larochelle , A. Beygelzimer , F. d ’ Alché-Buc , E. B . Fox , and R. Garnett,451 editors , Advances in Neural Information Processing Systems 32 : Annual Conference on Neu-452 ral Information Processing Systems 2019 , NeurIPS 2019 , December 8-14 , 2019 , Vancouver , BC,453 Canada , pages 9536–9546 , 2019 . URL https : //proceedings.neurips.cc/paper/2019/hash/454 e1054bf2d703bca1e8fe101d3ac5efcd-Abstract.html.455 [ 29 ] G. Leclerc and A. Madry . The two regimes of deep network training . CoRR , abs/2002.10376 , 2020 . URL456 https : //arxiv.org/abs/2002.10376.457 [ 30 ] A. Lewkowycz . How to decay your learning rate . CoRR , abs/2103.12682 , 2021 . URL https : //arxiv.458 org/abs/2103.12682.459 [ 31 ] A. Lewkowycz , Y. Bahri , E. Dyer , J. Sohl-Dickstein , and G. Gur-Ari . The large learning rate phase of deep460 learning : the catapult mechanism . CoRR , abs/2003.02218 , 2020 . URL https : //arxiv.org/abs/2003.461 02218.462 [ 32 ] Y. Li , C. Wei , and T. Ma . Towards explaining the regularization effect of initial large learning rate in463 training neural networks . In H. M. Wallach , H. Larochelle , A. Beygelzimer , F. d ’ Alché-Buc , E. B. Fox,464 and R. Garnett , editors , Advances in Neural Information Processing Systems 32 : Annual Conference465 on Neural Information Processing Systems 2019 , NeurIPS 2019 , December 8-14 , 2019 , Vancouver , BC,466 Canada , pages 11669–11680 , 2019 . URL https : //proceedings.neurips.cc/paper/2019/hash/467 bce9abf229ffd7e570818476ee5d7dde-Abstract.html.468 [ 33 ] Z. Li and S. Arora . An exponential learning rate schedule for deep learning . In 8th International Conference469 on Learning Representations , ICLR 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020 . OpenReview.net,470 2020 . URL https : //openreview.net/forum ? id=rJg8TeSFDH.471 [ 34 ] I. Loshchilov and F. Hutter . SGDR : stochastic gradient descent with warm restarts . In 5th International472 Conference on Learning Representations , ICLR 2017 , Toulon , France , April 24-26 , 2017 , Conference473 Track Proceedings . OpenReview.net , 2017 . URL https : //openreview.net/forum ? id=Skq89Scxx.474 [ 35 ] I. Loshchilov and F. Hutter . Decoupled weight decay regularization . In 7th International Conference on475 Learning Representations , ICLR 2019 , New Orleans , LA , USA , May 6-9 , 2019 . OpenReview.net , 2019.476 URL https : //openreview.net/forum ? id=Bkg6RiCqY7.477 [ 36 ] J. Ma and D. Yarats . On the adequacy of untuned warmup for adaptive optimization . CoRR , abs/1910.04209,478 2019 . URL http : //arxiv.org/abs/1910.04209.479 [ 37 ] R. Müller , S. Kornblith , and G. E. Hinton . When does label smoothing help ? In H. M.480 Wallach , H. Larochelle , A. Beygelzimer , F. d ’ Alché-Buc , E. B . Fox , and R. Garnett , edi-481 tors , Advances in Neural Information Processing Systems 32 : Annual Conference on Neural482 Information Processing Systems 2019 , NeurIPS 2019 , December 8-14 , 2019 , Vancouver , BC,483 Canada , pages 4696–4705 , 2019 . URL https : //proceedings.neurips.cc/paper/2019/hash/484 f1748d6b0fd9d439f71450117eba2725-Abstract.html.485 [ 38 ] V. Nair and G. E. Hinton . 3d object recognition with deep belief nets . In Y. Bengio , D. Schuur-486 mans , J. D. Lafferty , C. K. I. Williams , and A. Culotta , editors , Advances in Neural Information Pro-487 cessing Systems 22 : 23rd Annual Conference on Neural Information Processing Systems 2009 . Pro-488 ceedings of a meeting held 7-10 December 2009 , Vancouver , British Columbia , Canada , pages 1339–489 1347 . Curran Associates , Inc. , 2009 . URL https : //proceedings.neurips.cc/paper/2009/hash/490 6e7b33fdea3adc80ebd648fffb665bb8-Abstract.html.491 [ 39 ] P. Nakkiran , G. Kaplun , Y. Bansal , T. Yang , B. Barak , and I. Sutskever . Deep double descent : Where492 bigger models and more data hurt . In 8th International Conference on Learning Representations , ICLR493 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020 . OpenReview.net , 2020 . URL https : //openreview.494 net/forum ? id=B1g5sA4twr.495 [ 40 ] A. Neelakantan , L. Vilnis , Q. V. Le , I. Sutskever , L. Kaiser , K. Kurach , and J. Martens . Adding gradient496 noise improves learning for very deep networks . CoRR , abs/1511.06807 , 2015 . URL http : //arxiv.497 org/abs/1511.06807.498 [ 41 ] M. Raghu , J. Gilmer , J. Yosinski , and J. Sohl-Dickstein . SVCCA : singular vector canonical correlation499 analysis for deep learning dynamics and interpretability . In I. Guyon , U. von Luxburg , S. Bengio , H. M.500 Wallach , R. Fergus , S. V. N. Vishwanathan , and R. Garnett , editors , Advances in Neural Information501 Processing Systems 30 : Annual Conference on Neural Information Processing Systems 2017 , December502 4-9 , 2017 , Long Beach , CA , USA , pages 6076–6085 , 2017 . URL https : //proceedings.neurips.cc/503 paper/2017/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html.504 [ 42 ] N. Rahaman , A. Baratin , D. Arpit , F. Draxler , M. Lin , F. A. Hamprecht , Y. Bengio , and A. C. Courville.505 On the spectral bias of neural networks . In K. Chaudhuri and R. Salakhutdinov , editors , Proceedings of the506 36th International Conference on Machine Learning , ICML 2019 , 9-15 June 2019 , Long Beach , California,507 USA , volume 97 of Proceedings of Machine Learning Research , pages 5301–5310 . PMLR , 2019 . URL508 http : //proceedings.mlr.press/v97/rahaman19a.html.509 [ 43 ] M. Rolinek and G. Martius . L4 : practical loss-based stepsize adaptation for deep learn-510 ing . In S. Bengio , H. M. Wallach , H. Larochelle , K. Grauman , N. Cesa-Bianchi , and R. Gar-511 nett , editors , Advances in Neural Information Processing Systems 31 : Annual Conference on512 Neural Information Processing Systems 2018 , NeurIPS 2018 , December 3-8 , 2018 , Montréal,513 Canada , pages 6434–6444 , 2018 . URL https : //proceedings.neurips.cc/paper/2018/hash/514 98b17f068d5d9b7668e19fb8ae470841-Abstract.html.515 [ 44 ] K. Simonyan and A. Zisserman . Very deep convolutional networks for large-scale image recognition . In516 Y. Bengio and Y. LeCun , editors , 3rd International Conference on Learning Representations , ICLR 2015,517 San Diego , CA , USA , May 7-9 , 2015 , Conference Track Proceedings , 2015 . URL http : //arxiv.org/518 abs/1409.1556.519 [ 45 ] L. N. Smith and N. Topin . Super-convergence : Very fast training of residual networks using large learning520 rates . CoRR , abs/1708.07120 , 2017 . URL http : //arxiv.org/abs/1708.07120.521 [ 46 ] S. L. Smith , B. Dherin , D. G. T. Barrett , and S. De . On the origin of implicit regularization in stochastic522 gradient descent . CoRR , abs/2101.12176 , 2021 . URL https : //arxiv.org/abs/2101.12176.523 [ 47 ] N. Srivastava , G. E. Hinton , A. Krizhevsky , I. Sutskever , and R. Salakhutdinov . Dropout : a simple524 way to prevent neural networks from overfitting . J. Mach . Learn . Res. , 15 ( 1 ) :1929–1958 , 2014 . URL525 http : //dl.acm.org/citation.cfm ? id=2670313.526 [ 48 ] M. Tan and Q. V. Le . Efficientnet : Rethinking model scaling for convolutional neural networks . In527 K. Chaudhuri and R. Salakhutdinov , editors , Proceedings of the 36th International Conference on Machine528 Learning , ICML 2019 , 9-15 June 2019 , Long Beach , California , USA , volume 97 of Proceedings of529 Machine Learning Research , pages 6105–6114 . PMLR , 2019 . URL http : //proceedings.mlr.press/530 v97/tan19a.html.531 [ 49 ] A. C. Wilson , R. Roelofs , M. Stern , N. Srebro , and B. Recht . The marginal value of adaptive gradient532 methods in machine learning . In I. Guyon , U. von Luxburg , S. Bengio , H. M. Wallach , R. Fergus,533 S. V. N. Vishwanathan , and R. Garnett , editors , Advances in Neural Information Processing Systems 30:534 Annual Conference on Neural Information Processing Systems 2017 , December 4-9 , 2017 , Long Beach,535 CA , USA , pages 4148–4158 , 2017 . URL https : //proceedings.neurips.cc/paper/2017/hash/536 81b3833e2504647f9d794f7d7b9bf341-Abstract.html.537 [ 50 ] H. Xiao , K. Rasul , and R. Vollgraf . Fashion-mnist : a novel image dataset for benchmarking machine538 learning algorithms . CoRR , abs/1708.07747 , 2017 . URL http : //arxiv.org/abs/1708.07747.539 [ 51 ] Z. Xu , A. M. Dai , J. Kemp , and L. Metz . Learning an adaptive learning rate schedule . CoRR,540 abs/1909.09712 , 2019 . URL http : //arxiv.org/abs/1909.09712.541 [ 52 ] G. Zhang , C. Wang , B. Xu , and R. B. Grosse . Three mechanisms of weight decay regularization . In 7th542 International Conference on Learning Representations , ICLR 2019 , New Orleans , LA , USA , May 6-9 , 2019.543 OpenReview.net , 2019 . URL https : //openreview.net/forum ? id=B1lz-3Rct7.544 [ 53 ] H. Zhang , Y. N. Dauphin , and T. Ma . Fixup initialization : Residual learning without normalization . In545 7th International Conference on Learning Representations , ICLR 2019 , New Orleans , LA , USA , May 6-9,546 2019 . OpenReview.net , 2019 . URL https : //openreview.net/forum ? id=H1gsz30cKX.547 [ 54 ] Z. Zhu , J. Wu , B. Yu , L. Wu , and J. Ma . The anisotropic noise in stochastic gradient descent : Its behavior548 of escaping from sharp minima and regularization effects . In K. Chaudhuri and R. Salakhutdinov , editors,549 Proceedings of the 36th International Conference on Machine Learning , ICML 2019 , 9-15 June 2019,550 Long Beach , California , USA , volume 97 of Proceedings of Machine Learning Research , pages 7654–7663.551 PMLR , 2019 . URL http : //proceedings.mlr.press/v97/zhu19e.html.552 Checklist553 1 . For all authors ... 554 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s contribu-555 tions and scope ? [ Yes ] 556 ( b ) Did you describe the limitations of your work ? [ Yes ] This is discussed in Section 5.557 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] This is discussed in558 Section 5559 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to them ? [ Yes ] 560 2 . If you are including theoretical results ... 561 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] The assumptions are562 stated with the results.563 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Complete proofs can be found564 in the APPENDIX565 3 . If you ran experiments ... 566 ( a ) Did you include the code , data , and instructions needed to reproduce the main experimental567 results ( either in the supplemental material or as a URL ) ? [ Yes ] A reference implementation is568 provided under a free license.569 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were chosen ) ? 570 [ Yes ] Training details are specified with the experiments.571 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experiments572 multiple times ) ? [ Yes ] Errors bars are reported for our own experiments , where applicable.573 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs,574 internal cluster , or cloud provider ) ? [ Yes ] We specify the used compute in the APPENDIX.575 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 576 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 577 ( b ) Did you mention the license of the assets ? [ No ] We only use freely available data sets.578 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] No new579 assets / data sets were used.580 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re us-581 ing/curating ? [ No ] No additional data was used.582 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable infor-583 mation or offensive content ? [ No ] For the data sets we use , this is already discussed in the584 community.585 5 . If you used crowdsourcing or conducted research with human subjects ... 586 ( a ) Did you include the full text of instructions given to participants and screenshots , if applicable ? 587 [ N/A ] Not applicable.588 ( b ) Did you describe any potential participant risks , with links to Institutional Review Board ( IRB ) 589 approvals , if applicable ? [ N/A ] Not applicable.590 ( c ) Did you include the estimated hourly wage paid to participants and the total amount spent on591 participant compensation ? [ N/A ] Not applicable.592

Summary: This papers aims to explore the impact of learning rate schedules in neural network training, through   a newly proposed measure denoted the “activation pattern temperature” which examines the probability that ReLU activations will change between the 0 and linear regions. Thus this measure aims to examine the changes in non-linear behavior of the network. Several learning rate schedules are examined through this lens, and a learning rate schedule based on reducing this temperature is proposed and examined.