# RALR: Random Amplify Learning Rates for Training Neural Networks
## 3 Related Works  
Training deep neural networks is a challenging optimization task. Many researchers try to achieve better network performance and accelerate the training speed. Some works are described briefly in the section below.  
Adaptive Learning Rates: In order to improve the convergence speed of gradient descent learning algorithms, momentum [7] is proposed to accelerate SGD in the relevant direction. Instead of utilizing the gradient of the current step individually to guide the direction, momentum also accumulates the gradient of the previous step. As one of the early adaptive methods, AdaGrad [8] is a dynamic learning rate decreasing algorithm. AdaGrad adapts different learning rates associated with different features, i.e., frequently occurring features updated using small learning rates and infrequently occurring features updated using large learning rates. RMSprop is introduced by [PERSON] in Lecture 6e of his Coursera Class ([[http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)]([http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)), accessed on 25 December 2021). RMSprop divides the learning rate by an exponentially decaying average of squared gradients. AMSGrad [15] pinpoints the exponential moving average of past squared gradients as a reason for the poor generalization behavior of earlier adaptive learning rate methods and uses the maximum of past squared gradients rather than the exponential average to update the parameters.  
In recent years, several optimizers have been proposed. These include AdamW [16], which fixes weight decay in Adam; QHAdam [17], which averages a standard SGD step with a momentum SGD step; and AggMo [18], which combines multiple momentum terms; and others.  
Cycling learning rates: CLR and SGDR: The idea of cycling increasing the global learning rate has been explored by algorithms such as SGDR [19] and CLR [20]. SGDR reset the global learning rate to a large value in a periodic manner. After this, the global learning rate declines to a small value following a cosine annealing schedule. In addition, CLR lets the global learning rate cyclically vary between reasonable boundary values. [PERSON] [20] argued that we could estimate a reasonable learning rate by initially training the model with a very small learning rate and increasing the global learning rate linearly or exponentially at each iteration in CLR.  
Unlike the above techniques, RALR amplifies the global learning rate based on the given probability. It prevents the model from overfitting by escaping saddle points or  
Figure 2: The activation values changing of a network ResNet-110 trained for 300 epochs. The blue column indicates activation values of the network ResNet-110 obtained at the 200 th epoch, and the red column indicates activation values of the network ResNet-110 obtained at the 300 th epoch. **(a)** Baseline. **(b)** RALR.  
local minima. Further experiments demonstrate the effectiveness of the RALR in achieving better performance.  
Others: [PERSON] et al. [21] stressed that the learning rate of the gradient descent strongly affects performance in neural networks and propose the Alrao algorithm. [PERSON] et al. believed that each unit or feature in the network gets its learning rate sampled from a random distribution spanning several orders of magnitude, hoping that enough units will get a close-to-optimal learning rate.  
Unlike Alrao (the experimental results of Alrao show that Alrao cannot effectively improve the network performance on CIFAR and ImageNet datasets. Besides, the code of Alrao is not publicly available. Therefore, we are not able to provide the comparison results with ALRAO here), RALR is easy to implement and takes the same learning rate for all units or features of the network.