# RALR: Random Amplify Learning Rates for Training Neural Networks
## 1 Introduction  
After decades of lower activity, studies in artificial neural networks were revitalized after a 2006 breakthrough [1] in the research on deep learning. Deep neural networks have been a tremendous practical success in many fields, such as image recognition [2], object detection [3], face recognition [4], object tracking [5], and speech recognition [6]. However, a deep neural network is a complex mathematical function, consisting of millions of parameters that need to be computed to solve a problem. In addition, the parameters of deep neural networks include weight parameters and hyper-parameters. Each hyper-parameter has its impact on training deep neural networks. Deep neural networks are usually learned to perform a specific task, and the goal during training is to minimize the objective function \(f=\mathbb{R}^{n}\rightarrow\mathbb{R}\), typically updated by stochastic gradient descent (SGD). The basic procedure to optimize \(f\) is:  
\[\theta_{t+1}=\theta_{t}-\eta_{t}\
abla f_{t}(\theta_{t}) \tag{1}\]  
Here, \(\theta_{t}\in\mathbb{R}^{n}\) is the weight parameter vector at the \(t\)-th iteration, \(\
abla f_{t}(\theta_{t})\) is the loss gradient information obtained on a relatively small batch of training datasets at the \(t\)-th iteration, and \(\eta_{t}\) is the learning rate at the \(t\)-th iteration. According to Equation (1), we perceive that the learning rate is a crucial hyperparameter that controls how much we are adjusting the weight parameter vector of deep neural networks concerning the loss gradient information obtained by training datasets. It is well known that a small learning rate will make the training procedure converge slowly and more prone to get stuck in saddle points or local minima, and a large learning rate will cause the training procedure to diverge. Traditionally, the learning rate should be a single value that monotonically decreases during training at specific epochs. This paper describes the remarkable phenomenon that is randomly amplifying the learning rate during training is beneficial overall. Hence, we propose randomly magnifying the global learning rate between reasonable boundary values based on the traditional monotonically decaying way, named random amplify learning rates (RALR). The RALR effectively makes the networks' performance near the optimal classification accuracy compared to the conventional monotonically decaying way. Furthermore, unlike the adaptive learning rates methods, the RALR procedure requires virtually no additional computation. To better understand the potential profits of RALR, we visually plot the training and validation loss, validation accuracy, and the global learning rate changing of both our RALR method and traditional method while training the CIFAR-100 dataset using network ResNet-110 in Figure 1. The blue curve in Figure 1b shows the baseline of ResNet-110p; the baseline of ResNet-110 reaches the best classification accuracy of 73.94% after 300 epochs. In contrast, it is possible to achieve 2.11% gain in the same epochs using the RALR technique, and the orange curve in Figure 1 shows the validation accuracy while training using RALR, the network ResNet-110 reaches the best classification accuracy of 76.05% at the 291 st epoch. In Figure 1a, we found that training with traditional strictly decreasing methods (blue line) is more prone to overfitting compared with training using RALR (orange line). As shown in Figure 1, enlarging the global learning rate may temporarily reduce the classification accuracy and increase the training and verification loss. Still, the classification accuracy perhaps reaches a higher level after the global learning rate is restored. The contributions of this paper are:  
1. A methodology for setting the global learning rates for training deep neural networks makes the performance near-optimal classification accuracy compared with traditional monotonically decaying methods, with essentially no additional computation.