# RALR: Random Amplify Learning Rates for Training Neural Networks
## 2 Motivation  
Training deep neural networks involves using an optimization algorithm to find the weight parameter vector to best map inputs and outputs. Many researchers discover how to accelerate the learning speed, reduce overfitting, and obtain optimal networks' performance with deep learning models. The training procedure is challenging, not least because the optimization function is non-convex and can be falling into the local minima or saddle points problems, as shown in Figure 1b. Training with conventionally decaying way cannot receive better performance after 159 epochs (blue line). Yet, training with RALR is less prone to overfitting, which improves the networks' performance continuously almost in the whole training process (orange line), and the network gains the best classification accuracy at the 291 st epoch. The training procedure of RALR is more effective and can speed up convergence ability and mitigate against overfitting (see Figure 1b). The validation accuracy of the baseline reaches the highest 65.25% at the 139 th epoch in the first stage. In comparison, the validation accuracy of RALR achieves 66.02% at the 31 st epoch and reaches the highest 73.30% at the 142 nd epoch in the first stage.  
The global learning rate is a hyper-parameter that controls how much to change the model. Traditionally, the global learning rate decreases at specific epochs, i.e., the learning rate may become smaller and smaller, and the learning rate might be too small after some epochs, resulting in the optimizer sticking in saddle points or local minima that may slow down the training speed and restrict the network performance in a long time [14]. The x-axis on each subfigure of Figure 2 indicates the value of the activation units, and the y-axis indicates the number of activation units under this specific value. As shown in Figure 2a, the activation values of test images or the network parameters are substantially unchanged for the last 100 epochs, which indicates, the network cannot update effectively in the remaining training time. Increasing the learning rate between reasonable boundary values at the specific probability can potentially help the training escape saddle points and local minima and find flatter minima with good generalization performance by the leap forward of the network's weights updates. Figure 2b shows that the network's activation values and weight parameters are updated effectively with the RALR for the last 100 epochs. That means the network may restructure the parameter weight vector to gain near-optimal performance. Each column of Figure 2 indicates the activation values of different convolutional layers for one given image. The first, second, and last column indicate the activation values of the layer1, layer2, and layer3 of ResNet-110.  
From the above discussion, if the training procedure does not improve the network performance anymore, we can amplify the learning rate between reasonable boundary values to accelerate the network's training and achieve better performance. Increasing the learning rate allows the optimizer of deep neural networks to escape saddle points and local minima effectively. In this way, it is possible to further explore the optimization algorithm and leads the network to better generalization, as illustrated in Figure 2.  
We propose the RALR method to alter the learning rate around the global one. Amplifying the learning rate at some epochs can help the optimization algorithm escape local minima or saddle points and quickly find the correct descent direction. Particularly, the larger learning rate will make the optimizer explore the larger hidden space, which indicates the network could be updated more effectively. As a result, we speed the training up and train the network more effectively in comparison with the previous one.