# RALR: Random Amplify Learning Rates for Training Neural Networks
## 5 Experiment
### RALR on Different Tasks
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Baseline** & **RALR** \\ \hline ResNet-20 & 69.88 \(\pm\) 0.06 & **70.48 \(\pm\) 0.03** \\ ResNet-56 & 73.69 \(\pm\) 0.07 & **74.54 \(\pm\) 0.04** \\ ResNet-110 & 74.53 \(\pm\) 0.41 & **75.87 \(\pm\) 0.07** \\ \hline WRN-28-10 & 78.81 \(\pm\) 0.24 & **79.54 \(\pm\) 0.11** \\ \hline DesNet-BC-100-12 & 77.22 \(\pm\) 0.15 & **77.95 \(\pm\) 0.04** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Validation accuracy (%) over different architectures on CIFAR-100. Fine-grained classification: The fine-grained image classification task focuses on differentiating between hard to distinguish object classes, such as species of birds, flowers, or animals, and identifying the makes or models of vehicles. Since collecting fine-grained samples often requires expert-level domain knowledge, most fine-grained datasets are hard to extend to a large scale, i.e., the fine-grained datasets are usually insufficient to gain reliable classification accuracy of deep neural networks when adopting the random initialization method. In the fine-grained classification tasks, the deep learning models are always fine-tuned on the parameters trained on ImageNet. We test the effectiveness of RALR on several fine-grained benchmark datasets: CUB-200-2011 [28], StandfordDogs [29], and StandfordCars [30]. In the experiments, we evaluate the performance of the fine-grained classification tasks above using ResNet-50 pre-trained on ImageNet, and the model is fine-tuned by the optimizer SGD. 'The model is trained for 95 epochs, and the learning rate is initially set to 0.001 and decayed at 30 th, 60 th, and 90 th epoch. The comparison result is shown in Table 3. For all the cases in Table 3, models trained using RALR achieve significant boosts over baseline. Object Detection: Object detection is a task to predict the bounding boxes of each object of an image, with associated real-value confidence. SSD [31] is one object detection architecture. In this section, we use SSD ([[https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Object-Detection](https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Object-Detection)]([https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Object-Detection](https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Object-Detection)), accessed on 25 December 2021) to evaluate the effectiveness of RALR on object detection. For comparison, the backbone of our experiments is VGG-16 and using VOC 2007 and VOC 2012 as the training dataset simultaneously. The results are shown in Table 4. RALR can achieve further detection performance compared with the baseline one even trained with the network VGG-16 pre-trained on ImageNet-2012. Machine Translation - WMT 2014: We evaluate the capability of RALR technique on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs. Each training batch contained a set of sentence pairs containing approximately 25,000 source tokens and 25,000 target tokens. The backbone is from the paper [32]. Table 5 shows the comparison results ([[https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Machine-Translation/](https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Machine-Translation/)]([https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Machine-Translation/](https://github.com/sgrvnond/a-PyTorch-Tutorial-to-Machine-Translation/)), accessed on 25 December 2021). The results in Table 5 show that the RALR technique outperforms the baseline and even uses the factor \(low=1\), \(high=3\), and \(p_{r}=0.5\) which gained in computer vision tasks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Backbone** & **+RALR** & **Acc.** \\ \hline SSD-VGG16 & \(\times\) & 77.0 \\ \(\surd\) & **77.7** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison results of SSD on PASCAL VOC 2012.