# RALR: Random Amplify Learning Rates for Training Neural Networks
## 5 Experiment
### RALR on Different Tasks  
CIFAR-10 and CIFAR-100: Both of the CIFAR datasets [24] consist of 60,000, 32 \(\times\) 32 color images. CIFAR-10 has ten classes, such as dog, cat, car, and boat, with 6000 images per class, including 5000 training images per class and 1000 validation images per class. There are 50,000 training images and 10,000 validation images in total. The CIFAR-100 dataset is similar the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 validation images per class. CIFAR-100 requires much more fine-grained recognition compared to CIFAR-10, as some classes are very visually similar. For example, it contains five different species of flowers: orchids, poppies, roses, sunflowers, and tulips. In the experiment on the CIFAR dataset, the images are first zero-padded with 4 pixels on each side to get the \(40\times 40\) pixel images. Then, \(32\times 32\) pixel images are cropped randomly. Besides, images are randomly mirrored horizontally at a probability of \(p=0.5\). We compare the classification accuracy of some different architectures on the CIFAR-10 dataset, including deep residual networks (ResNet) [25], dense network(DenseNet) [26], and wide residual network (WRN) [27]. The networks includs (1) ResNet-56 and ResNet-110: a ResNet with a depth of 56 and 110; (2) DenseNet-BC-100-12: a DenseNet with a depth of 100 and a growth rate of 12; and (3) WRN-28-10, a WRN with a depth of 28 and a widening factor of 10. We first show the feature embedding in Figure 4, which is the results of network ResNet-110 updated by SGD and trained on the CIFAR-10 dataset with the cross-entropy loss. Compare with the baseline one, the model trained using the RALR technique brings stronger intra-class compactness and larger inter-class separability. This means our RALR strategy can efficiently improve the network performance. We trained for 300 epochs on the CIFAR-10 dataset for the architectures listed in Table 1. For all architectures, we trained with batches of 128 images using SGD. The hyper-parameter of Nesterov momentum is set to 0.9, weight decay is \(1\times 10^{-4}\), and the global learning rate is initially set to 0.1 and decreased by \(10\times\) at the 150 th and 225 th epoch. In Table 1, we observe that the RALR technique consistently outperforms the baseline scheme. We trained for 300 epochs on the CIFAR-100 dataset for architectures listed in Table 2. For ResNets and WRNs, we trained with batches of 128 images using SGD, and we trained with batches of 64 images using SGD for DenseNet. The hyper-parameter of Nesterov momentum is set to 0.9, weight decay is \(1\times 10^{-4}\), and the global learning rate is initially set to 0.1 and decreased by \(10\times\) at the 150 th and 225 th epoch for all architectures. In Table 2, we observe that the RALR technique consistently outperforms the baseline scheme. For example, our approach achieves 75.87% accuracy when applying network ResNet-110, which is a 1.34% improvement over baseline. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Baseline** & **RALR** \\ \hline ResNet-20 & 91.41 \(\pm\) 0.08 & **92.28 \(\pm\) 0.05** \\ ResNet-56 & 93.77 \(\pm\) 0.06 & **94.35 \(\pm\) 0.05** \\ ResNet-110 & 94.42 \(\pm\) 0.06 & **95.05 \(\pm\) 0.04** \\ \hline WRN-28-10 & 95.82 \(\pm\) 0.03 & **96.36 \(\pm\) 0.03** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Validation accuracy (%) over different architectures on CIFAR-10. Figure 4: The feature embedding of ResNet-110 on the CIFAR-10 dataset. The network is trained with the cross-entropy loss and updated by SGD. Each color on the picture represents a category of the CIFAR-10 dataset. (**a**) Baseline. (**b**) Random learning rates.