# RALR: Random Amplify Learning Rates for Training Neural Networks
## 5 Experiment
### Ablation Study  
Effect of the random interval factor: We explore the effect of the random interval factor _low_ and _high_ according to the classification performance. We change the random interval factor setting _low_ from 1 to 5 and _high_ from 2 to 9; the probability to take RALR method is fixed to \(p_{r}=0.5\). In addition, we evaluate the classification performance of network ResNet-110 on the CIFAR-100 dataset using the above factors. Partial validation accuracy is demonstrated in Table 8. As shown in Table 8, with the increase of random factor _low_, the classification accuracy decreases after _low_\(\geq 2\). With the increase of random factor _high_ when setting the random factor to \(low=1\), the accuracy raises and reaches the highest when \(high=3\), and then the accuracy decreases.  
Effect of the probability: Another important hyper-parameter of RALR is amplification probability \(p_{r}\). We do some experiments to explore the influence of probability \(p_{r}\) with a different threshold. We set amplification factor \(p_{r}\) from 0.1 to 0.9, and evaluate the accuracy of network ResNet-110 on the CIFAR-100 dataset. As illustrated in Table 9, the accuracy reaches the highest at amplification probability \(p_{r}=0.5\), and variance reaches the lowest at the same time. The variance reaches the lowest and accuracy reaches the highest when \(p_{r}=0.5\), \(low=1\), and \(high=3\).  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **+RALR** & **ResNet-56** & **ResNet-110** \\ \hline \multirow{3}{*}{+baseline} & \(\times\) & 73.69 \(\pm\) 0.07 & 74.53 \(\pm\) 0.41 \\  & \(\surd\) & **74.54 \(\pm\) 0.04** & **75.87 \(\pm\) 0.07** \\ \cline{2-4}  & \(\times\) & 74.55 \(\pm\) 0.05 & 75.34 \(\pm\) 0.12 \\  & \(\surd\) & **75.02 \(\pm\) 0.02** & **76.11 \(\pm\) 0.05** \\ \cline{2-4}  & \(\times\) & 75.66 \(\pm\) 0.07 & 77.89 \(\pm\) 0.05 \\ \cline{2-4}  & \(\surd\) & **76.02 \(\pm\) 0.04** & **78.26 \(\pm\) 0.05** \\ \cline{2-4}  & \(\times\) & 76.67 \(\pm\) 0.11 & 77.71 \(\pm\) 0.13 \\ \cline{2-4}  & \(\surd\) & **77.13 \(\pm\) 0.03** & **78.52 \(\pm\) 0.07** \\ \cline{2-4}  & \(\times\) & 74.05 \(\pm\) 0.11 & 75.01 \(\pm\) 0.11 \\ \cline{2-4}  & \(\surd\) & **74.94 \(\pm\) 0.07** & **75.98 \(\pm\) 0.04** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Validation accuracy (%) over different techniques based on ResNets trained on CIFAR-100.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Low** & **High** & **Acc.** \\ \hline \multirow{8}{*}{ResNet-110} & 1 & 2 & 75.37 \(\pm\) 0.82 \\  & 1 & 3 & **75.87 \(\pm\) 0.07** \\ \cline{1-1}  & 1 & 5 & 75.35 \(\pm\) 0.22 \\ \cline{1-1}  & 1 & 7 & 70.17 \(\pm\) 1.58 \\ \cline{1-1} \cline{2-4}  & 2 & 3 & 75.71 \(\pm\) 0.57 \\ \cline{1-1}  & 2 & 5 & 75.81 \(\pm\) 0.11 \\ \cline{1-1}  & 2 & 7 & 75.13 \(\pm\) 0.45 \\ \cline{1-1} \cline{2-4}  & 3 & 5 & 71.88 \(\pm\) 2.11 \\ \cline{1-1}  & 3 & 7 & 71.79 \(\pm\) 3.01 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Impact on parameters of different _low_, _high_ with fixed probability of \(p_{r}=0.5\) when taking RALR method.