# Implicit Bias of Deep Learning in the Large Learning Rate Phase: A Data Separability Perspective
## 5 Experiment
### Effectiveness on Synthetic and Real-World Datasets
To further enhance our theoretical finding and complement the effectiveness of the general annealing methods, we conducted experiments on the large-scale real-world dataset WebVision [25] as it is a large-scale dataset of images that has been specifically designed to evaluate the performance of computer vision algorithms under noise. We followed the \"Mini\" setting in [24; 59] that only takes the first 50 classes of the resized Google image subset. We evaluated the trained networks on the same 50 classes of the WebVision validation set, considered as a clean validation. We trained a ResNet-50 [14] using SGD for 250 epochs with a Nesterov momentum of 0.9, a weight decay of \(3\times 10^{-5}\), and a batch size of 512. We resized the images to 224 \(\times\) 224. Typical data augmentations, including random width/height shift, colour jittering and random horizontal flip, were applied. The accuracies on the clean WebVision validation set (e.g., only the first 50 classes) are reported in Table 5. As a result, the large learning rate annealing methods (one-step annealing and exponential learning rate annealing) provided better generalization. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multirow{2}{*}{**Methods**} & \multirow{2}{*}{**Clean (\(\tau\) = 0.0)**} & \multicolumn{4}{c}{**Symmetric Noise Rate (\(\tau\))**} \\  & & & **0.1** & **0.2** & **0.3** & **0.4** \\ \hline \multirow{3}{*}{MNIST} & SL & 99.09 \(\pm\) 0.02 & 98.60 \(\pm\) 0.04 & 98.29 \(\pm\) 0.08 & 97.88 \(\pm\) 0.12 & 97.57 \(\pm\) 0.09 \\  & OS & 99.33 \(\pm\) 0.04 & 99.22 \(\pm\) 0.57 & 98.71 \(\pm\) 0.10 & 98.34 \(\pm\) 0.07 & 97.96 \(\pm\) 0.16 \\  & EXP & 99.40 \(\pm\) 0.03 & 98.85 \(\pm\) 0.21 & 98.84 \(\pm\) 0.10 & 98.63 \(\pm\) 0.20 & 98.48 \(\pm\) 0.03 \\ \hline \multirow{3}{*}{CIFAR-10} & SL & 86.37 \(\pm\) 0.05 & 82.01 \(\pm\) 0.19 & 78.70 \(\pm\) 0.29 & 75.83 \(\pm\) 0.28 & 71.58 \(\pm\) 0.08 \\  & OS & 91.38 \(\pm\) 0.07 & 86.87 \(\pm\) 0.15 & 83.95 \(\pm\) 0.24 & 81.72 \(\pm\) 0.08 & 78.70 \(\pm\) 0.25 \\  & EXP & 91.63 \(\pm\) 0.15 & 85.52 \(\pm\) 0.22 & 82.94 \(\pm\) 0.32 & 81.57 \(\pm\) 0.99 & 79.66 \(\pm\) 2.20 \\ \hline \multirow{3}{*}{CIFAR-100} & SL & 48.10 \(\pm\) 0.14 & 42.31 \(\pm\) 0.44 & 38.10 \(\pm\) 0.65 & 34.10 \(\pm\) 0.25 & 31.21 \(\pm\) 1.01 \\  & OS & 70.50 \(\pm\) 1.07 & 62.66 \(\pm\) 1.51 & 57.31 \(\pm\) 2.09 & 52.08 \(\pm\) 1.63 & 47.22 \(\pm\) 0.88 \\ \cline{1-1}  & EXP & 70.14 \(\pm\) 0.82 & 63.67 \(\pm\) 0.26 & 55.70 \(\pm\) 0.24 & 49.67 \(\pm\) 1.95 & 43.39 \(\pm\) 1.11 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Test accuracy (%) of different methods on benchmark datasets with clean or symmetric label noise (\(\tau\in[0.1,0.4]\)). The results (mean\(\pm\)std) are reported over three random runs. SL refers to a training schedule with a small learning rate that remains constant throughout the training process. OS (one-step) denotes a training schedule where the learning rate starts at a high value and then drops to a smaller value after a specified number of training steps. Exp refers to a training schedule where the learning rate decreases exponentially as the training progresses. In terms of computational complexity, the actual process of changing the magnitude of the learning rate during training is typically straightforward and computationally inexpensive. The real computational cost of learning rate annealing comes from the additional training iterations required to allow the model to converge more precisely towards the optimal solution. Overall, the actual computational cost of learning rate annealing can depend on a variety of factors, including the size of the dataset, the complexity of the model, and the specific annealing schedule used. However, in general, the computational cost of learning rate annealing is relatively small compared to the overall cost of training a deep learning model.