SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Theorem 3 below we prove a linear convergence rate in the case of  $f$  satisfying a PL condition. We say that a function  $f: \mathbb{R}^n \to \mathbb{R}$  satisfies the PL condition if there exists  $\mu > 0$  such that,  $\forall w \in \mathbb{R}^n: \| \nabla f(w) \|^2 \geq 2\mu (f(w) - f(w^*))$ . The proof of this theorem extends Theorem 3.6 of Loizou et al. [2021] to the use of a stochastic nonmonotone line search. Again here, the presence of a nonmonotone term does not modify the constants controlling the speed of convergence ( $a_2$  below can be chosen arbitrarily small) as long as  $\xi$  is small enough. The conditions on  $c$  and on  $\eta^{\max}$  are the same as those of Theorem 3.6 of Loizou et al. [2021] (see the proof).
Theorem 3. Let  $C_k$  and  $\eta_k$  be defined in (2), with  $\eta_{k,0}$  defined in (6). We assume interpolation, the PL condition on  $f$  and that  $f_{i_k}$  are  $L_{i_k}$ -Lipschitz smooth. Given  $0 < a_2 \coloneqq \frac{4\mu c(1 - c) - L_{max}}{4\delta c(1 - c)} + \frac{1}{2\eta^{max}}$  and assuming  $\frac{2\delta(1 - c)}{L_{max}} < \bar{\eta}^{\min}$ ,  $\eta^{\max} < \frac{2\delta c(1 - c)}{L_{max} - 4\mu c(1 - c)}$ ,  $\frac{L_{max}}{4\mu} < c < 1$  and  $\xi < \frac{a_2c}{a_2c + L_{max}}$ , we have
$$
\mathbb {E} \left[ f \left(w _ {k + 1}\right) - f \left(w ^ {*}\right) + a _ {2} \eta^ {\max } \left(C _ {k} - f \left(w ^ {*}\right)\right) \right] \leq d _ {2} ^ {k} \left(1 + a _ {2} \eta^ {\max }\right) \left(f \left(w _ {0}\right) - f \left(w ^ {*}\right)\right)
$$
where  $d_2 \coloneqq \min \left\{\nu, b_2\right\} \in (0,1)$ ,  $\nu \coloneqq \eta^{max}\left(\frac{L_{max} - 4\mu c(1 - c)}{2\delta c(1 - c)} + a_2\right) \in (0,1)$ ,  $b_2 \coloneqq \left(1 + \frac{L_{max}}{a_2c}\right)\xi \in (0,1)$ .
Theorem 3 is particularly meaningful for modern neural networks because of the recent work of Liu et al. [2022]. More precisely, their Theorem 8 and Theorem 13 prove that, respectively given a wide-enough fully-connected or convolutional/skip-connected neural network, the corresponding function satisfies a local version of the PL condition. As a consequence, nonmonotone line search methods achieve linear convergence for training of non-convex neural networks if the assumptions of both Theorem 3 above and Theorem 8 from Liu et al. [2022] hold and if the initial point  $w_0$  is close enough to  $w^*$ .
In this section, we benchmark PoNoS against state-of-the-art methods for training different overparametrized models. We focus on various deep learning architectures for multi-class image classification problems, while in Section 5.3 we consider transformers for language modelling and kernel models for binary classification. In the absence of a better estimate, we assume  $f_{i_k}(w^*) = 0$  for all the experiments. Indeed the final loss is in many problems very close to 0.
Concerning datasets and architectures for the image classification task, we follow the setup by Luo et al. [2019], later employed also in Vaswani et al. [2019], Loizou et al. [2021]. In particular, we focus on the datasets MNIST, FashionMNIST, CIFAR10, CIFAR100 and SVHN, addressed with the architectures MLP [Luo et al., 2019], EfficientNet-b1 [Tan and Le, 2019], ResNet-34 [He et al., 2016], DenseNet-121 [Huang et al., 2017] and WideResNet [Zagoruyko and Komodakis, 2016]. Because of space limitation, we will not show all the experiments here, but report on the first three datasets combined respectively with the first three architectures. We refer to the supplementary materials for the complete benchmark (see Figures I-XII) and the implementation details (Section C). In Section C of the supplementary materials, we also discuss the sensitivity of the results to the selection of hyper-parameters.
In Figure 1 we compare SGD [Robbins and Monro, 1951], Adam [Kingma and Ba, 2015], SLS [Vaswani et al., 2019], SPS [Loizou et al., 2021] and PoNoS. We present train loss (log scale), test accuracy and average step size within the epoch (log scale). Note that the learning rate of SGD and Adam has been chosen through a grid-search as the one achieving the best performance on each problem. Based on Figure 1, we can make the following observations:
- PoNoS achieves the best performances both in terms of train loss and test accuracy. In particular, it often terminates by reaching a loss of  $10^{-6}$  and it always achieves the highest test accuracy.
- SLS does not always achieve the same best accuracy as PoNoS. In terms of training loss, SLS is particularly slow on mnist|mlp and fashion|effb1, while competitive on the others. On these two problems, both SLS and SPS employ a step size that is remarkably smaller than that of PoNoS. On the same problems, SPS behaves similarly as SLS because the original Polyak step size is larger than the "smoothing/resetting" value (3), i.e.,  $\tilde{\eta}_{k,0} > \eta_{k - 1}\gamma^{b / M}$  and thus the step size is controlled by (3). On the other hand, the proposed nonmonotone line search selects a larger step size, achieving faster convergence and better generalization [Nacson et al., 2022].
- SLS encounters some cancellation errors (e.g., around epoch 120 in cifar10|resnet34) that lead the step size to be reduced drastically, sometimes reaching values below  $10^{-7}$ . These events are caused by the fact that  $f_{i_k}(w_k)$  and  $f_{i_k}(w_{k + 1})$  became numerically identical. Thanks to the nonmonotone term  $C_k$  replacing  $f_{i_k}(w_k)$ , PoNoS avoids cancellations errors by design.
- SGD, Adam and SPS never achieve the best test accuracy nor the best training loss. In accordance with the results in Vaswani et al. [2019], the line-search-based algorithms outperform the others.
Because of space limitation, we defer the comparison between PoNoS and its monotone counterpart to the supplementary materials (Section E.2). There we show that PoNoS outperforms the other line search techniques, including a tractable stochastic adaptation of the method by Grippo et al. [1986].
Figure 1: Comparison between the proposed method (PoNoS) and the-state-of-the-art. Each column focus on a dataset. First row: train loss. Second row: test accuracy. Third row: step size.
5.1 A New Resetting Technique
In this subsection, we compare different initial step sizes and resetting techniques. We fix the line search to be (2), but similar comments can be made on (1) (see the Section D.2 of the supplementary materials). In Figure 2 we compare PoNoS and PoNoS_reset0 (without (5)) with zhang_reset2 (initial step size (3)) and:
- zhang reset3:  $\eta_{k,0} = \eta_{k - 1}\frac{||\nabla f_{i_{k - 1}}(w_{k - 1})||^2}{||\nabla f_{i_k}(w_k)||^2}$ , adapted to SGD from Nocedal and Wright [2006],
- zhang_reset4:  $\eta_{k,0} = \frac{2\left(f_{i_{k - 1}}(w_{k - 1}) - f_{i_{k - 1}}(w_k)\right)}{||\nabla f_{i_{k - 1}}(w_{k - 1})||}$ . adapted to SGD from Nocedal and Wright [2006],
- zhang EVERY2: same as PoNoS_reset0, but a new step is computed only every 2 iterations.
In Figure 2, we report train loss (log scale), the total amount of backtracks per epoch and the average step size within the epoch (log scale). From Figure 2, we can make the following observations:
- PoNoS and PoNoS_reset0 achieve very similar performances. In fact, the two algorithms yield step sizes that are almost always overlapping. An important difference between PoNoS_reset0 and PoNoS can be noticed in the amount of backtracks that the two algorithms require. The plots show a sum of  $\frac{M}{b}$  elements, with  $\frac{M}{b} = 391$  or 469 depending on the problem and PoNoS's line hits exactly this value. This means that PoNoS employs a median of 1 backtrack per iteration for the first 5-25 epochs, while PoNoS_reset0 needs more backtracks in this stage (around 1500-3000 per-epoch, see Section D.2 of the supplementary materials). After this initial phase, both PoNoS_reset0 and PoNoS reduce the amount of backtracks until it reaches a (almost) constant value of 0.
- zhang EVERY2 does not achieve the same good performance as PoNoS or PoNoS_reset0. The common belief that step sizes can be used in many subsequent iterations does not find confirmation here. In fact, zhang EVERY2 shows that we cannot skip the application of a line search if we want to maintain the same good performances.
- All the other initial step sizes achieve poor performances on both train loss and test accuracy. In many cases, the algorithms yield step sizes (also before the line search) that are too small w.r.t. (4).
5.2 Time Comparison
In this subsection, we show runtime comparisons corresponding to Figure 1. In the first row of Figure 3, we plot the train loss as in the first row of Figure 1. However, the  $x$ -axis of Figure 3 measures the cumulative epoch time of the average of 5 different runs of the same algorithm with different seeds. In the second row of Figure 3, we report the runtime per-epoch (with shaded error bars) on
Figure 2: Comparison between different initial step sizes and resetting techniques. Each column focus on a dataset. First row: train loss. Second row: # backtracks. Third row: step size.
the  $y$ -axis and epochs on the  $x$ -axis. From Figures 1 and 3, it is clear that PoNoS is not only faster than the other methods in terms of epochs but also in terms of total computational time. In particular, PoNoS is faster than SGD, despite the fact the second achieves the lowest per-epoch time. Again from the second row of Figure 3, we can observe that PoNoS's per-epoch time makes a transition from the phase of a median of 1 backtrack (first 5-25 epochs) to the phase of a median of 0 backtracks where its time is actually overlapping with that of SLS (always a median of 0 backtracks). In the first case, PoNoS requires less than twice the time of SGD, and in the second, this time is lower than 1.5 that of SGD. Given these measures, PoNoS becomes faster than SGD/Adam in terms of per-epoch time as soon as a grid-search (or any other hyper-parameter optimization) is employed to select the best-performing learning rate.
To conclude, let us recall that any algorithm based on a stochastic line search always requires one additional forward pass if compared with SGD. In fact, both  $f_{i_k}(w_k)$  and  $f_{i_k}(w_{k + 1})$  are computed at each  $k$  and each backtrack requires one additional forward pass. On the other hand, if we consider that one backward pass costs roughly two forward passes and that SGD needs one forward and one backward pass, any additional forward pass costs roughly one-third of SGD. These rough calculations have been verified in Section E.6 of the supplementary materials, where we profiled the single iteration of PoNoS. Following these calculations and referring to the two phases of Figure 3, one iteration of PoNoS only costs  $\frac{5}{3}$  that of SGD in the first phase and  $\frac{4}{3}$  in the second.
5.3 Experiments for Convex Losses and for Transformers
As a last benchmark, we take into account a set of convex problems from Vaswani et al. [2019], Loizou et al. [2021] and the transformers [Vaswani et al., 2017] trained from scratch in Kunstner et al. [2023]. In Figure 4 we show one convex problem (first column) and two transformers (last two column). We leave the fine-tuning of transformers to future works. Our experiments take into account binary classification problems addressed with a RBF kernel model without regularization. We show the results achieved on the dataset mushrooms, while leaving those on ijCNN, rcv1 and w8a to Section D.4 of the supplementary materials. Given the high amount of iterations, the smoothed version of the train loss will be reported. The test accuracy is also only reported in Section D.4 since (almost) all the methods achieve the best test accuracy in all the problems within the first few iterations. From the left subplot of Figure 4, we can observe that PoNoS obtains very good performances also in this setting (see supplementary materials). In Figure 4, PoNoS achieves a very low train loss  $(10^{-4})$  within the first 200 iterations. Only SLS is able to catch up, but this takes 6 times the iterations of PoNoS. On this problem, SLS is the only method reaching the value  $(10^{-6})$ . The methods SLS and SPS behave
Figure 3: Time comparison (s) between the proposed method (PoNoS) and the-state-of-the-art. Each column focus on a dataset. First row: train loss vs runtime. Second row: epoch runtime.
very similarly on all the datasets (see the supplementary materials) since in both cases (3) controls the step size. As clearly shown by the comparison with PoNoS, this choice is suboptimal and the Polyak step size is faster. Because of (3), both SLS and SPS encounter slow convergence issues in many of problems of this setting. As in Figure 1, SGD and Adam are always slower than PoNoS.
We consider training transformers on language modeling datasets. In particular, we train a Transformer Encoder [Vaswani et al., 2017] on PTB [Marcus et al., 1993] and a Transformer-XL [Dai et al., 2019] on Wikitext2 [Merit et al., 2017]. In contrast to the case of convolutional neural networks, the most popular method for training transformers is Adam and not SGD [Pan and Li, 2022, Kunstner et al., 2023]. For this reason, we use a preconditioned version of PoNoS, SLS, and SPS (respectively PoNoS_prec, SLS_prec, and SPS_prec, see Section D.5 of the supplementary materials for details). In fact, Adam can be considered a preconditioned version of SGD with momentum [Vaswani et al., 2020]. As in Kunstner et al. [2023], we focus on the training procedure and defer the generalization properties to the supplementary materials. From Figure 4, we can observe that the best algorithms in this setting are preconditioned-based and not SGD-based, in accordance with the literature. Moreover, PoNoS_prec achieves similar performances as Adam's. In particular from the central subplot of Figure 4, we observe that PoNoS_prec is the only algorithm as fast as Adam, while all the others have difficulties achieving loss below 1. Taking the right subplot of Figure 4 into account, we can notice that PoNoS_prec is slower than Adam on this problem. On the other hand, there is not one order of difference between the two final losses (i.e.,  $\sim$  1.5 points). Moreover, we should keep in mind that Adam's learning rate has been fine-tuned separately for each problem, while PoNoS_prec has been used off-the-shelf.
Figure 4: Train loss comparison between the new method (PoNoS) and the state-of-the-art on convex kernel models (first column) and transformers (last two columns).
In this work, we showed that modern DL models can be efficiently trained by nonmonotone line search methods. More precisely, nonmonotone techniques have been shown to outperform the monotone line searches existing in the literature. A stochastic Polyak step size with resetting has been employed as the initial step size for the nonmonotone line search, showing that the combined method is faster than the version without line search. Moreover, we presented the first runtime comparison
between line-search-based methods and SGD/Adam. The results show that the new line search is overall computationally faster than the state-of-the-art. A new resetting technique is developed to reduce the amount of backtracks to almost zero on average, while still maintaining a large initial step size. To conclude, the similar behavior of SLS_prec and Adam on the rightmost subplot of Figure 4 suggests that other initial step sizes might also be suited for training transformers. We leave such exploration (e.g., a stochastic BB like in Tan et al. [2016], Liang et al. [2019]) to future works.
We proved three convergence rate results for stochastic nonmonotone line search methods under interpolation and under either strong-convexity, convexity, or the PL condition. Our theory matches its monotone counterpart despite the use of a nonmonotone term. In the future, we plan to explore the conditions of the theorems in Liu et al. [2022] and to study a bridge between their local PL results and our global PL assumption. To conclude, it is worth mentioning that nonmonotone line search methods might be connected to the edge of stability phenomenon described in Cohen et al. [2021], because of the very similar behavior they induce in the decrease of (deterministic) objective functions. However, a rigorous study remains for future investigations.
The work was conducted within the KI-Starter project "Robustness and Generalization in Training Deep Neural Networks" funded by the Ministry of Culture and Science Nordrhein-Westfalen, Germany and partially supported by the Canada CIFAR AI Chair Program.
Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of Mathematics, 16(1):1-3, 1966. Cited on 1
Hilal Asi and John C Duchi. The importance of better models in stochastic optimization. Proceedings of the National Academy of Sciences, 116(46):22924-22930, 2019. Cited on 2
Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA Journal of Numerical Analysis, 8(1):141-148, 1988. Cited on 2
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations, 2017. Cited on 1
Stefania Bellavia, Natasa Krejic, Benedetta Morini, and Simone Rebegoldi. A stochastic first-order trust-region method with inexact restoration for finite-sum minimization. arXiv preprint arXiv:2107.03129, 2021. Cited on 2, 3
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Training neural networks for and by interpolation. In International Conference on Machine Learning, pages 799-809. PMLR, 2020. Cited on 2, 5
E. G. Birgin, J. M. Martínez, and M. Raydan. Nonmonotone spectral projected gradient methods on convex sets. SIAM Journal on Optimization, 10(4):1196-1211, 2000. Cited on 2
R. Bollapragada, D. Mudigere, J. Nocedal, H.-J.M. Shi, and P.T.P. Tang. A progressive batching L-BFGS method for machine learning. In International Conference on Machine Learning, volume 2, pages 989-1013, 2018. Cited on 1
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223-311, 2018. Cited on 4, 5
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. Cited on 1, 10
Yu-Hong Dai. On the nonmonotone line search. Journal of Optimization Theory and Applications, 112(2):315-330, 2002. Cited on 3
Conghui Tan, Shiqian Ma, Yu-Hong Dai, and Yuqiu Qian. Barzilai-Borwein step size for stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 685-693, 2016. Cited on 10
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019. Cited on 6
Tuyen Trung Truong and Hang-Tuan Nguyen. Backtracking gradient descent method and some applications in large scale optimisation. Part 2: Algorithms and experiments. Applied Mathematics & Optimization, 84(3):2557-2586, 2021. Cited on 1
Tuyen Trung Truong and Tuan Hang Nguyen. Backtracking gradient descent method for general  $C^1$  functions, with applications to deep learning. arXiv preprint arXiv:1808.05160, 2018. Cited on 1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Cited on 3, 8, 9
Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. In Advances in Neural Information Processing Systems, pages 3732-3745, 2019. Cited on 1, 2, 3, 4, 5, 6, 8
Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search). arXiv preprint arXiv:2006.06835, 2020. Cited on 1, 9
Yifei Wang, Zeyu Jia, and Zaiwen Wen. The search direction correction makes first-order methods faster. arXiv preprint arXiv:1905.06507, 2019. Cited on 2
Yifei Wang, Zeyu Jia, and Zaiwen Wen. Search direction correction with normalized gradient makes first-order methods faster. SIAM Journal on Scientific Computing, 43(5):A3184-A3211, 2021. Cited on 2
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In *British Machine Vision Conference* 2016. British Machine Vision Association, 2016. Cited on 6
Hongchao Zhang and William W Hager. A nonmonotone line search technique and its application to unconstrained optimization. SIAM Journal on Optimization, 14(4):1043-1056, 2004. Cited on 1, 2, 3, 4, 5
Jian L Zhou and AL Tits. Nonmonotone line search for minimax problems. Journal of Optimization Theory and Applications, 76(3):455-476, 1993. Cited on 2
Weijun Zhou. A short note on the global convergence of the unmodified PRP method. Optimization Letters, 7(6):1367-1372, 2013. Cited on 2