Title: Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: line search

nonmonotone

stochastic gradient descent

over-parametrized models

Polyak step size

optimization
Abstract: Recent works have shown that line search methods can speed up Stochastic Gradient Descent (SGD) and Adam in modern over-parameterized settings. However, existing line searches may take steps that are smaller than necessary since they require a monotone decrease of the (mini-)batch objective function. We explore nonmonotone line search methods to relax this condition and possibly accept larger step sizes. Despite the lack of a monotonic decrease, we prove the same fast rates of convergence as in the monotone case. Our experiments show that nonmonotone methods improve the speed of convergence and generalization properties of SGD/Adam even beyond the previous monotone line searches. We propose a POlyak NOnmonotone Stochastic (PoNoS) method, obtained by combining a nonmonotone line search with a Polyak initial step size. Furthermore, we develop a new resetting technique that in the majority of the iterations reduces the amount of backtracks to zero while still maintaining a large initial step size. To the best of our knowledge, a first runtime comparison shows that the epoch-wise advantage of line-search-based methods gets reflected in the overall computational time.

Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models

Abstract

Recent works have shown that line search methods can speed up Stochastic Gradient Descent (SGD) and Adam in modern over-parameterized settings. However, existing line searches may take steps that are smaller than necessary since they require a monotone decrease of the (mini-)batch objective function. We explore nonmonotone line search methods to relax this condition and possibly accept larger step sizes. Despite the lack of a monotonic decrease, we prove the same fast rates of convergence as in the monotone case. Our experiments show that nonmonotone methods improve the speed of convergence and generalization properties of SGD/Adam even beyond the previous monotone line searches. We propose a POlyak NOnmonotone Stochastic (PoNoS) method, obtained by combining a nonmonotone line search with a Polyak initial step size. Furthermore, we develop a new resetting technique that in the majority of the iterations reduces the amount of backtracks to zero while still maintaining a large initial step size. To the best of our knowledge, a first runtime comparison shows that the epoch-wise advantage of line-search-based methods gets reflected in the overall computational time.

1 Introduction

Stochastic Gradient Descent (SGD) [Robbins and Monro, 1951] is the workhorse for the whole Deep Learning (DL) activity today. Even though its simplicity and low memory requirements seem crucial for dealing with these huge models, the success of SGD is strongly connected to the choice of the learning rate. In the field of deterministic optimization [Nocedal and Wright, 2006], this problem is addressed [Armijo, 1966] by employing a line search technique to select the step size. Following this approach, a recent branch of research has started to re-introduce the use of line search techniques for training DL models [Mahsereci and Hennig, 2017, Paquette and Scheinberg, 2018, Bollapragada et al., 2018, Truong and Nguyen, 2018, Vaswani et al., 2019]. These methods have been shown to speed up SGD [Vaswani et al., 2019] and Adam [Duchi et al., 2011, Vaswani et al., 2020] in the over-parameterized regime. However, the existing stochastic line searches may take step sizes that are smaller than necessary since they require a monotone decrease in the (mini-)batch objective function. In particular, the highly non-linear non-convex landscapes of DL training losses suggest that it may be too restrictive to impose such a condition [Grippo et al., 1986]. In addition, the stochastic nature of SGD discourages imposing a monotone requirement on a function that is not directly the one we are minimizing [Ferris and Lucidi, 1994] (mini- vs full-batch function). Furthermore, the recent research on the edge of stability phenomenon [Cohen et al., 2021, Nacson et al., 2022] shows that the best performance for training DL models with Gradient Descent (GD) is obtained when large step sizes yield a nonmonotone decrease of the loss function. In this paper, we propose the use of nonmonotone line search methods [Grippo et al., 1986, Zhang and Hager, 2004] to possibly accept increases in the objective function and thus larger step sizes.

In parallel with line search techniques, many other methods [Baydin et al., 2017, Luo et al., 2019, Mutschler and Zell, 2020, Truong and Nguyen, 2021] have been recently developed to automatically

Mark Schmidt

select the step size. In Asi and Duchi [2019], Berrada et al. [2020], Loizou et al. [2021], Gower et al. [2021, 2022], Li et al. [2022], the Polyak step size from Polyak [1969] has been adapted to SGD and successfully employed to train DL models. However, in this setting, the Polyak step size may become very large and it may lead to divergence or, in more favorable cases, induce the (mini-batch) function to decrease nonmonotonically. Starting from Berrada et al. [2020], the Polyak step size has been upper-bounded to avoid divergence and 'smoothed' to prevent large fluctuations [Loizou et al., 2021]. However, this 'smoothing' technique is a heuristic and, as we will clarify below, may reduce the Polyak step more than necessary. In this paper, we instead combine the Polyak initial step from Loizou et al. [2021] with a nonmonotone line search. In fact, nonmonotone methods are well suited for accepting as often as possible a promising (nonmontone) step size, while still controlling its growth. In the deterministic setting, the same optimization recipe has been used in the seminal papers Raydan [1997] (for the Barzilai and Borwein [1988] (BB) step size) and Grippo et al. [1986] (for the unitary step in the case of Newton).

In the over-parameterized regime, the number of free parameters dominates the number of training samples. This provides modern DL models with the ability of exactly fitting the training data. In practice, over-parameterized models are able to reduce to zero the full-batch loss and consequently also all the individual losses. This is mathematically captured by the interpolation assumption, which allows SGD with non-diminishing step sizes to achieve GD-like convergence rates [Vaswani et al., 2019]. We develop the first rates of convergence for stochastic nonmonotone line search methods under interpolation. In the non-convex case, we prove a linear rate of convergence under the Polyak-Lojasiewicz (PL) condition. In fact in Liu et al. [2022], it has been shown that wide neural networks satisfy a local version of the PL condition, making this assumption more realistic for DL models than the Strong Growth Condition (SGC) [Schmidt and Le Roux, 2013].

When considering line search methods for training DL models one has to take into account that they require one additional forward step for each backtrack (internal iteration) and each mini-batch iteration requires an unknown amount (usually within 5) of backtracks. A single forward step may require up to one-third the runtime of SGD (see Section E.6 of the supplementary materials), thus, too many of them may become a computational burden. Le Roux et al. [2012] proposed a 'resetting' technique that is able to (almost) always reduce the number of backtracking steps to zero by shrinking the initial step size [Vaswani et al., 2019]. In this paper, we will show that this technique may have a negative impact on the speed of convergence, even when only employed as a safeguard (as in the 'smoothing' technique [Loizou et al., 2021]). As a third contribution, we develop a new resetting technique for the Polyak step size that reduces (almost) always the number of backtracks to zero, while still maintaining a large initial step size. To conclude, we will compare the runtime of different algorithms to show that line search methods equipped with our new technique outperform non-line-search-based algorithms.

2 Related Works

The first nonmonotone technique was proposed by Grippo et al. [1986] to globalize the Newton method without enforcing a monotonic decrease of the objective function. After that, nonmonotone techniques have been employed to speed up various optimization methods by relaxing this monotone requirement. A few notable examples are the spectral gradient in Raydan [1997], the spectral projected gradient in Birgin et al. [2000], sequential quadratic programming methods in Zhou and Tits [1993], and the Polak-Ribière-Polyak conjugate gradient in Zhou [2013].

In the pre-deep-learning era [Plagianakos et al., 2002], shallow neural networks have been trained by combining the nonmonotone method [Grippo et al., 1986] with GD. In the context of speech recognition [Keskar and Saon, 2015], one fully-connected neural network has been trained with a combination of a single-pass full-batch nonmonotone line search [Grippo et al., 1986] and SGD. The Fast Inertial Relaxation Engine (FIRE) algorithm is combined with the nonmonotone method [Zhang and Hager, 2004] in Wang et al. [2019, 2021] for solving sparse optimization methods and training logistic regression models, while a manually adapted learning rate is employed to train DL models. In Kreji´ c and Krklec Jerinki´ c [2015, 2019], Bellavia et al. [2021], the nonmonotone line search by Li and Fukushima [1999] is used together with SGD to solve classical optimization problems. In concurrent work [Hafshejani et al., 2023], the nonmonotone line search by Grippo et al. [1986] has been adapted to SGD for training small-scale kernel models. This line search maintains a nonmonotone window W (usually 10) of weights in memory and it computes the current mini-batch

function value on all of them at every iteration. The method proposed in Hafshejani et al. [2023] is computationally very expensive and impractical to train modern deep learning models due to the need to store previous weights. In this paper, we instead propose the first stochastic adaptation of the nonmonotone line search method by Zhang and Hager [2004]. Thanks to this line search, no computational overhead is introduced in computing the nonmonotone term which is in fact a linear combination of the previously computed mini-batch function values. To the best of our knowledge, we propose the first stochastic nonmonotone line search method to train modern convolutional neural networks and transformers [Vaswani et al., 2017].

Our paper is the first to combine a line search technique with the Stochastic Polyak Step size (SPS) [Loizou et al., 2021]. In fact, the existing line search methods [Vaswani et al., 2019, Paquette and Scheinberg, 2018, Mahsereci and Hennig, 2017] do not address the problem of the selection of a suitable initial step size. In Vaswani et al. [2019], the authors focus on reducing the amount of backtracks and propose a 'resetting' technique (3) that ends up also selecting the initial step size. In this paper, we consider these two problems separately and tackle them with two different solutions: a Polyak initial step size and a new resetting technique (5) to reduce the amount of backtracks. Regarding the latter, a similar scheme was presented in Grapiglia and Sachs [2021], however we modify it to be combined with an independent initial step size (e.g., Polyak). This modification allows the original initial step size not to be altered by the resetting technique.

In this paper, we extend Vaswani et al. [2019] by modifying the monotone line search proposed there with a nonmonotone line search. Our theoretical results extend the theorems presented in Vaswani et al. [2019] to the case of stochastic nonmonotone line search methods. Previous results in a similar context were given by Kreji´ c and Krklec Jerinki´ c [2015, 2019], Bellavia et al. [2021] that assumed 1) the difference between the nonmonotone and monotone terms to be geometrically converging to 0 and 2) the batch-size to be geometrically increasing. In this paper, we replace both hypotheses with a weaker assumption (i.e., interpolation) [Meng et al., 2020] and we actually prove that the nonmonotone and monotone terms converge to the same value. In Hafshejani et al. [2023], related convergence theorems are proved under the interpolation assumption for the use of a different nonmonotone line search (i.e., Grippo et al. [1986] instead of Zhang and Hager [2004]). However in Hafshejani et al. [2023], no explanation is given on how to use an asymptotic result (Lemma 4) for achieving their non-asymptotic convergence rates (Theorem 1 and 2). Addressing this problem represents one of the main challenges of adapting the existing nonmonotone theory [Grippo et al., 1986, Dai, 2002, Zhang and Hager, 2004] to the stochastic case.

In Vaswani et al. [2019], the rates provided in the non-convex case are developed under an SGC, while we here assume the more realistic PL condition [Liu et al., 2022]. In fact in Liu et al. [2022], not only wide networks are proven to satisfy PL but also networks with skip connections (e.g., ResNet [He et al., 2016], DenseNet [Huang et al., 2017]). For this result, we extend Theorem 3.6 of Loizou et al. [2021] for stochastic nonmonotone line search methods by exploiting our proof technique. In Loizou et al. [2021], various rates are developed for SGD with a Polyak step size and it is possible to prove that they hold also for PoNoS. In fact, the step size yielded by PoNoS is by construction less or equal than Polyak's. However, the rates presented in this paper are more general since they are developed for nonmonotone line search methods and do not assume the use of any specific initial step size (e.g., they hold in combination with a BB step size).

3 Methods

Training machine learning models (e.g., neural networks) entails solving the finite sum problem min w ∈ R n f ( w ) = 1 M ∑ M i =1 f i ( w ) , where w is the parameter vector and f i corresponds to a single instance of the M points in the training set. We assume that f is lower-bounded by some value f ∗ , that f is L -smooth and that it either satisfies the PL condition, convexity or strong-convexity. Vanilla SGD can be described by the step w k +1 = w k -η ∇ f ik ( w k ) , where i k ∈ { 1 , . . . , M } is one instance randomly sampled at iteration k , ∇ f ik ( w k ) is the gradient computed only w.r.t. the i k -th instance and η > 0 is the step size. The mini-batch version of this method modifies i k to be a randomly selected subset of instances, i.e., i k ⊂ { 1 , . . . , M } , with ∇ f ik ( w k ) being the averaged gradient on this subset and | i k | = b the mini-batch size. Through the whole paper we assume that each stochastic function f i k and gradient ∇ f ik ( w ) evaluations are unbiased, i.e., E ik [ f i k ( w )] = f ( w ) and E ik [ ∇ f ik ( w )] = ∇ f ( w ) , ∀ w ∈ R n . Note that E ik represents the conditional expectation w.r.t. w k , i.e., E ik [ · ] = E [ ·| w k ] . In other words, E ik is the expectation computed w.r.t. ν k , that is the

random variable associated with the selection of the sample (or subset) at iteration k (see Bottou et al. [2018] for more details).

We interpret the over-parameterized setting in which we operate to imply the interpolation property, i.e., let w ∗ ∈ argmin w ∈ R n f ( w ) , then w ∗ ∈ argmin w ∈ R n f i ( w ) ∀ 1 ≤ i ≤ M . This property is crucial for the convergence results of SGD-based methods because it can be combined either with the Lipschitz smoothness of f i k or with a line search condition to achieve the bound E ik ∥ ∇ f ik ( w k ) ∥ 2 ≤ a ( f ( w k ) -f ( w ∗ )) with a > 0 (see Lemma 4 of the supplementary materials for the proof). This bound on the variance of the gradient results in a r.h.s. which is independent of i k and, in the convex and strongly convex cases, it can be used to replace the SGC (see Fan et al. [2023] for more details).

As previously stated, our algorithm does not employ a constant learning rate η , but the step size η k is instead determined at each iteration k by a line search method. Given an initial step size η k, 0 and δ ∈ (0 , 1) , the Stochastic Line Search (SLS) condition [Vaswani et al., 2019] select the smallest l k ∈ N such that η k = η k, 0 δ l k satisfies the following condition

where c ∈ (0 , 1) and ∥ · ∥ is the Euclidean norm. Each internal step of the line search is called backtrack since the procedure starts with the largest value η k, 0 and reduces (cuts) it until the condition is fulfilled. Note that (1) requires a monotone decrease of f i k . In this paper, we follow Zhang and Hager [2004] and propose to replace f i k with a nonmonotone term C k ,

where ξ ∈ [0 , 1] , C 0 = Q 0 = 0 and C -1 = f i 0 ( w 0 ) . The value ˜ C k is a linear combination of the previously computed function values f i 0 ( w 0 ) , . . . , f i k ( w k ) and it ranges between the strongly nonmonotone term 1 k ∑ k j =0 f i j ( w j ) (with ξ = 1 ) and the monotone value f i k ( w k ) (with ξ = 0 ). The maximum in C k is needed to ensure the use of a value that is always greater or equal than the monotone term f i k ( w k ) . The activation of f i k ( w k ) instead of ˜ C k happens only in the initial iterations and it is rare in deterministic problems, but becomes more common in the stochastic case. The computation of C k does not introduce overhead since the linear combination is accumulated in C k .

Given γ > 1 , the initial step size employed in Vaswani et al. [2019] is the following

In this paper, we propose to use the following modified version of SPS [Loizou et al., 2021]

In Loizou et al. [2021], (4) is not directly employed, but equipped with the same resetting technique (3) to control the growth of ˜ η k, 0 , i.e. η k, 0 = min { ˜ η k, 0 , η k -1 γ b/M , η max } .

Given l k -1 the amount of backtracks at iteration k -1 , our newly introduced resetting technique stores this value for using it in the following iteration. In particular, to relieve the line search at iteration k from the burden of cutting the new step size l k -1 times, the new initial step size is pre-scaled by δ l k -1 . In fact, l k -1 is a good estimate for l k (see Section E.3 of the supplementary materials). However, to allow the original η k, 0 to be eventually used non-scaled, we reduce l k -1 by one. We thus redefine η k as

In Section 5, our experiments show that thanks to (5) we can save many backtracks and reduce them to zero in the majority of the iterations. At the same time, the original η k, 0 is not modified and the resulting step size η k is always a scaled version of η k, 0 . Moreover, the presence of the -1 in (5) keeps pushing δ ¯ l k towards zero, so that the initial step size is not cut more than necessary. Note that (5) can be combined with any initial step size and it is not limited to the case of (4).

4 Rates of Convergence

We present the first rates of convergence for nonmonotone stochastic line search methods under interpolation (see Section B of the supplementary materials for the proofs). The three main theorems are given under strong-convexity, convexity and the PL condition. Our results do not prove convergence only for PoNoS, but more generally for methods employing (2) and any bounded initial step size, i.e.

Many step size rules in the literature fulfill the above requirement [Berrada et al., 2020, Loizou et al., 2021, Liang et al., 2019] since when applied to non-convex DL models the step size needs to be upper bounded to achieve convergence [Bottou et al., 2018, Berrada et al., 2020].

In Theorem 1 below, we show a linear rate of convergence in the case of a strongly convex function f , with each f i k being convex. We are able to recover the same speed of convergence of Theorem 1 in Vaswani et al. [2019], despite the presence of the nonmonotone term C k instead of f i k ( w k ) . The challenges of the new theorem originate from studying the speed of convergence of C k to f ( w ∗ ) and from combining the two interconnected sequences { w k } and { C k } . It turns out that if ξ is small enough (i.e., such that b < (1 -η min µ ) ), then the presence of the nonmonotone term does not alter the speed of convergence of { w k } to w ∗ , not even in terms of constants.

Theorem 1. Let C k and η k be defined in (2) , with η k, 0 defined in (6) . We assume interpolation, f i k convex, f µ -strongly convex and f i k L i k -Lipschitz smooth. Assuming c > 1 2 and ξ < 1 ( 1+ η max η min (2 c -1) ) , we have

Comparing the above result with the corresponding deterministic rate (Theorem 3.1 from Zhang and Hager [2004]), we notice that the constants of the two rates are different. In particular, the proof technique employed in Theorem 3.1 from Zhang and Hager [2004] cannot be reused in Theorem 1 since a few bounds are not valid in the stochastic case (e.g., ∥ ∇ f ( w k +1 ) ∥ ≤ ∥ ∇ f ( w k ) ∥ or C k +1 ≤ C k -∥∇ f ik ( w k ) ∥ ). The only common aspect in these proofs is the distinction between the two different cases defining C k (i.e., being ˜ C k or f i k ( w k ) ). Moreover, in both theorems ξ needs to be small enough to allow the final constant to be less than 1.

In Theorem 2 we show a O ( 1 k ) rate of convergence in the case of a convex function f . Interestingly, in both Theorems 1 and 2 (and also in the corresponding monotone versions from Vaswani et al. [2019]), c is required to be larger than 1 2 . The same lower bound is also required for achieving Q-super-linear convergence for the Newton method in the deterministic case, but it is often considered too large in practice and the default value (also in the case of first-order methods) is 0 . 1 or smaller [Nocedal and Wright, 2006]. In this paper, we numerically tried both values and found out that c = 0 . 1 is too small since it may indeed lead to divergence (see Section E.1 of the supplementary materials).

Theorem 2. Let C k and η k be defined in (2) , with η k, 0 defined in (6) . We assume interpolation, f convex and f i k L i k -Lipschitz smooth. Given a constant a 1 such that 0 < a 1 < ( 2 -1 c ) and assuming c > 1 2 and ξ < a 1 2 , we have

In Theorem 3 below we prove a linear convergence rate in the case of f satisfying a PL condition. We say that a function f : R n → R satisfies the PL condition if there exists µ > 0 such that, ∀ w ∈ R n : ∥ ∇ f ( w ) ∥ 2 ≥ 2 µ ( f ( w ) -f ( w ∗ )) . The proof of this theorem extends Theorem 3.6 of Loizou et al. [2021] to the use of a stochastic nonmonotone line search. Again here, the presence of a nonmonotone term does not modify the constants controlling the speed of convergence ( a 2 below can be chosen arbitrarily small) as long as ξ is small enough. The conditions on c and on η max are the same as those of Theorem 3.6 of Loizou et al. [2021] (see the proof).

Theorem 3. Let C k and η k be defined in (2) , with η k, 0 defined in (6) . We assume interpolation, the PL condition on f and that f i k are L i k -Lipschitz smooth. Given 0 < a 2 := 4 µc (1 -c ) -Lmax 4 δc (1 -c ) + 1 2 η max and assuming 2 δ (1 -c ) Lmax < ¯ η min , η max < 2 δc (1 -c ) Lmax -4 µc (1 -c ) , Lmax 4 µ < c < 1 and ξ < a 2 c a 2 c + Lmax , we have

Theorem 3 is particularly meaningful for modern neural networks because of the recent work of Liu et al. [2022]. More precisely, their Theorem 8 and Theorem 13 prove that, respectively given a wide-enough fully-connected or convolutional/skip-connected neural network, the corresponding function satisfies a local version of the PL condition. As a consequence, nonmonotone line search methods achieve linear convergence for training of non-convex neural networks if the assumptions of both Theorem 3 above and Theorem 8 from Liu et al. [2022] hold and if the initial point w 0 is close enough to w ∗ .

5 Experiments

In this section, we benchmark PoNoS against state-of-the-art methods for training different overparametrized models. We focus on various deep learning architectures for multi-class image classification problems, while in Section 5.3 we consider transformers for language modelling and kernel models for binary classification. In the absence of a better estimate, we assume f i k ( w ∗ ) = 0 for all the experiments. Indeed the final loss is in many problems very close to 0 .

Concerning datasets and architectures for the image classification task, we follow the setup by Luo et al. [2019], later employed also in Vaswani et al. [2019], Loizou et al. [2021]. In particular, we focus on the datasets MNIST, FashionMNIST, CIFAR10, CIFAR100 and SVHN, addressed with the architectures MLP [Luo et al., 2019], EfficientNet-b1 [Tan and Le, 2019], ResNet-34 [He et al., 2016], DenseNet-121 [Huang et al., 2017] and WideResNet [Zagoruyko and Komodakis, 2016]. Because of space limitation, we will not show all the experiments here, but report on the first three datasets combined respectively with the first three architectures. We refer to the supplementary materials for the complete benchmark (see Figures I-XII) and the implementation details (Section C). In Section C of the supplementary materials, we also discuss the sensitivity of the results to the selection of hyper-parameters.

In Figure 1 we compare SGD [Robbins and Monro, 1951], Adam [Kingma and Ba, 2015], SLS [Vaswani et al., 2019], SPS [Loizou et al., 2021] and PoNoS. We present train loss (log scale), test accuracy and average step size within the epoch (log scale). Note that the learning rate of SGD and Adam has been chosen through a grid-search as the one achieving the best performance on each problem. Based on Figure 1, we can make the following observations:

PoNoS achieves the best performances both in terms of train loss and test accuracy. In particular, it often terminates by reaching a loss of 10 -6 and it always achieves the highest test accuracy.

SLS does not always achieve the same best accuracy as PoNoS. In terms of training loss, SLS is particularly slow on mnist|mlp and fashion|effb1 , while competitive on the others. On these two problems, both SLS and SPS employ a step size that is remarkably smaller than that of PoNoS. On the same problems, SPS behaves similarly as SLS because the original Polyak step size is larger than the 'smoothing/resetting' value (3), i.e., ˜ η k, 0 > η k -1 γ b/M and thus the step size is controlled by (3). On the other hand, the proposed nonmonotone line search selects a larger step size, achieving faster convergence and better generalization [Nacson et al., 2022].

SLS encounters some cancellation errors (e.g., around epoch 120 in cifar10|resnet34 ) that lead the step size to be reduced drastically, sometimes reaching values below 10 -7 . These events are caused by the fact that f i k ( w k ) and f i k ( w k +1 ) became numerically identical. Thanks to the nonmonotone term C k replacing f i k ( w k ) , PoNoS avoids cancellations errors by design.

SGD, Adam and SPS never achieve the best test accuracy nor the best training loss. In accordance with the results in Vaswani et al. [2019], the line-search-based algorithms outperform the others.

Because of space limitation, we defer the comparison between PoNoS and its monotone counterpart to the supplementary materials (Section E.2). There we show that PoNoS outperforms the other line search techniques, including a tractable stochastic adaptation of the method by Grippo et al. [1986].

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

mnist | mlp

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

cifar10 | resnet34

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

fashion | efficientnet-b1

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

0.950

0.955

0.960

0.965

0.970

0.975

0.980

0.985

0.990

test accuracy

mnist | mlp

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

0.86

0.88

0.90

0.92

0.94

test accuracy

cifar10 | resnet34

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

0.88

0.89

0.90

0.91

0.92

0.93

test accuracy

fashion | efficientnet-b1

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

10

1

10 0

10 1

average step size (log)

mnist | mlp

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

10

9

10

7

10

5

10

3

10

1

10 1

average step size (log)

cifar10 | resnet34

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

10

9

10

7

10

5

10

3

10

1

10 1

average step size (log)

fashion | efficientnet-b1

Adam

SPS

SGD

SLS

PoNoS

5.1 A New Resetting Technique

In this subsection, we compare different initial step sizes and resetting techniques. We fix the line search to be (2), but similar comments can be made on (1) (see the Section D.2 of the supplementary materials). In Figure 2 we compare PoNoS and PoNoS_reset0 (without (5)) with zhang_reset2 (initial step size (3)) and:

zhang_reset3: η k, 0 = η k -1 ||∇ f i k -1 ( w k -1 ) || 2 || ∇ fi k ( w k ) || 2 , adapted to SGD from Nocedal and Wright [2006],

zhang_reset4: η k, 0 = 2 ( f i k -1 ( w k -1 ) -f i k -1 ( w k ) ) ||∇ f i k -1 ( w k -1 ) || . adapted to SGD from Nocedal and Wright [2006],

zhang_every2: same as PoNoS_reset0, but a new step is computed only every 2 iterations.

In Figure 2, we report train loss (log scale), the total amount of backtracks per epoch and the average step size within the epoch (log scale). From Figure 2, we can make the following observations:

PoNoS and PoNoS_reset0 achieve very similar performances. In fact, the two algorithms yield step sizes that are almost always overlapping. An important difference between PoNoS_reset0 and PoNoS can be noticed in the amount of backtracks that the two algorithms require. The plots show a sum of M b elements, with M b = 391 or 469 depending on the problem and PoNoS's line hits exactly this value. This means that PoNoS employs a median of 1 backtrack per iteration for the first 5-25 epochs, while PoNoS_reset0 needs more backtracks in this stage (around 1500-3000 per-epoch, see Section D.2 of the supplementary materials). After this initial phase, both PoNoS_reset0 and PoNoS reduce the amount of backtracks until it reaches a (almost) constant value of 0.

zhang_every2 does not achieve the same good performance as PoNoS or PoNoS_reset0. The common belief that step sizes can be used in many subsequent iterations does not find confirmation here. In fact, zhang_every2 shows that we cannot skip the application of a line search if we want to maintain the same good performances.

All the other initial step sizes achieve poor performances on both train loss and test accuracy. In many cases, the algorithms yield step sizes (also before the line search) that are too small w.r.t. (4).

5.2 Time Comparison

In this subsection, we show runtime comparisons corresponding to Figure 1. In the first row of Figure 3, we plot the train loss as in the first row of Figure 1. However, the x -axis of Figure 3 measures the cumulative epoch time of the average of 5 different runs of the same algorithm with different seeds. In the second row of Figure 3, we report the runtime per-epoch (with shaded error bars) on

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

mnist | mlp

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

cifar10 | resnet34

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

fashion | efficientnet-b1

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

0

100

200

300

400

500

# backtracks

mnist | mlp

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

0

100

200

300

400

500

# backtracks

cifar10 | resnet34

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

0

100

200

300

400

500

# backtracks

fashion | efficientnet-b1

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

10

3

10

2

10

1

10 0

10 1

average step size (log)

mnist | mlp

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

10

3

10

2

10

1

10 0

10 1

average step size (log)

cifar10 | resnet34

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

0

25

50

75

100

125

150

175

200

10

7

10

5

10

3

10

1

10 1

average step size (log)

fashion | efficientnet-b1

zhang_reset2

zhang_reset3

zhang_reset4

zhang_every2

PoNoS

PoNoS_reset0

the y -axis and epochs on the x -axis. From Figures 1 and 3, it is clear that PoNoS is not only faster than the other methods in terms of epochs but also in terms of total computational time. In particular, PoNoS is faster than SGD, despite the fact the second achieves the lowest per-epoch time. Again from the second row of Figure 3, we can observe that PoNoS's per-epoch time makes a transition from the phase of a median of 1 backtrack (first 5-25 epochs) to the phase of a median of 0 backtracks where its time is actually overlapping with that of SLS (always a median of 0 backtracks). In the first case, PoNoS requires less than twice the time of SGD, and in the second, this time is lower than 1 . 5 that of SGD. Given these measures, PoNoS becomes faster than SGD/Adam in terms of per-epoch time as soon as a grid-search (or any other hyper-parameter optimization) is employed to select the best-performing learning rate.

To conclude, let us recall that any algorithm based on a stochastic line search always requires one additional forward pass if compared with SGD. In fact, both f i k ( w k ) and f i k ( w k +1 ) are computed at each k and each backtrack requires one additional forward pass. On the other hand, if we consider that one backward pass costs roughly two forward passes and that SGD needs one forward and one backward pass, any additional forward pass costs roughly one-third of SGD. These rough calculations have been verified in Section E.6 of the supplementary materials, where we profiled the single iteration of PoNoS. Following these calculations and referring to the two phases of Figure 3, one iteration of PoNoS only costs 5 3 that of SGD in the first phase and 4 3 in the second.

5.3 Experiments for Convex Losses and for Transformers

As a last benchmark, we take into account a set of convex problems from Vaswani et al. [2019], Loizou et al. [2021] and the transformers [Vaswani et al., 2017] trained from scratch in Kunstner et al. [2023]. In Figure 4 we show one convex problem (first column) and two transformers (last two column). We leave the fine-tuning of transformers to future works. Our experiments take into account binary classification problems addressed with a RBF kernel model without regularization. We show the results achieved on the dataset mushrooms, while leaving those on ijcnn, rcv1 and w8a to Section D.4 of the supplementary materials. Given the high amount of iterations, the smoothed version of the train loss will be reported. The test accuracy is also only reported in Section D.4 since (almost) all the methods achieve the best test accuracy in all the problems within the first few iterations. From the left subplot of Figure 4, we can observe that PoNoS obtains very good performances also in this setting (see supplementary materials). In Figure 4, PoNoS achieves a very low train loss ( 10 -4 ) within the first 200 iterations. Only SLS is able to catch up, but this takes 6 times the iterations of PoNoS. On this problem, SLS is the only method reaching the value ( 10 -6 ). The methods SLS and SPS behave

0

100

200

300

400

500

600

cumulative runtime (s)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

mnist | mlp

Adam

SPS

SGD

SLS

PoNoS

0

500

1000

1500

2000

2500

3000

3500

cumulative runtime (s)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

cifar10 | resnet34

Adam

SPS

SGD

SLS

PoNoS

0

2000

4000

6000

8000

10000

cumulative runtime (s)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

fashion | efficientnet-b1

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

runtime (s)

mnist | mlp

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

12

14

16

18

20

22

runtime (s)

cifar10 | resnet34

Adam

SPS

SGD

SLS

PoNoS

0

25

50

75

100

125

150

175

200

30

35

40

45

50

55

60

65

70

runtime (s)

fashion | efficientnet-b1

Adam

SPS

SGD

SLS

PoNoS

very similarly on all the datasets (see the supplementary materials) since in both cases (3) controls the step size. As clearly shown by the comparison with PoNoS, this choice is suboptimal and the Polyak step size is faster. Because of (3), both SLS and SPS encounter slow convergence issues in many of problems of this setting. As in Figure 1, SGD and Adam are always slower than PoNoS.

Weconsider training transformers on language modeling datasets. In particular, we train a Transformer Encoder [Vaswani et al., 2017] on PTB [Marcus et al., 1993] and a Transformer-XL [Dai et al., 2019] on Wikitext2 [Merity et al., 2017]. In contrast to the case of convolutional neural networks, the most popular method for training transformers is Adam and not SGD [Pan and Li, 2022, Kunstner et al., 2023]. For this reason, we use a preconditioned version of PoNoS, SLS, and SPS (respectively PoNoS_prec, SLS_prec, and SPS_prec, see Section D.5 of the supplementary materials for details). In fact, Adam can be considered a preconditioned version of SGD with momentum [Vaswani et al., 2020]. As in Kunstner et al. [2023], we focus on the training procedure and defer the generalization properties to the supplementary materials. From Figure 4, we can observe that the best algorithms in this setting are preconditioned-based and not SGD-based, in accordance with the literature. Moreover, PoNoS_prec achieves similar performances as Adam's. In particular from the central subplot of Figure 4, we observe that PoNoS_prec is the only algorithm as fast as Adam, while all the others have difficulties achieving loss below 1. Taking the right subplot of Figure 4 into account, we can notice that PoNoS_prec is slower than Adam on this problem. On the other hand, there is not one order of difference between the two final losses (i.e., ∼ 1 . 5 points). Moreover, we should keep in mind that Adam's learning rate has been fine-tuned separately for each problem, while PoNoS_prec has been used off-the-shelf.

0

250

500

750

1000

1250

1500

1750

2000

iter

10

6

10

5

10

4

10

3

10

2

10

1

10 0

train loss (log)

mushrooms | RBF kernel

Adam

SPS

SGD

SLS

PoNoS

0

20

40

60

80

100

10 0

10 1

train loss (log)

ptb | transformer_xl

Adam

SPS

SGD

SLS

SLS_prec

SPS_prec

PoNoS_prec

PoNoS

0

20

40

60

80

100

10 1

3×10 0

4×10 0

6×10 0

train loss (log)

wikitext2 | transformer_encoder

Adam

SPS

SGD

SLS

SLS_prec

SPS_prec

PoNoS_prec

PoNoS

6 Conclusion

In this work, we showed that modern DL models can be efficiently trained by nonmonotone line search methods. More precisely, nonmonotone techniques have been shown to outperform the monotone line searches existing in the literature. A stochastic Polyak step size with resetting has been employed as the initial step size for the nonmonotone line search, showing that the combined method is faster than the version without line search. Moreover, we presented the first runtime comparison

between line-search-based methods and SGD/Adam. The results show that the new line search is overall computationally faster than the state-of-the-art. A new resetting technique is developed to reduce the amount of backtracks to almost zero on average, while still maintaining a large initial step size. To conclude, the similar behavior of SLS_prec and Adam on the rightmost subplot of Figure 4 suggests that other initial step sizes might also be suited for training transformers. We leave such exploration (e.g., a stochastic BB like in Tan et al. [2016], Liang et al. [2019]) to future works.

We proved three convergence rate results for stochastic nonmonotone line search methods under interpolation and under either strong-convexity, convexity, or the PL condition. Our theory matches its monotone counterpart despite the use of a nonmonotone term. In the future, we plan to explore the conditions of the theorems in Liu et al. [2022] and to study a bridge between their local PL results and our global PL assumption. To conclude, it is worth mentioning that nonmonotone line search methods might be connected to the edge of stability phenomenon described in Cohen et al. [2021], because of the very similar behavior they induce in the decrease of (deterministic) objective functions. However, a rigorous study remains for future investigations.

Acknowledgments

The work was conducted within the KI-Starter project 'Robustness and Generalization in Training Deep Neural Networks' funded by the Ministry of Culture and Science Nordrhein-Westfalen, Germany and partially supported by the Canada CIFAR AI Chair Program.

References

Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of Mathematics , 16(1):1-3, 1966. Cited on 1

Hilal Asi and John C Duchi. The importance of better models in stochastic optimization. Proceedings of the National Academy of Sciences , 116(46):22924-22930, 2019. Cited on 2

Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA Journal of Numerical Analysis , 8(1):141-148, 1988. Cited on 2

Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations , 2017. Cited on 1

Stefania Bellavia, Natasa Krejic, Benedetta Morini, and Simone Rebegoldi. A stochastic firstorder trust-region method with inexact restoration for finite-sum minimization. arXiv preprint arXiv:2107.03129 , 2021. Cited on 2, 3

Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Training neural networks for and by interpolation. In International Conference on Machine Learning , pages 799-809. PMLR, 2020. Cited on 2, 5

G. Birgin, J. M. Martínez, and M. Raydan. Nonmonotone spectral projected gradient methods on convex sets. SIAM Journal on Optimization , 10(4):1196-1211, 2000. Cited on 2

Bollapragada, D. Mudigere, J. Nocedal, H.-J.M. Shi, and P.T.P. Tang. A progressive batching LBFGS method for machine learning. In International Conference on Machine Learning , volume 2, pages 989-1013, 2018. Cited on 1

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018. Cited on 4, 5

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2021. Cited on 1, 10

Yu-Hong Dai. On the nonmonotone line search. Journal of Optimization Theory and Applications , 112(2):315-330, 2002. Cited on 3

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, 2019. Cited on 9

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(7), 2011. Cited on 1

Chen Fan, Christos Thrampoulidis, and Mark Schmidt. Fast convergence of random reshuffling under over-parameterization and the polyak-\ l ojasiewicz condition. arXiv preprint arXiv:2304.00459 , 2023. Cited on 4

MCFerris and S Lucidi. Nonmonotone stabilization methods for nonlinear equations. Journal of Optimization Theory and Applications , 81(1):53-71, 1994. Cited on 1

Robert M Gower, Aaron Defazio, and Michael Rabbat. Stochastic Polyak stepsize with a moving target. arXiv preprint arXiv:2106.11851 , 2021. Cited on 2

Robert M Gower, Mathieu Blondel, Nidham Gazagnadou, and Fabian Pedregosa. Cutting some slack for SGD with adaptive polyak stepsizes. arXiv preprint arXiv:2202.12328 , 2022. Cited on 2

Geovani N Grapiglia and Ekkehard W Sachs. A generalized worst-case complexity analysis for non-monotone line searches. Numerical Algorithms , 87:779-796, 2021. Cited on 3

Luigi Grippo, Francesco Lampariello, and Stefano Lucidi. A nonmonotone line search technique for Newton's method. SIAM Journal on Numerical Analysis , 23(4):707-716, 1986. Cited on 1, 2, 3, 6

Sajad Fathi Hafshejani, Daya Gaur, Shahadat Hossain, and Robert Benkoczi. Fast armijo line search for stochastic gradient descent. Research Square preprint https://doi.org/10.21203/rs.3.rs2285238/v1 , 2023. Cited on 2, 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. Cited on 3, 6

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017. Cited on 3, 6

Nitish Shirish Keskar and George Saon. A nonmonotone learning rate strategy for sgd training of deep neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4974-4978. IEEE, 2015. Cited on 2

D.P. Kingma and J.L. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015. Cited on 6

Nataša Kreji´ c and Nataša Krklec Jerinki´ c. Nonmonotone line search methods with variable sample size. Numerical Algorithms , 68(4):711-739, 2015. Cited on 2, 3

Nataša Kreji´ c and Nataša Krklec Jerinki´ c. Spectral projected gradient method for stochastic optimization. Journal of Global Optimization , 73(1):59-81, 2019. Cited on 2, 3

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations , 2023. Cited on 8, 9

Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential convergence _rate for finite training sets. Advances in neural information processing systems , 25, 2012. Cited on 2

Donghui Li and Masao Fukushima. A globally and superlinearly convergent Gauss-Newton-based BFGS method for symmetric nonlinear equations. SIAM Journal on Numerical Analysis , 37(1): 152-172, 1999. Cited on 2

Shuang Li, William J Swartworth, Martin Takáˇ c, Deanna Needell, and Robert M Gower. SP2: A second order stochastic Polyak method. arXiv preprint arXiv:2207.08171 , 2022. Cited on 2

Jinxiu Liang, Yong Xu, Chenglong Bao, Yuhui Quan, and Hui Ji. Barzilai-Borwein-based adaptive learning rate for deep learning. Pattern Recognition Letters , 128:197-203, 2019. Cited on 5, 10

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59: 85-116, 2022. Cited on 2, 3, 6, 10

Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics , pages 1306-1314. PMLR, 2021. Cited on 2, 3, 4, 5, 6, 8

Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. In International Conference on Learning Representations , 2019. Cited on 1, 6

Maren Mahsereci and Philipp Hennig. Probabilistic line searches for stochastic optimization. The Journal of Machine Learning Research , 18(1):4262-4320, 2017. Cited on 1, 3

Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The Penn Treebank. Computational Linguistics 19.2 , pages 313-330, 1993. Cited on 9

Si Yi Meng, Sharan Vaswani, Issam Hadj Laradji, Mark Schmidt, and Simon Lacoste-Julien. Fast and furious convergence: Stochastic second order methods under interpolation. In International Conference on Artificial Intelligence and Statistics , pages 1375-1386. PMLR, 2020. Cited on 3

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017. Cited on 9

Maximus Mutschler and Andreas Zell. Parabolic approximation line search for DNNs. Advances in Neural Information Processing Systems , 33:5405-5416, 2020. Cited on 1

Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning , pages 16270-16295. PMLR, 2022. Cited on 1, 6

Jorge Nocedal and Stephen Wright. Numerical optimization . Springer Science & Business Media, 2006. Cited on 1, 5, 7

Yan Pan and Yuanzhi Li. Toward understanding why Adam converges faster than SGD for transformers. In Optimization for Machine Learning (NeurIPS Workshop) , 2022. Cited on 9

Courtney Paquette and Katya Scheinberg. A stochastic line search method with convergence rate analysis. arXiv preprint arXiv:1807.07994 , 2018. Cited on 1, 3

Vassilis P Plagianakos, George D Magoulas, and Michael N Vrahatis. Deterministic nonmonotone strategies for effective training of multilayer perceptrons. IEEE Transactions on Neural Networks , 13(6):1268-1284, 2002. Cited on 2

Boris Teodorovich Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics , 9(3):14-29, 1969. Cited on 2

Raydan. The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM Journal on Optimization , 7(1):26-33, 1997. Cited on 2

Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400-407, 1951. Cited on 1, 6

Mark Schmidt and Nicolas Le Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370 , 2013. Cited on 2

Conghui Tan, Shiqian Ma, Yu-Hong Dai, and Yuqiu Qian. Barzilai-Borwein step size for stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 685-693, 2016. Cited on 10

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019. Cited on 6

Tuyen Trung Truong and Hang-Tuan Nguyen. Backtracking gradient descent method and some applications in large scale optimisation. Part 2: Algorithms and experiments. Applied Mathematics &Optimization , 84(3):2557-2586, 2021. Cited on 1

Tuyen Trung Truong and Tuan Hang Nguyen. Backtracking gradient descent method for general C 1 functions, with applications to deep learning. arXiv preprint arXiv:1808.05160 , 2018. Cited on 1

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. Cited on 3, 8, 9

Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon LacosteJulien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. In Advances in Neural Information Processing Systems , pages 3732-3745, 2019. Cited on 1, 2, 3, 4, 5, 6, 8

Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon LacosteJulien. Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search). arXiv preprint arXiv:2006.06835 , 2020. Cited on 1, 9

Yifei Wang, Zeyu Jia, and Zaiwen Wen. The search direction correction makes first-order methods faster. arXiv preprint arXiv:1905.06507 , 2019. Cited on 2

Yifei Wang, Zeyu Jia, and Zaiwen Wen. Search direction correction with normalized gradient makes first-order methods faster. SIAM Journal on Scientific Computing , 43(5):A3184-A3211, 2021. Cited on 2

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016 . British Machine Vision Association, 2016. Cited on 6

Hongchao Zhang and William W Hager. A nonmonotone line search technique and its application to unconstrained optimization. SIAM Journal on Optimization , 14(4):1043-1056, 2004. Cited on 1, 2, 3, 4, 5

Jian L Zhou and AL Tits. Nonmonotone line search for minimax problems. Journal of Optimization Theory and Applications , 76(3):455-476, 1993. Cited on 2

Weijun Zhou. A short note on the global convergence of the unmodified PRP method. Optimization Letters , 7(6):1367-1372, 2013. Cited on 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the use of nonmonotone line search methods to speed up the optimization process of modern deep learning models, specifically Stochastic Gradient Descent (SGD) and Adam, in over-parameterized settings. The proposed method relaxes the condition of a monotonic decrease in the objective function and allows for larger step sizes. The authors introduce a new resetting technique that reduces the number of backtracks to zero while still maintaining a large initial step size. The proposed POlyak NOnmonotone Stochastic (PoNoS) method combines a nonmonotone line search with a Polyak initial step size. The paper proves the same rates of convergence as in the monotone case. The experiments show that nonmonotone methods outperform the rate of convergence and also generalization properties of SGD/Adam.\n\nSTRENGTHS:\n- **Originality:** The use of nonmonotone line search methods to relax the condition of a monotonic decrease in the objective function is a stochastic generalization of [Zhang and Hager 2004] which was proposed initially for deterministic optimization. The initial step size is chosen on the basis of previous work [Vaswani et al 2019]. The paper also introduces originally a new resetting technique that reduces the amount of backtracks to zero while still maintaining a large initial step size. Overall, the paper's originality is a significant strength. \n\n- **Quality:** The paper provides rigorous proof that the proposed nonmonotone line search method has the same rate of convergence as in the monotone case despite the lack of a monotonic decrease. The experiments show that nonmonotone method has a larger speed of convergence and better generalization properties of SGD and Adam. Computational time comparison experiments also show the outperformance of the proposed method. The theory is solid and the experimental results are strong.\n\n- **Clarity:** The paper is well-written and easy to understand, with clear explanations of technical terms and concepts. Qualitative explanations of the theorems are provided to help the readers understand the main messages. The authors provide detailed descriptions of the proposed method and the experiments conducted to evaluate its performance. Comparisons with other methods are presented clearly.\n\n- **Significance:** The proposed method shows the outperformance of existing state-of-the-art algorithms in both computational time and generalization properties.\n\nWEAKNESSES:\n- The proposed method includes many parameters to be chosen artificially, such as $\\eta_{\\rm max}$, $c$, $c_p$, $\\delta$, and $\\xi$. Although the ranges of them are provided in the theorems, influences on the performance of the proposed method due to different choices of these parameters are not clear. Are the specific values used in a real experiment not so important? If so, to what extent?\n\nQUESTIONS:\nSame as stated in the **Weakness** part.\n\nLIMITATIONS:\nThe limitations of the proposed method are stated not sufficiently. Only future perspectives are stated. For example, considering the local PL assumption. The claims that the proposed method outperforms many other the-state-of-the-art methods from several perspectives are quite strong. Are there any drawbacks, or points to be improved, of the proposed method?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a non-monotonic line search method for choosing step sizes in stochastic optimization. Convergence rates are proved for strongly convex, convex, and PL functions, and the rates match those of previous work. Experimental results show that (1) for MLPs and CNNs, the proposed algorithm outperforms SGD, Adam, and previous line search methods, and (2) for kernel models and transformers, the proposed algorithm outperforms SGD and previous line search methods, and is competitive with Adam.\n\nSTRENGTHS:\n1. The question is significant. Given the observations of the \"edge of stability\" and non-monotonic decreases in loss when training deep networks, it seems natural that incorporating non-monotonicity into line search methods may yield significant performance improvements.\n2. The presentation is clear and easy to follow.\n3. The theoretical results (Theorems 1, 2, 3) can recover convergence rates from previous work.\n4. The experimental evaluation is very broad, covering many datasets and neural network architectures.\n\nWEAKNESSES:\n1. The proposed algorithm appears to be a direct combination of existing techniques (non-monotonic line search with Polyak initial step size). While this isn't necessarily a problem in itself, as a result the technical novelty of the paper is not very high.\n2. The theoretical results recover the previous convergence rate, but they do not exhibit any improvement over baselines. Recovering the previous convergence rates is natural and the proofs don't appear to contain any new techniques. Therefore, the theoretical contribution is not significant.\n3. The main text contains no information about the tuning procedure for baselines or for the proposed algorithm, and the appendix contains very little information about hyperparameters. It's uncertain whether the experimental comparison is fair, and since the theoretical results do not exhibit improvement over baselines, the experimental performance is the only substantial contribution. Some previous baselines require additional hyperparameters (e.g. SPS with $\\gamma$), but there is no assurance that this parameter was properly tuned. It is also difficult to see whether PoNoS was tuned more extensively than baselines, which would of course not be a fair comparison.\n4. Evaluation for RBF kernel models and transformers does not compare by wall-clock time, and does not include results for test loss. Since PoNoS is competitive with Adam when measured by epochs, it is natural to assume that PoNoS lags behind Adam in terms of wall-clock time, which begs the question whether existing line search methods are useful for training Transformers. Line 344 says that only the training procedure is considered (following previous work), but this is not completely satisfying to me. If we compared test performance for MLPs, and CNNs, why not also for RBFs and Transformers?\n\nQUESTIONS:\n1. Why is interpolation important to achieve the theoretical results? The introduction discusses interpolation in detail, but Section 4 only mentions interpolation as a condition for the theorems, and does not discuss why interpolation is necessary. Do previous line search methods also require interpolation to recover the same convergence rates?\n2. How were the hyperparameters chosen? In particular, were all algorithms fairly tuned?\n2. Why does PoNoS outperform other line search methods empirically when the theoretical guarantees of PoNoS do not improve over baselines?\n3. How much do the two individual components of PoNoS (non-monotonicity and choice of initial step size) affect the performance? In particular, how would PoNoS perform if we used non-monotonicity with a classical initial step size?\n\nAs a miscellaneous suggestion, please use lower resolution images for Figures 1-3, or if you're using a vector format please remove some data points. The provided PDF is unusually large and viewing the figures in Chrome's PDF viewer is quite slow.\n\nLIMITATIONS:\nThe authors include some discussion of limitations and future work in the conclusion, though it would be nice to see some more discussion of the weaknesses of the proposed method instead of just directions for future work. Discussion of potential negative societal impact is, in my opinion, not necessary for this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a non-monotone line search method for optimizing over-parameterized models. The method is equipped with some theoretical support for strongly convex, convex and the PL condition. Furthermore, experimentally, the method is shown to have favorable performance when optimizing various deep learning models of practical interest.\n\n==> post rebuttal: increased score from 4 to 5.\n\nSTRENGTHS:\nObtaining convergence results with non-monotone line search strategies appears to be a novel contribution - though, I don't know if some variant of this result has appeared in existing literature dealing with non-monotone line search.\n\nWEAKNESSES:\n- The proposed approach seems incremental compared to existing approaches.\n- The theory doesn't adequately capture why the proposed method outperforms existing approaches. The bounds suggest identical convergence rates as ones that use monotone line search methods. This suggests all these bounds are fairly worst case (loose upper bounds) that do not help quantify why these methods work well in practice in the first place.\n\nQUESTIONS:\n- In the theorem, particularly for the PL case, the degree of (non-) monotonicity that can be tolerated seems to be rather small (the permitted \\zeta appears to be super small)? Can the authors clarify this?\n\n- The theorem for the strongly convex case doesn't appear to be particularly applicable in the over-parameterized case unless a regularized objective is being solved for, which doesn't reflect on how the behavior manifests on the loss function we truly care for.\n\n- Can the authors clarify why the proposed stepsizes tend to increase as a function of iterations? Is this related to using normalization layers (e.g. batch norm/layer norm) within the architectures? How do these stepsizes look like for convex case in the experiments conducted in the paper?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This submission proposed a new linear search method to ensure convergence without the monotone decrease condition of the (mini-)batch objective function. The method is quite suitable for the modem DNN training, which prefers the larger training learning rate.

STRENGTHS:
- The explanation of motivation is very clear.
- The related work has been extensively discussed.

WEAKNESSES:
- Discussing the difference between the proposed and the previous methods is inadequate, especially since the existing methods inspire some steps.
- The theoretical benefits of the proposed method are not shown/discussed in the convergence rate results. The proposed method seems to share a similar rate with the previous results. However, since PoNoS prefers a larger learning rate, its rate at least can demonstrate the advantage of a constant level.
- There is no numerical comparison in the main part, and just having the curve figures cannot fully display the results.

QUESTIONS:
- The experimental results are not entirely convincing. With the development of neural network training technology in recent years, the Adam optimizer can achieve better training losses than SGD in most cases. It is strange that SGD beat Adam constantly, especially on the transformer model for the NLP task.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new line search method for determining the step size in SGD within the interpolation regime. In contrast to the previous approach called SLS, which relies on the monotonically decreasing Amijo condition, the proposed method adopts the non-monotone Zhang & Hager line search. The authors establish the convergence guarantees for the proposed Stochastic Zhang & Hager line search when an upper bound is placed on the initial step size, considering strongly convex, convex, and PL problems. Additionally, they introduce several enhancements to improve empirical performance: (1) utilizing the Stochastic Polyak Step (SPS) to set the initial step size of the line search and (2) introducing a new resetting technique to reduce the number of backtracking steps.\n\nSTRENGTHS:\n- Numerical experiments are extensive and the performance of PoNoS looks quite promising. It is great to see a provably convergent optimization algorithm (with other heuristics) working well in large-scale experiments while incurring only a minor computation overhead.\n- The proof techniques look new and interesting.\n\nWEAKNESSES:\n- There are certain expressions within theorems that may lead to confusion. I have elaborated on these concerns in the \"Questions\" section of my review.\n- The Polyak step and the resetting technique are only heuristics. It would be valuable to provide further analysis regarding how these heuristics influence the convergence properties of the algorithm.\n\nQUESTIONS:\n- While reading the main text, I was initially skeptical about Theorem 1 and 2 due to their requirement of Lipschitz continuity for function $f$. It has been shown that the bounded gradient assumption contradicts the strong convexity [1]. Upon reviewing the proof in the appendix, it seems that the Lipschitz continuity of $f$ may not be necessary for Theorem 1 and 2. Additionally, in Theorem 3, it is unclear why the smoothness of $f$ needs to be separately assumed given that $f_i$ is $L_i$-smooth. It would be helpful if the authors could provide clarification.\n- Furthermore, the notation \"LC\" used in Theorem 2 and 3 lacks a definition in the main text. The corresponding definition can be found in Appendix B. \n- In theory, how can you ensure that backtracking (Lines 10 - 12) halts in finite steps?\n\n[1] Nguyen, Lam, et al. \"SGD and Hogwild! convergence without the bounded gradients assumption.\" International Conference on Machine Learning. ICML, 2018.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

comments: SUMMARY:
The paper presents a proposed Polyak nonmonotone stochastic (PoNoS) method which combines a nonmonotone line search with a Polyak initial step size. It builds on the work of Vaswani et al. [2019] by modifying the monotone line search to incorporate a nonmonotone approach.

STRENGTHS:
Originality:
The introduction of nonmonotone line search applied to Deep Learning is noteworthy.
Clarity:
The paper is generally well-written. However, the authors should emphasize their specific methodological innovation and contribution to the conclusion section to highlight the significance.

Quality: The overall quality is good. The paper demonstrates the outperformance of the proposed methods over other methods in regard to runtime.

Significance:
The experiments show the advantages of the methods in terms of convergence speed and runtime when applied to Deep Learning.

WEAKNESSES:
The authors admitted the adverse impact on convergence speed of the proposed method. This matter should be further investigated to have a more comprehensive intuition of the robustness of the method.

Additionally, the authors should conduct a further comparison with the state-of-the-art line search approaches and the based model of Vaswani et al. [2019]
?a Figurethis

QUESTIONS:
1.	Lines 12-23 are about a reduction of the number of backtracks to zero. Can you specify which parts of the experiments to support the argument.
2.	Did you have comparison with he based model of Vaswani et al. [2019]?
3.	Can you explain further or provide intuition about the observation in lines 317-319?
4.	In figure 4, data for SLS-prec, SPS-pre and PoNoS-pres seems to be missing. Do you have any reason for not to put the data?

LIMITATIONS:
The authors do address some of the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-02