Title: Adaptive Proximal Gradient Method for Convex Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: adaptive methods

gradient descent

proximal gradient method
Abstract: In this paper, we explore two fundamental first-order algorithms in convex optimization, namely, gradient descent (GD) and proximal gradient method (ProxGD). Our focus is on making these algorithms entirely adaptive by leveraging local curvature information of smooth functions. We propose adaptive versions of GD and ProxGD that are based on observed gradient differences and, thus, have no added computational costs. Moreover, we prove convergence of our methods assuming only local Lipschitzness of the gradient. In addition, the proposed versions allow for even larger stepsizes than those initially suggested in [MM20].

Adaptive Proximal Gradient Method for Convex Optimization

Abstract

In this paper, we explore two fundamental first-order algorithms in convex optimization, namely, gradient descent (GD) and proximal gradient method (ProxGD). Our focus is on making these algorithms entirely adaptive by leveraging local curvature information of smooth functions. We propose adaptive versions of GD and ProxGD that are based on observed gradient differences and, thus, have no added computational costs. Moreover, we prove convergence of our methods assuming only local Lipschitzness of the gradient. In addition, the proposed versions allow for even larger stepsizes than those initially suggested in [MM20].

1 Intro

In this paper, we address a convex minimization problem

We are interested in the cases when either F is differentiable and then we will use notation F = f , or it has a composite additive structure as F = f + g . Here, f represents a convex and differentiable function, while g is convex, lower semi-continuous (lsc), and prox-friendly. Throughout the paper, we will interchangeably refer to the smoothness of f and the Lipschitzness of ∇ f , occasionally with the adjective "locally," indicating that it is restricted to a bounded set. We will refer to this property as smoothness, without mentioning the Lipschitzness of f , so we hope there will be no confusion in this regard.

For simplicity, in most of the introduction, we consider only the simpler problem min x f ( x ) . We study one of the most classical optimization algorithms gradient descent -

Its simplicity and the sole prerequisite of knowing the gradient of f make it appealing for diverse applications. This method is central in modern continuous optimization, forming the bedrock for numerous extensions.

Given the initial point x 0 , the only thing we need to implement (1) is to choose a stepsize α k (also known as a learning rate in machine learning literature). This seemingly tiny detail is crucial for the method convergence and performance. When a user invokes GD as a solver, the standard approach would be to pick an arbitrary value for α k , run the algorithm, and observe its behavior. If it diverges at some point, the user would try a smaller stepsize and repeat the same procedure. If, on the other hand, the method takes too much time to converge, the user might try to increase the stepsize. In practice, this approach is not very efficient, as we have no theoretical guarantees for a randomly guessed stepsize, and the divergence may occur after a long time. Both underestimating and overestimating the stepsize can, thus, lead to a large overhead.

Konstantin Mishchenko

Samsung AI Center, UK

Below we briefly list possible approaches to choosing or estimating the stepsize and we provide a more detailed literature overview in Section 5.

Fixed stepsize. When f is L -smooth, GD can utilize a fixed stepsize α k = α < 2 L and values larger than 2 L will provably lead to divergence. Consequently, in such scenarios, the rate of convergence is given by f ( x k ) -f ∗ = O ( 1 αk ) , clearly indicating a direct dependence on the stepsize. Nevertheless, several drawbacks emerge from this approach:

(a) L is not available in many practical scenarios;

(b) if the curvature of f changes a lot, GD with the global value of L may be too conservative;

(c) f may be not globally L -smooth.

For illustration, consider the following functions. Firstly, when dealing with f ( x ) = 1 2 ∥ Ax -b ∥ 2 , where A ∈ ❘ n × d and b ∈ ❘ n , estimating L involves evaluating the largest eigenvalue of A ⊤ A . Second, the logistic loss f ( x ) = log(1 + exp( -ba ⊤ x )) , with a ∈ ❘ d , b ∈ {-1 , 1 } , is almost flat for large x , yet for values of x closer to 0 , it has L = 1 4 ∥ a ∥ 2 . Thus, if the solution is far from 0 , gradient descent with a constant stepsize would be too conservative. Finally, consider f ( x ) = x 4 . While this simple objective is not globally L -smooth for any value of L , on any bounded set it is smooth, and we would hope we can still minimize objectives like that.

Linesearch. Also known as backtracking in the literature. In the k -th iteration we compute x k +1 with a certain stepsize α k and check a specific condition. If the condition holds, we accept x k +1 and proceed to the next iteration; otherwise we halve α k and recompute x k +1 using this reduced stepsize. This approach, while the most robust and theoretically sound, incurs substantially higher computational costs compared to regular GD due to the linesearch procedure.

Adagrad-type algorithms. These are the methods of the type 1

where v -1 ≥ 0 is some constants, and d k is an estimate of ∥ x 0 -x ∗ ∥ for some solution x ∗ . While such methods indeed have certain nice properties, d k is usually either constant or quickly converges to a constant value, so a quick glance at (2) will reveal that its stepsizes are decreasing. Therefore, despite the name, we cannot expect true adaptivity of this method to the local curvature of f .

Heuristics. Numerous heuristics exist for selecting α k based on local properties of f and ∇ f , with the Barzilai-Borwein method [BB88] being among the most widely popular. However, it is crucial to note that we are not particularly interested in such approaches, as they lack consistency and may even lead to divergence, even for simple convex problems.

We have already mentioned adaptivity a few times, without properly introducing it. Now let us try to properly understand its meaning in the context of gradient descent. Besides the initial point x 0 , GD has only one degree of freedom - its stepsize. From the analysis we know that it has to be approximately an inverse of the local smoothness. We call a method adaptive , if it automatically adapts a stepsize to this local smoothness without additional expensive computation and the method does not deteriorate the rate of the original method in the worst case. In our case, the original method is GD with a fixed stepsize.

By this definition, GD with linesearch is not adaptive, because it finds the right stepsize with some extra evaluations of f or ∇ f . GD with diminishing steps (as in subgradient or Adagrad methods) is also not adaptive, because decreasing steps cannot in general represent well the function's curvature; also the rate of the subgradient method is definitely worse. It goes without saying, that for a good

method its rate must experience improvement when we confine the class of smooth convex functions to the strongly convex ones.

CONTRIBUTION

In a previous work [MM20], which serves as the cornerstone for the current paper, the authors proposed an adaptive gradient method named 'Adaptive Gradient Descent without Descent' (AdGD). In the current paper, we

deepen our understanding of AdGD and identify its limitations;

refine its theory to accommodate even larger steps;

extend the revised algorithm from unconstrained to the proximal case.

The analysis in the last two cases is not a trivial extension, and we were rather pleasantly surprised that this was possible at all. After all, the theory of GD is well-established and we thought it to be too well-explored for us to discover something new.

Continuous point of view. It is instructive for some time to switch from the discrete setting to the continuous and to compare gradient descent (GD) with its parent - gradient flow (GF)

where t is the time variable and x ′ ( t ) denotes the derivative of x ( t ) with respect to t . To guarantee the existence and uniqueness of a trajectory x ( t ) of GF, it is sufficient to assume that ∇ f is locally Lipschitz-continuous. Then one can prove convergence of x ( t ) to a minimizer of f in just a few lines. For GD, on the other hand, the central assumption is global Lipschitzness of ∇ f . Our analysis of gradient descent makes it level: local Lipschitzness suffices for both. Or to put it differently, we provide an adaptive discretization of GF that converges under the same assumptions as the original continuous problem (3).

Proximal case. We emphasize that there is already an excellent extension by Latafat et al. [Lat+23] of the work [MM20] to the additive composite case. Our proposed result, however, is based on an improved unconstrained analysis and uses a different (and simpler) proof. We believe that both these facts will be of interest. We don't have a good understanding why, but for us finding the proof for the proximal case was quite challenging. It does not follow the standard lines of arguments and uses a novel Lyapunov energy in the analysis.

Nonconvex problems. We believe that our algorithm will be no less important in the nonconvex case, where gradients are rarely globally Lipschitz continuous and where the curvature may change more drastically. It is true that our analysis applies only to the convex case, but, as far as we know, limited theory has never yet prevented practitioners from using methods in a broader setting. And based on our (speculative) experience, we found it challenging to identify nonconvex functions where the method did not converge to a local solution.

Outline. In Section 2, we begin by revisiting AdGD from [MM20], examining its limitations, and demonstrating a simple way to enhance it. This section maintains an informal tone, making it easily accessible for quick reading and classroom presentation. In Section 3, we further improve the method and provide all formal proofs, most of which we move to the Appendix. Section 4 extends the improved method to the proximal case. In Section 5 we put our finding in the perspective and compare it to some existing works. Lastly, in Section 6 (see also Appendix D), we conduct experiments to evaluate the proposed method against different linesearch variants.

1.1 Preliminaries

We say that a mapping is locally Lipschitz if it is Lipschitz over any compact set of its domain. A function f : ❘ d → ❘ is (locally) smooth if its gradient ∇ f is (locally) Lipschitz.

A convex L -smooth function f is characterized by the following inequality

This is equivalently of saying that ∇ f is a 1 L -cocoercive operator, that is

For a convex differentiable f that is not L -smooth one can only say that ∇ f is monotone , that is

We use notation [ t ] + = max { t, 0 } and for any a > 0 we suppose that a 0 = + ∞ . With a slight abuse of notation, we write [ n ] to denote the set { 1 , . . . , n } . A solution and the value of the optimization problem min f ( x ) are denoted by x ∗ and f ∗ , respectively.

2 Adaptive gradient descent: better analysis

Let us start with the simpler problem of min x f ( x ) with a convex, locally smooth f : ❘ d → ❘ . To solve it, in [MM20], the authors proposed a method called adaptive gradient descent without descent (AdGD), whose update is given below:

Similarly to the standard GD, this method leads to O (1 /k ) convergence rate. However, unlike the former, it doesn't require any knowledge about Lipschitz constant of ∇ f and doesn't even require a global Lipschitz continuity of ∇ f .

The update for α k has two ingredients. The first bound α k ⩽ √ 1 + θ k -1 α k -1 sets how fast steps may increase from iteration to iteration. The second α k ⩽ ∥ x k -x k -1 ∥ 2 ∥∇ f ( x k ) -∇ f ( x k -1 ) ∥ corresponds to the estimate of local Lipschitzness of ∇ f .

It is important to understand how essential these bounds are. Do we really need to control the growth rate of α k or is it an artifact of our analysis? For the second bound, it is not clear whether 2 in the denominator is necessary. For example, given L -smooth f , our scheme (7) does not encompass a standard GD with α k = 1 L for all k .

First bound. Answering the first question is relatively easy. Consider the following function

1

x

f

(

x

)

1

where parameters a, b > 0 are chosen to ensure that f ( ± 1) and f ′ ( ± 1) are well-defined, namely a = 2 and b = 2log 2 -3 2 , see Lemma 3 in Appendix A.

From an optimization point of view, f is a nice function. In particular, it is convex (even locally strongly convex) and its gradient is 1 -Lipschitz, see Lemma 3. This means that both GD and AdGD linearly converge on it. However, if we remove the first condition for α k in AdGD, this new modified algorithm will fail to converge. We can prove an even stronger statement. Specifically, let c ⩾ 1 , α 0 = 1 and consider the following method

In other words, the update in (9) is the same as in (7) except we removed the first constraint for α k in (7) and introduced a constant factor c to make the second one more general.

Theorem 1. For any c ⩾ 1 there exists x 0 such that the method (9) applied to f defined in (8) diverges.

The formal proof of this statement is in Appendix A, but its main idea should be intuitively clear. First, observe that for x with a large absolute value, f ( x ) behaves mostly like a linear function. However, f ′ ( x ) approaches -1 when x →-∞ and +1 when x → + ∞ . Therefore, if x k and x k -1 have the same sign, the local smoothness estimate will be too optimistic and x k +1 will 'leapfrog' the optimum. In contrast, if the signs of x k and x k -1 are different, then x k +1 will fail to get sufficiently close to the optimum. It is interesting to remark that on this function both versions of the Barzilai-Borwein method will diverge as well.

Consequently, the answer to the first question is affirmative: we do need some extra condition for the stepsize α k .

Second bound. The answer to the second question is the opposite: it is indeed an artifact of our previous analysis. In the next section, we propose an improvement over the previous version [MM20]. We give a concise presentation in an informal way. We keep a more formal style for section 3 where an even better version (also slightly more complicated) will be presented.

2.1 Improving AdGD

The analysis of GD usually starts from the standard identity, followed by convexity inequality

In [MM20] the only 'nontrivial' step in the proof was upper bounding α 2 k ∥∇ f ( x k ) ∥ 2 , that is ∥ x k +1 -x k ∥ 2 . Now we do it in a slightly different way. First, we need the following fact.

Lemma 1. For GD iterates ( x k ) with arbitrary positive stepsizes, it holds

Proof. This is just monotonicity of ∇ f in disguise:

Now we are going to bound ∥ x k +1 -x k ∥ 2 . For convenience, denote the approximate local Lipschitz constant as

Let α k satisfy α k ∥∇ f ( x k ) - ∇ f ( x k -1 ) ∥ ⩽ γ ∥ x k -x k -1 ∥ for some γ > 0 , that is α k L k ⩽ γ . Using ∥ u ∥ 2 = ∥ u -v ∥ 2 -∥ v ∥ 2 +2 ⟨ u, v ⟩ , we have

where the last inequality follows from convexity of f . For γ < 1 we can rewrite (12) as

Substituting this inequality into (10) gives us

As we want to telescope the above inequality, we require

On the other hand, we have already used that α k L k ⩽ γ . These two conditions lead to the bound

where γ ∈ (0 , 1) can be arbitrary. Now by playing with different values of γ , we obtain different instances of adaptive gradient descent method. For instance, by setting γ = 1 √ 2 , we get

which is a strict improvement upon the original version in [MM20]. A simple reason why this is possible is that, unlike in [MM20], we did not resort to the Cauchy-Schwarz inequality and instead relied on transformation (12) and Lemma 1.

Algorithm 1 Adaptive gradient descent

Algorithm 2 Adaptive gradient descent-2 1: Input: x 0 ∈ ❘ d , θ 0 = 1 3 , α 0 > 0 2: x 1 = x 0 -α 0 ∇ f ( x 0 ) 3: for k = 1 , 2 , . . . do 4: L k = ∥∇ f ( x k ) -∇ f ( x k -1 ) ∥ ∥ x k -x k -1 ∥ 5: α k = min {√ 2 3 + θ k -1 α k -1 , α k -1 √ [2 α 2 k -1 L 2 k -1] + } 6: x k +1 = x k -α k ∇ f ( x k ) 7: θ k = α k α k -1

We summarize the new scheme in Algorithm 1. We do not provide a formal proof for this scheme and hope that inequality (13) should be sufficient for the curious reader to complete the proof. In any case, the next section will contain a further improvement with all the missing proofs.

Remark 1. One might notice that we have used several times monotonicity of ∇ f , where we actually could use a stronger property of cocoercivity (5). That is true, but we just prefer simplicity. We recommend work [Lat+23] that exploits cocoercivity in this framework.

3 Adaptive gradient descent: larger stepsize

In this section, we modify Algorithm 1 to use even larger steps resulting in Algorithm 2. This, however, will require a slightly more complex analysis.

Recall the notation [ t ] + = max { t, 0 } and note that the second bound α k ⩽ α k -1 √ [2 α 2 k -1 L 2 k -1] + in step 5 of Algorithm 2 is equivalent to

which obviously allows for a larger range of α k than α 2 k L 2 k ⩽ 1 2 in Algorithm 1. On the other hand, the first bound α k ⩽ √ 2 3 + θ k -1 α k -1 is definitely worse. At the moment, it is not even clear whether it allows α k to increase.

Remark 2. A notable distinction between Algorithm 2 and Algorithm 1 is that the former allows to use a standard fixed step α k = 1 L , provided that f is L -smooth. For instance, if we start from α 0 = 1 L and use L ⩾ L k in every iteration (we can always use a larger value), then it follows from (15) and θ k -1 ⩾ 1 3 that α k = 1 L for all k ⩾ 1 .

Algorithm 2 requires an initial stepsize α 0 . While the algorithm converges for any value α 0 > 0 , it is important to choose initial step α 0 wisely. We suggest to do the following

The upper bound ensures that α 0 is not too large, while the lower ensures that it is not too small either. In most scenarios, this requires to run a linesearch, but we emphasize that it is only needed for the first iteration. Further discussion on this topic is in Appendix B.1.

We first prove that the sequence ( x k ) is bounded and then derive the convergence result. Both statements are proved in Appendix B.2.

Lemma 2. The sequence ( x k ) is bounded. In particular, for any solution x ∗ we have x k ∈ B ( x ∗ , R ) , where

Theorem 2. Let f be convex with a locally Lipschitz gradient ∇ f , x 0 ∈ ❘ d , and α 0 > 0 . Then the sequence ( x k ) generated by Algorithm 2 converges to a solution of min x f ( x ) and

where R is defined as in (17) . In particular, if α 0 satisfies (16) , then

where L is the Lipschitz constant of ∇ f over B ( x ∗ , R ) .

Of course, the important bound here is (18). The second bound only shows that our choice of stepsizes α k cannot be too bad. The bound in (19) is stronger than the bound √ 3 LR 2 2 k , which could be obtained as a direct consequence of Lemma 7 with simple analysis. The derivation of the sharper bound as in (19) is presented in Appendix B.3 with, unfortunately, much more involved analysis.

4 Adaptive proximal gradient method

In this section, we turn to a more general problem of composite optimization,

where g : ❘ d → ( -∞ , + ∞ ] is a proper convex lsc function and f : ❘ d → ❘ is a convex differentiable function with locally Lipschitz ∇ f . Additionally, we assume that g is prox-friendly, that is we can efficiently compute its proximal mapping prox g = (Id + ∂g ) -1 .

We present Algorithm 3 that is a verbatim adaptation of Algorithm 2 with the proximal operator applied on top of the main update (similarly, it could be applied to Algorithm 1). However, its analysis is not a straightforward generalization. We encountered two issues in the proof:

combining previous analysis of AdGD and the prox-mapping. As shown even in (13), we operate with the vectors x k and x k -1 in terms of f . However, using the prox-inequality gives us the value g ( x k +1 ) , which is not straightforward to combine with f ( x k ) and f ( x k -1 ) .

proving convergence of ( x k ) . The challenge arises from having a non-linear update due to the prox-mapping and allowing α k to go to ∞ , making the proof quite different from the traditional approach.

We define R in the same way as in (17)

where ˜ ∇ F ( x 0 ) denotes a subgradient of F at x 0 .

Algorithm 3 Adaptive proximal gradient method

Theorem 3. Let f be convex with a locally Lipschitz gradient ∇ f , g be convex lsc, x 0 ∈ ❘ d , and α 0 > 0 . Then the sequence ( x k ) generated by Algorithm 3 converges to a solution of (20) and

In particular, if α 0 satisfies (16) , then

where L is the Lipschitz constant of ∇ f over B ( x ∗ , R ) .

5 Literature and discussion

Linesearch. There are many variants of linesearch procedures that go back to celebrated works of Goldstein [Gol62] and Armijo [Arm66]. We discuss an efficient implementation of the latter in detail in the next section. For other variants of linesearch, we refer to [BN16; Sal17].

Adagrad-type methods. Original Adagrad algorithm was proposed simultaneously in [DHS11] and [MS10]. The method has had a stunning impact on machine learning applications. It has also spawned a stream of various extensions that retain the same idea of using eventually decreasing steps. Because of this, its adaptivity is more prominent in the non-smooth regime, where stepsizes must be diminishing to guarantee convergence. Recent works [DM23; IHC23] have proposed ways to increase d k in the update (2) and [KMJ23] even proved convergence of some Adagrad-type methods on smooth objectives. However, the stepsize in these methods eventually stops increasing, making them less adaptive.

In addition, Adagrad-type methods are usually sensitive to the initialization, as they either degrade in performance when d k = D and D is not chosen carefully, or their convergence rate depends multiplicatively on log( ∥ x 0 -x ∗ ∥ /d 0 ) . In contrast, in our methods, the cost of estimating α 0 to satisfy condition (16) is additive and its impact vanishes as the total number of iterations increases.

Refined results on GD with a fixed stepsize. Paper [TV22] summarizes quite well the difficulty of GD analysis with large steps. In it, the authors derive sharp convergence bounds separately for two cases αL ∈ (0 , 1] and αL ∈ (1 , 2) , and the latter case is considerably harder. In our analysis it is even harder, since the steps can go far beyond the global upper bound 2 L . A surprising recent result [Gri23] showcases how little is understood in this case.

Small gradient. The lack-of-descent property makes it hard to deduce the O (1 /k ) rate for the last-iterate ∥∇ f ( x k ) ∥ , which is known for GD with a fixed stepsize. We leave it as an open problem to establish a rate.

Extensions. Because the analysis of the algorithm is so special, it is not easy to extend it to basic generalizations of GD. However, some works have already built upon it. In [VMC21], the authors

consider a convex smooth minimization subject to linear constraints and combined the adaptive GD [MM20] with the Chambolle-Pock algorithm [CP10]. The authors of [Lat+23] went even further and considered a more general composite minimization problem subject to linear constraints, where the same two ideas as before were combined with a novel way of handling the prox mapping.

If we consider variational inequalities settings in the monotone case, then it is not clear how such adaptivity can help, since the most natural extension, the forward-backward method will diverge. Furthermore, the adaptive golden ratio algorithm [Mal19], which inspired the development of AdProxGD, already includes all the features that AdProxGD has.

6 Experiments

In the experiments 2 we compare our method to the ProxGD with Armijo's linesearch. We believe it is the best and arguably the most popular alternative to our method. An efficient implementation of Armijo's linesearch requires two parameters, s > 1 and r < 1 . In the k -th iteration, the first iteration of linesearch starts from α k = sα k -1 , that is, we want to try a slightly larger step than in the previous iteration. If linesearch does not terminate, we start decreasing a stepsize geometrically with a ratio r . Formally, we are looking for the largest α k = sr i α k -1 , for i = 0 , 1 , . . . , such that for x k +1 = prox α k g ( x k -α k ∇ f ( x k )) it holds that

It is evident that each iteration of this linesearch requires one evaluation of f and prox g . However, it is important to highlight that in some cases, the last evaluation of f ( x k +1 ) (during linesearch) may not incur any additional costs, as certain expensive operations, such as matrix-vector multiplication, can be reused to compute the next gradient ∇ f ( x k +1 ) . Throughout our comparisons, we consistently took these factors into account and reported only essential operations that cannot be further reused.

Our legend will stay the same for all plots:

where each pair of numbers represents ( s, r ) for ProxGD with linesearch described above. As we will see, the choice of ( s, r ) matters a lot. More experiments are provided in the Appendix D.

Maximum likelihood estimate of the information matrix. We consider [BV04, Equation (7.5)], where our goal is to estimate the inverse of a covariance matrix Y subject to eigenvalue bounds. Formally, this problem can be formulated as follows

where ❙ n denotes the space of n -byn symmetric matrices and A ≼ B means that B -A is positive semidefinite.

Computing projection onto the constraint set C = { X : lI ≼ X ≼ uI } requires computing matrix eigendecomposition. However, it is noteworthy that once the eigendecomposition is computed, both the objective and gradient evaluations can be carried out at a low cost. Consequently, when comparing methods, we only emphasized the number of projections conducted. We generated a random y ∈ ❘ n with entries from N (0 , 10) and δ i ∈ ❘ n with entries from N (0 , 1) , and then set y i = y + δ i , for i = 1 , . . . , M . Then we computed Y = 1 M ∑ M i =1 y i y ⊤ i . The results are presented in Figure 1. For two scenarios we generated, the proposed method converged faster than any of the linesearch versions.

Acknowledgments and Disclosure of Funding

The authors would like to thank Puya Latafat, who found a subtle error in the convergence proof of ( x k ) in Theorem 3 in the first version of this manuscript. Thanks to him, we were able to simplify our proof considerably. We also thank anonymous reviewers who found many typos and inaccuracies. Yura Malitsky's research was partially funded by the Austrian Science Fund (FWF) [10.55776/STA223].

AdProxGD

(1.2, 0.5)

(1.5, 0.8)

(1.1, 0.5)

(1.2, 0.9)

(1.1, 0.9)

(1.5, 0.5)

(1.2, 0.8)

(1.1, 0.8)

(1.5, 0.9)

(a)

n

= 100

,

l

= 0

.

1

,

u

= 10

,

M

= 50

(b)

n

= 50

,

l

= 0

.

1

,

u

= 1000

,

M

= 100

References

[Arm66] L. Armijo. 'Minimization of functions having Lipschitz continuous first partial derivatives'. In: Pacific J. Math. 16.1 (1966), pp. 1-3. DOI: 10.2140/pjm.1966.16.1 .

[BB88] J. Barzilai and J. M. Borwein. 'Two-point step size gradient methods'. In: IMA J Numer Anal 8.1 (1988), pp. 141-148. DOI: 10.1093/imanum/8.1.141 .

[BN16] J. Y. Bello Cruz and T. T. Nghia. 'On the convergence of the forward-backward splitting method with linesearches'. In: Optim. Methods Softw. 31.6 (2016), pp. 1209-1238.

[BV04] S. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2004. DOI: 10.1017/cbo9780511804441 .

[CP10] A. Chambolle and T. Pock. 'A first-order primal-dual algorithm for convex problems with applications to imaging'. In: J Math Imaging Vis 40.1 (2010), pp. 120-145. DOI: 10.1007/s10851010-0251-1 .

[DM23] A. Defazio and K. Mishchenko. 'Learning-rate-free learning by D-adaptation'. In: Proceedings of the 40th International Conference on Machine Learning . Vol. 202. 2023, pp. 7449-7479.

[DHS11] J. Duchi, E. Hazan, and Y. Singer. 'Adaptive subgradient methods for online learning and stochastic optimization'. In: J. Mach. Learn. Res. 12 (2011), pp. 2121-2159.

[Gol62] A. A. Goldstein. 'Cauchy's method of minimization'. In: Numer. Math. 4.1 (1962), pp. 146-150. DOI: 10.1007/bf01386306 .

[Gri23] B. Grimmer. 'Provably faster gradient descent via long steps'. In: (2023). arXiv: 2307.06324 .

[IHC23] M. Ivgi, O. Hinder, and Y. Carmon. 'DoG is SGD's best friend: A parameter-free dynamic step size schedule'. In: Proceedings of the 40th International Conference on Machine Learning . Vol. 202. 2023, pp. 14465-14499.

[KMJ23] A. Khaled, K. Mishchenko, and C. Jin. 'DoWG unleashed: An efficient universal parameter-free gradient descent method'. In: (2023). arXiv: 2305.16284 .

[Lat+23] P. Latafat, A. Themelis, L. Stella, and P. Patrinos. 'Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient'. In: (2023). arXiv: 2301.04431 .

[Mal19] Y. Malitsky. 'Golden ratio algorithms for variational inequalities'. In: Math. Program. 184.1-2 (2019), pp. 383-410. DOI: 10.1007/s10107-019-01416-w . arXiv: 1803.08832 .

[MM20] Y. Malitsky and K. Mishchenko. 'Adaptive gradient descent without descent'. In: Proceedings of the 37th International Conference on Machine Learning . Vol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 6702-6712. arXiv: 1910.09529 .

[MS10] H. B. McMahan and M. Streeter. 'Adaptive bound optimization for online convex optimization'. In: Proceedings of the 23rd Annual Conference on Learning Theory (COLT) . 2010.

[Sal17] S. Salzo. 'The variable metric forward-backward splitting algorithm under mild differentiability assumptions'. In: SIAM J. Optim. 27.4 (2017), pp. 2153-2181. DOI: 10.1137/16m1073741 .

[TV22] M. Teboulle and Y. Vaisbourd. 'An elementary approach to tight worst case complexity analysis of gradient based methods'. In: Math. Program. 201.1-2 (2022), pp. 63-96. DOI: 10.1007/s10107022-01899-0 .

[VMC21] M.-L. Vladarean, Y. Malitsky, and V. Cevher. 'A first-order primal-dual method with adaptivity to local smoothness'. In: NeurIPS . Vol. 34. 2021, pp. 6171-6182. arXiv: 2110.15148 .

Appendix

A Counterexample

Lemma 3. The function f defined in (8) satisfies the following properties:

f is convex.

f ′ is L -Lipschitz with L = 1 .

f is locally strongly convex, i.e., for any bounded set X there exists a constant µ X > 0 such that | f ′ ( x ) -f ′ ( y ) | ⩾ µ X | x -y | for any x, y ∈ X .

| f ′ ( x ) | ⩽ G with G = 2 .

f is 2 -Lipschitz.

Proof. First, let us find f ′ and f ′′ :

so indeed a = 2 and b = 2log 2 -3 2 . Convexity of f follows from the fact that f ′′ ( x ) > 0 for any x . Lipschitzness of f ′ follows directly from the bound f ′′ ( x ) ⩽ 1 for all x . Similarly, local strong convexity follows from the bound f ′′ ( x ) ≥ 1 max z ∈X (1+ | z | ) 2 =: µ X for any x ∈ X . Finally, the last two properties trivially follow from the expression for f ′ ( x ) . ■

Proof of Theorem 1. Let us choose x 0 = r +2 with a sufficiently large r > 6 c . This readily implies that

Our goal is to show that the iterates follow a very specific pattern. Namely, we prove that for all k ⩾ 0 ,

̸

If this condition holds true, then the sequence ( x k ) must be divergent.

First, observe that if | x k | , | x k -1 | ⩾ r and sign( x k ) = sign( x k -1 ) , then the smoothness estimate admits a simple expression:

̸

Therefore, in that case α k | f ′ ( x k ) | > r (1+ | x k | ) 2 c | f ′ ( x k ) | = r | x k | c > 3 | x k | . Since sign( f ′ ( x k )) = sign( x k ) , it implies that | x k +1 | > 2 | x k | and sign( x k +1 ) = sign( x k ) .

̸

Next, if | x k | , | x k -1 | ⩾ r > 3 with | x k | ⩾ 2 | x k -1 | and sign( x k ) = sign( x k -1 ) , then we have 2 | x k | 1+ | x k | > 3 2 and

This implies α k < | x k | 2 c ⩽ | x k | 2 . Since sign( f ′ ( x k )) = sign( x k ) and α k 1+ | x k | ⩽ | x k | 2(1+ | x k | ) < 1 2 , we conclude that sign( x k +1 ) = sign( x k ) and

As x 0 and x 1 satisfy the first case, by induction we deduce that all iterates ( x k ) follow the described pattern. ■

B Analysis of Algorithm 2

B.1 Initial stepsize (expanded discussion)

Algorithm 2 requires an initial stepsize α 0 . While the algorithm converges for any value α 0 > 0 with the rate

(see eq. (17) for the definition of R ), the choice of α 0 will impact further steps due to the bound α k ⩽ √ 2 / 3 + θ k -1 α k -1 . Because of this reason, we do not want to choose α 0 too small. On the other hand, too large α 0 will make R large. To counterbalance these two extremes, we suggest to do the following:

The upper bound ensures that α 0 is not too large, while the lower ensures that it is not too small either. In most scenarios, this requires to run a linesearch, but we emphasize one more time: it is only needed for the first iteration. In some sense, our condition (16) is similar to classical Goldstein's rule [Gol62] on selecting the stepsize: not too small and not too big.

Of course, if we start with a very small α 0 , only the first bound for α k will be active for some time, and we will eventually reach a reasonable range for a stepsize. However, linesearch with a more aggressive factor (say, 10 ) will allow us to reach this range faster. If we start with α 0 = 10 -8 when in fact a reasonable range for steps in this region is [1 , 10] , then we will need at least 100 iterations of our method, while linesearch with a factor 10 will find it in less than 10 iterations.

It may happen that the problem is degenerated in the sense that for any α 0 , α 0 L 1 < 1 √ 2 . In other words, increasing α 0 leads to decreasing L 1 and linesearch may never stop. In this case we should terminate a linesearch after α 0 reaches any prescribed value, say 1 .

B.2 Analysis of Algorithm 2

Lemma 4. For iterates ( x k ) of Algorithm 2 it holds

Before we continue, let us give some intuition for this lemma. Its analysis follows mostly the same steps as in (12). However, now we will split α 2 k ∥∇ f ( x k -1 ) ∥ 2 into two parts and use one of it to improve the smoothness bound for α k .

Proof. We start from the third line in (12) and then apply the above-mentioned splitting:

Convexity of f completes the proof.

■

Lemma 5. For iterates ( x k ) of Algorithm 2 and any solution x ∗ it holds

Proof. From (26) we have

Using this inequality in (10), we get

which is equivalent to (28).

■

Proof of Lemma 2. The first bound for α k in Algorithm 2 gives us 3 α k θ k ⩽ (2 + 3 θ k -1 ) α k -1 . We use it in (28) and telescope then to obtain

This immediately implies that ( x k ) is bounded, but we would like to obtain the bound without an intermediate iterate x 1 . From (10) we know that

Combining it with (30), we deduce

Using that θ 0 = 1 3 completes the proof.

■

Remark 3. We could have used θ 0 = 0 as we did in Algorithm 1 which would have improved the final constant R . However, since the first bound for α k is worse this time, we would need a more complicated initial bound for α 0 . We decided to keep it simple.

Notation. For brevity, we write α k ↦→ 1 to denote that in the k -th iteration α k satisfies the first bound, that is α k = √ 2 3 + θ k -1 . Similarly, for α k ↦→ 2 . Also let L be the Lipschitz constant of ∇ f over the set B ( x ∗ , R ) . This means that L k ⩽ L for all k .

Next few statements are not very important for the first reading, as they only concern with a lower bound of ∑ k i =1 α i . The main statement in Theorem 2 is valid independently of them, so the reader can go directly there.

Lemma 6. If α k ↦→ 2 , then α k ⩾ 1 √ 2 L and α k -1 + α k ⩾ 2 L .

Proof. Note that in this case α k = α k -1 √ 2 α 2 k -1 L 2 k -1 , and hence 1 α 2 k -1 + 1 α 2 k = 2 L 2 k . This implies that α k ⩾ 1 √ 2 L k ⩾ 1 √ 2 L . By AM-GM inequality,

and the conclusion α k -1 + α k ⩾ √ 8 2 L 2 k = 2 L k follows. ■

Lemma 7. If α 0 satisfies (16), then α k ⩾ 1 √ 3 L for all k ⩾ 1 .

Proof. We use induction. For k = 1 , we have either α 1 = √ 2 3 + θ 0 α 0 ⩾ 1 √ 2 L or α 1 ↦→ 2 , which in view of Lemma 6 also implies α 1 ⩾ 1 √ 2 L .

Suppose that α k -1 ⩾ 1 √ 3 L and we must show that α k ⩾ 1 √ 3 L . If α k ↦→ 2 , then we are done. Therefore, suppose that α k ↦→ 1 . Consider two options for α k -1 . If α k -1 ↦→ 1 , then θ k -1 ⩾ √ 2 / 3 . Thus, for α k we have that

If α k -1 ↦→ 2 , then α k -1 ⩾ 1 √ 2 L and hence

which completes the proof.

■

Remark 4. It is clear from above proof that condition α 0 ⩾ 1 √ 2 L 1 from (16) was used only to give us the basis for induction. Without that condition, one can still show in the same way that α k ⩾ min { α 0 , 1 √ 3 L } .

Summing this result from 1 to k yields ∑ k i =1 α i ⩾ k √ 3 L . The stepsize in the previous section is lower bounded by a k √ 2 L , so it is natural to wonder: why does the current section contain a 'larger stepsize'? The answer is that while we cannot show that each individual step is larger, we still show in the next theorem that its total length will be lower bounded by the same quantity.

Proof of Theorem 2. We proceed in the same way as in Lemma 2, but this time we keep all the terms that were discarded earlier. Specifically, summing (28) over all k yields

where the last two bounds follow from the same arguments as in Lemma 2. Note that each factor ( α k (2 + 3 θ k ) -3 α k +1 θ k +1 ) is nonnegative and their sum is

Hence, we readily obtain that

In particular, if α 0 satisfies (16), then inequality (19) is a direct consequence of Lemma 11, which we prove in the next section.

It remains to prove that ( x k ) converges to a solution. The next arguments will be similar to the ones in [MM20]. We have already proved that ( x k ) is bounded. As f is L -smooth over B ( x ∗ , R ) , we have

Using this sharper bound instead of plain convexity in (10) and repeating the same arguments as in Lemma 5, we end up with the same inequality plus the extra term

Now, by telescoping this inequality we infer that ∑ k i =1 α i L ∥∇ f ( x i ) ∥ 2 ⩽ R 2 . Since the sequence ( α k ) is separated from 0 (note that this is independent of condition (16) by Remark 4), we conclude that ∇ f ( x k ) → 0 as k → ∞ . Hence, all limit points of ( x k ) are solutions. Applying 3 θ k α k ⩽ (2 + 3 θ k -1 ) α k -1 in (32) we get

where b k = ∥ x k -x k -1 ∥ 2 + α k -1 (2 + 3 θ k -1 )( f ( x k -1 ) -f ∗ ) . Then the convergence of ( x k ) to a solution follows from the standard Opial-type arguments. ■

B.3 Better bounds for the sum of stepsizes

In this section, we prove the bound ∑ k i =1 α i ⩾ k √ 2 L .

Lemma 8. If θ k < 1 3 , then α k ↦→ 2 and α k -1 L k > √ 5 , α k -2 L k ⩾ 3 2 , α k -3 L k ⩾ 1 .

Proof. By definition, α k ↦→ 1 means that α k = √ 2 3 + θ k -1 α k -1 and thus θ k ⩾ √ 2 3 . Hence, α k ↦→ 2 . Then we have that θ k = 1 √ 2 α 2 k -1 L 2 k -1 < 1 3 which implies α k -1 L k > √ 5 . Since we get a large α k -1 , the first bound on stepsizes does not allow previous steps to be much smaller. That is the idea we shall use.

For any k , we have that θ k ⩽ √ 2 3 + θ k -1 . As θ 0 ⩽ 1 , it is trivial to prove that θ k ⩽ 1+ √ 11 3 2 =: t 0 , which is the root of t -√ 2 3 + t = 0 . From α k -1 L k > √ 5 , it follows that

Hence, to prove α k -2 L k ⩾ 3 2 , it only remains to check that √ 5 t 0 ⩾ 3 2 .

Similarly, we have

And to prove α k -3 L k ⩾ 1 , we must check that 3 2 t 0 ⩾ 1 .

■

Given the sequence ( α k ) k ≥ 1 , we call its element α m a breakpoint , if θ m < 1 3 and α m < 1 L . The next lemma says that a small step can only occur shortly after a breakpoint.

Lemma 9. If α k < 1 √ 2 L , then exactly one of the following holds

(i) α k -1 is a breakpoint;

(ii) α k -1 < α k and α k -2 is a breakpoint.

Proof. In view of Lemma 6, the statement implies that α k ↦→ 1 . Suppose that α k -1 is not a breakpoint, since otherwise we are done. This means that either (a) α k -1 ⩾ 1 L or (b) α k -1 < 1 L and θ k -1 ⩾ 1 3 . In the first case we immediately get a contradiction, since α k = √ 2 3 + θ k -1 α k -1 ⩾ √ 2 3 1 L > 1 √ 2 L . Then if we consider (b), the bound θ k -1 ⩾ 1 3 implies that α k -1 ⩽ α k < 1 √ 2 L . Then we can apply the same arguments as above, but to α k -1 . This means that either α k -2 will be a breakpoint or we will have a chain α k -2 ⩽ α k -1 ⩽ α k < 1 √ 2 L . However, the latter option cannot occur, because using θ k -1 ⩾ 1 and α k -1 ⩾ 1 √ 3 L ensure us that

Although a breakpoint indicates that we are in the region with a small stepsize, Lemma 8 guarantees that previous steps were quite large. The next lemma shows that in total we make significant progress.

Lemma 10. If α m is a breakpoint, then ∑ 2 j = -2 α m + j > 5 L .

Proof. If α m is a breakpoint, then on one hand Lemma 8 implies that α m -1 ⩾ √ 5 L m , α m -2 ⩾ 3 2 L m . On the other hand, we have that α m ⩾ 1 √ 2 L , α m +1 ⩾ 1 √ 3 L , and α m +2 ⩾ 1 √ 3 L . Combining, we get

Lemma 11. If α 0 satisfies (16), then for any k ⩾ 1 we have

Proof. Let M = { m is a breakpoint: α m +1 < 1 √ 2 L } . We can split ∑ k i =1 α i into two terms as

We claim that elements in the 'rest' are greater or equal than 1 √ 2 L . Indeed, if α i < 1 √ 2 L is in the 'rest' term, then either α i -1 is a breakpoint or α i -1 < 1 √ 2 L and α i -2 is a breakpoint, as Lemma 9 suggests. In either case, α i must be included in the first sum, by the definition of M .

Now let us estimate both terms. The first sum in (34) is greater than 5 |M| L > 5 |M| √ 2 L , by Lemma 10. The total sum in the 'rest' term is not less than k -5 √ |M| 2 L . Hence, the desired inequality follows. It has to be only noted that if k -1 ∈ M , we have to additionally consider the sum ∑ 1 j = -2 α k -1+ j ⩾ 4 √ 2 L , for which the bound follows from the same arguments as in Lemma 10. ■

Remark 5. It is obvious that our analysis was not optimal. For instance, whenever α k ↦→ 2 , we could use α k -1 + α k ⩾ 2 L instead of more conservative 2 √ 2 L . Similarly, we got a much better bound for every breakpoint. However, we did not want to overcomplicate an already tedious examination. We leave it as an open question if one can provide a bound closer to k L (or better?) with a readable proof.

C Adaptive proximal gradient method

Recall that the second bound for the stepsize α k is equivalent to

We can rewrite x k +1 = prox α k g ( x k -α k ∇ f ( x k )) as an implicit equation

where ˜ ∇ g ( x k +1 ) is a certain subgradient of g at x k +1 , that is ˜ ∇ g ( x k +1 ) ∈ ∂g ( x k +1 ) . For this particular subgradient we will also use the notation

First, we adapt our basic inequality (10) to the more general case. By prox-inequality, we have

■

Then we set x = x ∗ above and transform it into

This standard inequality is at the heart of the analysis of the proximal gradient method. To complete the full proof, or rather to get the final inequality, the classical analysis only requires applying one convexity inequality and one descent lemma to function f . Our analysis, however, will be different. The main nuisance is that in the k -th iteration the proximal map yields us g ( x k +1 ) -g ( x ∗ ) term, while our adaptivity approach works with f ( x k ) -f ( x ∗ ) , as we remember from before. Thus, our first obstacle is to understand how to combine these two terms.

First, we estimate the term ⟨∇ f ( x k ) , x ∗ -x k +1 ⟩ in the RHS of (38). We have

where in the last inequality we used separately convexity of f and g . Applying this inequality in (38) yields

As we see, the final inequality is very much in the spirit of (10).

Lemma 12 (Compare to Lemma 1) . For iterates ( x k ) with arbitrary stepsizes, it holds

Proof. As before, this is just monotonicity of ∇ f in disguise:

The next lemma is special for the composite case. Although it looks like this fact should be known in the literature, we were not able to identify it.

Lemma 13. For iterates ( x k ) of the proximal gradient method with arbitrary stepsizes, it holds

Proof. This time it is just a monotonicity of ∂g in disguise:

where the last inequality follows from monotonicity of ∂g and (36). ■

In Section 3 we estimated ∥ x k +1 -x k ∥ 2 = α 2 k ∥∇ f ( x k ) ∥ 2 . This time, ∥ x k +1 -x k ∥ 2 and α 2 k ∥ ˜ ∇ F ( x k ) ∥ 2 are different and it is the latter term that matters to us.

Lemma 14 (Compare to Lemma 4) . For iterates ( x k ) of Algorithm 3 it holds

Proof. The main idea of the proof is exactly the same as in Lemma 4. However, the presence of ˜ ∇ g ( x k ) make it slightly more cumbersome. The previous two lemmata are instrumental on our way. We have

Convexity of F completes the proof.

■

Lemma 15 (Compare to Lemma 5) . For iterates ( x k ) of Algorithm 3 and any solution x ∗ it holds

Proof. The same as in Lemma 5.

Recall that we define R as

Lemma 16. The sequence ( x k ) is bounded. In particular, for any solution x ∗ of (20) we have x k ∈ B ( x ∗ , R ) .

Proof. The same as in Lemma 2. We use (42) to telescope until k = 1 and then apply (39) with k = 0 to bound ∥ x 1 -x ∗ ∥ 2 . ■

Proof of Theorem 3. The proof of inequalities (22) and (23) is almost identical to the one in Theorem 2. The proof of convergence of ( x k ) to a solution is, however, more nuanced. The nontrivial part is to show that all limit points of ( x k ) are solutions. While on the surface, it should be no harder than before, the fact that lim k → + ∞ α k can be + ∞ complicates things a bit.

Let x ∗ be a solution of (20). By L -smoothness of f over B ( x ∗ , R ) , we have

Using this improved bound, similarly to how it was done in (32), we get

■

By telescoping this inequality as before, we can now additionally infer that

and thus, ∥∇ f ( x k ) -∇ f ( x ∗ ) ∥ → 0 . Specifically, this implies ∇ f ( x k ) -∇ f ( x k -1 ) → 0 as k →∞ .

We want to prove that all limit points of ( x k ) are solutions. To this end, we will use prox-inequality (37) rewritten as

which in turn, by convexity of f , leads to

The left-hand side has two terms, and the second term evidently tends to 0 as ∇ f ( x k +1 ) -∇ f ( x k ) → 0 . If we can show the same for the first term, it will imply that all limit points of ( x k ) are solutions.

Consider (46) again, but this time we set x = x k . This yields

We manipulate the inequality above as follows

where in the last inequality we applied Cauchy-Schwarz and Young's inequalities. From this we deduce that

Note that the sequence ( α k ∥∇ f ( x k ) -∇ f ( x ∗ ) ∥ 2 ) is summable by (45). Also, the sequence ( δ k ) is summable, since ( x k ) is bounded and g ( x k ) is lower-bounded: g ( x k ) ⩾ F ∗ -f ( x k ) > -∞ for all k . Hence, ∑ k 1 α k ∥ x k +1 -x k ∥ 2 < + ∞ and, thus, 1 α k ∥ x k +1 -x k ∥ 2 → 0 as k → + ∞ . Given that ( α k ) is separated from zero, it immediately follows that 1 α k ∥ x k +1 -x k ∥ → 0 as well.

Therefore, we have proved that all limit points of ( x k ) are solutions. The proof of convergence of the whole sequence ( x k ) runs as before in Theorem 2. ■

Remark 6. We didn't derive a linear convergence of the adaptive proximal gradient, when F is strongly convex. We only mention that it is quite straightforward and goes along the same lines as the original AdGD in [MM20, Theorem 2] in the strongly convex regime.

D Additional experiments

Low-rank matrix completion. We consider a famous low-rank matrix completion problem in the form

where Ω is a subset of indices ( i, j ) and r is the supposed maximum rank. To project onto the spectral ball C = { X : ∥ X ∥ ∗ ⩽ r } , computing the singular value decomposition (SVD) is required, making it the most computationally expensive operation in this setting.

We created matrix A by multiplying matrices U and V ⊤ , where U and V are n -byr matrices with entries sampled from a normal distribution. The subset Ω was randomly chosen as a fraction of 1 5 n 2 entries from [ n ] × [ n ] . The obtained results are depicted in Figure 2, where we solely compared the number of computed SVDs. For two scenarios we generated, the proposed method was always faster than any of the linesearch versions.

AdProxGD

(1.2, 0.5)

(1.5, 0.8)

(1.1, 0.5)

(1.2, 0.9)

(1.1, 0.9)

(1.5, 0.5)

(1.2, 0.8)

(1.1, 0.8)

(1.5, 0.9)

(a)

n

= 100

,

r

= 20

(b)

n

= 200

,

r

= 20

Minimal length piecewise-linear curve subject to linear constraints. We consider [BV04, Example 10.4], where we want to minimize the length of a piecewise-linear curve passing through n points in R 2 with coordinates (1 , x 1 ) , . . . , ( n, x n ) while satisfying linear constraints Ax = b , where x = ( x 1 , . . . , x n ) . Given A ∈ ❘ m × n and b ∈ ❘ m , this can be modeled as

While applying the proximal gradient method, the most computationally expensive operation is computing the projection onto C = { x : Ax = b } . Assuming that A is full rank with m ⩽ n , this projection can be computed as P C z = z -A ⊤ ( AA ⊤ ) -1 ( Az -b ) .

In comparison, we focused solely on the number of computed projections. We generated a random m -byn matrix A and random vector w with entries sampled from a normal distribution and set b = Aw . In Figure 3, we can see again that the proposed method converged faster than any of the linesearch versions.

AdProxGD

(1.2, 0.5)

(1.5, 0.8)

(1.1, 0.5)

(1.2, 0.9)

(1.1, 0.9)

(1.5, 0.5)

(1.2, 0.8)

(1.1, 0.8)

(1.5, 0.9)

(a)

m

= 50

,

n

= 200

(b)

m

= 50

,

n

= 500

Nonnegative matrix factorization. We want to solve the matrix factorization problem subject to nonnegative constraints:

where A is a given n -byn low-rank matrix. Although nonconvex, this problem is famously welltackled by first-order methods. In each iteration, the gradient ∇ f ( x ) involves 3 matrix-matrix

multiplications, whereas evaluating the objective f ( x ) only requires 1. Note that for the last iteration of the linesearch, the computed matrix product can be reused to compute the gradient for the next iteration.

We created matrix A by multiplying matrices B and C ⊤ , where B and C are n -byr matrices with entries sampled from a normal distribution. Negative entries in both matrices B and C were then set to zero. The results are presented in Figure 4, where the number of gradients roughly means the number of 3 matrix-matrix multiplications. In both cases we generated, the proposed method converged faster than any of the linesearch versions.

AdProxGD

(1.2, 0.5)

(1.5, 0.8)

(1.1, 0.5)

(1.2, 0.9)

(1.1, 0.9)

(1.5, 0.5)

(1.2, 0.8)

(1.1, 0.8)

(1.5, 0.9)

(a)

n

= 100

,

r

= 20

(b)

n

= 100

,

r

= 30

Dual of the entropy maximization problem. Consider the entropy maximization problem subject to linear constraints

where A ∈ ❘ m × n . Its dual problem is given by

where a i ∈ ❘ m is the i -th column of A (the derivation is provided in [BV04, Chapter 5.1.6]).

AdProxGD

(1.2, 0.5)

(1.5, 0.8)

(1.1, 0.5)

(1.2, 0.9)

(1.1, 0.9)

(1.5, 0.5)

(1.2, 0.8)

(1.1, 0.8)

(1.5, 0.9)

(a)

m

= 500

,

n

= 100

(b)

m

= 100

,

n

= 500

It is the latter problem (52) that we solved. We generated m -byn matrix A with entries sampled from a normal distribution. Then we generated a random w ∈ ❘ n from the unit simplex and set

b = Aw . Each gradient requires two matrix-vector multiplications, while the objective evaluation requires only one (and, as before, the last one can be reused for the next gradient). The results are presented in Figure 5, where the number of gradients roughly means the number of 2 matrix-vector multiplications. In the first scenario, the proposed method is faster than all the linesearch versions, while in the second one, only one version of linesearch was comparable in performance.

Conclusion. Based on our preliminary experiments, it is evident that AdProxGD indeed performs better. To our surprise, a few specific pairs ( r, s ) consistently outperform the rest among ProxGD with linesearch. We are not aware of any theoretical finding that would confirm this evidence. Also, from a numerical point of view, the linesearch implementation is not always robust. In particular, when we are already close to a solution, the linesearch condition can sometimes fail because the numbers it operates on are all very small.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction describes our contributions in details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the Literature and Discussion section, as well as in other various parts of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each statement has a proof either in the main text or in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe in details how a random data was generated for the experiments. The code is publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: It is described in details which parameters were used.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: There are no error bars for the experiments. The paper is mostly of theoretical interest.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: This is not important for our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This is a theoretical paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work that does not have any foreseeable societal impacts. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All experiments were randomly generated.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not contain a study involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not contain a study involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper refines the analysis of AdGD to accommodate larger stepsizes, an approach that exploits local curvature instead of global smoothness. The technique is later extended to ProxAdGD. Experiments demonstrate the superiority of ProxAdGD over Armijo’s linesearh.

STRENGTHS:
1. The paper unfolds by providing intuition and examples, helping the readers to deepen their understanding beyond the technical results.
2. Better guarantees that exploits local properties of the objective are desirable, both in deterministic and stochastic settings.

WEAKNESSES:
3. A more detailed comparison to the results of [MM20] is required. Does the improvement merely tighten the constants or enable a drastically different behaviour than [MM20]?
4. Experiments comparing the stepsize choices of the paper and [MM20] are sorely missing. Both the difference between the stepsizes along the optimization and the optimization error of each method.

QUESTIONS:
5. See weaknesses.
6. Did the authors consider the stochastic case? Either by a theoretical analysis or by experiments. [MM20] provided both a theoretical guarantee (without adaptivity to local curvature) and experiments with neural networks. (The reviewer does not ask the authors to conduct new experiments with neural networks but merely asks if such were previously performed.)

Overall, the paper presents a clear picture of AdGD which helps deepen our understanding, includes an improved stepsize selection and the new AdProxGD method. That being said, the quantification of the improvement is unclear due to limited comparison with [MM20]. Such a comparison will move along way toward establishing the value of this work.

Typos: line 54 - some some, line 272 - experimentswe.

LIMITATIONS:
Discussed in weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces new algorithms for solving convex optimization problems, where a step size parameter adapts to the underlying objective function. Adaptation is achieved by making good use of the already available gradient information, and does not have a further computational cost. The authors propose multiple variants. One of those rely on a generalized derivation of an algorithm from a recent paper, to achieve an improved step sizes. Another provides yet larger stepsizes, and a third one allows non-differentiable (but still convex) objective functions, provided that can be split into a smooth and a non-smooth function where the proximity operator of the non-smooth function is feasible to realize. They compare their algorithm against a simple alternative on a specific problem to demonstrate how it can reduce computational cost.\n\nSTRENGTHS:\nThe first algorithm alone, which relies on a generalized derivation of a past algorithm, is of interest. The two additional algorithms also allow to further expand the usefulness of the paper. In particular, the third algorithm, which allows handling of non-smooth functions, would be useful in practice. In particular, it extends the applicability of the algorithm to constrained problems (provided projections to the constraint set is feasible in practice), or potentially, problems with sparsity constraints, which can be enforced by convex functions like some variant of an $\\ell_1$ norm. I think such problems are pretty relevant for ML, and I'd expect the paper to be of interest to a wide community.\n\nWEAKNESSES:\nThe attention is restricted to convex problems. To be fair, this is perhaps welcome, as it allows the authors to provide a clean analysis of their algorithms. In practice, many convex algorithms found their way to non-convex problems with success, sometimes without justification.\n\nAt times, I wished the authors would complete their arguments fully. I have a few comment below.\n\nThe numerical problem didn't feel very fair for the competitor algorithm (I probably would not have used such an algorithm due to how expensive its steps are -- see below).\n\nQUESTIONS:\n- eqn 13 : I don't see how a simple substitution of the previous inequality to (10) gets you this. If you're using additional steps, please either not, or include details -- it's perfectly understandable to push some content to the appendix, but the manuscript is perfectly brings the development to this stage to suddenly switch gears. Since this subsection is meant to be simple, I'd suggest including details.\n- line 191, \"note that the second bound...\" : please note this is the \"second bound in step 5 of Alg. 2\".\n- eqn 15 : In this inequality, should $\\alpha_k$ and $\\alpha_{k-1}$ be interchanged? In that case, the following interpretation is also not correct.\n- eqn 16 : For a wildly changing function $L_1$ won't capture the behavior globally anyway. Is this meant for a relatively \"well-behaved\" function?\n- line 221 : I'd suggest writing out the definition of the proximity operator $arg min_t 0.5 \\|x - t\\|_2^2 + g(t)$ instead of relying on short hand. The text is perfectly readable without knowing what a proximity operator is. In fact, you could also include $\\alpha_k$ in there, since that's used in the algorithm description.\n- line 222, \"Algorithm 3, presented in Appendix C\" : Please include the algorithm as part of the main text.\n- Numerical experiment on the constrained problem : I understand the motivation behind comparing the objective wrt the projections. However, this is particularly a problem where a line search would be feasible due to the high computational cost. It would be interesting to compare against a basic forward backward splitting method with a fairly reasonable estimate of the local $L$. If that's not possible, and the baseline algorithm is the most suitable, it'd be good to include an argument.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers optimization problems where the function is convex and possibly composite. Gradient Descent and the Proximal Gradient method are studied, and one of the main motivations of this work is the paper [MM20], which developed a locally adaptive step size and an associated algorithm 'adaptive gradient descent without descent AdGD'. In the current work, the authors consider whether the bounds on the step for AdGD are essential, and they find that they are, but can be improved upon. They present Algorithm 2, which employs their refined step size, and Theorem 2 details the convergence properties. \nAs well as improving the step size criteria, this work also extends that of [MM20] by presenting an adaptive proximal variant. The algorithm is given in Algorithm 3, and convergence properties are presented in Theorem 3.\nThe authors compare their algorithm with others in the literature in Section 5, including detailing possible extensions, and why they might prove tricky. \nFinally, numerical experiments are presented in Section 6 (with proofs and further experiments in the appendix).\n\nSTRENGTHS:\nWriting. The paper was well written and I enjoyed reading it. I liked the writing style because the authors made efforts to explain *why* they were doing what they were (not just what they were doing).\nI liked that the authors opted to keep the exposition simple and easy to follow (e.g. see Remark 1 regarding the potential use of cocoercivity).\nContribution. Gradient descent is one of the most important algorithms in this field, and the step size has a major impact on the practical performance of the algorithm. This work presents an step size selection condition that is conceptually simple, and is cheap to evaluate, which is a nice contribution. The extension to the proximal setting is also welcome.\n\nWEAKNESSES:\nIt would be good if the authors tightened up the wording of, for example, their theorem statements. For example, for Theorem 2, I think it would be better if they mentioned \"given an initial point $x_0$, and an initial step size $\\alpha_0>0$. Also, the phrase \"converges to a solution\" is used, but it would be better if they also referred to the problem they were trying to solve and said what they mean by a solution (and they should also specify that $F=f$, define $f_*$, etc). They should then check the other theorem statements and update them appropriately. I feel it is really important for them to be precise. \n\nIn Lemma 1 I think it should be mentioned that $\\alpha_k>0$ (i.e., the step size can be 'arbitrary but positive').\n\nNumerical experiments. The authors presented several numerical experiments, which is good. However, it would have been good to have had more discussion explaining to the reader what the plots were showing, i.e., more emphasis describing the observed practical behaviour of the algorithms, and specifically commenting on the performance for the parameter choices.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
In the paper, the authors explore two adaptive first-order algorithms in convex optimization, namely gradient descent and proximal gradient methods, which are based on observed gradient differences. With a novel Lyapunov energy, the authors prove its convergences assuming only local Lipschitz condition of the gradient and extend it to the proximal case. In addition, the methods allow larger initial stepsizes than those in previous work.

STRENGTHS:
For many other methods, such as Gradient Descent, we should assume the problems are globally L-smooth and calculate the value L. However, we just ensure the object functions are locally L-smooth if using the methods in this paper. 

The same as Barzilai-Borwein stepsize, the methods do not include linesearch procedure. It means they are just based on observed gradient differences and do not need high computational cost in each iteration. However, in the contrast to the Barzilai-Borwein stepsize, the paper could guarantee its convergences theoretically in convex and proximal case.

WEAKNESSES:
1.We may not guarantee its convergences in the nonconvex case. 
2. In the second experiment, the first term in problem may be (1+x_1^2 )^(1/2), rather than (1+x_1 )^(1/2).

QUESTIONS:
1. What about the result by using stochastic gradient?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10