Adaptive First-Order Methods Revisited: Convex Minimization without Lipschitz Requirements

1 Introduction . Owing to their wide applicability and flexibility , first-order methods continue to occupy the forefront of research in learning theory and continuous optimization . Their analysis typically revolves around two basic regularity conditions for the problem at hand : ( i ) Lipschitz continuity of the problem ’ s objective , and / or ( ii ) Lipschitz continuity of its gradient ( also referred to as Lipschitz smoothness ) . Depending on which of these conditions holds , the lower bounds for first-order methods with perfect gradient input are Θ ( 1/ √ T ) and Θ ( 1/T 2 ) after T gradient queries , and they are achieved by gradient descent and Nesterov ’ s fast gradient algorithm respectively [ 36 , 37 , 48 ] . By contrast , if the optimizer only has access to stochastic gradients ( as is often the case in machine learning and distributed control ) , the corresponding lower bound is Θ ( 1/ √ T ) for both classes [ 14 , 34 , 37 ] . This disparity in convergence rates has led to a surge of interest in adaptive methods that can seamlessly interpolate between these different regimes . Two state-of-the-art methods of this type are the ACCELEGRAD and UNIXGRAD algorithms of Levy et al . [ 26 ] and Kavis et al . [ 24 ] : both algorithms concurrently achieve an O ( 1/ √ T ) value convergence rate in non-smooth problems , an O ( 1/T 2 ) rate in smooth problems , and an O ( 1/ √ T ) average rate if run with bounded , unbiased stochastic gradients ( the smoothness does not affect the rate in this case ) . In this regard , UNIXGRAD and ACCELEGRAD both achieve a “ best of all worlds ” guarantee which makes them ideal as off-theshelf solutions for applications where the problem class is not known in advance – e.g. , as in online traffic routing , game theory , etc . At the same time , there have been considerable efforts in the literature to account for problems that do not adhere to these Lipschitz regularity requirements – such as Fisher markets , quantum tomography , D-design , Poisson deconvolution / inverse problems , and many other examples [ 7 , 10 , 11 , 28 , 29 , 45 ] . The reason that the Lipschitz framework fails in this case is that , even when the problem ’ s domain 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is bounded , the objective function exhibits singularities at the boundary , so it can not be Lipschitz continuous or smooth . As a result , no matter how small we pick the step-size of a standard gradient method ( adaptive or otherwise ) , the existence of domains with singular gradients can – and typically does – lead to catastrophic oscillations ( especially in the stochastic case ) . A first breakthrough in this area was provided by Birnbaum et al . [ 10 ] and , independently , Bauschke et al . [ 7 ] and Lu et al . [ 29 ] , who considered a “ Lipschitz-like ” gradient continuity condition for problems with singularities.1 At around the same time , Lu [ 28 ] and Teboulle [ 45 ] introduced a “ relative continuity ” condition which plays the same role for Lipschitz continuity . Collectively , instead of using a global norm as a metric , these conditions employ a Bregman divergence as a measure of distance , and they replace gradient descent with mirror descent [ 14 , 34 ] . In these extended problem classes , non-adaptive mirror descent methods achieve an O ( 1/ √ T ) value convergence rate in relatively continuous problems [ 2 , 28 ] , an O ( 1/T ) rate in relatively smooth problems [ 7 , 10 ] , and an O ( 1/ √ T ) average rate if run with stochastic gradients in relatively continuous problems [ 2 , 28 ] . Importantly , the O ( 1/T ) rate for relatively smooth problems does not match the O ( 1/T 2 ) rate for standard Lipschitz smooth problems : in fact , even though [ 20 ] proposed a tentative path towards faster convergence in certain non-Lipschitz problems , Dragomir et al . [ 17 ] recently established an Ω ( 1/T ) lower bound for problems that are relatively-but-not-Lipschitz smooth . Our contributions . Our aim in this paper is to provide an adaptive , parameter-agnostic method that simultaneously achieves order-optimal rates in the above “ non-Lipschitz ” framework . By design , the proposed method – which we call adaptive mirror descent ( ADAMIR ) – has the following desirable properties : 1 . When run with perfect gradients , the trajectory of queried points converges , and the method ’ s rate of convergence in terms of function values interpolates between O ( 1/ √ T ) and O ( 1/T ) for relatively continuous and relatively smooth problems respectively . 2 . When run with stochastic gradients , the method attains anO ( 1/ √ T ) average rate of convergence . 3 . The method applies to both constrained and unconstrained problems , without requiring a finite Bregman diameter or knowledge of a compact domain containing a solution . The only thing we assume known in the above is the reference Bregman function with respect to which the problem ’ s objective is relatively continuous/smooth ; other than that , we assume no prior information on the problem ’ s regularity class / modulus and/or the oracle model being accessed ( deterministic or stochastic ) .2 The enabling apparatus for these properties is an adaptive step-size policy in the spirit of [ 6 , 18 , 30 ] . However , a major difference – and technical difficulty – is that the relevant definitions can not be stated in terms of global norms , because the variation of nonLipschitz function explodes at the boundary of the problem ’ s domain ( put differently , gradients may be unbounded even over bounded domains ) . For this reason , our policy relies on the aggregation of a suitable sequence of “ Bregman residuals ” that stabilizes seamlessly when approaching a smooth solution , thus enabling the method to achieve faster convergence rates . Related work . Beyond the references cited above , problems with singular objectives were treated in a recent series of papers in the context of online and stochastic optimization [ 2 , 49 ] ; however , the proposed methods are not adaptive , and they do not interpolate between different problem classes . In the context of adaptive methods , the widely used ADAGRAD algorithm of Duchi et al . [ 18 ] and McMahan and Streeter [ 30 ] was recently shown to interpolate between an O ( 1/ √ T ) and O ( 1/T ) rate of convergence [ 26 , 27 ] . More precisely , Li and Orabona [ 27 ] showed that a specific , “ one-lagbehind ” variant of ADAGRAD with prior knowledge of the smoothness modulus achieves an O ( 1/T ) 1This condition was first examined by Birnbaum et al . [ 10 ] in the context of Fisher markets . The analysis of Bauschke et al . [ 7 ] and Lu et al . [ 29 ] is much more general , but several ideas are already present in [ 10 ] . 2The reference function has to do with the type of gradient singularities that develop at the boundary of the problem ’ s domain . For example , the Fisher market models that we discuss later in the paper always exhibit Θ ( log x ) gradient singularities at the boundary , so the appropriate choice for such models is the negative entropy , and this is true whether the specific market model is everywhere differentiable or not . Likewise , Poisson inverse problems always exhibit Θ ( 1/x ) gradient singularities at the boundary of the positive orthant , which means that the log-entropy is the appropriate reference function for such problems . rate in smooth , unconstrained problems ; concurrently , Levy et al . [ 26 ] obtained the same rate in a parameter-agnostic context . In either case , ADAGRAD achieves an O ( 1/ √ T ) rate of convergence in stochastic problems ( though with somewhat different assumptions for the randomness ) . In terms of rate optimality for Lipschitz smooth problems , ADAGRAD is outperformed by ACCELEGRAD [ 26 ] and UNIXGRAD [ 24 ] : these methods both achieve an O ( 1/T 2 ) value convergence rate in Lipschitz smooth problems , and an O ( 1/ √ T ) rate in stochastic problems with bounded gradient noise . By employing an efficient line-search step , the universal primal gradient descent ( UPGD ) algorithm of Nesterov [ 39 ] achieves order-optimal guarantees in the wider class of Hölder continuous problems ( which includes the Lipschitz continuous and smooth cases as extreme cases ) ; however , UPGD does not cover stochastic problems or problems with relatively continuous / smooth objectives . As far as we are aware , the closest works to our own are the generalized mirror-prox ( GMP ) algorithm of [ 44 ] and the ADAPROX method of [ 2 ] , both designed for variational inequality problems . The GMP algorithm can achieve interpolation between different classes of Hölder continuous problems and can adapt to the problem ’ s relative smoothness modulus , but it does not otherwise interpolate between the relatively smooth and relatively continuous classes . Moreover , GMP requires knowledge of a “ domain of interest ” containing a solution of the problem ; in this regard , it is similar to ACCELEGRAD [ 26 ] ( though it does not require an extra projection step ) . The recently proposed ADAPROX method of Antonakopoulos et al . [ 3 ] also achieves a similar interpolation result under a set of assumptions that are closely related – but not equivalent – to the relatively continuous/smooth setting of our paper . Moreover , neither of these papers covers the stochastic case ; to the best of our knowledge , ADAMIR is the first method in the literature capable of adapting to relatively continuous / smooth objectives , even under uncertainty . For convenience , we detail these related works in Table 1 above . 2 Problem setup and preliminaries . Problem statement . Throughout the sequel , we will focus on convex minimization problems of the general form minimize f ( x ) , subject to x ∈ X . ( Opt ) In the above , X is a convex subset of a normed d-dimensional space V ∼= Rd , and f : V → R ∪ { ∞ } is a proper lower semi-continuous ( l.s.c . ) convex function with dom f = { x ∈ V : f ( x ) < ∞ } = X . Compared to standard formulations , we stress that X is not assumed to be compact , bounded , or even closed . This lack of closedness will be an important feature for our analysis because we are interested in objectives that may develop singularities at the boundary of their domain ; for a class of relevant examples of this type , see [ 1 , 2 , 7 , 9–11 , 22 , 29 , 49 ] and references therein . To formalize our assumptions for ( Opt ) , we will write ∂f ( x ) for the subdifferential of f at x , and X◦ ≡ dom ∂f = { x ∈ X : ∂f ( x ) ̸= ∅ } for the domain of subdifferentiability of f . Formally , elements of ∂f will be called subgradients , and we will treat them throughout as elements of the dual space V∗ of V . By standard results , we have riX ⊆ X◦ ⊆ X , and any solution x∗ of ( Opt ) belongs to X◦ [ 42 , Chap . 26 ] ; to avoid trivialities , we will make the following blanket assumption . Assumption 1 . The solution set X ∗ ≡ argmin f ⊆ X◦ of ( Opt ) is nonempty . Two further assumptions that are standard in the literature ( but which we relax in the sequel ) are : 1 . Lipschitz continuity : there exists some G > 0 such that |f ( x′ ) − f ( x ) | ≤ G∥x′ − x∥ for all x , x′ ∈ X . ( LC ) 2 . Lipschitz smoothness : there exists some L > 0 such that f ( x′ ) ≤ f ( x ) + ⟨v , x′ − x⟩+ L 2 ∥x′ − x∥2 for all x , x′ ∈ X and all v ∈ ∂f ( x ) . ( LS ) Remark . For posterity , we note that the smoothness requirement ( LS ) does not imply that ∂f ( x ) is a singleton.3 The reason for this more general definition is that we want to concurrently treat problems with smooth and non-smooth objectives , and also feasible domains that are contained in lower-dimensional subspaces of V . We also note that we will be mainly interested in cases where the above requirements all fail because f and/or its derivatives blow up at the boundary of X . By this token , we will not treat ( LC ) / ( LS ) as “ blanket assumptions ” ; we discuss this in detail in the sequel . The oracle model . From an algorithmic point of view , we aim to solve ( Opt ) by using iterative methods that require access to a stochastic first-order oracle ( SFO ) [ 37 ] . This means that , at each stage of the process , the optimizer can query a black-box mechanism that returns an estimate of the objective ’ s gradient ( or subgradient ) at the queried point . Formally , when called at x ∈ X , an SFO is assumed to return a random ( dual ) vector g ( x ; ω ) ∈ V∗ where ω belongs to some ( complete ) probability space ( Ω , F , P ) . In practice , the oracle will be called repeatedly at a ( possibly ) random sequence of points Xt ∈ X generated by the algorithm under study . Thus , once Xt has been generated at stage t , the oracle draws an i.i.d . sample ωt ∈ Ω and returns the dual vector : gt ≡ g ( Xt ; ωt ) = ∇f ( Xt ) + Ut ( 1 ) with Ut ∈ V∗ denoting the “ measurement error ” relative to some selection ∇f ( Xt ) of ∂f ( Xt ) . In terms of measurability , we will write Ft for the history ( natural filtration ) of Xt ; in particular , Xt is Ft-adapted , but ωt , gt and Ut are not . Finally , we will also make the statistical assumption that E [ Ut | Ft ] = 0 and ∥Ut∥2∗ ≤ σ2 for all t = 1 , 2 , . . . ( SFO ) where ∥·∥∗ denotes the dual norm on V∗ . This assumption is standard in the analysis of parameteragnostic adaptive methods , cf . [ 6 , 24 , 26 , 46 ] and references therein . For concreteness , we will refer to the case σ = 0 as deterministic – since , in that case , Ut = 0 for all t. Otherwise , if lim inft∥Ut∥∗ > 0 , the noise will be called persistent and the model will be called stochastic .

Summary: - In the paper, the authors propose an adaptive parameter agnostic algorithm AdaMir for online and stochastic convex optimization problems. - AdaMir is essentially like the popular Adagrad algorithm with Bregman distances involved in controlling the step-size. - Their setting involves non-Lipschitz gradients, contrary to the traditional Lipschitz smooth setting.