A first-order primal-dual method with adaptivity to local smoothness

1 Introduction . In this paper we study a particular instance of the composite minimization problem min x∈X f ( x ) + g ( Ax ) , ( 1 ) where f and g are convex , proper and lower-semicontinuous ( l.s.c . ) , and A is a linear operator . Problems of the form ( 1 ) have been studied in the literature under various assumptions on f and g. For the particular instances where g ◦ A is proximal-friendly and f is L-smooth , the objective is suitable for applying forward-backward splitting algorithms like the Proximal Gradient algorithm and its accelerated counterpart [ Nesterov , 2013 , Beck and Teboulle , 2009 ] . In general , however , the proximal operator of g ◦A is not easily computable and in such cases a popular approach is to decouple A and g by reformulating problem ( 1 ) as a convex-concave saddle-point problem : min x∈X max y∈Y 〈Ax , y〉+ f ( x ) − g∗ ( y ) , ( 2 ) where g∗ denotes the Fenchel conjugate of g. Objective ( 2 ) is typically addressed by primal-dual splitting algorithms which , under strong duality , can recover the solution to the original problem ( 1 ) . In the particular case when f and g are proximal-friendly and possibly non-smooth , a very popular method is the Primal-Dual Hybrid Gradient proposed in [ Chambolle and Pock , 2011 ] , which was further extended to handle an additional L-smooth component with the Condat-Vũ algorithm [ Condat , 2013 , Vũ , 2013 ] . Convergence rates for the latter are studied in [ Chambolle and Pock , 2016a ] . Together , these classes of algorithms cover a broad range of problems in diverse fields such as signal processing , machine learning , inverse problems , telecommunications and many others . As a result , a great amount of research effort has gone into addressing practical concerns such as robustness 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to inexact oracles , acceleration and automation of stepsize selection . For a comprehensive list of examples and theoretical details we refer the reader to review papers [ Combettes and Pesquet , 2011 , Parikh and Boyd , 2014 , Komodakis and Pesquet , 2015 , Chambolle and Pock , 2016b ] . In this work , we focus on the line of investigation studying stepsize regime automation for primal-dual algorithms targeting problem ( 2 ) . In their basic form , primal-dual methods require as input stepsize parameters belonging to a designated interval of stability , which depends on problem specific constants like the global smoothness parameter L and ‖A ‖ . Dependence on such constants is undesirable because they may be costly to compute and oftentimes one can only access upper-bound estimates , thus leading to overly-conservative stepsizes and slower convergence . Moreover , the need to know L for setting the stepsizes prevents these methods from being applied to functions which are not globally smooth . Consequently , recent efforts have gone towards devising methods with adaptive stepsizes [ Goldstein et al. , 2013 , 2015 , Malitsky and Pock , 2018 , Pedregosa and Gidel , 2018 ] . These approaches resort to linesearch for finding good stepsizes at every iteration , and exhibit improved practical performance . It thus appears that better convergence comes at the cost of an indeterminate number of extra steps spent in subprocedures aimed at finding appropriate stepsizes . In this work , we study problem ( 2 ) under the assumption that∇f is locally Lipschitz continuous and g is proximal-friendly . To illustrate the motivation of our framework , we take a prototypical example in image processing : min x 1 2 ‖Kx− b ‖2 + λ ‖Dx ‖2,1 , K ∈ Rm×d , D ∈ R2d×d , where x is an image , K is a problem-specific measurement operator , b is the vector of ( possibly noisy ) observations and D is the discrete gradient operator and the regularization term represents the isotropic TV norm . In order to apply any of the aforementioned primal-dual algorithms , one needs to first choose how to decouple the linear operators . There are three options : decoupling with respect to K leaves us with having to compute the proximal operator of the TV norm for the primal step , which is an iterative procedure [ Chambolle , 2004 ] . Decoupling D implies performing gradient steps on f , since in general its proximal operator is not efficient . Finally , decoupling with respect to both implies increasing the dimensionality of the dual variable to m+ 2d , which is problematic for large d and m. The sensible choice is the second one ( i.e. , decoupling D ) , and the question we seek to answer with this work is : Does there exist a method for solving ( 2 ) that adapts to the local problem geometry without resorting to linesearch ? Our contribution is to propose a first-order primal-dual scheme that answers this question in the affirmative and is accompanied by theoretical convergence guarantees . Using standard analysis techniques we show an ergodic convergence of O ( k−1 ) when ∇f is locally Lipschitz and g is proximal-friendly , and a linear convergence rate for the case when f is in addition locally strongly convex and A has full row rank . We provide numerical experiments for sparse logistic regression and image inpainting , as well as use our method as a heuristic for TV-regularized nonconvex phase retrieval . The rest of the paper is structured as follows : Section 2 provides details about related work ; Section 3 introduces notation , along with technical preliminaries and assumptions to be used in our analysis ; Section 4 reports the main theoretical results alongside partial proofs ; finally , partial numerical results are provided in Section 5 with the rest being deferred to the appendix due to lack of space . 2 Related Work . Adaptive Gradient Descent ( GD ) methods . Arguably the most widespread of optimization methods , GD presents similar shortcomings for setting the stepsize as those described in the previous section . In particular , much research effort has gone in devising variants of the algorithm that remove the need to estimate the global smoothness constant L. In a recent work , Malitsky and Mishchenko [ 2020 ] propose an extremely simple and effective alternative for setting the stepsize τk adaptively at every iteration , as follows : τk = min { τk−1 √ 1 + τk−1 τk−2 , ‖xk − xk−1 ‖ 2 ‖∇f ( xk ) −∇f ( xk−1 ) ‖ } . ( 3 ) Adaptivity essentially comes ‘ for free ’ in ( 3 ) , as it involves solely quantities which have already been computed . Moreover , the method requires only the weaker assumption of local smoothness , thus extending the reach of provably-convergent GD to a wider class of differentiable functions while maintaining the standard O ( k−1 ) convergence rate . In this work we show that the above technique can be extended to the analysis of primal-dual methods , where it gives rise to an algorithm whose stepsizes adapt to the local geometry of the objective ’ s ( locally ) smooth component f . Adaptive monotone variational inequality ( VI ) methods . Malitsky [ 2020 ] proposes an algorithm for solving monotone VIs with a stepsize that adapts to local smoothness similarly to ( 3 ) . This method solves the very general formulation of finding u∗ such that 〈F ( u∗ ) , u−u∗〉+h ( u ) −h ( u∗ ) ≥ 0 , ∀u for a given monotone operator F which is locally Lipschitz continuous . Our template ( 2 ) can be recovered from theirs by setting u = ( x , y ) , with F ( u ) = F ( x , y ) = [ ∇f ( x ) +AT y −Ax ] , and h ( u ) = g∗ ( y ) . The advantages of this approach are the relaxed requirement of local Lipschitz continuity for F and the fact that knowledge of ‖A‖ is not required . However , since the VI framework is very general and does not take advantage of the problem structure ( e.g . the fact that 〈Ax , y〉 is a bilinear term ) , the method comes with worse convergence bounds than algorithms specifically designed to solve ( 2 ) . In addition , the algorithm requires as input an upper bound on the stepsizes , despite them being set in accordance to the estimated local smoothness . First order primal dual algorithms and adaptive versions . A popular method for solving ( 2 ) when f is L-smooth is the Condat-Vũ algorithm ( CVA ) [ Condat , 2013 , Vũ , 2013 ] . The method ’ s convergence is subject to a global stepsize validity condition given by ( 1 τ − L ) 1 σ ≥ ‖A ‖ 2 , where τ and σ are the primal and dual stepsizes , respectively . Another approach to solving problem ( 2 ) is via the Primal–Dual Fixed-Point algorithm based on the Proximity Operator ( PDFP2O ) or the Proximal Alternating Predictor–Corrector ( PAPC ) methods [ Loris and Verhoeven , 2011 , Chen et al. , 2013 , Drori et al. , 2015 ] . This approach comes with less restrictive stepsize conditions than CVA owing to a different iteration style , but which nevertheless depend on the global smoothness constant L and ‖A ‖ and have to be carefully chosen . In order to alleviate the burden of choosing the stepsize parameters in CVA , Malitsky and Pock [ 2018 ] propose a linesearch procedure involving only dual variable updates and which , for certain problems such as regularized least squares , does not require any additional matrix-vector multiplications . A characteristic of this algorithm is that it maintains a constant ratio between primal and dual stepsizes through a hyperparameter β — a setup which we also use in this work . 3 Preliminaries . Consider problem ( 2 ) and let X , Y be finite dimensional real vector spaces equipped with the standard inner product 〈· , ·〉 and the associated Euclidean norm ‖ · ‖ = √ 〈· , ·〉 . We denote by g∗ the Fenchel conjugate of g in ( 1 ) defined as g∗ ( y ) = supx { 〈x , y〉− g ( x ) } . In order to not overload the ∗ notation , we use AT to denote the adjoint operator of A . One can easily see that ( 2 ) is a primal-dual formulation of the following primal and dual optimization problems , of which the former is the same as ( 1 ) : min x∈X f ( x ) + g ( Ax ) , max y∈Y − ( f∗ ( −AT y ) + g∗ ( y ) ) . A saddle-point ( x∗ , y∗ ) ∈ X × Y of problem ( 2 ) satisfies the following optimality conditions : −AT y∗ = ∇f ( x∗ ) , Ax∗ ∈ ∂g∗ ( y∗ ) . ( 4 ) For ( x′ , y′ ) ∈ X × Y we define the following quantities : Px′ , y′ ( x ) : = f ( x ) − f ( x′ ) + 〈x− x′ , AT y′〉 , Dx′ , y′ ( y ) : = g ∗ ( y ) − g∗ ( y′ ) − 〈Ax′ , y − y′〉 , Gx′ , y′ ( x , y ) : = Px′ , y′ ( x ) +Dx′ , y′ ( y ) . These functions are convex for fixed ( x′ , y′ ) and whenever ( x′ , y′ ) = ( x∗ , y∗ ) , it holds that Px∗ , y∗ ( x ) ≥ 0 , Dx∗ , y∗ ( x ) ≥ 0 and Gx∗ , y∗ ( x , y ) ≥ 0 , with the latter quantity representing the primal-dual gap . We also define the gap restricted to a bounded subset B1 ×B2 ⊂ X × Y as : GB1×B2 ( x , y ) : = sup ( x′ , y′ ) ∈B1×B2 Px′ , y′ ( x ) +Dx′ , y′ ( y ) , and note that it is non-negative whenever B1 ×B2 contains a saddle-point . Given a function f : X → R and L > 0 , we say that f is L-smooth if its gradient ∇f is Lipschitz continuous : ‖∇f ( x ) − ∇f ( y ) ‖ ≤ L‖x− y‖ , ∀x , y . Furthermore , f is locally smooth if ∇f is Lipschitz continuous on any compact subset C : ∀C ⊂ X , ∃LC > 0 such that ‖∇f ( x ) −∇f ( y ) ‖ ≤ LC‖x− y‖ , ∀x , y ∈ C. We also say that f is µ-strongly convex if f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉 + µ2 ‖x− y ‖ 2 , ∀x , y . Similarly , f is locally strongly convex if it is strongly convex on any compact subset C : ∀C ⊂ X , ∃µC > 0 such that f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉+ µC2 ‖x− y ‖ 2 , ∀x , y ∈ C. We define the proximal operator of a convex function g : X → R ∪ { ∞ } as proxg ( x ) = argminz { g ( z ) + 12 ‖x− z ‖ 2 } , and say that g is ‘ proximal-friendly ’ if proxg ( x ) has a closed form solution or can be efficiently computed to high accuracy . Finally , the following two blanket assumptions will hold throughout the paper : Assumption 3.1 . Function f is convex and locally smooth , while g convex , l.s.c. , and proximalfriendly . Assumption 3.2 . A saddle-point exists for problem ( 2 ) and thus strong duality holds . We note that Assumption 3.2 is standard in the literature ( see e.g. , [ Chambolle and Pock , 2011 ] ) . Assumption 3.1 , on the other hand , is weaker than the usual global L-smoothness premise and thus enlarges the category of admissible functions f with instances such as x 7→ exp ( x ) . To illustrate , consider the aforementioned function defined on the reals : the global smoothness assumption clearly does not hold , however for any fixed interval [ a , b ] ⊂ R the smoothness constant can be chosen as exp ( b ) . For showing linear convergence of our method , we will add the following assumption : Assumption 3.3 . Function f is locally strongly convex and operator A has full row-rank .

Summary: This paper studies adaptivity of first-order primal-dual methods with respect to the smoothness or local smoothness of the problem. The manuscript builds upon a previous publication (Yura Malitsky and Konstantin Mishchenko. Adaptive gradient descent without descent. In Interna347 tional Conference on Machine Learning, pages 6702–6712. PMLR, 2020) to primal dual setup. Ergodic convergence is shown for the proposed method, where no knowledge of L is needed. The results is based on local smoothness which is defined for pairs of iterates of the sequence of points generated by the algorithm. The case of strongly convex function is also studied.