A first-order primal-dual method with adaptivity to local smoothness

1 Introduction . In this paper we study a particular instance of the composite minimization problem min x∈X f ( x ) + g ( Ax ) , ( 1 ) where f and g are convex , proper and lower-semicontinuous ( l.s.c . ) , and A is a linear operator . Problems of the form ( 1 ) have been studied in the literature under various assumptions on f and g. For the particular instances where g ◦ A is proximal-friendly and f is L-smooth , the objective is suitable for applying forward-backward splitting algorithms like the Proximal Gradient algorithm and its accelerated counterpart [ Nesterov , 2013 , Beck and Teboulle , 2009 ] . In general , however , the proximal operator of g ◦A is not easily computable and in such cases a popular approach is to decouple A and g by reformulating problem ( 1 ) as a convex-concave saddle-point problem : min x∈X max y∈Y 〈Ax , y〉+ f ( x ) − g∗ ( y ) , ( 2 ) where g∗ denotes the Fenchel conjugate of g. Objective ( 2 ) is typically addressed by primal-dual splitting algorithms which , under strong duality , can recover the solution to the original problem ( 1 ) . In the particular case when f and g are proximal-friendly and possibly non-smooth , a very popular method is the Primal-Dual Hybrid Gradient proposed in [ Chambolle and Pock , 2011 ] , which was further extended to handle an additional L-smooth component with the Condat-Vũ algorithm [ Condat , 2013 , Vũ , 2013 ] . Convergence rates for the latter are studied in [ Chambolle and Pock , 2016a ] . Together , these classes of algorithms cover a broad range of problems in diverse fields such as signal processing , machine learning , inverse problems , telecommunications and many others . As a result , a great amount of research effort has gone into addressing practical concerns such as robustness 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to inexact oracles , acceleration and automation of stepsize selection . For a comprehensive list of examples and theoretical details we refer the reader to review papers [ Combettes and Pesquet , 2011 , Parikh and Boyd , 2014 , Komodakis and Pesquet , 2015 , Chambolle and Pock , 2016b ] . In this work , we focus on the line of investigation studying stepsize regime automation for primal-dual algorithms targeting problem ( 2 ) . In their basic form , primal-dual methods require as input stepsize parameters belonging to a designated interval of stability , which depends on problem specific constants like the global smoothness parameter L and ‖A ‖ . Dependence on such constants is undesirable because they may be costly to compute and oftentimes one can only access upper-bound estimates , thus leading to overly-conservative stepsizes and slower convergence . Moreover , the need to know L for setting the stepsizes prevents these methods from being applied to functions which are not globally smooth . Consequently , recent efforts have gone towards devising methods with adaptive stepsizes [ Goldstein et al. , 2013 , 2015 , Malitsky and Pock , 2018 , Pedregosa and Gidel , 2018 ] . These approaches resort to linesearch for finding good stepsizes at every iteration , and exhibit improved practical performance . It thus appears that better convergence comes at the cost of an indeterminate number of extra steps spent in subprocedures aimed at finding appropriate stepsizes . In this work , we study problem ( 2 ) under the assumption that∇f is locally Lipschitz continuous and g is proximal-friendly . To illustrate the motivation of our framework , we take a prototypical example in image processing : min x 1 2 ‖Kx− b ‖2 + λ ‖Dx ‖2,1 , K ∈ Rm×d , D ∈ R2d×d , where x is an image , K is a problem-specific measurement operator , b is the vector of ( possibly noisy ) observations and D is the discrete gradient operator and the regularization term represents the isotropic TV norm . In order to apply any of the aforementioned primal-dual algorithms , one needs to first choose how to decouple the linear operators . There are three options : decoupling with respect to K leaves us with having to compute the proximal operator of the TV norm for the primal step , which is an iterative procedure [ Chambolle , 2004 ] . Decoupling D implies performing gradient steps on f , since in general its proximal operator is not efficient . Finally , decoupling with respect to both implies increasing the dimensionality of the dual variable to m+ 2d , which is problematic for large d and m. The sensible choice is the second one ( i.e. , decoupling D ) , and the question we seek to answer with this work is : Does there exist a method for solving ( 2 ) that adapts to the local problem geometry without resorting to linesearch ? Our contribution is to propose a first-order primal-dual scheme that answers this question in the affirmative and is accompanied by theoretical convergence guarantees . Using standard analysis techniques we show an ergodic convergence of O ( k−1 ) when ∇f is locally Lipschitz and g is proximal-friendly , and a linear convergence rate for the case when f is in addition locally strongly convex and A has full row rank . We provide numerical experiments for sparse logistic regression and image inpainting , as well as use our method as a heuristic for TV-regularized nonconvex phase retrieval . The rest of the paper is structured as follows : Section 2 provides details about related work ; Section 3 introduces notation , along with technical preliminaries and assumptions to be used in our analysis ; Section 4 reports the main theoretical results alongside partial proofs ; finally , partial numerical results are provided in Section 5 with the rest being deferred to the appendix due to lack of space . 2 Related Work . Adaptive Gradient Descent ( GD ) methods . Arguably the most widespread of optimization methods , GD presents similar shortcomings for setting the stepsize as those described in the previous section . In particular , much research effort has gone in devising variants of the algorithm that remove the need to estimate the global smoothness constant L. In a recent work , Malitsky and Mishchenko [ 2020 ] propose an extremely simple and effective alternative for setting the stepsize τk adaptively at every iteration , as follows : τk = min { τk−1 √ 1 + τk−1 τk−2 , ‖xk − xk−1 ‖ 2 ‖∇f ( xk ) −∇f ( xk−1 ) ‖ } . ( 3 ) Adaptivity essentially comes ‘ for free ’ in ( 3 ) , as it involves solely quantities which have already been computed . Moreover , the method requires only the weaker assumption of local smoothness , thus extending the reach of provably-convergent GD to a wider class of differentiable functions while maintaining the standard O ( k−1 ) convergence rate . In this work we show that the above technique can be extended to the analysis of primal-dual methods , where it gives rise to an algorithm whose stepsizes adapt to the local geometry of the objective ’ s ( locally ) smooth component f . Adaptive monotone variational inequality ( VI ) methods . Malitsky [ 2020 ] proposes an algorithm for solving monotone VIs with a stepsize that adapts to local smoothness similarly to ( 3 ) . This method solves the very general formulation of finding u∗ such that 〈F ( u∗ ) , u−u∗〉+h ( u ) −h ( u∗ ) ≥ 0 , ∀u for a given monotone operator F which is locally Lipschitz continuous . Our template ( 2 ) can be recovered from theirs by setting u = ( x , y ) , with F ( u ) = F ( x , y ) = [ ∇f ( x ) +AT y −Ax ] , and h ( u ) = g∗ ( y ) . The advantages of this approach are the relaxed requirement of local Lipschitz continuity for F and the fact that knowledge of ‖A‖ is not required . However , since the VI framework is very general and does not take advantage of the problem structure ( e.g . the fact that 〈Ax , y〉 is a bilinear term ) , the method comes with worse convergence bounds than algorithms specifically designed to solve ( 2 ) . In addition , the algorithm requires as input an upper bound on the stepsizes , despite them being set in accordance to the estimated local smoothness . First order primal dual algorithms and adaptive versions . A popular method for solving ( 2 ) when f is L-smooth is the Condat-Vũ algorithm ( CVA ) [ Condat , 2013 , Vũ , 2013 ] . The method ’ s convergence is subject to a global stepsize validity condition given by ( 1 τ − L ) 1 σ ≥ ‖A ‖ 2 , where τ and σ are the primal and dual stepsizes , respectively . Another approach to solving problem ( 2 ) is via the Primal–Dual Fixed-Point algorithm based on the Proximity Operator ( PDFP2O ) or the Proximal Alternating Predictor–Corrector ( PAPC ) methods [ Loris and Verhoeven , 2011 , Chen et al. , 2013 , Drori et al. , 2015 ] . This approach comes with less restrictive stepsize conditions than CVA owing to a different iteration style , but which nevertheless depend on the global smoothness constant L and ‖A ‖ and have to be carefully chosen . In order to alleviate the burden of choosing the stepsize parameters in CVA , Malitsky and Pock [ 2018 ] propose a linesearch procedure involving only dual variable updates and which , for certain problems such as regularized least squares , does not require any additional matrix-vector multiplications . A characteristic of this algorithm is that it maintains a constant ratio between primal and dual stepsizes through a hyperparameter β — a setup which we also use in this work . 3 Preliminaries . Consider problem ( 2 ) and let X , Y be finite dimensional real vector spaces equipped with the standard inner product 〈· , ·〉 and the associated Euclidean norm ‖ · ‖ = √ 〈· , ·〉 . We denote by g∗ the Fenchel conjugate of g in ( 1 ) defined as g∗ ( y ) = supx { 〈x , y〉− g ( x ) } . In order to not overload the ∗ notation , we use AT to denote the adjoint operator of A . One can easily see that ( 2 ) is a primal-dual formulation of the following primal and dual optimization problems , of which the former is the same as ( 1 ) : min x∈X f ( x ) + g ( Ax ) , max y∈Y − ( f∗ ( −AT y ) + g∗ ( y ) ) . A saddle-point ( x∗ , y∗ ) ∈ X × Y of problem ( 2 ) satisfies the following optimality conditions : −AT y∗ = ∇f ( x∗ ) , Ax∗ ∈ ∂g∗ ( y∗ ) . ( 4 ) For ( x′ , y′ ) ∈ X × Y we define the following quantities : Px′ , y′ ( x ) : = f ( x ) − f ( x′ ) + 〈x− x′ , AT y′〉 , Dx′ , y′ ( y ) : = g ∗ ( y ) − g∗ ( y′ ) − 〈Ax′ , y − y′〉 , Gx′ , y′ ( x , y ) : = Px′ , y′ ( x ) +Dx′ , y′ ( y ) . These functions are convex for fixed ( x′ , y′ ) and whenever ( x′ , y′ ) = ( x∗ , y∗ ) , it holds that Px∗ , y∗ ( x ) ≥ 0 , Dx∗ , y∗ ( x ) ≥ 0 and Gx∗ , y∗ ( x , y ) ≥ 0 , with the latter quantity representing the primal-dual gap . We also define the gap restricted to a bounded subset B1 ×B2 ⊂ X × Y as : GB1×B2 ( x , y ) : = sup ( x′ , y′ ) ∈B1×B2 Px′ , y′ ( x ) +Dx′ , y′ ( y ) , and note that it is non-negative whenever B1 ×B2 contains a saddle-point . Given a function f : X → R and L > 0 , we say that f is L-smooth if its gradient ∇f is Lipschitz continuous : ‖∇f ( x ) − ∇f ( y ) ‖ ≤ L‖x− y‖ , ∀x , y . Furthermore , f is locally smooth if ∇f is Lipschitz continuous on any compact subset C : ∀C ⊂ X , ∃LC > 0 such that ‖∇f ( x ) −∇f ( y ) ‖ ≤ LC‖x− y‖ , ∀x , y ∈ C. We also say that f is µ-strongly convex if f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉 + µ2 ‖x− y ‖ 2 , ∀x , y . Similarly , f is locally strongly convex if it is strongly convex on any compact subset C : ∀C ⊂ X , ∃µC > 0 such that f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉+ µC2 ‖x− y ‖ 2 , ∀x , y ∈ C. We define the proximal operator of a convex function g : X → R ∪ { ∞ } as proxg ( x ) = argminz { g ( z ) + 12 ‖x− z ‖ 2 } , and say that g is ‘ proximal-friendly ’ if proxg ( x ) has a closed form solution or can be efficiently computed to high accuracy . Finally , the following two blanket assumptions will hold throughout the paper : Assumption 3.1 . Function f is convex and locally smooth , while g convex , l.s.c. , and proximalfriendly . Assumption 3.2 . A saddle-point exists for problem ( 2 ) and thus strong duality holds . We note that Assumption 3.2 is standard in the literature ( see e.g. , [ Chambolle and Pock , 2011 ] ) . Assumption 3.1 , on the other hand , is weaker than the usual global L-smoothness premise and thus enlarges the category of admissible functions f with instances such as x 7→ exp ( x ) . To illustrate , consider the aforementioned function defined on the reals : the global smoothness assumption clearly does not hold , however for any fixed interval [ a , b ] ⊂ R the smoothness constant can be chosen as exp ( b ) . For showing linear convergence of our method , we will add the following assumption : Assumption 3.3 . Function f is locally strongly convex and operator A has full row-rank .

Summary: The authors propose an adaptive stepsize updating rule for the primal-dual Chambolle-Pock method that solves min f(x)+g(Ax) for locally-smooth f and cheap-prox-step g. The updating rule for adaptive stepsizes is simple in the sense that it only uses local gradients and ||A||, without needing any further subroutines like line search. They provide theoretical convergence rate for their proposed method under the local-smooth assumption and also with further local-strongly-convex assumption. They also corroborated their findings with experiment results that shows favorable  behavior of the new step-sizes.