Universal Safeguarded Learned Convex Optimization with Guaranteed Convergence

1 INTRODUCTION . Solving scientific computing problems often requires application of efficient and scalable optimization algorithms . Despite the ever improving rates of convergence of state-of-the-art general purpose algorithms , their ability to apply to real-time applications is still limited due to the relatively large number of iterations that must be computed . To circumvent this shortcoming , a growing number of researchers use machine learning to develop task-specific algorithms from general-purpose algorithms . For example , inspired by the iterative shrinkage thresholding algorithm ( ISTA ) for solving the LASSO problem , a sparse coding problem , Gregor & LeCun ( 2010 ) proposed to learn the weights in the matrices of the ISTA updates that worked best for a given data set , rather than leave these parameters fixed . They then truncated the method to K iterations , making their Learned ISTA ( LISTA ) algorithm form a K-layer feed-forward neural network . Empirically , their examples showed roughly a 20-fold reduction in computational cost compared to the traditional algorithms . Several related works followed , also demonstrating numerical success ( discussed below ) . While classic optimization results often provide worst-case convergence rates , limited theory exists pertaining to such instances of data drawn from a common distribution ( e.g. , data supported on a lowdimensional manifold ) . As a step toward providing such theory , this work addresses the question : Does there exist a universal method that encompasses all L2O algorithms and generates iterates that approach the solution set with guarantees ? We provide an affirmative answer to this question by prescribing and proving properties of neural networks generated within our L2O framework . Convergence is established by including any choice among several practical safeguarding procedures , including nonmonotone options . Nonmonotone safeguarding enables sequences to traverse portions of the underlying space where the objective function value may increase for a few successive iterations as long as , on average , the sequence approaches the solution set . Although counterintuitive , this ability may lead to faster convergence . Furthermore , we provide a practical guide in our discussion for how practitioners may use our framework to create and apply L2O schemes to their own problems . The theoretical portion of this work is presented in the context of fixed point theory . This is done to be sufficiently general and provide the desired convergence result for the wide class of optimization methods that can be expressed as special cases of the Krasnosel ’ skiı̆-Mann ( KM ) method . For concreteness and ease of application , we then provide the special-case results to several well-known methods ( e.g. , proximal-gradient , Douglas-Rachford splitting , and ADMM ) . Related Works . Learning to learn methods date back decades ( e.g. , see ( Thrun & Pratt , 1998 ) for a survey of earlier works and references ) . A seminal L2O work in the context of sparse coding was by Gregor & LeCun ( 2010 ) . Numerous follow-up papers also demonstrated empirical success at constructing rapid regressors approximating iterative sparse solvers , compression , ` 0 encoding , combining sparse coding with clustering models , nonnegative matrix factorization , compressive sensing MRI , and other applications ( Sprechmann et al. , 2015 ; Wang et al. , 2016a ; b ; c ; d ; Hershey et al. , 2014 ; Yang et al. , 2016 ) . A nice summary of unfolded optimization procedures for sparse recovery is given by Ablin et al . ( 2019 ) in Table A.1 . However , the majority of L2O works pertain to sparse coding and provide limited theoretical results . Some works have interpreted LISTA in various ways to provide proofs of different convergence properties ( Giryes et al. , 2018 ; Moreau & Bruna , 2017 ) . Others have investigated structures related to LISTA ( Xin et al. , 2016 ; Blumensath & Davies , 2009 ; Borgerding et al. , 2017 ; Metzler et al. , 2017 ) , providing varying results dependent upon the assumptions made . Chen et al . ( 2018 ) introduced necessary conditions for the LISTA weight structure to asymptotically achieve a linear convergence rate . This was followed by Liu et al . ( 2019a ) , which proved linear convergence of their ALISTA method for the LASSO problem and provided a result stating that , with high probability , the convergence rate of LISTA is at most linear . The mentioned results are useful , yet can require intricate assumptions and proofs specific to the relevant sparse coding problems . L2O works have also taken other approaches . For example , the paper by Li & Malik ( 2016 ) used reinforcement learning with an objective function f and a stochastic policy π∗ that encodes the updates , which takes existing optimization algorithms as special cases . Our work is related to theirs ( cf . Method 1 below and Algorithm 1 in that paper ) , with the distinction that we include safeguarding and work in the fixed point setting . The idea of Andrychowicz et al . ( 2016 ) is to use long short term memory ( LSTM ) units in recurrent neural networks ( RNNs ) . Additional learning approaches have been applied in the discrete setting ( Dai et al. , 2018 ; Li et al. , 2018 ; Bengio et al. , 2018 ) . Balcan et al . ( 2019 ) reveal how many samples are needed for the average algorithm performance on the training set to generalize over the entire distribution . This is practical for choosing training data and may be used for training L2O networks within our framework . Our Contribution . This is the first work to merge ideas from machine learning , safeguarded optimization , and fixed point theory into a general framework for incorporating data-driven updates into iterative convex optimization algorithms . In particular , given a collection of data and an update operator from an established method ( e.g. , ADMM or proximal gradient ) for solving an optimization problem , we present procedures for creating a neural network that can be used to quickly infer solution estimates . The first novelty of this framework is the ability to incorporate several safeguarding procedures in a general setting . The second is that we present a procedure for utilizing machine learning methods to incorporate knowledge from particular data sets . However , our most significant contribution to the L2O literature is to combine these results into a single , general framework for use by practitioners on any convex optimization problem . Outline . We first provide a brief overview of the fixed point setting of this work in Section 2 . Then we present the SKM method and convergence results in Section 3 . The incorporation of the SKM method into a neural network and subsequent training approach is presented in Section 4 . This is followed in Section 5 by numerical examples , discussion in Section 6 , and conclusions in Section 7 . 2 FIXED POINT METHODS . Let H be a finite dimensional Hilbert space ( e.g. , the Euclidean space Rn ) with inner product 〈· , ·〉 and norm ‖ · ‖ . Denote the set of fixed points of each operator T : H → H by Fix ( T ) : = { x ∈ H : Tx = x } . In this work , for an operator T with a nonempty fixed point set ( i.e. , Fix ( T ) 6= ∅ ) , the primary problem considered is the fixed point problem : Find x ? ∈ Fix ( T ) . ( 1 ) Convex minimization problems , both constrained and unconstrained , may be equivalently rewritten as the problem ( 1 ) for an appropriate mapping T . The method chosen for solving the minimization problem determines the operator T in ( 1 ) ( e.g. , see Table 1 below for examples ) . We focus on the fixed point formulation to provide a general approach , given T , for creating a sequence that converges to a solution of ( 1 ) and , thus , also of the corresponding optimization problem . The following definitions will be used in the sequel . A mapping T : H → H is nonexpansive if ‖Tx− Ty‖ ≤ ‖x− y‖ , for all x , y ∈ H. ( 2 ) An operator T : H → H is α-averaged if α ∈ ( 0 , 1 ) and there is a nonexpansive operator Q : H → H such that T = ( 1 − α ) Id + αQ , where Id is the identity operator . If the constant α is not important , then T may for brevity be called averaged . We also denote the distance between a point x ∈ H and a set C by dC ( x ) : = inf { ‖x− y‖ : y ∈ C } . ( 3 ) Two operators frequently used in optimization are constructed from monotone relations . Letting α > 0 and f : H → R be a function , the resolvent of the ( possibly ) multi-valued subgradient ∂f is Jα∂f : = ( Id + α∂f ) −1 ( 4 ) and the reflected resolvent of ∂f is Rα∂f : = 2Jα∂f − Id . ( 5 ) If f is closed , convex , and proper , then the resolvent is precisely the proximal operator , i.e. , Jα∂f = proxαf ( x ) : = arg min z∈H αf ( z ) + 1 2 ‖z − x‖2 . ( 6 ) From these definitions , it can be shown that Rα∂f is nonexpansive and Jα∂f is averaged . Results above may all be found in Bauschke & Combettes ( 2017 ) ( e.g. , see Prop . 4.4 , Thm . 20.25 , Example 23.3 , and Prop . 23.8 ) . See Table 1 for examples of these operators in optimization methods . A classic theorem states that sequences generated by successively applying an averaged operator converges to a fixed point . This method comes from Krasnosel ’ skii ( 1955 ) and Mann ( 1953 ) , which yielded adoption of the name Krasnosel ’ skiı̆-Mann ( KM ) method . This result is stated below and can be found with various forms and proofs in many works ( e.g. , see ( Bauschke & Combettes , 2017 , Thm . 5.14 ) , ( Byrne , 2008 , Thm . 5.2 ) , ( Cegielski , 2012 , Thm . 3.5.4 ) , and ( Reich , 1979 , Thm . 2 ) ) . Theorem 2.1 . If an averaged operator T : H → H has a nonempty fixed point set and a sequence { xk } k∈N with arbitrary x1 ∈ H satisfies the update relation xk+1 = T ( xk ) , for all k ∈ N , ( 7 ) then there is x ? ∈ Fix ( T ) such that { xk } k∈N converges to x ? , i.e. , xk → x ? . There are pathological cases where the result fails for operators that are only nonexpansive ( e.g. , when x1 6= 0 and either T = −Id or T is a rotation ) . However , this is easily remedied since any convex combination of a nonexpansive operator with the identity is averaged . 3 SAFEGUARDED KM METHOD . This section generalizes the classic KM iteration in ( 7 ) . We accomplish this by defining an envelope of operators TL2O ( · ; · ) . For a parameter ζ chosen from an appropriate set , we let TL2O ( · ; ζ ) define an operator on H. Changing ζ may define a new operator with different properties . We do not impose restrictions on TL2O ( · ; ζ ) other than it be well-defined , meaning TL2O ( · ; ζ ) may fail to be averaged and/or fail to have a fixed point . This is illustrated by the following two examples . Example 3.1 . Let Q : H → H be nonexpansive . Then define TL2O : H× R→ R by TL2O ( x ; ζ ) : = ( 1− ζ ) x+ ζQ ( x ) . ( 8 ) For ζ ∈ ( 0 , 1 ) , the operator TL2O ( · ; ζ ) defined in ( 8 ) is ζ-averaged . Although using ζ 1 may result in an operator that fails to be averaged , this can be useful in accelerating the convergence of a method ( e.g. , see ( Giselsson et al. , 2016 ) ) . 4 Example 3.2 . Let f : H → R be closed , convex and proper , and define TL2O : H× ( 0 , ∞ ) → R by TL2O ( x ; ζ ) : = proxζf ( x ) . ( 9 ) For fixed ζ ∈ ( 0 , ∞ ) , the operator TL2O ( · ; ζ ) is averaged . 4 The practicality of TL2O is discussed and demonstrated in Sections 4 and 5 , respectively . In the remainder of this work , each operator T : H → H is assumed to be averaged and we set S : = Id−T . Our proposed method below is called the Safeguarded Krasnosel ’ skiı̆-Mann ( SKM ) Method . Explanation of the SKM Method is as follows . In Line 1 , the initial iterate and parameter δ are initialized . A common choice for the initial iterate is x1 = 0 . The for loop on Lines 2 to 11 generates each update xk+1 in the sequence { xk } k∈N . The choice of parameter ζk in Line 3 may be any value that results in a well-defined operator TL2O ( · ; ζk ) in Line 5 . The choice of µk in Line 4 defines the safeguarding procedure that is used to ensure convergence . Safeguarding is implemented through a descent condition inequality in Line 6 . When the inequality in Line 6 holds , TL2O ( xk ; ζk ) is used to update xk via Line 7 . Otherwise , a KM update is used to update xk via Line 9 . Notice also the iteration indexed parameters may all be chosen dynamically ( rather than precomputed ) . Below are several standard assumptions used to prove our convergence result in Theorem 3.1 . Assumption 1 . The operator T is nonexpansive with a nonempty fixed point set . The following assumption ensures boundedness of sequences generated by the SKM method . Assumption 2 . The operator S is coercive , i.e. , lim ‖x‖→∞ ‖S ( x ) ‖ =∞ . ( 10 ) Remark 3.1 . Assumption 2 does not hold , in general , for nonexpansive operators . For example , if T is the gradient operator ( Id− α∇f ) for some α > 0 and f is a constant function , then S ( x ) = 0 for all x ∈ H. However , a minor perturbation to the f enables Assumption 2 to hold . In this example , if one fixes small ε > 0 and sets f̃ ( x ) : = f ( x ) + ε2‖x‖ 2 , then the associated S̃ satisfies ‖S̃ ( x ) ‖ = ε‖x‖ . This idea generalizes and , since this works for arbitrarily small ε , in practice it may be reasonable to assume Assumption 2 holds when applying the SKM Method . Algorithm 2 Learned SKM ( LSKM ) 1 : Stage 1 : Initialization/Training . 2 : Choose envelope TL2O ( · ; · ) and network structure C , parameterized by Θ = ( ζk ) Kk=1 3 : Choose training loss function φd 4 : Choose ‘ optimal ’ parameter Θ ? ∈ arg min Θ∈C Ed∼D [ φd ( x K ) ] , assuming µk =∞ at each layer k 5 : Choose δ and safeguarding scheme for { µk } Kk=1 6 : Define the neural networkM =MΘ ? , δ , µk . 7 : Stage 2 : Inference . 8 : For input d return x =M ( d ) Assumption 3 . If the inequality in Line 6 is satisfied infinitely many times , then the sequence { µk } k∈N converges to zero . Remark 3.2 . Assumption 3 may be enforced by using various choices that are dependent upon combinations of the previous residuals . This is illustrated by Table 2 and Corollary 3.1 below . Our main convergence result is Theorem 3.1 below ( proven in the Appendix ) . Theorem 3.1 . If { xk } k∈N is a sequence generated by the SKM method and Assumptions 1 to 3 hold , then lim k→∞ dFix ( T ) ( xk ) = 0 . ( 11 ) And , if { xk } k∈N contains a single cluster point , then { xk } k∈N converges to a point x ? ∈ Fix ( T ) . We propose several methods for choosing the sequence { µk } k∈N in Table 2 . These methods are adaptive in the sense that each update to an iterate µk depends upon the current iterate xk and ( possibly ) previous iterates . These update schemes enable each µk to trail the value of the residual norm ‖S ( xk ) ‖ . This implies there may exist j ∈ N for which ‖S ( xj ) ‖ < ‖S ( xj+1 ) ‖ = ‖S ( TL2O ( x j ; ζj ) ) ‖ ≤ ( 1− δ ) µj . ( 12 ) Such leniency is desirable since it is possible that , even though the residual norms converge to zero , constructing a sequence of iterates along the quickest route to the solution set requires traversing portions of the underlying space where the residual norms increase for a few successive iterations . The safeguarding schemes in Table 2 are justified by the Corollary below ( proven in the Appendix ) . Corollary 3.1 . If { xk } k∈N is a sequence generated by the SKM method and Assumptions 1 and 2 hold and { µk } k∈N is generated using a scheme outlined in Table 2 , then Assumption 3 holds and , by Theorem 3.1 , the limit ( 33 ) holds .

Summary: This paper proposes a framework to unfold the safeguarded Krasnosel’ski˘ı-Mann (SKM) method for the learn to optimization (L2O) schemes. First, SKM is proposed in Algorithm 1 with convergence guarantee established in Theorem 3.1 and Corollary 3.1. Then, SKM is unfolded and executed with a neural network summarized in Algorithm 2. Experiments on the Lasso and nonnegative least squares show the efficiency of the proposed method as well as the effectiveness of safeguarding compared to traditional L2O methods.