Simple steps are all you need: Frank-Wolfe and generalized self-concordant functions

1 Introduction . Constrained convex optimization is the cornerstone of many machine learning problems . We consider such problems , formulated as : min x∈X 𝑓 ( x ) , ( 1.1 ) where 𝑓 : ℝ𝑛 → ℝ ∪ { +∞ } is a generalized self-concordant function and X ⊆ ℝ𝑛 is a compact convex set . When computing projections onto the feasible regions as required in , e.g. , projected gradient descent , is prohibitive , Frank-Wolfe ( FW ) ( Frank & Wolfe , 1956 ) algorithms ( a.k.a . Conditional Gradients ( CG ) ( Levitin & Polyak , 1966 ) ) are the algorithms of choice , relying on Linear Minimization Oracles ( LMO ) at each iteration to solve Problem ( 1.1 ) . The analysis of their convergence often relies on the assumption that the gradient is Lipschitzcontinuous . This assumption does not necessarily hold for generalized self-concordant functions , an important class of functions for which the growth can be unbounded . 1.1 Related work . In the classical analysis of Newton ’ s method , when the Hessian of 𝑓 is assumed to be Lipschitz continuous and the function is strongly convex , one arrives at a convergence rate for the algorithm that depends on the Euclidean structure of ℝ𝑛 , despite the fact that the algorithm is affine-invariant . This motivated the introduction of self-concordant functions in Nesterov & Nemirovskii ( 1994 ) , functions for which the third derivative is bounded by the second-order derivative , with which one can obtain an affine-invariant convergence rate for the aforementioned algorithm . More importantly , many of the barrier functions used in interior-point methods are self-concordant , which extended the use of polynomial-time interior-point methods to many settings of interest . ∗Georgia Institute of Technology . E-mail : alejandro.carderera @ gatech.edu †Zuse Institute Berlin . E-mail : besancon @ zib.de ‡Zuse Institute Berlin and Technische Universität Berlin . E-mail : pokutta @ math.tu-berlin.de 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Self-concordant functions have received strong interest in recent years due to the attractive properties that they allow to prove for many statistical estimation settings ( Marteau-Ferey et al. , 2019 ; Ostrovskii & Bach , 2021 ) . The original definition of self-concordance has been expanded and generalized since its inception , as many objective functions of interest have self-concordant-like properties without satisfying the strict definition of self-concordance . For example , the logistic loss function used in logistic regression is not strictly self-concordant , but it fits into a class of pseudo-self-concordant functions , which allows one to obtain similar properties and bounds as those obtained for self-concordant functions ( Bach , 2010 ) . This was also the case in Ostrovskii & Bach ( 2021 ) and Tran-Dinh et al . ( 2015 ) , in which more general properties of these pseudo-self-concordant functions were established . This was fully formalized in Sun & Tran-Dinh ( 2019 ) , in which the concept of generalized self-concordant functions was introduced , along with key bounds , properties , and variants of Newton methods for the unconstrained setting which make use of this property . Most algorithms that aim to solve Problem ( 1.1 ) assume access to second-order information , as this often allows the algorithms to make monotonous progress , remain inside the domain of 𝑓 , and often , converge quadratically when close enough to the optimum . Recently , several lines of work have focused on using Frank-Wolfe algorithm variants to solve these types of problems in the projection-free setting , for example constructing second-order approximations to a self-concordant 𝑓 using first and second-order information , and minimizing these approximations over X using the Frank-Wolfe algorithm ( Liu et al. , 2020 ) . Other approaches , such as the ones presented in Dvurechensky et al . ( 2020a ) ( later extended in Dvurechensky et al . ( 2020b ) ) , apply the Frank-Wolfe algorithm to a generalized self-concordant 𝑓 , using first and second-order information about the function to guarantee that the step sizes are so that the iterates do not leave the domain of 𝑓 , and monotonous progress is made . An additional Frank-Wolfe variant in that work , in the spirit of Garber & Hazan ( 2016 ) , utilizes first and second order information about 𝑓 , along with a Local Linear Optimization Oracle for X , to obtain a linear convergence rate in primal gap over polytopes given in inequality description . The authors in Dvurechensky et al . ( 2020b ) also present an additional FrankWolfe variant which does not use second-order information , and uses the backtracking line search of Pedregosa et al . ( 2020 ) to estimate local smoothness parameters at a given iterate . Other specialized Frank-Wolfe algorithms have been developed for specific problems involving generalized self-concordant functions , such as the Frank-Wolfe variant developed for marginal inference with concave maximization ( Krishnan et al. , 2015 ) , the variant developed in Zhao & Freund ( 2020 ) for 𝜃-homogeneous barrier functions , or the application for phase retrieval in Odor et al . ( 2016 ) , where the Frank-Wolfe algorithm is shown to converge on a self-concordant non-Lipschitz smooth objective . 1.2 Contribution . The contributions of this paper are detailed below and summarized in Table 1 . Simple FW for generalized self-concordant functions . We show that a small variation of the original Frank-Wolfe algorithm ( Frank & Wolfe , 1956 ) with an open-loop step size of the form 𝛾𝑡 = 2/ ( 𝑡+2 ) , where 𝑡 is the iteration count is all that is needed to achieve a convergence rate of O ( 1/𝑡 ) in primal gap ; this also answers an open question posed in Dvurechensky et al . ( 2020b ) . Our variation ensures monotonous progress while employing an open-loop strategy which , together with the iterates being convex combinations , ensures that we do not leave the domain of 𝑓 . In contrast to other methods that depend on either a line search or second-order information , our variant uses only a linear minimization oracle , zeroth-order and first-order information and a domain oracle for 𝑓 ( x ) . The assumption of the latter oracle is very mild and was also implicitly assumed in several of the algorithms presented in Dvurechensky et al . ( 2020b ) . As such , our iterations are much cheaper than those in previous work , while essentially achieving the same convergence rates for Problem ( 1.1 ) . Moreover , our variant relying on the open-loop step size 𝛾𝑡 = 2/ ( 𝑡 +2 ) allows us to establish a O ( 1/𝑡 ) convergence rate for the Frank-Wolfe gap , is agnostic , i.e. , does not need to estimate local smoothness parameters , and is parameter-free , leading to convergence rates and oracle complexities that are independent of any tuning parameters . Faster rates in common special cases . We also obtain improved convergence rates when the optimum is contained in the interior of X ∩ dom ( 𝑓 ) , or when the set X is uniformly or strongly convex , using the backtracking line search of Pedregosa et al . ( 2020 ) . We also show that the Away-Step Frank-Wolfe algorithm ( Lacoste-Julien & Jaggi , 2015 ; Wolfe , 1970 ) can use the aforementioned line search to achieve linear rates over polytopes . For clarity we want to stress that any linear rate over polytopes has to depend also on the ambient dimension of the polytope ; this applies to our linear rates and those in Table 1 established elsewhere ( see Diakonikolas et al . ( 2020 ) ) . In contrast the O ( 1/𝜀 ) rates are dimension-independent . Numerical experiments . We provide numerical experiments that showcase the performance of the algorithms on generalized self-concordant objectives to complement the theoretical results . In particular , they highlight that the simple step size strategy we propose is competitive with and sometimes outperforms other variants on many instances . After publication of our initial draft , in a revision of their original work , Dvurechensky et al . ( 2020b ) added an analysis of the Away-Step Frank-Wolfe algorithm which is complementary to ours ( considering a slightly different setup and regimes ) and was conducted independently ; we have updated the tables to include these additional results . 1.3 Preliminaries and Notation . We denote the domain of 𝑓 as dom ( 𝑓 ) def= { x ∈ ℝ𝑛 , 𝑓 ( x ) < +∞ } and the ( potentially nonunique ) minimizer of Problem ( 1.1 ) by x∗ . Moreover , we denote the primal gap and the Frank-Wolfe gap at x ∈ X ∩dom ( 𝑓 ) as ℎ ( x ) def= 𝑓 ( x ) − 𝑓 ( x∗ ) and 𝑔 ( x ) def= maxv∈X ⟨∇ 𝑓 ( x ) , x − v⟩ , respectively . We use ∥·∥ , ∥·∥𝐻 , and ⟨· , ·⟩ to denote the Euclidean norm , the matrix norm induced by a symmetric positive definite matrix 𝐻 ∈ ℝ𝑛×𝑛 , and the Euclidean inner product , respectively . We denote the diameter of X as 𝐷 def= maxx , y∈X ∥x − y∥ . Given a non-empty set X ⊂ ℝ𝑛 we refer to its boundary as Bd ( X ) and to its interior as Int ( X ) . We use Δ𝑛 to denote the probability simplex of dimension 𝑛 . Given a compact convex set C ⊆ dom ( 𝑓 ) we denote 𝐿C 𝑓 = max u∈C , d∈ℝ𝑛 ∥d∥2∇2 𝑓 ( u ) /∥d∥22 and 𝜇C𝑓 = minu∈C , d∈ℝ𝑛 ∥d∥ 2 ∇2 𝑓 ( u ) /∥d∥22 . We assume access to : 1 . Domain Oracle ( DO ) : Given x ∈ X , return true if x ∈ dom ( 𝑓 ) , false otherwise . 2 . Zeroth-Order Oracle ( ZOO ) : Given x ∈ dom ( 𝑓 ) , return 𝑓 ( x ) . 3 . First-Order Oracle ( FOO ) : Given x ∈ dom ( 𝑓 ) , return ∇ 𝑓 ( x ) . 4 . Linear Minimization Oracle ( LMO ) : Given d ∈ ℝ𝑛 , return argminx∈X ⟨x , d⟩ . The FOO and LMO oracles are standard in the FW literature . The ZOO oracle is often implicitly assumed to be included with the FOO oracle ; we make this explicit here for clarity . Finally , the DO oracle is motivated by the properties of generalized self-concordant functions . It is reasonable to assume the availability of the DO oracle : following the definition of the function codomain , one could simply evaluate 𝑓 at x and assert 𝑓 ( x ) < +∞ , thereby combining the DO and ZOO oracles into one oracle . However , in many cases testing the membership of x ∈ dom ( 𝑓 ) is computationally less demanding than the function evaluation . Remark 1.1 . Requiring access to a zeroth-order and domain oracle are mild assumptions , that were also implicitly assumed in one of the three FW-variants presented in Dvurechensky et al . ( 2020b ) when computing the step size according to the strategy from Pedregosa et al . ( 2020 ) ; see Line 3 in Algorithm 6 in the Appendix . The remaining two variants ensure that x ∈ dom ( 𝑓 ) by using second-order information about 𝑓 , which we explicitly do not rely on . The following example motivates the use of Frank-Wolfe algorithms in the context of generalized self-concordant functions . We present more examples in the computational results . Example 1.2 ( Intersection of a convex set with a polytope ) . Consider Problem ( 1.1 ) where X = P ∩ C , P is a polytope over which we can minimize a linear function efficiently , and C is a convex compact set for which one can easily build a barrier function . Solving a linear optimization problem over X may be extremely expensive . In light of this , we can incorporate C into the problem through the use of a barrier penalty in the objective function , minimizing instead 𝑓 ( x ) + 𝜇ΦC ( x ) where ΦC ( x ) is a log-barrier function for C and 𝜇 is a parameter controlling the penalization . This reformulation is illustrated in Figure 1 . Note that if the original objective function is generalized self-concordant , so is the new objective function ( see Proposition 1 in Sun & Tran-Dinh ( 2019 ) ) . We assume that computing the gradient of 𝑓 ( x ) + 𝜇ΦC ( x ) is roughly as expensive as computing the gradient for 𝑓 ( x ) and solving an LP over P is inexpensive relative to solving an LP over P ∩ C. The 𝜇 parameter can be driven down to 0 after a solution converges in a warm-starting procedure similar to interior-point methods , ensuring convergence to the true optimum . An additional advantage of this transformation of the problem is the solution structure . Running Frank-Wolfe on the set P ∩ C could potentially select a large number of extremal points from Bd ( C ) if C is non-polyhedral . In contrast , P has a finite number of vertices , a small subset of which will be selected throughout the optimization procedure . The same solution as that of the original problem can thus be constructed as a convex combination of a small number of vertices of P , improving sparsity and interpretability in many applications . The following definition formalizes the setting of Problem ( 1.1 ) . Definition 1.3 ( Generalized self-concordant function ) . Let 𝑓 ∈ 𝐶3 ( dom ( 𝑓 ) ) be a closed convex function with dom ( 𝑓 ) ⊆ ℝ𝑛 open . Then 𝑓 is ( 𝑀 , 𝜈 ) generalized self-concordant if : | 〈D3 𝑓 ( x ) [ w ] u , u〉 | ≤ 𝑀 ∥u∥2∇2 𝑓 ( x ) ∥w∥𝜈−2∇2 𝑓 ( x ) ∥w∥3−𝜈2 , for any x ∈ dom ( 𝑓 ) and u , w ∈ ℝ𝑛 , where D3 𝑓 ( x ) [ w ] = lim 𝛼→0 𝛼−1 ( ∇2 𝑓 ( x + 𝛼w ) − ∇2 𝑓 ( x ) ) .

Summary: This work studies simple step size, i.e., 2/(t+2), in FW for generalized self-concordant functions. The main idea is to wait the step size to shrink until f(x_{t+1}) <= f(x_t). By doing so, one can avoid relying on second order information or line search.