Unifying Width-Reduced Methods for Quasi-Self-Concordant Optimization

1 Introduction . We study a class of constrained optimization problems of the following form : min Ax=b X i f ( Px ) i ( 1 ) for convex f : R ! R , where A 2 Rd⇥n , b 2 Rd , P 2 Rm⇥n , with d n m. Specifically , we are interested in the case where f satisfies a certain higher-order smoothness-like condition known as M -quasi-self-concordance ( q.s.c . ) , i.e. , |f 000 ( x ) | Mf 00 ( x ) for all x 2 R. Several problems of significant interest in machine learning and numerical methods meet this condition , including logistic regression [ Bac10 , KSJ18 ] , as well as softmax ( often used to approximate ` 1 regression ) [ Nes05 , CKM+11 , EV19 , Bul20 ] and ( regularized ) ` p regression [ BCLL18 , AKPS19 ] . A very useful optimization technique , first introduced by [ CKM+11 ] for faster approximate maximum flow and later by [ CMMP13 ] for regression , is that of width reduction , whereby they used it to improve the iteration complexity dependence on m , the number of rows of the design matrix from O ( m1/2 ) to Õ ( m1/3 ) , and where each iteration requires a linear system solve . Later work by [ AKPS19 ] for high-accuracy ` p regression , building on an O ( m1/2 ) -iteration result from [ BCLL18 ] , again showed 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . how width reduction could lead to improved Õ ( m1/3 ) -iteration algorithms . As a drawback , however , these approaches rely on potential methods and analyses specifically tailored to each problem . Building on these results , we present the first unified approach to achieving m1/3-type rates , at the heart of which lies a more general width reduction scheme . Notably , our method goes beyond these previously considered problems to capture quasi-self-concordant losses , thereby further including well-studied problems such as logistic regression , among others . By doing so , we directly achieve m 1/3-type rates in the constrained setting without relying on explicit acceleration schemes [ MS13 ] , thus complementing recent work based on a ball-oracle approach [ CJJ+20 ] . We additionally note that , given the ways in which our results achieve improvements similar to those of [ CJJ+20 ] , we believe our work hints at a deeper , though to our knowledge not yet fully understood , connection between the techniques of width reduction and Monteiro-Svaiter acceleration . 1.1 Main Results and Applications . We first present in Section 3 a width-reduced method for obtaining a crude approximation to ( 1 ) for quasi-self-concordant f . At a high level , our algorithm returns an approximate solution x̃ that both satisfies the linear constraints and is bounded in ` 1-norm by O ( R ) , where R is a bound on the norm of the optimal solution . Following from Theorem 3.3 , the result below shows how , for the problem of minimizing softmax ( parameterized by ⌫ > 0 ) , i.e. , smax⌫ ( Px ) = ⌫ log ⇣P i e ( Px ) i ⌫ ⌘ , we can bound the norm of the solution by ( 1 + ⌫ ) R. Theorem 1.1 . Let x ? denote the optimum of minAx=b smax⌫ ( Px ) . Algorithm 1 when applied to the function f ( Px ) = P i e ( Px ) i ⌫ with ✏ = ⌫ , returns ex such that Aex = b , and smax⌫ ( Pex ) ( 1 + eO ( ⌫ ) ) smax⌫ ( Px ? ) , in at most eO ( m1/3⌫ 5/3 ) calls to a linear system solver . As a consequence of Theorem 1.1 when taking ⌫ = ⌦ ⇣ ✏/ logO ( 1 ) ( m ) ⌘ , we have by Theorem 5.2 a ( 1 + ✏ ) approximate solution to the problem of ` 1 regression with Õ ( m1/3✏ 5/3 ) calls to a linear system solver . Further , we show the following result which can use the approximate solution returned by Theorem 1.1 as an initial point for achieving a high-accuracy solution . We also present in Appendix A a natural extension of our results to minimizing general-self-concordant ( g.s.c . ) functions . Theorem 1.2 . For M -q.s.c . f , ✏ > 0 , and x ( 0 ) such that Ax ( 0 ) = b and kx ( 0 ) k1 R , Algorithm 2 finds ex such that Aex = b and f ( ex ) ✏+ f ( x ? ) in eO MRm 1/3 log ( MR ) log ✓ f ( x ( 0 ) ) f ( x ? ) ✏ ◆ ! calls to a linear system solver . Resulting from the theorem above , as detailed in Section 5 , are guarantees given by Theorems 5.4 and 5.5 which establish convergence rates of eO ( p2µ 1/ ( p 2 ) m1/3R ) and eO ( m1/3R ) , respectively , for µ-regularized ` p regression and logistic regression . We emphasize that the latter is , to our knowledge , the first such use of width reduction for directly solving constrained logistic regression problems . 1.2 Related Works . Quasi-self-concordance and higher-order smoothness . [ Bac10 ] showed how to analyze Newton ’ s method for quasi-self-concordant functions , with an emphasis on its application to logistic regression . Later , notions of local , or Hessian , stability which follow from quasi-self-concordance gave rise to methods with better dependence on various conditioning parameters along with a linear rate of convergence [ KSJ18 , MFBR19 , CJJ+20 ] . In the work by [ KSJ18 ] , the authors show how a trust-region-based Newton method [ NW06 ] achieves linear convergence for locally stable functions without requiring , e.g. , strong convexity . Meanwhile , after noting that quasi-self-concordance implies Hessian stability , [ CJJ+20 ] further improve the dependence on the distance to the optimum by leveraging Monteiro-Svaiter acceleration [ MS13 ] , which has proven useful in the context of near-optimal methods for higher-order acceleration [ ASS19 , GDG+19 ] . However , in general these methods , which assume higher-order smoothness , require access to an oracle which minimizes a higher-order Taylor expansion model , though in some cases this may be relaxed to requiring linear system solves [ Bul20 ] . Width reduction and ` p regression . The notion of width is common in the multiplicative weights literature [ PST95 , Fle00 , GK07 ] . Most of these algorithms repeatedly solve a certain subproblem , and “ width ” is defined as an upper bound on the ` 1-norm of the solution to these subproblems . The runtime of such algorithms depends linearly on the width , and since this quantity can have a large value , several approaches have been proposed to reduce the width . The technique of width-reduction first came to prominence in seminal work by [ CKM+11 ] for achieving faster approximate maximum flow , being the first to achieve an improved m1/3 dependence . At a high level , the idea behind the approach is to solve a sequence of weighted ` 2-minimizing flow subproblems , whereby at each iteration one of two cases occurs : either the proposed step is added to the current solution ( a `` flow '' step ) along with the weights , or else there exist some set of coordinates that exceed a certain threshold , and so their weights are updated accordingly ( a `` width reduction '' step ) . Several works have since adapted this approach to regression problems [ CMMP13 , AKPS19 , EV19 , ABKS21 ] and matrix scaling [ AZLOW17 ] . In particular , when comparing with [ EV19 ] , we note that the update steps for the weights are different from our algorithm . We also note that the number of width reduction steps in our algorithm are restricted by ✏ 2/3 iterations , which is similar to [ EV19 ] , but our algorithm requires ✏ 5/3 flow steps . In addition to their importance in machine learning , regression methods capture several fundamental problems in scientific computing and signal processing . A recent line of work initiated by [ BCLL18 ] showed how to attain high-accuracy solutions for ` p regression using Op ( m1/2 1/p ) linear system solves , thus going beyond what is achievable via self-concordance . Building on this work , [ AKPS19 ] showed how width reduction could be applied to this setting to achieve , as in the case of approximate maximum flow [ CKM+11 ] , a similar improvement from Op ( m1/2 ) to Op ( m1/3 ) ( for p ! 1 ) . Further developments by [ KPSW19 , AS20 ] for graph problems showed almost-linear time solutions for ` p regression for p ⇡ p log ( n ) which have since been a critical part of recent advances in high-accuracy maximum flow on unit-capacity graphs [ LS20 , KLS20 ] . Accelerated methods . Recent developments by [ CJJ+20 ] have shown several advantages that arise in the case of unconstrained minimization for smooth , quasi-self-concordant problems . By considering a certain ball oracle method ( whereby each call to the oracle returns the minimizer of the function inside an ` 2 ball of radius r ) , [ CJJ+20 ] implement an accelerated scheme which returns a solution to the unconstrained smooth convex minimization problem in ( R/r ) 2/3 calls to the oracle , where R is the initial ` 2-norm distance to the optimum , and they further show a matching lower bound under this oracle model . We note here that while our method obtains rates in terms of the ` 1-norm of the optimum rather than the ` 2-norm , in the worst case , we can have kx ? k2 = p mkx ? k1 , which results in both rates being essentially the same . While the approach of [ CJJ+20 ] transfers its difficulty to implementing the oracle , a key insight from their work involves showing this can be done efficiently for smooth quasi-self-concordant functions when r is sufficiently small , where the allowed size depends on the quasi-self-concordance parameter . One limitation to their results is that they apply directly to unconstrained optimization problems and require the function to be smooth , and so we complement these results in the quasi-self-concordant setting by establishing comparable rates for directly optimizing a large class of constrained convex problems without requiring smoothness of the function . 1.3 Outline of the Paper . After establishing the potential functions at the heart of our width reduction techniques , we present in Section 3 our oracle for roughly approximating a solution to problem ( 1 ) . We then show in Section 4 how we may attain a high-accuracy solution by using the crude approximation as a starting point . Here , the key idea is to considering a sequence of optimization problems inside ` 1 balls of manageable size , similar to [ CMTV17 , CJJ+20 ] . As in the case of the crude oracle , our primary advantage comes from carefully handling a pair of coupled potentials which are amenable to the large class of quasi-self-concordant problems , and in Section 5 we further show how our results may be applied to several problems of interest , including logistic and ` p regression . 2 Preliminaries . Notation : We use boldface lowercase letters to denote vectors or functions and boldface uppercase letters for matrices . Scalars are non-bold letters . Our functions are univariate , and we overload function notation to act on a vector coordinate-wise , i.e . f ( x ) = P i f ( x i ) . The notation x y for vectors refers to entry-wise inequality . Refer to the algorithm boxes for definitions of certain algorithm specific parameters that appear in lemma and theorem statements .

Summary: The authors give an algorithm for solving optimization problems of the form $\min_{x \in \mathbb{R}^m, Ax =b } \sum_i f( (Px)_i)$, where $A,b,P$ are given and $f$ is convex and $\textit{quasi-self concordant}$. This property, introduced by Bach in [Bac10], is an alternative notion to the standard notion of self-concordance and implies that the Hessian of f(x) is multiplicatively stable inside of a box. As several standard regression objectives (softmax, logistic loss, l_p norms) are quasi-self concordant, the study of algorithms for such objective functions is of obvious theoretical and practical importance. The previous state of the art algorithm by [CJJ+20] solves qSC regression problems in $O(m^{1/3})$ linear system solves-- this improves upon previous work of [Bac10] which achieved a complexity of $O(m^{1/2})$ . However, [CJJ+20]'s algorithm (which combines an accelerated gradient method of [MS13] with an efficient algorithm to solve trust region problems) is fairly complex and inherently requires multiple iterative loops.  The present work improves upon [CJJ+20] by providing a simple $O(m^{1/3})$-iteration algorithm for this problem, based on a framework of [CKMST11] and [EV19].