Oblivious Sketching-based Central Path Method for Solving Linear Programming Problems

1 Introduction . Linear programming is one of the fundamental models widely used in both theory and practice . It has been extensively applied in many fields such as economics Tintner ( 1955 ) ; Dorfman et al . ( 1987 ) , operations research Delson & Shahidehpour ( 1992 ) , compressed sensing Donoho ( 2006 ) ; Candes et al . ( 2006 ) , medical studies Mangasarian et al . ( 1990 ; 1995 ) , adversarial deep learning Wong & Kolter ( 2018 ) ; Weng et al . ( 2018 ) , etc. , due to its simple and intuitive structure . The problem of solving linear programmings has been studied since the 19-th century Sierksma & Zwols ( 2015 ) . Consider solving a general linear program in standard form minAx=b , x≥0 c > x of size A ∈ Rd×n without redundant constraints . For the generic case d = Ω ( n ) we considered in this paper , the state of art results take a total running time of O∗ ( nω + n2.5−α/2 + n2+1/6 ) 1 to obtain a solution of δ accuracy in current matrix multiplication time Cohen et al . ( 2019b ) ; Lee et al . ( 2019 ) , where ω is the exponent of matrix multiplication whose current value is roughly 2.373 Williams ( 2012 ) ; Le Gall ( 2014 ) , and α is the dual exponent of matrix multiplication whose current value is 0.31 Le Gall & Urrutia ( 2018 ) . The breakthrough work due to Cohen , Lee , and Song Cohen et al . ( 2019b ) improves the long standing running time of O∗ ( n2.5 ) since 1989 Vaidya ( 1989 ) . For the current ω and α , Cohen et al . ( 2019b ) algorithm takes O∗ ( n2.373 ) time . For the current state-of-art results , the work Cohen et al . ( 2019b ) involves a non-oblivious sampling technique , whose sampling set and size changes along the iterations . It avoids the possibilities of implementing expensive calculations in the preprocessing stage and also makes it harder to extend to other classical optimization problems . On the other hand , the work Lee et al . ( 2019 ) only maintains an infeasible update in each iteration and requires the usage of dense sketching matrices , which will ruin the potential sparsity structure of the original linear programs . Thus , a natural question to ask is : 1We use O∗ hides no ( 1 ) and logO ( 1 ) ( 1/δ ) factors . Is there an oblivious and feasible algorithm for solving linear programming programs in fast running time ( i.e . current matrix multiplication time ) ? In this work , we propose a both oblivious and feasible ( per iteration ) 2 method that solves linear programs in the same running time as the state of art . The algorithm we propose is a sketching-based short step central path method . The classical short step method follows the central path in the interior of the feasible region . It decreases the complementarity gap uniformly by roughly a 1− 1/√n factor in each iteration and takes O∗ ( √ n ) iterations to converge . This results in O∗ ( √ n ) × n = O∗ ( n1.5 ) coordinate updates throughout the algorithm Vaidya ( 1989 ) . Compared to Cohen et al . ( 2019b ) ; Lee et al . ( 2019 ) , our randomized algorithm improves this amount ( in Vaidya ( 1989 ) ) of updates via a different approach . We only updating a O∗ ( √ n ) -dimensional subspace in each iteration while keeping the same number of iterations √ n through an Oblivious Coordinate-wise Embedding ( OCE ) technique . Thus , our method updates O∗ ( n ) dimensions in total , which is nearly optimal . The coordinate-wise embedding we introduce in this work is a distribution of matrices R ∈ Rbsketch×n with bsketch n such that , for any inner product g > h between two ndimensional vector g , h ∈ Rn , with ” high ” probability g > R > Rh approximates g > h well . In the case of solving linear programmings , we approximate the calculation of matrix-vector multiplication Ph in each iteration by PR > Rh through OCE , such that the resulting random vector is close to previous one in each coordinate , i.e. , ( PR > Rh ) i ≈ ( Ph ) i for all i ∈ [ n ] . Combining with lazy update and low-rank update techniques to maintain the query structure PR > Rh for any input vector h ∈ Rn , we can ensure the new random path is still close to the central path throughout the iterations . Therefore , our method decrease the average running time per iteration while keeping the same number of iterations . Furthermore , the sketching matrix R in our approach can be chosen in an oblivious way since it does not depend on the algorithm updates . Compare to previous work Lee et al . ( 2019 ) , our approximation form PR > Rh also helps admit a closed form solution in each iteration for solving LP . Thus , our approach takes the advantages of being oblivious and feasible , compared to other state of art results Cohen et al . ( 2019b ) ; Lee et al . ( 2019 ) . We state our main result as follows : Theorem 1.1 ( Main result , informal ) . Given a linear program minAx=b , x≥0 c > x with no redundant constraints . Let δlp denotes precision . Our algorithm takes O ( n 2.373 log ( n/δlp ) ) time to solve this LP . 1.1 Related works . Linear programming . Linear programmings have been studied for nearly a century . One of the first and most popular LP algorithm is the simplex algorithm Dantzig ( 1947 ) . Despite it works well in practical small size problems , the simplex algorithm is known to be an exponential time algorithm in the worst case of Klee-Minty cube Klee & Minty ( 1972 ) . The first polynomial time algorithm for solving LP is the ellipsoid method Khachiyan ( 1980 ) proposed by Khachiyan . Although this algorithm runs in polynomial time in theory , but in practice this algorithm runs much slower than the simplex algorithm . The interior point type of methods Karmarkar ( 1984 ) have both polynomial running time in theory and fast and stable performance in practice . In the case of d = Ω ( n ) considered in this work , Karmarkar ’ s algorithm Karmarkar ( 1984 ) takes O∗ ( n3.5 ) running time . Then it was improved to O∗ ( n3 ) in the work Renegar ( 1988 ) ; Vaidya ( 1987 ) . In 1989 , Vaidya further proposed an algorithm that takes a running of O∗ ( n2.5 ) . This result hasn ’ t been improved until recent work due to Cohen , Lee and Song Cohen et al . ( 2019b ) . Sketching . Classical sketching methodology proposed by Clarkson & Woodruff ( 2013 ) is the so-called ” sketch and solve ” . The most standard and well-known applications are 2In each iteration , we approximate the central path by solving a linear system . Our approach constructs a randomized oblivious system equation which can be solved exactly . While previous work Cohen et al . ( 2019b ) constructs a non-oblivious one , and Lee et al . ( 2019 ) doesn ’ t solve the system exactly . linear regression Clarkson & Woodruff ( 2013 ) ; Nelson & Nguyên ( 2013 ) ; Andoni et al . ( 2018 ) ; Clarkson et al . ( 2019 ) ; Song et al . ( 2019a ) and low-rank approximation Clarkson & Woodruff ( 2013 ) ; Nelson & Nguyên ( 2013 ) ; Boutsidis & Woodruff ( 2014 ) ; Clarkson & Woodruff ( 2015b ; a ) ; Razenshteyn et al . ( 2016 ) ; Song et al . ( 2017 ; 2019b ; c ) . It further generalizes to subspace Wang & Woodruff ( 2019 ) ; Li et al . ( 2020a ) , positive semi-definite matrices Clarkson & Woodruff ( 2017 ) , total least regression Diao et al . ( 2019 ) , quantile regression Li et al . ( 2020b ) , tensor regression Li et al . ( 2017 ) ; Diao et al . ( 2018 ) , tensor decomposition Song et al . ( 2019d ) . The sketching method we deploy in this work is called ” ‘ iterate and sketch ” Song ( 2019 ) . The major difference between classical “ sketch and solve “ , and “ iterating and sketch “ is : the first one only applied the sketch once at very beginning to reduce the dimension of problem , while does not modify the solver ; the second one opens up and modifies the solver by applying sketching techniques iteratively in each iteration . The idea of “ iterate and sketch ” has been applied to a number of problems , e.g . computing John Ellipsoid Cohen et al . ( 2019a ) , Newton method Pilanci & Wainwright ( 2016 ; 2017 ) , tensor decomposition Wang et al . ( 2015 ) ; Song et al . ( 2016 ) , training deep neural network Brand et al . ( 2020 ) . Empirical risk minimization Empirical risk minimization ( ERM ) problem is a fundamental question in statistical machine learning . Extensive literature has been devoted to study this topic Nesterov ( 1983 ) ; Vapnik ( 1992 ) ; Nesterov ( 1998 ) ; Polyak & Juditsky ( 1992 ) ; Nemirovski et al . ( 2009 ) ; Nesterov ( 2013 ) ; Vapnik ( 2013 ) . First-order methods and a series of accelerated gradient descent algorithms for ERM are well-developed and studied Jin et al . ( 2018 ) ; Johnson & Zhang ( 2013 ) ; Nesterov & Stich ( 2017 ) ; Xiao & Zhang ( 2014 ) ; Allen-Zhu ( 2018 ) . These rates depend polynomially on the smoothness/strong convexity of the objective in order to achieve a log ( 1/ ) dependence on the error parameter . Notations For a positive integer n , we use [ n ] to denote set { 1 , 2 , · · · , n } . For vectors x , z ∈ Rn and parameter ∈ ( 0 , 1 ) , we use x ≈ z to denote ( 1− ) zi ≤ xi ≤ ( 1+ ) zi , ∀i ∈ [ n ] . For any scalar t , we use a ≈ t to denote ( 1− ) t ≤ ai ≤ ( 1 + ) t , ∀i ∈ [ n ] . Given diagonal matrices X = diag ( x ) ∈ Rn×n , S = diag ( s ) ∈ Rn×n , we use XS to denote the diagonal matrix with ( XS ) i , i = xi/si , ∀i ∈ [ n ] . 2 Technique overview . In this section , we discuss the key ideas of our approach based on the central path method . 2.1 Short Step Central Path Method . Consider the following standard primal and dual problems of linear programmings : min Ax=b , x≥0 c > x ( primal ) and max A > y+s=c , x , s≥0 b > y ( dual ) where A ∈ Rd×n is full rank with d = O ( n ) . Then ( x , y , s ) is an optimal solution if and only if it satisfies the following optimality conditions Vanderbei et al . ( 2015 ) : Ax = b , x ≥ 0 ( primal feasibility ) A > y + s = c , s ≥ 0 ( dual feasibility ) xisi = 0 for all i ( complementary slackness ) The classical interior point method finds an optimal solution by following the central path in the interior of the feasible region , which is defined as the tuple ( x , y , s , t ) that satisfies : Ax = b , x > 0 A > y + s = c , s > 0 ( 1 ) xisi = t for all i where t > 0 is called the complementarity gap . It has been shown we can obtain an initialization point on the central path with t = 1 according to Ye et al . ( 1994 ) . Then in each iteration , the classical algorithm deceases the complementarity gap uniformly from t to ηt with η < 1 , and solves Eq . ( 1 ) . As t approaches 0 , the central path will converge to an optimal solution . The short step central path method approximately solves Eq . ( 1 ) by the following linear system : Xδs + Sδx = δµ , Aδx = 0 , ( 2 ) A > δy + δs = 0 , where X = diag ( x ) , S = diag ( s ) and we update the solution by x = x + δx , s = s + δs and y = y + δy . Denote the actual complementarity gap µ ∈ Rn defined under Eq . ( 2 ) as µi = xisi for i ∈ [ n ] . Then Eq . ( 2 ) maintains the feasibility conditions while approximately moving the gap from µ to µ+ δµ . As long as the actual complementarity gap µ is always close to the aiming complementarity gap t during the algorithm , we can assure the actual complementarity gap µ will converge to 0 as t goes to 0 , which leads us to an optimal solution . To solve the linear system ( 2 ) , note when A is full-rank , it has an unique solution explicitly given by : δx = X√ XS ( I − P ) 1√ XS δµ and δs = S√ XS P 1√ XS δµ , ( 3 ) where P = √ X S A > ( AXS A > ) −1 A √ X S is an orthogonal projection matrix . Literature Vaidya ( 1989 ) shows we can choose η to be roughly 1− 1√ n , and the algorithm converges in O∗ ( √ n ) iterations . Therefore , the total running time needed of solving LP by explicit solution Eq . ( 3 ) is O∗ ( nω+1/2 ) .

Summary: The paper is based on the works of Lee et al and Cohen et al. Building upon these works, the paper comes up with an interior point method that matches the state-of-the-art in LPs. The paper's contribution is in a new type of sketching used inside the interior point method, that demonstrates some advantages over those of Lee et al and Cohen et al. Notable among these advantages are (1) the ability to preserve sparsity of an LP, and (2) exact solutions to the system of linear equations obtained from optimality conditions.