Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradient

1 Introduction . First-order methods ( FOMs ) , which use gradient and not Hessian information , are now applied as standard practice in many areas of optimization [ 12 ] . A known weakness of FOMs is the tailing-off effect , where FOMs quickly find moderately accurate solutions , but progress towards an optimal solution slows down over time . While moderately accurate solutions are often sufficient for large machine learning applications , other applications traditionally demand higher precision . One such area is Linear Programming ( LP ) , the focus of this work . LP is a fundamental class of optimization problems in applied mathematics , operations research , and computer science with a huge range of applications , including mixed-integer programming , scheduling , network flow , chip design , budget allocation , and many others [ 17 , 22 , 65 , 69 ] . Software ∗Part of the work was done at Cornell and Google Research . †Part of the work was done at Google Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for solving LP problems , called LP solvers , originated in the earliest days of computing , predating the invention of operating systems [ 55 ] . The state-of-the-art methods for LP , namely Dantzig ’ s simplex method [ 22,23 ] and interior-point ( or barrier ) methods [ 51 ] , are quite mature and reliable at delivering highly accurate solutions . These widely successful methods have left little room for FOMs to make inroads . Furthermore , practitioners who use LP solvers are not accustomed to reasoning about the trade-off between accuracy and computing times typically intrinsic to FOMs . In this paper , we provide evidence that , if properly enhanced , FOMs can obtain high quality solutions to LP problems quickly . Indeed , there ’ s reason to expect this , as authors have developed FOMs for LP with linear rates of convergence [ 24 , 32 , 47 , 70 , 71 ] . On the other hand , the linear rates depend on potentially loose and hard-to-compute constants ; hence , tailing off may still be observed in practice . To our knowledge , ours is the first work to combine both theoretical enhancements with practical heuristics , demonstrating their combined effectiveness with extensive computational experiments on standard benchmark instances . In fact , our experiments will expose a substantial gap between algorithms presented in the literature and what ’ s needed to obtain good performance . Starting from a baseline primal-dual hybrid gradient ( PDHG ) method [ 19 ] applied to a saddle point formulation of LP , we develop a series of algorithmic improvements . These enhancements include adaptive restarting [ 7 ] , dynamic primal-dual step size selection [ 36 , 37 ] , presolving techniques [ 1 ] , and diagonal preconditioning ( data equilibration ) [ 33 ] . Most of these enhancements , while inspired by existing literature , are novel . We name our collection of enhancements PDLP ( PDHG for LP ) . The impact of these improvements is substantial . For example , on 383 LP instances derived from the MIPLIB 2017 collection [ 34 ] , our implementation of a baseline version of PDHG solved only 50 problems to 10−8 relative accuracy given a limit of approximately 100,000 iterations per problem . By contrast , PDLP solves 283 of the 383 problems under the same conditions . We demonstrate that PDLP outperforms FOM baselines and , in a small number of cases , obtains performance competitive with a commercial LP solver . Although not the focus of this paper , we believe that our results open the door to a new set of possibilities and computational trade-offs when solving LP problems . PDLP has the potential to solve extremely large scale instances where the simplex method and interior-point methods are unable to run because of their reliance on matrix factorization . Since PDLP uses matrix-vector operations at its core , it can effectively run on multi-threaded CPUs , GPUs [ 68 ] , or distributed clusters [ 26 ] . Furthermore , a GPU implementation of PDLP could efficiently solve batches of similar problems , a setup that has already been successfully applied with other optimization algorithms in applications like strong branching [ 46 ] and training neural networks that contain optimization layers [ 5 ] . Outline . The remainder of this section focuses on related work . Section 2 introduces LP and PDHG . Section 3 describes the set of enhancements that define PDLP . Section 4 presents numerical experiments , and Section 5 concludes and outlines future directions . 1.1 Literature review . PDHG PDHG was first developed by Zhu and Chan [ 72 ] , with subsequent analysis and extension by a number of authors [ 3,18,19,21,27,38,60 ] . PDHG is closely related to the Arrow-Hurwicz method [ 8 ] . PDHG is a form of operator-splitting [ 11 , 64 ] and can be interpreted as a variant of the alternating directions method of multipliers ( ADMM ) and Douglas-Rachford splitting ( DRS ) [ 16 , 25 , 56 ] , which themselves are both instantiations of the proximal point method [ 25 , 58 , 62 ] . As opposed to ADMM or DRS , PDHG is ‘ matrix-free ’ in that the data matrix is only used for matrix-vector multiplications . This allows PDHG to scale to problems even larger than those tackled by these other techniques , and to make better use of parallel and distributed computation . FOM-based solvers Recent interest in large-scale cone programming has sparked the development several first-order solvers based on competing methods . ProxSDP [ 66 ] is a solver for semidefinite programming based on PDHG . Solvers based on Nesterov ’ s accelerated gradients [ 49 ] include TFOCS [ 14 ] , and FOM which is a suite of solvers employing both gradient and proximal algorithms [ 13 ] . Solvers based on operator splitting techniques like ADMM include SCS [ 52–54 ] , OSQP [ 67 ] , POGS [ 28 ] , and COSMO [ 30 ] . Of these both SCS and POGS offer a matrix-free implementation where the linear system , that arises from the proximal operator used in ADMM , is solved using the conjugate gradient method . However , we shall show experimentally that our method can be significantly faster and more robust than this approach . Finally , [ 4 ] considers applying a truncated semismooth Newton method to the system of equations defining a fixed point of the SCS operator . FOMs for LP Lan , Lu and Monteiro [ 40 ] and Renegar [ 61 ] develop FOMs for LP as a special case of semidefinite programming , with sublinear convergence rates . The FOM-based solvers above all apply to more general problem classes like cone programming or quadratic programming . In contrast , some of the enhancements that constitute PDLP are specialized , either in theory or practice , for LP ( namely restarts [ 7 ] and presolving ) . A number of authors [ 24 , 32 , 47 , 70 , 71 ] have proposed linearly convergent FOMs for LP ; to our knowledge , none have been subject of a comprehensive computational study . ECLIPSE [ 10 ] solves huge-scale industrial LP problems by accelerated gradient descent , without presenting comparisons on standard test problems . Lin et al . [ 42 ] propose an ADMM-based interior point method . In contrast with PDLP which solves to high accuracy ( i.e. , 10−8 relative error ) , [ 42 ] perform experiments with 10−3 and 10−5 relative error . SNIPAL [ 41 ] is a semismooth Newton method based on the proximal augmented Lagrangian . SNIPAL has fast asymptotic convergence , yet , to get good performance , the authors use ADMM for warm-starts . Given PDLP ’ s favorable comparisons with SCS , it ’ s plausible that PDLP could provide a more effective warm-start . Finally , Pock and Chambolle [ 59 ] apply PDHG with diagonal preconditioning to a limited set of test LP problems and Applegate et al . [ 6 ] show how to extract infeasibility certificates when applying PDHG to LP . 2 Preliminaries . In this section , we introduce the notation we use throughout the paper , summarize the LP formulations we solve , and introduce the baseline PDHG algorithm . Notation . Let R denote the set of real numbers , R+ the set of nonnegative real numbers , and R− the set of nonpositive real numbers . Let N denote the set of natural numbers ( starting from one ) . Let ‖ · ‖p denote the ` p norm for a vector , and let ‖ · ‖2 denote the spectral norm for a matrix . For a vector v ∈ Rn , we use v+ and v− for their positive and negative parts , i.e. , v+i = max { 0 , vi } and v−i = min { 0 , vi } . The symbol v1 : m denotes the vector with the first m components of v. The symbols Ki , · and K· , j correspond to the ith column and jth row of the matrix K , respectively . The symbol 1 denotes the vector of all ones . Given a convex set X , we use projX to denote the map that projects onto X . Linear Programming . We solve primal-dual LP problems of the form : minimize x∈Rn c > x subject to : Gx ≥ h Ax = b l ≤ x ≤ u maximize y∈Rm1+m2 , λ∈Rn q > y + l > λ+ − u > λ− subject to : c−K > y = λ y1 : m1 ≥ 0 λ ∈ Λ , ( 1 ) where G ∈ Rm1×n , A ∈ Rm2×n , c ∈ Rn , h ∈ Rm1 , b ∈ Rm2 , l ∈ ( R∪ { −∞ } ) n , u ∈ ( R∪ { ∞ } ) n , K > = ( G > , A > ) , q > : = ( h > , b > ) , and Λ = Λ1 × · · · × Λn Λi : = { 0 } li = −∞ , ui =∞ , R− li = −∞ , ui ∈ R R+ li ∈ R , ui =∞ R otherwise is the set of variables λ such that the dual objective is finite . This pair of primal-dual problems is equivalent to the saddle-point problem : min x∈X max y∈Y L ( x , y ) : = c > x− y > Kx+ q > y ( 2 ) with X : = { x ∈ Rn : l ≤ x ≤ u } , and Y : = { y ∈ Rm1+m2 : y1 : m1 ≥ 0 } . PDHG . When specialized to ( 2 ) , the PDHG algorithm takes the form : xk+1 = proj X ( xk − τ ( c−K > yk ) ) yk+1 = proj Y ( yk + σ ( q −K ( 2xk+1 − xk ) ) ) ( 3 ) where τ , σ > 0 are primal and dual step sizes , respectively . PDHG is known to converge to an optimal solution when τσ‖K‖22 ≤ 1 [ 20 , 21 ] . We reparameterize the step sizes by τ = η/ω and σ = ωη with η ∈ ( 0 , ∞ ) and ω ∈ ( 0 , ∞ ) . ( 4 ) We call ω ∈ ( 0 , ∞ ) the primal weight , and η ∈ ( 0 , ∞ ) the step size . Under this reparameterization PDHG converges for all η ≤ 1/‖K‖2 . This allows us to control the scaling between the primal and dual iterates with a single parameter ω . We use the term primal weight to describe ω because it weights the primal variables in the following norm : ‖z‖ω : = √ ω‖x‖22 + ‖y‖22 ω . This norm plays a role in the theory for PDHG [ 20 ] and later algorithmic discussions . For the baseline PDHG algorithm that we use for comparisons , we consider two simple choices for η and ω . For the step size , set η = 0.9/‖K‖2 where ‖K‖2 is estimated via power iteration , and for the primal weight we set ω = 1 ; this is similar to the default parameters in the standard PDHG implementation in ODL [ 2 ] .

Summary: The paper applies the primal-dual hybrid-gradient (PDHG) method (also sometimes referred to as Chambolle-Pock algorithm) to solve very large linear programs. The motivation is to solve problems which are out of reach for commerical LP solvers that are based on simplex or interior-point approaches. Moreover, the algorithm is admissible to highly parallel computation due to the simple update rules of PDHG.   The main contribution is a careful combination of existing heuristic which are known to speed up PDHG in practice: restarts, "line search" and preconditioning.  In experiments, this combination of heuristics is shown to substantially improve a basic PDHG baseline. Contrary to existing wisdom, it is shown that FOMs, when enhanced with good heuristics, can indeed find highly accurate solutions to large-scale problems.