Repository: scipy/scipy

Issue:
linprog benchmarking and proposal to make 'interior-point' the default method
In support of #9263, I benchmarked the new revised simplex implementation against the other `scipy.optimize.linprog` methods and two other open source LP solvers with Python interfaces: [CVXOPT ](https://cvxopt.org/) and [GLPK](https://www.gnu.org/software/glpk/). I ask the community for feedback on this PR and suggest that in addition to adding the revised simplex method, interior-point should replace the original simplex as the new *default* method for linprog.

### Background
To my knowledge, CVXOPT is the only other popular, open source linear programming solver with a Python interface that is actually *implemented* in Python. GLPK is implemented in ANSI C but CVXOPT provides a Python interface. Both are released under the GNU GPL 3. 

I intended to test [COIN CBC](https://projects.coin-or.org/Cbc), but of the four Python interface options I could find, [PuLP ](https://pythonhosted.org/PuLP/) and [Pyomo ](http://www.pyomo.org/) could not directly interpret input from MPS files or numpy arrays, [CyLP](https://github.com/coin-or/CyLP) is incompatible with Python 3, and installation of [yaposib](https://pypi.org/project/yaposib/), which has not been updated since 2012, failed.

Other popular open source LP solvers exist, but testing was restricted to those with Python interfaces, and of those GLPK and COIN CBC were the fastest and most robust according to [this report](https://prod-ng.sandia.gov/techlib-noauth/access-control.cgi/2013/138847.pdf) prepared at Sandia National Laboratories in late 2013.

### Methods
All linprog methods, CVXOPT, and GLPK were tested on the 92 Netlib problems without `RANGES` sections in their definitions. (RANGES is just an alternative way of expressing inequality constraints in the MPS format; most of the Netlib problems express inequality constraints otherwise.) Tests were performed on Macbook Pro (Early 2013 , 2.6 GHz intel Core i5, 8 GB 1600 MHz DDR3 RAM) with macOS 10.13.6 and the Anaconda distribution of Python 3.6.6. 

`scipy.optimize.linprog` was tested in a virtual environment with a development version of SciPy and relevant dependencies NumPy 1.15.0, MKL 2018.0.3, and SuiteSparse 5.2.0. This virtual environment was cloned for testing of CVXOPT/GLPK, as installing CVXOPT seemed to break my Spyder installation in the past. Installation of CVXOPT according to the [official instructions](http://cvxopt.org/install/) downgraded SuiteSparse to 4.5.6 and linked NumPy and SuiteSparse to OpenBLAS 0.3.3.

Solution times were measured using `time.perf_counter`; they include the time of loading the problem from the relevant MPS (GLPK and CVXOPT) or .npz file (linprog) and calling the solver within a `try`-`except` block.

### Results

**Dense Solvers**
Although the Netlib benchmark problems tend to be quite sparse (see [problem summary table](http://www.netlib.org/lp/data/readme)), I solve them with dense matrix techniques options to estimate performance on problems that are inherently dense. Results for GLPK are not shown because GLPK does not seem to use dense techniques when controlled via CVXOPT's interface. That is, CVXOPT exposes an option that is supposed to provide this control, but GLPK is consistently faster in 'dense' mode than 'sparse' mode for every problem. This and the fact that the speed difference is usually quite small suggests that only one solution technique is actually being used, and given its speed, it is likely employing sparse matrix techniques.

*Speed*
Bar graphs show elapsed time in seconds for each problem and each solver if:

- the solver reported success, and 
- the reported value of the objective function was within 0.1% of the accepted value.

Otherwise, the value shown is 0.
Each colored bar represents a combination of a problem, indicated by the bar's horizontal location, and solver, indicated by the bar's color; the elapsed time is represented by the position of the _top_ of each bar on the vertical axis. (This is _not_ the visible length of each bar; the bars are layered in order of speed, as if in a third dimension, with the fastest solver - and shortest bar - for the particular problem in front.)

<div>
    <a href="https://plot.ly/~mdhaber/104/?share_key=Qr3EsljqwnMBSGP4xAF0Xv" target="_blank" title="Dense LP Solver Benchmarking" style="display: block; text-align: center;"><img src="https://plot.ly/~mdhaber/104.png" alt="Dense LP Solver Benchmarking" style="max-width: 100%;width: 1250px;"  width="1250" onerror="this.onerror=null;this.src='https://plot.ly/404.png';" /></a>
</div>

 - linprog (interior point) is faster than any other solver for every problem it solves.
 - linprog (simplex) solves only a few problems. For each of these, linprog (revised simplex) is faster.
 - Of the problems solved by both linprog (revised simplex) and CVXOPT, linprog (revised simplex) tends  to be faster for problems that take more than a second to solve.
 - There are a few problems that linprog (revised simplex) solves that no other solver solves.
 - There are a few problems that only CVXOPT solves with default options. Note, however, that at least one SciPy solver can solve each of these with appropriate settings.

 
*Reliability*
Pie charts depict the percentage of problems for which the solver:

- reported success and achieved reasonable accuracy (True Positive),
- reported failure yet achieved reasonable accuracy (False Negative),
- reported failure and failed to achieve reasonable accuracy (True Negative), and
- reported success yet failed to achieve reasonable accuracy (False Positive),

where 'reasonable accuracy' means that the objective function is within 0.1% of the accepted value.

<div>
    <a href="https://plot.ly/~mdhaber/106/?share_key=oNuP6TSKuRiWNn06BxQnp8" target="_blank" title="Dense LP Solver Benchmarking Reliability" style="display: block; text-align: center;"><img src="https://plot.ly/~mdhaber/106.png" alt="Dense LP Solver Benchmarking Reliability" style="max-width: 100%;width: 1250px;"  width="1250" onerror="this.onerror=null;this.src='https://plot.ly/404.png';" /></a>
</div>

* linprog (interior-point) is the most reliable.
* There is a considerably percentage of problems for which linprog (interior-point) reports failure but is reasonably accurate. This typically occurs because linprog (interior-point)'s internal criterion for success is much stricter than the 0.1% tolerance used here.
* linprog (revised simplex) is arguably less reliable than CVXOPT, but certainly more reliable than linprog (simplex)

**Sparse Solvers**
The results for the sparse solvers are represented in the same way.  Due to the apparent error in CVXOPT's interface to GLPK, results are for GLPK with the 'dense' option as it is always faster than the 'sparse' option.

*Speed*
<div>
    <a href="https://plot.ly/~mdhaber/108/?share_key=cJRgt4ZsFXDOhuqy0sxjaZ" target="_blank" title="Sparse LP Solver Benchmarking" style="display: block; text-align: center;"><img src="https://plot.ly/~mdhaber/108.png" alt="Sparse LP Solver Benchmarking" style="max-width: 100%;width: 1250px;"  width="1250" onerror="this.onerror=null;this.src='https://plot.ly/404.png';" /></a>
</div>

- Of the problems solved by both linprog (interior point) and CVXOPT, linprog (interior point) is typically faster except for the smallest problems.
- Of the problems solved by both linprog (interior point) and GLPK, linprog(interior-point) is often faster, but not always.

*Reliability*

<div>
    <a href="https://plot.ly/~mdhaber/110/?share_key=ymKcQe0UqH6WKxBOXVd2AG" target="_blank" title="Sparse LP Solver Benchmarking Reliability" style="display: block; text-align: center;"><img src="https://plot.ly/~mdhaber/110.png" alt="Sparse LP Solver Benchmarking Reliability" style="max-width: 100%;width: 1250px;"  width="1250" onerror="this.onerror=null;this.src='https://plot.ly/404.png';" /></a>
</div>

* GLPK is the most reliable, but linprog (interior-point) is quite reliable, too. Both are considerably more reliable than CVXOPT.
* There is a considerably percentage of problems for which linprog (interior-point) reports failure but is reasonably accurate. This typically occurs because linprog (interior-point)'s internal criterion for success is much stricter than the 0.1% tolerance used here.

### Discussion and Conclusion
Although the methods in `scipy.optimize.linprog` are not always as fast and reliable as GLPK, the sparse interior point option certainly rivals GLPK's speed. Also, a closer look at the failure state messages for the interior point options suggests that linprog's internal success criterion may be too strict, as it frequently reports failure even though it achieves reasonable accuracy. Adjusted appropriately, the sparse solver may approach GLPK in terms of reliability. In any case, the interior point options are significantly faster and more reliable than CVXOPT, the only other popular open source linear programming solver implemented in Python. Although there is much room for improvement, the new dense revised simplex method is a valuable addition to linprog because it 

1. provides *basic* (vertex of the polytope defined by the constraints) solutions to a wider variety of problems faster than the original simplex implementation, 
2. offers greater accuracy than the interior point options, and 
3. solves some problems for which all other methods fail or report failure.

An elusive bug in the linprog docstring is causing #9263 the documentation build test to fail, but I suggest that once it is passing, a final commit should make 'interior point' the default linprog method, and #9263 should be merged.

_This work was supported by a NumFOCUS Small Development Grant_



Discussion:
Such a nice read. If we relax the solution success strictness, does it also increase the false positives? Or do we have a rough idea to what it would lead on ill-conditioned problems? 
> I intended to test COIN CBC, but of the four Python interface options I could find, PuLP and Pyomo could not directly interpret input from MPS files or numpy arrays, CyLP is incompatible with Python 3, and installation of yaposib, which has not been updated since 2012, failed.

@mdhaber Have you considered [google `or-tools`](https://developers.google.com/optimization/) for benchmarking? It is written in C++ and the python interface which it provides is [python 3 compatible](https://developers.google.com/optimization/install/python/linux#python-3), provides [`mps_reader`](https://developers.google.com/optimization/reference/lp_data/mps_reader/) through the C++ interface, and is in [active development](https://github.com/google/or-tools/releases).


Very nice @mdhaber :)

> An elusive bug in the linprog docstring is causing #9263 the documentation build test to fail, but I suggest that once it is passing, a final commit should make 'interior point' the default linprog method,

This seems reasonable based on these results. Technically that's a backwards compatibility break, but I think it's a minor one and it will make the default behavior a lot better.
@ilayn Thanks! These false positives didn't occur in the original benchmarks for the 'interior-point' PR #7123, so I think they started when the different linprog methods started to share pre- and post-processing. We let something slip through the cracks - now interior point is terminating when mathemagical quantities fall below a default tolerance of 1e-8 yet post-processing declares failure if any constraint is not satisfied to 1e-12. Clearly there's a double standard we need to address! I think we already started talking about increasing this 1e-12 tolerance for other reasons, too. So I think there is a happy medium for these numbers that will not cause as many false negatives and won't introduce false positives for reasonable problems.

@jGaboardi now that you mention it I remember running across that before, but not when I was looking for popular Python-interface LP solvers recently. So I might give that a shot. Thanks!
@mdhaber I've and a look at the tolerance and don't think anything has "slipped" through - rather I set a lower tolerance as default. The original interior-point's tolerance was 1e-9 in presolve,1e-8 in the body and sqrt(1e-8) postsolve.  When setting the tolerance I seem to have copied the tolerance used in presolve, not the tolerance used in the body of the interior-point method.

I've also increased the tolerance in the post processing slightly, from ``sqrt(tol)`` to ``sqrt(tol)  * 10``, this was required for some of the simplex cases to pass. Could you try set `tol=1e-8` explicitly and see if this improves anything?
@kai-striega i didn't mean that you made a  mistake! Sorry if it sounded like that.
Looks like I misread the code when I was writing that last post, too :-\ I thought that the lower tolerance 1e-12 was being used for interior point postsolve. thank for setting me straight. I need to dig into this.
@mdhaber no offense taken =). This *actually* was a minor mistake, the tolerance should have been set to 1e-8 as it was in interior-point, not 1e-9 as it was in pre-solve. At this point, `tol` is rather poorly defined. The simplex documentation defines tol as:

> The tolerance which determines when a solution is “close enough” to zero in Phase 1 to be considered a basic feasible solution or close enough to positive to serve as an optimal solution.

note that this does not mention pre/post solving. Maybe (in another issue) we need to discuss how the tolerance should be defined (e.g. one for pre/post solve and one exposed to the user) and assess the reliability of the solvers at different tolerance levels.

Resolving patch:
diff --git a/scipy/optimize/_linprog.py b/scipy/optimize/_linprog.py
--- a/scipy/optimize/_linprog.py
+++ b/scipy/optimize/_linprog.py
@@ -162,7 +162,7 @@ def linprog_terse_callback(res):
 
 
 def linprog(c, A_ub=None, b_ub=None, A_eq=None, b_eq=None,
-            bounds=None, method='simplex', callback=None,
+            bounds=None, method='interior-point', callback=None,
             options=None, x0=None):
     r"""
     Linear programming: minimize a linear objective function subject to linear
@@ -219,7 +219,7 @@ def linprog(c, A_ub=None, b_ub=None, A_eq=None, b_eq=None,
         (all decision variables are non-negative).
         If a single tuple ``(min, max)`` is provided, then ``min`` and
         ``max`` will serve as bounds for all decision variables.
-    method : str, optional
+    method : {'interior-point', 'revised simplex', 'simplex'}, optional
         The algorithm used to solve the standard form problem.
         :ref:`'interior-point' <optimize.linprog-interior-point>` (default),
         :ref:`'revised simplex' <optimize.linprog-revised_simplex>`, and
@@ -461,15 +461,32 @@ def linprog(c, A_ub=None, b_ub=None, A_eq=None, b_eq=None,
     >>> x1_bounds = (-3, None)
     >>> from scipy.optimize import linprog
     >>> res = linprog(c, A_ub=A, b_ub=b, bounds=[x0_bounds, x1_bounds])
+
+    Note that the default method for `linprog` is 'interior-point', which is
+    approximate by nature.
+
+    >>> print(res)
+         con: array([], dtype=float64)
+         fun: -21.99999984082494 # may vary
+     message: 'Optimization terminated successfully.'
+         nit: 6 # may vary
+       slack: array([3.89999997e+01, 8.46872439e-08] # may vary
+      status: 0
+     success: True
+           x: array([ 9.99999989, -2.99999999]) # may vary
+
+    If you need greater accuracy, try 'revised simplex'.
+
+    >>> res = linprog(c, A_ub=A, b_ub=b, bounds=[x0_bounds, x1_bounds], method='revised simplex')
     >>> print(res)
          con: array([], dtype=float64)
-         fun: -22.0
+         fun: -22.0 # may vary
      message: 'Optimization terminated successfully.'
-         nit: 5 # may vary
+         nit: 1 # may vary
        slack: array([39.,  0.]) # may vary
       status: 0
      success: True
-           x: array([10., -3.])
+           x: array([10., -3.]) # may vary
 
     """
     meth = method.lower()


Test patch: