Repository: scipy/scipy

Issue:
Highs solver did not provide a solution nor did it report a failure
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

My issue is about `scipy.optimize.linprog(c, A_ub=A_ub, b_ub=b_ub, A_eq=A_eq, b_eq=b_eq, method='highs')` telling me to submit a bug report.

#### Reproducing code example:
See minimal reproducer in a comment
https://github.com/scipy/scipy/issues/13610#issuecomment-785620302

#### Error message:
```
/home/chn/env/lib/python3.8/site-packages/scipy/optimize/_linprog_highs.py:316: OptimizeWarning: model_status is not optimal, using scaled_model_status instead.
  res = _highs_wrapper(c, A.indptr, A.indices, A.data, lhs, rhs,
The solver did not provide a solution nor did it report a failure. Please submit a bug report.
```

#### Scipy/Numpy/Python version information:

```
1.6.1 1.18.3 sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0)
```



Discussion:
Testcase is around 600k unzipped. Sorry to not trying to minimize the test case.

[reproducer.py.gz](https://github.com/scipy/scipy/files/6038754/reproducer.py.gz)

@kalvdans thanks very much for reporting this! 
Would you be willing to add this as a [gist](https://gist.github.com/) instead of the zipped file?
This is actually an interesting case I don't think I ever expected to see (`model_status != scaled_model_status`, that is), so I'm interested to see what this problem is.  A gist would be very helpful
Ok.. but it has 20000 lines: https://gist.github.com/kalvdans/087c71acda7fb1064d0e4cdc07e85759
One thing is that these arrays are sparse but they're provided as dense. HiGHS might be choking on that. I'm trying to convert them `csc_matrix` but Spyder is hanging : )
_Update: I do suggest using `scipy.sparse.csc_matrix` in the future, but that did not fix this problem._
A little background; I tried to follow https://stackoverflow.com/questions/17067351/polynomial-fitting-using-l1-norm#answer-17071311 which results in huge diagonal matrices. Thanks for the tip about using sparse matrices.
Haven't started looking at this -- just fishing for information.  Does this fail for the other linprog methods other than highs?
I've been running it with "interior-point" using sparse matrices. It's definitely too big for the other methods, so I didn't bother, and "interior-point" is still running.
The best I can do tonight is this simpler reproducer -- I'll be able to dig in more tomorrow.  The trouble seems to be from the size of `A_ub`.  It breaks for me when `A_ub` is `(130, 73)` or larger (when `n > 64` in the reproducer below).

Note: I can get higher with `highs-ipm`, the trouble is with `highs-ds` I believe.  Again -- I'll have more time to look at this tomorrow.  In the meantime try switching to `highs-ipm` and see what happens

```python
'''Simple LAD regression.'''

import numpy as np
from scipy.optimize import linprog
from scipy.sparse import lil_matrix, csc_matrix

if __name__ == '__main__':
    # Training data is {(x0, y0), (x1, y2), ..., (xn-1, yn-1)}
    #     x \in R^d
    #     y \in R
    #
    # n: number of training samples
    # d: dimension of x, i.e. x \in R^d
    # phi: feature map R^d -> R^m
    # m: dimension of feature space

    # size of features and feature space
    m, d = 8, 9

    # number of samples (this is where the trouble is!)
    # 64: works
    # 65: reproduces the strange behavior
    n = 64

    # any feature mapping will do -- why not a random one
    np.random.seed(0) # reproducibility
    phi = np.random.normal(0, 1, size=(m, d))

    # we are omnipotent, so we get to know what the truth is
    w_true = np.random.randn(m)

    # any features will do
    x = np.random.normal(0, 1, size=(d, n))

    # corresponding measurements -- with a little spice
    y = w_true @ (phi @ x) + np.random.normal(0, 1e-5, size=n)

    # construct the problem
    c = np.ones(m+n)
    c[:m] = 0
    A_ub = lil_matrix((2*n, n+m))
    idx = 0
    for ii in range(n):
        A_ub[idx, :m] = phi @ x[:, ii]
        A_ub[idx, m+ii] = -1
        A_ub[idx+1, :m] = -1*phi @ x[:, ii]
        A_ub[idx+1, m+ii] = -1
        idx += 2
    A_ub = A_ub.tocsc()
    b_ub = np.zeros(2*n)
    b_ub[0::2] = y
    b_ub[1::2] = -y
    bnds = [(None, None)]*m + [(0, None)]*n

    # solve
    res = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=bnds, method='highs')  # use 'highs-ipm' to avoid issues -- seems to work!

    # did strange things happen?
    if res.x is None and res.success:
        raise ValueError('Strange things are the foot')

    # check
    assert np.allclose(res.x[:m], w_true, atol=1e-5)
    print('We did it!')
```
Update: using the HiGHS standalone solver, the LAD regression problem doesn't show the warning about model and scaled model status difference, so it appears to be an issue with how the problem is passed through by the Cython bindings
@kalvdans I have a fix coming (see #13626) but it's currently blocked by an [upstream bug](https://github.com/ERGO-Code/HiGHS/pull/481).  In the meantime, you can try running your model with the `'disp': True` option to see if it gives you some hints about primal/dual feasibility tolerance adjustments.  Tightening those tolerances removes the scaled/unscaled model status warning and might help give you a solution for dual simplex
> I don't think I ever expected to see (model_status != scaled_model_status

This happens surprisingly frequently in solvers. 

CPLEX has a great status [`CPX_STAT_OPTIMAL_INFEAS`](https://www.ibm.com/support/knowledgecenter/SSSA5P_20.1.0/ilog.odms.cplex.help/refcallablelibrary/macros/CPX_STAT_OPTIMAL_INFEAS.html), which means it found an optimal solution to the scaled model, but that the solution was infeasible in the unscaled model, and CPLEX could not repair the solution to feasibility. I guess HiGHS is doing something similar here as well.
> CPLEX has a great status CPX_STAT_OPTIMAL_INFEAS, which means it found an optimal solution to the scaled model, but that the solution was infeasible in the unscaled model, and CPLEX could not repair the solution to feasibility. I guess HiGHS is doing something similar here as well.

Thanks for pointing this out.  This is certainly a case that we can easily catch and might be worth creating a new `linprog` status code for -- thoughts @mdhaber ? 
We have a catch-all code 4 which, in the rendered documentation, says "numerical difficulties encountered". It is the code that @kalvdans would have received along with the error message. It's also used for all the solvers when we do a sanity check of the "optimal" solution and find that it doesn't satisfy the constraints. I think it's reasonable to provide a more specific error message, but I'm not sure if we want a new status code (unless we want to get more specific with status codes in general).
Yep, that makes sense @mdhaber, no new status code needed.  In the PR I have provided a (hopefully) more useful message when this case does occur

Resolving patch:
diff --git a/scipy/optimize/_highs/setup.py b/scipy/optimize/_highs/setup.py
--- a/scipy/optimize/_highs/setup.py
+++ b/scipy/optimize/_highs/setup.py
@@ -105,6 +105,9 @@ def configuration(parent_package='', top_path=None):
     # highs_wrapper:
     ipx_sources = _get_sources('src/CMakeLists.txt', 'set(ipx_sources\n', ')')
     highs_sources = _get_sources('src/CMakeLists.txt', 'set(sources\n', ')')
+    # filter out MIP sources until MIP is officially supported
+    highs_sources = [s for s in highs_sources
+                     if pathlib.Path(s).parent.name != 'mip']
     ext = config.add_extension(
         '_highs_wrapper',
         sources=['cython/src/_highs_wrapper.cxx'] + highs_sources + ipx_sources,
@@ -132,32 +135,6 @@ def configuration(parent_package='', top_path=None):
     # Add c++11/14 support:
     ext._pre_build_hook = pre_build_hook
 
-    # wrapper around HiGHS writeMPS:
-    ext = config.add_extension(
-        '_mpswriter',
-        sources=[
-            # we should be using using highs shared library;
-            # next best thing is compiling minimal set of sources
-            'cython/src/_mpswriter.cxx',
-            'src/util/HighsUtils.cpp',
-            'src/io/HighsIO.cpp',
-            'src/io/HMPSIO.cpp',
-            'src/lp_data/HighsModelUtils.cpp',
-            'src/util/stringutil.cpp',
-        ],
-        include_dirs=[
-            'cython/src/',
-            'src/',
-            'src/io/',
-            'src/lp_data/',
-        ],
-        language='c++',
-        libraries=['basiclu'],
-        define_macros=DEFINE_MACROS,
-        undef_macros=UNDEF_MACROS,
-    )
-    ext._pre_build_hook = pre_build_hook
-
     # Export constants and enums from HiGHS:
     ext = config.add_extension(
         '_highs_constants',
diff --git a/scipy/optimize/_linprog_highs.py b/scipy/optimize/_linprog_highs.py
--- a/scipy/optimize/_linprog_highs.py
+++ b/scipy/optimize/_linprog_highs.py
@@ -311,11 +311,10 @@ def _linprog_highs(lp, solver, time_limit=None, presolve=True,
             simplex_dual_edge_weight_strategy_enum,
         'simplex_strategy': HIGHS_SIMPLEX_STRATEGY_DUAL,
         'simplex_crash_strategy': HIGHS_SIMPLEX_CRASH_STRATEGY_OFF,
+        'ipm_iteration_limit': maxiter,
+        'simplex_iteration_limit': maxiter,
     }
 
-    options['ipm_iteration_limit'] = maxiter
-    options['simplex_iteration_limit'] = maxiter
-
     # np.inf doesn't work; use very large constant
     rhs = _replace_inf(rhs)
     lhs = _replace_inf(lhs)
@@ -338,16 +337,26 @@ def _linprog_highs(lp, solver, time_limit=None, presolve=True,
     # this needs to be updated if we start choosing the solver intelligently
     solvers = {"ipm": "highs-ipm", "simplex": "highs-ds", None: "highs-ds"}
 
+    # HiGHS will report OPTIMAL if the scaled model is solved to optimality
+    # even if the unscaled original model is infeasible;
+    # Catch that case here and provide a more useful message
+    if ((res['status'] == MODEL_STATUS_OPTIMAL) and
+            (res['unscaled_status'] != res['status'])):
+        _unscaled_status, unscaled_message = statuses[res["unscaled_status"]]
+        status, message = 4, ('An optimal solution to the scaled model was '
+                              f'found but was {unscaled_message} in the '
+                              'unscaled model. For more information run with '
+                              'the option `disp: True`.')
+    else:
+        status, message = statuses[res['status']]
+
     sol = {'x': np.array(res['x']) if 'x' in res else None,
            'slack': slack,
-           # TODO: Add/test dual info like:
-           # 'lambda': res.get('lambda'),
-           # 's': res.get('s'),
            'fun': res.get('fun'),
            'con': con,
-           'status': statuses[res['status']][0],
+           'status': status,
            'success': res['status'] == MODEL_STATUS_OPTIMAL,
-           'message': statuses[res['status']][1],
+           'message': message,
            'nit': res.get('simplex_nit', 0) or res.get('ipm_nit', 0),
            'crossover_nit': res.get('crossover_nit'),
            }


Test patch:
diff --git a/scipy/optimize/tests/test_linprog.py b/scipy/optimize/tests/test_linprog.py
--- a/scipy/optimize/tests/test_linprog.py
+++ b/scipy/optimize/tests/test_linprog.py
@@ -199,6 +199,41 @@ def nontrivial_problem():
     return c, A_ub, b_ub, A_eq, b_eq, x_star, f_star
 
 
+def l1_regression_prob(seed=0, m=8, d=9, n=100):
+    '''
+    Training data is {(x0, y0), (x1, y2), ..., (xn-1, yn-1)}
+        x in R^d
+        y in R
+    n: number of training samples
+    d: dimension of x, i.e. x in R^d
+    phi: feature map R^d -> R^m
+    m: dimension of feature space
+    '''
+    np.random.seed(seed)
+    phi = np.random.normal(0, 1, size=(m, d))  # random feature mapping
+    w_true = np.random.randn(m)
+    x = np.random.normal(0, 1, size=(d, n))  # features
+    y = w_true @ (phi @ x) + np.random.normal(0, 1e-5, size=n)  # measurements
+
+    # construct the problem
+    c = np.ones(m+n)
+    c[:m] = 0
+    A_ub = scipy.sparse.lil_matrix((2*n, n+m))
+    idx = 0
+    for ii in range(n):
+        A_ub[idx, :m] = phi @ x[:, ii]
+        A_ub[idx, m+ii] = -1
+        A_ub[idx+1, :m] = -1*phi @ x[:, ii]
+        A_ub[idx+1, m+ii] = -1
+        idx += 2
+    A_ub = A_ub.tocsc()
+    b_ub = np.zeros(2*n)
+    b_ub[0::2] = y
+    b_ub[1::2] = -y
+    bnds = [(None, None)]*m + [(0, None)]*n
+    return c, A_ub, b_ub, bnds
+
+
 def generic_callback_test(self):
     # Check that callback is as advertised
     last_cb = {}
@@ -1638,6 +1673,7 @@ def test_crossover(self):
         # there should be nonzero crossover iterations for IPM (only)
         assert_equal(res.crossover_nit == 0, self.method != "highs-ipm")
 
+
 ################################
 # Simplex Option-Specific Tests#
 ################################
@@ -1955,6 +1991,19 @@ class TestLinprogHiGHSSimplexDual(LinprogHiGHSTests):
     method = "highs-ds"
     options = {}
 
+    def test_lad_regression(self):
+        '''The scaled model should be optimal but unscaled model infeasible.'''
+        c, A_ub, b_ub, bnds = l1_regression_prob()
+        res = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=bnds,
+                      method=self.method, options=self.options)
+        assert_equal(res.status, 4)
+        assert_('An optimal solution to the scaled '
+                'model was found but' in res.message)
+        assert_(res.x is not None)
+        assert_(np.all(res.slack > -1e-6))
+        assert_(np.all(res.x <= [np.inf if u is None else u for l, u in bnds]))
+        assert_(np.all(res.x >= [-np.inf if l is None else l for l, u in bnds]))
+
 
 ###################################
 # HiGHS-IPM Option-Specific Tests #