Repository: scipy/scipy

Issue:
linprog: Differents solution with scipy 0.19.1 vs 1.2.1
<!-- 
Thank you for taking the time to report a SciPy issue.
Please describe the issue in detail, and for bug reports
fill in the fields below. You can delete the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
-->

I have a simple linear programmin problem with only equality constraints. When I try to solve it with scipy (v. 0.19.1), I get a succesful solution:
![imagen](https://user-images.githubusercontent.com/45110745/60055103-2324e680-96a2-11e9-945a-a7f1efb9d76a.png)

However, whe I try to solve the same problem (the same code below) i get an error with a linear combination in matrix A.

![imagen](https://user-images.githubusercontent.com/45110745/60055067-09839f00-96a2-11e9-8b9c-080ea49b1a65.png)


### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be marked as a code block automatically 
-->


```
import numpy as np
import scipy.optimize as opt

A_eq = np.array([[1,1,0,0,0,0],
                [0,0,1,1,0,0],
                [0,0,0,0,1,1],
                [1,0,1,0,0,0],
                [0,0,0,1,1,0],
                [0,1,0,0,0,1]])
b_eq=np.array([221,210,10,141,198,102])
c=np.concatenate((0,1,np.zeros(4)),axis=None)

sol=opt.linprog(c,A_eq=A_eq,b_eq=b_eq)
sol

```


### Scipy/Numpy/Python version information: 
Working: 
Scipy: 0.19.1 
Numpy: 1.15.4 


Not working:
Scipy: 1.2.1 
Numpy: 1.16.4 


<!-- You can simply run the following and paste the result in a code block
```
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
```
-->



Discussion:
The solution given by v0.19 `linprog` (by default `method = 'simplex'`) is wrong.
```
x = [129,92,12,198,0,10]
A_eq@x - b_eq
```
I get:
```
array([   0,    0,    0, -131,    0,    0])
```
So your equality constraints were not satisfied.
The current version of `linprog` is telling you that it detects in presolve that the problem is infeasible. First it notices that the `A_eq` matrix is not full rank:
```
import numpy as np
np.linalg.matrix_rank(A_eq)
```
The output is `5`, so there is a linear combination of five of the rows of `A_eq` that equals the sixth row. `linprog` finds that linear combination and notices that the same linear combination of the corresponding five elements of `b_eq` does not equal the sixth element of `b_eq`, and therefore it is impossible to satisfy the constraints.
> 
> 
> The solution given by v0.19 `linprog` (by default `method = 'simplex'`) is wrong.
> 
> ```
> x = [129,92,12,198,0,10]
> A_eq@x - b_eq
> ```
> 
> I get:
> 
> ```
> array([   0,    0,    0, -131,    0,    0])
> ```
> 
> So your equality constraints were not satisfied.

Sorry, my mistake. The A_eq matrix had an error, but I have already updated it.

Now, i check the condition

```
x = [129,92,12,198,0,10]
A_eq@x - b_eq
```

I get

```
array([0, 0, 0, 0, 0, 0])
```
OK. Thanks for finding this. I'll have to take a closer look at the presolve routine. It is probably a numerical tolerance issue.

In the meantime, try:
```
sol=opt.linprog(c,A_eq=A_eq,b_eq=b_eq, options={'presolve':False})
```
The solution you posted is feasible for the new constraint matrix, but not optimal. The optimal solution seems to be:
```
[119., 102.,  22., 188.,  10.,   0.]
```
@zapatacc If you find any other examples, please let me know. 
@zapatacc have you tried this in version 1.3? I'm not able to reproduce this error in the current development version.
I can reproduce this is in 1.2.x but not 1.3.x or the current development version. If I recall, 1.2 is a long-term support release so if you need this fix (whatever it was... I can't remember think of what it might have been now, though) backported, let me know. Otherwise, please try 1.3.
@mdhaber If it is important to backport please do put the backport-candidate label on the appropriate PR. I do remember making one or two decisions to avoid 1.2.x backports where the merge conflicts were truly gruesome (not sure if this was one), but if it is important we can likely do it.
@mdhaber I *can* reproduce this on the current dev version, plus I've got some extra warnings, maybe those will be useful? I haven't looked at this in depth, but, if you can't reproduce it, I can take a more thorough look at this.

**EDIT** @zapatacc edited his problem slightly (c2: -1 -> 1)  I've edited my examples to match the new input.

```python
Python 3.7.3 (default, Apr  3 2019, 19:16:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np; np.__version__                                                                                                                                                                          
Out[1]: '1.16.4'

In [2]: import scipy as sp; sp.__version__                                                                                                                                                                          
Out[2]: '1.4.0.dev0+7d1481e'
```

Running the problem:

```python
(scipy-dev) kai@kaiT470:~/scipy-git$ ipython
Python 3.7.3 (default, Apr  3 2019, 19:16:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np                                                                                                                                                                                          

In [2]: import scipy.optimize as opt                                                                                                                                                                                

In [3]: A_eq = np.array([[1,1,0,0,0,0], 
   ...:                 [0,0,1,1,0,0], 
   ...:                 [0,0,0,0,1,1], 
   ...:                 [1,0,1,0,0,0], 
   ...:                 [0,0,0,1,1,0], 
   ...:                 [0,1,0,0,0,1]])                                                                                                                                                                             

In [4]: b_eq=np.array([221,210,10,141,198,102])                                                                                                                                                                     

In [5]: c=np.concatenate((0,1,np.zeros(4)),axis=None)                                                                                                                                                              

In [6]: opt.linprog(c,A_eq=A_eq,b_eq=b_eq)                                                                                                                                                                          
/home/kai/envs/scipy-dev/bin/ipython:1: OptimizeWarning: A_eq does not appear to be of full row rank. To improve performance, check the problem formulation for redundant equality constraints.
  #!/home/kai/envs/scipy-dev/bin/python3.7
Out[6]: 
     con: array([221., 210.,  10., 141., 198., 102.])
     fun: 0.0
 message: 'There is a linear combination of rows of A_eq that results in zero, suggesting a redundant constraint. However the same linear combination of b_eq is nonzero, suggesting that the constraints conflict and the problem is infeasible.'
     nit: 0
   slack: array([], dtype=float64)
  status: 2
 success: False
       x: array([0., 0., 0., 0., 0., 0.])

In [7]: opt.linprog(c, A_eq=A_eq, b_eq=b_eq, options={'presolve': False})                                                                                                                                           
/home/kai/envs/scipy-dev/bin/ipython:1: OptimizeWarning: Solving system with option 'cholesky':True failed. It is normal for this to happen occasionally, especially as the solution is approached. However, if you see this frequently, consider setting option 'cholesky' to False.
  #!/home/kai/envs/scipy-dev/bin/python3.7
/home/kai/envs/scipy-dev/bin/ipython:1: OptimizeWarning: Solving system with option 'sym_pos':True failed. It is normal for this to happen occasionally, especially as the solution is approached. However, if you see this frequently, consider setting option 'sym_pos' to False.
  #!/home/kai/envs/scipy-dev/bin/python3.7
/home/kai/scipy-git/scipy/optimize/_linprog_ip.py:110: LinAlgWarning: Ill-conditioned matrix (rcond=1.72451e-17): result may not be accurate.
  return sp.linalg.solve(M, r, sym_pos=sym_pos)
/home/kai/scipy-git/scipy/optimize/_linprog_ip.py:110: LinAlgWarning: Ill-conditioned matrix (rcond=1.41574e-17): result may not be accurate.
  return sp.linalg.solve(M, r, sym_pos=sym_pos)
/home/kai/scipy-git/scipy/optimize/_linprog_ip.py:110: LinAlgWarning: Ill-conditioned matrix (rcond=1.12073e-17): result may not be accurate.
  return sp.linalg.solve(M, r, sym_pos=sym_pos)
Out[7]: 
     con: array([6.43106262e-08, 6.10796747e-08, 2.34924435e-09, 4.08180938e-08,
       5.75562069e-08, 2.93652533e-08])
     fun: 91.99999997512728
 message: 'Optimization terminated successfully.'
     nit: 6
   slack: array([], dtype=float64)
  status: 0
 success: True
       x: array([1.29000000e+02, 9.20000000e+01, 1.20000000e+01, 1.98000000e+02,
       2.14329427e-09, 1.00000000e+01])
```
Thanks for trying it @kai-striega. Sounds even more like a tolerance issue. Soon I'll suggest a few lines to query that would help debug.
NM looks like this is the same as #10038/#10059. Maybe another factor of 10 is in order?
> 
> 
> OK. Thanks for finding this. I'll have to take a closer look at the presolve routine. It is probably a numerical tolerance issue.
> 
> In the meantime, try:
> 
> ```
> sol=opt.linprog(c,A_eq=A_eq,b_eq=b_eq, options={'presolve':False})
> ```
> 
> The solution you posted is feasible for the new constraint matrix, but not optimal. The optimal solution seems to be:
> 
> ```
> [119., 102.,  22., 188.,  10.,   0.]
> ```

@mdhaber: Again, my mistake, I'm so sorry. I'm researching both min and max problem, so when I did the copy/paste process to started the discussion, I forget change the vector c.

So, the real optimal solution for **min** problem is
```
array([129.,  92.,  12., 198.,   0.,  10.])
```
and the real optimal solution for **max** problem is
```
array([119., 102.,  22., 188.,  10.,   0.])
```

Now, with scipy 1.2.1 and with the next line, I get the real solution for both problems:
```
sol=opt.linprog(c,A_eq=A_eq,b_eq=b_eq, options={'presolve':False})
```

I'll try to solve it with 1.3 version.

Thank you!
@mdhaber in version 1.3.0 I get similar results like @Kai-Striega with warnings but the solver computed the real optimal solutions:
![imagen](https://user-images.githubusercontent.com/45110745/60129921-c8e85c00-975b-11e9-8c08-3f3b4a3f5819.png)
![imagen](https://user-images.githubusercontent.com/45110745/60129946-d9003b80-975b-11e9-83a5-2b05e7221a99.png)

OK. As suspected this is a tolerance issue. We bumped up the tolerance by a factor of 10 between 1.2 and 1.3, and that's why we're having more luck in 1.3. However, @Kai-Striega reports that the factor of 10 might not have been enough, so we can bump it up more.
 
The warnings that happen with presolve off are to be expected, as the constraint matrix is singular. Part of the the point of presolve is to help avoid those issues with singular matrices. Unfortunately, that is the part of presolve that was causing you trouble in 1.2. 

Going forward, know that in addition to the new default `method="interior-point"`, you can also specify `method="revised simplex"`, which will give you a more accurate solution without the warnings, at least on simple problems like this.
Ok, thanks @mdhaber.
Super-easy first PR: 

1. Before fixing the tolerance, add the problem above as a test in [`test_linprog.py`](https://github.com/scipy/scipy/blob/master/scipy/optimize/tests/test_linprog.py). 
2. Create a PR; confirming that it _fails_ at least one continuous integration check with the current tolerance.
3. Increase tolerance (see the change made in #10059) by an additional factor of 10. Confirm that all tests pass.

 It would be great to let someone participating in the SciPy Conference sprint fix this.
Hi Matt! 
I'd love to take on this issue if no one else has :) 
Would you mind letting me know if I need to set up anything besides my environment for [dev](https://scipy.github.io/devdocs/dev/contributor/quickstart_mac.html#quickstart-mac) and [testing](https://github.com/scipy/scipy/blob/master/INSTALL.rst.txt#testing)?
Yes, that sounds great, and yes, that should be all you need! Let me know how it goes following that Mac Quickstart guide. For testing, I suggest you take a look at the information [here](http://scipy.github.io/devdocs/dev/contributor/contributor_toc.html#unit-tests) instead. Thanks!
Thanks a lot for the references! I was able to set everything up following the guides :) 
For testing everything passed except `TestQRdelete_F.test_delete_last_p_col` with a similar error message as #5335:

```Python
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________ TestQRdelete_F.test_delete_last_p_col ___________________________________________________________________________________

self = <scipy.linalg.tests.test_decomp_update.TestQRdelete_F object at 0x123e020b8>

    def test_delete_last_p_col(self):
        a, q, r = self.generate('tall', 'full')
        q1, r1 = qr_delete(q, r, 0, a.shape[1], 'col')
>       assert_unitary(q1)

a          = array([[0.36852294+0.7065659j , 0.9763755 +0.8815727j ,
        0.73179656+0.5390478j , 0.6848747 +0.2577209j ,
      ...0.8069605j ,
        0.2923851 +0.9023696j , 0.08459105+0.7541054j ,
        0.8391204 +0.23152167j]], dtype=complex64)
q          = array([[-0.11519074-0.22085421j,  0.4031076 +0.25545135j,
         0.20641464-0.02725353j,  0.04916074+0.12054591j,
  ....17945503j,  0.10257003-0.11285198j,
        -0.05898733+0.03267026j,  0.5694973 -0.06679238j]],
      dtype=complex64)
q1         = array([[-0.11519074-0.22085421j,  0.4031076 +0.25545135j,
         0.20641464-0.02725353j,  0.04916074+0.12054591j,
  ....17945503j,  0.10257003-0.11285198j,
        -0.05898733+0.03267026j,  0.5694973 -0.06679238j]],
      dtype=complex64)
r          = array([[-3.1992414 +0.j        , -2.1899545 +0.23892963j,
        -1.6467261 +0.47293147j, -2.4033258 +0.41776413j,
  ...        ,
         0.        +0.j        ,  0.        +0.j        ,
         0.        +0.j        ]], dtype=complex64)
r1         = array([], shape=(12, 0), dtype=complex64)
self       = <scipy.linalg.tests.test_decomp_update.TestQRdelete_F object at 0x123e020b8>

scipy/linalg/tests/test_decomp_update.py:327: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = array([[-0.11519074-0.22085421j,  0.4031076 +0.25545135j,
         0.20641464-0.02725353j,  0.04916074+0.12054591j,
  ....17945503j,  0.10257003-0.11285198j,
        -0.05898733+0.03267026j,  0.5694973 -0.06679238j]],
      dtype=complex64)
rtol = 0.0001, atol = 2.384185791015625e-07, assert_sqr = True

    def assert_unitary(a, rtol=None, atol=None, assert_sqr=True):
        if rtol is None:
            rtol = 10.0 ** -(np.finfo(a.dtype).precision-2)
        if atol is None:
            atol = 2*np.finfo(a.dtype).eps
    
        if assert_sqr:
            assert_(a.shape[0] == a.shape[1], 'unitary matrices must be square')
        aTa = np.dot(a.T.conj(), a)
>       assert_allclose(aTa, np.eye(a.shape[1]), rtol=rtol, atol=atol)
E       AssertionError: 
E       Not equal to tolerance rtol=0.0001, atol=2.38419e-07
E       
E       Mismatch: 1.39%
E       Max absolute difference: 3.57627869e-07
E       Max relative difference: inf
E        x: array([[ 1.000000e+00+0.000000e+00j, -1.676381e-08+1.490116e-08j,
E                0.000000e+00+1.490116e-08j, -2.793968e-08-2.980232e-08j,
E               -2.235174e-08-1.862645e-08j, -2.095476e-08+5.587935e-09j,...
E        y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
E              [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
E              [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...

a          = array([[-0.11519074-0.22085421j,  0.4031076 +0.25545135j,
         0.20641464-0.02725353j,  0.04916074+0.12054591j,
  ....17945503j,  0.10257003-0.11285198j,
        -0.05898733+0.03267026j,  0.5694973 -0.06679238j]],
      dtype=complex64)
aTa        = array([[ 1.0000001e+00+0.0000000e+00j, -1.6763806e-08+1.4901161e-08j,
         0.0000000e+00+1.4901161e-08j, -2.793967...3484e-08-4.4703484e-08j,
        -8.3819032e-09-2.9802322e-08j,  9.9999994e-01+0.0000000e+00j]],
      dtype=complex64)
assert_sqr = True
atol       = 2.384185791015625e-07
rtol       = 0.0001

scipy/linalg/tests/test_decomp_update.py:21: AssertionError
```

@zchelseal I've gotten that error recently, too! I assumed it was just my computer's problem. Please submit a bug report.


Resolving patch:
diff --git a/scipy/optimize/_remove_redundancy.py b/scipy/optimize/_remove_redundancy.py
--- a/scipy/optimize/_remove_redundancy.py
+++ b/scipy/optimize/_remove_redundancy.py
@@ -432,7 +432,7 @@ def _remove_redundancy(A, b):
                        "off redundancy removal, or try turning off presolve "
                        "altogether.")
             break
-        if np.any(np.abs(v.dot(b)) > tol * 10):  # factor of 10 to fix 10038
+        if np.any(np.abs(v.dot(b)) > tol * 100):  # factor of 100 to fix 10038 and 10349
             status = 2
             message = ("There is a linear combination of rows of A_eq that "
                        "results in zero, suggesting a redundant constraint. "


Test patch:
diff --git a/scipy/optimize/tests/test_linprog.py b/scipy/optimize/tests/test_linprog.py
--- a/scipy/optimize/tests/test_linprog.py
+++ b/scipy/optimize/tests/test_linprog.py
@@ -1312,6 +1312,25 @@ def test_bug_10124(self):
                       method=self.method, options=self.options)
         _assert_success(res, desired_x=[10, -3], desired_fun=-22)
 
+    def test_bug_10349(self):
+        """
+        Test for redundancy removal tolerance issue
+        https://github.com/scipy/scipy/issues/10349
+        """
+        A_eq = np.array([[1, 1, 0, 0, 0, 0],
+                         [0, 0, 1, 1, 0, 0],
+                         [0, 0, 0, 0, 1, 1],
+                         [1, 0, 1, 0, 0, 0],
+                         [0, 0, 0, 1, 1, 0],
+                         [0, 1, 0, 0, 0, 1]])
+        b_eq = np.array([221, 210, 10, 141, 198, 102])
+        c = np.concatenate((0, 1, np.zeros(4)), axis=None)
+        with suppress_warnings() as sup:
+            sup.filter(OptimizeWarning, "A_eq does not appear...")
+            res = linprog(c, A_ub, b_ub, A_eq, b_eq, bounds,
+                          method=self.method, options=self.options)
+        _assert_success(res, desired_x=[129, 92, 12, 198, 0, 10], desired_fun=92)
+
 #########################
 # Method-specific Tests #
 #########################