Repository: lmfit/lmfit-py

Issue:
Passing jac_sparsity to least_squares breaks minimize()
#### Description

Calls of the form:

```python
lmfit.minimize(..., method="least_squares", jac_sparsity=obj)
```

result in the Jacobian being returned as a sparse matrix, which breaks the calculation of the Hessian:

https://github.com/lmfit/lmfit-py/blob/d7b408baa64821f4356cff9c32a199bc53becd93/lmfit/minimizer.py#L1430

This should be replaced with something like:

```python
if scipy.sparse.issparse(ret.jac):
    jac = ret.jac.todense()
else:
    jac = ret.jac
np.matmul(jac.T, jac)
```

or, perhaps faster but python 3 only:

```python
hess = ret.jac.T @ ret.jac
if scipy.sparse.issparse(hess):
    hess = hess.todense()
```

#### A Minimal, Complete, and Verifiable example

```python
import lmfit

def f(params):
    return params['x'] - 1, params['y'] - 1

params = lmfit.Parameters()
params.add('x', value=0)
params.add('y', value=0)
out = lmfit.minimize(f, params, method='least_squares', jac_sparsity=[[1, 1], [1, 1]])
```

#### Error message:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Admin/.pyenv/versions/3.7.3/Python.framework/Versions/3.7/lib/python3.7/site-packages/lmfit/minimizer.py", line 2448
, in minimize
    return fitter.minimize(method=method)
  File "/Users/Admin/.pyenv/versions/3.7.3/Python.framework/Versions/3.7/lib/python3.7/site-packages/lmfit/minimizer.py", line 2117
, in minimize
    return function(**kwargs)
  File "/Users/Admin/.pyenv/versions/3.7.3/Python.framework/Versions/3.7/lib/python3.7/site-packages/lmfit/minimizer.py", line 1430
, in least_squares
    hess = np.matmul(ret.jac.T, ret.jac)
ValueError: matmul: Input operand 0 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)->(n?,m?) requi
res 1)
```

#### Version information

```
Python: 3.7.3 (default, May 26 2019, 09:41:28)
[Clang 10.0.1 (clang-1001.0.46.4)]
lmfit: 0.9.14, scipy: 1.3.0, numpy: 1.16.3, asteval: 0.9.14, uncertainties: 3.1, six: 1.12.0
```


Discussion:
@ezwelty  You raised a Github issue instead of asking a question on the mailing list.  You cannot have missed the instructions that clearly advise not doing this.  

Anyway, you raise yet another incompatible set of options for `least_squares`.  This does not surprise me or really even bother me that much.   The API and docs for `least_squares` are complete messes with numerous interdependent options and changes in the meaning of one option based on the value of another option.  We support what we can.  Supporting every matrixed way of using `least_squares` is not a high priority.  And sparse problems don't seem to be a common occurrence for lmfit.

Using `tr_solver='lsmr'` because the jacobian is sparse should be a separate solving method, and certainly not conflated with Levenberg-Marquardt.   I would be more inclined to drop generic support for `least_squares` in general and replace it with explicit support for `trf`, meaning precisely `trf exact`.  I would drop support for `loss` in the bargain, as it is completely unneeded and turns `least_squares` into "not least squares".

I am skeptical that many people actually know when their jacobian is actually sparse.  Perhaps you do know this.  

You can make a PR for this if you want.   It might be that your suggestion is the right approach.  Or it might be that the exceptions for that `try: np.matul()` block should allow more than `LinAlgError` to say that if you have sparse jacobians, you won't be able to get uncertainties...  I'm not sure.  

One might think that a discussion about this has would have been a  good idea.  If only we had a forum for that.   Next time, do not ignore the instructions for contributing.

@newville I read the contributor instructions and, considering this a bug, submitted a GitHub issue.

> If you find a bug with the code or documentation, please use GitHub Issues to submit a bug report.

The documentation for [scipy.optimize.least_squares](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html) clearly indicates that the Jacobian is sometimes returned as a sparse matrix or a scipy.sparse.linalg.LinearOperator:

> ndarray, sparse matrix or LinearOperator, shape (m, n)

So a complete handling of this spec could look like this:

```python
if scipy.sparse.issparse(ret.jac):
    jac = ret.jac.todense()
elif isinstance(ret.jac, scipy.sparse.linalg.LinearOperator):
   jac = ret.jac * np.eye(ret.jac.shape[1], dtype=ret.jac.dtype)
else:
    jac = ret.jac
hess = np.matmul(jac.T, jac)
```

This does not change the math in any way, only deals correctly with the expected range of return values. Passing a sparsity matrix (`jac_sparsity`) to the Jacobian calculation in `least_squares()` often yields the same Jacobian (as in the example below), but at a fraction of the computational cost. 

```python
import numpy as np
import scipy

# Initialize model
x = np.random.normal(0, 1, 50)
y = np.random.normal(0, 1, 50)
def f(params):
    return np.concatenate((
        params[0] - x,
        params[1] - y,
        params[0] - x * params[1] - y))

# Solve with sparsity matrix
sparsity = np.row_stack((
    np.tile([[1, 0]], (50, 1)),
    np.tile([[0, 1]], (50, 1)),
    np.tile([[1, 1]], (50, 1))))
ret = scipy.optimize.least_squares(fun=f, x0=(1, 1), jac_sparsity=sparsity)
hess = np.matmul(ret.jac.T.todense(), ret.jac.todense())
np.linalg.inv(hess)
```

> array([[ 0.01013523, -0.00119951], [-0.00119951,  0.01063987]])

```python
# Solve without sparsity matrix
ret = scipy.optimize.least_squares(fun=f, x0=(1, 1))
hess = np.matmul(ret.jac.T, ret.jac)
np.linalg.inv(hess)
```

> array([[ 0.01013523, -0.00119951], [-0.00119951,  0.01063987]]) 
@ezwelty 
> @newville I read the contributor instructions and, considering this a bug, submitted a GitHub issue.
>
>>  If you find a bug with the code or documentation, please use GitHub Issues to submit a bug report.

Hm, yeah...  we also say:

> If you are at all unsure whether to use the mailing list or open an Issue, please start a conversation on the mailing list.

Perhaps we are being too polite. 

> The documentation for scipy.optimize.least_squares clearly indicates that the Jacobian is sometimes returned as a sparse matrix or a scipy.sparse.linalg.LinearOperator:

The doc for `least_squares` is a study in complex, interdependent options. To predict the behavior of `least_square`, one must know the value of **all** of the arguments and the complex ways in how they interact.

The type of `OptimizeResult.jac` is documented as

> ndarray, sparse matrix or LinearOperator, shape (m, n)
> Modified Jacobian matrix at the solution, in the sense that J^T J is a Gauss-Newton approximation of the Hessian of the cost function. The type is the same as the one used by the algorithm.

That is, one must know "the type used by the algorithm".  OK, how do we know that, inside lmfit? What is the **code** to determine whether `jac` will be sparse?   I think we have to know the actual solver used.  But that is not returned.  We could examine the function arguments and guess its behavior.  Yuck.   Does `tr_solver`, and maybe `tr_options` impact this?   What happens if one specifies a method of `lm` or `dogbox`?  I do not know. 

As you suggest we could examine the type of `jac` and act accordingly.   Note that the doc is then is not helping at all, we're just guessing types and responding.  That might be the right approach: treat `least_squares()` as a black box and inspect what comes out downstream.   

> Passing a sparsity matrix (jac_sparsity) to the Jacobian calculation in least_squares() often yields the same Jacobian (as in the example below), but at a fraction of the computational cost.

Hm, maybe...  I think I've never had a problem with a sparse jacobian.

Anyway, you can submit a PR.  Since I really have no experience with sparse jacobians,  I'm not inclined to do the work or support this feature without help from others. 





Resolving patch:
diff --git a/lmfit/minimizer.py b/lmfit/minimizer.py
--- a/lmfit/minimizer.py
+++ b/lmfit/minimizer.py
@@ -29,6 +29,8 @@
 from scipy.optimize import leastsq as scipy_leastsq
 from scipy.optimize import minimize as scipy_minimize
 from scipy.optimize import shgo as scipy_shgo
+from scipy.sparse import issparse
+from scipy.sparse.linalg import LinearOperator
 from scipy.stats import cauchy as cauchy_dist
 from scipy.stats import norm as norm_dist
 from scipy.version import version as scipy_version
@@ -1408,7 +1410,21 @@ def least_squares(self, params=None, **kws):
 
             # calculate the cov_x and estimate uncertainties/correlations
             try:
-                hess = np.matmul(ret.jac.T, ret.jac)
+                if issparse(ret.jac):
+                    hess = (ret.jac.T * ret.jac).toarray()
+                elif isinstance(ret.jac, LinearOperator):
+                    identity = np.eye(ret.jac.shape[1], dtype=ret.jac.dtype)
+                    # TODO: Remove try-except when scipy < 1.4.0 support dropped
+                    try:
+                        # For scipy >= 1.4.0 (with Linear Operator transpose)
+                        # https://github.com/scipy/scipy/pull/9064
+                        hess = (ret.jac.T * ret.jac) * identity
+                    except AttributeError:
+                        # For scipy < 1.4.0 (without Linear Operator transpose)
+                        jac = ret.jac * identity
+                        hess = np.matmul(jac.T, jac)
+                else:
+                    hess = np.matmul(ret.jac.T, ret.jac)
                 result.covar = np.linalg.inv(hess)
                 self._calculate_uncertainties_correlations()
             except LinAlgError:


Test patch:
diff --git a/tests/test_least_squares.py b/tests/test_least_squares.py
--- a/tests/test_least_squares.py
+++ b/tests/test_least_squares.py
@@ -2,6 +2,8 @@
 import numpy as np
 from numpy.testing import assert_allclose
 import pytest
+from scipy.sparse import bsr_matrix
+from scipy.sparse.linalg import aslinearoperator
 
 import lmfit
 from lmfit.models import VoigtModel
@@ -116,3 +118,52 @@ def test_least_squares_solver_options(peakdata, capsys):
 
     assert 'Iteration' in captured.out
     assert 'final cost' in captured.out
+
+
+def test_least_squares_jacobian_types():
+    """Test support for Jacobian of all types supported by least_squares."""
+    # Build function
+    # f(x, y) = (x - a)^2 + (y - b)^2
+    np.random.seed(42)
+    a = np.random.normal(0, 1, 50)
+    np.random.seed(43)
+    b = np.random.normal(0, 1, 50)
+
+    def f(params):
+        return (params['x'] - a)**2 + (params['y'] - b)**2
+
+    # Build analytic Jacobian functions with the different possible return types
+    # numpy.ndarray, scipy.sparse.spmatrix, scipy.sparse.linalg.LinearOperator
+    # J = [ 2x - 2a , 2y - 2b ]
+    def jac_array(params, *args, **kwargs):
+        return np.column_stack((2 * params[0] - 2 * a, 2 * params[1] - 2 * b))
+
+    def jac_sparse(params, *args, **kwargs):
+        return bsr_matrix(jac_array(params, *args, **kwargs))
+
+    def jac_operator(params, *args, **kwargs):
+        return aslinearoperator(jac_array(params, *args, **kwargs))
+    # Build parameters
+    params = lmfit.Parameters()
+    params.add('x', value=0)
+    params.add('y', value=0)
+    # Solve model for numerical Jacobian and each analytic Jacobian function
+    result = lmfit.minimize(f, params, method='least_squares')
+    result_array = lmfit.minimize(
+        f, params, method='least_squares',
+        jac=jac_array)
+    result_sparse = lmfit.minimize(
+        f, params, method='least_squares',
+        jac=jac_sparse)
+    result_operator = lmfit.minimize(
+        f, params, method='least_squares',
+        jac=jac_operator)
+    # Check that all have uncertainties
+    assert result.errorbars
+    assert result_array.errorbars
+    assert result_sparse.errorbars
+    assert result_operator.errorbars
+    # Check that all have ~equal covariance matrix
+    assert_allclose(result.covar, result_array.covar)
+    assert_allclose(result.covar, result_sparse.covar)
+    assert_allclose(result.covar, result_operator.covar)