Repository: scipy/scipy

Issue:
optimize.minimize 'trust' methods and finite difference Hessian strategies potentially breaking due to NumPy broadcasting. 
I'm writing this issue specifically pertaining to the "trust-krylov" method in scipy.optimize.minimize in conjunction with built-in finite difference {'2-point', '3-point', 'cs'} Hessian methods. However, similar but different issues seem to also arise, reproduced
by the same code below for trust-exact and trust-ncg. 

#### Reproducing code example:
```
import sys
from scipy.optimize import minimize
import numpy as np


def main():
    def _minfunc(x):
        return (x**2 - 1).sum()

    def _minjac(x):
        # ravel included just for my sanity
        return (2 * x).ravel()

    x0 = 4*np.ones(32)
    x_final = minimize(_minfunc, x0, method='trust-krylov', jac=_minjac, hess='cs')
    return x_final


if __name__ == '__main__':
    sys.exit(main())
```

#### Error message:

```
Traceback (most recent call last):
  File "scipy-trust-broadcasting-error.py", line 21, in <module>
    sys.exit(main())
  File "scipy-trust-broadcasting-error.py", line 16, in main
    x_final = minimize(_minfunc, x0, method='trust-krylov', jac=_minjac, hess='cs')
  File "C:\Users\Matt\Desktop\env\lib\site-packages\scipy\optimize\_minimize.py", line 641, in minimize
    callback=callback, **options)
  File "C:\Users\Matt\Desktop\env\lib\site-packages\scipy\optimize\_trustregion_krylov.py", line 57, in _minimize_trust_krylov
    **trust_region_options)
  File "C:\Users\Matt\Desktop\env\lib\site-packages\scipy\optimize\_trustregion.py", line 189, in _minimize_trust_region
    p, hits_boundary = m.solve(trust_radius)
  File "_trlib.pyx", line 113, in scipy.optimize._trlib._trlib.TRLIBQuadraticSubproblem.solve
TypeError: float() argument must be a string or a number, not '_ScaledLinearOperator'
```

#### Scipy/Numpy/Python version information:
```
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.6.2 1.20.1 sys.version_info(major=3, minor=7, micro=6, releaselevel='final', serial=0)
```

#### Environment information

I am running on
```
Microsoft Windows 10 Education
Version 10.0.19042 Build 19402
```

For testing purposes, I created a virtual environment using the following commands in command prompt. 

```
$ python -m venv env
$ cd env\Scripts
$ activate
$(env) python -m pip --upgrade pip
$(env) python -m pip install scipy
$(env) python -m pip list

Package    Version
---------- -------
numpy      1.20.1
pip        21.0.1
scipy      1.6.2
setuptools 41.2.0
```

#### Relevant Debugging Information

https://github.com/scipy/scipy/blob/d03be7c789097a994507e6ce8b6be14d450eeac8/scipy/optimize/_trustregion.py#L63

In the BaseQuadraticSubproblem method 

```
def hessp(self, p):
    if self._hessp is not None:
        return self._hessp(self._x, p)
    else:
        return np.dot(self.hess, p)
```

In the minimal code example I provided, the issue is occurring during the evaluation of ```np.dot(self.hess, p)```. More
specifically, I believe that the issue is due to NumPy broadcasting, as what is being returned is an array whose elements are of _ScaledLinearOperator type.

#### Proposed/Possible Fix

I do not know enough about SciPy/NumPy internals but the following is at least an incredibly simple change that could be
considered.  I believe that a method for fixing this would be to evaluate the dot product by not appealing to numpy, but rather, appealing to the ```dot``` method instead, which I presume (perhaps incorrectly) must exist for all ndarray / LinearOperator types
that would possibly arise here. 

```
def hessp(self, p):
    if self._hessp is not None:
        return self._hessp(self._x, p)
    else:
        return self.hess.dot(p)
```

I have attached an image of using Pycharm's evaluation tool during debugging, showing the results of the two different evaluations; namely, the current code vs. code that could possibly resolve the issue. 

![scipy_issue_debug_evaluation](https://user-images.githubusercontent.com/41150404/112709214-b4377b80-8e8d-11eb-91ca-02fdb7029a3d.png)

#### Contact Info

For questions feel free to contact me at matthew.gudorf@gmail.com. Thanks!


Discussion:
Additional comment: 

Error was generated via 
```
$(env) python scipy-trust-broadcasting-error.py
```
where ```scipy-trust-broadcasting-error.py``` was the name of the script containing the minimal code example.  
Thanks for reporting. Well actually I think this is not implemented at all. In the other methods like `newton-cg`, there is a call to `_prepare_scalar_function` which allows the options `cs`, etc. So for now you can only pass a function to the hessian. The doc should be updated or this implemented.

@andyfaff what do you think should be done here?
As far as I can tell from the git history of `_trustregion.py` (which serves *trust-krylov, trust-ncg, trust-exact*), those trust region methods (`trust-constr` is different) have always required a callable `hess` in order to work properly.
It does say in the documentation

> Finite-difference options {‘2-point’, ‘3-point’, ‘cs’} and HessianUpdateStrategy are available only for ‘trust-constr’ method.

, which means that they're not allowed for *trust-krylov, trust-ncg, trust-exact*. This indirectly means that you need a callable `hess`. So the documentation is already there, @mgudorf how could the documentation be made clearer?

One change to the code could be to make a check that will raise an appropriate error is hess isn't callable.

If `2-point`, `3-point`, `cs`, or `HessianUpdateStrategy` are provided for hess then `ScalarFunction.H` is either a `LinearOperator` or `HessianUpdateStrategy`. It *may* be possible to use a Hessian estimated from that to give to the trustregion methods, but that's not my area of expertise.

@tupui @andyfaff 

My apologies, I missed that sentence; however, more testing on my end still yielded unexpected results; notably providing the Hessian as a `LinearOperator` and two dimensional `np.ndarray` have disparate results for `trust-constr` and `trust-krylov`; the
latter breaking down in the same place as the original issue. 

### Using a `LinearOperator` with `trust-krylov`

```
import sys
from scipy.optimize import minimize
import numpy as np
from scipy.sparse.linalg import LinearOperator

def main():
    def _minfunc(x):
        # The scalar function to minimize
        return (x**2 - 1).sum()

    def _minjac(x):
        # The scalar function's gradient
        return (2 * x).ravel()

    def _minhess(x):
        # The Hessian provided as a LinearOperator (exact hessian is 2 * identity)
        mv = lambda y : 2 * y
        return LinearOperator(shape=(x.size, x.size), matvec=mv, rmatvec=mv)

    x0 = 4*np.ones(32)
    x_final = minimize(_minfunc, x0, jac=_minjac, hess=_minhess, method='trust-krylov')
    return x_final

if __name__ == '__main__':
    sys.exit(main())
```

#### Error message:

```
Traceback (most recent call last):
  File "scipy-trust-broadcasting-error.py", line 29, in <module>
    sys.exit(main())
  File "scipy-trust-broadcasting-error.py", line 24, in main
    x_final = minimize(_minfunc, x0, jac=_minjac, hess=_minhess, method='trust-krylov')
  File "C:\Users\Matt\Desktop\scipy-test\lib\site-packages\scipy\optimize\_minimize.py", line 641, in minimize
    callback=callback, **options)
  File "C:\Users\Matt\Desktop\scipy-test\lib\site-packages\scipy\optimize\_trustregion_krylov.py", line 57, in _minimize_trust_krylov
    **trust_region_options)
  File "C:\Users\Matt\Desktop\scipy-test\lib\site-packages\scipy\optimize\_trustregion.py", line 189, in _minimize_trust_region
    p, hits_boundary = m.solve(trust_radius)
  File "_trlib.pyx", line 113, in scipy.optimize._trlib._trlib.TRLIBQuadraticSubproblem.solve
TypeError: float() argument must be a string or a number, not '_ScaledLinearOperator'
```

### Using a `LinearOperator` with `trust-constr`

```
import sys
from scipy.optimize import minimize
import numpy as np
from scipy.sparse.linalg import LinearOperator

def main():
    def _minfunc(x):
        # The scalar function to minimize
        return (x**2 - 1).sum()

    def _minjac(x):
        # The scalar function's gradient
        return (2 * x).ravel()

    def _minhess(x):
        # The Hessian provided as a LinearOperator (exact hessian is 2 * identity)
        mv = lambda y : 2 * y
        return LinearOperator(shape=(x.size, x.size), matvec=mv, rmatvec=mv)

    x0 = 4*np.ones(32)
    x_final = minimize(_minfunc, x0, jac=_minjac, hess=_minhess, method='trust-constr')
    return x_final

if __name__ == '__main__':
    sys.exit(main())
```

#### Output

```
         cg_niter: 3
     cg_stop_cond: 4
           constr: []
      constr_nfev: []
      constr_nhev: []
      constr_njev: []
   constr_penalty: 1.0
 constr_violation: 0
   execution_time: 0.008998870849609375
              fun: -32.0
             grad: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
              jac: []
  lagrangian_grad: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
          message: '`gtol` termination condition is satisfied.'
           method: 'equality_constrained_sqp'
             nfev: 4
             nhev: 4
              nit: 4
            niter: 4
             njev: 4
       optimality: 0.0
           status: 1
          success: True
        tr_radius: 102.39191898578666
                v: []
                x: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
```

Meanwhile, providing the Hessian as an array works fine for both. 

### Using a `ndarray` Hessian with `trust-krylov`

```
import sys
from scipy.optimize import minimize
import numpy as np
from scipy.sparse.linalg import LinearOperator

def main():
    def _minfunc(x):
        # The scalar function to minimize
        return (x**2 - 1).sum()

    def _minjac(x):
        # The scalar function's gradient
        return (2 * x).ravel()

    def _minhess(x):
        return 2*np.eye(x.size)

    x0 = 4*np.ones(32)
    x_final = minimize(_minfunc, x0, jac=_minjac, hess=_minhess, method='trust-krylov')
    return x_final


if __name__ == '__main__':
    sys.exit(main())
```

#### Output
``` 
     fun: -32.0
    hess: array([[2., 0., 0., ..., 0., 0., 0.],
       [0., 2., 0., ..., 0., 0., 0.],
       [0., 0., 2., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 2., 0., 0.],
       [0., 0., 0., ..., 0., 2., 0.],
       [0., 0., 0., ..., 0., 0., 2.]])
     jac: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
 message: 'Optimization terminated successfully.'
    nfev: 6
    nhev: 5
     nit: 5
    njev: 6
  status: 0
 success: True
       x: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
```
## `trust-constr` output using Hessian as 2-d array. 
```
import sys
from scipy.optimize import minimize
import numpy as np
from scipy.sparse.linalg import LinearOperator

def main():
    def _minfunc(x):
        # The scalar function to minimize
        return (x**2 - 1).sum()

    def _minjac(x):
        # The scalar function's gradient
        return (2 * x).ravel()

    def _minhess(x):
        return 2*np.eye(x.size)

    x0 = 4*np.ones(32)
    x_final = minimize(_minfunc, x0, jac=_minjac, hess=_minhess, method='trust-constr')
    return x_final


if __name__ == '__main__':
    sys.exit(main())

```
### Output
``` 
       cg_niter: 3
   cg_stop_cond: 4
         constr: []
    constr_nfev: []
    constr_nhev: []
    constr_njev: []
 constr_penalty: 1.0
constr_violation: 0
 execution_time: 0.009998798370361328
            fun: -32.0
           grad: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
     0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
            jac: []
lagrangian_grad: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
     0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
        message: '`gtol` termination condition is satisfied.'
         method: 'equality_constrained_sqp'
           nfev: 4
           nhev: 4
            nit: 4
          niter: 4
           njev: 4
     optimality: 0.0
         status: 1
        success: True
      tr_radius: 102.39191898578666
              v: []
              x: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
     0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
```

If I understood @andyfaff correctly then I think the issue stands, albeit for a different reason now, and I believe would be fixed by the same correction I originally proposed. 
In the examples where you're providing an "`np.ndarray`", what you're actually providing to `minimize` is a callable `hess`, `callable(hess) is True`. A callable hess has always been accepted for the trust region minimizers, and was mandatory for all of them except trust-constr.

As you suggest, it should be possible to amend `_trustregion.py`, around about [here](https://github.com/scipy/scipy/blob/master/scipy/optimize/_trustregion.py#L163) such that if a callable `hessp` is not provided, then the function you suggest above can be used to create a `hessp` from the LinearOperator/HessianUpdateOperator that's returned from `ScalarFunction.hess`. Whilst this would be suitable for trust-krylov and trust-ncg the documentation suggests it wouldn't work for trust-exact.



Resolving patch:
diff --git a/scipy/optimize/_minimize.py b/scipy/optimize/_minimize.py
--- a/scipy/optimize/_minimize.py
+++ b/scipy/optimize/_minimize.py
@@ -108,16 +108,16 @@ def minimize(fun, x0, args=(), method=None, jac=None, hess=None,
         obey any specified `bounds`.
     hess : {callable, '2-point', '3-point', 'cs', HessianUpdateStrategy}, optional
         Method for computing the Hessian matrix. Only for Newton-CG, dogleg,
-        trust-ncg,  trust-krylov, trust-exact and trust-constr. If it is
-        callable, it should return the  Hessian matrix:
+        trust-ncg, trust-krylov, trust-exact and trust-constr. If it is
+        callable, it should return the Hessian matrix:
 
             ``hess(x, *args) -> {LinearOperator, spmatrix, array}, (n, n)``
 
         where x is a (n,) ndarray and `args` is a tuple with the fixed
-        parameters. LinearOperator and sparse matrix returns are
-        allowed only for 'trust-constr' method. Alternatively, the keywords
+        parameters. LinearOperator and sparse matrix returns are only allowed
+        for 'trust-constr' method. Alternatively, the keywords
         {'2-point', '3-point', 'cs'} select a finite difference scheme
-        for numerical estimation. Or, objects implementing
+        for numerical estimation. Or, objects implementing the
         `HessianUpdateStrategy` interface can be used to approximate
         the Hessian. Available quasi-Newton methods implementing
         this interface are:
@@ -129,8 +129,8 @@ def minimize(fun, x0, args=(), method=None, jac=None, hess=None,
         the Hessian cannot be estimated with options
         {'2-point', '3-point', 'cs'} and needs to be
         estimated using one of the quasi-Newton strategies.
-        Finite-difference options {'2-point', '3-point', 'cs'} and
-        `HessianUpdateStrategy` are available only for 'trust-constr' method.
+        'trust-exact' cannot use a finite-difference scheme, and must be used
+        with a callable returning an (n, n) array.
     hessp : callable, optional
         Hessian of objective function times an arbitrary vector p. Only for
         Newton-CG, trust-ncg, trust-krylov, trust-constr.
diff --git a/scipy/optimize/_trustregion.py b/scipy/optimize/_trustregion.py
--- a/scipy/optimize/_trustregion.py
+++ b/scipy/optimize/_trustregion.py
@@ -5,7 +5,8 @@
 import scipy.linalg
 from .optimize import (_check_unknown_options, _wrap_function, _status_message,
                        OptimizeResult, _prepare_scalar_function)
-
+from scipy.optimize._hessian_update_strategy import HessianUpdateStrategy
+from scipy.optimize._differentiable_functions import FD_METHODS
 __all__ = []
 
 
@@ -157,8 +158,25 @@ def _minimize_trust_region(fun, x0, args=(), jac=None, hess=None, hessp=None,
     sf = _prepare_scalar_function(fun, x0, jac=jac, hess=hess, args=args)
     fun = sf.fun
     jac = sf.grad
-    if hess is not None:
+    if callable(hess):
         hess = sf.hess
+    elif callable(hessp):
+        # this elif statement must come before examining whether hess
+        # is estimated by FD methods or a HessianUpdateStrategy
+        pass
+    elif (hess in FD_METHODS or isinstance(hess, HessianUpdateStrategy)):
+        # If the Hessian is being estimated by finite differences or a
+        # Hessian update strategy then ScalarFunction.hess returns a
+        # LinearOperator or a HessianUpdateStrategy. This enables the
+        # calculation/creation of a hessp. BUT you only want to do this
+        # if the user *hasn't* provided a callable(hessp) function.
+        hess = None
+        def hessp(x, p, *args):
+            return sf.hess(x).dot(p)
+    else:
+        raise ValueError('Either the Hessian or the Hessian-vector product '
+                         'is currently required for trust-region methods')
+
     # ScalarFunction doesn't represent hessp
     nhessp, hessp = _wrap_function(hessp, args)
 


Test patch:
diff --git a/scipy/optimize/tests/test_trustregion.py b/scipy/optimize/tests/test_trustregion.py
--- a/scipy/optimize/tests/test_trustregion.py
+++ b/scipy/optimize/tests/test_trustregion.py
@@ -5,10 +5,14 @@
   nosetests test_optimize.py
 
 """
+import itertools
+from copy import deepcopy
 import numpy as np
-from scipy.optimize import (minimize, rosen, rosen_der, rosen_hess,
-                            rosen_hess_prod)
 from numpy.testing import assert_, assert_equal, assert_allclose
+from scipy.optimize import (minimize, rosen, rosen_der, rosen_hess,
+                            rosen_hess_prod, BFGS)
+from scipy.optimize._differentiable_functions import FD_METHODS
+import pytest
 
 
 class Accumulator:
@@ -102,3 +106,31 @@ def test_trust_exact_start_in_optimum(self):
         r = minimize(rosen, x0=self.x_opt, jac=rosen_der, hess=rosen_hess,
                      tol=1e-8, method='trust-exact')
         assert_allclose(self.x_opt, r['x'])
+
+    def test_finite_differences(self):
+        # if the Hessian is estimated by finite differences or
+        # a HessianUpdateStrategy (and no hessp is provided) then creation
+        # of a hessp is possible.
+        # GH13754
+        methods = ["trust-ncg", "trust-krylov", "dogleg"]
+        product = itertools.product(
+            FD_METHODS + (BFGS,),
+            methods
+        )
+        # a hessian needs to be specified for trustregion
+        for method in methods:
+            with pytest.raises(ValueError):
+                minimize(rosen, x0=self.x_opt, jac=rosen_der, method=method)
+
+        # estimate hessian by finite differences. In _trustregion.py this
+        # creates a hessp from the LinearOperator/HessianUpdateStrategy
+        # that's returned from ScalarFunction.
+        for fd, method in product:
+            hess = fd
+            if fd == BFGS:
+                # don't want to use the same object over and over
+                hess = deepcopy(fd())
+
+            r = minimize(rosen, x0=self.x_opt, jac=rosen_der, hess=hess,
+                         tol=1e-8, method=method)
+            assert_allclose(self.x_opt, r['x'])