Repository: scipy/scipy

Issue:
Incorrect Hessian returned by optimize.minimize using BFGS algorithm
One of the keys of the dictionary returned by optimize.minimize using the BFGS algorithm is called 'hess' and the corresponding value is an array of the right size to be the Hessian.  However the array that is returned is neither the correct Hessian nor inverse Hessian at the minima.  Is this a bug in the Hessian calculation or is the 'hess' item confusingly not in fact the Hessian?  

```
import scipy.optimize as op
import numpy as np
import numdifftools as nd

def sum_squares(x): 
    ''' 
    Returns the sum of the squared values of the components of x.  
    The Hessian of this function should have diagonal elements of
    2 and non-diagonal elements of 0.  
    '''
    return np.sum(x ** 2)

fit = op.minimize(sum_squares, np.array([2., 3.])  ,method = 'BFGS')

fit['hess']    # Matrix that is neither the correct Hessian or inverse Hessian.

Hess = nd.Hessian(sum_squares)(np.array([0.,0.])) # The correct Hessian.
```



Discussion:
Thanks for the report. The returned value seems to be the quasi-newton approximation to the inverse Hessian.
The name `hess` is also misleading as it's the inverse, and moreover, an approximation (the Q-N approximations also often are not very good). I think it should be renamed.

@dlax: ping

Pauli Virtanen a écrit :

> Thanks for the report. The returned value seems to be the quasi-newton
> approximation to the inverse Hessian.
> The name |hess| is also misleading as it's the inverse, I think it
> should be renamed.

Agreed. I'd suggest one of hess-inv or hess_inv.


Resolving patch:
diff --git a/scipy/optimize/_minimize.py b/scipy/optimize/_minimize.py
--- a/scipy/optimize/_minimize.py
+++ b/scipy/optimize/_minimize.py
@@ -159,7 +159,8 @@ def minimize(fun, x0, args=(), method='BFGS', jac=None, hess=None,
     Method *BFGS* uses the quasi-Newton method of Broyden, Fletcher,
     Goldfarb, and Shanno (BFGS) [5]_ pp. 136. It uses the first derivatives
     only. BFGS has proven good performance even for non-smooth
-    optimizations
+    optimizations. This method also returns an approximation of the Hessian
+    inverse, stored as `hess_inv` in the Result object.
 
     Method *Newton-CG* uses a Newton-CG algorithm [5]_ pp. 168 (also known
     as the truncated Newton method). It uses a CG method to the compute the
diff --git a/scipy/optimize/optimize.py b/scipy/optimize/optimize.py
--- a/scipy/optimize/optimize.py
+++ b/scipy/optimize/optimize.py
@@ -82,8 +82,9 @@ class Result(dict):
         underlying solver. Refer to `message` for details.
     message : str
         Description of the cause of the termination.
-    fun, jac, hess : ndarray
-        Values of objective function, Jacobian and Hessian (if available).
+    fun, jac, hess, hess_inv : ndarray
+        Values of objective function, Jacobian, Hessian or its inverse (if
+        available).
     nfev, njev, nhev : int
         Number of evaluations of the objective functions and of its
         Jacobian and Hessian.
@@ -775,7 +776,7 @@ def fmin_bfgs(f, x0, fprime=None, args=(), gtol=1e-5, norm=Inf,
     res = _minimize_bfgs(f, x0, args, fprime, callback=callback, **opts)
 
     if full_output:
-        retlist = (res['x'], res['fun'], res['jac'], res['hess'],
+        retlist = (res['x'], res['fun'], res['jac'], res['hess_inv'],
                    res['nfev'], res['njev'], res['status'])
         if retall:
             retlist += (res['allvecs'], )
@@ -917,7 +918,7 @@ def _minimize_bfgs(fun, x0, args=(), jac=None, callback=None,
             print("         Function evaluations: %d" % func_calls[0])
             print("         Gradient evaluations: %d" % grad_calls[0])
 
-    result = Result(fun=fval, jac=gfk, hess=Hk, nfev=func_calls[0],
+    result = Result(fun=fval, jac=gfk, hess_inv=Hk, nfev=func_calls[0],
                     njev=grad_calls[0], status=warnflag,
                     success=(warnflag == 0), message=msg, x=xk)
     if retall:


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -102,7 +102,7 @@ def test_bfgs(self, use_wrapper=False):
                                     options=opts)
 
             params, fopt, gopt, Hopt, func_calls, grad_calls, warnflag = \
-                    res['x'], res['fun'], res['jac'], res['hess'], \
+                    res['x'], res['fun'], res['jac'], res['hess_inv'], \
                     res['nfev'], res['njev'], res['status']
         else:
             retval = optimize.fmin_bfgs(self.func, self.startparams, self.grad,