Repository: scipy/scipy

Issue:
optimize.approx_fprime & jacobians
Hello,
While trying to run scipy.optimize.check_grad  function in the following code
# !/usr/bin/python

'''Fitting the function y = c + r*sin(phi + alpha)'''
from math import sin, cos, pi
import scipy.optimize as op
import numpy as np

def f(als, c=1., r=1., phi=15*pi/180.):
    return np.array([y(a, c, r, phi) for a in als])

def y(a,  c=1., r=1., phi=15*pi/180.):
    return c + r \* sin(phi + a)

def y_grad(als, c=1., r=1., phi=15*pi/180):
    return np.array([y_prime(a, c, r, phi) for a in als])

def y_prime(a, c=1., r=1., phi=15*pi/180):
    return -r \* cos(phi + a)

def diffsquares(y_exp, y_fit):
    return (y_exp - y_fit)**2

step = 2_pi / 360.
a = np.arange(0., 2_pi + step, step)

result = op.check_grad(f, y_grad, a)

print 'result', result

on Ubuntu 12.04 (scipy version '0.9.0') I get the following error:

Traceback (most recent call last):
  File "./fit.py", line 28, in <module>
    result = op.check_grad(f, y_grad, a)
  File "/usr/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 382, in check_grad
    return sqrt(sum((grad(x0,_args)-approx_fprime(x0,func,_epsilon,_args))*_2))
  File "/usr/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 377, in approx_fprime
    grad[k] = (f(_((xk+ei,)+args)) - f0)/epsilon
ValueError: setting an array element with a sequence.

looking in the source code I think there is an obvious typo there, the numerator is not indexed, I have checked the code on gitHub and the bug seems to be still present (line 610 https://github.com/scipy/scipy/blob/master/scipy/optimize/optimize.py) .

line 610
grad[k] = (f(*((xk + d,) + args)) - f0) / d[k]

should probably read
grad[k] = (f(*((xk + d,) + args))[k] - f0[k]) / d[k]

Cheers,
Martin

optimize.check_grad does not work for complex-valued functions
The function ``optimize.check_grad`` does not work for complex-valued functions.

Here is an adapted example from the [documentation of check_grad](http://scipy.github.io/devdocs/generated/scipy.optimize.check_grad.html):

    def func(x):
        return x[0]**2 - 0.5j * x[1]**3
    
    def grad(x):
        return [2 * x[0], -1.5j * x[1]**2]
    
    from scipy.optimize import check_grad
    print(check_grad(func, grad, [1.5, -1.5]))

Expected: a (real) float close to zero.
Got: a ComplexWarning is raised and the returned value is non-zero.

This should be solved by replacing the last line:

    return sqrt(sum((grad(x0, *args) - approx_fprime(x0, func, step, *args))**2))

by:

    return sqrt(sum(numpy.abs((grad(x0, *args) - approx_fprime(x0, func, step, *args))**2))) 

(take the absolute value)



Edit : A long term solution would be using optimize._numdiff.check_derivative instead. However the function optimize._numdiff._dense_difference has to be adapted to complex valued function too.


Discussion:
f is supposed to return a scalar, see docstring

From what I remember when I looked at the function last time, everything worked as expected.

Yes, it works correctly. It evaluates only gradients, however, not Jacobians.

Can be seen as a documentation issue. From reading the docstring of `check_gradient` for the first time, I didn't figure out whether `f` is supposed to be a a scalar function of a vector argument (and what is being checked is indeed a gradient), or is the requirement that `x0` is the array just a way of vectorizing a computation of a scalar derivative of a scalar function at a series of points. 

The docstring of `approx_fprime` is way clearer. A quick fix could be to add an example to the docstring. Something as simple as

```
import scipy.optimize as op
import numpy as np

def fv(x):
    return x*x

def fv_prime(x):
    return 2.*x

def fs(x):
    return np.sum(x*x)

print op.check_grad(fs, fv_prime, np.array([3]))
```

On a related note, I wonder is there's a pressing reason for `check_grad` refusing to deal with scalar arguments, not arrays:

```
op.check_gradient(fs, fv_prime, 3.)
```

Ok, yep, the documentation is a bit confusing here.

@MartinSavko: in principle it would be OK to extend `approx_fprime` to evaluate also Jacobians of vector-valued functions, and not only gradients of scalar functions. If you decide to do that, please file a second pull request, thanks!

At the present time It would be better to expose `optimize._differentiable_functions.VectorFunction`, or `optimize._numdiff.approx_derivative` as a public interface.
The latter function can do either gradients (scalar functions) or Jacobians (vector valued functions).
I'm in the process of making `approx_fprime` rely on `approx_derivative` for calculation.
Based on the comments, looks like the doc fix is a good first issue.
So it looks like `approx_fprime` does rely on `approx_derivative` now.  Moreover, it only performs two main steps: enforcing that `x` is an array and enforcing that `f(x)` is scalar.  Then it calls `approx_derivative`, which _can_ handle vector-valued functions.

So the question is, I could (a) make `check_grad` call `approx_derivative` directly, or (b) remove the step in `approx_fprime` that restricts `f(x)` to be a scalar.  I would imagine that there's a reason it was made to be a scalar to support some other higher-level functions.  On the other hand, not sure if you want `check_grad` to directly call the non-public `optimize._numdiff.approx_derivative`.
It might be better to keep `approx_derivative` private at this point.

`approx_fprime` could probably do with an overhaul. For example, the function itself is no longer used in minimisation, so the last sentence in the notes section is out of date.

It should be fairly straightforward to allow `approx_fprime` to calculate Jacobians (R^m -> R^n) instead of solely being used to calculate gradients (R^m -> 1), would you like to add that functionality, it'll be a worthwhile PR.

`check_grad` could be changed to call `approx_derivative` directly as well, but it's not clear to me that `check_grad` is actually all that useful.
One thing to keep in mind if someone's willing to work on finite differencing: it might be better to separate scalar derivatives, gradients of vector functions and jacobians early on because of broadcasting. 


Resolving patch:
diff --git a/scipy/optimize/_optimize.py b/scipy/optimize/_optimize.py
--- a/scipy/optimize/_optimize.py
+++ b/scipy/optimize/_optimize.py
@@ -908,17 +908,30 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
 
 
 def approx_fprime(xk, f, epsilon=_epsilon, *args):
-    """Finite-difference approximation of the gradient of a scalar function.
+    """Finite difference approximation of the derivatives of a
+    scalar or vector-valued function.
+
+    If a function maps from :math:`R^n` to :math:`R^m`, its derivatives form
+    an m-by-n matrix
+    called the Jacobian, where an element :math:`(i, j)` is a partial
+    derivative of f[i] with respect to ``xk[j]``.
 
     Parameters
     ----------
     xk : array_like
         The coordinate vector at which to determine the gradient of `f`.
     f : callable
-        The function of which to determine the gradient (partial derivatives).
-        Should take `xk` as first argument, other arguments to `f` can be
-        supplied in ``*args``. Should return a scalar, the value of the
-        function at `xk`.
+        Function of which to estimate the derivatives of. Has the signature
+        ``f(xk, *args)`` where `xk` is the argument in the form of a 1-D array
+        and `args` is a tuple of any additional fixed parameters needed to
+        completely specify the function. The argument `xk` passed to this
+        function is an ndarray of shape (n,) (never a scalar even if n=1).
+        It must return a 1-D array_like of shape (m,) or a scalar.
+
+        .. versionchanged:: 1.9.0
+            `f` is now able to return a 1-D array-like, with the :math:`(m, n)`
+            Jacobian being estimated.
+
     epsilon : {float, array_like}, optional
         Increment to `xk` to use for determining the function gradient.
         If a scalar, uses the same finite difference delta for all partial
@@ -930,7 +943,7 @@ def approx_fprime(xk, f, epsilon=_epsilon, *args):
 
     Returns
     -------
-    grad : ndarray
+    jac : ndarray
         The partial derivatives of `f` to `xk`.
 
     See Also
@@ -946,9 +959,6 @@ def approx_fprime(xk, f, epsilon=_epsilon, *args):
         f'[i] = ---------------------------------
                             epsilon[i]
 
-    The main use of `approx_fprime` is in scalar function optimizers like
-    `fmin_bfgs`, to determine numerically the Jacobian of a function.
-
     Examples
     --------
     >>> from scipy import optimize
@@ -964,15 +974,7 @@ def approx_fprime(xk, f, epsilon=_epsilon, *args):
 
     """
     xk = np.asarray(xk, float)
-
     f0 = f(xk, *args)
-    if not np.isscalar(f0):
-        try:
-            f0 = f0.item()
-        except (ValueError, AttributeError) as e:
-            raise ValueError("The user-provided "
-                             "objective function must "
-                             "return a scalar value.") from e
 
     return approx_derivative(f, xk, method='2-point', abs_step=epsilon,
                              args=args, f0=f0)
@@ -988,7 +990,7 @@ def check_grad(func, grad, x0, *args, epsilon=_epsilon,
     func : callable ``func(x0, *args)``
         Function whose derivative is to be checked.
     grad : callable ``grad(x0, *args)``
-        Gradient of `func`.
+        Jacobian of `func`.
     x0 : ndarray
         Points to check `grad` against forward difference approximation of grad
         using `func`.
@@ -1002,6 +1004,7 @@ def check_grad(func, grad, x0, *args, epsilon=_epsilon,
         are used to check `grad` against forward difference approximation
         using `func`. By default it is ``'all'``, in which case, all
         the one hot direction vectors are considered to check `grad`.
+        If `func` is a vector valued function then only ``'all'`` can be used.
     seed : {None, int, `numpy.random.Generator`,
             `numpy.random.RandomState`}, optional
 
@@ -1049,12 +1052,16 @@ def g(w, func, x0, v, *args):
         return func(x0 + w*v, *args)
 
     if direction == 'random':
+        _grad = np.asanyarray(grad(x0, *args))
+        if _grad.ndim > 1:
+            raise ValueError("'random' can only be used with scalar valued"
+                             " func")
         random_state = check_random_state(seed)
         v = random_state.normal(0, 1, size=(x0.shape))
         _args = (func, x0, v) + args
         _func = g
         vars = np.zeros((1,))
-        analytical_grad = np.dot(grad(x0, *args), v)
+        analytical_grad = np.dot(_grad, v)
     elif direction == 'all':
         _args = args
         _func = func
@@ -1064,8 +1071,9 @@ def g(w, func, x0, v, *args):
         raise ValueError("{} is not a valid string for "
                          "``direction`` argument".format(direction))
 
-    return sqrt(sum((analytical_grad -
-                     approx_fprime(vars, _func, step, *_args))**2))
+    return np.sqrt(np.sum(np.abs(
+        (analytical_grad - approx_fprime(vars, _func, step, *_args))**2
+    )))
 
 
 def approx_fhess_p(x0, p, fprime, epsilon, *args):


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -76,6 +76,11 @@ def der_x_sinx(x):
                   x_sinx, der_x_sinx, x0,
                   direction='random_projection', seed=1234)
 
+    # checking can be done for derivatives of vector valued functions
+    r = optimize.check_grad(himmelblau_grad, himmelblau_hess, himmelblau_x0,
+                            direction='all', seed=1234)
+    assert r < 5e-7
+
 
 class CheckOptimize:
     """ Base test case for a simple constrained entropy maximization problem
@@ -2568,3 +2573,13 @@ def func(x):
             assert hasattr(result, "fun")
             assert hasattr(result, "nfev")
             assert hasattr(result, "nit")
+
+
+def test_approx_fprime():
+    # check that approx_fprime (serviced by approx_derivative) works for
+    # jac and hess
+    g = optimize.approx_fprime(himmelblau_x0, himmelblau)
+    assert_allclose(g, himmelblau_grad(himmelblau_x0), rtol=5e-6)
+
+    h = optimize.approx_fprime(himmelblau_x0, himmelblau_grad)
+    assert_allclose(h, himmelblau_hess(himmelblau_x0), rtol=5e-6)