Repository: scipy/scipy

Issue:
diff_step in least_squares does absolute step while docs say relative step
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

The `least_squares` method of `scipy.optimize` has a keyword argument `diff_step`, which allows the user to define the relative step size to be used in computing the numerical Jacobian. The doc strings says: The actual step is computed as ``x * diff_step``. But it, unfortunately, doesn't. It takes an absolute step. In example below the initial parameters are 1000 and 0.01. The `diff_step` is 1e-4, so the Jacobian should be computed with three parameter sets: (1000, 0.01), (1000.1, 0.01), and (1000, 0.010001). But it doesn't do a relative step, it simply adds 1e-4. 

#### Reproducing code example:
```
xobs = np.array([0, 1, 2, 3])
yobs = np.array([0.1, 1005, 1993, 3015])

def residuals(p, xobs=xobs, yobs=yobs):
    print(p)
    return yobs - (p[0] * xobs + p[1])

popt = so.least_squares(residuals, [1000, 0.01], diff_step=1e-4, 
        max_nfev=1)  # just 1 step to show the problem
```

Output (which is incorrect):
```
[1.e+03 1.e-02]
[1.0001e+03 1.0000e-02]
[1.00e+03 1.01e-02]
```

#### Error message:
<!-- If any, paste the *full* error message inside a code block
as above (starting from line Traceback)
-->

#### Scipy/Numpy/Python version information:
```
1.4.1 1.17.2 sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
```


Discussion:
The finite difference is done here: https://github.com/scipy/scipy/blob/e291b10ec85378cf87afea03befb8a69d16e7ad6/scipy/optimize/_numdiff.py#L210. If the parameter is less than one then the step is calculated slightly differently. Perhaps the documentation could up updated slightly.
Thanks. That explains it. Not sure if that is desirable (at least not for my problem).
Turns out `diff_step` can also be an array, so you can trick the problem by specifying different values if you know the approximate parameter values. The following code does what I want to do (at least for the initial parameters)
```
popt = so.least_squares(residuals, [1000, 0.01], diff_step=[1e-4, 1e-6], max_nfev=1)
```
Output:
```
[1.e+03 1.e-02]
[1.0001e+03 1.0000e-02]
[1.0000e+03 1.0001e-02]
```
And in case the doc string gets updated. The description of `max_nfev` isn't quite accurate either. `max_nfev=1` does 1 iteration in the least_squares optimization, which for the case with 2 parameters and a 2-point Jacobian, equals 3 function evaluations. So something like

```Maximum number of iterations in the least_squares optimization before termination```.

No big deal probably, but may as well get it right.
I think that behaviour is chosen to balance formula and truncation errors. If X got really small should one still have a rel step?
When tracking function evaluations it's always going to be difficult to limit to the exact amount because of the problem you've just described. However the number of iterations and the number of function evaluations are two different things.
`rel_step * sign(x0) * max(1, abs(x0))` is not useful if the optimal value of x0 is small. 
My optimal value is around 1e-4, so `least_squares` can not compute a reasonable numerical derivative. I suggest to change the computation of the numerical derivative and/or make it an option not to do the multiplication with `max(1, abs(x0))` but just with `abs(x0)`
I understand that the number of iterations is different from the number of function evaluations.
As far as I can see, the maximum number of function evaluations is now used to set the maximum number of iterations. So I propose to change the doc string (although the name of the keyword argument would be a bit confusing). 
The existing behaviour makes sure that the finite difference absolute step is scaled correctly for the default situation where no `rel_step` is specified, which is `sqrt(EPS) * np.abs(x0)` for `x0 >=1`, for 2-sided FD.  When `x0 < 1` then an absolute step of ~ `sqrt(EPS)` should be used because otherwise the step size is too small and runs into floating point issues.

The [absolute step size calculation](https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L99) could possibly be changed to:

```
def _compute_absolute_step(rel_step, x0, method):
    sign_x0 = (x0 >= 0).astype(float) * 2 - 1
    if rel_step is None:
        rel_step = relative_step[method]
        abs_step = rel_step * sign_x0 * np.maximum(1.0, np.abs(x0))
    else:
        abs_step = rel_step * sign_x0
    return abs_step
```
which would ensure that the user-specified rel_step size was used.
I think you forgot the `np.abs(x0)` term. After the `else` it should be
```
        abs_step = rel_step * sign_x0 * np.abs(x0)
```  
This would surely help cases where the optimal value is small, for example, like my case, where it is between 1e-3 and 1e-5. 

Question: Why not simply `sign_x0 = np.sign(x0)`? Only diff I can see is if `x0=0`. 
You're correct about my suggested change. From memory the calculation of `sign_x0` is necessary, but I cant immediately remember why. That could do with a comment.
> Why not simply sign_x0 = np.sign(x0)? Only diff I can see is if x0=0.

Thinking about it more, I think we want `sign_x0` to be 1 when `x0 == 0`, not 0.

Resolving patch:
diff --git a/scipy/optimize/_numdiff.py b/scipy/optimize/_numdiff.py
--- a/scipy/optimize/_numdiff.py
+++ b/scipy/optimize/_numdiff.py
@@ -166,10 +166,28 @@ def _compute_absolute_step(rel_step, x0, f0, method):
     smaller floating point dtypes (e.g. np.float32), then the absolute
     step size will be calculated from the smallest floating point size.
     """
-    if rel_step is None:
-        rel_step = _eps_for_method(x0.dtype, f0.dtype, method)
+    # this is used instead of np.sign(x0) because we need
+    # sign_x0 to be 1 when x0 == 0.
     sign_x0 = (x0 >= 0).astype(float) * 2 - 1
-    return rel_step * sign_x0 * np.maximum(1.0, np.abs(x0))
+
+    rstep = _eps_for_method(x0.dtype, f0.dtype, method)
+
+    if rel_step is None:
+        abs_step = rstep * sign_x0 * np.maximum(1.0, np.abs(x0))
+    else:
+        # User has requested specific relative steps.
+        # Don't multiply by max(1, abs(x0) because if x0 < 1 then their
+        # requested step is not used.
+        abs_step = rel_step * sign_x0 * np.abs(x0)
+
+        # however we don't want an abs_step of 0, which can happen if
+        # rel_step is 0, or x0 is 0. Instead, substitute a realistic step
+        dx = ((x0 + abs_step) - x0)
+        abs_step = np.where(dx == 0,
+                            rstep * sign_x0 * np.maximum(1.0, np.abs(x0)),
+                            abs_step)
+
+    return abs_step
 
 
 def _prepare_bounds(bounds, x0):
@@ -285,18 +303,19 @@ def approx_derivative(fun, x0, method='3-point', rel_step=None, abs_step=None,
                      analytically continued to the complex plane. Otherwise,
                      produces bogus results.
     rel_step : None or array_like, optional
-        Relative step size to use. The absolute step size is computed as
-        ``h = rel_step * sign(x0) * max(1, abs(x0))``, possibly adjusted to
-        fit into the bounds. For ``method='3-point'`` the sign of `h` is
-        ignored. If None (default) then step is selected automatically,
-        see Notes.
+        Relative step size to use. If None (default) the absolute step size is
+        computed as ``h = rel_step * sign(x0) * max(1, abs(x0))``, with
+        `rel_step` being selected automatically, see Notes. Otherwise
+        ``h = rel_step * sign(x0) * abs(x0)``. For ``method='3-point'`` the
+        sign of `h` is ignored. The calculated step size is possibly adjusted
+        to fit into the bounds.
     abs_step : array_like, optional
         Absolute step size to use, possibly adjusted to fit into the bounds.
         For ``method='3-point'`` the sign of `abs_step` is ignored. By default
         relative steps are used, only if ``abs_step is not None`` are absolute
         steps used.
     f0 : None or array_like, optional
-        If not None it is assumed to be equal to ``fun(x0)``, in  this case
+        If not None it is assumed to be equal to ``fun(x0)``, in this case
         the ``fun(x0)`` is not called. Default is None.
     bounds : tuple of array_like, optional
         Lower and upper bounds on independent variables. Defaults to no bounds.
@@ -360,9 +379,9 @@ def approx_derivative(fun, x0, method='3-point', rel_step=None, abs_step=None,
     s=3 for '3-point' method. Such relative step approximately minimizes a sum
     of truncation and round-off errors, see [1]_. Relative steps are used by
     default. However, absolute steps are used when ``abs_step is not None``.
-    If any of the absolute steps produces an indistinguishable difference from
-    the original `x0`, ``(x0 + abs_step) - x0 == 0``, then a relative step is
-    substituted for that particular entry.
+    If any of the absolute or relative steps produces an indistinguishable
+    difference from the original `x0`, ``(x0 + dx) - x0 == 0``, then a
+    automatic step size is substituted for that particular entry.
 
     A finite difference scheme for '3-point' method is selected automatically.
     The well-known central difference scheme is used for points sufficiently


Test patch:
diff --git a/scipy/optimize/tests/test__numdiff.py b/scipy/optimize/tests/test__numdiff.py
--- a/scipy/optimize/tests/test__numdiff.py
+++ b/scipy/optimize/tests/test__numdiff.py
@@ -40,7 +40,7 @@ def test_group_columns():
     assert_equal(groups_1, groups_2)
 
 
-def test_correct_eps():
+def test_correct_fp_eps():
     # check that relative step size is correct for FP size
     EPS = np.finfo(np.float64).eps
     relative_step = {"2-point": EPS**0.5,
@@ -717,7 +717,7 @@ def test_exception(self):
                       method='2-point', bounds=(1, np.inf))
 
 
-def test_absolute_step():
+def test_absolute_step_sign():
     # test for gh12487
     # if an absolute step is specified for 2-point differences make sure that
     # the side corresponds to the step. i.e. if step is positive then forward
@@ -768,3 +768,46 @@ def f(x):
         f, [-1, -1], method='2-point', abs_step=-1e-8, bounds=(-1, np.inf)
     )
     assert_allclose(grad, [-1.0, 1.0])
+
+
+def test__compute_absolute_step():
+    # tests calculation of absolute step from rel_step
+    methods = ['2-point', '3-point', 'cs']
+
+    x0 = np.array([1e-5, 0, 1, 1e5])
+
+    EPS = np.finfo(np.float64).eps
+    relative_step = {
+        "2-point": EPS**0.5,
+        "3-point": EPS**(1/3),
+        "cs": EPS**0.5
+    }
+    f0 = np.array(1.0)
+
+    for method in methods:
+        rel_step = relative_step[method]
+        correct_step = np.array([rel_step,
+                                 rel_step * 1.,
+                                 rel_step * 1.,
+                                 rel_step * np.abs(x0[3])])
+
+        abs_step = _compute_absolute_step(None, x0, f0, method)
+        assert_allclose(abs_step, correct_step)
+
+        sign_x0 = (-x0 >= 0).astype(float) * 2 - 1
+        abs_step = _compute_absolute_step(None, -x0, f0, method)
+        assert_allclose(abs_step, sign_x0 * correct_step)
+
+    # if a relative step is provided it should be used
+    rel_step = np.array([0.1, 1, 10, 100])
+    correct_step = np.array([rel_step[0] * x0[0],
+                             relative_step['2-point'],
+                             rel_step[2] * 1.,
+                             rel_step[3] * np.abs(x0[3])])
+
+    abs_step = _compute_absolute_step(rel_step, x0, f0, '2-point')
+    assert_allclose(abs_step, correct_step)
+
+    sign_x0 = (-x0 >= 0).astype(float) * 2 - 1
+    abs_step = _compute_absolute_step(rel_step, -x0, f0, '2-point')
+    assert_allclose(abs_step, sign_x0 * correct_step)
diff --git a/scipy/optimize/tests/test_least_squares.py b/scipy/optimize/tests/test_least_squares.py
--- a/scipy/optimize/tests/test_least_squares.py
+++ b/scipy/optimize/tests/test_least_squares.py
@@ -241,7 +241,6 @@ def test_diff_step(self):
         assert_allclose(res3.x, 0, atol=1e-4)
         assert_equal(res1.x, res2.x)
         assert_equal(res1.nfev, res2.nfev)
-        assert_(res2.nfev != res3.nfev)
 
     def test_incorrect_options_usage(self):
         assert_raises(TypeError, least_squares, fun_trivial, 2.0,