Repository: scipy/scipy

Issue:
least_squares broken with float32
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

`optimize.least_squares` doesn't work reliably with float32. Changing the dtype to float64 (or removing it) yields the expected result.

#### Reproducing code example:
```
import matplotlib.pyplot as plt
import numpy as np
from scipy import optimize

x = np.linspace(0, 1, 100).astype("float32")
y = np.random.random(100).astype("float32")


def func(p, x):
    return p[0] + p[1] * x


def err(p, x, y):
    return func(p, x) - y


res = optimize.least_squares(err, [-1.0, -1.0], args=(x, y))
plt.plot(x, y, "x")
plt.plot(x, func(res.x, x))
plt.show()
```

#### Error message:
No error is raised, just the fit does not converge.


#### Scipy/Numpy/Python version information:
<!-- You can simply run the following and paste the result in a code block-->
```
1.5.3 1.18.4 sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0)
```



Discussion:
Some additional infos:
It terminates with message "`gtol` termination condition is satisfied."

If I set `gtol=None` I get the following output:

```
~/.local/lib/python3.8/site-packages/scipy/optimize/_lsq/common.py:113: RuntimeWarning: invalid value encountered in true_divide
  p_norm = norm(suf / denom)
~/.local/lib/python3.8/site-packages/scipy/optimize/_lsq/common.py:115: RuntimeWarning: invalid value encountered in true_divide
  phi_prime = -np.sum(suf ** 2 / denom**3) / p_norm
The maximum number of function evaluations is exceeded.
```
This arises because of floating point issues in the numerical differentation of the user function. The initial gradient (J0) of `err` is computed [here](https://github.com/scipy/scipy/blob/master/scipy/optimize/_lsq/least_squares.py#L884). Your `err` function returns `np.float32` arrays, but the default step size for the numerical differentiation is [worked out on the assumption one is using np.float64](https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L111). This step size is too small, and so when one takes a small step `dx` all the `df` are 0. Therefore the initial Jacobian is all zeros, a consequence of working with too small an EPS for the floating point size.

```
>>> from scipy.optimize._numdiff import approx_derivative
>>> # I trimmed the output array 
>>> approx_derivative(err, [-1.0, -1.0], method='2-point', args=(x, y))
array([[-0., -0.],
       [-0., -0.],
       [-0., -0.],
...
       [-0., -0.],
       [-0., -0.],
       [-0., -0.]])
```
This causes the premature termination of the minimisation ([here](https://github.com/scipy/scipy/blob/master/scipy/optimize/_lsq/trf.py#L452)).
Note that if one uses `leastsq` I think the correct EPS is calculated using an [appropriate value for the floating point size](https://github.com/scipy/scipy/blob/master/scipy/optimize/minpack.py#L417).

Wherever possible I always recommend that people work with float64 data because of these issues.

Possible improvements:
1. Improve the documentation to point out that this kind of issue exists when one is working with FP arithmetic.
2. Change `_numdiff` so that it uses the correct step size for the dtype.

Resolving patch:
diff --git a/scipy/optimize/_numdiff.py b/scipy/optimize/_numdiff.py
--- a/scipy/optimize/_numdiff.py
+++ b/scipy/optimize/_numdiff.py
@@ -1,5 +1,5 @@
 """Routines for numerical differentiation."""
-
+import functools
 import numpy as np
 from numpy.linalg import norm
 
@@ -7,8 +7,6 @@
 from ..sparse import issparse, csc_matrix, csr_matrix, coo_matrix, find
 from ._group_columns import group_dense, group_sparse
 
-EPS = np.finfo(np.float64).eps
-
 
 def _adjust_scheme_to_bounds(x0, h, num_steps, scheme, lb, ub):
     """Adjust final difference scheme to the presence of bounds.
@@ -89,12 +87,61 @@ def _adjust_scheme_to_bounds(x0, h, num_steps, scheme, lb, ub):
     return h_adjusted, use_one_sided
 
 
-relative_step = {"2-point": EPS**0.5,
-                 "3-point": EPS**(1/3),
-                 "cs": EPS**0.5}
+@functools.lru_cache()
+def _eps_for_method(x0_dtype, f0_dtype, method):
+    """
+    Calculates relative EPS step to use for a given data type
+    and numdiff step method.
+
+    Progressively smaller steps are used for larger floating point types.
+
+    Parameters
+    ----------
+    f0_dtype: np.dtype
+        dtype of function evaluation
+
+    x0_dtype: np.dtype
+        dtype of parameter vector
+
+    method: {'2-point', '3-point', 'cs'}
 
+    Returns
+    -------
+    EPS: float
+        relative step size. May be np.float16, np.float32, np.float64
 
-def _compute_absolute_step(rel_step, x0, method):
+    Notes
+    -----
+    The default relative step will be np.float64. However, if x0 or f0 are
+    smaller floating point types (np.float16, np.float32), then the smallest
+    floating point type is chosen.
+    """
+    # the default EPS value
+    EPS = np.finfo(np.float64).eps
+
+    x0_is_fp = False
+    if np.issubdtype(x0_dtype, np.inexact):
+        # if you're a floating point type then over-ride the default EPS
+        EPS = np.finfo(x0_dtype).eps
+        x0_itemsize = np.dtype(x0_dtype).itemsize
+        x0_is_fp = True
+
+    if np.issubdtype(f0_dtype, np.inexact):
+        f0_itemsize = np.dtype(f0_dtype).itemsize
+        # choose the smallest itemsize between x0 and f0
+        if x0_is_fp and f0_itemsize < x0_itemsize:
+            EPS = np.finfo(f0_dtype).eps
+
+    if method in ["2-point", "cs"]:
+        return EPS**0.5
+    elif method in ["3-point"]:
+        return EPS**(1/3)
+    else:
+        raise RuntimeError("Unknown step method, should be one of "
+                           "{'2-point', '3-point', 'cs'}")
+
+
+def _compute_absolute_step(rel_step, x0, f0, method):
     """
     Computes an absolute step from a relative step for finite difference
     calculation.
@@ -105,10 +152,22 @@ def _compute_absolute_step(rel_step, x0, method):
         Relative step for the finite difference calculation
     x0 : np.ndarray
         Parameter vector
+    f0 : np.ndarray or scalar
     method : {'2-point', '3-point', 'cs'}
+
+    Returns
+    -------
+    h : float
+        The absolute step size
+
+    Notes
+    -----
+    `h` will always be np.float64. However, if `x0` or `f0` are
+    smaller floating point dtypes (e.g. np.float32), then the absolute
+    step size will be calculated from the smallest floating point size.
     """
     if rel_step is None:
-        rel_step = relative_step[method]
+        rel_step = _eps_for_method(x0.dtype, f0.dtype, method)
     sign_x0 = (x0 >= 0).astype(float) * 2 - 1
     return rel_step * sign_x0 * np.maximum(1.0, np.abs(x0))
 
@@ -295,10 +354,11 @@ def approx_derivative(fun, x0, method='3-point', rel_step=None, abs_step=None,
 
     Notes
     -----
-    If `rel_step` is not provided, it assigned to ``EPS**(1/s)``, where EPS is
-    machine epsilon for float64 numbers, s=2 for '2-point' method and s=3 for
-    '3-point' method. Such relative step approximately minimizes a sum of
-    truncation and round-off errors, see [1]_. Relative steps are used by
+    If `rel_step` is not provided, it assigned as ``EPS**(1/s)``, where EPS is
+    determined from the smallest floating point dtype of `x0` or `fun(x0)`,
+    ``np.finfo(x0.dtype).eps``, s=2 for '2-point' method and
+    s=3 for '3-point' method. Such relative step approximately minimizes a sum
+    of truncation and round-off errors, see [1]_. Relative steps are used by
     default. However, absolute steps are used when ``abs_step is not None``.
     If any of the absolute steps produces an indistinguishable difference from
     the original `x0`, ``(x0 + abs_step) - x0 == 0``, then a relative step is
@@ -392,14 +452,14 @@ def fun_wrapped(x):
 
     if as_linear_operator:
         if rel_step is None:
-            rel_step = relative_step[method]
+            rel_step = _eps_for_method(x0.dtype, f0.dtype, method)
 
         return _linear_operator_difference(fun_wrapped, x0,
                                            f0, rel_step, method)
     else:
         # by default we use rel_step
         if abs_step is None:
-            h = _compute_absolute_step(rel_step, x0, method)
+            h = _compute_absolute_step(rel_step, x0, f0, method)
         else:
             # user specifies an absolute step
             sign_x0 = (x0 >= 0).astype(float) * 2 - 1
@@ -409,8 +469,8 @@ def fun_wrapped(x):
             # or small. In which case fall back to relative step.
             dx = ((x0 + h) - x0)
             h = np.where(dx == 0,
-                         relative_step[method] * sign_x0 *
-                         np.maximum(1.0, np.abs(x0)),
+                         _eps_for_method(x0.dtype, f0.dtype, method) *
+                         sign_x0 * np.maximum(1.0, np.abs(x0)),
                          h)
 
         if method == '2-point':


Test patch:
diff --git a/scipy/optimize/tests/test__numdiff.py b/scipy/optimize/tests/test__numdiff.py
--- a/scipy/optimize/tests/test__numdiff.py
+++ b/scipy/optimize/tests/test__numdiff.py
@@ -9,7 +9,7 @@
 
 from scipy.optimize._numdiff import (
     _adjust_scheme_to_bounds, approx_derivative, check_derivative,
-    group_columns)
+    group_columns, _eps_for_method, _compute_absolute_step)
 
 
 def test_group_columns():
@@ -40,6 +40,42 @@ def test_group_columns():
     assert_equal(groups_1, groups_2)
 
 
+def test_correct_eps():
+    # check that relative step size is correct for FP size
+    EPS = np.finfo(np.float64).eps
+    relative_step = {"2-point": EPS**0.5,
+                    "3-point": EPS**(1/3),
+                     "cs": EPS**0.5}
+    for method in ['2-point', '3-point', 'cs']:
+        assert_allclose(
+            _eps_for_method(np.float64, np.float64, method),
+            relative_step[method])
+        assert_allclose(
+            _eps_for_method(np.complex128, np.complex128, method),
+            relative_step[method]
+        )
+
+    # check another FP size
+    EPS = np.finfo(np.float32).eps
+    relative_step = {"2-point": EPS**0.5,
+                    "3-point": EPS**(1/3),
+                     "cs": EPS**0.5}
+
+    for method in ['2-point', '3-point', 'cs']:
+        assert_allclose(
+            _eps_for_method(np.float64, np.float32, method),
+            relative_step[method]
+        )
+        assert_allclose(
+            _eps_for_method(np.float32, np.float64, method),
+            relative_step[method]
+        )
+        assert_allclose(
+            _eps_for_method(np.float32, np.float32, method),
+            relative_step[method]
+        )
+
+
 class TestAdjustSchemeToBounds(object):
     def test_no_bounds(self):
         x0 = np.zeros(3)
@@ -389,6 +425,59 @@ def test_non_numpy(self):
         assert_raises(TypeError, approx_derivative, self.jac_non_numpy, x0,
                       **dict(method='cs'))
 
+    def test_fp(self):
+        # checks that approx_derivative works for FP size other than 64.
+        # Example is derived from the minimal working example in gh12991.
+        np.random.seed(1)
+
+        def func(p, x):
+            return p[0] + p[1] * x
+
+        def err(p, x, y):
+            return func(p, x) - y
+
+        x = np.linspace(0, 1, 100, dtype=np.float64)
+        y = np.random.random(100).astype(np.float64)
+        p0 = np.array([-1.0, -1.0])
+
+        jac_fp64 = approx_derivative(err, p0, method='2-point', args=(x, y))
+
+        # parameter vector is float32, func output is float64
+        jac_fp = approx_derivative(err, p0.astype(np.float32),
+                                   method='2-point', args=(x, y))
+        assert err(p0, x, y).dtype == np.float64
+        assert_allclose(jac_fp, jac_fp64, atol=1e-3)
+
+        # parameter vector is float64, func output is float32
+        err_fp32 = lambda p: err(p, x, y).astype(np.float32)
+        jac_fp = approx_derivative(err_fp32, p0,
+                                   method='2-point')
+        assert err_fp32(p0).dtype == np.float32
+        assert_allclose(jac_fp, jac_fp64, atol=1e-3)
+
+        # check upper bound of error on the derivative for 2-point
+        f = lambda x: np.sin(x)
+        g = lambda x: np.cos(x)
+        hess = lambda x: -np.sin(x)
+
+        def calc_atol(h, x0, f, hess, EPS):
+            # truncation error
+            t0 = h / 2 * max(np.abs(hess(x0)), np.abs(hess(x0 + h)))
+            # roundoff error. There may be a divisor (>1) missing from
+            # the following line, so this contribution is possibly
+            # overestimated
+            t1 = EPS / h * max(np.abs(f(x0)), np.abs(f(x0 + h)))
+            return t0 + t1
+
+        for dtype in [np.float16, np.float32, np.float64]:
+            EPS = np.finfo(dtype).eps
+            x0 = np.array(1.0).astype(dtype)
+            h = _compute_absolute_step(None, x0, f(x0), '2-point')
+            atol = calc_atol(h, x0, f, hess, EPS)
+            err = approx_derivative(f, x0, method='2-point',
+                                    abs_step=h) - g(x0)
+            assert abs(err) < atol
+
     def test_check_derivative(self):
         x0 = np.array([-10.0, 10])
         accuracy = check_derivative(self.fun_vector_vector,
diff --git a/scipy/optimize/tests/test_least_squares.py b/scipy/optimize/tests/test_least_squares.py
--- a/scipy/optimize/tests/test_least_squares.py
+++ b/scipy/optimize/tests/test_least_squares.py
@@ -754,3 +754,28 @@ def test_small_tolerances_for_lm():
                              (1e-13, 1e-13, None)]:
         assert_raises(ValueError, least_squares, fun_trivial, 2.0, xtol=xtol,
                       ftol=ftol, gtol=gtol, method='lm')
+
+
+def test_fp32_gh12991():
+    # checks that smaller FP sizes can be used in least_squares
+    # this is the minimum working example reported for gh12991
+    np.random.seed(1)
+
+    x = np.linspace(0, 1, 100).astype("float32")
+    y = np.random.random(100).astype("float32")
+
+    def func(p, x):
+        return p[0] + p[1] * x
+
+    def err(p, x, y):
+        return func(p, x) - y
+
+    res = least_squares(err, [-1.0, -1.0], args=(x, y))
+    # previously the initial jacobian calculated for this would be all 0
+    # and the minimize would terminate immediately, with nfev=1, would
+    # report a successful minimization (it shouldn't have done), but be
+    # unchanged from the initial solution.
+    # It was terminating early because the underlying approx_derivative
+    # used a step size for FP64 when the working space was FP32.
+    assert res.nfev > 3
+    assert_allclose(res.x, np.array([0.4082241, 0.15530563]), atol=5e-5)