Repository: scipy/scipy

Issue:
ENH: optimize: maintain user dtype
### Is your feature request related to a problem? Please describe.

The functions defined in `optimize/_differentiable_functions.py` all cast the inputs to float64 before evaluation. This is a problem if you want to work at an arbitrarily chosen precision format or change the precision format dynamically to speed up computation. Moreover these casts generates a significant overhead if the inputs in purposely not already in float64. 

### Describe the solution you'd like.

Remove all the the (useless) calls to `astype(float)` from `optimize/_differentiable_functions.py `

### Describe alternatives you've considered.

Add a param allowing to keep the format provided by the user

### Additional context (e.g. screenshots, GIFs)

NA


Discussion:
All the code in `_differentiable_functions` is part of the private API that's used to support `optimize` (you can't import by `from scipy.optimize import ScalarFunction`), none of the functions are public. External code should not really be using them, or relying on them to remain the same.

Rather the question might be "why doesn't this `minimize` method respect the float width of my `x0` array". The answer to that is varied. Frequently minimisers are compiled code which means they need to work with float64. Others rely on double precision for estimating gradients, etc. Most of them would require double precision to work properly.



>All the code in _differentiable_functions is part of the private API that's used to support optimize (you can't import by from scipy.optimize import ScalarFunction), none of the functions are public. External code should not really be using them, or relying on them to remain the same.

Yes but it still heavily impacts a public API (optimizers): this private API is used in lots of Public API because it's wrapping calls to gradients and objective functions. I might have poorly explained it but a simple look at the source code of any optimizers like trust-region ones, makes it clear that any inputs to this optimizers is then casted in float64.

> Frequently minimisers are compiled code which means they need to work with float64. Others rely on double precision for estimating gradients, etc. Most of them would require double precision to work properly.

Erm, no: this overusage of float64 is a thing only in scipy. Lots of modern optimization libraries such as all the deep learning frameworks use single precision by default or sometimes [even lower](https://arxiv.org/abs/1511.00363) in order to optimize the throughput.  I get that scipy runs on CPU so the constraints are different but still: memory-wise it's aweful! For instance, [this paper](https://arxiv.org/abs/1812.03467) prooves that you actually can use traditional optimizers in lower precision formats than float64 
Moreover, compiled code can obviously handle dynamic typing if well written...
Finally, even considering that for some significantly big input dimension, using float32 or lower might significantly impact the convergence of the optimizers, isn't it the responsibility of the user to choose the right input format? 

A note on:

> Moreover, compiled code can obviously handle dynamic typing if well written...

Yes but it's more maintenance/knowledge to have and also can produce larger binaries. But if you have concrete solutions, we are happy to evaluate them.
For `minimize` what is the biggest set of parameters that you envisage will be used? Even at the million parameter mark one is only talking about 8Mb instead of 4 Mb, which is miniscule in RAM terms.

I'm not saying that one shouldn't be thoughtful about creating code that respects float width, it's just that I've not heard noise where using supplying float64 to the objective function has really caused issues.

If you look at [Nelder-Mead](https://github.com/scipy/scipy/blob/main/scipy/optimize/_optimize.py#L635), the code there should respect float width. However, it's just not sane to start modifying code in e.g. [SLSQP](https://github.com/scipy/scipy/blob/main/scipy/optimize/slsqp/slsqp_optmz.f) to do the same thing.

We always welcome people willing to help improve optimize, so if you know how to implement the kinds of changes you'd like to see (and evidence for there being performance penalties), then please help to improve the library.


Resolving patch:
diff --git a/scipy/optimize/_differentiable_functions.py b/scipy/optimize/_differentiable_functions.py
--- a/scipy/optimize/_differentiable_functions.py
+++ b/scipy/optimize/_differentiable_functions.py
@@ -3,6 +3,7 @@
 from ._numdiff import approx_derivative, group_columns
 from ._hessian_update_strategy import HessianUpdateStrategy
 from scipy.sparse.linalg import LinearOperator
+from scipy._lib._array_api import atleast_nd, array_namespace
 
 
 FD_METHODS = ('2-point', '3-point', 'cs')
@@ -103,8 +104,15 @@ def __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step,
                              "to be estimated using one of the "
                              "quasi-Newton strategies.")
 
-        # the astype call ensures that self.x is a copy of x0
-        self.x = np.atleast_1d(x0).astype(float)
+        self.xp = xp = array_namespace(x0)
+        _x = atleast_nd(x0, ndim=1, xp=xp)
+        _dtype = xp.float64
+        if xp.isdtype(_x.dtype, "real floating"):
+            _dtype = _x.dtype
+
+        # promotes to floating
+        self.x = xp.astype(_x, _dtype)
+        self.x_dtype = _dtype
         self.n = self.x.size
         self.nfev = 0
         self.ngev = 0
@@ -231,7 +239,9 @@ def update_x(x):
                 self.g_prev = self.g
                 # ensure that self.x is a copy of x. Don't store a reference
                 # otherwise the memoization doesn't work properly.
-                self.x = np.atleast_1d(x).astype(float)
+
+                _x = atleast_nd(x, ndim=1, xp=self.xp)
+                self.x = self.xp.astype(_x, self.x_dtype)
                 self.f_updated = False
                 self.g_updated = False
                 self.H_updated = False
@@ -240,7 +250,8 @@ def update_x(x):
             def update_x(x):
                 # ensure that self.x is a copy of x. Don't store a reference
                 # otherwise the memoization doesn't work properly.
-                self.x = np.atleast_1d(x).astype(float)
+                _x = atleast_nd(x, ndim=1, xp=self.xp)
+                self.x = self.xp.astype(_x, self.x_dtype)
                 self.f_updated = False
                 self.g_updated = False
                 self.H_updated = False
@@ -323,7 +334,16 @@ def __init__(self, fun, x0, jac, hess,
                              "be estimated using one of the quasi-Newton "
                              "strategies.")
 
-        self.x = np.atleast_1d(x0).astype(float)
+        self.xp = xp = array_namespace(x0)
+        _x = atleast_nd(x0, ndim=1, xp=xp)
+        _dtype = xp.float64
+        if xp.isdtype(_x.dtype, "real floating"):
+            _dtype = _x.dtype
+
+        # promotes to floating
+        self.x = xp.astype(_x, _dtype)
+        self.x_dtype = _dtype
+
         self.n = self.x.size
         self.nfev = 0
         self.njev = 0
@@ -492,14 +512,16 @@ def update_x(x):
                 self._update_jac()
                 self.x_prev = self.x
                 self.J_prev = self.J
-                self.x = np.atleast_1d(x).astype(float)
+                _x = atleast_nd(x, ndim=1, xp=self.xp)
+                self.x = self.xp.astype(_x, self.x_dtype)
                 self.f_updated = False
                 self.J_updated = False
                 self.H_updated = False
                 self._update_hess()
         else:
             def update_x(x):
-                self.x = np.atleast_1d(x).astype(float)
+                _x = atleast_nd(x, ndim=1, xp=self.xp)
+                self.x = self.xp.astype(_x, self.x_dtype)
                 self.f_updated = False
                 self.J_updated = False
                 self.H_updated = False
@@ -569,7 +591,16 @@ def __init__(self, A, x0, sparse_jacobian):
 
         self.m, self.n = self.J.shape
 
-        self.x = np.atleast_1d(x0).astype(float)
+        self.xp = xp = array_namespace(x0)
+        _x = atleast_nd(x0, ndim=1, xp=xp)
+        _dtype = xp.float64
+        if xp.isdtype(_x.dtype, "real floating"):
+            _dtype = _x.dtype
+
+        # promotes to floating
+        self.x = xp.astype(_x, _dtype)
+        self.x_dtype = _dtype
+
         self.f = self.J.dot(self.x)
         self.f_updated = True
 
@@ -578,7 +609,8 @@ def __init__(self, A, x0, sparse_jacobian):
 
     def _update_x(self, x):
         if not np.array_equal(x, self.x):
-            self.x = np.atleast_1d(x).astype(float)
+            _x = atleast_nd(x, ndim=1, xp=self.xp)
+            self.x = self.xp.astype(_x, self.x_dtype)
             self.f_updated = False
 
     def fun(self, x):
diff --git a/scipy/optimize/_numdiff.py b/scipy/optimize/_numdiff.py
--- a/scipy/optimize/_numdiff.py
+++ b/scipy/optimize/_numdiff.py
@@ -6,6 +6,7 @@
 from scipy.sparse.linalg import LinearOperator
 from ..sparse import issparse, csc_matrix, csr_matrix, coo_matrix, find
 from ._group_columns import group_dense, group_sparse
+from scipy._lib._array_api import atleast_nd, array_namespace
 
 
 def _adjust_scheme_to_bounds(x0, h, num_steps, scheme, lb, ub):
@@ -438,7 +439,15 @@ def approx_derivative(fun, x0, method='3-point', rel_step=None, abs_step=None,
     if method not in ['2-point', '3-point', 'cs']:
         raise ValueError("Unknown method '%s'. " % method)
 
-    x0 = np.atleast_1d(x0)
+    xp = array_namespace(x0)
+    _x = atleast_nd(x0, ndim=1, xp=xp)
+    _dtype = xp.float64
+    if xp.isdtype(_x.dtype, "real floating"):
+        _dtype = _x.dtype
+
+    # promotes to floating
+    x0 = xp.astype(_x, _dtype)
+
     if x0.ndim > 1:
         raise ValueError("`x0` must have at most 1 dimension.")
 
@@ -453,6 +462,11 @@ def approx_derivative(fun, x0, method='3-point', rel_step=None, abs_step=None,
                          "`as_linear_operator` is True.")
 
     def fun_wrapped(x):
+        # send user function same fp type as x0. (but only if cs is not being
+        # used
+        if xp.isdtype(x.dtype, "real floating"):
+            x = xp.astype(x, x0.dtype)
+
         f = np.atleast_1d(fun(x, *args, **kwargs))
         if f.ndim > 1:
             raise RuntimeError("`fun` return value has "


Test patch:
diff --git a/scipy/optimize/tests/test__numdiff.py b/scipy/optimize/tests/test__numdiff.py
--- a/scipy/optimize/tests/test__numdiff.py
+++ b/scipy/optimize/tests/test__numdiff.py
@@ -450,10 +450,11 @@ def err(p, x, y):
 
         # parameter vector is float64, func output is float32
         def err_fp32(p):
+            assert p.dtype == np.float32
             return err(p, x, y).astype(np.float32)
-        jac_fp = approx_derivative(err_fp32, p0,
+
+        jac_fp = approx_derivative(err_fp32, p0.astype(np.float32),
                                    method='2-point')
-        assert err_fp32(p0).dtype == np.float32
         assert_allclose(jac_fp, jac_fp64, atol=1e-3)
 
         # check upper bound of error on the derivative for 2-point
diff --git a/scipy/optimize/tests/test_differentiable_functions.py b/scipy/optimize/tests/test_differentiable_functions.py
--- a/scipy/optimize/tests/test_differentiable_functions.py
+++ b/scipy/optimize/tests/test_differentiable_functions.py
@@ -362,6 +362,20 @@ def test_lowest_x(self):
         assert_equal(sf._lowest_f, 0.0)
         assert_equal(sf._lowest_x, [1.0, 1.0, 1.0])
 
+    def test_float_size(self):
+        x0 = np.array([2, 3, 4]).astype(np.float32)
+
+        # check that ScalarFunction/approx_derivative always send the correct
+        # float width
+        def rosen_(x):
+            assert x.dtype == np.float32
+            return rosen(x)
+
+        sf = ScalarFunction(rosen_, x0, (), '2-point', rosen_hess,
+                            None, (-np.inf, np.inf))
+        res = sf.fun(x0)
+        assert res.dtype == np.float32
+
 
 class ExVectorialFunction:
 
@@ -373,12 +387,12 @@ def __init__(self):
     def fun(self, x):
         self.nfev += 1
         return np.array([2*(x[0]**2 + x[1]**2 - 1) - x[0],
-                         4*(x[0]**3 + x[1]**2 - 4) - 3*x[0]])
+                         4*(x[0]**3 + x[1]**2 - 4) - 3*x[0]], dtype=x.dtype)
 
     def jac(self, x):
         self.njev += 1
         return np.array([[4*x[0]-1, 4*x[1]],
-                         [12*x[0]**2-3, 8*x[1]]])
+                         [12*x[0]**2-3, 8*x[1]]], dtype=x.dtype)
 
     def hess(self, x, v):
         self.nhev += 1
@@ -655,6 +669,19 @@ def test_x_storage_overlap(self):
             assert_equal(vf.fun(x0), ex.fun(x0))
             assert x0 is not vf.x
 
+    def test_float_size(self):
+        ex = ExVectorialFunction()
+        x0 = np.array([1.0, 0.0]).astype(np.float32)
+
+        vf = VectorFunction(ex.fun, x0, ex.jac, ex.hess, None, None,
+                            (-np.inf, np.inf), None)
+
+        res = vf.fun(x0)
+        assert res.dtype == np.float32
+
+        res = vf.jac(x0)
+        assert res.dtype == np.float32
+
 
 def test_LinearVectorFunction():
     A_dense = np.array([