Repository: scipy/scipy

Issue:
DOC: dual_annealing optimizer does not pass bounds to minimizer when minimizer_kwargs is partially given
### Issue with current documentation:

Regarding the [documentation of ```dual_annealing```](https://scipy.github.io/devdocs/reference/generated/scipy.optimize.dual_annealing.html#scipy.optimize.dual_annealing):

The current implementation of ```dual_annealing``` only generates the new ```kwargs``` dict for the local minimizer when there is NO ```minimizer_kwargs``` provided.
https://github.com/scipy/scipy/blob/4ee0de31b758f81be85435a0ad72d315a840441d/scipy/optimize/_dual_annealing.py#L403C1-L413C70

This causes issues during local search, such as below:
```python
dual_annealing(
    func,
    x0 = x0,
    args = args,
    bounds = bounds,
    minimizer_kwargs = dict(options=dict(eps=1e-5))
)
```
The only kwargs items passed to the minimizer is ```{'eps': 1e-5}```, where users like myself might expect ```{'eps': 1e-5, 'bounds': bounds}``` to be passed.
This results in an extremely inefficient local search, since the local minimizer finds some coordinates outside the bounds only to get rejected by the sanity check code (below).
https://github.com/scipy/scipy/blob/4ee0de31b758f81be85435a0ad72d315a840441d/scipy/optimize/_dual_annealing.py#L428C1-L431C43

To avoid this, current workaround is to add ```bounds``` to ```minimizer_kwargs```:
```python
dual_annealing(
    func,
    x0 = x0,
    args = args,
    bounds = bounds,
    minimizer_kwargs = dict(bounds=bounds, options=dict(eps=1e-5))
)
```

### Idea or request for content:

It will be a great help for users to explicitly mention in the document (or change the implementation so that default kwargs get updated by user kwargs: see below) that minimizer_kwargs will be the ONLY kwargs that get passed to the minimizer. 
Current description of ```minimizer_kwargs``` option is

> Extra keyword arguments to be passed to the local minimizer

and this is misleading, since it gives an impression that only the modification to the default values (such as ```eps``` parameter in the example above) are needed, in which case users may think ```bounds``` is not necessary.


---------
Alternatively, changing the implementation (of the kwargs generator code I linked above) to something like this
```python
minimizer_kwargs_default = {
    'method': 'L-BFGS-B',
    'maxiter': ls_max_iter,
    'bounds': list(zip(self.lower, self.upper))
}

minimizer_kwargs_updated = minimizer_kwargs_default.copy()
if self.kwargs:
    minimizer_kwargs_updated.update(self.kwargs)

```
might match the expected behavior.

### Additional context (e.g. screenshots, GIFs)

_No response_


Discussion:
Thanks for the report @SterlingYM .

I understand that the API might be counter intuitive but passing bounds automatically to the local optimizer would break backwards compatibility. Another issue is that not all local optimizers support bounds.

I suggest that we add this behavior to the docstring explicitly. What do you think?
@dschmitz89 I understand changing the implementation comes with some difficulties. At least I think adding docstrings is worth it, since I spent many hours debugging this issue and I am sure someone else would do that in the future.

A few alternative ideas:
1. Add a boolean option (something like ```pass_bounds=False```) bounds are added to the user-defined kwargs when True. We can set it to False by default to maintain the backward compatibility.
2. Make additional option (```minimizer_kwargs_extra={}```) so that when this is used, these are strictly _additional_ kwargs as the original docstrings suggest, not the _only_ kwargs to be passed. If this is used only for updating the existing kwargs (whether user-defined or default), this is also backward compatible.
~~~python
self.kwargs = minimizer_kwargs
self.kwargs_extra = minimizer_kwargs_extra
if not self.kwargs:
    # define kwargs, same as the original implementation, and add the following
    self.kwargs.update(self.kwargs_extra)
else:
    self.kwargs.update(self.kwargs_extra) # this is not necessary but just in case
# (or we can have the kwargs.updateI() line outside the if statement)
~~~ 

I am suggesting this because the current workaround to specify bounds twice in a single function call of ```dual_annealing``` seems silly and redundant, but again I understand if making such changes involves too much effort for this small issue. If that's the case, you can close the issue once the docstrings are changed.
> @dschmitz89 I understand changing the implementation comes with some difficulties. At least I think adding docstrings is worth it, since I spent many hours debugging this issue and I am sure someone else would do that in the future.
> 
> A few alternative ideas:
> 
> 1. Add a boolean option (something like `pass_bounds=False`) bounds are added to the user-defined kwargs when True. We can set it to False by default to maintain the backward compatibility.
> 2. Make additional option (`minimizer_kwargs_extra={}`) so that when this is used, these are strictly _additional_ kwargs as the original docstrings suggest, not the _only_ kwargs to be passed. If this is used only for updating the existing kwargs (whether user-defined or default), this is also backward compatible.
> 
> ```python
> self.kwargs = minimizer_kwargs
> self.kwargs_extra = minimizer_kwargs_extra
> if not self.kwargs:
>     # define kwargs, same as the original implementation, and add the following
>     self.kwargs.update(self.kwargs_extra)
> else:
>     self.kwargs.update(self.kwargs_extra) # this is not necessary but just in case
> # (or we can have the kwargs.updateI() line outside the if statement)
> ```
> 
> I am suggesting this because the current workaround to specify bounds twice in a single function call of `dual_annealing` seems silly and redundant, but again I understand if making such changes involves too much effort for this small issue. If that's the case, you can close the issue once the docstrings are changed.

I fully agree that the current API seems silly. In fact, `scipy.optimize` is not exactly an example of state of the art API design (3 different ways to specify bounds, arbitrary strings instead of enums just to name two). I have spent my fair share of time debugging API related issues as well .. class based optimizers would probably be the way improve things in the future but realistically, I do not see it happening. `scipy.optimize` is used a LOT so downstream users/libraries will likely resist changes.

Adding one more option makes this already complex API even harder to grasp though in my opinion. Slightly related question: did you encounter the same issue with `optimize.shgo` by any chance? It uses a similar API from what I know. Whatever we decide to do, it should be applied to `shgo` as well.

Resolving patch:
diff --git a/scipy/optimize/_dual_annealing.py b/scipy/optimize/_dual_annealing.py
--- a/scipy/optimize/_dual_annealing.py
+++ b/scipy/optimize/_dual_annealing.py
@@ -395,6 +395,9 @@ def __init__(self, search_bounds, func_wrapper, *args, **kwargs):
         self.func_wrapper = func_wrapper
         self.kwargs = kwargs
         self.jac = self.kwargs.get('jac', None)
+        self.hess = self.kwargs.get('hess', None)
+        self.hessp = self.kwargs.get('hessp', None)
+        self.kwargs.pop("args", None)
         self.minimizer = minimize
         bounds_list = list(zip(*search_bounds))
         self.lower = np.array(bounds_list[0])
@@ -411,10 +414,19 @@ def __init__(self, search_bounds, func_wrapper, *args, **kwargs):
                 'maxiter': ls_max_iter,
             }
             self.kwargs['bounds'] = list(zip(self.lower, self.upper))
-        elif callable(self.jac):
-            def wrapped_jac(x):
-                return self.jac(x, *args)
-            self.kwargs['jac'] = wrapped_jac
+        else:
+            if callable(self.jac):
+                def wrapped_jac(x):
+                    return self.jac(x, *args)
+                self.kwargs['jac'] = wrapped_jac
+            if callable(self.hess):
+                def wrapped_hess(x):
+                    return self.hess(x, *args)
+                self.kwargs['hess'] = wrapped_hess
+            if callable(self.hessp):
+                def wrapped_hessp(x, p):
+                    return self.hessp(x, p, *args)
+                self.kwargs['hessp'] = wrapped_hessp
 
     def local_search(self, x, e):
         # Run local search from the given x location where energy value is e
@@ -464,10 +476,15 @@ def dual_annealing(func, bounds, args=(), maxiter=1000,
     maxiter : int, optional
         The maximum number of global search iterations. Default value is 1000.
     minimizer_kwargs : dict, optional
-        Extra keyword arguments to be passed to the local minimizer
-        (`minimize`). Some important options could be:
-        ``method`` for the minimizer method to use and ``args`` for
-        objective function additional arguments.
+        Keyword arguments to be passed to the local minimizer
+        (`minimize`). An important option could be ``method`` for the minimizer
+        method to use.
+        If no keyword arguments are provided, the local minimizer defaults to
+        'L-BFGS-B' and uses the already supplied bounds. If `minimizer_kwargs`
+        is specified, then the dict must contain all parameters required to
+        control the local minimization. `args` is ignored in this dict, as it is
+        passed automatically. `bounds` is not automatically passed on to the
+        local minimizer as the method may not support them.
     initial_temp : float, optional
         The initial temperature, use higher values to facilitates a wider
         search of the energy landscape, allowing dual_annealing to escape


Test patch:
diff --git a/scipy/optimize/tests/test__dual_annealing.py b/scipy/optimize/tests/test__dual_annealing.py
--- a/scipy/optimize/tests/test__dual_annealing.py
+++ b/scipy/optimize/tests/test__dual_annealing.py
@@ -361,9 +361,11 @@ def func(x):
         assert_allclose(ret_bounds_list.fun, ret_bounds_class.fun, atol=1e-9)
         assert ret_bounds_list.nfev == ret_bounds_class.nfev
 
-    def test_callable_jac_with_args_gh11052(self):
+    def test_callable_jac_hess_with_args_gh11052(self):
         # dual_annealing used to fail when `jac` was callable and `args` were
         # used; check that this is resolved. Example is from gh-11052.
+
+        # extended to hess as part of closing gh20614
         rng = np.random.default_rng(94253637693657847462)
         def f(x, power):
             return np.sum(np.exp(x ** power))
@@ -371,9 +373,27 @@ def f(x, power):
         def jac(x, power):
             return np.exp(x ** power) * power * x ** (power - 1)
 
+        def hess(x, power):
+            # calculated using WolframAlpha as d^2/dx^2 e^(x^p)
+            return np.diag(
+                power * np.exp(x ** power) * x ** (power - 2) *
+                (power * x ** power + power - 1)
+            )
+
+        def hessp(x, p, power):
+            return hess(x, power) @ p
+
         res1 = dual_annealing(f, args=(2, ), bounds=[[0, 1], [0, 1]], seed=rng,
                               minimizer_kwargs=dict(method='L-BFGS-B'))
         res2 = dual_annealing(f, args=(2, ), bounds=[[0, 1], [0, 1]], seed=rng,
                               minimizer_kwargs=dict(method='L-BFGS-B',
                                                     jac=jac))
+        res3 = dual_annealing(f, args=(2, ), bounds=[[0, 1], [0, 1]], seed=rng,
+                              minimizer_kwargs=dict(method='newton-cg',
+                                                    jac=jac, hess=hess))
+        res4 = dual_annealing(f, args=(2, ), bounds=[[0, 1], [0, 1]], seed=rng,
+                              minimizer_kwargs=dict(method='newton-cg',
+                                                    jac=jac, hessp=hessp))
         assert_allclose(res1.fun, res2.fun, rtol=1e-6)
+        assert_allclose(res3.fun, res2.fun, rtol=1e-6)
+        assert_allclose(res4.fun, res2.fun, rtol=1e-6)