Repository: scipy/scipy

Issue:
BUG: Scipy Optimize with Nelder-Mead method has issues when specifying initial guess as upper bound value
### Describe your issue.

Hi,
      I ran across the following peculiar issue with scipy optimize using Nelder Mead with bounds . I believe a code snippet will be more useful here:


Now we know that minimum is indeed when x = 0 . If we initialize Nelder Mead method above, with max bound value = initial guess, the algorithm returns 4 as the optimum value. However , if we change either the max bound value or initial guess then it correctly converges. I am not sure why we are getting this peculiar case and why it seems to be specific to Nelder_Mead case. This can ,be depending on context, a benign to a fairly insidious error (in my case as I am trying to optimize wear on a wall lining and hence initial value would be equal to max known wall thickness)

Please let me know if you can reproduce the above error and whether it can be fixed readily.
The work around for me rn is to just increase the bounds by a small epsilon

### Reproducing Code Example

```python
import numpy as np
from scipy.optimize import minimize
def d(x):
    return x**2-3


w = minimize(d,np.array([4]),bounds=[[0,4]],method="Nelder-Mead")
```


### Error message

```shell
N/A as  it's a logical bug
```


### SciPy/NumPy/Python version and system information

```shell
1.11.1 1.24.1 sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SKYLAKEX MAX_THREADS=2
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.21.dev
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SKYLAKEX MAX_THREADS=2
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.21.dev
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.10.4
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.35
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  pythran:
    include directory: C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-gwuhujd5\overlay\Lib\site-packages/pythran
    version: 0.13.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-t4t9eut3\cp310-win_amd64\build\venv\Scripts\python.exe
  version: '3.10'
```



Discussion:
I'm able to reproduce this issue on SciPy version 1.12.0.

I spent some time looking at the code with a debugger. Here's what I found.

When Nelder-Mead starts optimizing this function, it needs a simplex or polytope which describes the search space it is looking at. If this simplex is not provided via the `initial_simplex` solver option, then it is created from x0 with N + 1 points, where N is the number of dimensions.

One point is x0. The remaining N points are x0, but varied in one of each of the N dimensions. It varies x0[i] by 5% upward if x0[i] is non zero, or by 0.00025 if it is.

This is the initial simplex that it chooses:

```
(Pdb) sim
array([[4. ],
       [4.2]])
```

Next, it tries to ensure that the simplex lies inside of the bounds. It does this by clipping all of the points within the simplex to the bounds.

https://github.com/scipy/scipy/blob/488db3d39dae9f65d61d6dc1f4bccdcea92321cd/scipy/optimize/_optimize.py#L755

This is the simplex after clipping:

```
(Pdb) sim
array([[4.],
       [4.]])
```

One of the termination conditions for Nelder-Mead is that the simplex that represents the search area has become too small and flat.

https://github.com/scipy/scipy/blob/488db3d39dae9f65d61d6dc1f4bccdcea92321cd/scipy/optimize/_optimize.py#L824-L828

More specifically, it checks if the first point within the simplex has every coordinate within `xatol` of every other point in the simplex, and that the function evaluated at these points varies by no more than `fatol`. Since the simplex has zero size, it passes both checks, and Nelder-Mead terminates.

I'm not sure how to fix this issue.
Wow thanks a lot for the debugging and going into details for this one. Some really nice insights you provided on how they form the initial simplex if unspecified and how that is potentially the problematic point.

As a **potential fix,** I imagine varying by **either +5% or -5%** conditionally for each element in the dimension should suffice. 
E.g if original_x[i] == clip_ub(x_varied) , i.e after clipping upper bound we get the same value in that dimension as before varying (+5%) then perhaps vary -5% (vary down)  . That should cater for the any **n dimensional real number** case (as each **element** in the array is 1D/scalar). In the case that if after lower clipping we get original_x[i] == clip_lb(x_varied) then **upper bound = lower bound** must be the situation and thus the algorithm would still work as expected 

Again this is just a pretty simplistic suggested approach and from the top of my head so it's definitely possible I might be missing something here. Let me know what you think
 
I think that's a smart suggestion. I think there's one additional corner case - what happens if a coordinate of x0 is 0 while against the upper bound?

Suppose you have a one variable optimization problem, with x0 = [0], upper bounds of [0], and lower bounds of [-inf].

In this case, the inital simplex it would choose is

```
[[0]
 [0.00025]]
```

Then those points get clipped to (-inf, 0)

```
[[0]
 [0]]
```

so you have a 0 size simplex again. You could fix this by negating the starting point.

----

Some other ideas I have about this, as an alternative to changing the behavior of this method:

- Warn the user if their initial simplex is smaller than the termination conditions, or set an abnormal termination condition.
- Warn the user if their x0 is exactly equal to the bounds.
This occurs for those entries in `x0` are equal to the upper bound. 

I think the preferred way will be [instead of clipping the simplex on L800](https://github.com/scipy/scipy/blob/main/scipy/optimize/_optimize.py#L800), perform a reflective correction, followed by a clip. e.g. if an upper bound is 4, then the process would be 4.25 --> 3.75. If the reflected value is less than the lower bound a final clip would make it inside the bound. Hopefully that would address all the corner cases here.



Resolving patch:
diff --git a/scipy/optimize/_optimize.py b/scipy/optimize/_optimize.py
--- a/scipy/optimize/_optimize.py
+++ b/scipy/optimize/_optimize.py
@@ -797,6 +797,15 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
             maxfun = np.inf
 
     if bounds is not None:
+        # The default simplex construction may make all entries (for a given
+        # parameter) greater than an upper bound if x0 is very close to the
+        # upper bound. If one simply clips the simplex to the bounds this could
+        # make the simplex entries degenerate. If that occurs reflect into the
+        # interior.
+        msk = sim > upper_bound
+        # reflect into the interior
+        sim = np.where(msk, 2*upper_bound - sim, sim)
+        # but make sure the reflection is no less than the lower_bound
         sim = np.clip(sim, lower_bound, upper_bound)
 
     one2np1 = list(range(1, N + 1))


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -499,6 +499,29 @@ def test_neldermead_initial_simplex_bad(self):
                               full_output=True, disp=False, retall=False,
                               initial_simplex=simplex)
 
+    def test_neldermead_x0_ub(self):
+        # checks whether minimisation occurs correctly for entries where
+        # x0 == ub
+        # gh19991
+        def quad(x):
+            return np.sum(x**2)
+
+        res = optimize.minimize(
+            quad,
+            [1],
+            bounds=[(0, 1.)],
+            method='nelder-mead'
+        )
+        assert_allclose(res.x, [0])
+
+        res = optimize.minimize(
+            quad,
+            [1, 2],
+            bounds=[(0, 1.), (1, 3.)],
+            method='nelder-mead'
+        )
+        assert_allclose(res.x, [0, 1])
+
     def test_ncg_negative_maxiter(self):
         # Regression test for gh-8241
         opts = {'maxiter': -1}