Repository: scipy/scipy

Issue:
BUG: Regression in 1.11.2: optimize.least_squares with method='trf' will sometimes fail if initial guess is chosen on boundary
### Describe your issue.

When I updated from 1.11.1 to 1.11.2, I noticed various errors from our test suite when using `optimize.curve_fit`. I could drill this down to problems when choosing an initial guess that shared points with the boundary conditions. Upon inspecting the release notes, I suspected this change in `optimize.least_squares` https://github.com/scipy/scipy/pull/18896 Indeed upon using `make_strictly_feasible` on our initial guess, all was good again.

### Proposed solution
I see the problem in https://github.com/scipy/scipy/issues/18793 as well. I would therefore suggest to use instead `make_strictly_feasible` with `rstep=0`, which defaults to moving from the boundary with a stepsize calculated from `np.nextafter` (one floating point difference). This prevents any problems with initial guesses on boundaries and reduces boundary effects to a minimum. I can make a PR if you want. 

### Background
Interior point optimization problems usually require strictly feasible points. This is outlined by the original author of this code here: https://nmayorov.wordpress.com/2015/06/19/trust-region-reflective-algorithm/ (step 3). 

### Reproducing Code Example

```python
import numpy as np
from scipy.optimize import curve_fit


def exponential(x, A, B, x0, y0):
    return A * np.exp(B * (x - x0)) + y0


ydata = np.array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                  1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 1.,
                  1., 1., 1., 0., 0., 0., 1., 0., 0., 2., 1.,
                  0., 3., 1., 6., 5., 0., 0., 2., 8., 4., 4.,
                  6., 9., 7., 2., 7., 8., 2., 13., 9., 8., 11.,
                  10., 13., 14., 19., 11., 15., 18., 26., 19., 32., 29.,
                  28., 36., 32., 35., 36., 43., 52., 32., 58., 56., 52.,
                  67., 53., 72., 88., 77., 95., 94., 84., 86., 101., 107.,
                  108., 118., 96., 115., 138., 137., 139., 126., 162., 188., 169.,
                  157., 197., 193., 185.])
xdata = np.arange(0, ydata.size) * 0.1
p0 = [0.01, 1., 5., 0.]
bounds = ((0.01, 0, 0, 0), (np.inf, 10, 20.9, 0.5))
res = curve_fit(exponential, xdata, ydata, p0=p0, bounds=bounds)
print(res)
```


### Error message

```shell
/home/hielke/.local/lib/python3.9/site-packages/scipy/optimize/_lsq/trf.py:234: RuntimeWarning: divide by zero encountered in divide
  Delta = norm(x0 * scale_inv / v**0.5)
/home/hielke/.local/lib/python3.9/site-packages/scipy/optimize/_lsq/trf.py:234: RuntimeWarning: invalid value encountered in divide
  Delta = norm(x0 * scale_inv / v**0.5)
/home/hielke/.local/lib/python3.9/site-packages/scipy/optimize/_lsq/common.py:398: RuntimeWarning: invalid value encountered in cast
  return min_step, np.equal(steps, min_step) * np.sign(s).astype(int)
Traceback (most recent call last):
  File "/home/hielke/tmp/test_scipy.py", line 41, in <module>
    res = curve_fit(exponential, xdata, ydata, p0=p0, bounds=bounds)
  File "/home/hielke/.local/lib/python3.9/site-packages/scipy/optimize/_minpack_py.py", line 978, in curve_fit
    raise RuntimeError("Optimal parameters not found: " + res.message)
RuntimeError: Optimal parameters not found: The maximum number of function evaluations is exceeded.
```


### SciPy/NumPy/Python version and system information

```shell
1.11.2 1.24.0 sys.version_info(major=3, minor=9, micro=2, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.21.dev
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.21.dev
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.10.4
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.36
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  pythran:
    include directory: ../../tmp/pip-build-env-ucip512m/overlay/lib/python3.9/site-packages/pythran
    version: 0.13.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/python/cp39-cp39/bin/python
  version: '3.9'
```



Discussion:
I have the same issues after upgrading Scipy from v1.11.1 to v1.11.2.

/mambaforge/lib/python3.10/site-packages/scipy/optimize/_lsq/trf.py:234: RuntimeWarning: divide by zero encountered in divide
Delta = norm(x0 * scale_inv / v**0.5)
/mambaforge/lib/python3.10/site-packages/scipy/optimize/_lsq/trf.py:363: RuntimeWarning: invalid value encountered in scalar divide
alpha *= Delta / Delta_new
/mambaforge/lib/python3.10/site-packages/scipy/optimize/_lsq/common.py:49: RuntimeWarning: invalid value encountered in scalar divide
t2 = c / q

Hope this can be fixed quickly!
>  I can make a PR if you want.

Yes please, and you may want to ping @nmayorov if they are indeed the original code author. Try to add a regression test--I'd say it is pretty bad if such a test does not exist in our suite at the moment and the break is affecting multiple people.

If you need to change the test that was added with https://github.com/scipy/scipy/pull/18896, then please also ping folks involved with that decision: @ellieLitwack and @adrianusler 

Resolving patch:
diff --git a/scipy/optimize/_lsq/least_squares.py b/scipy/optimize/_lsq/least_squares.py
--- a/scipy/optimize/_lsq/least_squares.py
+++ b/scipy/optimize/_lsq/least_squares.py
@@ -12,7 +12,7 @@
 
 from .trf import trf
 from .dogbox import dogbox
-from .common import EPS, in_bounds
+from .common import EPS, in_bounds, make_strictly_feasible
 
 
 TERMINATION_MESSAGES = {
@@ -821,6 +821,9 @@ def least_squares(
 
     ftol, xtol, gtol = check_tolerance(ftol, xtol, gtol, method)
 
+    if method == 'trf':
+        x0 = make_strictly_feasible(x0, lb, ub, rstep=0)
+
     def fun_wrapped(x):
         return np.atleast_1d(fun(x, *args, **kwargs))
 


Test patch:
diff --git a/scipy/optimize/tests/test_least_squares.py b/scipy/optimize/tests/test_least_squares.py
--- a/scipy/optimize/tests/test_least_squares.py
+++ b/scipy/optimize/tests/test_least_squares.py
@@ -822,3 +822,31 @@ def chi2(x):
     # if we choose an initial condition that is close to the solution
     # we shouldn't return an answer that is further away from the solution
     assert_allclose(res.x, answer, atol=initial_guess-answer)
+
+
+def test_gh_19103():
+    # Checks that least_squares trf method selects a strictly feasible point,
+    # and thus succeeds instead of failing,
+    # when the initial guess is reported exactly at a boundary point.
+    # This is a reduced example from gh191303
+
+    ydata = np.array([0.] * 66 + [
+        1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 1.,
+        1., 1., 1., 0., 0., 0., 1., 0., 0., 2., 1.,
+        0., 3., 1., 6., 5., 0., 0., 2., 8., 4., 4.,
+        6., 9., 7., 2., 7., 8., 2., 13., 9., 8., 11.,
+        10., 13., 14., 19., 11., 15., 18., 26., 19., 32., 29.,
+        28., 36., 32., 35., 36., 43., 52., 32., 58., 56., 52.,
+        67., 53., 72., 88., 77., 95., 94., 84., 86., 101., 107.,
+        108., 118., 96., 115., 138., 137.,
+    ])
+    xdata = np.arange(0, ydata.size) * 0.1
+
+    def exponential_wrapped(params):
+        A, B, x0 = params
+        return A * np.exp(B * (xdata - x0)) - ydata
+
+    x0 = [0.01, 1., 5.]
+    bounds = ((0.01, 0, 0), (np.inf, 10, 20.9))
+    res = least_squares(exponential_wrapped, x0, method='trf', bounds=bounds)
+    assert res.success