Repository: scipy/scipy

Issue:
BUG: Regression in 1.11.3 can still fail for `optimize.least_squares` with `method='trf'`, maybe due to 0 lower bounds?
### Describe your issue.

Hi @tylerjereddy (thanks for all the maintenance work!)

This issue follows up on https://github.com/scipy/scipy/issues/19103, maybe also echoes https://github.com/scipy/scipy/issues/19309 (but the error message is not linked to division by zero).

Optimization with `optimize.least_squares` and `method='trf'` fails since `1.11.2`, and for some cases it was not fixed in `1.11.3` (but some were). We noticed this from our CI test suite in SciKit-GStat (https://github.com/mmaelicke/scikit-gstat).
I'm not sure what the origin is, maybe related to lower bounds equal to zero? 
Reproducible example below.

### Reproducing Code Example

Here's an example that:
- Passes with SciPy 1.11.1,
- Fails with SciPy 1.11.2,
- Still fails with SciPy 1.11.3 despite the fix in https://github.com/scipy/scipy/pull/19111,

```python
import numpy as np
from scipy.optimize import curve_fit
from functools import wraps
import math

def variogram(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        if hasattr(args[0], '__iter__'):
            new_args = args[1:]
            mapping = map(lambda h: func(h, *new_args, **kwargs), args[0])
            return np.fromiter(mapping, dtype=float)
        else:
            return func(*args, **kwargs)
    return wrapper

def spherical(h, r, c0, b=0):
    a = r / 1.

    if h <= r:
        return b + c0 * ((1.5 * (h / a)) - (0.5 * ((h / a) ** 3.0)))
    else:
        return b + c0

def gaussian(h, r, c0, b=0):
    # prepare parameters
    a = r / 2.

    return b + c0 * (1. - math.exp(- (h ** 2 / a ** 2)))

@variogram
def model_sum(h, r1, c1, r2, c2):

    return spherical(h, r1, c1) + gaussian(h, r2, c2)

f = model_sum

xdata = np.array([ 12.64307503,  25.28615006,  37.9292251 ,  50.57230013,
                   63.21537516,  75.85845019,  88.50152522, 101.14460026,
                   113.78767529, 126.43075032])
ydata = np.array([10.33885898, 20.33271591, 17.6010188 , 18.12232903, 19.60947486,
                  16.96256666, 17.08413617, 18.35997835, 13.34964295, 16.91325055])
p0 = np.array([126.43075032,  20.33271591, 126.43075032,  20.33271591])
bounds = (0, [126.43075032075254, 20.33271591029816, 126.43075032075254, 20.33271591029816])

curve_fit(f, xdata, ydata, p0=p0, bounds=bounds)
```

### Error message


As for previous issues on this topic, the error message is (in both 1.11.2 and 1.11.3):

```
RuntimeError: Optimal parameters not found: The maximum number of function evaluations is exceeded.
```
### SciPy/NumPy/Python version and system information

SciPy 1.11.1 to 1.11.3.

Thanks for looking into this!


Discussion:
I had a pretty good suspicion that the problem wasn't entirely fixed as you can see on the release discussion here: https://github.com/scipy/scipy/pull/19279#issuecomment-1737968436, and obviously from the issue you linked, which we knew about before I did the `1.11.3` release.

To get these issues squashed more decisively I probably need some domain expertise to chime in. I think we usually ping @nmayorov for this corner of `optimize`.

@mdhaber also has more knowledge of optimization algorithms than I do in general (pretty sure at least). Any thoughts on the sequence of patches below and the issues that remain open?

Patch 1: https://github.com/scipy/scipy/pull/18896
Patch 2: https://github.com/scipy/scipy/pull/19111
(both sets of regression tests passed when combined)

New issues that remain and/or popped up as a result of merging those patches:
https://github.com/scipy/scipy/issues/19309
https://github.com/scipy/scipy/issues/19351


Hmm thanks but unfortunately I don't know much about these algorithms, either. I would defer to @nmayorov, too.
I think the wisest decision is to restore the code prior to #18896 

The original complaint (about x not being exactly 0) I believe is not substantial or at least not critical. It is rather a small issue or a special property of the algorithm, certainly not a bug.

Maybe the docs could briefly explain the special property so folks don't think it is a bug?
I plan to create a PR with the revert + the test adjustment + small clarification of what to expect from the algorithms.

Resolving patch:
diff --git a/scipy/optimize/_lsq/least_squares.py b/scipy/optimize/_lsq/least_squares.py
--- a/scipy/optimize/_lsq/least_squares.py
+++ b/scipy/optimize/_lsq/least_squares.py
@@ -268,7 +268,9 @@ def least_squares(
         arguments, as shown at the end of the Examples section.
     x0 : array_like with shape (n,) or float
         Initial guess on independent variables. If float, it will be treated
-        as a 1-D array with one element.
+        as a 1-D array with one element. When `method` is 'trf', the initial
+        guess might be slightly adjusted to lie sufficiently within the given
+        `bounds`.
     jac : {'2-point', '3-point', 'cs', callable}, optional
         Method of computing the Jacobian matrix (an m-by-n matrix, where
         element (i, j) is the partial derivative of f[i] with respect to
@@ -822,7 +824,7 @@ def least_squares(
     ftol, xtol, gtol = check_tolerance(ftol, xtol, gtol, method)
 
     if method == 'trf':
-        x0 = make_strictly_feasible(x0, lb, ub, rstep=0)
+        x0 = make_strictly_feasible(x0, lb, ub)
 
     def fun_wrapped(x):
         return np.atleast_1d(fun(x, *args, **kwargs))


Test patch:
diff --git a/scipy/optimize/tests/test_least_squares.py b/scipy/optimize/tests/test_least_squares.py
--- a/scipy/optimize/tests/test_least_squares.py
+++ b/scipy/optimize/tests/test_least_squares.py
@@ -10,7 +10,7 @@
 
 from scipy.optimize import least_squares, Bounds
 from scipy.optimize._lsq.least_squares import IMPLEMENTED_LOSSES
-from scipy.optimize._lsq.common import EPS, make_strictly_feasible
+from scipy.optimize._lsq.common import EPS, make_strictly_feasible, CL_scaling_vector
 
 
 def fun_trivial(x, a=0):
@@ -811,17 +811,30 @@ def err(p, x, y):
     assert_allclose(res.x, np.array([0.4082241, 0.15530563]), atol=5e-5)
 
 
-def test_gh_18793():
+def test_gh_18793_and_19351():
     answer = 1e-12
     initial_guess = 1.1e-12
 
     def chi2(x):
         return (x-answer)**2
 
-    res = least_squares(chi2, x0=initial_guess, bounds=(0, np.inf))
+    gtol = 1e-15
+    res = least_squares(chi2, x0=initial_guess, gtol=1e-15, bounds=(0, np.inf))
+    # Original motivation: gh-18793
     # if we choose an initial condition that is close to the solution
     # we shouldn't return an answer that is further away from the solution
-    assert_allclose(res.x, answer, atol=initial_guess-answer)
+
+    # Update: gh-19351
+    # However this requirement does not go well with 'trf' algorithm logic.
+    # Some regressions were reported after the presumed fix.
+    # The returned solution is good as long as it satisfies the convergence
+    # conditions.
+    # Specifically in this case the scaled gradient will be sufficiently low.
+
+    scaling, _ = CL_scaling_vector(res.x, res.grad,
+                                   np.atleast_1d(0), np.atleast_1d(np.inf))
+    assert res.status == 1  # Converged by gradient
+    assert np.linalg.norm(res.grad * scaling, ord=np.inf) < gtol
 
 
 def test_gh_19103():