Repository: scipy/scipy

Issue:
trust-constr error when attempting to keep bound constrained solutions feasible
I encountered this issue when running some simulations with 1000s of calls to minimize(method='trust-constr'), where the initial starting points always satisfy all inequality (including bound) constraints, but are not necessarily good starting conditions and are close to boundaries. If you run this printing out x from objective function calls you will see the function gets evaluated at [1.23093548, -0.74322811] so this is not an issue with a tolerance on the bounds in the finite difference calculations.

#### Reproducing code example:


```
import numpy as np
import scipy.optimize as opt
of = lambda x: np.exp(x[0])*(4*x[0]**2 + 2*x[1]**2 + 4*x[0]*x[1] + 2*x[1] + 1)
nce = lambda x: x[0]**2 + x[1]
nci = lambda x: x[0]*x[1]
x0 = np.array((0.99, -0.99)) # This causes the bug - using a better starting position, [-1, 1], does not
nlcs = [opt.NonlinearConstraint(nci, -10, np.inf), opt.NonlinearConstraint(nce, 1, 1)]
bnds = opt.Bounds(lb = np.array((-1, -1)), ub = np.array((1, 1)), keep_feasible=True)
tst = opt.minimize(fun =  of, x0 = x0, method = 'trust-constr', bounds = bnds, constraints = nlcs)
```

#### Error message:
<!-- If any, paste the *full* error message inside a code block
as above (starting from line Traceback)
-->

```
Traceback (most recent call last):
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-e67f037b5be0>", line 9, in <module>
    tst = opt.minimize(fun =  of, x0 = x0, method = 'trust-constr', bounds = bnds, constraints = nlcs)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 612, in minimize
    callback=callback, **options)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py", line 519, in _minimize_trustregion_constr
    factorization_method)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_trustregion_constr/tr_interior_point.py", line 329, in tr_interior_point
    factorization_method, trust_lb, trust_ub, subprob.scaling)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_trustregion_constr/equality_constrained_sqp.py", line 148, in equality_constrained_sqp
    f_next, b_next = fun_and_constr(x_next)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_trustregion_constr/tr_interior_point.py", line 83, in function_and_constraints
    f = self.fun(x)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 179, in fun
    self._update_x_impl(x)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 153, in update_x
    self._update_hess()
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 174, in _update_hess
    self._update_hess_impl()
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 138, in update_hess
    self._update_grad()
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 169, in _update_grad
    self._update_grad_impl()
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 90, in update_grad
    **finite_diff_options)
  File "/Users/dom/anaconda3/lib/python3.7/site-packages/scipy/optimize/_numdiff.py", line 362, in approx_derivative
    raise ValueError("`x0` violates bound constraints.")
ValueError: `x0` violates bound constraints.
  ...
```

#### Scipy/Numpy/Python version information:
<!-- You can simply run the following and paste the result in a code block
```
1.3.1 1.17.2 sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
```
-->


Discussion:
pinging @antonior92 who may have a better idea of what's going on here.
Thank you @dpoerio for spotting this problem.

At some point, the function "of'' is evaluated outside of the region. The following experiment show this happening:
```
import numpy as np
import scipy.optimize as opt
def of(x):
    print(x)
    return np.exp(x[0])*(4*x[0]**2 + 2*x[1]**2 + 4*x[0]*x[1] + 2*x[1] + 1)
nce = lambda x: x[0]**2 + x[1]
nci = lambda x: x[0]*x[1]
x0 = np.array((0.99, -0.99)) # This causes the bug - using a better starting position, [-1, 1], does not
nlcs = [opt.NonlinearConstraint(nci, -10, np.inf), opt.NonlinearConstraint(nce, 1, 1)]
bnds = opt.Bounds(lb = np.array((-1, -1)), ub = np.array((1, 1)), keep_feasible=True)
tst = opt.minimize(fun =  of, x0 = x0, method = 'trust-constr', bounds = bnds, constraints = nlcs, options={'verbose':2})
```

OUTPUT:
```
[ 0.99 -0.99]
[ 0.99000001 -0.99      ]
[ 0.99       -0.99000001]
| niter |f evals|CG iter|  obj func   |tr radius |   opt    |  c viol  |
|-------|-------|-------|-------------|----------|----------|----------|
|   1   |   3   |   0   | +2.6379e+00 | 1.00e+00 | 1.30e-03 | 1.01e+00 |
[ 0.97836707 -0.95143086]
[ 0.97836708 -0.95143086]
[ 0.97836707 -0.95143088]
|   2   |   6   |   1   | +2.6947e+00 | 7.00e+00 | 5.69e-03 | 9.94e-01 |
[ 1.23093548 -0.74322811]
```
-----------

I think the error is caused by the combination: approximating the gradient using finite differences and the hessian using quasi-Newton. Looking at the error message I see that :
1. The ``VectorFunction``, when faced with a new point, calls ``_update_x_impl()``:
https://github.com/scipy/scipy/blob/27a5729600854d8ed05cbbd6c68d70f9451c1793/scipy/optimize/_differentiable_functions.py#L181
2. And inside ``_update_x_impl()`` it tries to update the hessian by calling``_update_hess()``.
3. And inside ``_update_hess()``, since it is using a Quasi-newton approximation, it calls ``self._update_grad()``
4.  Finally,  it to try to compute the gradient at this point using numerical differentiation. Since the point ``x = [ 1.23093548 -0.74322811]`` contradicts the bounds that are being passed to the numeric differentiation function, it to throw the ``ValueError``.

-------

Summarizing, the method ``trust-constr``, by itself, is implemented in a way that, when faced with a point outside the bounds, it will just go back and try a new point closer to the current iteration. The problem here is that the function implemented in ``_differentiable_functions.py``
when given a new point try to update the quasi-Newton approximation.  And, in order to do that, the gradient computation is required. And, it fails to evaluate the new point due to how numerical differentiation for computing the gradient is implemented

--------

I think the solution is to include something that avoids the gradient being computed for quasi-Newton Hessian approximation for points outside the bounds. Maybe change:

```
        elif isinstance(hess, HessianUpdateStrategy):
            self.H = hess
            self.H.initialize(self.n, 'hess')
            self.H_updated = True
            self.x_prev = None
            self.g_prev = None

            def update_hess():
                self._update_grad()
                self.H.update(self.x - self.x_prev, self.g - self.g_prev)

        self._update_hess_impl = update_hess
```
to:


```
        elif isinstance(hess, HessianUpdateStrategy):
            self.H = hess
            self.H.initialize(self.n, 'hess')
            self.H_updated = True
            self.x_prev = None
            self.g_prev = None

            def update_hess():
                if ``x inside bounds``:
                    self._update_grad()
                    self.H.update(self.x - self.x_prev, self.g - self.g_prev)

        self._update_hess_impl = update_hess
```
https://github.com/scipy/scipy/blob/27a5729600854d8ed05cbbd6c68d70f9451c1793/scipy/optimize/_differentiable_functions.py#L132
@antonior92  This is more or less what I implemented in #11650 (along with some other details like returning inf for the objective function evaluation in this case to shrink the trust region).
That is great! Thank you for that @dpoerio 

I will help reviewing the pull request.

Resolving patch:
diff --git a/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py b/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py
--- a/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py
+++ b/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py
@@ -517,7 +517,7 @@ def grad_and_jac(x):
             xtol, state, initial_barrier_parameter,
             initial_barrier_tolerance,
             initial_constr_penalty, initial_tr_radius,
-            factorization_method)
+            factorization_method, finite_diff_bounds)
 
     # Status 3 occurs when the callback function requests termination,
     # this is assumed to not be a success.
diff --git a/scipy/optimize/_trustregion_constr/tr_interior_point.py b/scipy/optimize/_trustregion_constr/tr_interior_point.py
--- a/scipy/optimize/_trustregion_constr/tr_interior_point.py
+++ b/scipy/optimize/_trustregion_constr/tr_interior_point.py
@@ -32,7 +32,7 @@ def __init__(self, x0, s0, fun, grad, lagr_hess, n_vars, n_ineq, n_eq,
                  constr, jac, barrier_parameter, tolerance,
                  enforce_feasibility, global_stop_criteria,
                  xtol, fun0, grad0, constr_ineq0, jac_ineq0, constr_eq0,
-                 jac_eq0):
+                 jac_eq0, finite_diff_bounds):
         # Store parameters
         self.n_vars = n_vars
         self.x0 = x0
@@ -54,6 +54,8 @@ def __init__(self, x0, s0, fun, grad, lagr_hess, n_vars, n_ineq, n_eq,
         self.constr0 = self._compute_constr(constr_ineq0, constr_eq0, s0)
         self.jac0 = self._compute_jacobian(jac_eq0, jac_ineq0, s0)
         self.terminate = False
+        self.lb = finite_diff_bounds[0]
+        self.ub = finite_diff_bounds[1]
 
     def update(self, barrier_parameter, tolerance):
         self.barrier_parameter = barrier_parameter
@@ -78,9 +80,19 @@ def function_and_constraints(self, z):
         # Get variables and slack variables
         x = self.get_variables(z)
         s = self.get_slack(z)
-        # Compute function and constraints
-        f = self.fun(x)
-        c_eq, c_ineq = self.constr(x)
+        # Compute function and constraints,
+        # making sure x is within any strict bounds
+        if np.any((x < self.lb) | (x > self.ub)):
+            # If x is out of the strict bounds, set f = inf,
+            # and just set both equality and inequality
+            # constraints to 0 since we can't evaluate
+            # them separately.
+            f = np.inf
+            c_eq = np.full(self.n_eq, 0)
+            c_ineq = np.full(self.n_ineq, 0)
+        else:
+            f = self.fun(x)
+            c_eq, c_ineq = self.constr(x)
         # Return objective function and constraints
         return (self._compute_function(f, c_ineq, s),
                 self._compute_constr(c_ineq, c_eq, s))
@@ -272,7 +284,8 @@ def tr_interior_point(fun, grad, lagr_hess, n_vars, n_ineq, n_eq,
                       initial_tolerance,
                       initial_penalty,
                       initial_trust_radius,
-                      factorization_method):
+                      factorization_method,
+                      finite_diff_bounds):
     """Trust-region interior points method.
 
     Solve problem:
@@ -305,7 +318,7 @@ def tr_interior_point(fun, grad, lagr_hess, n_vars, n_ineq, n_eq,
         x0, s0, fun, grad, lagr_hess, n_vars, n_ineq, n_eq, constr, jac,
         barrier_parameter, tolerance, enforce_feasibility,
         stop_criteria, xtol, fun0, grad0, constr_ineq0, jac_ineq0,
-        constr_eq0, jac_eq0)
+        constr_eq0, jac_eq0, finite_diff_bounds)
     # Define initial parameter for the first iteration.
     z = np.hstack((x0, s0))
     fun0_subprob, constr0_subprob = subprob.fun0, subprob.constr0


Test patch:
diff --git a/scipy/optimize/tests/test_minimize_constrained.py b/scipy/optimize/tests/test_minimize_constrained.py
--- a/scipy/optimize/tests/test_minimize_constrained.py
+++ b/scipy/optimize/tests/test_minimize_constrained.py
@@ -679,6 +679,34 @@ def opt(x):
     minimize(opt, 2*[1], constraints = lin_cons)  # just checking that there are no errors
 
 
+def test_gh11649():
+    bnds = Bounds(lb=[-1, -1], ub=[1, 1], keep_feasible=True)
+
+    def assert_inbounds(x):
+        assert np.all(x >= bnds.lb)
+        assert np.all(x <= bnds.ub)
+
+    def obj(x):
+        assert_inbounds(x)
+        return np.exp(x[0])*(4*x[0]**2 + 2*x[1]**2 + 4*x[0]*x[1] + 2*x[1] + 1)
+
+    def nce(x):
+        assert_inbounds(x)
+        return x[0]**2 + x[1]
+
+    def nci(x):
+        assert_inbounds(x)
+        return x[0]*x[1]
+
+    x0 = np.array((0.99, -0.99))
+    nlcs = [NonlinearConstraint(nci, -10, np.inf),
+            NonlinearConstraint(nce, 1, 1)]
+
+    res = minimize(fun=obj, x0=x0, method='trust-constr',
+                   bounds=bnds, constraints=nlcs)
+    assert res.success
+
+
 class TestBoundedNelderMead:
 
     @pytest.mark.parametrize('bounds, x_opt',