Repository: scipy/scipy

Issue:
Optimize raises "ValueError: `x0` violates bound constraints" for parameters that are within bounds
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

[My related Stack Overflow question](https://stackoverflow.com/questions/57615235/scipy-optimize-raises-valueerror-despite-x0-being-within-bounds), unanswered

Hi, I have been running into a recurring issue for a while where the scipy optimize function throws a ValueError during fitting, claiming that x0 violates the bound constraints, despite them being valid and within bounds at the time optimize is called. It has been suggested to me that this error is possibly being raised due to x0 being changed during the calculation of the Jacobian, but I am not positive about the behind-the-scenes working of optimize. 

My current workaround is going in and toggling the bounds a bit for the `k` parameter, but this has to be done manually each time the code crashes until the full dataset is imported, and likely leading to variations and inconsistencies in fits that I would like to avoid. It seems that in many cases the "ideal" fit for the data is one that involves a `k` value higher than the bounds (since I am fitting a logistic function to data that sometimes includes sharp stair-steps, leading to an infinitely high `k` value), but I am forced to limit this to ensure reasonable convergence times. I believe that the optimize function might be trying to update the parameter value to one outside the bounds.

It appears that this error is OS or environment specific, as other Stack Overflow users were not encountering the error while running the reproduction example provided below, whereas it crashes and throws the given error for me 100% of the time. I am running on 64-bit Windows 10 Pro (Version 10.0.18363 Build 18363) and Python 3.6.6

#### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be rendered as a code block. 
-->

```python
import numpy as np
import scipy as sp
from scipy.special import expit, logit
import scipy.optimize

def f(x,x0,g,c,k):
    y = c*expit(k*10.*(x-x0)) + g*(1.-c)
    return y

#               x0                      g                       c                       k
p0 = np.array([8.841357069490852e-01, 4.492363462957287e-19, 5.547073496706608e-01, 7.435378446218519e+00])
bounds = np.array([[-1.,1.], [0.,1.], [0.,1.], [0.,20.]])
x = np.array([1.0, 1.0, 1.0, 1.0, 1.0, 0.8911796599834791, 1.0, 1.0, 1.0, 0.33232919909076103, 1.0])
y = np.array([0.999, 0.999, 0.999, 0.999, 0.999, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001])
s = np.array([0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9])

print([pval >= b[0] and pval <= b[1] for pval,b in zip(p0,bounds)])

fit,cov = sp.optimize.curve_fit(f,x,y,p0=p0,sigma=s,bounds=([b[0] for b in bounds],[b[1] for b in bounds]),method='dogbox',tr_solver='exact')

print(fit)
print(cov)
```

#### Error message:
<!-- If any, paste the *full* error message inside a code block
as above (starting from line Traceback)
-->

```
c:\Users\user\Documents\LogRegProj\bin>python optimize_error.py
[True, True, True, True]
Traceback (most recent call last):
  File "optimize_error.py", line 19, in <module>
    fit,cov = sp.optimize.curve_fit(f,x,y,p0=p0,sigma=s,bounds=([b[0] for b in bounds],[b[1] for b in bounds]),method='dogbox',tr_solver='exact')
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\scipy\optimize\minpack.py", line 775, in curve_fit
    **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\scipy\optimize\_lsq\least_squares.py", line 928, in least_squares
    tr_solver, tr_options, verbose)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\scipy\optimize\_lsq\dogbox.py", line 310, in dogbox
    J = jac(x, f)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\scipy\optimize\_lsq\least_squares.py", line 875, in jac_wrapped
    kwargs=kwargs, sparsity=jac_sparsity)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\scipy\optimize\_numdiff.py", line 362, in approx_derivative
    raise ValueError("`x0` violates bound constraints.")
ValueError: `x0` violates bound constraints.
```

#### Scipy/Numpy/Python version information:
<!-- You can simply run the following and paste the result in a code block
```
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
```
-->
scipy: 1.4.1 
numpy: 1.18.1 
sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)

Thanks in advance, as this error has been driving me in circles for months.


Discussion:
Unfortunately, I'm not able to reproduce it on my Windows machine or a Linux environment I can access, so I can't help track it down right now.
Thanks for your fast response! That is reasonable and seems to be the case with all others I have asked to try and reproduce this error.

Can you think of anything that would lead this to only be happening on my local machine, like a local cache that I could delete or anything along those lines that I could try? I tried reinstalling numpy/scipy with force-reinstall as well but the error persists. Happy to try just about anything at this point.
> Happy to try just about anything at this point.

If I were you, I would try tracking down the error by putting breakpoints in the code involved in the stack trace. How is your problem turned into an optimization problem? Does it seem to be a bug in `curve_fit` or the underlying optimizer? Are the values it's complaining about far out of bounds, or is it just a tolerance issue? 
This example works for me with numpy 1.17.4, scipy 1.5.0.dev0+cb42f2f, macOS 10.13, Python 3.7.
When I say that it works, I mean there are no problems.
Sorry that `curve_fit` gave this issue. Can you try with `optimize.least_squares` to do this minimisation?
I'm getting the same error when I run the code above. 

I arrived here because I'm experiencing a similar issue in a leastsquares optimization. In my own example one of the parameter values is just outside of the bounds passed to leastsquares (in the order of 1e-11 beyond the upper limit). When I change the `method` keyword argument to `method="trf"`, I do not get the error...

__Version info:__
Python 3.7.3 
scipy 1.2.1
numpy 1.18.2
Windows 10 Pro Version 1909 (Build 18363.720)
I came across this issue in google as I was having the same problem but with the SLSQP minimizer.  I was just using different initial guesses/seeds to work around it, however, I went back to some old code and found that it did not work, when previously it did.  The only thing that had changed was that I upgraded scipy to the version 1.5.0, and sure enough downgrading it to 1.4.1 made it work again.

However, when using the example code above I receive the same error for all the versions of scipy that I have tried (the latest versions of each major release from 1.1 onwards).  

I'm not sure how helpful these observations are, but I figured there might be the small possibility that it could help for isolating the cause of the issue.  Unfortunately my code that throws this error with 1.5.0 but not with 1.4.1 is a bit long and relies on data from an outside file, so I can't really post it here as an example.

Version Info:
Python 3.7.0
Jupyter notebook 5.6.0
Numpy 1.19.0
scipy - various versions mentioned
Windows 10 Education Version 1909 OS build 18363.900

After upgrading to 1.5.0, I'm experiencing a similar behavior with the SLSQP minimizer. When I reverse the upgrade back to 1.4.1 the issue disappears.

**Version info:**
Python 3.7.7
Numpy 1.18.5
scipy - various versions mentioned
Microsoft Windows 10 Enterprise 10.0.18362 Build 18362
With scipy 1.5.1 and Python 3.8.3 on Linux, I am experiencing a similar issue with the SLSQP minimizer as well.

For example, in this case, my initial solution is: `[-10.8623 -22.7164  11.3582 -22.7164]`
The bounds are: `Bounds(array([-260.3  , -260.3  , -123.159, -104.635]), array([ 12.35, -12.35,  12.35, -12.35]), keep_feasible=True)`

I don't get the error with scipy 1.4.1
For those using slsqp can you give an example which can reproduce the issue?
I bumped into this too, when upgrading from scipy 1.1 to 1.5.2. I'm on Windows, but it seems Linux users are seeing the same. 

> For those using slsqp can you give an example which can reproduce the issue?

@andyfaff, here is a simple example:

``` python
from scipy.optimize import minimize, Bounds
import numpy as np
import sys

working = True
while working:
    bounds = Bounds(np.array([0.1]), np.array([1.0]))
    n_inputs = len(bounds.lb)
    x0 = np.array(bounds.lb + (bounds.ub-bounds.lb) * np.random.random(n_inputs))
    try:
        minimize(lambda x: np.linalg.norm(x), x0, method='SLSQP', bounds=bounds)
        print('.', end='')
    except:
        ex = sys.exc_info()
        print('\nBang!', ex[0], ex[1])
        working = False
        
x0, bounds
```



numpy: '1.19.1' , scipy master

The SLSQP behaviour is an interesting one... I first of all modified `optimize._numdiff.approx_derivative` on L390 to print out `x0` and `lb`:

```
with np.printoptions(precision=40, floatmode='fixed'):
    print("AD: ", x0 < lb, x0, lb)
```

I then run the following code.
```
from scipy.optimize import minimize, Bounds
import numpy as np

bounds = Bounds(np.array([0.1]), np.array([1.0]))
n_inputs = len(bounds.lb)
x0 = np.array(bounds.lb + (bounds.ub-bounds.lb) * np.random.random(n_inputs))
f = lambda x: np.linalg.norm(x)

with np.printoptions(precision=40, floatmode='fixed'):
        print(x0 < bounds.lb, x0, bounds.lb)

minimize(f, x0, method='SLSQP', bounds=bounds)
```

I get the following:

```
BEFORE: [False] [0.9426587714197141032457238907227292656898] [0.1000000000000000055511151231257827021182]
AD:  [False] [0.9426587714197141032457238907227292656898] [0.1000000000000000055511151231257827021182]
AD:  [ True] [0.0999999999999999777955395074968691915274] [0.1000000000000000055511151231257827021182]
```
Followed by a "ValueError: `x0` violates bound constraints." being [raised here](https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L390)

You can see that the initial guess is inside the bounds, but then the SLSQP code is requesting a gradient for `x0` in a location that is below the lower bound by 2 ULP.

pinging @pv for advice because he's examined the guts of SLSQP before.

[EDITED: the example was using random initialisation, so I shouldn't necessarily have expected it to fail everytime]
@mdhaber this may be the example you are looking for in https://github.com/scipy/scipy/issues/3056#issuecomment-523307547.
Good memory!
Hello, 

I am experiencing a similar problem with optimize basinghoping using the TNC local minimiser. It used to work but I updated everything 2 months ago.
Here is the raised error:
```
File "...\scipy\optimize\_basinhopping.py", line 686, in basinhopping
    new_global_min = bh.one_cycle()
  File "...\scipy\optimize\_basinhopping.py", line 152, in one_cycle
    accept, minres = self._monte_carlo_step()
  File "...\scipy\optimize\_basinhopping.py", line 104, in _monte_carlo_step
    minres = self.minimizer(x_after_step)
  File "...\scipy\optimize\_basinhopping.py", line 284, in __call__
    return self.minimizer(self.func, x0, **self.kwargs)
  File "...\scipy\optimize\_minimize.py", line 621, in minimize
    **options)
  File "...\scipy\optimize\tnc.py", line 377, in _minimize_tnc
    bounds=new_bounds)
  File "...\scipy\optimize\optimize.py", line 262, in _prepare_scalar_function
    finite_diff_rel_step, bounds, epsilon=epsilon)
  File "...\scipy\optimize\_differentiable_functions.py", line 95, in __init__
    self._update_grad()
  File "...\scipy\optimize\_differentiable_functions.py", line 171, in _update_grad
    self._update_grad_impl()
  File "...\scipy\optimize\_differentiable_functions.py", line 92, in update_grad
    **finite_diff_options)
  File "...\scipy\optimize\_numdiff.py", line 391, in approx_derivative
    raise ValueError("`x0` violates bound constraints.")

ValueError: `x0` violates bound constraints.
```

Here is what I do:

```
import numpy as np
from scipy import optimize

def sinFun(xx,a,f,phi,b):
    return a*np.sin(xx*f+phi)+b

def fun2Mini(par,xx,data):
    amp ,freq,delay,offset = par
    model = sinFun(xx,amp,freq,delay,offset)
    return np.sum((model-data)**2)

x = np.linspace(0,5,500)
y = np.random.rand()*np.sin(x*(np.random.rand()-0.5)*10+(np.random.rand()-0.5)*2*np.pi)

args = ( x, y)    
p1 = [0,2,0.05*np.pi,0]
bound = [(0,1),(0,5),(0,np.pi),(-1,1)]
minimizer_args = dict(args = args,method='TNC',bounds=bound)
globalRes = optimize.basinhopping(fun2Mini,p1
                                  ,minimizer_kwargs=minimizer_args
                                  ,niter_success=200)
```
Removing the local minimizer lift the error. Down grading scipy and numpy also fix it.
Cheers,
If this is useful, I did this code that can be used to test the bug, I run it in 'Linux-5.3.0-1032-aws-x86_64-with-debian-buster-sid'
and 'Windows-10-10.0.18362-SP0' both with scipy 1.5.2 and numpy 1.19.0:

```py
import pandas as pd 
import numpy as np
from scipy.optimize import minimize, LinearConstraint, Bounds


# this a portfolio optimization problem (a dummy one)
# The bounds are from 0 to 1 but I use -1e-8 as a value near to 0

bounds = Bounds([
    -1.e-08, 
    -1.e-08, 
    -1.e-08, 
    -1.e-08, 
    -1.e-08, 
    -1.e-08, 
    -1.e-08,
    -1.e-08, 
    -1.e-08], 
    [1.00000001, 
    1.00000001, 
    1.00000001, 
    1.00000001, 
    1.00000001,
    1.00000001, 
    1.00000001, 
    1.00000001, 
    1.00000001])

# this is the initial point, It's produced by "another process".
x0 = np.array([0.16383494, 0.1355977, 0.12064842, 0.12048256, 0.11785921, 0.10407136, 0.09826595, 0.07088149, 0.06835838])

# this is a previous set of weights generated by the "another process"
prev_x = np.array([0.16088057, 0.14069885, 0.12976232, 0.11833196, 0.11268556, 0.1008778, 0.10168758, 0.07268326, 0.])

# The sum of the weights must be equal to 1
A = np.array([np.array([1] * len(x0))])
lb = np.array([0.9])
ub = np.array([1.1])
constraints = [LinearConstraint(A, lb, ub)]

# Checking the initial point and that both function has the same results
print(np.all(A.dot(x0) <= ub))
print(np.all(A.dot(x0) >= lb))
print(np.all(x0 >= bounds.lb))
print(np.all(x0 <= bounds.ub))
print((x0 - prev_x).dot(x0 - prev_x) == np.sum((x0 - prev_x) ** 2))

# the objective function is a dummy transaction cost function

# This minimization work
a = minimize(
    fun=lambda x: 206.30358083892196 * np.sum((x - prev_x) ** 2),
    x0=x0,
    bounds=bounds,
    constraints=constraints,
    method='SLSQP',
)
print("Passed")

# but this one for some weird reason gave ValueError: `x0` violates bound constraints.
a = minimize(
    fun=lambda x: 206.30358083892196 * (x - prev_x).dot(x - prev_x),
    x0=x0,
    bounds=bounds,
    constraints=constraints,
    method='SLSQP',
)

print(a.x)
```
Same here with scipy v1.5.2 and `SLSQP`. When using `trust-constr` I don't get the `x0 violates bound constraints` error, but `x` is set outside bounds starting from the first iteration. When using `L-BFGS-B`, it works fine.
Additionally to the example above, I have set Bounds to `keep_feasible=True`. Still not working.
Also confirming that I'm getting this bug with SLSQP
Also confirmed from another project (scikit-hep/pyhf#1146). Happy to help debug this. 
@lukasheinrich (and other interested parties) in scipy 1.5 the underlying numerical differentiation function for the `minimize` methods (such as SLSQP), and `optimize.approx_fprime`, was changed to `scipy.optimize._numdiff.approx_derivative`. This is a much more robust and feature rich numerical differentiation routine than previously used.

One repeat issue was that the minimizers were asking the objective to be evaluated outside lower and upper bounds, and for many systems that's a big issue. This was occurring because the previous numerical differentiation code, when a value was at the very upper bound, would take a forward (differences) step above that upper bound. With 1.5 `approx_derivative` is supplied with these lower and upper bounds, and it does not take finite difference steps beyond either of the limits. For example a forward step is turned into a backwards step if up against an upper bound.

For this issue:
There are various minimizer codes (such as the SLSQP Fortran routine) that are responsible for generating steps towards a solution. They call back to Python to ask for function and gradient evaluation at new `x` locations. Unfortunately some of those locations seem to be *just* outside one or more of the bounds, we're talking one or two ULP in some cases. `approx_derivative` then complains that the `x` for which it's being asked to evaluate gradients is outside the limits. 

Note: the `keep_feasible` arguments for `optimize.Bounds` *only applies* to ***, it has no effect on `minimize` methods. This needs to be documented.

The best solution would be for the external codes (SLSQP a major culprit) to suggest steps that are strictly inside the lower/upper bounds.
I am not a Fortran expert, and am not able to delve into the code to try and fix it. If someone has the expertise, it'd be welcome. 
One possible fix is to clip the suggested`x` to be inside the bounds (around about [here](https://github.com/scipy/scipy/blob/master/scipy/optimize/slsqp.py#L421)), but that's not the cleanest fix.

EDITED to reflect @mdhaber comment.
> Note: the keep_feasible arguments for optimize.Bounds only applies to optimize.least_squares

I was under the impression it worked for trust-constr. I haven't checked in detail, but I see the `minimize_trustregion_constr` function mentioned `bounds.keep_feasible`. Is it worth a closer look?
You're probably right Matt.
@Teusia I don't think your example is really a bug. The `basinhopping` function is sending an initial `x0` to the TNC local minimizer that is outside the bounds you specify. It just so happens that the first piece of code that doesn't like that is the numerical differentiation code, with the error you see. The behaviour of TNC could be improved here. If `x0` is outside the `bounds` it should either be clipped to the bounds, or an Exception should be raised.
@andyfaff confirmed #13009 does fix our problem. Your explanation does sync up with my initial gut reaction (must be something with the steps) which was helpful. For reference, we see the warning pop out:

```
$ python toys_crash.py 
Background-like:  36%|████████████████████████████████████████████████████████████████▊                                                                                                                    | 179/500 [00:03<00:06, 48.58toy/s]/Users/kratsg/.pyenv/versions/pyhf-scipy/lib/python3.8/site-packages/scipy/optimize/optimize.py:275: RuntimeWarning: Values in x were outside bounds during an SLSQP step, clipping to bounds
  warnings.warn("Values in x were outside bounds during an "
```

Do you know if the fix will slow down the optimization routine at all because of the extra check that it does?
Something like a microsecond per slsqp iteration. It would be possible to clip regardless, but I thought it might be wise to warn if clipping is actually doing something.
@nmayorov, these users have a question about `least_squares`. The issue arises when a bounded `dogbox` problem wants to evaluate a Jacobian with numerical differentiation (`approx_derivative`). The `x` array that is sent to `approx_derivative` has entries that are outside the lower/upper bounds (see OP MWE). `trf_bounds` doesn't show the issue.

One difference between `trf_bounds` and `dogbox` is that trf employs a function called `make_strictly_feasible` which shifts a point to the interior of a feasible region, dogbox doesn't do that.

The generation of the point outside the bounds is going to be happening around about [L263](https://github.com/scipy/scipy/blob/master/scipy/optimize/_lsq/dogbox.py#L263). Could the fix be as simple as using `make_strictly_feasible`, or is there an underlying reason why the step generation doesn't obey the bounds that needs to be fixed?
Please note that the SLSQP portion of this issue is fixed by #13009
@LoaneShark, @dbrakenhoff, are you still experiencing issues with 'dogbox', either through `curve_fit` or `least_squares`? I do have a couple of routes that might fix the original issue that was reported, but I'm having problems generating an example that I can use for test purposes. 
I can't reproduce the error using @LoaneShark's example anymore. I haven't been working with "dogbox" much recently, so I don't know whether the issue still occurs for me... If I run into anything, I'll share it here.

My setup now is:
**Version info:**
Python 3.7.6
Numpy 1.19.2
scipy 1.5.2
Ubuntu 20.04.1 LTS

Resolving patch:
diff --git a/scipy/optimize/_lsq/dogbox.py b/scipy/optimize/_lsq/dogbox.py
--- a/scipy/optimize/_lsq/dogbox.py
+++ b/scipy/optimize/_lsq/dogbox.py
@@ -260,7 +260,9 @@ def dogbox(fun, jac, x0, f0, J0, lb, ub, ftol, xtol, gtol, max_nfev, x_scale,
             elif tr_solver == 'lsmr':
                 predicted_reduction = -evaluate_quadratic(Jop, g, step)
 
-            x_new = x + step
+            # gh11403 ensure that solution is fully within bounds.
+            x_new = np.clip(x + step, lb, ub)
+
             f_new = fun(x_new)
             nfev += 1
 


Test patch:
diff --git a/scipy/optimize/tests/test_slsqp.py b/scipy/optimize/tests/test_slsqp.py
--- a/scipy/optimize/tests/test_slsqp.py
+++ b/scipy/optimize/tests/test_slsqp.py
@@ -214,11 +214,11 @@ def test_minimize_bounded_constraint(self):
         # jacobian doesn't try to exceed that bound using a finite difference.
         # gh11403
         def c(x):
-            assert 0 <= x[0] <= 1 and 0 <= x[1] <=1, x
+            assert 0 <= x[0] <= 1 and 0 <= x[1] <= 1, x
             return x[0] ** 0.5 + x[1]
 
         def f(x):
-            assert 0 <= x[0] <= 1 and 0 <= x[1] <=1, x
+            assert 0 <= x[0] <= 1 and 0 <= x[1] <= 1, x
             return -x[0] ** 2 + x[1] ** 2
 
         cns = [NonlinearConstraint(c, 0, 1.5)]