Repository: scipy/scipy

Issue:
BUG: L-BFGS-B does two more iterations than specified in maxiter
If the optimization does not finish early, L-BFGS-B runs 1 iteration more than requested.



### Reproducing code example:
```
import itertools
from scipy import optimize

def f(x):
  x, y = x
  return (1 - x)**2 + 100 * (y - x**2)**2  # Rosenbrock function

counter = itertools.count()

def c(x):
  print next(counter), x, f(x)

optimize.minimize(f, [0, 0], method='l-bfgs-b', callback=c,
                  options={'maxiter': 5})

# Output
0 [  1.74855120e-01  -8.73576809e-08] 0.774343468431
1 [ 0.26113558  0.04906611] 0.582499810584
2 [ 0.32394677  0.11056373] 0.460208907475
3 [ 0.4240506   0.16523825] 0.352977278528
4 [ 0.51688677  0.23667522] 0.326403357767
5 [ 0.52242281  0.26249422] 0.238961333065 <-- This should be the optimized function value
     nit: 6  <-- This is already one too many
 success: False
 message: 'STOP: TOTAL NO. of ITERATIONS EXCEEDS LIMIT'
       x: array([ 0.60246124,  0.35628737])
     fun: 0.16248886013148453 <-- But this is less than reported in the last callback.
     jac: array([ 0.81281454, -1.33443389])
    nfev: 30
  status: 1
```

### Scipy/Numpy/Python version information:
```
('0.15.1',
 '1.11.1',
 sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0))
```

The wrong logic is [here](https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/lbfgsb.py#L317). It is probably enough to increase the `n_iterations` counter (and call the callback!) **before** checking it against `maxiter`. And the check should be a `<=`, not a `<`.

BUG: L-BFGS-B does two more iterations than specified in maxiter
If the optimization does not finish early, L-BFGS-B runs 1 iteration more than requested.



### Reproducing code example:
```
import itertools
from scipy import optimize

def f(x):
  x, y = x
  return (1 - x)**2 + 100 * (y - x**2)**2  # Rosenbrock function

counter = itertools.count()

def c(x):
  print next(counter), x, f(x)

optimize.minimize(f, [0, 0], method='l-bfgs-b', callback=c,
                  options={'maxiter': 5})

# Output
0 [  1.74855120e-01  -8.73576809e-08] 0.774343468431
1 [ 0.26113558  0.04906611] 0.582499810584
2 [ 0.32394677  0.11056373] 0.460208907475
3 [ 0.4240506   0.16523825] 0.352977278528
4 [ 0.51688677  0.23667522] 0.326403357767
5 [ 0.52242281  0.26249422] 0.238961333065 <-- This should be the optimized function value
     nit: 6  <-- This is already one too many
 success: False
 message: 'STOP: TOTAL NO. of ITERATIONS EXCEEDS LIMIT'
       x: array([ 0.60246124,  0.35628737])
     fun: 0.16248886013148453 <-- But this is less than reported in the last callback.
     jac: array([ 0.81281454, -1.33443389])
    nfev: 30
  status: 1
```

### Scipy/Numpy/Python version information:
```
('0.15.1',
 '1.11.1',
 sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0))
```

The wrong logic is [here](https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/lbfgsb.py#L317). It is probably enough to increase the `n_iterations` counter (and call the callback!) **before** checking it against `maxiter`. And the check should be a `<=`, not a `<`.



Discussion:
In case it was not clear above, the issues are that:
 * n + 2 iterations are done, and
 * the callback is called n + 1 times.
Any updates from the scipy maintainers?
I've had my eye on this one, but have had difficulty finding the time.
In case it was not clear above, the issues are that:
 * n + 2 iterations are done, and
 * the callback is called n + 1 times.
Any updates from the scipy maintainers?
I've had my eye on this one, but have had difficulty finding the time.

Resolving patch:
diff --git a/scipy/optimize/lbfgsb.py b/scipy/optimize/lbfgsb.py
--- a/scipy/optimize/lbfgsb.py
+++ b/scipy/optimize/lbfgsb.py
@@ -335,24 +335,22 @@ def func_and_grad(x):
             f, g = func_and_grad(x)
         elif task_str.startswith(b'NEW_X'):
             # new iteration
-            if n_iterations > maxiter:
-                task[:] = 'STOP: TOTAL NO. of ITERATIONS EXCEEDS LIMIT'
+            n_iterations += 1
+            if callback is not None:
+                callback(x)
+
+            if n_iterations >= maxiter:
+                task[:] = 'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
             elif n_function_evals[0] > maxfun:
                 task[:] = ('STOP: TOTAL NO. of f AND g EVALUATIONS '
                            'EXCEEDS LIMIT')
-            else:
-                n_iterations += 1
-                if callback is not None:
-                    callback(x)
         else:
             break
 
     task_str = task.tostring().strip(b'\x00').strip()
     if task_str.startswith(b'CONV'):
         warnflag = 0
-    elif n_function_evals[0] > maxfun:
-        warnflag = 1
-    elif n_iterations > maxiter:
+    elif n_function_evals[0] > maxfun or n_iterations >= maxiter:
         warnflag = 1
     else:
         warnflag = 2


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -573,6 +573,31 @@ def fun(x):
         assert_allclose(self.func(params), self.func(self.solution),
                         atol=1e-6)
 
+    def test_l_bfgs_b_maxiter(self):
+        # gh7854
+        # Ensure that not more than maxiters are ever run.
+        class Callback(object):
+            def __init__(self):
+                self.nit = 0
+                self.fun = None
+                self.x = None
+
+            def __call__(self, x):
+                self.x = x
+                self.fun = optimize.rosen(x)
+                self.nit += 1
+
+        c = Callback()
+        res = optimize.minimize(optimize.rosen, [0., 0.], method='l-bfgs-b',
+                                callback=c, options={'maxiter': 5})
+
+        assert_equal(res.nit, 5)
+        assert_almost_equal(res.x, c.x)
+        assert_almost_equal(res.fun, c.fun)
+        assert_equal(res.status, 1)
+        assert_(res.success is False)
+        assert_equal(res.message.decode(), 'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT')
+
     def test_minimize_l_bfgs_b(self):
         # Minimize with L-BFGS-B method
         opts = {'disp': False, 'maxiter': self.maxiter}