Repository: scipy/scipy

Issue:
scipy.optimize maxiter option not working as expected
When minimizing a function through scipy.optimize and setting maxiter:n and disp:True as options, the program outputs Iterations: n+1. With no value it runs a maximum of 101 iterations, so I guess the default value is 100. Either there is something wrong with the printing or with computing the number of iterations. 

Either way one would expect that setting maxiter:n, one would see Iterations: i, where i <= n

Thank you for your work, you make my work much easier



Discussion:
Which "minimizing function" are you referring to exactly?

This is the actual code, I am solving a quadratic programming problem

``` python
def loss(r):
    return (np.dot(r.T, np.dot(H, r)) / 2 + np.dot(c, r) + c0)
def jac(r):
    return np.dot(r.T, H) + c
cons = {'type':'ineq', 'fun':lambda x: b - np.dot(A, x), 'jac':lambda x: -A}
opt = {'disp':True,'maxiter':203}
res_cons = optimize.minimize(loss, x0, jac = jac, constraints = cons, method = 'SLSQP', options = opt, tol = 1e-16)
```

Sorry for off-topic, but is `H` positive semi-definite? I think [cvxpy](https://github.com/cvxgrp/cvxpy) is better for these 'structurally convex' problems.

@argriffing I am actually using the cvxopt package to solve this, I was using the scipy.optimize.minimize to compare results.

Confirmed for `slsqp` and `trust-constr`. Didn't try others.
```
import numpy as np
import scipy.optimize
m, n = 100, 200
np.random.seed(0)
A = np.random.rand(m, n)
b = np.random.rand(m)
c = np.random.rand(n)
x0 = np.random.rand(n)
def loss(r):
    return (np.sum(r))
cons = {'type':'ineq', 'fun':lambda x: b - np.dot(A, x), 'jac':lambda x: -A}
opt = {'disp':True,'maxiter':5}
res_cons = scipy.optimize.minimize(loss, x0, constraints = cons, method = 'trust-constr', options = opt, tol = 1e-16)
```
```
The maximum number of function evaluations is exceeded.
Number of iterations: 6
```
@antonior92 Since you implemented `trust-constr` relatively recently (compared to this issue being posted) and it has the same behavior, I wonder if this was a conscious choice?
For `trust-constr`, I think this is happening because it stops when `niter > maxiter`. So it does count the iteration when the optimization process gets interrupted after the counter is increased, hence it exceeds the value by one. I imagine SLSQP have a similar implementation.

I wouldn't say this is a (completely) conscious choice. But I probably looked over this detail to make the implementation more convenient. Nevertheless, I do agree that is a very weird choice, and I agree that it might be a good idea to fix that (even though I think it is a quite low priority issue).




 
OK, we'll leave this open. Since this will be quite easy to fix for the Python-based solvers, I'm marking it as a good first issue. In addition to solving the off-by-one issue, a unit test should be added to confirm that all the solvers obey the iteration limit.
@mdhaber Hi. This issue cannot be closed by #10780 ?
@AtsushiSakai no, I don't think so because `SLSQP` and `TNC` still don't pass the test. Right? See the conversation and how  `SLSQP` and `TNC` are commented out in the test.

Resolving patch:
diff --git a/scipy/optimize/slsqp.py b/scipy/optimize/slsqp.py
--- a/scipy/optimize/slsqp.py
+++ b/scipy/optimize/slsqp.py
@@ -172,7 +172,7 @@ def fmin_slsqp(func, x0, eqcons=(), f_eqcons=None, ieqcons=(), f_ieqcons=None,
          6 : Singular matrix C in LSQ subproblem
          7 : Rank-deficient equality constraint subproblem HFTI
          8 : Positive directional derivative for linesearch
-         9 : Iteration limit exceeded
+         9 : Iteration limit reached
 
     Examples
     --------
@@ -242,7 +242,7 @@ def _minimize_slsqp(func, x0, args=(), jac=None, bounds=None,
         automatically.
     """
     _check_unknown_options(unknown_options)
-    iter = maxiter
+    iter = maxiter - 1
     acc = ftol
     epsilon = eps
 
@@ -305,7 +305,7 @@ def cjac(x, *args):
                    6: "Singular matrix C in LSQ subproblem",
                    7: "Rank-deficient equality constraint subproblem HFTI",
                    8: "Positive directional derivative for linesearch",
-                   9: "Iteration limit exceeded"}
+                   9: "Iteration limit reached"}
 
     # Transform x0 into an array.
     x = asfarray(x0).flatten()
diff --git a/scipy/optimize/tnc.py b/scipy/optimize/tnc.py
--- a/scipy/optimize/tnc.py
+++ b/scipy/optimize/tnc.py
@@ -266,7 +266,7 @@ def fmin_tnc(func, x0, fprime=None, args=(), approx_grad=0,
             'offset': offset,
             'mesg_num': mesg_num,
             'maxCGit': maxCGit,
-            'maxiter': maxfun,
+            'maxfun': maxfun,
             'eta': eta,
             'stepmx': stepmx,
             'accuracy': accuracy,
@@ -286,7 +286,8 @@ def _minimize_tnc(fun, x0, args=(), jac=None, bounds=None,
                   eps=1e-8, scale=None, offset=None, mesg_num=None,
                   maxCGit=-1, maxiter=None, eta=-1, stepmx=0, accuracy=0,
                   minfev=0, ftol=-1, xtol=-1, gtol=-1, rescale=-1, disp=False,
-                  callback=None, finite_diff_rel_step=None, **unknown_options):
+                  callback=None, finite_diff_rel_step=None, maxfun=None,
+                  **unknown_options):
     """
     Minimize a scalar function of one or more variables using a truncated
     Newton (TNC) algorithm.
@@ -311,9 +312,9 @@ def _minimize_tnc(fun, x0, args=(), jac=None, bounds=None,
         iteration. If maxCGit == 0, the direction chosen is
         -gradient if maxCGit < 0, maxCGit is set to
         max(1,min(50,n/2)). Defaults to -1.
-    maxiter : int
-        Maximum number of function evaluation. If None, `maxiter` is
-        set to max(100, 10*len(x0)). Defaults to None.
+    maxiter : int, optional
+        Maximum number of function evaluations. This keyword is deprecated
+        in favor of `maxfun`. Only if `maxfun` is None is this keyword used.
     eta : float
         Severity of the line search. If < 0 or > 1, set to 0.25.
         Defaults to -1.
@@ -350,9 +351,11 @@ def _minimize_tnc(fun, x0, args=(), jac=None, bounds=None,
         possibly adjusted to fit into the bounds. For ``method='3-point'``
         the sign of `h` is ignored. If None (default) then step is selected
         automatically.
+    maxfun : int
+        Maximum number of function evaluations. If None, `maxfun` is
+        set to max(100, 10*len(x0)). Defaults to None.
     """
     _check_unknown_options(unknown_options)
-    maxfun = maxiter
     fmin = minfev
     pgtol = gtol
 
@@ -407,7 +410,10 @@ def _minimize_tnc(fun, x0, args=(), jac=None, bounds=None,
         offset = array([])
 
     if maxfun is None:
-        maxfun = max(100, 10*len(x0))
+        if maxiter is not None:
+            maxfun = maxiter
+        else:
+            maxfun = max(100, 10*len(x0))
 
     rc, nf, nit, x = moduleTNC.minimize(func_and_grad, x0, low, up, scale,
                                         offset, messages, maxCGit, maxfun,


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -836,7 +836,7 @@ def routine(*a, **kw):
                 return optimize.minimize(*a, **kw)
 
             if method == 'tnc':
-                kwargs['options'] = dict(maxiter=100)
+                kwargs['options'] = dict(maxfun=100)
             else:
                 kwargs['options'] = dict(maxiter=5)
 
@@ -907,8 +907,7 @@ def cons(x):
         assert_allclose(res.x, np.array([0., 2, 5, 8])/3, atol=1e-12)
 
     @pytest.mark.parametrize('method', ['Nelder-Mead', 'Powell', 'CG', 'BFGS',
-                                        'Newton-CG', 'L-BFGS-B',
-                                        # 'TNC', 'SLSQP',
+                                        'Newton-CG', 'L-BFGS-B', 'SLSQP',
                                         'trust-constr', 'dogleg', 'trust-ncg',
                                         'trust-exact', 'trust-krylov'])
     def test_respect_maxiter(self, method):
@@ -932,11 +931,15 @@ def test_respect_maxiter(self, method):
             kwargs['hess'] = sf.hess
 
         sol = optimize.minimize(sf.fun, x0, **kwargs)
-        assert sol.nit <= MAXITER
+        assert sol.nit == MAXITER
         assert sol.nfev >= sf.nfev
         if hasattr(sol, 'njev'):
             assert sol.njev >= sf.ngev
 
+        # method specific tests
+        if method == 'SLSQP':
+            assert sol.status == 9  # Iteration limit reached
+
     def test_respect_maxiter_trust_constr_ineq_constraints(self):
         # special case of minimization with trust-constr and inequality
         # constraints to check maxiter limit is obeyed when using internal
diff --git a/scipy/optimize/tests/test_tnc.py b/scipy/optimize/tests/test_tnc.py
--- a/scipy/optimize/tests/test_tnc.py
+++ b/scipy/optimize/tests/test_tnc.py
@@ -22,7 +22,7 @@ class TestTnc(object):
     """
     def setup_method(self):
         # options for minimize
-        self.opts = {'disp': False, 'maxiter': 200}
+        self.opts = {'disp': False, 'maxfun': 200}
 
     # objective functions and Jacobian for each test
     def f1(self, x, a=100.0):