Repository: scipy/scipy

Issue:
default maxiter/maxfev in minimize_neldermead over rule user
scipy.optimize.minimize() with method='Nelder-Mead' has maxiter and maxfev options to end optimization early. If either of them is unset then it gets set to 200 times the number of input variables. For slow to converge functions this may not be enough. Currently if the user increases only one of these, then the optimization will stop early due to the default value of the other.

For example the following function requires about 4000 evaluations to minimize 

```
from numpy import sqrt, sin, arctan2
import scipy.optimize

def test_func(v):
    x,y = v
    r,t = sqrt(x**2+y**2), arctan2(x,y)
    a,b  = 20, 0.5
    pen = sin(r*a + t)+r*b
    return pen

print "1) opts {}"
res = scipy.optimize.minimize(test_func, [1,1], method='Nelder-Mead')
print res
print "\n2) opts {'maxiter':5000}"
res = scipy.optimize.minimize(test_func, [1,1], method='Nelder-Mead', options={"maxiter":5000})
print res
print "\n3) opts {'maxfev':5000}"
res = scipy.optimize.minimize(test_func, [1,1], method='Nelder-Mead', options={"maxfev":5000})
print res
print "\n4) opts {'maxiter':5000, 'maxfev':5000}"
res = scipy.optimize.minimize(test_func, [1,1], method='Nelder-Mead', options={"maxiter":5000, "maxfev":5000})
print res
```

Calls 2 and 3 will stop after 400 evaluations or iterations, even though the user has asked to increase the number. Only by increasing both can the function be minimized.

I propose that if the user specifies maxiter, then the function wont halt after any number of evaluations, and vice versa.

Patch to follow.



Discussion:
_minimize_powell() has the same issue. Once you are happy with the solution I shall make a patch for that as well (or I could put that in the same patch).

My guess is that people normally set a limit on `maxfun` before a limit on `maxiter`. This is because they often know how long it takes to perform one function evaluation. By limiting `maxfun` you are limiting the upper time bound on the minimisation.
I'd note here that the docstring does not specify what `maxfun` and `maxiter` default to, you have to read the code to do that. (as a minimum the docstring should be fixed).
It was suggested in #5968 that one could set `maxfun` and `maxiter` to `np.inf`. This way the minimization would keep going until it converged. Setting `maxfun` and `maxiter` would then put a clear specified limit on how long the minimization kept going for.
Perhaps that's not a bad idea, but are there any cases where the simplex doesn't converge? That would be bad because the minimisation would never stop.
I think any solutions proposed should be floated on the scipy-dev mailing list.

As it currently is if setting maxfun (maxfev) does the same thing, i.e. the 3rd example above stops after 400 iterations (731 function calls) even though the user specified maxfev=5000.

As suggested I made a simpler patch using np.inf https://github.com/samtygier/scipy/commit/bf1d89fc06e24aeb58e940a9bf1341595efc3ffd

With either patch if the user specifies neither then the minimization stops after N*200 iterations or function calls (which ever happens first). If the user specifies just 1 of maxfev or maxiter, then that becomes the only stopping condition. If they specify both, then it stops with which ever happens first. I think this is the least surprising behavior.

I could update the doc string to say what happens in the case of neither or both being specified.

One question is whether there is a safeguard inside each iteration to terminate. For example, is there a linesearch that could become endless if there are nans or other weird things.
(I wonder whether there was an algorithmic reason to add both criteria in the first place.)

My first thought was that this breaks backwards compatibility, If I use maxiter and maxfev in the sense of a minimum condition, stop if the tighter criterion is satisfied, then this will change the number of iterations.
On the other hand, when I'm using fmin I usually have to alternate with increasing maxiter and maxfev in many cases until I get convergence.
For statsmodels I recommend and use nelder mead as preliminary estimators when the starting values might be bad and usually limit with maxiter, because individual function evaluations are relatively cheap.. 

Nelder-Mead has no loop inside the iteration, but an iteration can make between 1 and 3 calls to the function. So there is no risk of infinite looping if only maxiter is set.
(I am not sure I can see a logical reason why some minimizers, offer maxiter, some maxfev, and some both.)

In terms of backwards compatibility:
If you have set both maxiter and maxfev, then the behavior is the same, it will stop if either limit is reached.
If you have set neither, then the behavior is the same, it will stop when either limit reaches N*200 (maxfev will be hit first).
If you have set just one of maxiter or maxfev, then previously the search could stop much before that limit, now it will honor that limit, resulting it a better chance of convergence, though with a longer run time. I'd consider this to be a bug fix, as the code now does what it was asked.

I noticed one issue with my second implementation ( https://github.com/samtygier/scipy/commit/bf1d89fc06e24aeb58e940a9bf1341595efc3ffd ). If you call it with maxiter or maxfev =None, then it will treat None as zero, and stop after the first iteration. I'll rework it so that the defaults in the definition are None, and set to np.inf when needed.

> I'd consider this to be a bug fix, as the code now does what it was asked.

Not the way I see it.

If I set maxfev to a large number because I don't care about it and just want maxiter to be binding, then this would change. I and statsmodels, AFAIR, don't have code like this and the new pattern sounds fine with me. (for example I don't think the change in the criteria is binding or affects the results in any of our current use.)

In contrast, I would be very upset if numpy allclose would get a similar change where if I set atol or rtol but not both the other one wouldn't use the default anymore.
or similarly, if precision in convergence criteria would switch from "or" to "and" or the other way around it would affect our test results.


Resolving patch:
diff --git a/scipy/optimize/optimize.py b/scipy/optimize/optimize.py
--- a/scipy/optimize/optimize.py
+++ b/scipy/optimize/optimize.py
@@ -415,10 +415,12 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
     -------
     disp : bool
         Set to True to print convergence messages.
-    maxiter : int
-        Maximum number of iterations to perform.
-    maxfev : int
-        Maximum number of function evaluations to make.
+    maxiter, maxfev : int
+        Maximum allowed number of iterations and function evaluations.
+        Will default to ``N*200``, where ``N`` is the number of
+        variables, if neither `maxiter` or `maxfev` is set. If both
+        `maxiter` and `maxfev` are set, minimization will stop at the
+        first reached.
     initial_simplex : array_like of shape (N + 1, N)
         Initial simplex. If given, overrides `x0`.
         ``initial_simplex[j,:]`` should contain the coordinates of
@@ -430,6 +432,7 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
     fatol : number, optional
         Absolute error in func(xopt) between iterations that is acceptable for
         convergence.
+
     """
     if 'ftol' in unknown_options:
         warnings.warn("ftol is deprecated for Nelder-Mead,"
@@ -490,10 +493,22 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
     if retall:
         allvecs = [sim[0]]
 
-    if maxiter is None:
+    # If neither are set, then set both to default
+    if maxiter is None and maxfun is None:
         maxiter = N * 200
-    if maxfun is None:
         maxfun = N * 200
+    elif maxiter is None:
+        # Convert remaining Nones, to np.inf, unless the other is np.inf, in
+        # which case use the default to avoid unbounded iteration
+        if maxfun == np.inf:
+            maxiter = N * 200
+        else:
+            maxiter = np.inf
+    elif maxfun is None:
+        if maxiter == np.inf:
+            maxfun = N * 200
+        else:
+            maxfun = np.inf
 
     one2np1 = list(range(1, N + 1))
     fsim = numpy.zeros((N + 1,), float)
@@ -2384,10 +2399,12 @@ def _minimize_powell(func, x0, args=(), callback=None,
         Relative error in solution `xopt` acceptable for convergence.
     ftol : float
         Relative error in ``fun(xopt)`` acceptable for convergence.
-    maxiter : int
-        Maximum number of iterations to perform.
-    maxfev : int
-        Maximum number of function evaluations to make.
+    maxiter, maxfev : int
+        Maximum allowed number of iterations and function evaluations.
+        Will default to ``N*1000``, where ``N`` is the number of
+        variables, if neither `maxiter` or `maxfev` is set. If both
+        `maxiter` and `maxfev` are set, minimization will stop at the
+        first reached.
     direc : ndarray
         Initial set of direction vectors for the Powell method.
 
@@ -2402,10 +2419,22 @@ def _minimize_powell(func, x0, args=(), callback=None,
     if retall:
         allvecs = [x]
     N = len(x)
-    if maxiter is None:
+    # If neither are set, then set both to default
+    if maxiter is None and maxfun is None:
         maxiter = N * 1000
-    if maxfun is None:
         maxfun = N * 1000
+    elif maxiter is None:
+        # Convert remaining Nones, to np.inf, unless the other is np.inf, in
+        # which case use the default to avoid unbounded iteration
+        if maxfun == np.inf:
+            maxiter = N * 1000
+        else:
+            maxiter = np.inf
+    elif maxfun is None:
+        if maxiter == np.inf:
+            maxfun = N * 1000
+        else:
+            maxfun = np.inf
 
     if direc is None:
         direc = eye(N, dtype=float)


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -1161,6 +1161,64 @@ def test_brute(self):
         assert_allclose(resbrute[1], self.func(self.solution, *self.params),
                         atol=1e-3)
 
+class TestIterationLimits(TestCase):
+    # Tests that optimisation does not give up before trying requested
+    # number of iterations or evaluations. And that it does not succeed
+    # by exceeding the limits.
+    def setUp(self):
+        self.funcalls = 0
+
+    def slow_func(self, v):
+        self.funcalls += 1
+        r,t = np.sqrt(v[0]**2+v[1]**2), np.arctan2(v[0],v[1])
+        return np.sin(r*20 + t)+r*0.5
+
+    def test_neldermead_limit(self):
+        self.check_limits("Nelder-Mead", 200)
+
+    def test_powell_limit(self):
+        self.check_limits("powell", 1000)
+
+    def check_limits(self, method, default_iters):
+        for start_v in [[0.1,0.1], [1,1], [2,2]]:
+            for mfev in [50, 500, 5000]:
+                self.funcalls = 0
+                res = optimize.minimize(self.slow_func, start_v,
+                      method=method, options={"maxfev":mfev})
+                assert_(self.funcalls == res["nfev"])
+                if res["success"]:
+                    assert_(res["nfev"] < mfev)
+                else:
+                    assert_(res["nfev"] >= mfev)
+            for mit in [50, 500,5000]:
+                res = optimize.minimize(self.slow_func, start_v,
+                      method=method, options={"maxiter":mit})
+                if res["success"]:
+                    assert_(res["nit"] <= mit)
+                else:
+                    assert_(res["nit"] >= mit)
+            for mfev,mit in [[50,50], [5000,5000],[5000,np.inf]]:
+                self.funcalls = 0
+                res = optimize.minimize(self.slow_func, start_v,
+                      method=method, options={"maxiter":mit, "maxfev":mfev})
+                assert_(self.funcalls == res["nfev"])
+                if res["success"]:
+                    assert_(res["nfev"] < mfev and res["nit"] <= mit)
+                else:
+                    assert_(res["nfev"] >= mfev or res["nit"] >= mit)
+            for mfev,mit in [[np.inf,None], [None,np.inf]]:
+                self.funcalls = 0
+                res = optimize.minimize(self.slow_func, start_v,
+                      method=method, options={"maxiter":mit, "maxfev":mfev})
+                assert_(self.funcalls == res["nfev"])
+                if res["success"]:
+                    if mfev is None:
+                        assert_(res["nfev"] < default_iters*2)
+                    else:
+                        assert_(res["nit"] <= default_iters*2)
+                else:
+                    assert_(res["nfev"] >= default_iters*2 or
+                        res["nit"] >= default_iters*2)
 
 if __name__ == "__main__":
     run_module_suite()