Repository: scipy/scipy

Issue:
ENH: Nelder-Mead to optionally return final simplex
Add the return_simplex option to
scipy.optimize.optimize._minimize_neldermead to also return all nodes of
the final simplex and their function values.
This is helpful for error estimation, see e.g.
http://www.scholarpedia.org/article/Nelder-Mead_algorithm#Postprocessing_in_parameter_estimation_problems



Discussion:
Thanks!
I'm sure this needs a smoke test, some better names (what are "sim" and "fsim" and how are they different). Also a question of do we need a separate kwarg, or can we add these this to `retall`? 

Good points. For me it is perfect to just add it to `retall` (did not occur to me before, thanks for the hint). I also changed the `sim` label to `final_simplex` and `fsim` to `final_simplex_function_values`.
However, I do not really know where / how to add the smoketest, as I have not found any test with `retall` set to `True` so far. Should we just start one?

https://github.com/scipy/scipy/blob/master/scipy/optimize/tests/test_optimize.py#L220 looks like a possible place to me, but I'd ask @dlax.

final_simplex_function_values looks a bit excessive as a name.

If this is a _final_ result, this shouldn't go under the `retall` flag, which controls whether all iteration trials of the optimization are returned or not.

Concerning the test, another assertion within `test_neldermead` is certainly fine.

Hi @dlax, thanks for considering this request. Would it also make sense to include the whole error estimation procedure from the final simplex as detailed in the original Nelder--Mead paper? Or should I just add a flag such as ''return_final_simplex'' to return the final simplex?

I don't know about the "whole error estimation procedure" you're mentioning (actually, I'm not using much this algorithm myself). If you have a use case for something, you can go ahead.

About this final simplex information, I think you can just add them to `result` independently of the `retall` flag and that there's no need for another flag for this (after all, this does not imply any additional computation).

Also, I agree that `final_simplex_function_values` is a bit excessive. How about just `final_simplex` and `f_final_simplex`? Or maybe packing the two values into a tuple `result['final_simplex'] = (sim, fsim)`.


Resolving patch:
diff --git a/scipy/optimize/optimize.py b/scipy/optimize/optimize.py
--- a/scipy/optimize/optimize.py
+++ b/scipy/optimize/optimize.py
@@ -541,7 +541,7 @@ def _minimize_neldermead(func, x0, args=(), callback=None,
 
     result = OptimizeResult(fun=fval, nit=iterations, nfev=fcalls[0],
                             status=warnflag, success=(warnflag == 0),
-                            message=msg, x=x)
+                            message=msg, x=x, final_simplex=(sim, fsim))
     if retall:
         result['allvecs'] = allvecs
     return result


Test patch:
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -224,9 +224,9 @@ def test_neldermead(self):
                     'return_all': False}
             res = optimize.minimize(self.func, self.startparams, args=(),
                                     method='Nelder-mead', options=opts)
-            params, fopt, numiter, func_calls, warnflag = (
+            params, fopt, numiter, func_calls, warnflag, final_simplex = (
                     res['x'], res['fun'], res['nit'], res['nfev'],
-                    res['status'])
+                    res['status'], res['final_simplex'])
         else:
             retval = optimize.fmin(self.func, self.startparams,
                                         args=(), maxiter=self.maxiter,