Repository: scipy/scipy

Issue:
Scipy.opimize.fmin_powell direc argument syntax unclear
There is no information on how the direc argument of fmin-powell is supposed to be entered. All the scipy documentation for fmin_powell says is

```
direc : ndarray, optional
           Initial direction set.
```

I thought that by giving direc=(0.1,0.1,1), I was telling it to start with step sizes of 0.1 for the first two fitting parameters and 1 for the third, which are needed in my case since the 3rd parameter is not sensitive to step sizes of 0.1. However, with this code it starts with 0.1 for all of the fitting parameters. If I try direc=(1,0.1,1), it uses an initial step of 1 for all parameters which destroys the fit, as the second parameter has a range of (0,1) and results in a division by zero if it ever goes negative (if the step is 0.1, it sees the bad negative if it hits it and steps back to positive). 

We need a clearer explanation of how this argument works.



Discussion:
I looked at the docs, and maybe `direc` is supposed to be an ndarray of direction vectors.  So in your example it might be like `direc=np.array([[0.1, 0, 0], [0, 0.1, 0], [0, 0, 1]])` or something.  I think that numpy broadcasting might be causing your code to effectively be like `direc=np.array([[0.1, 0.1, 0.1], [0.1, 0.1, 0.1], [1, 1, 1]])`.

That almost works. Actually takes `direc=([1,0,0],[0,0.1,0],[0,0,1])`. However it leaves open the question of exactly how this is arranged. What, for example, would `direc=([1,0.1,0],[1,0.1,0],[0,0,1])` do?

The `direc` argument is a sequence of directions.  So `direc=([1,0.1,0],[1,0.1,0],[0,0,1])` presumably would cause the algorithm to not try the default scheme of line-searching along each parameter of your objective function separately, but would instead start out by trying a direction that is a combination of parameters.  I'm not sure if the code expects `direc` to have length exactly equal to the number of parameters, or if it expects the `direc` vectors to span the parameter space; the default "direc := identity matrix" has both of those properties.

This should be clarified and improved documentation submitted. It is impossible to figure this out looking at the doc page on fmin_powell. If I knew the method for submitting updated documentation, I would go do it once I played with this is bit.

@Void2258: either submit it by editing the code ([see here](https://github.com/scipy/scipy/blob/master/HACKING.rst.txt#contributing-new-code) --- for DOC edits you can also use the "edit" buttons on this Github web site).
Or, write you suggested improved docstring piece here, and someone can check it in.

@Void2258: I was about to reopen the ticket, but @pv beat me to it.  I agree that the documentation should be improved!  We might as well keep this issue open until it _is_ improved.

> This should be clarified and improved documentation submitted.

Yes I also agree.  Right now it's pretty much impossible to understand what the `direc` argument means without reading the source code.

direc : ndarray, optional
    Initial fitting step and parameter order set as an NxN vector, where N is the number of fitting parameters in x0. Defaults to step size 1.0 fitting all parameters simultaneously `[[1]*N]*N`. To prevent initial consideration of values in a step or to change initial step size, set to 0 or desired step size in the Jth position in the Mth block, where J is the position in x0 and M is the desired evaluation step, with steps being evaluated in index order. Step size and ordering will change freely as minimization proceeds.

Not sure how to state examples cases in general for usage. Also there are not generally individual examples cases in the parameters section. This sounds very dense to me, probably not the best way to say it.

Sorry, to re-open this.  I would really prefer Void2258's documentation.  I've just used a single N-elements vector and it kind of worked, only in the special case where `(fx > fx2)` and `if t < 0.0` it breaks.  Which is difficult to understand without this thread.
@srtlg Would you mind opening a PR for this? It would also trigger a discussion that would improve even further.

Resolving patch:
diff --git a/scipy/optimize/optimize.py b/scipy/optimize/optimize.py
--- a/scipy/optimize/optimize.py
+++ b/scipy/optimize/optimize.py
@@ -2404,8 +2404,9 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
                 maxfun=None, full_output=0, disp=1, retall=0, callback=None,
                 direc=None):
     """
-    Minimize a function using modified Powell's method. This method
-    only uses function values, not derivatives.
+    Minimize a function using modified Powell's method.
+
+    This method only uses function values, not derivatives.
 
     Parameters
     ----------
@@ -2415,12 +2416,6 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
         Initial guess.
     args : tuple, optional
         Extra arguments passed to func.
-    callback : callable, optional
-        An optional user-supplied function, called after each
-        iteration.  Called as ``callback(xk)``, where ``xk`` is the
-        current parameter vector.
-    direc : ndarray, optional
-        Initial direction set.
     xtol : float, optional
         Line-search error tolerance.
     ftol : float, optional
@@ -2430,12 +2425,25 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
     maxfun : int, optional
         Maximum number of function evaluations to make.
     full_output : bool, optional
-        If True, fopt, xi, direc, iter, funcalls, and
-        warnflag are returned.
+        If True, ``fopt``, ``xi``, ``direc``, ``iter``, ``funcalls``, and
+        ``warnflag`` are returned.
     disp : bool, optional
         If True, print convergence messages.
     retall : bool, optional
         If True, return a list of the solution at each iteration.
+    callback : callable, optional
+        An optional user-supplied function, called after each
+        iteration.  Called as ``callback(xk)``, where ``xk`` is the
+        current parameter vector.
+    direc : ndarray, optional
+        Initial fitting step and parameter order set as an (N, N) array, where N
+        is the number of fitting parameters in `x0`.  Defaults to step size 1.0
+        fitting all parameters simultaneously (``np.ones((N, N))``).  To
+        prevent initial consideration of values in a step or to change initial
+        step size, set to 0 or desired step size in the Jth position in the Mth
+        block, where J is the position in `x0` and M is the desired evaluation
+        step, with steps being evaluated in index order.  Step size and ordering
+        will change freely as minimization proceeds.
 
     Returns
     -------
@@ -2459,7 +2467,7 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
     See also
     --------
     minimize: Interface to unconstrained minimization algorithms for
-        multivariate functions. See the 'Powell' `method` in particular.
+        multivariate functions. See the 'Powell' method in particular.
 
     Notes
     -----
@@ -2467,13 +2475,12 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
     a function of N variables. Powell's method is a conjugate
     direction method.
 
-    The algorithm has two loops. The outer loop
-    merely iterates over the inner loop. The inner loop minimizes
-    over each current direction in the direction set. At the end
-    of the inner loop, if certain conditions are met, the direction
-    that gave the largest decrease is dropped and replaced with
-    the difference between the current estimated x and the estimated
-    x from the beginning of the inner-loop.
+    The algorithm has two loops.  The outer loop merely iterates over the inner
+    loop. The inner loop minimizes over each current direction in the direction
+    set. At the end of the inner loop, if certain conditions are met, the
+    direction that gave the largest decrease is dropped and replaced with the
+    difference between the current estimated x and the estimated x from the
+    beginning of the inner-loop.
 
     The technical conditions for replacing the direction of greatest
     increase amount to checking that
@@ -2484,6 +2491,15 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
        fraction of the decrease in the function value from that iteration of
        the inner loop.
 
+    References
+    ----------
+    Powell M.J.D. (1964) An efficient method for finding the minimum of a
+    function of several variables without calculating derivatives,
+    Computer Journal, 7 (2):155-162.
+
+    Press W., Teukolsky S.A., Vetterling W.T., and Flannery B.P.:
+    Numerical Recipes (any edition), Cambridge University Press
+
     Examples
     --------
     >>> def f(x):
@@ -2499,15 +2515,6 @@ def fmin_powell(func, x0, args=(), xtol=1e-4, ftol=1e-4, maxiter=None,
     >>> minimum
     array(0.0)
 
-    References
-    ----------
-    Powell M.J.D. (1964) An efficient method for finding the minimum of a
-    function of several variables without calculating derivatives,
-    Computer Journal, 7 (2):155-162.
-
-    Press W., Teukolsky S.A., Vetterling W.T., and Flannery B.P.:
-    Numerical Recipes (any edition), Cambridge University Press
-
     """
     opts = {'xtol': xtol,
             'ftol': ftol,


Test patch: