Repository: scipy/scipy

Issue:
Differential_evolution not giving a jacobean
I'm using the <code>differential_evolution</code> algorithm in scipy to fit some data with various exponential functions convolved with gaussian functions - this in itself is not a problem, the function fits it well.

However, it is not giving the <code>jacobian</code> in the result dictionary (which I would like to use to calculate the errors on my fit constants), despite the fact that I have set <code>polish</code> (i.e. use scipy.optimize.minimize with the L-BFGS-B method to polish the best population member at the end) to <code>True</code>, and thus the documentation states it should give the <code>jacobian</code>. My function takes the gaussian width and any number of exponents, and is being fit like so:

<code>result = differential_evolution(exponentialfit, bounds, args=(avgspectra, c, fitfrom, errors, numcomponents, 1), tol=0.000000000001, disp=True, polish=True)</code>

Version of scipy being used is 0.15.1. Is this a bug, or have I done something obvious but wrong?



Discussion:
You are not mistaken.  What the code is doing and what the documentation states are slightly divergent. The docstring states that if the `polish` was employed, then OptimizeResult should contain the `jac` attribute. However, currently the [polishing code](https://github.com/scipy/scipy/blob/master/scipy/optimize/_differentialevolution.py#L552) only uses the output from the polishing step if the L-BFGS-B strictly improves the objective function. So in your example if the polishing step doesn't improve the minimisation then the `jac` attribute isn't filled out. 
There are two ways of approaching this:
1. Change the documentation to say that `jac` is only filled out if the polishing improves the objective function. (i.e. what the code actually does).
2. Change the code to fill out the `jac` attribute regardless.  What I'm not sure about with this option is what L-BFGS-B does if it starts a minimisation already very close to, if not already at, the minimum.

Do you find the polishing option useful?

Well, if it gives a reasonable jacobean for errors, yes.
On 9 Aug 2015 08:40, "Andrew Nelson" notifications@github.com wrote:

> You are not mistaken. What the code is doing and what the documentation
> states are slightly divergent. The docstring states that if the polish
> was employed, then OptimizeResult should contain the jac attribute.
> However, currently the polishing code
> https://github.com/scipy/scipy/blob/master/scipy/optimize/_differentialevolution.py#L552
> only uses the output from the polishing step if the L-BFGS-B strictly
> improves the objective function. So in your example if the polishing step
> doesn't improve the minimisation then the jac attribute isn't filled out.
> There are two ways of approaching this:
> 1. Change the documentation to say that jac is only filled out if the
>    polishing improves the objective function. (i.e. what the code actually
>    does).
> 2. Change the code to fill out the jac attribute regardless. What I'm
>    not sure about with this option is what L-BFGS-B does if it starts a
>    minimisation already very close to, if not already at, the minimum.
> 
> Do you find the polishing option useful?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scipy/scipy/issues/5121#issuecomment-129128597.

Would there be another way to get the covariance matrix for my data from
differential evolution without employing the jacobean from polish then?

Thanks,

Joe Beckwith
On 9 Aug 2015 09:11, "Joe Beckwith" jsbeckwith@gmail.com wrote:

> Well, if it gives a reasonable jacobean for errors, yes.
> On 9 Aug 2015 08:40, "Andrew Nelson" notifications@github.com wrote:
> 
> > You are not mistaken. What the code is doing and what the documentation
> > states are slightly divergent. The docstring states that if the polish
> > was employed, then OptimizeResult should contain the jac attribute.
> > However, currently the polishing code
> > https://github.com/scipy/scipy/blob/master/scipy/optimize/_differentialevolution.py#L552
> > only uses the output from the polishing step if the L-BFGS-B strictly
> > improves the objective function. So in your example if the polishing step
> > doesn't improve the minimisation then the jac attribute isn't filled
> > out.
> > There are two ways of approaching this:
> > 1. Change the documentation to say that jac is only filled out if the
> >    polishing improves the objective function. (i.e. what the code actually
> >    does).
> > 2. Change the code to fill out the jac attribute regardless. What I'm
> >    not sure about with this option is what L-BFGS-B does if it starts a
> >    minimisation already very close to, if not already at, the minimum.
> > 
> > Do you find the polishing option useful?
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/scipy/scipy/issues/5121#issuecomment-129128597.

> Would there be another way to get the covariance matrix for my data from
> differential evolution without employing the jacobean from polish then?

In general you'd be better off using `leastsq` if you're after a covariance
matrix.

I was under the impression that leastsq was only for fitting x vs y data (i.e. not a grid of data points as I have here). Also I don't think you can use weighted residuals for it?

> I was under the impression that leastsq was only for fitting x vs y data
> (i.e. not a grid of data points as I have here). Also I don't think you can
> use weighted residuals for it?

It's more generalised than you think. You can use it to fit N dimensional
surfaces if you want. You can also weight the residuals if you want to. The
only requirement is that you return a vector of (possibly weighted)
residuals.
For example you might want to fit a dataset with two independent variables
m and n: z=f (m,n,x) where x is the vector of parameters that you want to
find. If you have a total of 100 observed points (e.g. a 10 by 10 grid over
m and n), then the func you supply to leastsq should return an array that
is 100 points long. You pass in the independent variables in the `args`
parameter.

So you just return the weighted residuals?
On 10 Aug 2015 11:24, "Andrew Nelson" notifications@github.com wrote:

> > I was under the impression that leastsq was only for fitting x vs y data
> > (i.e. not a grid of data points as I have here). Also I don't think you can
> > use weighted residuals for it?
> 
> It's more generalised than you think. You can use it to fit N dimensional
> surfaces if you want. You can also weight the residuals if you want to. The
> only requirement is that you return a vector of (possibly weighted)
> residuals.
> For example you might want to fit a dataset with two independent variables
> m and n: z=f (m,n,x) where x is the vector of parameters that you want to
> find. If you have a total of 100 observed points (e.g. a 10 by 10 grid over
> m and n), then the func you supply to leastsq should return an array that
> is 100 points long. You pass in the independent variables in the `args`
> parameter.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scipy/scipy/issues/5121#issuecomment-129377820.

You just need to divide the residual by the standard deviation of each data
point.

No, I know what weighted residuals are, I meant you can give least squares
the weighted residuals as the output of your function and it works with
this?
On 10 Aug 2015 11:45, "Andrew Nelson" notifications@github.com wrote:

> You just need to divide the residual by the standard deviation of each data
> point.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scipy/scipy/issues/5121#issuecomment-129384864.

As I was under the impression one had to return the fitted data.
On 10 Aug 2015 11:47, "Joe Beckwith" jsbeckwith@gmail.com wrote:

> No, I know what weighted residuals are, I meant you can give least squares
> the weighted residuals as the output of your function and it works with
> this?
> On 10 Aug 2015 11:45, "Andrew Nelson" notifications@github.com wrote:
> 
> > You just need to divide the residual by the standard deviation of each
> > data
> > point.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/scipy/scipy/issues/5121#issuecomment-129384864.

Yes, you can return that from `func`
On 10 Aug 2015 7:48 pm, "JBeckwith" notifications@github.com wrote:

> No, I know what weighted residuals are, I meant you can give least squares
> the weighted residuals as the output of your function and it works with
> this?
> On 10 Aug 2015 11:45, "Andrew Nelson" notifications@github.com wrote:
> 
> > You just need to divide the residual by the standard deviation of each
> > data
> > point.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/scipy/scipy/issues/5121#issuecomment-129384864.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scipy/scipy/issues/5121#issuecomment-129385651.

Cool, thanks!
On 10 Aug 2015 11:53, "Andrew Nelson" notifications@github.com wrote:

> Yes, you can return that from `func`
> On 10 Aug 2015 7:48 pm, "JBeckwith" notifications@github.com wrote:
> 
> > No, I know what weighted residuals are, I meant you can give least
> > squares
> > the weighted residuals as the output of your function and it works with
> > this?
> > On 10 Aug 2015 11:45, "Andrew Nelson" notifications@github.com wrote:
> > 
> > > You just need to divide the residual by the standard deviation of each
> > > data
> > > point.
> > > 
> > > —
> > > Reply to this email directly or view it on GitHub
> > > https://github.com/scipy/scipy/issues/5121#issuecomment-129384864.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/scipy/scipy/issues/5121#issuecomment-129385651.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scipy/scipy/issues/5121#issuecomment-129388133.


Resolving patch:
diff --git a/scipy/optimize/_differentialevolution.py b/scipy/optimize/_differentialevolution.py
--- a/scipy/optimize/_differentialevolution.py
+++ b/scipy/optimize/_differentialevolution.py
@@ -122,8 +122,9 @@ def differential_evolution(func, bounds, args=(), strategy='best1bin',
         Important attributes are: ``x`` the solution array, ``success`` a
         Boolean flag indicating if the optimizer exited successfully and
         ``message`` which describes the cause of the termination. See
-        `OptimizeResult` for a description of other attributes. If `polish`
-        was employed, then OptimizeResult also contains the `jac` attribute.
+        `OptimizeResult` for a description of other attributes.  If `polish`
+        was employed, and a lower minimum was obtained by the polishing, then
+        OptimizeResult also contains the ``jac`` attribute.
 
     Notes
     -----
@@ -446,9 +447,9 @@ def solve(self):
             Important attributes are: ``x`` the solution array, ``success`` a
             Boolean flag indicating if the optimizer exited successfully and
             ``message`` which describes the cause of the termination. See
-            `OptimizeResult` for a description of other attributes. If polish
-            was employed, then OptimizeResult also contains the ``hess_inv`` and
-            ``jac`` attributes.
+            `OptimizeResult` for a description of other attributes.  If `polish`
+            was employed, and a lower minimum was obtained by the polishing,
+            then OptimizeResult also contains the ``jac`` attribute.
         """
 
         nfev, nit, warning_flag = 0, 0, False


Test patch: