Repository: lmfit/lmfit-py

Issue:
Add emcee
## Submission checklist:
##  1. Follow PEP-8 as closely as possible
Yes, except for line lengths. Also removed unrelated unused variable from previous author to make more pep-8 compliant.
##  2. Reference Issue #
N/A
##  3. Add a test or explain why this is not needed.
Doesn't change any existing test coverage (i.e. only adds a new way to access existing code)

##  4. Consider if an example is needed.
```
fitter = Minimizer(fcn, params, fcn_args, fcn_kws, **fit_kws)
fitter.minimize(method='emcee')
```
##  5. Include changes to main documentation if the API changes or a new feature is added.
Updated docstrings accordingly.

# Describe changes here

There are several class methods of Minimizer that return a MinimizerResult object, and set the method attribute of that object. Currently, `Minimizer.emcee` is the only one that is not accessible by passing the `method` kwarg to `Minimizer.minimize`.

The reason why `'emcee'` should be added as an option for the `method` kwarg to `Minimizer.minimize` is because when you want to run a fit on a `Model` object by calling `Model.fit`, that calls `Minimizer.minimize` under the hood. There is no way to run emcee directly on a `Model` class (since there is no transparent access to the `Minimizer` class from the `Model` object) without first running any other fit method and then calling the emcee method of `ModelResult` object.

Note: MCMC sampling process is not strictly a minimization routine, but under certain constraints the maximum likelihood estimate should be identical to the output of a least-squares routine that minimizes Chi-square. In other words, it is possible to use emcee to "fit" to the model in addition to just calculating better parameter covariances.

# Describe changes to tests and/or docs here
Updated docstrings to show the expanded cabability.


Discussion:
@FaustinCarter Thanks!  I think this is very useful.   There should probably be a test that using `minimize(..., method='emcee',...)` works for an easy test case.   Maybe that could also be encapsulated in an example?

@reneeotten Even though I tagged 0.9.12.rc1, I would suggest that this PR should probably be included as 0.9.12rc2, and then we release 0.9.12  a week after this is merged.   Does that seem OK to you?
I'll try to have one more PR for doc/info/license files after this.  



@newville I think that the code in `examples/doc_fitting_emcee.py` could be modified (and simplified) to include this as a test. Specifically, lines 47-57 could be replaced by:

`res = lmfit.minimize(residual, p, method='emcee', nan_policy='omit', burn=300, steps=1000, thin=20, params=mi.params)`

Or, if you want to test both the logic branch where the `userfcn` returns a single float and also the branch where the `userfcn` returns a vector of residuals, the above line could be added as an additional example after line 57 to show that it returns the same result as lines 47-57 with the` __lnsigma` parameter replacing the `noise` parameter.

Thoughts? Let me know and I'll work on adding this.
@FaustinCarter 

> @newville I think that the code in examples/doc_fitting_emcee.py could be modified (and 
> simplified) to include this as a test. Specifically, lines 47-57 could be replaced by:
>
> res = lmfit.minimize(residual, p, method='emcee', nan_policy='omit', burn=300, steps=1000, 
>  thin=20, params=mi.params)

Yes, that sounds great.

> Or, if you want to test both the logic branch where the userfcn returns a single float and also 
> the branch where the userfcn returns a vector of residuals, the above line could be added as
> an additional example after line 57 to show that it returns the same result as lines 47-57 with 
> the __lnsigma parameter replacing the noise parameter.

Doing both would be really nice, but at doing either one would be OK too.
Thanks!

This all sounds good. I'll work on putting the tests together. What is the timeline for getting this into the next release?
@FaustinCarter Well, we were aiming for about now, but I've been tied up with other stuff too, and have a meeting and then travel coming up so it can slip a couple weeks.  Is that reasonable for you? 
Ok, I have rewritten the emcee example to use `lmfit.Model.fit(method='emcee')`. This effectively tests all the ways of doing it, since the Model fit method calls `Minimizer.minimize(method='emcee')`, which calls `Minimizer.emcee`.

Please run this on your computers to check and make sure you are getting sensible results. If everything looks good, I can update the docs example file (/doc/fitting.rst) to reflect this example.
An actual test should also be added for this, but I'm not sure where the best place to put it is. Does it need its own test file (/tests/test_emcee.py), or should it be added to one of the existing test files?
@FaustinCarter Great, Thanks!   For a test location, the other `emcee` tests are in the catchall (and now poorly named) `test_nose.py`.  That would be a place to put it.  If you want to make a `test_emcee.py` that would also be fine.  At some point we could then move the current `emcee`-using tests there as well.   Either way is fine.
@FaustinCarter @newville thanks for the work, but I am not sure if we want to change the example you adapted (the ones starting with ```doc_``` are actually used in the documentation... 

Assuming that one starts reading the documentation in a logical order, one would read the ```minimize``` section before getting to the Model interface and such. Therefore, in that part I would prefer to leave it "as is" - or if you want just change the function call from ```res = mini.emcee()``` to use ```res = mini.minimze(method='emcee', .....)``` to show the new possible usage that would be fine with me. The usage in the Model class could be done either somewhere when describing the class (i.e., in "model" or "built-in models"), or just as an example that does not end up in the actual documentation (as happens for many) - that's at least my opinion. The fact that this "effectively tests all the ways of doing it...." as you state is good, but that's what tests are for and shouldn't dictate how to write the documentation.

Additionally, it seems to me that there are some unexpected/unaddressed issue with using ```emcee``` from the Minimizer interface. For example, looking at the fit_report shows this (I don't think that's what we want):
```
[[Fit Statistics]]
    # fitting method   = emcee
    # function evals   = 0
    # data points      = 0
    # variables        = 5
    chi-square         = unknown
    reduced chi-square = unknown
    Akaike info crit   = unknown
    Bayesian info crit = unknown
```
(As a minor point: is there a reason to change from ```noise``` to ```__lnsigma```?)

Additionally, for me it crashes with the following message (on Python 3.7, it seems that 2.7 works but haven't investigated further):
```
Traceback (most recent call last):
  File "doc_fitting_emcee.py", line 73, in <module>
    truths=result_emcee.params.valuesdict().values())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/corner/corner.py", line 251, in corner
    if truths is not None and truths[i] is not None:
TypeError: 'odict_values' object does not support indexing
```

Finally, if the decision is to change the example anyway (or add it elsewhere) - please check the code with ```flake8```, ```pylint``` or alike and make sure that it follows the coding standards.


This is a long reply, so I've broken it into sections. As soon as there is consensus I will run with it and process the updates.

## Documentation issue:

@reneeotten @newville I did some digging around, and it looks like this particular example is presented independently several times:

 * in the file doc/fitting.rst (as you mentioned)
 * as an ipython notebook in examples/lmfit_emcee_example.ipynb
 * as a python file in examples/doc_fitting_emcee.py (it looks like this is just the code from the cells in the ipynb file above)

There is also an independent emcee example at examples/lmfit_emcee_model_selection.ipynb

I looked for scripts that autogenerate any of the above from one another, but did not see anything. It looks like they will all need to be updated to match by hand (can someone confirm?).

For things like this there should really be a single ipython notebook that autogenerates the rst file (and appropriate images) in the docs and also the pure python example file at build (typically via nbconvert). This way the examples have a single point of creation and can easily be tested and updated as things change. This is probably worthy of opening a new issue, but I'll defer to you guys on this.

### Proposal:
To get this into the next release quickly, for now I should just take @reneeotten's suggestion and revert to the original example, but only slightly modified to use the `method=emcee` hook. I will update all three locations above as appropriate. (As an aside, the choice to use `__lnsigma` instead of `noise` as a parameter was because that is the name the emcee method auto-assigns when it gets an unweighted residual function, which it does when using the Model class to wrap a function, and I wanted to place bounds on that auto-generated parameter.)

The code I have now that tests this using the Model class will get moved to examples/example_emcee_with_Model.py.

Finally, there is one other edit to the docs: the table of supported methods starting on line 128 of doc/fitting.rst will need to be updated.

## Empty Fit Report

Regarding the fit report looking empty, that is because the emcee method never runs `_calculate_statistics` (which all the others do). For that to work, we would need to decide what constitutes an appropriate residual for the emcee method. I think for continuity, we could just generate `self.residual` from the parameter values that are in result.params at the end of the MCMC run. An alternate choice would be to use the maximum likelihood estimate params. Additionally, the number of function evaluations could reasonably considered to be `ntemps*nwalkers*steps`.

### Proposal:
Update `self.residual` and `nfev` from within the `emcee` method as outlined above and then call `_calculate_statistics` to get a decent fit report.

## Python 3 crash

The issue with Python 3.7 is because dict().values() is a generator in Python 3 and not a list. Casting it explicitly to a list fixes the problem. I'll update that when I move the file to the new location.
@FaustinCarter @newville thank you for the comprehensive response!

- Documentation
There is indeed some duplication in the specific ```emcee``` example... At some point I reorganized the example and as part of that made sure that there are ```doc_``` files for all example presented in the documentation. Most of them are included using a ```literalinclude```, but for ```emcee``` there is some explanatory text in between the actual code that make this impossible (I think). So changing the examples in the documentation mostly involves adapting the Python code in the examples, but in some cases it involves adding the ```.rst``` files as well in addition to remaking the figures. If you can come up with a more automated way of doing this that would be great (but that's indeed outside of the scope of this PR)! Additionally, it might be beneficial to change all (perhaps with the exception of ```doc_```?) examples to IPython notebooks - I think we talked about this before (#442)... Finally, I agree with your proposal on this point.

- Fit report
I agree with your suggestions. It makes sense to me to use ```result.params``` for the calculation of ```self.residual``` - that's how all other methods do it. According to the documentation for emcee these "represent the median (50th percentile) of all the samples". Perhaps @andyfaff has some insights on this?

- Tests
Tests still need to be included to make sure everything works as intended, that presumably would have shown the differences between Python 2.7 and 3.x.
@FaustinCarter @reneeotten  I'm OK with reverting the example and all these suggestions. 

I agree that thinking about renaming / reorganizing all the "doc examples" is a fine idea, but that should probably wait for 1.0 release.


@FaustinCarter thanks - there are still some failures with Travis though. Ideally you would run the test-suite locally and fix possible issues before pushing changes. I will need to look a little bit closer at the code to see whether changing ```_calculate_statistics``` is the best thing to do and at your changes in the ```emcee``` method. But assuming that these things are all good, it would help to fix the failures to see if it actually works as intended.
@FaustinCarter after fixing the syntax errors locally, it seems that the calculation of the statistics doesn't work as intended. In your example, ```self.residual``` is actually an ```numpy.ndarray``` so in the function ```_calculate_statistics``` it doesn't make it to the code that you added. In this case you were adding emcee-specific code to this general function, I would personally just do this in the ```emcee``` function and consider it a special case.

@newville opinions? 
@reneeotten @FaustinCarter right, maybe the `float_behavior` and reduction of ndarray to scalar "cost" should be in `emcee` instead of `_calculate_statistics`.

Well, I don't know if other scalar methods might also be returning values meant to be something other than chi-square.  Perhaps that is an important consideration and we should look into the matter, but I'm not sure we're going to (or should) solve that here.  
@reneeotten @newville Hey sorry about the failed build. I had to do a computer swap and was without python and couldn't run tests... I'll get the failed builds sorted in the next couple of days.

Regarding the `_calculate_statistics` function: The `emcee` process (as written) allows three ways to do things:

 * pass in a function that returns a residual
 * pass in a function that returns sum(residual^2)/weight^2 (i.e. float method is chi2)
 * pass in a function that returns the log likelihood (i.e. float method is lnprob). In this final case, it is not possible to back out a value for chi-squared unless it is assumed that the weight is unity.

It should be possible to just use the user-supplied function to calculate one of the three above quantities from `result.params` and then call calculate statistics, provided the last case (the special one) is handled appropriately. This is a small change to the function and I think it is within the scope of this pull request. I also think I see where I went wrong, so look for a fix soon.

However, in all of these cases, if the weights are known ahead of time, then one should use the extended formula for `_neg2_log_like` and properly apply the weights rather than assuming that the maximum likelihood value of the weights is just the sum of squared residuals, but that would require the `_calculate_statistics` function to have access to the weighting information. In general I agree that messing too much with this function is outside the scope of this pull request.

In any case, give me another day or two so sort out the build issues and check the calcs and finish updating the docs and we can go from there.
@FaustinCarter  For the `float_behavior='posterior'`case in `_calculate_statistics()`, why not set

```
self.chisqr =  np.exp(_neg2_log_likel)
``` 

(to be clear, `self.ndata =1`) ?   That seems more useful that `NaN` to me.   Yes, that does assume the residual is scaled properly, but all the code here also assumes that.


@FaustinCarter okay, I do see now the three options in ```emcee``` and how you incorporated that in the ```_calculate_statistics``` function. Sorry for not reading careful enough before! I am still not sure though if that function is the right place to handle this as it's specific to only the ```emcee``` method, but it probably doesn't hurt either... I'll defer to @newville for an executive decision ;) 

In any case, please do incorporate these three different scenarios in the tests that still need to be added to make sure all options work as intended (as I said earlier your example returns an array and doesn't reach the code you added).

Minor details: please do not add trailing whitespace to newlines and remove the newline before ```result._calculate_statistics(float_behavior)```.
@FaustinCarter @reneeotten In general, I think that it would be OK for `calculate_statistics` for scalar minimizers to be somewhat more general than it is so that statistics other than chi-square was being minimized.   

With this PR, the `float_behavior` is unique to `emcee` -- that code could be in `emcee`.      
If I'm reading this correctly (please correct if not!), it seems that `float_behavior='posterior'` is a bit of an oddity, as it should be maximized, not minimized.  Currently, there are a few different ways of reducing a residual array to the cost value for the scalar minimizers -- but they are all meant to be minimized.   It might be good for `calculate_statistics()` to support these different cost functions (neg-entropy, neg-log-cauchy which is sort of like -neg log likelihood).    Perhaps these should all be rationalized?

That is out-of-scope for this PR, but it sort of suggests that we may want to revisit some of these methods in the near term.   I think that argues for *not altering*  `calculate_statisctics` in this PR and just moving that code to calculate the statistics within `emcee()`.   That would mean we won't change the API for `calculate_statistics()` only to change it again soon....

Does that seem reasonable to both of you?  Are there other things to consider here?


that sounds like a good plan to me. I certainly agree that ```_calculate_statistics``` should be able to handle other scenarios/cost-functions than chisqr. If that can all be generalized into one function that would be great, but probably needs a bit more time to get right.
Sorry for the build failures. Somehow I managed to leave `emcee` out of my test environment. Fixed that. I think everything here is ready to go with the exception of updating the documentation. I think we should probably open a new issue for discussing how to generalize the `_calculate_statistics` function and then that can go in a separate PR (unless you guys feel strongly about putting in this one).
@FaustinCarter Thanks -- I think this looks great, and thanks for moving the statistics calculations!
@reneeotten does this OK to you? 
Sorry for the delay on the updates. Had to drive from Chicago to Santa Monica...

I think this last round of edits should sort out the code side of things. I'm currently adding a few tests, but that will be straightforward. There are still a few open questions regarding the documentation that we should resolve. Once I have answers to those I'll do the edits. The goal here is to minimize the number of places the same exact thing shows up down to one, so that future edits don't slip through the cracks.

In no particular order:

 * The doc_fitting_emcee.py file is never actually included anywhere (unlike the other doc_foo_bar.py files). This suggests it ought to be renamed to example_fitting_emcee.py or something like that. It is clear that the reason it is not literally included is because the original author of the documentation desired to show some output images in between code blocks, which is understandable. One way to accomplish this would be to use the line-numbers option to the literalinclude rst directive, and include the file in pieces with the images in between. This is probably the simplest option for moving forward.

 * The lmfit_emcee_example.ipynb file is essentially the same code flow, but it has pictures inline. This is great, and pretty much what is desired. One option is to add a script that converts this notebook (via nbconvert) to an rst file, dumps the images in static as appropriate and then includes that rst file inline at the appropriate place in fitting.rst (line 405 or thereabouts). This script could be called from the conf.py file. Thoughts?

 * There are some other example files that involve emcee, but I think they can be left as-is for now. Yes?
Once I sort out the docs and add the tests and everyone is happy with the result, I'll squash commits in prep for the merge.
@FaustinCarter thanks a lot, this looks very good! 

You marked my comment regarding line 343 in ```minimizer.py``` (see above) as resolved, but didn't change it back to what I suggested ```self.redchi = self.chisqr / max(1, self.nfree)```. Any specific reason, it could very well be that I'm wrong about that... if you think that's the case could you elaborate?

Regarding the documentation/example-code-duplication. The ```doc_fitting_emcee.py``` is intended to reflect the code in the documentation and therefore should keep its name. You already made your changes there so that's good to go. I wasn't aware of the possibility to specify linenumbers with ```literalinclude```, but that seem exactly what we want for now - I am in favor of that suggestion. Yes, there is duplication indeed and ideally we clean that up in a later PR. Let's indeed not worry about other examples, there are multiple ways of doing the same way in lmfit and if that's reflected in the documentation/examples that's fine with me.
@FaustinCarter @reneeotten FWIW, I would be generally in favor of converting the emcee notebook to RST for the docs.

I would really like to get 0.9.12 released.  Three weeks ago we had 0.9.12rc1 and this seemed like it was very close (and a great addition!) so that we could wait for this for 0.9.12.   It has gone slower than I expected and seems to have needed several reviews.  I thought it was ready a week ago, and now there appears to be more discussion of what to do.

This is very close, passing tests, and this does not have to be perfect -- we know the "calculate statistics" bit is going to need some work for example.  I think we should either merge this in the next few days (could be as-is as far as I'm concerned) or it will miss 0.9.12.   Is that OK?


@newville I agree with all your statements Matt. As far as I am concerned, the last few changed I would like are:
1. the change of line 343 in minimizer.py is reverted
2. line 315 in minimizer.py is removed
3. the few small changes in fitting.rst describing the emcee method are made. It doesn't matter to me whether "linenumber and literalinclude" are used or things are just manually changed for now. I thought it was obvious that all further discussion on documentation and how to do this in the future should not be part of this PR so I don't understand why this keeps on coming up... 
4. commits are squashed in a few small ones with useful commit messages

I can add tests for this at a later stage, we know this does work as advertised. I consider points 1 and 4 as requirements before merging and releasing 0.9.12. However, the above should take more than 30 minutes to accomplish so I'm fine with saying "if this is not finished by Tue night it will not go into 0.9.12". @FaustinCarter if you don't have time to finish this off I can take care of it tonight, just let me know.

@reneeotten @FaustinCarter Thanks.  Yes, I should have also said: would it be OK to merge this soon-ish, and then have Renee and/or I make a few small cleanups before 0.9.12 release?  That is Faustin has already done "the work", and we're just asking for cleanups based on the rest of the code.
@FaustinCarter please let us know how you want to proceed. I can take care of the requested changes and squash the commits today, but don't want to interfere if you've been doing work on this... If I don't hear anything from you this afternoon I will assume that I can go ahead tonight and make the changes myself and squash the commits so that this PR is ready to be merged, okay?
@reneeotten @newville  I don't have time to finish this up today, but tomorrow I think I'll have some time to clean up the last few details and get it squashed up for a pull request. Is that soon enough?

Important question: Do you want me to rebase this off the most recent master prior to squashing to avoid a merge commit, or should I just squash down to one commit and then do a merge? It seems like most folks have strong preferences either for or against merge commits, but I remain agnostic.
@FaustinCarter @reneeotten I am going to merge this around 24 hours from now, and tag 0.9.12rc2.  
If there are things that need to be improved in this PR, this is the time to do it.  You can also say here what else needs to be done.  I may make some of the changes discussed above. 

I intend to tag it as 0.9.12 before Dec 1.  If nothing is broken with 0.9.12rc2 (small fixes related to this PR would be OK), that will be 0.9.12.   Otherwise, we'll use the current master branch (a8d29f3feb5da57a2e99a6d55cb86fc5576ccaa3) as 0.9.12.




Resolving patch:
diff --git a/examples/doc_fitting_emcee.py b/examples/doc_fitting_emcee.py
--- a/examples/doc_fitting_emcee.py
+++ b/examples/doc_fitting_emcee.py
@@ -22,7 +22,7 @@
 y = (3.0*np.exp(-x/2) - 5.0*np.exp(-(x-0.1) / 10.) +
      0.1*np.random.randn(len(x)))
 if HASPYLAB:
-    plt.plot(x, y)
+    plt.plot(x, y, 'b')
     # plt.savefig('../doc/_images/emcee_dbl_exp.png')
     plt.show()
 
@@ -39,8 +39,8 @@ def residual(p):
 lmfit.printfuncs.report_fit(mi.params, min_correl=0.5)
 if HASPYLAB:
     plt.figure()
-    plt.plot(x, y)
-    plt.plot(x, residual(mi.params) + y, 'k')
+    plt.plot(x, y, 'b')
+    plt.plot(x, residual(mi.params) + y, 'r')
     # plt.savefig('../doc/_images/emcee_dbl_exp2.png')
     plt.show()
 
@@ -73,7 +73,7 @@ def residual(p):
 
 if HASPYLAB:
     plt.figure()
-    plt.plot(x, y)
+    plt.plot(x, y, 'b')
     plt.plot(x, residual(mi.params) + y, 'r', label='Nelder-Mead')
     plt.plot(x, residual(res.params) + y, 'k--', label='emcee')
     plt.legend()
diff --git a/lmfit/minimizer.py b/lmfit/minimizer.py
--- a/lmfit/minimizer.py
+++ b/lmfit/minimizer.py
@@ -341,7 +341,7 @@ def _calculate_statistics(self):
             self.chisqr = self.residual
             self.ndata = 1
             self.nfree = 1
-        self.redchi = self.chisqr / self.nfree
+        self.redchi = self.chisqr / max(1, self.nfree)
         # this is -2*loglikelihood
         _neg2_log_likel = self.ndata * np.log(self.chisqr / self.ndata)
         self.aic = _neg2_log_likel + 2 * self.nvarys
@@ -1312,7 +1312,6 @@ def emcee(self, params=None, steps=1000, nwalkers=100, burn=0, thin=1,
         # Handle special case unique to emcee:
         # This should eventually be moved into result._calculate_statistics.
         elif float_behavior == 'posterior':
-            result.nvarys = len(result.init_vals)
             result.ndata = 1
             result.nfree = 1
 


Test patch:
diff --git a/tests/test_nose.py b/tests/test_nose.py
--- a/tests/test_nose.py
+++ b/tests/test_nose.py
@@ -431,6 +431,20 @@ def test_emcee(self):
 
         check_paras(out.params, self.p_true, sig=3)
 
+    @dec.slow
+    def test_emcee_method_kwarg(self):
+        # test with emcee as method keyword argument
+        if not HAS_EMCEE:
+            return True
+
+        np.random.seed(123456)
+        out = self.mini.minimize(method='emcee', nwalkers=100, steps=200,
+                                 burn=50, thin=10)
+        assert out.method == 'emcee'
+        assert out.nfev == 100*200
+
+        check_paras(out.params, self.p_true, sig=3)
+
     @dec.slow
     def test_emcee_PT(self):
         # test emcee with parallel tempering