API/BUG/ENH: ewmvar/cov debiasing factors; add 'adjust' to ewmvar/std/vol/cov/corr; ewm*() min_periods
State: closed

Closes https://github.com/pydata/pandas/issues/7911.
Closes https://github.com/pydata/pandas/issues/7912.
Closes https://github.com/pydata/pandas/issues/7977.
Closes https://github.com/pydata/pandas/issues/7900.

These are all fixed simultaneously in order to enable consistency testing across the various functions.

Enhancement -- add `adjust` to `ewm*()` (https://github.com/pydata/pandas/issues/7911):
This code adds `adjust` arguments to `ewmvar`, `ewmstd`, `ewmvol`, `ewmcov`, and `ewmcorr` (`ewma` already had it). If ok, I'd like to reorder the parameters to be more logical and consistent, but for now I just added `adjust=True` to the end of the arguments list.

Bug fix -- `ewmvar/std/cov` debiasing factors (https://github.com/pydata/pandas/issues/7912):
This code corrects the bias correction factor for exponentially weighted (co)variance calculations. The prior factor was the asymptotic value for an infinite number of observations.

As a result of this change, `ewmvar/cov` will now return `NaN` for the (co)variance of a single value when `bias=False`, and `0` when `bias=True`. Previously they always returned `0` for a single value. See also https://github.com/pydata/pandas/issues/7900.

API -- meaning of `min_periods` (https://github.com/pydata/pandas/issues/7977):
The meaning of the `min_periods` argument to the `ewm*()` functions is now consistent with that of the `expanding_*()` functions, i.e. an entry will be `NaN` if the count of non-`NaN`  input values through that point is `<min_periods`. Note that in view of this change, https://github.com/pydata/pandas/issues/7884 / https://github.com/pydata/pandas/pull/7898 is no longer relevant.

Bug fix -- `rolling/expanding_var/std` for a single value (https://github.com/pydata/pandas/issues/7900)


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
CC'ing @jaimefrio, in case of interest.

I think in an ideal world there would be a separate `algos.ewmcov` function, but for now I kept the existing framework that calculates `ewmcov(x,y)` as `ewma(x*y) - ewma(x)*ewma(y)`, and modified `algos.ewma` so that it either returns the moving average (`variance_debiasing_factors=False`) or the variance debiasing factors (`variance_debiasing_factors=True`). This isn't as efficient or as numerically stable as having a good `algos.ewmcov` function, but for now it should do. Need to check that performance hit isn't too bad.

Per https://github.com/pydata/pandas/wiki/Performance-Testing, vbench is not compatible with Python 3. Is this still the case?

hmm. never tried it on py3. It probably isn't as it hasn't been updated in a while (its a separate project). I imagine you could patch it 2 work / fix it pretty easily (if you want).

Does vbench, as per https://github.com/pydata/pandas/wiki/Performance-Testing, really depend on `affinity`, which per https://pypi.python.org/pypi/affinity was last updated in 2006 for Python 2.4?

it should work w/o 
though u may have to hack around it

OK. I'm trying to proceed as per https://github.com/pydata/pandas/wiki/Performance-Testing, but when I go to my pandas directory and run `./test_perf.sh -b master -t HEAD`, I see a pop-up window that does something with git but disappears quickly and the script exits immediately. I'm assuming it didn't work, but (a) is there a log somewhere?, and (b) if it did work, where would I see the results? Afraid I can't find very clear instructions at either https://github.com/pydata/pandas/wiki/Performance-Testing or https://github.com/pydata/vbench.

you prob need to so something slightly different as that script runs in Linux and prob won't work in windows

I'm slowly getting the hint that one isn't supposed to do development on Windows...

hahah

FYI u might want to setup a vagrant box

free and they are stable
u can just ssh to them (and store your data locally)

BTW, are there any automated (Travis?) vbench results?

no
though that could be done as an optional build (well a limited subset could be done)
as this takes a while and has to compile twice (for the compares)
but could be done

Vagrant sounds interesting. I might give it a shot.

Re: Vagrant. Any suggestions for what box to use? (It's been many years since I've used linux/unix.) Also, do you have a vagrantfile that provisions all the necessary Python stuff?

Perhaps add such a Vagrantfile (or files?) to pandas?

https://github.com/pydata/pandas/blob/d7c6eb136dd5c0736149278cb40eb97d1d5e99f1/ci/speedpack/Vagrantfile

but you are better off searching the web for examples to create the environement. This is part of an automated process to actually create wheels.

I've updated the code that (a) adds "adjust=True" to all the `ewm*()` functions that didn't have them; and (b) implements an efficient, numerically stable `algos.ewmcov()` with proper debiasing factors (when `bias=False`).

I added a whole bunch of tests for consistency among the `ewm*()` functions, but still need to think about how to verify that the variance debiasing factors are correct.

@jaimefrio, note that I wrote a new `algos.ewmcov()`, and reimplemented `ewmvar()` and `ewmcov()` in terms of it (as opposed to, for example, `ewmcov(x,y) = ewma(x*y) - ewma(x)*ewma(y)`). You might similarly want to replace `algos.roll_var()` with an `algos.roll_cov()`, and reimplement `rolling_cov()` (and `rolling_var()`) in terms of it.

@jreback, @jaimefrio: unless anyone objects, I'd like to write rolling versions of all the `ewm*()` functions. But I'm not going to tackle that until all the current open issues re: moments.py are finished...

I think this is pretty much ready, other than updating release notes.

ok, go ahead and put in a release note. What does this actually change/adjust?

cc @jaimefrio want to have a look?

I added these release notes, which describe the changes:

```
- :func:`ewmstd`, :func:`ewmvol`, :func:`ewmvar`, :func:`ewmcov`, and :func:`ewmcorr`
  now have an optional ``adjust`` argument, just like :func:`ewma` does,
  affecting how the weights are calculated.
  The default value of ``adjust`` is ``True``, which is backwards-compatible. (:issue:`7911`)

- Bug in :func:`ewmstd`, :func:`ewmvol`, :func:`ewmvar`, and :func:`ewmcov`
  calculation of de-biasing factors when ``bias=False`` (the default).
  Previously an incorrect constant factor was used, based on ``adjust=True``, ``ignore_na=True``,
  and an infinite number of observations.
  Now a different factor is used for each entry, based on the actual weights used up to that point
  (and so taking account of the values of ``adjust`` and ``ignore_na``, and the number of observations).
  See http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance
  and the formula under ``gsl_stats_wvariance`` at http://www.gnu.org/software/gsl/manual/html_node/Weighted-Samples.html
  for details.
  In particular, for a single point a value of ``NaN`` is returned when ``bias=False``
  (analogous to the usual ``N/(N-1)`` factor with ``N=1``),
  whereas previously a value of ``0`` was returned. (:issue:`7912`)
```

can u update the docs in moments.rst with this type of info as well (then in the release note just provide a link to the docs)

Where is moments.rst? I don't see it under doc/source.

sorry meant computation.rst

Got it. Will try to update computation.rst tomorrow.

I updated computation.rst. A few things:
1. How do I include a link to computation.rst in the release notes? (Sorry if stupid question...)
2. I can't actually build the help files (on Windows), so decent chance some editing is needed.
3. All the function names in computation.rst appear as `''ewma''`. Should they appear as `:func:'ewma'`?

look in v0.15.0 the :ref: are examples of how to link 

OK, I put in some references. I hope they work. :-)

@seth-p the docs are also build on travis, so if you can't build them yourself, you can always look at the log output (not the html output, but still, the errors there can be some hints of things that are wrong) on travis. It is the 3th job, if you click on that on then scroll down, you can expand the docs log output (click on '$ if [ -f /tmp/doc.log ]; then cat /tmp/doc.log; fi')

Eg, there is one relevant warning now I think: `/tmp/doc/source/computation.rst:572: ERROR: Content block expected for the "note" directive; none found.`

@jorisvandenbossche Thank you. Will take a look. Should such a documentation error not break the build?