ENH: add optional ddof param to DataFrame.cov()
State: closed

 - [ ] closes #xxxx (searched, didn't see any issues on this topic)
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [x] whatsnew entry

Noticed that the .cov() methods on DataFrame and Series assume the user wants to calculate sample variance and normalize by N-1. In contrast, the .var() and .std() methods on DataFrame, Series, DataFrameGroupBy, and SeriesGroupBy take a ddof parameter. Its default value is 1, calculating the sample std/var, but the user can specify ddof=0 to calculate the population std/var.

Simply added a ddof param (default value 1) to the .cov() methods. The value is then passed into other method calls as required. Did have to modify 1 bit of Cython code in algos.pyx, the nancorr() function.

Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=h1) Report
> Merging [#16227](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4a748ccf6086a07e5ce18066099c835b2beef31f?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16227/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16227      +/-   ##
==========================================
- Coverage   90.25%   90.23%   -0.02%     
==========================================
  Files         162      162              
  Lines       50882    50882              
==========================================
- Hits        45925    45915      -10     
- Misses       4957     4967      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.02% <100%> (-0.01%)` | :arrow_down: |
| #single | `40.31% <60%> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.58% <100%> (-0.1%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.8% <100%> (ø)` | :arrow_up: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `98.2% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `90.68% <0%> (-0.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=footer). Last update [4a748cc...a93f443](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=h1) Report
> Merging [#16227](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d0d28fec180ee61de17921fe5068ecde95adae8a?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `88.88%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16227/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16227      +/-   ##
==========================================
+ Coverage   91.01%   91.01%   +<.01%     
==========================================
  Files         162      162              
  Lines       49567    49580      +13     
==========================================
+ Hits        45111    45125      +14     
+ Misses       4456     4455       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <88.88%> (+0.02%)` | :arrow_up: |
| #single | `40.23% <16.66%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `95% <100%> (ø)` | :arrow_up: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `97.48% <85.71%> (-0.18%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.68% <90%> (-0.14%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.05% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=footer). Last update [d0d28fe...e80054d](https://codecov.io/gh/pandas-dev/pandas/pull/16227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you rebase and update?
Made changes to docstrings, etc. based on comments. Rebased and updated.
@edhalter did you push your changes? There seems to be a conflict in the whatsnew file. You may need to force push to your branch.
Already pushed -- the conflicting file has my most recent changes. Someone just needs to edit it and append my new line to the other new lines under "Other Enhancements".
@edhalter pls rebase on master. this will present a conflict which you fix, then push.
Rebased on master.
Added some checks in DataFrame.cov() and the nancov() func called by Series.cov():

- sanity check on input ddof
- warn when N - ddof <= 0

For the latter, alternatively, could throw an Exception, or just silently return nan. The warning seemed better -- that's what numpy's cov() does:

`warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)`

However, the Cython nancorr() func called by DataFrame.cov() does just silently return nan.
> warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)

can you change UserWarning -> RuntimeWarning to be consistent

> However, the Cython nancorr() func called by DataFrame.cov() does just silently return nan.

can you change (and test) this as well?
if you can rebase and update
can you rebase
Hello @edhalter! Thanks for updating the PR.











Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on August 21, 2017 at 08:35 Hours UTC
Returning to this.... Changed warning type to RuntimeWarning, and added a single warning that will throw in cython from nancorr() if at least 1 result in the matrix is set to NaN due to N - ddof <= 0.

Manual tests looked good. Also added some more automated tests in DataFrame's test_cov().
closing as stale, but if you'd like to continue work, pls rebase and ping.