add biweight midcorrelation as option for series correlation
State: closed

Add [Biweight Midcorrelation](http://download-v2.springer.com/static/pdf/95/chp%253A10.1007%252F978-3-642-39678-6_14.pdf?token2=exp=1428360548~acl=%2Fstatic%2Fpdf%2F95%2Fchp%25253A10.1007%25252F978-3-642-39678-6_14.pdf*~hmac=980c2edddba12bbb73ebbadaa682092cde9d534bc8738d65785647402b683c21), an alternative correlation method to Pearson that is robust to outliers, to the series correlation method repertoire.


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
This looks nice! It needs unit tests and updates to the documentation. See: http://pandas-docs.github.io/pandas-docs-travis/contributing.html#contributing-to-the-code-base

Thanks! Just wanted to get the idea out there before I started working on
the tests

On Mon, Apr 6, 2015, 9:39 PM Stephan Hoyer notifications@github.com wrote:

> This looks nice! It needs unit tests and updates to the documentation.
> See:
> http://pandas-docs.github.io/pandas-docs-travis/contributing.html#contributing-to-the-code-base
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9826#issuecomment-90356292.

Might be worth waiting for feedback from another core dev before you get too far. A library like stats models would probably be a more natural fit. But on the other hand, this seems likely to be useful to other pandas users, barely expands the API and has a straightforward implementation. So it seems pretty reasonable to me.

On Mon, Apr 6, 2015 at 10:14 PM, Olga Botvinnik notifications@github.com
wrote:

> Thanks! Just wanted to get the idea out there before I started working on
> the tests
> On Mon, Apr 6, 2015, 9:39 PM Stephan Hoyer notifications@github.com wrote:
> 
> > This looks nice! It needs unit tests and updates to the documentation.
> > See:
> > http://pandas-docs.github.io/pandas-docs-travis/contributing.html#contributing-to-the-code-base
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/pydata/pandas/pull/9826#issuecomment-90356292.
> > 
> > ---
> > 
> > Reply to this email directly or view it on GitHub:
> > https://github.com/pydata/pandas/pull/9826#issuecomment-90362192

this looks reasonable. `.corr` is already well supported in pandas and this is a straightforward extension.
- needs a test
- needs a release note in v0.16.1

+1 on including this as well. I typically draw the line as anything doing inference belongs in statsmodels.

Running the test suite gets me this `TimeDelta` error that I'm not too sure about:

Test command:

```
nosetests pandas/tests/test_nanops.py:TestnanopsDataFrame.test_nancorr_bicor
```

Test output:

```
E
======================================================================
ERROR: Failure: ValueError (cannot construct a TimeDelta from the passed arguments, allowed keywords are [days, seconds, microseconds, milliseconds, minutes, hours, weeks])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/olga/workspace-git/pandas/pandas/__init__.py", line 47, in <module>
    import pandas.core.config_init
  File "/Users/olga/workspace-git/pandas/pandas/core/config_init.py", line 17, in <module>
    from pandas.core.format import detect_console_encoding
  File "/Users/olga/workspace-git/pandas/pandas/core/format.py", line 24, in <module>
    from pandas.tseries.period import PeriodIndex, DatetimeIndex
  File "/Users/olga/workspace-git/pandas/pandas/tseries/period.py", line 8, in <module>
    import pandas.tseries.frequencies as frequencies
  File "/Users/olga/workspace-git/pandas/pandas/tseries/frequencies.py", line 9, in <module>
    from pandas.tseries.offsets import DateOffset
  File "/Users/olga/workspace-git/pandas/pandas/tseries/offsets.py", line 2152, in <module>
    class Nano(Tick):
  File "/Users/olga/workspace-git/pandas/pandas/tseries/offsets.py", line 2153, in Nano
    _inc = Timedelta(nanoseconds=1)
  File "tslib.pyx", line 1722, in pandas.tslib.Timedelta.__new__ (pandas/tslib.c:27355)
ValueError: cannot construct a TimeDelta from the passed arguments, allowed keywords are [days, seconds, microseconds, milliseconds, minutes, hours, weeks]

----------------------------------------------------------------------
Ran 1 test in 0.025s

FAILED (errors=1)
```

I rebased and still get this error

I'm pretty sure this is an indication that some of the cython is not compiled properly for the version of pandas you're using. Running "make clean" and "make" in the main pandas install directory should fix this.

On Tue, Apr 14, 2015 at 11:30 AM, Olga Botvinnik notifications@github.com
wrote:

> Running the test suite gets me this `TimeDelta` error that I'm not too sure about:
> Test command:
> 
> ```
> nosetests pandas/tests/test_nanops.py:TestnanopsDataFrame.test_nancorr_bicor
> ```
> 
> Test output:
> 
> ```
> E
> ======================================================================
> ERROR: Failure: ValueError (cannot construct a TimeDelta from the passed arguments, allowed keywords are [days, seconds, microseconds, milliseconds, minutes, hours, weeks])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/loader.py", line 414, in loadTestsFromName
>     addr.filename, addr.module)
>   File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
>     return self.importFromDir(dir_path, fqname)
>   File "/Users/olga/anaconda/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
>     mod = load_module(part_fqname, fh, filename, desc)
>   File "/Users/olga/workspace-git/pandas/pandas/__init__.py", line 47, in <module>
>     import pandas.core.config_init
>   File "/Users/olga/workspace-git/pandas/pandas/core/config_init.py", line 17, in <module>
>     from pandas.core.format import detect_console_encoding
>   File "/Users/olga/workspace-git/pandas/pandas/core/format.py", line 24, in <module>
>     from pandas.tseries.period import PeriodIndex, DatetimeIndex
>   File "/Users/olga/workspace-git/pandas/pandas/tseries/period.py", line 8, in <module>
>     import pandas.tseries.frequencies as frequencies
>   File "/Users/olga/workspace-git/pandas/pandas/tseries/frequencies.py", line 9, in <module>
>     from pandas.tseries.offsets import DateOffset
>   File "/Users/olga/workspace-git/pandas/pandas/tseries/offsets.py", line 2152, in <module>
>     class Nano(Tick):
>   File "/Users/olga/workspace-git/pandas/pandas/tseries/offsets.py", line 2153, in Nano
>     _inc = Timedelta(nanoseconds=1)
>   File "tslib.pyx", line 1722, in pandas.tslib.Timedelta.__new__ (pandas/tslib.c:27355)
> ValueError: cannot construct a TimeDelta from the passed arguments, allowed keywords are [days, seconds, microseconds, milliseconds, minutes, hours, weeks]
> ----------------------------------------------------------------------
> Ran 1 test in 0.025s
> FAILED (errors=1)
> ```
> 
> ## I rebased and still get this error
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/9826#issuecomment-93009777

@shoyer - yes that did it, thank you! Now onto fixing the test :)

corner case: what do you do when MAD is zero?

this looked fine the last time I saw it. pls rebase/squash and can put this in after a quick look.

This also needs an update to the doc strings for `Series.corr`/`DataFrame.corr` and the doc page on computing correlations:
http://pandas.pydata.org/pandas-docs/stable/computation.html#correlation

@olgabot can you update?

closing, but if you would like to update, pls reopen

Apologies for the delay.. I've made the suggested changes 
- `Series.corr()` and `DataFrame.corr()` docstrings
- Move release notes to v0.17.1

RE when MAD is 0 - then all samples in one of the vectors are equal, and their similarity to anything else is undefined, and thus this would return nan.

I don't see a way to reopen this. On [my github page](https://github.com/olgabot/pandas/tree/biweight_midcorrelation) I see an option to "Compare and Pull Request." Is re-opening an issue only something that project owners can do?

@olgabot reopened (apparantly the PR submitter can't do this him/herself ..)

@olgabot ok, pls rebase, then can give comments

I rebased to pydata/pandas:master and merged on
olgabot/pandas:biweight_midcorrelation

On Tue, Aug 18, 2015 at 5:31 PM Jeff Reback notifications@github.com
wrote:

> @olgabot https://github.com/olgabot ok, pls rebase, then can give
> comments
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9826#issuecomment-132402856.

BTW, working on a wikipedia article too: https://en.wikipedia.org/wiki/Draft:Biweight_midcorrelation
so whenever that is approved, I'll replace the pubmed link in the release notes

@olgabot you still have a rebase/merge mixup -- notice the 33 commits in this PR.

You might want to rebuild the branch (that's what I would do) or perhaps @jorisvandenbossche or @jreback knows the magical git command that would fix this one.

Oh hmm. 33 is many more commits than I actually did. Can you specify what you mean by "rebuild the branch"? 

@olgabot you should be able to simply `git rebase -i origin/master` and it will just pick out your commits.

Does that imply that "origin" is my branch? Currently, my origin is set as
git@github.com:olgabot/pandas, and upstream is git@github.com:pydata/pandas

On Wed, Aug 19, 2015 at 8:41 AM Jeff Reback notifications@github.com
wrote:

> @olgabot https://github.com/olgabot you should be able to simply git
> rebase -i origin/master and it will just pick out your commits.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9826#issuecomment-132642574.

`origin` is `upstream` (I usually use that as the name of the remote)

e.g.

`git remote -v` will show you

@olgabot that is correct. Typically origin is your fork (olgabot/pandas), upstream is the main repo (pydata/pandas)

So basically, doing:

```
git fetch upstream master
git rebase -i upstream/master
```

should always work

Okay rebased.. waiting for test results now!

@olgabot you have a failing test on Travis:

```
======================================================================
FAIL: test_nancorr_bicor (pandas.tests.test_nanops.TestnanopsDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_nanops.py", line 631, in test_nancorr_bicor
    method='bicor')
  File "/home/travis/build/pydata/pandas/pandas/tests/test_nanops.py", line 462, in check_nancorr_nancov_2d
    tm.assert_almost_equal(targ0, res00)
  File "testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3248)
  File "testing.pyx", line 197, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3035)
AssertionError: expected -0.06666 but got -0.48860, with decimal 5
```

Checks have passed so far. Is there anything else to do?

what does this do in the presence of Nan's? can u show (and test)

Can you show an example of a test that uses fixed numbers? I've got some examples from R but I'm wondering if they should use the exact random vectors generated by the random seed in  that file.

Sure -- for Pearson correlation coefficient, a good example would be Anscombe's quartet: https://en.wikipedia.org/wiki/Anscombe%27s_quartet

But it's useful also just to test trivial cases, where the result can be determined analytically, e.g., correlation between `[0, 0, 1]` and `[0, 1, 1]`.

You should certainly use seeds (or better yet, `np.random.RandomState`) for randomly generated tests.

In your wikipedia article (which is now live, congrats!) there is a question in the middle of the derivation: [How is the function denoted capital I defined?]