DataFrame.corrwith() method now supports 'how', 'method' and 'min_periods' params
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

A method `DataFrame.corrmatrix()` creates a correlation matrix between
all columns of the first DataFrame object and all columns of the second
DataFrame object. It differs from `corrwith()` in that way it computes correlations between all pairs of first and second columns (not only that has the same names).


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @Mottl! Thanks for updating the PR.



 - In the file [`pandas/tests/frame/test_analytics.py`](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py), following are the PEP8 issues :

> [Line 471:46](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L471): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 472:52](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L472): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 472:55](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L472): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 472:60](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L472): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 483:45](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L483): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 483:63](https://github.com/pandas-dev/pandas/blob/a2b71008a4e6e5176e89f6cdc2cf3172a0ed31e8/pandas/tests/frame/test_analytics.py#L483): [E226](https://duckduckgo.com/?q=pep8%20E226) missing whitespace around arithmetic operator



##### Comment last updated on December 25, 2018 at 11:53 Hours UTC
what is the utility of this? we are loathe to add additional api
@jreback
The utility is simple:  
during the feature selection process it is not desirable to spend time to get a full correlation matrix when the number of features is huge. It's more important to calculate a correlation matrix between M number of features and Y number of targets (we have a variety of different targets and we want to understand how features correlate to each of them) and `corrmatrix()` makes it simple.

So `corrmatrix()` is a simple form of `corr()` but without unnecessary calculations which will be dropped anyway.
This is how most people solve this problem:
https://stackoverflow.com/questions/34896455/how-to-do-pearson-correlation-of-selected-columns-of-a-pandas-data-frame
viewed: 24,989 times!

They calculate the full correlation matrix and drop all the unnecessary columns or make a lambda function, but it works only for correlation between DataFrame and Series, not with two DataFrames.



# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=h1) Report
> Merging [#22622](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fc7bc3f74d1f5702bf66c519ad190126538a8f5b?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `96.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22622/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22622      +/-   ##
==========================================
+ Coverage    92.3%    92.3%   +<.01%     
==========================================
  Files         163      163              
  Lines       51943    51962      +19     
==========================================
+ Hits        47946    47964      +18     
- Misses       3997     3998       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.71% <96.66%> (ø)` | :arrow_up: |
| #single | `42.99% <3.33%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22622/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.87% <96.66%> (-0.04%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/22622/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.31% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/22622/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.62% <0%> (ø)` | :arrow_up: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/22622/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `85.58% <0%> (+0.26%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=footer). Last update [fc7bc3f...a2b7100](https://codecov.io/gh/pandas-dev/pandas/pull/22622?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

-1 on this api as it will make things even more confusing
@jreback, how about adding new param to the existing `.corrwith()` function to compute all pair correlations between DataFrames instead of name-like?
@Mottl would be more ameanable to an extension to an existing method. though pls put up an example of what you want to do with copy-pastable code (and the 'work-around')
We could add the `which` param to `DataFrame.corrwith()` method.
`which='same'` for the current behaviour (returns a Series with pair-wise correlations between columns or rows with the same name/index)
`which='all'` to produce a correlation matrix (returns a DataFrame with pair-wise correlations between columns (or rows) of all possible combinations)

Example:
```python
 index = ['a', 'b', 'c', 'd', 'e']
 columns1 = ['one', 'two']
 columns2 = ['three', 'four']
 df1 = pd.DataFrame(np.random.randn(5, 2), index=index, columns=columns1)
 df2 = pd.DataFrame(np.random.randn(4, 2), index=index[:4], columns=columns2)
 df1.corrwith(df2, which='all')
```

@jreback, what are we doing with `df1.corrwith()`? Changing as I described above?
can u show the output of your example above with both options
@jreback

```python
import numpy as np
import pandas as pd
df1 = pd.DataFrame({'a':np.random.randn(10), 'b':np.random.randn(10)})
df2 = pd.DataFrame({'a':np.random.randn(10), 'c':np.random.randn(10)})

df1.corrwith(df2)  # default, which='same'
```
```
a    0.40657
b        NaN
c        NaN
dtype: float64
```
```python
df1.corrwith(df2, which='all')  # new
```
```
          a         c
a  0.406570 -0.005083
b -0.231165 -0.318797
```


@Mottl ok I think adding a parameter ``how='pairwise|all'`` might be ok here.

cc @jorisvandenbossche 
@jreback, what should I do with the cancelled Circleci jobs? Is the PR well enough since Travis is green?
i cancelled those jobs

I know
> @Mottl ok I think adding a parameter `how='pairwise|all'` might be ok here.
> cc @jorisvandenbossche

@jreback, @jorisvandenbossche
please review

conceptually this might be ok, but I think you have killed performance for the common case. Can you show the asv's.?
```python
import numpy as np
import pandas as pd
df1 = pd.DataFrame(np.random.randn(1000,30))
df2 = pd.DataFrame(np.random.randn(1000,30))

%timeit df1.corrwith(df2)
23 ms ± 1.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit df1.corrwith_orig(df2)
34.7 ms ± 2.17 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Actually, the **new implementation is 34% faster** than the old one since it uses `pd.Series.corr()` which in turn uses numpy's `corrcoef()` instead of `left - left.mean()`, `right - right.mean()`, etc.
try with a large number of columns
Same — **33% faster**:
```python
df1 = pd.DataFrame(np.random.randn(10000,3000))
df2 = pd.DataFrame(np.random.randn(10000,3000))

%timeit df1.corrwith(df2)
5.7 s ± 81.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit df1.corrwith_orig(df2)
8.45 s ± 152 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

>iterating over the pair of the columns is exactly what the nancorr routines seeks to avoid. can we just directly use that here? this is not scaling very well.

You want to duplicate lines from pd.Series.corr() to pd.DataFrame.corrwith()? What is the reason for that?

This is the `pd.Series.corr()` implementation and it looks just fine to reuse it:
```python
def corr(self, other, method='pearson', min_periods=None):
    this, other = self.align(other, join='inner', copy=False)
    if len(this) == 0:
        return np.nan

    if method in ['pearson', 'spearman', 'kendall']:
        return nanops.nancorr(this.values, other.values, method=method,
                              min_periods=min_periods)
```

@jreback 
Please, review
can you merge master and update to comments.
@jreback 
merged
@jreback
Let's do the things simpler — just fix what you want. 
@Mottl the point is that the community does these PRs
if you don’t want to complete then close this