Repository: pandas-dev/pandas

Issue:
performance regression in ewm.corr(pairwise=True)
#### Problem description

The deprecation of Panel in the 0.20.x releases has introduced a severe performance regression in ewm.corr(pairwise=True) for a common case when this function is called on a long time series (e.g. a dataframe with 1 million rows and 6 columns).  The issue is the last 3 lines of code in this section of core/window.py:

```python
                # TODO: not the most efficient (perf-wise)
                # though not bad code-wise
                from pandas import Panel, MultiIndex, concat

                with warnings.catch_warnings(record=True):
                    p = Panel.from_dict(results).swapaxes('items', 'major')
                    if len(p.major_axis) > 0:
                        p.major_axis = arg1.columns[p.major_axis]
                    if len(p.minor_axis) > 0:
                        p.minor_axis = arg2.columns[p.minor_axis]

                if len(p.items):
                    result = concat(
                        [p.iloc[i].T for i in range(len(p.items))],
                        keys=p.items)
```

The result is converted from a Panel to a DataFrame by running a concat along an axis that is typically very long.  This is killing performance for me compared to the 0.19 releases.

My solution was to replace the last 3 lines with:

```python
                      result = DataFrame(
                          p.values.reshape((p.shape[0], p.shape[1]*p.shape[2])),
                          index=p.items,
                          columns=MultiIndex.from_product((arg1.columns, arg2.columns))
                      )
                      result = result.stack(dropna=False)
```

This works for me but I'm no pandas internals expert, so perhaps this solution does not work in all cases.

I'd really appreciate getting a workaround in though - clearly from the TODO comment the developers were at least aware this was a performance issue when they added the code.  I'm happy to rejig the above if it has problems.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-327.22.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: 2.8.5
pip: 9.0.1
setuptools: 26.1.1
Cython: 0.26
numpy: 1.13.3
scipy: 0.19.1
xarray: 0.8.2
IPython: 6.0.0
sphinx: 1.3.5
patsy: 0.4.0
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.8.4
lxml: 3.5.0
bs4: 4.4.1
html5lib: None
sqlalchemy: 1.0.11
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
if you replace this does it pass the tests? want to do a PR? (would also appreciate an asv for this).
What's an ASV?
http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-performance-test-suite
Thanks.

I realized my patch above won't work if arg1.columns or arg2.columns is a MultiIndex.  I can fix it but I need a way of combining two indices into a MultiIndex where one (or both) of the indices could be a MultiIndex.

I.e. if idx1 and idx2 are Index objects, then I need an expression to compute idx equivalent to:

```python
idx = concat([DataFrame([], columns=idx2)] * len(idx1), keys=idx1).index
```

The above is too slow since it creates too many unnecessary empty DataFrames.

If someone can provide me with a fast way of doing that I can get this working.  Apologies, this is probably the wrong venue for this...I don't ever submit patches.


not really sure what you need, pls show an example
```
In [15]: idx1 = pd.Index(list('abc'))

In [16]: idx2 = pd.Index([1, 2, 3])

In [17]: pd.MultiIndex.from_arrays([idx1, idx2])
Out[17]: 
MultiIndex(levels=[['a', 'b', 'c'], [1, 2, 3]],
           labels=[[0, 1, 2], [0, 1, 2]])

```

Resolving patch:
diff --git a/asv_bench/benchmarks/rolling.py b/asv_bench/benchmarks/rolling.py
--- a/asv_bench/benchmarks/rolling.py
+++ b/asv_bench/benchmarks/rolling.py
@@ -11,8 +11,8 @@ class Methods(object):
               [10, 1000],
               ['int', 'float'],
               ['median', 'mean', 'max', 'min', 'std', 'count', 'skew', 'kurt',
-               'sum', 'corr', 'cov'])
-    param_names = ['constructor', 'window', 'dtype', 'method']
+               'sum'])
+    param_names = ['contructor', 'window', 'dtype', 'method']
 
     def setup(self, constructor, window, dtype, method):
         N = 10**5
@@ -23,6 +23,27 @@ def time_rolling(self, constructor, window, dtype, method):
         getattr(self.roll, method)()
 
 
+class Pairwise(object):
+
+    sample_time = 0.2
+    params = ([10, 1000, None],
+              ['corr', 'cov'],
+              [True, False])
+    param_names = ['window', 'method', 'pairwise']
+
+    def setup(self, window, method, pairwise):
+        N = 10**4
+        arr = np.random.random(N)
+        self.df = pd.DataFrame(arr)
+
+    def time_pairwise(self, window, method, pairwise):
+        if window is None:
+            r = self.df.expanding()
+        else:
+            r = self.df.rolling(window=window)
+        getattr(r, method)(self.df, pairwise=pairwise)
+
+
 class Quantile(object):
 
     sample_time = 0.2
diff --git a/pandas/core/reshape/pivot.py b/pandas/core/reshape/pivot.py
--- a/pandas/core/reshape/pivot.py
+++ b/pandas/core/reshape/pivot.py
@@ -99,19 +99,15 @@ def pivot_table(data, values=None, index=None, columns=None, aggfunc='mean',
 
     if not dropna:
         from pandas import MultiIndex
-        try:
+        if table.index.nlevels > 1:
             m = MultiIndex.from_arrays(cartesian_product(table.index.levels),
                                        names=table.index.names)
             table = table.reindex(m, axis=0)
-        except AttributeError:
-            pass  # it's a single level
 
-        try:
+        if table.columns.nlevels > 1:
             m = MultiIndex.from_arrays(cartesian_product(table.columns.levels),
                                        names=table.columns.names)
             table = table.reindex(m, axis=1)
-        except AttributeError:
-            pass  # it's a single level or a series
 
     if isinstance(table, ABCDataFrame):
         table = table.sort_index(axis=1)
diff --git a/pandas/core/window.py b/pandas/core/window.py
--- a/pandas/core/window.py
+++ b/pandas/core/window.py
@@ -1863,25 +1863,38 @@ def dataframe_from_int_dict(data, frame_template):
                             results[i][j] = f(*_prep_binary(arg1.iloc[:, i],
                                                             arg2.iloc[:, j]))
 
-                # TODO: not the most efficient (perf-wise)
-                # though not bad code-wise
-                from pandas import Panel, MultiIndex, concat
-
-                with warnings.catch_warnings(record=True):
-                    p = Panel.from_dict(results).swapaxes('items', 'major')
-                    if len(p.major_axis) > 0:
-                        p.major_axis = arg1.columns[p.major_axis]
-                    if len(p.minor_axis) > 0:
-                        p.minor_axis = arg2.columns[p.minor_axis]
-
-                if len(p.items):
+                from pandas import MultiIndex, concat
+
+                result_index = arg1.index.union(arg2.index)
+                if len(result_index):
+
+                    # construct result frame
                     result = concat(
-                        [p.iloc[i].T for i in range(len(p.items))],
-                        keys=p.items)
+                        [concat([results[i][j]
+                                 for j, c in enumerate(arg2.columns)],
+                                ignore_index=True)
+                         for i, c in enumerate(arg1.columns)],
+                        ignore_index=True,
+                        axis=1)
+                    result.columns = arg1.columns
+
+                    # set the index and reorder
+                    if arg2.columns.nlevels > 1:
+                        result.index = MultiIndex.from_product(
+                            arg2.columns.levels + [result_index])
+                        result = result.reorder_levels([2, 0, 1]).sort_index()
+                    else:
+                        result.index = MultiIndex.from_product(
+                            [range(len(arg2.columns)),
+                             range(len(result_index))])
+                        result = result.swaplevel(1, 0).sort_index()
+                        result.index = MultiIndex.from_product(
+                            [result_index] + [arg2.columns])
                 else:
 
+                    # empty result
                     result = DataFrame(
-                        index=MultiIndex(levels=[arg1.index, arg1.columns],
+                        index=MultiIndex(levels=[arg1.index, arg2.columns],
                                          labels=[[], []]),
                         columns=arg2.columns,
                         dtype='float64')
@@ -1890,9 +1903,9 @@ def dataframe_from_int_dict(data, frame_template):
                 # reset our column names to arg2 names
                 # careful not to mutate the original names
                 result.columns = result.columns.set_names(
-                    arg2.columns.names)
+                    arg1.columns.names)
                 result.index = result.index.set_names(
-                    arg1.index.names + arg1.columns.names)
+                    result_index.names + arg2.columns.names)
 
                 return result
 


Test patch:
diff --git a/pandas/tests/test_window.py b/pandas/tests/test_window.py
--- a/pandas/tests/test_window.py
+++ b/pandas/tests/test_window.py
@@ -14,6 +14,7 @@
 import pandas.tseries.offsets as offsets
 from pandas.core.base import SpecificationError
 from pandas.errors import UnsupportedFunctionCall
+from pandas.core.sorting import safe_sort
 import pandas.util.testing as tm
 import pandas.util._test_decorators as td
 from pandas.compat import range, zip
@@ -1645,7 +1646,7 @@ def compare(self, result, expected):
         result = result.dropna().values
         expected = expected.dropna().values
 
-        tm.assert_numpy_array_equal(result, expected)
+        tm.assert_numpy_array_equal(result, expected, check_dtype=False)
 
     @pytest.mark.parametrize('f', [lambda x: x.cov(), lambda x: x.corr()])
     def test_no_flex(self, f):
@@ -1670,15 +1671,19 @@ def test_no_flex(self, f):
     def test_pairwise_with_self(self, f):
 
         # DataFrame with itself, pairwise=True
-        results = [f(df) for df in self.df1s]
-        for (df, result) in zip(self.df1s, results):
+        # note that we may construct the 1st level of the MI
+        # in a non-motononic way, so compare accordingly
+        results = []
+        for i, df in enumerate(self.df1s):
+            result = f(df)
             tm.assert_index_equal(result.index.levels[0],
                                   df.index,
                                   check_names=False)
-            tm.assert_index_equal(result.index.levels[1],
-                                  df.columns,
-                                  check_names=False)
+            tm.assert_numpy_array_equal(safe_sort(result.index.levels[1]),
+                                        safe_sort(df.columns.unique()))
             tm.assert_index_equal(result.columns, df.columns)
+            results.append(df)
+
         for i, result in enumerate(results):
             if i > 0:
                 self.compare(result, results[0])
@@ -1716,9 +1721,8 @@ def test_pairwise_with_other(self, f):
             tm.assert_index_equal(result.index.levels[0],
                                   df.index,
                                   check_names=False)
-            tm.assert_index_equal(result.index.levels[1],
-                                  self.df2.columns,
-                                  check_names=False)
+            tm.assert_numpy_array_equal(safe_sort(result.index.levels[1]),
+                                        safe_sort(self.df2.columns.unique()))
         for i, result in enumerate(results):
             if i > 0:
                 self.compare(result, results[0])