Repository: pandas-dev/pandas

Issue:
PERF: 6x perf hit from using numexpr
1 million x 3 dataframe, using numexpr takes 142ms to do multiplication, with numexpr disabled, takes 21.6ms. I'm going to investigate, but would be helpful to know if you can reproduce this anywhere else.

```
In [10]: df = pd.DataFrame({"A": np.arange(1000000), "B": np.arange(1000000, 0, -1), "C": np.random.randn(1000000)})

In [11]: df * df
Out[11]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
A    1000000  non-null values
B    1000000  non-null values
C    1000000  non-null values
dtypes: float64(1), int64(2)

In [12]: %timeit df * df
10 loops, best of 3: 142 ms per loop

In [13]: pd.computation.expressions.set_use_numexpr(False)

In [14]: %timeit df * df
10 loops, best of 3: 21.6 ms per loop

In [15]: pd.computation.expressions.set_use_numexpr(True)

In [16]: %timeit df * df
10 loops, best of 3: 141 ms per loop

In [17]: len(df)
Out[17]: 1000000
```



Discussion:
```
INSTALLED VERSIONS
------------------
Python: 2.7.5.final.0
OS: Darwin 12.4.0 Darwin Kernel Version 12.4.0: Wed May  1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.12.0-1081-ge684bdc
Cython: 0.19.1
Numpy: 1.7.1
Scipy: 0.12.0
statsmodels: Not installed
    patsy: Not installed
scikits.timeseries: Not installed
dateutil: 2.1
pytz: 2013d
bottleneck: Not installed
PyTables: 3.0.0
    numexpr: 2.2.1
matplotlib: 1.3.0
openpyxl: 1.6.2
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: Not installed
sqlalchemy: 0.8.2
lxml: 3.2.3
bs4: 4.3.2
html5lib: 1.0b3
bigquery: Not installed
apiclient: Not installed
```

installing bottleneck doesn't help (not that it would really make sense if it did)

I can reproduce: python 2.7.4, pandas 0.12.0-1083-g9e2c3d6, numpy 1.9.0.dev-8a2728c, numexpr 2.2.2:

```
>>> pd.computation.expressions.set_use_numexpr(False)
>>> %timeit df * df
10 loops, best of 3: 28.4 ms per loop
>>> pd.computation.expressions.set_use_numexpr(True)
>>> %timeit df * df
10 loops, best of 3: 91.2 ms per loop
```

So prun suggests that infer_dtype is the cost, I'm looking into it but I'm guessing that it has to do with the different return type from numexpr

Still happens in 0.12  as well

Issue is with mixed dtypes, float only and int only are both faster as expected. Maybe the cols are getting pushed together.

This is the result of multiplying dataframes 10 times by themselves (didn't divide time by 10):

Frames:

``` python
(pd.DataFrame({"A": np.arange(1000000), "B": np.arange(1000000, 0, -1), # mixed
                    "C": np.random.randn(1000000)}))
(pd.DataFrame({"A": np.arange(1000000), "B": np.arange(1000000, 0, -1)})) # int
(pd.DataFrame({"C": np.random.randn(1000000)})) # float
```

```
Mixed dtype
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
A    1000000  non-null values
B    1000000  non-null values
C    1000000  non-null values
dtypes: float64(1), int64(2)
No numexpr:0.396526098251
With numexpr1.45339894295

Int only
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 2 columns):
A    1000000  non-null values
B    1000000  non-null values
dtypes: int64(2)
No numexpr:0.120020866394
With numexpr0.0645599365234

Float only
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
C    1000000  non-null values
dtypes: float64(1)
No numexpr:0.0512380599976
With numexpr0.0369510650635
```


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2810,11 +2810,28 @@ def _arith_op(left, right):
             return func(left, right)
 
         if this._is_mixed_type or other._is_mixed_type:
-            # XXX no good for duplicate columns
-            # but cannot outer join in align if dups anyways?
-            result = {}
-            for col in this:
-                result[col] = _arith_op(this[col].values, other[col].values)
+
+            # unique
+            if this.columns.is_unique:
+
+                def f(col):
+                    r = _arith_op(this[col].values, other[col].values)
+                    return self._constructor_sliced(r,index=new_index,dtype=r.dtype)
+
+                result = dict([ (col, f(col)) for col in this ])
+
+            # non-unique
+            else:
+
+                def f(i):
+                    r = _arith_op(this.iloc[:,i].values, other.iloc[:,i].values)
+                    return self._constructor_sliced(r,index=new_index,dtype=r.dtype)
+
+                result = dict([ (i,f(i)) for i, col in enumerate(this.columns) ])
+                result = self._constructor(result, index=new_index, copy=False)
+                result.columns = new_columns
+                return result
+
         else:
             result = _arith_op(this.values, other.values)
 
@@ -2890,10 +2907,12 @@ def _compare(a, b):
         # non-unique
         else:
             def _compare(a, b):
-                return [func(a.iloc[:,i], b.iloc[:,i]) for i, col in enumerate(a.columns)]
+                return dict([(i,func(a.iloc[:,i], b.iloc[:,i])) for i, col in enumerate(a.columns)])
             new_data = expressions.evaluate(_compare, str_rep, self, other)
-            return self._constructor(data=new_data, index=self.columns,
-                                     columns=self.index, copy=False).T
+            result = self._constructor(data=new_data, index=self.index,
+                                       copy=False)
+            result.columns = self.columns
+            return result
 
     def _compare_frame(self, other, func, str_rep):
         if not self._indexed_same(other):


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -3217,6 +3217,15 @@ def check(result, expected=None):
             this_df['A'] = index
             check(this_df, expected_df)
 
+        # operations
+        for op in ['__add__','__mul__','__sub__','__truediv__']:
+            df = DataFrame(dict(A = np.arange(10), B = np.random.rand(10)))
+            expected = getattr(df,op)(df)
+            expected.columns = ['A','A']
+            df.columns = ['A','A']
+            result = getattr(df,op)(df)
+            check(result,expected)
+
     def test_column_dups_indexing(self):
 
         def check(result, expected=None):