Repository: pandas-dev/pandas

Issue:
transform function is very slow in 0.14 - similar to issue #2121
I have a dataframe where I'm computing some summary items for groups and it's convenient to reconsolidate merge those back into the parent dataframe. Transform seems the most intuitive way to do this but I'm seeing much worse performance than doing a manual aggregation and merge. 

Starting from issue #2121 marked as closed: 

``` python
import numpy as np
import pandas as pd
from pandas import Index, MultiIndex, DataFrame

n_dates = 20000
n_securities = 50
n_columns = 2
share_na = 0.1

dates = pd.date_range('1997-12-31', periods=n_dates, freq='B')
dates = Index(map(lambda x: x.year * 10000 + x.month * 100 + x.day, dates))

secid_min = int('10000000', 16)
secid_max = int('F0000000', 16)
step = (secid_max - secid_min) // (n_securities - 1)
security_ids = map(lambda x: hex(x)[2:10].upper(), range(secid_min, secid_max + 1, step))

data_index = MultiIndex(levels=[dates.values],
    labels=[[i for i in xrange(n_dates) for _ in xrange(n_securities)]],
    names=['date'])
n_data = len(data_index)

columns = Index(['factor{}'.format(i) for i in xrange(1, n_columns + 1)])

data = DataFrame(np.random.randn(n_data, n_columns), index=data_index, columns=columns)

grouped = data.groupby(level='date')
```

which gives us a (1000000, 2) dataframe of floats

``` python
%timeit grouped.transform(max)
```

1 loops, best of 3: 8.16 s per loop

my alternative is to write something like this:

``` python
def mergeTransform(grouped, originalDF):
    item = grouped.max()
    colnames = item.columns
    item = originalDF.merge(item, how='outer', left_index=True, right_index=True).ix[:,-item.shape[1]:]
    item.columns = colnames
```

which seems to run much faster despite all the indexing going on:

``` python
%%timeit 
mergeTransform(grouped, data)
```

10 loops, best of 3: 81.1 **ms** per loop



Discussion:
see #6496 as well (it's possible the merge is faster in some cases)
depends on the group density - eg how many groups as a function of. the total size

would welcome a pull request for these solutions - it's not hard 

the existing transform will be faster of density is very high (eg lots of groups)

so may make sense to separate these 2 cases - but may not matter

#6496 is somewhat faster in this case (this only affects Series.transform really)

So this would work on `core/groupby.py/NDFrame.transform`

want to do a PR to see if you can do this type of merging soln in general?

e.g. put it in place and see if you can get the tests to pass?

we already have this benchmark in vbench, so you can see


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2701,27 +2701,7 @@ def _wrap_applied_output(self, keys, values, not_indexed_same=False):
             return self._concat_objects(keys, values,
                                         not_indexed_same=not_indexed_same)
 
-    def transform(self, func, *args, **kwargs):
-        """
-        Call function producing a like-indexed DataFrame on each group and
-        return a DataFrame having the same indexes as the original object
-        filled with the transformed values
-
-        Parameters
-        ----------
-        f : function
-            Function to apply to each subframe
-
-        Notes
-        -----
-        Each subframe is endowed the attribute 'name' in case you need to know
-        which group you are working on.
-
-        Examples
-        --------
-        >>> grouped = df.groupby(lambda x: mapping[x])
-        >>> grouped.transform(lambda x: (x - x.mean()) / x.std())
-        """
+    def _transform_general(self, func, *args, **kwargs):
         from pandas.tools.merge import concat
 
         applied = []
@@ -2763,6 +2743,66 @@ def transform(self, func, *args, **kwargs):
         concatenated.sort_index(inplace=True)
         return concatenated
 
+    def transform(self, func, *args, **kwargs):
+        """
+        Call function producing a like-indexed DataFrame on each group and
+        return a DataFrame having the same indexes as the original object
+        filled with the transformed values
+
+        Parameters
+        ----------
+        f : function
+            Function to apply to each subframe
+
+        Notes
+        -----
+        Each subframe is endowed the attribute 'name' in case you need to know
+        which group you are working on.
+
+        Examples
+        --------
+        >>> grouped = df.groupby(lambda x: mapping[x])
+        >>> grouped.transform(lambda x: (x - x.mean()) / x.std())
+        """
+
+        # try to do a fast transform via merge if possible
+        try:
+            obj = self._obj_with_exclusions
+            if isinstance(func, compat.string_types):
+                result = getattr(self, func)(*args, **kwargs)
+            else:
+                cyfunc = _intercept_cython(func)
+                if cyfunc and not args and not kwargs:
+                    result = getattr(self, cyfunc)()
+                else:
+                    return self._transform_general(func, *args, **kwargs)
+        except:
+            return self._transform_general(func, *args, **kwargs)
+
+        # a reduction transform
+        if not isinstance(result, DataFrame):
+            return self._transform_general(func, *args, **kwargs)
+
+        # nuiscance columns
+        if not result.columns.equals(obj.columns):
+            return self._transform_general(func, *args, **kwargs)
+
+        # a grouped that doesn't preserve the index, remap index based on the grouper
+        # and broadcast it
+        if not isinstance(obj.index,MultiIndex) and type(result.index) != type(obj.index):
+            results = obj.values.copy()
+            for (name, group), (i, row) in zip(self, result.iterrows()):
+                indexer = self._get_index(name)
+                results[indexer] = np.tile(row.values,len(indexer)).reshape(len(indexer),-1)
+            return DataFrame(results,columns=result.columns,index=obj.index).convert_objects()
+
+        # we can merge the result in
+        # GH 7383
+        names = result.columns
+        result = obj.merge(result, how='outer', left_index=True, right_index=True).ix[:,-result.shape[1]:]
+        result.columns = names
+        return result
+
     def _define_paths(self, func, *args, **kwargs):
         if isinstance(func, compat.string_types):
             fast_path = lambda group: getattr(group, func)(*args, **kwargs)
diff --git a/vb_suite/groupby.py b/vb_suite/groupby.py
--- a/vb_suite/groupby.py
+++ b/vb_suite/groupby.py
@@ -376,6 +376,7 @@ def f(g):
 """
 
 groupby_transform = Benchmark("data.groupby(level='security_id').transform(f_fillna)", setup)
+groupby_transform_ufunc = Benchmark("data.groupby(level='date').transform(np.max)", setup)
 
 setup = common_setup + """
 np.random.seed(0)
@@ -393,4 +394,4 @@ def f(g):
 df = DataFrame({ 'signal' : np.random.rand(N)})
 """
 
-groupby_transform2 = Benchmark("df['signal'].groupby(g).transform(np.mean)", setup)
+groupby_transform_series = Benchmark("df['signal'].groupby(g).transform(np.mean)", setup)


Test patch: