Repository: pandas-dev/pandas

Issue:
BUG: groupby.transform length mismatch under certain specifications
Replicating Example (pandas 15.2 and 0.16.0rc1-32-g5a417ec):

```
import numpy as np
df = pd.DataFrame({'col1':[1,1,2,2], 'col2':[1,2,3,np.nan]})

# Works fine
df.groupby('col1').transform(sum)['col2']

# Throws error
df.groupby('col1')['col2'].transform(sum)
```

Error:
    Traceback (most recent call last):

```
  File "<ipython-input-7-f969e26273d4>", line 8, in <module>
    df.groupby('col1')['col2'].transform(sum)

  File "/Users/Nick/GitHub/pandas/pandas/core/groupby.py", line 2418, in transform
    return self._transform_fast(cyfunc)

  File "/Users/Nick/GitHub/pandas/pandas/core/groupby.py", line 2459, in _transform_fast
    return self._set_result_index_ordered(Series(values))

  File "/Users/Nick/GitHub/pandas/pandas/core/groupby.py", line 497, in _set_result_index_ordered
    result.index = self.obj.index

  File "/Users/Nick/GitHub/pandas/pandas/core/generic.py", line 2061, in __setattr__
    return object.__setattr__(self, name, value)

  File "pandas/src/properties.pyx", line 65, in pandas.lib.AxisProperty.__set__ (pandas/lib.c:41404)
    obj._set_axis(self.axis, value)

  File "/Users/Nick/GitHub/pandas/pandas/core/series.py", line 268, in _set_axis
    self._data.set_axis(axis, labels)

  File "/Users/Nick/GitHub/pandas/pandas/core/internals.py", line 2211, in set_axis
    'new values have %d elements' % (old_len, new_len))

ValueError: Length mismatch: Expected axis has 3 elements, new values have 4 elements
```



Discussion:
Note that 

```
>>> df.groupby("col1")["col2"].transform(lambda x: np.nansum(x))
0    3
1    3
2    3
3    3
Name: col2, dtype: float64
```

works, so the bug must be in the fastpath.

More information: the use of `count` in `_transform_fast` doesn't seem right to me (that'll pick up the number of non-nan values in the grouped series, IIUC, whereas we want the size of the groups.)

We're also seeing things like

```
>>> df = pd.DataFrame({"c1": [1], "c2": [2]})
>>> df
   c1  c2
0   1   2
>>> df.groupby("c1").transform(sum)
   c2
0 NaN
1   2
```

which makes me think there's definitely been a confusion about the index.

yep, this https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L2456
should be `self.size().values`. Its the group size here that matters.

Switching count -> size seems to fix the original problem.  Doesn't pass the tests I made which revealed the index problem above.  Shall I submit a PR with the count/size switch here and open a new issue for that one?


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2453,7 +2453,7 @@ def _transform_fast(self, func):
         if isinstance(func, compat.string_types):
             func = getattr(self,func)
         values = func().values
-        counts = self.count().values
+        counts = self.size().values
         values = np.repeat(values, com._ensure_platform_int(counts))
 
         return self._set_result_index_ordered(Series(values))


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -1058,6 +1058,19 @@ def test_transform_function_aliases(self):
         expected = self.df.groupby('A')['C'].transform(np.mean)
         assert_series_equal(result, expected)
 
+    def test_transform_length(self):
+        # GH 9697
+        df = pd.DataFrame({'col1':[1,1,2,2], 'col2':[1,2,3,np.nan]})
+        expected = pd.Series([3.0]*4)
+        def nsum(x):
+            return np.nansum(x)
+        results = [df.groupby('col1').transform(sum)['col2'],
+                   df.groupby('col1')['col2'].transform(sum),
+                   df.groupby('col1').transform(nsum)['col2'],
+                   df.groupby('col1')['col2'].transform(nsum)]
+        for result in results:
+            assert_series_equal(result, expected)
+
     def test_with_na(self):
         index = Index(np.arange(10))