Repository: pandas-dev/pandas

Issue:
Unexpected `transform` behavior on grouped dataset
I have a simple longitudinal biomedical dataset that I am grouping according to the patient on which measurements are taken. Here are the first couple of groups:

```
1
   patient  obs  week  site  id  treat  age sex  twstrs  treatment
0        1    1     0     1   1  5000U   65   F      32          1
1        1    2     2     1   1  5000U   65   F      30          1
2        1    3     4     1   1  5000U   65   F      24          1
3        1    4     8     1   1  5000U   65   F      37          1
4        1    5    12     1   1  5000U   65   F      39          1
5        1    6    16     1   1  5000U   65   F      36          1

2
    patient  obs  week  site  id   treat  age sex  twstrs  treatment
6         2    1     0     1   2  10000U   70   F      60          2
7         2    2     2     1   2  10000U   70   F      26          2
8         2    3     4     1   2  10000U   70   F      27          2
9         2    4     8     1   2  10000U   70   F      41          2
10        2    5    12     1   2  10000U   70   F      65          2
11        2    6    16     1   2  10000U   70   F      67          2
```

However, when I try to `transform` these data, say by normalization, I get nonsensical results:

```
normalize = lambda x: (x - x.mean())/x.std()
normed = cdystonia_grouped.transform(normalize)
normed.head(10)

               patient  obs  week                 site                   id  \
0 -9223372036854775808   -1    -1 -9223372036854775808 -9223372036854775808   
1 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808   
2 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808   
3 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808   
4 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808   

                   age  twstrs            treatment  
0 -9223372036854775808       0 -9223372036854775808  
1 -9223372036854775808       0 -9223372036854775808  
2 -9223372036854775808      -1 -9223372036854775808  
3 -9223372036854775808       0 -9223372036854775808  
4 -9223372036854775808       1 -9223372036854775808  
```

The `normalize` function is straightforward, and works fine when applied to manually subsetted data:

```
normalize(cdystonia.twstrs[cdystonia.patient==1])

0   -0.181369
1   -0.544107
2   -1.632322
3    0.725476
4    1.088214
5    0.544107
Name: twstrs, dtype: float64
```

Any guidance here much appreciated. I'm hoping its something obvious.



Discussion:
looks like u have some uint dtypes
can u show df.info()

also try this on master I just fixed the cause of this 

On Jun 2, 2013, at 4:40 PM, Chris Fonnesbeck notifications@github.com wrote:

> I have a simple longitudinal biomedical dataset that I am grouping according to the patient on which measurements are taken. Here are the first couple of groups:
> 
> 1
>    patient  obs  week  site  id  treat  age sex  twstrs  treatment
> 0        1    1     0     1   1  5000U   65   F      32          1
> 1        1    2     2     1   1  5000U   65   F      30          1
> 2        1    3     4     1   1  5000U   65   F      24          1
> 3        1    4     8     1   1  5000U   65   F      37          1
> 4        1    5    12     1   1  5000U   65   F      39          1
> 5        1    6    16     1   1  5000U   65   F      36          1
> 
> 2
>     patient  obs  week  site  id   treat  age sex  twstrs  treatment
> 6         2    1     0     1   2  10000U   70   F      60          2
> 7         2    2     2     1   2  10000U   70   F      26          2
> 8         2    3     4     1   2  10000U   70   F      27          2
> 9         2    4     8     1   2  10000U   70   F      41          2
> 10        2    5    12     1   2  10000U   70   F      65          2
> 11        2    6    16     1   2  10000U   70   F      67          2
> However, when I try to transform these data, say by normalization, I get nonsensical results:
> 
> normalize = lambda x: (x - x.mean())/x.std()
> normed = cdystonia_grouped.transform(normalize)
> normed.head(10)
> 
> ```
>            patient  obs  week                 site                   id  \
> ```
> 
> 0 -9223372036854775808   -1    -1 -9223372036854775808 -9223372036854775808  
> 1 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808  
> 2 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808  
> 3 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808  
> 4 -9223372036854775808    0     0 -9223372036854775808 -9223372036854775808   
> 
> ```
>                age  twstrs            treatment  
> ```
> 
> 0 -9223372036854775808       0 -9223372036854775808  
> 1 -9223372036854775808       0 -9223372036854775808  
> 2 -9223372036854775808      -1 -9223372036854775808  
> 3 -9223372036854775808       0 -9223372036854775808  
> 4 -9223372036854775808       1 -9223372036854775808  
> The normalize function is straightforward, and works fine when applied to manually subsetted data:
> 
> normalize(cdystonia.twstrs[cdystonia.patient==1])
> 
> 0   -0.181369
> 1   -0.544107
> 2   -1.632322
> 3    0.725476
> 4    1.088214
> 5    0.544107
> Name: twstrs, dtype: float64
> Any guidance here much appreciated. I'm hoping its something obvious.
> 
> —
> Reply to this email directly or view it on GitHub.

Here is an example. This is in 0.11.0

```
In [16]: df = DataFrame(dict(A = Series([1]*5,dtype='uint8'), B = Series([0]*5,dtype='uint64'),C=np.random.randint(-10,10,size=5)))

In [17]: df
Out[17]: 
   A  B  C
0  1  0  8
1  1  0 -3
2  1  0  9
3  1  0  8
4  1  0  6

In [18]: df.values
Out[18]: 
array([[                   1,                    0,                    8],
       [                   1,                    0, 18446744073709551613],
       [                   1,                    0,                    9],
       [                   1,                    0,                    8],
       [                   1,                    0,                    6]], dtype=uint64)
```

0.11.1 this works

```
In [1]: df = DataFrame(dict(A = Series([1]*5,dtype='uint8'), B = Series([0]*5,dtype='uint64'),C=np.random.randint(-10,10,size=5)))

In [2]: df.values
Out[2]: 
array([[ 1,  0, -8],
       [ 1,  0,  2],
       [ 1,  0, -3],
       [ 1,  0,  5],
       [ 1,  0,  5]])

```

My DataFrame info is as follows:

```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 631 entries, 0 to 630
Data columns (total 10 columns):
patient      631  non-null values
obs          631  non-null values
week         631  non-null values
site         631  non-null values
id           631  non-null values
treat        631  non-null values
age          631  non-null values
sex          631  non-null values
twstrs       631  non-null values
treatment    631  non-null values
dtypes: int64(8), object(2)
```

So, `sex` and `treat` are string variables, but the others are valid int64, which ought to nornalize properly. I updated to the current master, but the result is the same.

can u give me a Dropbox link to the frame as a csv?

[Here it is](https://www.dropbox.com/s/kqp599stc502ecf/cdystonia.csv)


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1532,6 +1532,9 @@ def transform(self, func, *args, **kwargs):
         transformed : Series
         """
         result = self.obj.copy()
+        if hasattr(result,'values'):
+            result = result.values
+        dtype = result.dtype
 
         if isinstance(func, basestring):
             wrapper = lambda x: getattr(x, func)(*args, **kwargs)
@@ -1539,13 +1542,21 @@ def transform(self, func, *args, **kwargs):
             wrapper = lambda x: func(x, *args, **kwargs)
 
         for name, group in self:
+
+            group = com.ensure_float(group)
             object.__setattr__(group, 'name', name)
             res = wrapper(group)
-            # result[group.index] = res
             indexer = self.obj.index.get_indexer(group.index)
-            np.put(result, indexer, res)
+            if hasattr(res,'values'):
+                res = res.values
 
-        return result
+            # need to do a safe put here, as the dtype may be different
+            # this needs to be an ndarray
+            result,_ = com._maybe_upcast_indexer(result, indexer, res)
+
+        # downcast if we can (and need)
+        result = _possibly_downcast_to_dtype(result, dtype)
+        return self.obj.__class__(result,index=self.obj.index,name=self.obj.name)
 
 
 class NDFrameGroupBy(GroupBy):


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -616,6 +616,39 @@ def f(x):
             assert_series_equal(agged, expected, check_dtype=False)
             self.assert_(issubclass(agged.dtype.type, np.dtype(dtype).type))
 
+    def test_groupby_transform_with_int(self):
+
+        # GH 3740, make sure that we might upcast on item-by-item transform
+
+        # floats
+        df = DataFrame(dict(A = [1,1,1,2,2,2], B = Series(1,dtype='float64'), C = Series([1,2,3,1,2,3],dtype='float64'), D = 'foo'))
+        result = df.groupby('A').transform(lambda x: (x-x.mean())/x.std())
+        expected = DataFrame(dict(B = np.nan, C = Series([-1,0,1,-1,0,1],dtype='float64')))
+        assert_frame_equal(result,expected)
+
+        # int case
+        df = DataFrame(dict(A = [1,1,1,2,2,2], B = 1, C = [1,2,3,1,2,3], D = 'foo'))
+        result = df.groupby('A').transform(lambda x: (x-x.mean())/x.std())
+        expected = DataFrame(dict(B = np.nan, C = [-1,0,1,-1,0,1]))
+        assert_frame_equal(result,expected)
+
+        # int that needs float conversion
+        s = Series([2,3,4,10,5,-1])
+        df = DataFrame(dict(A = [1,1,1,2,2,2], B = 1, C = s, D = 'foo'))
+        result = df.groupby('A').transform(lambda x: (x-x.mean())/x.std())
+
+        s1 = s.iloc[0:3]
+        s1 = (s1-s1.mean())/s1.std()
+        s2 = s.iloc[3:6]
+        s2 = (s2-s2.mean())/s2.std()
+        expected = DataFrame(dict(B = np.nan, C = concat([s1,s2])))
+        assert_frame_equal(result,expected)
+
+        # int downcasting
+        result = df.groupby('A').transform(lambda x: x*2/2)
+        expected = DataFrame(dict(B = 1, C = [2,3,4,10,5,-1]))
+        assert_frame_equal(result,expected)
+
     def test_indices_concatenation_order(self):
 
         # GH 2808