Repository: pandas-dev/pandas

Issue:
 An apply on a DataFrame along axis=1 breaks aggregations on groupby's of that DataFrame
This seems to be a regression from 0.10.1 to 0.11.0

This works:

``` python
from pandas import DataFrame
df = DataFrame({'foo1' : ['one', 'two', 'two', 'three', 'one', 'two'],
                'foo2' : np.random.randn(6)})
grouped = df.groupby('foo1')
grouped.agg(['mean'])
```

This does not (notice the apply):

``` python
from pandas import DataFrame
df = DataFrame({'foo1' : ['one', 'two', 'two', 'three', 'one', 'two'],
                'foo2' : np.random.randn(6)})
df = df.apply(lambda x: x, axis=1)
grouped = df.groupby('foo1')
grouped.agg(['mean'])
```

The error raised is:

``` python
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-47-4df234603be2> in <module>()
----> 1 grouped.agg(['mean'])

/Users/gkoller/.virtualenvs/abo/lib/python2.6/site-packages/pandas/core/groupby.pyc in agg(self, func, *args, **kwargs)
    336     @Appender(_agg_doc)
    337     def agg(self, func, *args, **kwargs):
--> 338         return self.aggregate(func, *args, **kwargs)
    339 
    340     def _iterate_slices(self):

/Users/gkoller/.virtualenvs/abo/lib/python2.6/site-packages/pandas/core/groupby.pyc in aggregate(self, arg, *args, **kwargs)
   1688                 result = DataFrame(result)
   1689         elif isinstance(arg, list):
-> 1690             return self._aggregate_multiple_funcs(arg)
   1691         else:
   1692             cyfunc = _intercept_cython(arg)

/Users/gkoller/.virtualenvs/abo/lib/python2.6/site-packages/pandas/core/groupby.pyc in _aggregate_multiple_funcs(self, arg)
   1736             except SpecificationError:
   1737                 raise
-> 1738         result = concat(results, keys=keys, axis=1)
   1739 
   1740         return result

/Users/gkoller/.virtualenvs/abo/lib/python2.6/site-packages/pandas/tools/merge.pyc in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity)
    870                        ignore_index=ignore_index, join=join,
    871                        keys=keys, levels=levels, names=names,
--> 872                        verify_integrity=verify_integrity)
    873     return op.get_result()
    874 

/Users/gkoller/.virtualenvs/abo/lib/python2.6/site-packages/pandas/tools/merge.pyc in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity)
    911 
    912         if len(objs) == 0:
--> 913             raise Exception('All objects passed were None')
    914 
    915         # consolidate data

Exception: All objects passed were None
```



Discussion:
this doesn't work because:

```
In [8]: df.dtypes
Out[8]: 
foo1     object
foo2    float64
dtype: object

In [9]: df.apply(lambda x: x,axis=1).dtypes
Out[9]: 
foo1    object
foo2    object
dtype: object

```

I you think about it, your are aggregating across the columns which are mixed dtypes, so your output will be of dtype `object`. you can do this to convert back.

```
In [10]: df.apply(lambda x: x,axis=1).convert_objects()
Out[10]: 
    foo1      foo2
0    one -0.849480
1    two  0.038060
2    two  0.714368
3  three  0.522911
4    one -1.706384
5    two -0.694232

In [11]: df.apply(lambda x: x,axis=1).convert_objects().dtypes
Out[11]: 
foo1     object
foo2    float64
dtype: object
```

I am not sure this is a bug. We try to use the original dtypes after an apply, but it is not always possible.

Thank you for the clear explanation. Given this explanation I'm not sure either this is a bug; it seems logical and predictable.

However it did work in version 0.10.1 (I just verified it once more). So I presume version 0.10.1 did the conversion automatically. The change in behavior in 0.11.0  broke existing code. If it isn't a regression it should arguably be documented as a backwards incompatible change.

on 2nd thought I think this is easily fixible......


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -4210,7 +4210,7 @@ def _apply_standard(self, func, axis, ignore_failures=False):
 
             if axis == 1:
                 result = result.T
-            result = result.convert_objects(convert_dates=False)
+            result = result.convert_objects()
 
             return result
         else:


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -1826,6 +1826,13 @@ def convert_force_pure(x):
         self.assert_(result.dtype == np.object_)
         self.assert_(isinstance(result[0], Decimal))
 
+    def test_apply_with_mixed_dtype(self):
+        # GH3480, apply with mixed dtype on axis=1 breaks in 0.11
+        df = DataFrame({'foo1' : ['one', 'two', 'two', 'three', 'one', 'two'],
+                        'foo2' : np.random.randn(6)})
+        result = df.apply(lambda x: x, axis=1)
+        assert_series_equal(df.get_dtype_counts(), result.get_dtype_counts())
+
     def test_groupby_list_infer_array_like(self):
         result = self.df.groupby(list(self.df['A'])).mean()
         expected = self.df.groupby(self.df['A']).mean()