Repository: pandas-dev/pandas

Issue:
Inconsistent result with applying function between dataframe/groupby apply and 0.12/0.13/master
I stumbled upon an inconsistent behaviour in (groupby/dataframe) apply when updating some code from 0.12 to 0.13.

Say you have following custom function:

```
def P1(a):
    try:
        return np.percentile(a.dropna(), q=1)
    except:
        return np.nan

def P1_withouttry(a):
    return np.percentile(a.dropna(), q=1)
```

When you apply this function to a dataframe:

```
In [3]: df = pd.DataFrame({'col1':[1,2,3,4],'col2':[10,25,26,31],                   
     ...:                    'date':[dt.date(2013,2,10),dt.date(2013,2,10),dt.date(2013,2,11),dt.date(2013,2,11)]})

In [4]: df
Out[4]:
   col1  col2        date
0     1    10  2013-02-10
1     2    25  2013-02-10
2     3    26  2013-02-11
3     4    31  2013-02-11

In [136]: df.apply(P1)
Out[136]: 
col1     1.03
col2    10.45
date      NaN
dtype: float64

In [138]: df.apply(P1_withouttry)
Traceback (most recent call last):
  ...
TypeError: ("unsupported operand type(s) for *: 'datetime.date' and 'float'", u'occurred at index date')
```

this does work with `P1`, but not with `P1_withouttry`. So I constructed my original function with a try/except to be able to apply this on a dataframe with also non-numeric columns.

However, when applying this on a groupby, it does not work anymore like this:

```
In [6]: g = df.groupby('date')

In [7]: g.apply(P1)
Out[7]:
date
2013-02-10   NaN
2013-02-11   NaN
dtype: float64

In [8]: g.apply(P1_withouttry)
Traceback (most recent call last):
   ...
TypeError: can't compare datetime.date to long


In [8]: g.agg(P1)
Out[8]:
            col1  col2
date
2013-02-10   NaN   NaN
2013-02-11   NaN   NaN

In [143]: g.agg(P1_withouttry)
Out[143]: 
            col1   col2
date                   
2013-02-10  1.01  10.15
2013-02-11  3.01  26.05
```

So, with `apply` it does not work, with `aggregate` it does, but only the `P1_withouttry` that didn't work with `df.apply()`. 
When using `g.agg([P1]`) this does work again on master, but not with 0.13.1 (then it gives the same as `g.agg(P1)`), although this did work
in 0.12:

```
In [11]: g.agg([P1])
Out[11]:
            col1   col2
              P1     P1
date
2013-02-10  1.01  10.15
2013-02-11  3.01  26.05
```

It was this last pattern I was using in my code in 0.12 that does not work anymore in 0.13.1 (I had something like `g.agg([P1, P5, P10, P25, np.median, np.mean])`).



Discussion:
this is actually quite complicated why it does this. It is trying to detect whether the passed function modifies the input data, which is basically if the returned shape/index is different OR an exception is raised (this is done in cython). In which case it falls back to essentially a loop where the function is tried. An exception here WILL raise though.

So the first application succeeds because of the try: except: as you trap the error, the 2nd raises at the appropriate place.

I am not sure that I can fix easily. I have something that sort of works, but breaks other things which are 'supposed' to raise, so sorting them will take some effort.


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1081,10 +1081,13 @@ def apply(self, f, data, axis=0):
             try:
                 values, mutated = splitter.fast_apply(f, group_keys)
                 return group_keys, values, mutated
-            except Exception:
+            except (lib.InvalidApply):
                 # we detect a mutation of some kind
                 # so take slow path
                 pass
+            except (Exception) as e:
+                # raise this error to the caller
+                pass
 
         result_values = []
         for key, (i, group) in zip(group_keys, splitter):
@@ -2295,7 +2298,15 @@ def aggregate(self, arg, *args, **kwargs):
             if self.grouper.nkeys > 1:
                 return self._python_agg_general(arg, *args, **kwargs)
             else:
-                result = self._aggregate_generic(arg, *args, **kwargs)
+
+                # try to treat as if we are passing a list
+                try:
+                    assert not args and not kwargs
+                    result = self._aggregate_multiple_funcs([arg])
+                    result.columns = Index(result.columns.levels[0],
+                                           name=self._selected_obj.columns.name)
+                except:
+                    result = self._aggregate_generic(arg, *args, **kwargs)
 
         if not self.as_index:
             if isinstance(result.index, MultiIndex):


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -1946,6 +1946,29 @@ def test_grouping_ndarray(self):
         expected = self.df.groupby('A').sum()
         assert_frame_equal(result, expected, check_names=False)  # Note: no names when grouping by value
 
+    def test_agg_consistency(self):
+        # agg with ([]) and () not consistent
+        # GH 6715
+
+        def P1(a):
+            try:
+                return np.percentile(a.dropna(), q=1)
+            except:
+                return np.nan
+
+        import datetime as dt
+        df = DataFrame({'col1':[1,2,3,4],
+                        'col2':[10,25,26,31],
+                        'date':[dt.date(2013,2,10),dt.date(2013,2,10),dt.date(2013,2,11),dt.date(2013,2,11)]})
+
+        g = df.groupby('date')
+
+        expected = g.agg([P1])
+        expected.columns = expected.columns.levels[0]
+
+        result = g.agg(P1)
+        assert_frame_equal(result, expected)
+
     def test_apply_typecast_fail(self):
         df = DataFrame({'d': [1., 1., 1., 2., 2., 2.],
                         'c': np.tile(['a', 'b', 'c'], 2),