Repository: pandas-dev/pandas

Issue:
agg() swallows exceptions.
agg() appears to swallow exceptions.
This was the source of a hard-to-find bug today.
Here's a program that replicates the problem.

```
#!/usr/bin/env python                                                                                                                                                                                                                         
from pandas import *
from numpy.random import randint

df = DataFrame(randint(10, size=(20, 10)))

def raiseException(df):
  print '----------------------------------------'
  print df.to_string()
  raise TypeError

df.groupby(0).agg(raiseException)
```



Discussion:
Try raising `Exception` (instead of `TypeError`), and your example will work. ``TypeError` is caught internally in order to deal with issues like trying to perform a `sum` on non-numeric data. Prob ought to be a doc mention on this

I wasn't raising a TypeError myself; calling a function with an incorrect number of parameters raises a TypeError. Your solution is not acceptable.

what exactly are you trying to accomplish? maybe put up a sample frame, function that is closer to what you are doing

I have a custom aggregator function. This function calls another function; the parameter list was wrong. The error was ignored silently! It should not.

@wesm ?

agg tries to be flexible by applying the aggregator to the entire dataset
at once, and then fallback to doing an item-by-item apply.
This can yield a big performance win if vectorization can be employed, especially in 
groupby operations which tend to be large.

Unfortunately, it's hard to predict what kind of Error will be raised by the user-provided
aggregator if the first test fails, hence the broad exception clause.

Although I'm not very fond of this sort of magical behaviour for exactly the reason you
exemplified, I don't see a way to fix that without potentially breaking a lot of code, and 
`TypeError` is specifically something you would expect to see in this situation.

If you have a solution that does not break existing code, let's discuss it.

I like the current magic, but maybe there could be some kind of raise_exceptions flag (which defaults to False)...

This is a debugging issue, it doesn't make sense to crudify signatures with "dev-time only"
args, and that would open the flood gates to similiar signature noise across the lib.

IMO.

what about raising an exception if none of the items can be agg'ed while agg'ing item by item? It shouldn't break existing code (much) and it should solve the above case.
i know it's just adding on more magic but i can't really think of a general/clean solution off the top of my head.

Let me have a look at this soon and let you know what I think. 

There's one unit test that breaks by no longer suppressing the TypeError but on inspection I'd rather not support the behavior-- i.e. to be able to pass an aggregation function that ONLY works on the string columns (but fails on the numeric columns). The auto-exclusion of non-numeric data when doing, say, `df.groupby(keys).agg(np.sum)` was only conceived as a convenience to begin with.

Any objections?


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -438,7 +438,7 @@ def _try_cast(self, result, obj):
                 dtype = obj.dtype
 
             if _is_numeric_dtype(dtype):
-                
+
                 # need to respect a non-number here (e.g. Decimal)
                 if len(result) and issubclass(type(result[0]),(np.number,float,int)):
                     result = _possibly_downcast_to_dtype(result, dtype)
@@ -494,7 +494,7 @@ def _python_agg_general(self, func, *args, **kwargs):
                 values = result
                 if _is_numeric_dtype(values.dtype):
                     values = com.ensure_float(values)
-                
+
                 output[name] = self._try_cast(values[mask],result)
 
         return self._wrap_aggregated_output(output)
@@ -794,7 +794,7 @@ def wrapper(*args, **kwargs):
 
                 # need to curry our sub-function
                 func = wrapper
-                
+
         else:
             func = get_func(ftype)
 
@@ -1778,7 +1778,7 @@ def _aggregate_item_by_item(self, func, *args, **kwargs):
                 colg = SeriesGroupBy(obj[item], selection=item,
                                      grouper=self.grouper)
                 result[item] = colg.aggregate(func, *args, **kwargs)
-            except (ValueError, TypeError):
+            except ValueError:
                 cannot_agg.append(item)
                 continue
 
@@ -1917,7 +1917,7 @@ def transform(self, func, *args, **kwargs):
                     # if we make it here, test if we can use the fast path
                     try:
                         res_fast = fast_path(group)
-                    
+
                         # compare that we get the same results
                         if res.shape == res_fast.shape:
                             res_r = res.values.ravel()
@@ -1925,7 +1925,7 @@ def transform(self, func, *args, **kwargs):
                             mask = notnull(res_r)
                             if (res_r[mask] == res_fast_r[mask]).all():
                                 path = fast_path
-                
+
                     except:
                         pass
                 except TypeError:


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -121,7 +121,7 @@ def checkit(dtype):
             agged = grouped.aggregate([np.mean, np.std])
             agged = grouped.aggregate({'one': np.mean,
                                        'two': np.std})
-            
+
             group_constants = {
                 0: 10,
                 1: 20,
@@ -425,10 +425,11 @@ def test_aggregate_item_by_item(self):
         df['E'] = ['a'] * len(self.df)
         grouped = self.df.groupby('A')
 
-        def aggfun(ser):
-            return len(ser + 'a')
-        result = grouped.agg(aggfun)
-        self.assertEqual(len(result.columns), 1)
+        # API change in 0.11
+        # def aggfun(ser):
+        #     return len(ser + 'a')
+        # result = grouped.agg(aggfun)
+        # self.assertEqual(len(result.columns), 1)
 
         aggfun = lambda ser: ser.size
         result = grouped.agg(aggfun)
@@ -444,6 +445,19 @@ def aggfun(ser):
         self.assert_(isinstance(result, DataFrame))
         self.assertEqual(len(result), 0)
 
+    def test_agg_item_by_item_raise_typeerror(self):
+        from numpy.random import randint
+
+        df = DataFrame(randint(10, size=(20, 10)))
+
+        def raiseException(df):
+          print '----------------------------------------'
+          print df.to_string()
+          raise TypeError
+
+        self.assertRaises(TypeError, df.groupby(0).agg,
+                          raiseException)
+
     def test_basic_regression(self):
         # regression
         T = [1.0 * x for x in range(1, 10) * 10][:1095]
@@ -614,7 +628,7 @@ def f3(x):
         df = DataFrame({'a':[1,2,2,2],
                         'b':range(4),
                         'c':range(5,9)})
-        
+
         df2 = DataFrame({'a':[3,2,2,2],
                          'b':range(4),
                          'c':range(5,9)})
@@ -624,7 +638,7 @@ def f3(x):
         result1 = df.groupby('a').apply(f1)
         result2 = df2.groupby('a').apply(f1)
         assert_frame_equal(result1, result2)
-        
+
         # should fail (not the same number of levels)
         self.assertRaises(AssertionError, df.groupby('a').apply, f2)
         self.assertRaises(AssertionError, df2.groupby('a').apply, f2)