Repository: pandas-dev/pandas

Issue:
BUG: resample unexpectedly branching on type of `how` callable
I was about to answer an SO question giving the standard "use `lambda` or `functools.partial`" to create a new function with bound arguments when to my surprise my example didn't work with the OP's code.  After some experimentation, it turns out to be because we don't always get the same return type.  For example:

```
df = pd.DataFrame(np.arange(5, dtype=np.int64), 
                  index=pd.DatetimeIndex(start='2014/01/01', periods=5, freq='d'))

def f(x,a=1):
    print(type(x))
    return int(isinstance(x, pd.DataFrame)) + 1000*a
```

After which:

```
>>> df.resample("M", how=f)
<class 'pandas.core.series.Series'>
               0
2014-01-31  1000
>>> df.resample("M", how=lambda x: f(x,5))
<class 'pandas.core.series.Series'>
               0
2014-01-31  5000
>>> df.resample("M", how=partial(f, a=9))
<class 'pandas.core.frame.DataFrame'>
               0
2014-01-31  9001
```

`how` shouldn't care about how the function was constructed, only whether it's callable, and we should be getting the same result (whether Series or DataFrame) fed into `how` in every case.



Discussion:
groupby first calls the function as a DataFrame (to evaluate it as a single block)

then will call it series by series if that doesn't work

I don't think this is a bug but rather state being captured in the closure

but u would have to step thru and see

What does "it doesn't work" mean here?  In each case, the `how` function is called exactly once, _and it's called with a different type object_.  So it's definitely not branching on anything that function is doing.

ok, a very odd bug

so, every function has a `__name__` attribute, but:

`partial(f,...)` does not, this ends up raising internally and takes another path (that is wrong).

The reason for this is convience feature I think something like this:

```
df = DataFrame({'foo' : [1,2], 'bar' :[3,4]})
df.groupby('foo').aggregate([np.sum,lambda x: x+1])
```

e.g. it figures out the name of the `np.sum` function is `sum` (duh!). the lambdas name is '<lambda>'
but a partial function doesn't have a name (not sure why)...

@dsm054 want to take a stab at fixing?

search for `__name__` and need to deal with that if doesn't exist. Could see if its a partial then look inside the partial to get the name I guess.

Class instances don't have a default `__name__`, so `functools.partial` instances aren't different from any other class which supports `__call__`.  We shouldn't care what's providing us an answer, only that it does..

Okay, I've got a working branch with tests, but am I alone in thinking the API for this is actually a little strange?  If I pass a `lambda` or a `partial` I don't really have a good way of changing the name, because the dict syntax is already being used to mean columns.  Anyway, setting that aside:

(1) What do we want to do for names of things which don't really have them?  At the moment we can't pass multiple lambdas.

(2) Is there a good place to toss a small utility function like `get_callable_name`?

The problem is that they NEED to be put in a dict ATM, somewhat related here: https://github.com/pydata/pandas/issues/6515

I think it might be better to simply number the columns if they don't have a name (and de-annomize the lambdas too).

e.g. my example above with `[np.sum, lambda x: x+1]` instead of yielding `['sum','<lambda>']`
really should yield `['sum',1]` I think (or maybe `['sum',0]`)

or maybe `['sum','aggregate_1']`

I think this is why @wesm punted on this, have to name them _something_.

1) dunno (but if you create a naming scheme then you prob _can_ have multiple lambdas)
2) maybe `core/common.py`?

We could add a layer of nesting, e.g. `{"neat_name": {"foo": sum}, "neat_name2": {"bar": lambda x: x.sum()}}` but even though that seems natural enough to me it does flip the meaning of the outermost keys.  Probably better to leave it alone, and add a different method if necessary.

If we're fixing duplicate names of the empty variety, should we also allow multiple sums (e.g. `sum`, `np.sum`) and use the same convention applied to duplicate columns when reading (e.g. `sum, sum.1`)?  If so, then the result of passing `sum, np.sum, lambda x: x.sum()` will be `sum`, `sum.1`, `aggregate_0` (or 1?), which I could live with.  If those columns have duplicates (say, because you already have a column with one of those names), then simply raise and leave it to the user to fix.

yep, I would be ok with `aggregate_0`, maybe could allow a `Series` which solves the problem nicely

`df.groupby(....).agg(Series([np.sum,sum,lambda x:x.sum()],['numpy_num','reg_sum','lambda_sum']))`

or a dict too (I don't think you need a _nested_ dict?

I like the Series idea.  Any objection to my submitting a PR to fix the original bug first, and then opening a followup ENH?


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -9,6 +9,7 @@
 import csv
 import types
 from datetime import datetime, timedelta
+from functools import partial
 
 from numpy.lib.format import read_array, write_array
 import numpy as np
@@ -2432,7 +2433,22 @@ def _is_sequence(x):
     except (TypeError, AttributeError):
         return False
 
-
+def _get_callable_name(obj):
+    # typical case has name
+    if hasattr(obj, '__name__'):
+        return getattr(obj, '__name__')
+    # some objects don't; could recurse
+    if isinstance(obj, partial):
+        return _get_callable_name(obj.func)
+    # fall back to class name
+    if hasattr(obj, '__call__'):
+        return obj.__class__.__name__
+    # everything failed (probably because the argument
+    # wasn't actually callable); we return None
+    # instead of the empty string in this case to allow
+    # distinguishing between no name and a name of ''
+    return None
+    
 _string_dtypes = frozenset(map(_get_dtype_from_object, (compat.binary_type,
                                                         compat.text_type)))
 
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1221,7 +1221,8 @@ def apply(self, f, data, axis=0):
         group_keys = self._get_group_keys()
 
         # oh boy
-        if (f.__name__ not in _plotting_methods and
+        f_name = com._get_callable_name(f)
+        if (f_name not in _plotting_methods and
                 hasattr(splitter, 'fast_apply') and axis == 0):
             try:
                 values, mutated = splitter.fast_apply(f, group_keys)
@@ -2185,11 +2186,11 @@ def _aggregate_multiple_funcs(self, arg):
                 if isinstance(f, compat.string_types):
                     columns.append(f)
                 else:
-                    columns.append(f.__name__)
+                    # protect against callables without names
+                    columns.append(com._get_callable_name(f))
             arg = lzip(columns, arg)
 
         results = {}
-
         for name, func in arg:
             if name in results:
                 raise SpecificationError('Function names must be unique, '


Test patch:
diff --git a/pandas/tests/test_common.py b/pandas/tests/test_common.py
--- a/pandas/tests/test_common.py
+++ b/pandas/tests/test_common.py
@@ -38,6 +38,26 @@ def __getitem__(self):
 
     assert(not is_seq(A()))
 
+def test_get_callable_name():
+    from functools import partial
+    getname = com._get_callable_name
+
+    def fn(x): 
+        return x
+    lambda_ = lambda x: x
+    part1 = partial(fn)
+    part2 = partial(part1)
+    class somecall(object):
+        def __call__(self):
+            return x
+
+    assert getname(fn) == 'fn'
+    assert getname(lambda_)
+    assert getname(part1) == 'fn'
+    assert getname(part2) == 'fn'
+    assert getname(somecall()) == 'somecall'
+    assert getname(1) is None
+
 
 def test_notnull():
     assert notnull(1.)
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -25,6 +25,7 @@
 from pandas.core.panel import Panel
 from pandas.tools.merge import concat
 from collections import defaultdict
+from functools import partial
 import pandas.core.common as com
 import numpy as np
 
@@ -2910,6 +2911,24 @@ def test_multi_function_flexible_mix(self):
         assert_frame_equal(result, expected)
         assert_frame_equal(result2, expected)
 
+    def test_agg_callables(self):
+        # GH 7929
+        df = DataFrame({'foo' : [1,2], 'bar' :[3,4]}).astype(np.int64)
+
+        class fn_class(object):
+            def __call__(self, x):
+                return sum(x)
+
+        equiv_callables = [sum, np.sum,
+                           lambda x: sum(x),
+                           lambda x: x.sum(),
+                           partial(sum), fn_class()]
+        
+        expected = df.groupby("foo").agg(sum)
+        for ecall in equiv_callables:
+            result = df.groupby('foo').agg(ecall)
+            assert_frame_equal(result, expected)
+
     def test_set_group_name(self):
         def f(group):
             assert group.name is not None
@@ -4530,6 +4549,8 @@ def test_transform_doesnt_clobber_ints(self):
         tm.assert_frame_equal(result, expected)
 
 
+
+
 def assert_fp_equal(a, b):
     assert (np.abs(a - b) < 1e-12).all()
 
diff --git a/pandas/tseries/tests/test_resample.py b/pandas/tseries/tests/test_resample.py
--- a/pandas/tseries/tests/test_resample.py
+++ b/pandas/tseries/tests/test_resample.py
@@ -1,6 +1,7 @@
 # pylint: disable=E1101
 
 from datetime import datetime, timedelta
+from functools import partial
 
 from pandas.compat import range, lrange, zip, product
 import numpy as np
@@ -140,6 +141,30 @@ def _ohlc(group):
                 exc.args += ('how=%s' % arg,)
                 raise
 
+    def test_resample_how_callables(self):
+        # GH 7929
+        data = np.arange(5, dtype=np.int64)
+        ind = pd.DatetimeIndex(start='2014-01-01', periods=len(data), freq='d')
+        df = pd.DataFrame({"A": data, "B": data}, index=ind)
+        
+        def fn(x, a=1):
+            return str(type(x))
+
+        class fn_class:
+            def __call__(self, x):
+                return str(type(x))
+
+        df_standard = df.resample("M", how=fn)
+        df_lambda = df.resample("M", how=lambda x: str(type(x)))
+        df_partial = df.resample("M", how=partial(fn))
+        df_partial2 = df.resample("M", how=partial(fn, a=2))
+        df_class = df.resample("M", how=fn_class())
+
+        assert_frame_equal(df_standard, df_lambda)
+        assert_frame_equal(df_standard, df_partial)
+        assert_frame_equal(df_standard, df_partial2)
+        assert_frame_equal(df_standard, df_class)
+        
     def test_resample_basic_from_daily(self):
         # from daily
         dti = DatetimeIndex(
@@ -765,6 +790,7 @@ def test_resample_timegrouper(self):
             assert_frame_equal(result, expected)
 
 
+
 def _simple_ts(start, end, freq='D'):
     rng = date_range(start, end, freq=freq)
     return Series(np.random.randn(len(rng)), index=rng)