Repository: pandas-dev/pandas

Issue:
BUG: DataFrame.agg({'col': 'size'}) not working
`size` aggregation does not seem to work properly when used with ungrouped dataframes. When replacing `size` with `count` the examples run through. However, this change null semantics, counting nulls (`size`) in contrast to skipping nulls (`count`). The examples below are for Pandas version `0.20.1`.

When used with `.groupby(...)`, size aggregation works as expected:

```python
>>> pd.DataFrame({'g': [0, 0, 1], 'v': [1, 2, None]}).groupby('g').agg({'v': 'size'})
   v
g   
0  2
1  1
```

Without `.groupby`, it raises an recursion error:

```python
>>> pd.DataFrame({'g': [0, 0, 1], 'v': [1, 2, None]}).agg({'v': 'size'})
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
<ipython-input-15-5fc5c38cdb5f> in <module>()
----> 1 pd.DataFrame({'g': [0, 0, 1], 'v': [1, 2, 3]}).agg({'v': 'size'})

/Volumes/Home/venvs/py3-data-science/lib/python3.5/site-packages/pandas/core/frame.py in aggregate(self, func, axis, *args, **kwargs)
   4250                 pass
   4251         if result is None:
-> 4252             return self.apply(func, axis=axis, args=args, **kwargs)
   4253         return result
   4254 

/Volumes/Home/venvs/py3-data-science/lib/python3.5/site-packages/pandas/core/frame.py in apply(self, func, axis, broadcast, raw, reduce, args, **kwds)
   4322         # dispatch to agg
   4323         if axis == 0 and isinstance(func, (list, dict)):
-> 4324             return self.aggregate(func, axis=axis, *args, **kwds)
   4325 
   4326         if len(self.columns) == 0 and len(self.index) == 0:

... last 2 frames repeated, from the frame below ...

/Volumes/Home/venvs/py3-data-science/lib/python3.5/site-packages/pandas/core/frame.py in aggregate(self, func, axis, *args, **kwargs)
   4250                 pass
   4251         if result is None:
-> 4252             return self.apply(func, axis=axis, args=args, **kwargs)
   4253         return result
   4254 

RecursionError: maximum recursion depth exceeded
```


Discussion:
hmm, that doesn't seem friendly. I think the issue is that ``size`` needs to be special cases in ``.agg`` because its a property on a Series/DataFarme and a function in ``.groupby``.
Hey there, I started looking into this as part of the PyCon sprint.

Using the example:

```python
>>> pd.DataFrame({'g': [0, 0, 1], 'v': [1, 2, None]}).agg({'v': 'size'})
```
I tracked down where the error that gets passed by is. In `pandas/core/base.py`:
```python
def _try_aggregate_string_function(self, arg, *args, **kwargs):
        """
        if arg is a string, then try to operate on it:
        - try to find a function on ourselves
        - try to find a numpy function
        - raise
        """
        assert isinstance(arg, compat.string_types)

        f = getattr(self, arg, None)
        if f is not None:
            return f(*args, **kwargs)

        f = getattr(np, arg, None)
        if f is not None:
            return f(self, *args, **kwargs)

        raise ValueError("{} is an unknown string function".format(arg))
```
The arg in this case is `'size'`, which *is* an actual attribute of the Series, just not, as expected, a callable one.  A `TypeError` then bubbles up back to the original `pandas/core/frame.py`'s [`aggregate`](https://github.com/pandas-dev/pandas/blob/master/pandas/core/frame.py#L4310), which is caught and passed over. `Frame.aggregate` then calls `Frame.apply`, which immediately deploys back to `Frame.aggregate` (and thus, our eventual recursion error):
```python
 def apply(self, func, axis=0, broadcast=False, raw=False, reduce=None,
              args=(), **kwds):
        """
        Applies function along input axis of DataFrame.

        ...

        """
        axis = self._get_axis_number(axis)
        ignore_failures = kwds.pop('ignore_failures', False)

        # dispatch to agg
        if axis == 0 and isinstance(func, (list, dict)):
            return self.aggregate(func, axis=axis, *args, **kwds)

        ...
```

It seems like a good place to cut this off may be either `_try_aggregate_string_function` in base.py or in `aggregate` in series.py. Any executive decisions on this @TomAugspurger ? Left entierly to my own devices I'd probably do something along the lines:
```python
def _try_aggregate_string_function(self, arg, *args, **kwargs):
        """
        if arg is a string, then try to operate on it:
        - try to find a function on ourselves
        - try to find a numpy function
        - raise
        """
        assert isinstance(arg, compat.string_types)

        f = getattr(self, arg, None)
        if f is not None:
            return f(*args, **kwargs) if callable(f) else f  # Do people still hate the ternary operator?

        ...
```
But since this is actually in the base class, there could be some unforeseen consequences of this I can't see? (And barring those consequences, should this should just be restricted to calls of 'size'? Otherwise people could do potentially-unexpected-but-also-kinda-cool things like call `df.agg({v:'ndim', w:'name', x:'data'})`)

Appreciate any thoughts, thanks!


@pvomelveny what you did is reasonable, but let's write it out (yes *hate* the ternary operator :>)

maybe

```
if f is not None:
    if callable(f):
        return f(*args, **kwargs)
   assert len(args) == 0 and len(kwargs) == 0
   return f
```

and see if things pass (and add a comment about why doing this).

@pvomelveny can you submit a pull request with those changes? I'm curious to see if the tests pass.

Resolving patch:
diff --git a/pandas/core/base.py b/pandas/core/base.py
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -378,7 +378,7 @@ def aggregate(self, func, *args, **kwargs):
     def _try_aggregate_string_function(self, arg, *args, **kwargs):
         """
         if arg is a string, then try to operate on it:
-        - try to find a function on ourselves
+        - try to find a function (or attribute) on ourselves
         - try to find a numpy function
         - raise
 
@@ -387,7 +387,15 @@ def _try_aggregate_string_function(self, arg, *args, **kwargs):
 
         f = getattr(self, arg, None)
         if f is not None:
-            return f(*args, **kwargs)
+            if callable(f):
+                return f(*args, **kwargs)
+
+            # people may try to aggregate on a non-callable attribute
+            # but don't let them think they can pass args to it
+            assert len(args) == 0
+            assert len([kwarg for kwarg in kwargs
+                        if kwarg not in ['axis', '_level']]) == 0
+            return f
 
         f = getattr(np, arg, None)
         if f is not None:


Test patch:
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -635,3 +635,48 @@ def test_nuiscance_columns(self):
         expected = DataFrame([[6, 6., 'foobarbaz']],
                              index=['sum'], columns=['A', 'B', 'C'])
         assert_frame_equal(result, expected)
+
+    def test_non_callable_aggregates(self):
+
+        # GH 16405
+        # 'size' is a property of frame/series
+        # validate that this is working
+        df = DataFrame({'A': [None, 2, 3],
+                        'B': [1.0, np.nan, 3.0],
+                        'C': ['foo', None, 'bar']})
+
+        # Function aggregate
+        result = df.agg({'A': 'count'})
+        expected = pd.Series({'A': 2})
+
+        assert_series_equal(result, expected)
+
+        # Non-function aggregate
+        result = df.agg({'A': 'size'})
+        expected = pd.Series({'A': 3})
+
+        assert_series_equal(result, expected)
+
+        # Mix function and non-function aggs
+        result1 = df.agg(['count', 'size'])
+        result2 = df.agg({'A': ['count', 'size'],
+                          'B': ['count', 'size'],
+                          'C': ['count', 'size']})
+        expected = pd.DataFrame({'A': {'count': 2, 'size': 3},
+                                 'B': {'count': 2, 'size': 3},
+                                 'C': {'count': 2, 'size': 3}})
+
+        assert_frame_equal(result1, result2, check_like=True)
+        assert_frame_equal(result2, expected, check_like=True)
+
+        # Just functional string arg is same as calling df.arg()
+        result = df.agg('count')
+        expected = df.count()
+
+        assert_series_equal(result, expected)
+
+        # Just a string attribute arg same as calling df.arg
+        result = df.agg('size')
+        expected = df.size
+
+        assert result == expected
diff --git a/pandas/tests/series/test_apply.py b/pandas/tests/series/test_apply.py
--- a/pandas/tests/series/test_apply.py
+++ b/pandas/tests/series/test_apply.py
@@ -306,6 +306,22 @@ def test_reduce(self):
                           name=self.series.name)
         assert_series_equal(result, expected)
 
+    def test_non_callable_aggregates(self):
+        # test agg using non-callable series attributes
+        s = Series([1, 2, None])
+
+        # Calling agg w/ just a string arg same as calling s.arg
+        result = s.agg('size')
+        expected = s.size
+        assert result == expected
+
+        # test when mixed w/ callable reducers
+        result = s.agg(['size', 'count', 'mean'])
+        expected = Series(OrderedDict({'size': 3.0,
+                                       'count': 2.0,
+                                       'mean': 1.5}))
+        assert_series_equal(result[expected.index], expected)
+
 
 class TestSeriesMap(TestData):