Repository: pandas-dev/pandas

Issue:
BUG: value_counts / shift not in groupby dispatch whitelist (0.12-dev)
Copying comment from #4887 to a new issue for discussion purposes:

While testing pandas-master, hit a method not in the new groupby dispatch whitelist: `value_counts` (on a `SeriesGroupBy` object).  Another possible addition: `shift`, as used in this [SO answer](http://stackoverflow.com/a/17406083/243434).

Tried to generate a list of blacklisted methods for DataFrame and Series, see below -- needs further filtering, but may reveal useful blocked methods.  any thoughts on the remaining methods?

```
In [18]: sorted([x for x in set(dir(pd.DataFrame)) - pd.core.groupby._apply_whitelist - set(dir(pd.core.groupby.GroupBy)) if not x.startswith(('_', 'T', 'to', 'from', 'as'))])
Out[18]: ['abs', 'add', 'add_prefix', 'add_suffix', 'align', 'all', 'any', 'append', 'applymap', 'at', 'at_time', 'axes', 'between_time', 'bfill', 'blocks', 'bool', 'clip', 'clip_lower', 'clip_upper', 'columns', 'combine', 'combineAdd', 'combineMult', 'combine_first', 'compound', 'consolidate', 'convert_objects', 'copy', 'corr', 'corrwith', 'cov', 'delevel', 'diff', 'div', 'divide', 'dot', 'drop', 'drop_duplicates', 'dropna', 'dtypes', 'duplicated', 'empty', 'eq', 'eval', 'ffill', 'filter', 'first_valid_index', 'floordiv', 'ftypes', 'ge', 'get', 'get_dtype_counts', 'get_ftype_counts', 'get_value', 'get_values', 'groupby', 'gt', 'iat', 'icol', 'idxmax', 'idxmin', 'iget_value', 'iloc', 'index', 'info', 'insert', 'interpolate', 'irow', 'isin', 'isnull', 'iteritems', 'iterkv', 'iterrows', 'itertuples', 'ix', 'join', 'keys', 'kurt', 'kurtosis', 'last_valid_index', 'le', 'load', 'loc', 'lookup', 'lt', 'mad', 'mask', 'merge', 'mod', 'mode', 'mul', 'multiply', 'ndim', 'ne', 'notnull', 'pct_change', 'pivot', 'pivot_table', 'pop', 'pow', 'product', 'query', 'radd', 'rdiv', 'reindex', 'reindex_axis', 'reindex_like', 'rename', 'rename_axis', 'reorder_levels', 'replace', 'reset_index', 'rfloordiv', 'rmod', 'rmul', 'rpow', 'rsub', 'rtruediv', 'save', 'select', 'set_index', 'set_value', 'shape', 'shift', 'skew', 'sort', 'sort_index', 'sortlevel', 'squeeze', 'stack', 'sub', 'subtract', 'swapaxes', 'swaplevel', 'take', 'transpose', 'truediv', 'truncate', 'tshift', 'tz_convert', 'tz_localize', 'unstack', 'update', 'values', 'where', 'xs']

In [19]: sorted([x for x in set(dir(pd.Series)) - pd.core.groupby._apply_whitelist - set(dir(pd.core.groupby.GroupBy)) if not x.startswith(('_', 'T', 'to', 'from', 'as'))]) 
Out[19]: ['abs', 'add', 'add_prefix', 'add_suffix', 'align', 'all', 'any', 'append', 'argmax', 'argmin', 'argsort', 'at', 'at_time', 'autocorr', 'axes', 'base', 'between', 'between_time', 'bfill', 'blocks', 'bool', 'clip', 'clip_lower', 'clip_upper', 'combine', 'combine_first', 'compound', 'consolidate', 'convert_objects', 'copy', 'corr', 'cov', 'data', 'diff', 'div', 'divide', 'dot', 'drop', 'drop_duplicates', 'dropna', 'duplicated', 'empty', 'eq', 'ffill', 'filter', 'first_valid_index', 'flags', 'floordiv', 'ftype', 'ge', 'get', 'get_dtype_counts', 'get_ftype_counts', 'get_value', 'get_values', 'groupby', 'gt', 'iat', 'idxmax', 'idxmin', 'iget', 'iget_value', 'iloc', 'imag', 'index', 'interpolate', 'irow', 'is_time_series', 'isin', 'isnull', 'item', 'iteritems', 'iterkv', 'ix', 'keys', 'kurt', 'kurtosis', 'last_valid_index', 'le', 'load', 'loc', 'lt', 'mad', 'map', 'mask', 'mod', 'mode', 'mul', 'multiply', 'ndim', 'ne', 'nonzero', 'notnull', 'nunique', 'order', 'pct_change', 'pop', 'pow', 'product', 'ptp', 'put', 'radd', 'ravel', 'rdiv', 'real', 'reindex', 'reindex_axis', 'reindex_like', 'rename', 'rename_axis', 'reorder_levels', 'repeat', 'replace', 'reset_index', 'reshape', 'rfloordiv', 'rmod', 'rmul', 'round', 'rpow', 'rsub', 'rtruediv', 'save', 'select', 'set_value', 'shape', 'shift', 'skew', 'sort', 'sort_index', 'sortlevel', 'squeeze', 'str', 'strides', 'sub', 'subtract', 'swapaxes', 'swaplevel', 'take', 'transpose', 'truediv', 'truncate', 'tshift', 'tz_convert', 'tz_localize', 'unique', 'unstack', 'update', 'valid', 'value_counts', 'values', 'view', 'weekday', 'where', 'xs']
```



Discussion:
This seems like a good idea - would be nice to get a pull-request with tests that exercise the methods that we're readding.

you can of course use apply and use whatever methods u want

this magic of operating on the groupby is only for convience (and possibly cythonized methods)

That's fair and the exception is clear about using apply instead.  Just wanted to file this as it broke existing code without a deprecation warning.

@jreback wouldn't be _too_ hard to set up a DeprecationWarning for this...

``` python
   def _make_wrapper(self, name):
        if name not in _apply_whitelist:
            is_callable = callable(getattr(self.obj, name, None))
            kind = ' callable ' if is_callable else ' '
            warn("access to{0}attribute {1!r} of {2!r} objects is deprecated"
                 " and will be removed in a future version, try using the"
                 "'apply' method".format(kind, name, type(self).__name__))
```

@jreback I think we need to do the deprecation first, otherwise people are going to get confused. Are you okay with that?

sure.. (though i would add shift/value_counts to whitelist as well).

only thing with deprecation - i think it needs to be dynamic to avoid having the tab completion (the **dir** find those methods)

We could keep the whitelist for dir completions and then use the warning if
something isn't in the whitelist, right?

irc their is some kind of message their now.?

It's an error though right?

On Mon, Nov 11, 2013 at 5:19 PM, jreback notifications@github.com wrote:

> irc their is some kind of message their now.?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/5480#issuecomment-28245457
> .

i am confused now

that error message is already there

i would just add shift and value_counts. not big on other methoods as they should be in the inner apply anyhow (their may be some more exception)....but I think its beter to err on the side of a limited whitelist

its an AttributeError which is correct, not a Deprecation.....

its more explicit to use apply 

I think @gdraps brought this up not only for value_counts/shift, but also because there are probably a few people who relied on some part of this and will be surprised their existing code doesn't work (even if it was buggy/wrong - @gdraps please correct me if I've mischaracterized).  We generally deprecate everything before removing it, seems like that's relevant here too.

ok with the deprecation

but u then may need a blacklist that raises for sure - eg to_csv and such 

Okay, @gdraps, do you have time to put this together?

I took a stab at reviewing the full list of methods and would propose the following be added to the whitelist:

```
'value_counts',
'mad', 'median',
'any', 'all',
'irow', 'take',
'shift', 'tshift',
'ffill', 'bfill',
'pct_change', 'skew',
```

As far as the blacklist goes, am thinking all to_\* methods:

```
'to_clipboard', 'to_csv', 'to_dense', 'to_dict', 'to_excel', 'to_frame',
'to_gbq', 'to_hdf', 'to_html', 'to_json', 'to_latex', 'to_msgpack',
'to_panel', 'to_period', 'to_pickle', 'to_records', 'to_sparse', 'to_sql',
'to_stata', 'to_string', 'to_timestamp', 'to_wide',
```

When's the target for 0.13?  I can put a PR later this week.

Target is any day now. I'll put up a quick PR to enable the ones you listed.

median isalready a defined method

what i mean is tht the whitelist should not include methods which are explicity defined on a groupby (as they are already allowed)

Yep, that's what I understood, I took off median.


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -50,13 +50,30 @@
 # forwarding methods from NDFrames
 _plotting_methods = frozenset(['plot', 'boxplot', 'hist'])
 
-_apply_whitelist = frozenset(['last', 'first',
-                              'mean', 'sum', 'min', 'max',
-                              'cumsum', 'cumprod', 'cummin', 'cummax',
-                              'resample',
-                              'describe',
-                              'rank', 'quantile', 'count',
-                              'fillna', 'dtype']) | _plotting_methods
+_common_apply_whitelist = frozenset([
+    'last', 'first',
+    'head', 'tail', 'median',
+    'mean', 'sum', 'min', 'max',
+    'cumsum', 'cumprod', 'cummin', 'cummax', 'cumcount',
+    'resample',
+    'describe',
+    'rank', 'quantile', 'count',
+    'fillna',
+    'mad',
+    'any', 'all',
+    'irow', 'take',
+    'shift', 'tshift',
+    'ffill', 'bfill',
+    'pct_change', 'skew',
+    'corr', 'cov',
+]) | _plotting_methods
+
+_series_apply_whitelist = \
+    (_common_apply_whitelist - set(['boxplot'])) | \
+    frozenset(['dtype', 'value_counts'])
+
+_dataframe_apply_whitelist = \
+    _common_apply_whitelist | frozenset(['dtypes', 'corrwith'])
 
 
 class GroupByError(Exception):
@@ -185,6 +202,7 @@ class GroupBy(PandasObject):
     len(grouped) : int
         Number of groups
     """
+    _apply_whitelist = _common_apply_whitelist
 
     def __init__(self, obj, keys=None, axis=0, level=None,
                  grouper=None, exclusions=None, selection=None, as_index=True,
@@ -252,7 +270,7 @@ def _selection_list(self):
         return self._selection
 
     def _local_dir(self):
-        return sorted(set(self.obj._local_dir() + list(_apply_whitelist)))
+        return sorted(set(self.obj._local_dir() + list(self._apply_whitelist)))
 
     def __getattr__(self, attr):
         if attr in self.obj:
@@ -268,7 +286,7 @@ def __getitem__(self, key):
         raise NotImplementedError
 
     def _make_wrapper(self, name):
-        if name not in _apply_whitelist:
+        if name not in self._apply_whitelist:
             is_callable = callable(getattr(self.obj, name, None))
             kind = ' callable ' if is_callable else ' '
             msg = ("Cannot access{0}attribute {1!r} of {2!r} objects, try "
@@ -1605,6 +1623,7 @@ def _convert_grouper(axis, grouper):
 
 
 class SeriesGroupBy(GroupBy):
+    _apply_whitelist = _series_apply_whitelist
 
     def aggregate(self, func_or_funcs, *args, **kwargs):
         """
@@ -2401,6 +2420,7 @@ def add_indices():
 
 
 class DataFrameGroupBy(NDFrameGroupBy):
+    _apply_whitelist = _dataframe_apply_whitelist
 
     _block_agg_axis = 1
 


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -3221,10 +3221,67 @@ def test_groupby_whitelist(self):
                         'letters': Series(random_letters)})
         s = df.floats
 
-        blacklist = ['eval', 'query', 'abs', 'shift', 'tshift', 'where',
-                     'mask', 'align', 'groupby', 'clip', 'astype',
-                     'at', 'combine', 'consolidate', 'convert_objects',
-                     'corr', 'corr_with', 'cov']
+        df_whitelist = frozenset([
+            'last', 'first',
+            'mean', 'sum', 'min', 'max',
+            'head', 'tail',
+            'cumsum', 'cumprod', 'cummin', 'cummax', 'cumcount',
+            'resample',
+            'describe',
+            'rank', 'quantile', 'count',
+            'fillna',
+            'mad',
+            'any', 'all',
+            'irow', 'take',
+            'shift', 'tshift',
+            'ffill', 'bfill',
+            'pct_change', 'skew',
+            'plot', 'boxplot', 'hist',
+            'median', 'dtypes',
+            'corrwith', 'corr', 'cov',
+        ])
+        s_whitelist = frozenset([
+            'last', 'first',
+            'mean', 'sum', 'min', 'max',
+            'head', 'tail',
+            'cumsum', 'cumprod', 'cummin', 'cummax', 'cumcount',
+            'resample',
+            'describe',
+            'rank', 'quantile', 'count',
+            'fillna',
+            'mad',
+            'any', 'all',
+            'irow', 'take',
+            'shift', 'tshift',
+            'ffill', 'bfill',
+            'pct_change', 'skew',
+            'plot', 'hist',
+            'median', 'dtype',
+            'corr', 'cov',
+            'value_counts',
+        ])
+
+        for obj, whitelist in zip((df, s),
+                                  (df_whitelist, s_whitelist)):
+            gb = obj.groupby(df.letters)
+            self.assertEqual(whitelist, gb._apply_whitelist)
+            for m in whitelist:
+                getattr(gb, m)
+
+    def test_groupby_blacklist(self):
+        from string import ascii_lowercase
+        letters = np.array(list(ascii_lowercase))
+        N = 10
+        random_letters = letters.take(np.random.randint(0, 26, N))
+        df = DataFrame({'floats': N / 10 * Series(np.random.random(N)),
+                        'letters': Series(random_letters)})
+        s = df.floats
+
+        blacklist = [
+            'eval', 'query', 'abs', 'where',
+            'mask', 'align', 'groupby', 'clip', 'astype',
+            'at', 'combine', 'consolidate', 'convert_objects',
+        ]
         to_methods = [method for method in dir(df) if method.startswith('to_')]
 
         blacklist.extend(to_methods)
@@ -3319,8 +3376,12 @@ def test_tab_completion(self):
             'groups','hist','indices','last','max','mean','median',
             'min','name','ngroups','nth','ohlc','plot', 'prod',
             'size','std','sum','transform','var', 'count', 'head', 'describe',
-            'cummax', 'dtype', 'quantile', 'rank', 'cumprod', 'tail',
-            'resample', 'cummin', 'fillna', 'cumsum', 'cumcount'])
+            'cummax', 'quantile', 'rank', 'cumprod', 'tail',
+            'resample', 'cummin', 'fillna', 'cumsum', 'cumcount',
+            'all', 'shift', 'skew', 'bfill', 'irow', 'ffill',
+            'take', 'tshift', 'pct_change', 'any', 'mad', 'corr', 'corrwith',
+            'cov', 'dtypes',
+        ])
         self.assertEqual(results, expected)
 
 def assert_fp_equal(a, b):