Repository: pandas-dev/pandas

Issue:
CLN: generate groupby method wrapper code
```
$ grep -r "[^_]exec(" pandas
pandas/compat/__init__.py:    exec("""
pandas/core/groupby.py:        exec(_def_str)
pandas/core/groupby.py:        exec(_def_str)
```

The usage in `compat` is for `raise_with_traceback` where the py2 version is a syntax error in py3.  This is pretty straightforward to de-exec by putting the py2-specific code in a module that only gets imported in py2.  

The usage in `core.groupby` is tougher.  The string being exec'ed defines a method in some cases and property in others.  The property version I think I've got a working fix, but the method version has me stumped.

These `_def_str`s are constructed in `core.groupby._whitelist_method_generator`:

```
    method_wrapper_template = \
        """def %(name)s(%(sig)s) :
    \"""
    %(doc)s
    \"""
    f = %(self)s.__getattr__('%(name)s')
    return f(%(args)s)"""
    property_wrapper_template = \
        """@property
def %(name)s(self) :
    \"""
    %(doc)s
    \"""
    return self.__getattr__('%(name)s')"""
    for name in whitelist:
        # don't override anything that was explicitly defined
        # in the base class
        if hasattr(GroupBy, name):
            continue
        # ugly, but we need the name string itself in the method.
        f = getattr(klass, name)
        doc = f.__doc__
        doc = doc if type(doc) == str else ''
        if isinstance(f, types.MethodType):
            wrapper_template = method_wrapper_template
            decl, args = make_signature(f)
            # pass args by name to f because otherwise
            # GroupBy._make_wrapper won't know whether
            # we passed in an axis parameter.
            args_by_name = ['{0}={0}'.format(arg) for arg in args[1:]]
            params = {'name': name,
                      'doc': doc,
                      'sig': ','.join(decl),
                      'self': args[0],
                      'args': ','.join(args_by_name)}
        else:
            wrapper_template = property_wrapper_template
            params = {'name': name, 'doc': doc}
        yield wrapper_template % params
```



Discussion:
not sure this is a big deal. this is just a way to code generate.
For the `raise_with_traceback` I agree it's not that big a deal, I've just been indoctrinated to react to exec.  Let's forget about that one.

In the case of `method_wrapper_template` the fact that I couldn't reason about how it works makes it a problem for me.  Moving forward on the requests made by @gfyoung in #16913 is going to require poking around in there.  I'll take another look at it next week.
yeah what we usually do is to generate code (for cython) using tempita. We actually could do that here, its cleaner and more pythonic. (see the pxi.in -> pxi files).
> Moving forward on the requests made by @gfyoung in #16913 is going to require poking around in there.

@jbrockmendel : Could you explain why deprecations are requiring you to go through this refactoring?
> Could you explain why deprecations are requiring you to go through this refactoring?

The non-invasive part of the deprecation does not.  What I was referring to was the suggestion that removing these methods entirely might fix the problem of the circular import:

> [...] That might go away once those hist and plot methods are removed.

*Removing* these methods from `Series`/`DataFrame` does cascade into requiring changes in `core.groupby`.  Like with #16913, the stumbled-upon issue (`exec`) is more important than the original motivator (refactoring that's a ways down the road).

> The non-invasive part of the deprecation does not. What I was referring to was the suggestion that removing these methods entirely might fix the problem of the circular import:

Got it.  That makes sense. 👍 
Digging into this a bit further...

```
    method_wrapper_template = \
        """def %(name)s(%(sig)s) :
    \"""
    %(doc)s
    \"""
    f = %(self)s.__getattr__('%(name)s')
    return f(%(args)s)"""
```

The `%(self)s` there in _most_ cases resolves to just `self` (the exception is `hist`, which uses an accessor, which is another layer of redirection...).  So at first this looks like it should get into some kind of loop trying to access itself.  But it turns out `__getattr__` has been patched so that this actually accesses `self.obj.__getattr__('%(name)s')`.

Resolving patch:
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -11,7 +11,7 @@
 from functools import partial
 from textwrap import dedent
 import typing
-from typing import Any, Callable, FrozenSet, Iterator, Sequence, Type, Union
+from typing import Any, Callable, FrozenSet, Sequence, Type, Union
 import warnings
 
 import numpy as np
@@ -70,47 +70,63 @@
 ScalarResult = typing.TypeVar("ScalarResult")
 
 
-def whitelist_method_generator(
-    base_class: Type[GroupBy], klass: Type[FrameOrSeries], whitelist: FrozenSet[str]
-) -> Iterator[str]:
+def generate_property(name: str, klass: Type[FrameOrSeries]):
     """
-    Yields all GroupBy member defs for DataFrame/Series names in whitelist.
+    Create a property for a GroupBy subclass to dispatch to DataFrame/Series.
+
+    Parameters
+    ----------
+    name : str
+    klass : {DataFrame, Series}
+
+    Returns
+    -------
+    property
+    """
+
+    def prop(self):
+        return self._make_wrapper(name)
+
+    parent_method = getattr(klass, name)
+    prop.__doc__ = parent_method.__doc__ or ""
+    prop.__name__ = name
+    return property(prop)
+
+
+def pin_whitelisted_properties(klass: Type[FrameOrSeries], whitelist: FrozenSet[str]):
+    """
+    Create GroupBy member defs for DataFrame/Series names in a whitelist.
 
     Parameters
     ----------
-    base_class : Groupby class
-        base class
     klass : DataFrame or Series class
         class where members are defined.
-    whitelist : frozenset
+    whitelist : frozenset[str]
         Set of names of klass methods to be constructed
 
     Returns
     -------
-    The generator yields a sequence of strings, each suitable for exec'ing,
-    that define implementations of the named methods for DataFrameGroupBy
-    or SeriesGroupBy.
+    class decorator
 
+    Notes
+    -----
     Since we don't want to override methods explicitly defined in the
     base class, any such name is skipped.
     """
-    property_wrapper_template = """@property
-def %(name)s(self) :
-    \"""%(doc)s\"""
-    return self.__getattr__('%(name)s')"""
-
-    for name in whitelist:
-        # don't override anything that was explicitly defined
-        # in the base class
-        if hasattr(base_class, name):
-            continue
-        # ugly, but we need the name string itself in the method.
-        f = getattr(klass, name)
-        doc = f.__doc__
-        doc = doc if type(doc) == str else ""
-        wrapper_template = property_wrapper_template
-        params = {"name": name, "doc": doc}
-        yield wrapper_template % params
+
+    def pinner(cls):
+        for name in whitelist:
+            if hasattr(cls, name):
+                # don't override anything that was explicitly defined
+                #  in the base class
+                continue
+
+            prop = generate_property(name, klass)
+            setattr(cls, name, prop)
+
+        return cls
+
+    return pinner
 
 
 class NDFrameGroupBy(GroupBy):
@@ -747,13 +763,9 @@ def filter(self, func, dropna=True, *args, **kwargs):
         return self._apply_filter(indices, dropna)
 
 
+@pin_whitelisted_properties(Series, base.series_apply_whitelist)
 class SeriesGroupBy(GroupBy):
-    #
-    # Make class defs of attributes on SeriesGroupBy whitelist
-
     _apply_whitelist = base.series_apply_whitelist
-    for _def_str in whitelist_method_generator(GroupBy, Series, _apply_whitelist):
-        exec(_def_str)
 
     @property
     def _selection_name(self):
@@ -1368,15 +1380,11 @@ def pct_change(self, periods=1, fill_method="pad", limit=None, freq=None):
         return (filled / shifted) - 1
 
 
+@pin_whitelisted_properties(DataFrame, base.dataframe_apply_whitelist)
 class DataFrameGroupBy(NDFrameGroupBy):
 
     _apply_whitelist = base.dataframe_apply_whitelist
 
-    #
-    # Make class defs of attributes on DataFrameGroupBy whitelist.
-    for _def_str in whitelist_method_generator(GroupBy, DataFrame, _apply_whitelist):
-        exec(_def_str)
-
     _block_agg_axis = 1
 
     _agg_see_also_doc = dedent(
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -562,8 +562,6 @@ def __getattr__(self, attr):
             return object.__getattribute__(self, attr)
         if attr in self.obj:
             return self[attr]
-        if hasattr(self.obj, attr):
-            return self._make_wrapper(attr)
 
         raise AttributeError(
             "%r object has no attribute %r" % (type(self).__name__, attr)


Test patch: