Repository: pandas-dev/pandas

Issue:
groupby filtering is missing some groups
I brought this up on the mailing list. @cpcloud modified my example into very concise sample showing expected and resulting output:

```
In [20]: df  = pd.DataFrame([
    ['best', 'a', 'x'],
    ['worst', 'b', 'y'],
    ['best', 'c', 'x'],
    ['best','d', 'y'],
    ['worst','d', 'y'],
    ['worst','d', 'y'],
    ['best','d', 'z'],
], columns=['a', 'b', 'c'])

In [21]: pd.concat(v[v.a == 'best'] for _, v in df.groupby('c'))
Out[21]:
      a  b  c
0  best  a  x
2  best  c  x
3  best  d  y # <--- missing from the next statement
6  best  d  z

In [22]: df.groupby('c').filter(lambda g: g.a == 'best')
Out[22]:
      a  b  c
0  best  a  x
2  best  c  x
6  best  d  z
```



Discussion:
I'm not sure this is what `filter` is meant for.  I think it acts on the _groups_, not on elements of the groups.

If I'm right, then the bug is that this isn't tripping `raise TypeError("the filter must return a boolean result")`, not that the result differs from the concat result.  If not, then the docs are confusing.  (Maybe they are in any case.)

I'll repost how I thought it might've worked

```
In [20]: df
Out[20]:
       a  b  c
0   best  a  x
1  worst  b  y
2   best  c  x
3   best  d  y
4  worst  d  y
5  worst  d  y
6   best  d  z

In [21]: pd.concat(v[v.a == 'best'] for _, v in df.groupby('c'))
Out[21]:
      a  b  c
0  best  a  x
2  best  c  x
3  best  d  y
6  best  d  z

In [22]: df.groupby('c').filter(lambda g: g.a == 'best')
Out[22]:
      a  b  c
0  best  a  x
2  best  c  x
6  best  d  z
```

@dsm054 by groups do you mean `['x', 'y', 'z']` or the `(name, frame)` pairs?

I agree with @dsm054 here. This should return the SAME df, (as 'best' is in EACH group). The bug is that the complete groups are not being returned (because the filter function is tricking the detector).

I mean the subframes that `filter` is passed.  I think you're given a group and you're supposed to return a yea or nay bool based on the _group_, not an iterable of bools about your opinions on each row in the group.  

The example in the function itself is `(lambda x: x['A'].sum() + x['B'].sum() > 0)`, which is a reduction-to-bool operation.

well then this is very weird :)

pr coming

Just to be clear, I'm not sure I agree with @jreback that the right thing to do is to return the same `df` because "best" is in each group.  That's equivalent to saying that we're going to use the convention bool(array) == any(array), which is only going to work here.

Posted on ML before this. I think raise is the correct course of action here. Filter is for removing/including _entire_ groups. Picking all or any (or first!) of the boolean Series would be strange IMO...

> Function to apply to each subframe. Should return True or False.

Very insightful comments here. Much appreciated. I'm :+1: on raising an error, which I believe version 0.12 did.

I concur

Okay, this mainly had to do with the `slow_path`/`fast_path` business, where the slow path calls `apply` on the `group` (usually a frame) and the fast path just calls the function directly, which of course in the case of this:

```
In [8]: df
Out[8]:
   a       b
0  1  0.9712
1  1  0.7578
2  1  0.6300
3  2  1.7625
4  2 -0.0895

In [9]: df[df.a == 1].apply(lambda x: len(x) > 1)
Out[9]:
a    True
b    True
dtype: bool

In [10]: len(df[df.a == 1]) > 1
Out[10]: True
```

gives two different unrelated results and there were some hacks around trying to figure out which one was "correct". 

i gutted all that code and now just call the fast path, we'll see what happens on travis but this doesn't break anything locally.


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2945,48 +2945,34 @@ def filter(self, func, dropna=True, *args, **kwargs):
         >>> grouped = df.groupby(lambda x: mapping[x])
         >>> grouped.filter(lambda x: x['A'].sum() + x['B'].sum() > 0)
         """
-        from pandas.tools.merge import concat
 
         indices = []
 
         obj = self._selected_obj
         gen = self.grouper.get_iterator(obj, axis=self.axis)
 
-        fast_path, slow_path = self._define_paths(func, *args, **kwargs)
-
-        path = None
         for name, group in gen:
             object.__setattr__(group, 'name', name)
 
-            if path is None:
-                # Try slow path and fast path.
-                try:
-                    path, res = self._choose_path(fast_path, slow_path, group)
-                except Exception:  # pragma: no cover
-                    res = fast_path(group)
-                    path = fast_path
-            else:
-                res = path(group)
+            res = func(group)
 
-            def add_indices():
-                indices.append(self._get_index(name))
+            try:
+                res = res.squeeze()
+            except AttributeError:  # allow e.g., scalars and frames to pass
+                pass
 
             # interpret the result of the filter
-            if isinstance(res, (bool, np.bool_)):
-                if res:
-                    add_indices()
+            if (isinstance(res, (bool, np.bool_)) or
+                np.isscalar(res) and isnull(res)):
+                if res and notnull(res):
+                    indices.append(self._get_index(name))
             else:
-                if getattr(res, 'ndim', None) == 1:
-                    val = res.ravel()[0]
-                    if val and notnull(val):
-                        add_indices()
-                else:
-
-                    # in theory you could do .all() on the boolean result ?
-                    raise TypeError("the filter must return a boolean result")
+                # non scalars aren't allowed
+                raise TypeError("filter function returned a %s, "
+                                "but expected a scalar bool" %
+                                type(res).__name__)
 
-        filtered = self._apply_filter(indices, dropna)
-        return filtered
+        return self._apply_filter(indices, dropna)
 
 
 class DataFrameGroupBy(NDFrameGroupBy):


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -3968,6 +3968,32 @@ def test_filter_has_access_to_grouped_cols(self):
         filt = g.filter(lambda x: x['A'].sum() == 2)
         assert_frame_equal(filt, df.iloc[[0, 1]])
 
+    def test_filter_enforces_scalarness(self):
+        df  = pd.DataFrame([
+            ['best', 'a', 'x'],
+            ['worst', 'b', 'y'],
+            ['best', 'c', 'x'],
+            ['best','d', 'y'],
+            ['worst','d', 'y'],
+            ['worst','d', 'y'],
+            ['best','d', 'z'],
+        ], columns=['a', 'b', 'c'])
+        with tm.assertRaisesRegexp(TypeError, 'filter function returned a.*'):
+            df.groupby('c').filter(lambda g: g['a'] == 'best')
+
+    def test_filter_non_bool_raises(self):
+        df  = pd.DataFrame([
+            ['best', 'a', 1],
+            ['worst', 'b', 1],
+            ['best', 'c', 1],
+            ['best','d', 1],
+            ['worst','d', 1],
+            ['worst','d', 1],
+            ['best','d', 1],
+        ], columns=['a', 'b', 'c'])
+        with tm.assertRaisesRegexp(TypeError, 'filter function returned a.*'):
+            df.groupby('a').filter(lambda g: g.c.mean())
+
     def test_index_label_overlaps_location(self):
         # checking we don't have any label/location confusion in the
         # the wake of GH5375