Repository: pandas-dev/pandas

Issue:
Filter of grouped dataframe raises ValueError
see also #4527

When trying to filter a grouped dataframe with more than 2 columns raises an ValueError:

``` python

In [90]: dff = pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})

In [91]: dff.groupby('B').filter(lambda x: len(x) > 2)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-91-89d79df28299> in <module>()
----> 1 dff.groupby('B').filter(lambda x: len(x) > 2)

C:\Anaconda\lib\site-packages\pandas\core\groupby.pyc in filter(self, func, dropna, *args, **kwargs)
   2092                 res = path(group)
   2093
-> 2094             if res:
   2095                 indexers.append(self.obj.index.get_indexer(group.index))
   2096

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


In [93]: pd.__version__
Out[93]: '0.12.0'
```



Discussion:
good catch. Looking into this.

So far, I've found that the proximate problem is with the slow_path here, but fixing it for this case causes issues for other tests. Working on it... 

The problem arises with `len` specifically. For example, `count` is fine.

```
In [1]: dff = pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})

In [2]: dff.groupby('B').filter(lambda x: x.count() > 2)
Out[2]: 
   A  B  C
0  0  a  0
1  1  a  1
2  2  b  2
3  3  b  3
4  4  b  4
5  5  b  5
6  6  c  6
7  7  c  7
```

@danielballan you prob just need to deal with different return types/shapes from the filtered function;

this is BTD (bug test driven!)

@danielballan fyi... # 4657 will partially fix this (sort of) as it looks at the return value from the filter in a better way

@danielballan would be nice to fix for 0.13

I cannot reproduce the error. I think #4657 changed the path taken, so now the shape of the `len` result is interpreted correctly. On the current master:

```
In [19]: pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)}).groupby('B').filter(lambda x: len(x) > 2)
Out[19]: 
   A  B  C
2  2  b  2
3  3  b  3
4  4  b  4
5  5  b  5
```

If you can point me toward an extant problem, I will follow up.

If not, close?

hmm....maybe add some test cases that return different things in the function to try to break it:

e.g. scalar, series, frame, raise various exceptions, etc

and then can close this issue

OK, sounds good.

@danielballan tests for this?

Within a week should be no problem. (Not exactly related, but is there a target for 0.13?)

gr8!

not exactly sure of target date...still lots of issues to get thru...few weeks maybe

@danielballan ?

Added four tests in PR #5096 using len on a DataFrame and a Series. Will continue to explore various shapes. I have not discovered any other filter bugs so far. I will switch focus to #4621....


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -17,7 +17,7 @@
 from pandas.util.decorators import cache_readonly, Appender
 import pandas.core.algorithms as algos
 import pandas.core.common as com
-from pandas.core.common import _possibly_downcast_to_dtype, notnull
+from pandas.core.common import _possibly_downcast_to_dtype, isnull, notnull
 
 import pandas.lib as lib
 import pandas.algos as _algos
@@ -1605,8 +1605,19 @@ def filter(self, func, dropna=True, *args, **kwargs):
         else:
             wrapper = lambda x: func(x, *args, **kwargs)
 
-        indexers = [self.obj.index.get_indexer(group.index) \
-                    if wrapper(group) else [] for _ , group in self]
+        # Interpret np.nan as False.
+        def true_and_notnull(x, *args, **kwargs):
+            b = wrapper(x, *args, **kwargs)
+            return b and notnull(b)
+
+        try:
+            indexers = [self.obj.index.get_indexer(group.index) \
+                        if true_and_notnull(group) else [] \
+                        for _ , group in self]
+        except ValueError:
+            raise TypeError("the filter must return a boolean result")
+        except TypeError:
+            raise TypeError("the filter must return a boolean result")
 
         if len(indexers) == 0:
             filtered = self.obj.take([]) # because np.concatenate would fail
@@ -2124,7 +2135,8 @@ def add_indexer():
                     add_indexer()
             else:
                 if getattr(res,'ndim',None) == 1:
-                    if res.ravel()[0]:
+                    val = res.ravel()[0]
+                    if val and notnull(val):
                         add_indexer()
                 else:
 


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -2642,9 +2642,37 @@ def raise_if_sum_is_zero(x):
         s = pd.Series([-1,0,1,2])
         grouper = s.apply(lambda x: x % 2)
         grouped = s.groupby(grouper)
-        self.assertRaises(ValueError,
+        self.assertRaises(TypeError,
                           lambda: grouped.filter(raise_if_sum_is_zero))
 
+    def test_filter_bad_shapes(self):
+        df = DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})
+        s = df['B']
+        g_df = df.groupby('B')
+        g_s = s.groupby(s)
+
+        f = lambda x: x
+        self.assertRaises(TypeError, lambda: g_df.filter(f))
+        self.assertRaises(TypeError, lambda: g_s.filter(f))
+
+        f = lambda x: x == 1
+        self.assertRaises(TypeError, lambda: g_df.filter(f))
+        self.assertRaises(TypeError, lambda: g_s.filter(f))
+
+        f = lambda x: np.outer(x, x)
+        self.assertRaises(TypeError, lambda: g_df.filter(f))
+        self.assertRaises(TypeError, lambda: g_s.filter(f))
+
+    def test_filter_nan_is_false(self):
+        df = DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})
+        s = df['B']
+        g_df = df.groupby(df['B'])
+        g_s = s.groupby(s)
+
+        f = lambda x: np.nan
+        assert_frame_equal(g_df.filter(f), df.loc[[]])
+        assert_series_equal(g_s.filter(f), s[[]])
+
     def test_filter_against_workaround(self):
         np.random.seed(0)
         # Series of ints
@@ -2697,6 +2725,29 @@ def test_filter_against_workaround(self):
         new_way = grouped.filter(lambda x: x['ints'].mean() > N/20)
         assert_frame_equal(new_way.sort_index(), old_way.sort_index())
 
+    def test_filter_using_len(self):
+        # BUG GH4447
+        df = DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})
+        grouped = df.groupby('B')
+        actual = grouped.filter(lambda x: len(x) > 2)
+        expected = DataFrame({'A': np.arange(2, 6), 'B': list('bbbb'), 'C': np.arange(2, 6)}, index=np.arange(2, 6))
+        assert_frame_equal(actual, expected)
+
+        actual = grouped.filter(lambda x: len(x) > 4)
+        expected = df.ix[[]]
+        assert_frame_equal(actual, expected)
+
+        # Series have always worked properly, but we'll test anyway.
+        s = df['B']
+        grouped = s.groupby(s)
+        actual = grouped.filter(lambda x: len(x) > 2)
+        expected = Series(4*['b'], index=np.arange(2, 6))
+        assert_series_equal(actual, expected)
+
+        actual = grouped.filter(lambda x: len(x) > 4)
+        expected = s[[]]
+        assert_series_equal(actual, expected)
+
     def test_groupby_whitelist(self):
         from string import ascii_lowercase
         letters = np.array(list(ascii_lowercase))