Repository: pandas-dev/pandas

Issue:
Now boolean operators work with NDFrames?
Awkward side effect of the new and improved `Series` probably, I typed something wrong and was shocked to not get the usual `numpy` `ValueError` barf:

```
In [57]: a, b = Series(rand(10) > 0.5), Series(rand(10) > 0.5)

In [58]: a
Out[58]:
0     True
1     True
2    False
3    False
4     True
5    False
6    False
7     True
8     True
9     True
dtype: bool

In [59]: b
Out[59]:
0     True
1     True
2     True
3     True
4    False
5     True
6    False
7     True
8     True
9     True
dtype: bool

In [60]: a and b # what?! no way!!
Out[60]:
0     True
1     True
2     True
3     True
4    False
5     True
6    False
7     True
8     True
9     True
dtype: bool

In [61]: a & b # whew
Out[61]:
0     True
1     True
2    False
3    False
4    False
5    False
6    False
7     True
8     True
9     True
dtype: bool
```



Discussion:
```
In [11]: dfa, dfb = DataFrame(randn(10,2)),DataFrame(randn(10,2))

In [13]: dfa
Out[13]: 
          0         1
0  1.032736 -0.981451
1  0.154600  1.458225
2  0.688022  0.793732
3 -1.171007 -0.643027
4 -0.559731 -0.019199
5  1.079594 -0.549702
6 -1.803250  1.319622
7  0.602654  0.550604
8 -0.908063 -1.304829
9  0.103192  1.542427

In [14]: dfb
Out[14]: 
          0         1
0 -0.445965 -1.055636
1  0.929273  0.160086
2  0.302718 -0.653348
3 -1.227953 -1.627101
4  1.832395 -0.036561
5  0.806457  1.730636
6  1.805333 -0.299774
7  0.283562 -0.263815
8  0.749068  0.589955
9  0.523333 -2.085325

In [15]: dfa and dfb
Out[15]: 
          0         1
0 -0.445965 -1.055636
1  0.929273  0.160086
2  0.302718 -0.653348
3 -1.227953 -1.627101
4  1.832395 -0.036561
5  0.806457  1.730636
6  1.805333 -0.299774
7  0.283562 -0.263815
8  0.749068  0.589955
9  0.523333 -2.085325
```

This seems like a good thing. Now truthy-ness of Series/DataFrames is consistent, empty Series are Falsey.

```
In [6]: s1 = pd.Series([1])

In [7]: s_ = pd.Series()

In [8]: s_ or s
Out[8]: 
0    1
dtype: int64
```

_Not sure I can use this anywhere, but I'll try._

Consistency aside, this seems likely to lead to lots of subtle bugs when people use `df[a or b]` when they mean `df[a | b]`.

Along the lines of @mtkni's comment : is it a good thing that Series behaves more like a Python container vs a numpy container? 

Ew:

```
In [1]: bool(np.array([]))
Out[1]: False

In [2]: bool(np.array([1]))
Out[2]: True

In [3]: bool(np.array([1, 2]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-5ba97928842c> in <module>()
----> 1 bool(np.array([1, 2]))

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

That said, you're _probably_ right that and/or should raise. :(

You also get fun things like this:

```
>>> a = Series(rand(10))
>>> bool(a == 0)
True
>>> bool(a == 1)
True
```

I think it should raise as well.  It's doing numpy-like comparisons and broadcasting, not throwing an exception can allow subtle bugs to go undetected.

Possibly related: #1069 (old but apparently reverted), #4576, #4537

this like this I think are easily caught

I had to add a couple of coercion method which can handle these cases

eg **bool** , **int** etc

What's the use case for allowing `bool(s)` to return `True` for any series with nonzero length?  It has never been the case in previous releases, since `Series` inherited from `numpy.ndarray`.  This is new behavior, and IMO it does not add anything useful, only a source of bugs.

oh I agree

however we have a couple of tests that would fail otherwise so need to fix those

iIIRC there was a couple of places this is relied upon (which is just wrong)

@cpcloud pls mark this and we can take a look 

OK, thanks.

It looks like raising `ValueError` for truth tests on DataFrames (which had been introduced in PR #1073) was reverted by PR #3696.  I would argue that should be restored as well.

yeh these should all be consistent
I'll take alook tomorrow

```
In [17]: s = Series(randn(4))

In [18]: df = DataFrame(randn(10,2))

In [19]: s_empty = Series()

In [20]: df_empty = DataFrame()
```

This is from 0.12

```
In [21]: bool(s)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [22]: bool(s_empty)
Out[22]: False

In [23]: bool(df)
Out[23]: True

In [24]: bool(df_empty)
Out[24]: False
```

master

```
In [5]: bool(s)
Out[5]: True

In [6]: bool(s_empty)
Out[6]: False

In [7]: bool(df)
Out[7]: True

In [8]: bool(df_empty)
Out[8]: False
```

These are the same in 0.12 and master, e.g. a 1-element Series is special cased
to evaluate to return the boolean of a single value

```
In [9]: bool(Series([True]))
Out[9]: True

In [10]: bool(Series([False]))
Out[10]: False
```

I am going to argue that 0.12 is inconsistent and master is correct

However, one _could_ have a point of view that ALL of these should raise

any thoughts?

@wesm ?


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -531,7 +531,8 @@ def empty(self):
         return not all(len(self._get_axis(a)) > 0 for a in self._AXIS_ORDERS)
 
     def __nonzero__(self):
-        return not self.empty
+        raise ValueError("The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().")
+
     __bool__ = __nonzero__
 
     #----------------------------------------------------------------------
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2101,9 +2101,22 @@ def filter(self, func, dropna=True, *args, **kwargs):
             else:
                 res = path(group)
 
-            if res:
+            def add_indexer():
                 indexers.append(self.obj.index.get_indexer(group.index))
 
+            # interpret the result of the filter
+            if isinstance(res,(bool,np.bool_)):
+                if res:
+                    add_indexer()
+            else:
+                if getattr(res,'ndim',None) == 1:
+                    if res.ravel()[0]:
+                        add_indexer()
+                else:
+
+                    # in theory you could do .all() on the boolean result ?
+                    raise TypeError("the filter must return a boolean result")
+
         if len(indexers) == 0:
             filtered = self.obj.take([]) # because np.concatenate would fail
         else:
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -798,13 +798,6 @@ def __contains__(self, key):
     __long__ = _coerce_method(int)
     __int__ = _coerce_method(int)
 
-    def __nonzero__(self):
-        # special case of a single element bool series degenerating to a scalar
-        if self.dtype == np.bool_ and len(self) == 1:
-            return bool(self.iloc[0])
-        return not self.empty
-    __bool__ = __nonzero__
-
     # we are preserving name here
     def __getstate__(self):
         return dict(_data=self._data, name=self.name)


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -1519,11 +1519,11 @@ def test_table_values_dtypes_roundtrip(self):
         with ensure_clean(self.path) as store:
             df1 = DataFrame({'a': [1, 2, 3]}, dtype='f8')
             store.append('df_f8', df1)
-            assert df1.dtypes == store['df_f8'].dtypes
+            assert_series_equal(df1.dtypes,store['df_f8'].dtypes)
 
             df2 = DataFrame({'a': [1, 2, 3]}, dtype='i8')
             store.append('df_i8', df2)
-            assert df2.dtypes == store['df_i8'].dtypes
+            assert_series_equal(df2.dtypes,store['df_i8'].dtypes)
 
             # incompatible dtype
             self.assertRaises(ValueError, store.append, 'df_i8', df1)
@@ -1531,7 +1531,7 @@ def test_table_values_dtypes_roundtrip(self):
             # check creation/storage/retrieval of float32 (a bit hacky to actually create them thought)
             df1 = DataFrame(np.array([[1],[2],[3]],dtype='f4'),columns = ['A'])
             store.append('df_f4', df1)
-            assert df1.dtypes == store['df_f4'].dtypes
+            assert_series_equal(df1.dtypes,store['df_f4'].dtypes)
             assert df1.dtypes[0] == 'float32'
 
             # check with mixed dtypes
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -10607,13 +10607,10 @@ def test_index_namedtuple(self):
         df = DataFrame([(1, 2), (3, 4)], index=index, columns=["A", "B"])
         self.assertEqual(df.ix[IndexType("foo", "bar")]["A"], 1)
 
-    def test_bool_empty_nonzero(self):
+    def test_empty_nonzero(self):
         df = DataFrame([1, 2, 3])
-        self.assertTrue(bool(df))
         self.assertFalse(df.empty)
         df = DataFrame(index=['a', 'b'], columns=['c', 'd']).dropna()
-        self.assertFalse(bool(df))
-        self.assertFalse(bool(df.T))
         self.assertTrue(df.empty)
         self.assertTrue(df.T.empty)
 
diff --git a/pandas/tests/test_generic.py b/pandas/tests/test_generic.py
--- a/pandas/tests/test_generic.py
+++ b/pandas/tests/test_generic.py
@@ -73,7 +73,6 @@ def _construct(self, shape, value=None, **kwargs):
             arr = np.random.randn(*shape)
         return self._typ(arr,**kwargs)
 
-
     def _compare(self, result, expected):
         self._comparator(result,expected)
 
@@ -82,14 +81,14 @@ def test_rename(self):
         # single axis
         for axis in self._axes():
             kwargs = { axis : list('ABCD') }
-            o = self._construct(4,**kwargs)
+            obj = self._construct(4,**kwargs)
 
             # no values passed
             #self.assertRaises(Exception, o.rename(str.lower))
 
             # rename a single axis
-            result = o.rename(**{ axis : str.lower })
-            expected = o.copy()
+            result = obj.rename(**{ axis : str.lower })
+            expected = obj.copy()
             setattr(expected,axis,list('abcd'))
             self._compare(result, expected)
 
@@ -119,6 +118,41 @@ def test_get_numeric_data(self):
         self._compare(result, o)
 
         # _get_numeric_data is includes _get_bool_data, so can't test for non-inclusion
+    def test_nonzero(self):
+
+        # GH 4633
+        # look at the boolean/nonzero behavior for objects
+        obj = self._construct(shape=4)
+        self.assertRaises(ValueError, lambda : bool(obj == 0))
+        self.assertRaises(ValueError, lambda : bool(obj == 1))
+        self.assertRaises(ValueError, lambda : bool(obj))
+
+        obj = self._construct(shape=4,value=1)
+        self.assertRaises(ValueError, lambda : bool(obj == 0))
+        self.assertRaises(ValueError, lambda : bool(obj == 1))
+        self.assertRaises(ValueError, lambda : bool(obj))
+
+        obj = self._construct(shape=4,value=np.nan)
+        self.assertRaises(ValueError, lambda : bool(obj == 0))
+        self.assertRaises(ValueError, lambda : bool(obj == 1))
+        self.assertRaises(ValueError, lambda : bool(obj))
+
+        # empty
+        obj = self._construct(shape=0)
+        self.assertRaises(ValueError, lambda : bool(obj))
+
+        # invalid behaviors
+
+        obj1 = self._construct(shape=4,value=1)
+        obj2 = self._construct(shape=4,value=1)
+
+        def f():
+            if obj1:
+                print("this works and shouldn't")
+        self.assertRaises(ValueError, f)
+        self.assertRaises(ValueError, lambda : obj1 and obj2)
+        self.assertRaises(ValueError, lambda : obj1 or obj2)
+        self.assertRaises(ValueError, lambda : not obj1)
 
 class TestSeries(unittest.TestCase, Generic):
     _typ = Series
@@ -154,6 +188,14 @@ def test_get_numeric_data_preserve_dtype(self):
         expected = Series([],dtype='M8[ns]')
         self._compare(result, expected)
 
+    def test_nonzero_single_element(self):
+
+        s = Series([True])
+        self.assertRaises(ValueError, lambda : bool(s))
+
+        s = Series([False])
+        self.assertRaises(ValueError, lambda : bool(s))
+
 class TestDataFrame(unittest.TestCase, Generic):
     _typ = DataFrame
     _comparator = lambda self, x, y: assert_frame_equal(x,y)
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -296,12 +296,6 @@ def test_scalar_conversion(self):
         self.assert_(int(Series([1.])) == 1)
         self.assert_(long(Series([1.])) == 1)
 
-        self.assert_(bool(Series([True])) == True)
-        self.assert_(bool(Series([False])) == False)
-
-        self.assert_(bool(Series([True,True])) == True)
-        self.assert_(bool(Series([False,True])) == True)
-
     def test_astype(self):
         s = Series(np.random.randn(5),name='foo')
 
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -256,7 +256,7 @@ def test_indexing(self):
         df = DataFrame(randn(5,5),columns=['open','high','low','close','volume'],index=date_range('2012-01-02 18:01:00',periods=5,tz='US/Central',freq='s'))
         expected = df.loc[[df.index[2]]]
         result = df['2012-01-02 18:01:02']
-        self.assert_(result == expected)
+        assert_frame_equal(result,expected)
 
         # this is a single date, so will raise
         self.assertRaises(KeyError, df.__getitem__, df.index[2],)