Repository: pandas-dev/pandas

Issue:
Pandas 0.12 unexpected results when comparing dataframe to list or tuple
It seems that when comparing DataFrame to list or tuple of values (lenght of DataFrame columns), the resulting boolean DataFrame is incorrect.

`````` python
>>> import pandas
>>> print pandas.__version__
0.12.0
>>> import numpy as np
>>> print np.__version__
1.6.1
>>> df=pandas.DataFrame([ [-1, 0], [1, 2] ])
>>> df > (0, 1)
       0      1
0  False  False
1  False   True
>>> df > [0, 1]
       0      1
0  False  False
1  False   True

``'

Comparison with numpy array works as expected:
```python
>>> df > np.array([0, 1])
       0      1
0  False  False
1   True   True
``'

Note that the comparison behaved correctly at least in pandas 0.9.0:
```python
>>> import pandas
>>> print pandas.__version__
0.9.0
>>> df=pandas.DataFrame([ [-1, 0], [1, 2] ])
>>> df > (0, 1)
       0      1
0  False  False
1   True   True
>>> df > [0, 1]
       0      1
0  False  False
1   True   True
``'
``````



Discussion:
These yield different shapes (which is confusing as you are using a 2x2 frame)

A single list/tuple becomes a column, while a list-of-list yields rows

```
In [25]: df = DataFrame(np.arange(6).reshape(3,2))

In [26]: df>(2,2,2)
Out[26]: 
       0      1
0  False  False
1  False   True
2   True   True
I
n [30]: pd.DataFrame([2,2,2])
Out[30]: 
   0
0  2
1  2
2  2

In [31]: pd.DataFrame([[2,2,2]])
Out[31]: 
   0  1  2
0  2  2  2
```

```
In [29]: pd.DataFrame(np.array([2,2]))
Out[29]: 
   0
0  2
1  2

In [27]: df>np.array((2,2))
Out[27]: 
       0      1
0  False  False
1  False   True
2   True   True
```

OK I see my example was a bit shady because of the 2x2 size. Here's an example with 3x1 dataframe:

``` python
In [10]: df=pd.DataFrame(np.arange(6).reshape((3,2)))

In [11]: df
Out[11]:
   0  1
0  0  1
1  2  3
2  4  5
```

If list/tuple really becomes a column, why does not 1D array?

``` python
In [12]: df > [2, 2]
Out[12]:
      0     1
0  True  True
1  True  True
2  True  True
```

``` python
In [13]: df > np.array([2, 2])
Out[13]:
       0      1
0  False  False
1  False   True
2   True   True
```

I think both should result in the same since both the list and 1D array are... well, 1D objects.

This would also match what happens with numpy 2D arrays:

``` python
In [17]: df.values > [2, 2]
Out[17]:
array([[False, False],
       [False,  True],
       [ True,  True]], dtype=bool)
```

Similar example that confuses me:

``` python
In [24]: row_vector = np.atleast_2d([2,2])
In [25]: df > row_vector
Out[25]:
      0     1
0  True  True
1  True  True
2  True  True
```

``` python
In [26]: df.values > row_vector
Out[26]:
array([[False, False],
       [False,  True],
       [ True,  True]], dtype=bool)
```

Would not it be logical in this case that row_vector with shape (1,2) would be broadcasted to (3,2) before comparison?

EDIT: these examples were with pandas 0.12 and numpy 1.7.1

actually that's not correct

a passed `ndarray` is represented as it comes (as it defines rows/columns). 
A tuple or list (which is what you are presenting) is passed as a list-of-lists/tuples.
These are rows.

```
In [5]: pd.DataFrame([[1,2,3],[4,5,6]])
Out[5]: 
   0  1  2
0  1  2  3
1  4  5  6

In [6]: pd.DataFrame([1,2,3])
Out[6]: 
   0
0  1
1  2
2  3

In [7]: pd.DataFrame([[1,2,3]])
Out[7]: 
   0  1  2
0  1  2  3
```

This is exactly the same as numpy behavior. There isn't any implicit broadcasting,

```
In [13]: np.array([[1,2,3],[4,5,6]])
Out[13]: 
array([[1, 2, 3],
       [4, 5, 6]])

In [14]: np.array([[1,2,3],[4,5,6]]).shape
Out[14]: (2, 3)

In [15]: np.array([1,2,3])
Out[15]: array([1, 2, 3])

In [16]: np.array([1,2,3]).shape
Out[16]: (3,)

In [17]: np.array([[1,2,3]])
Out[17]: array([[1, 2, 3]])

In [18]: np.array([[1,2,3]]).shape
Out[18]: (1, 3)
```

You are just passing a list which is a column, that's it.

Remember that since you are not passing an index/columns, pandas has to follow
a defined behavior. If you had passed an index/columns then it WILL align on the index.

I would like to disagree about pandas following numpy behaviour here. Firstly, 1d numpy arrays do not have a defined direction - they are just 1d vectors. For example,

``` python
In [30]: a = np.array([1,2,3])

In [31]: a.shape
Out[31]: (3,)

In [32]: a
Out[32]: array([1, 2, 3])

In [33]: a.T
Out[33]: array([1, 2, 3])

In [34]: a==a.T
Out[34]: array([ True,  True,  True], dtype=bool)
```

Therefore, it is in my opinion rather dangerous to assume that lists or tuples don't have a shape but that 1d arrays would. I believe they should behave identically.

Second issue is that numpy does broadcasts with comparison operators, just as ssalonen showed above. I guess it would be OK if pandas didn't, but that should be an explicit and documented deviation from numpy semantics.

Third, regardless of the broadcasts, I believe the comparison operators in pandas are quite broken at the moment:

``` python
In [49]: df = pd.DataFrame(np.arange(6).reshape((3,2)))

In [50]: b = np.array([2, 2])

In [51]: b_r = np.atleast_2d([2,2])

In [52]: b_c = b_r.T

In [53]: df > b
Out[53]:
       0      1
0  False  False
1  False   True
2   True   True

In [54]: df > b_r
Out[54]:
      0     1
0  True  True
1  True  True
2  True  True

In [55]: df > b_c
Out[55]:
       0      1
0  False  False
1  False   True
2   True   True
```

I don't quite understand the element-wise comparisons made in the example above. Some broadcasts are necessarily made, but not in any logical fashion.

Also, the equality operator should work with the same semantics as greater than. However, it does not:

``` python
In [60]: df == b
Out[60]:
       0      1
0  False  False
1   True  False
2  False  False

In [61]: df == b_r
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[...]
TypeError: Could not compare [array([[2, 2]])] with block values

In [62]: df == b_c
Out[62]:
       0      1
0  False  False
1   True  False
2  False  False
```

To me it would seem that 1d and column vectors behave as broadcast row vectors in the comparison operators, while row vectors are more thoroughly broken. :-)

ok...

so the results from `df>b, df>b_r, df>b_c` should all be all the same?

`df>b_r` has a broadcasting issue (it is very strict and won't allow even a transpose, maybe I can relax it)


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -723,29 +723,65 @@ def eval(self, func, other, raise_on_error=True, try_cast=False):
         # make sure that we can broadcast
         is_transposed = False
         if hasattr(other, 'ndim') and hasattr(values, 'ndim'):
-            if values.ndim != other.ndim or values.shape == other.shape[::-1]:
-                values = values.T
-                is_transposed = True
+            if values.ndim != other.ndim:
+                    is_transposed = True
+            else:
+                if values.shape == other.shape[::-1]:
+                    is_transposed = True
+                elif values.shape[0] == other.shape[-1]:
+                    is_transposed = True
+                else:
+                    # this is a broadcast error heree
+                    raise ValueError("cannot broadcast shape [%s] with block values [%s]"
+                                     % (values.T.shape,other.shape))
+
+        transf = (lambda x: x.T) if is_transposed else (lambda x: x)
+
+        # coerce/transpose the args if needed
+        values, other = self._try_coerce_args(transf(values), other)
+
+        # get the result, may need to transpose the other
+        def get_result(other):
+            return self._try_coerce_result(func(values, other))
+
+        # error handler if we have an issue operating with the function
+        def handle_error():
 
-        values, other = self._try_coerce_args(values, other)
-        args = [values, other]
-        try:
-            result = self._try_coerce_result(func(*args))
-        except (Exception) as detail:
             if raise_on_error:
-                raise TypeError('Could not operate [%s] with block values [%s]'
+                raise TypeError('Could not operate %s with block values %s'
                                 % (repr(other), str(detail)))
             else:
                 # return the values
                 result = np.empty(values.shape, dtype='O')
                 result.fill(np.nan)
+                return result
+
+        # get the result
+        try:
+            result = get_result(other)
+
+        # if we have an invalid shape/broadcast error
+        # GH4576, so raise instead of allowing to pass thru
+        except (ValueError) as detail:
+            raise
+        except (Exception) as detail:
+            result = handle_error()
 
+        # technically a broadcast error in numpy can 'work' by returning a boolean False
         if not isinstance(result, np.ndarray):
-            raise TypeError('Could not compare [%s] with block values'
-                            % repr(other))
+            if not isinstance(result, np.ndarray):
+
+                # differentiate between an invalid ndarray-ndarray comparsion and
+                # an invalid type comparison
+                if isinstance(values, np.ndarray) and is_list_like(other):
+                    raise ValueError('Invalid broadcasting comparison [%s] with block values'
+                                     % repr(other))
+
+                raise TypeError('Could not compare [%s] with block values'
+                                % repr(other))
 
-        if is_transposed:
-            result = result.T
+        # transpose if needed
+        result = transf(result)
 
         # try to cast if requested
         if try_cast:


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -4709,6 +4709,75 @@ def test_float_none_comparison(self):
 
         self.assertRaises(TypeError, df.__eq__, None)
 
+    def test_boolean_comparison(self):
+
+        # GH 4576
+        # boolean comparisons with a tuple/list give unexpected results
+        df = DataFrame(np.arange(6).reshape((3,2)))
+        b = np.array([2, 2])
+        b_r = np.atleast_2d([2,2])
+        b_c = b_r.T
+        l = (2,2,2)
+        tup = tuple(l)
+
+        # gt
+        expected = DataFrame([[False,False],[False,True],[True,True]])
+        result = df>b
+        assert_frame_equal(result,expected)
+
+        result = df.values>b
+        assert_array_equal(result,expected.values)
+
+        result = df>l
+        assert_frame_equal(result,expected)
+
+        result = df>tup
+        assert_frame_equal(result,expected)
+
+        result = df>b_r
+        assert_frame_equal(result,expected)
+
+        result = df.values>b_r
+        assert_array_equal(result,expected.values)
+
+        self.assertRaises(ValueError, df.__gt__, b_c)
+        self.assertRaises(ValueError, df.values.__gt__, b_c)
+
+        # ==
+        expected = DataFrame([[False,False],[True,False],[False,False]])
+        result = df == b
+        assert_frame_equal(result,expected)
+
+        result = df==l
+        assert_frame_equal(result,expected)
+
+        result = df==tup
+        assert_frame_equal(result,expected)
+
+        result = df == b_r
+        assert_frame_equal(result,expected)
+
+        result = df.values == b_r
+        assert_array_equal(result,expected.values)
+
+        self.assertRaises(ValueError, lambda : df == b_c)
+        self.assert_((df.values == b_c) is False)
+
+        # with alignment
+        df = DataFrame(np.arange(6).reshape((3,2)),columns=list('AB'),index=list('abc'))
+        expected.index=df.index
+        expected.columns=df.columns
+
+        result = df==l
+        assert_frame_equal(result,expected)
+
+        result = df==tup
+        assert_frame_equal(result,expected)
+
+        # not shape compatible
+        self.assertRaises(ValueError, lambda : df == (2,2))
+        self.assertRaises(ValueError, lambda : df == [2,2])
+
     def test_to_csv_deprecated_options(self):
 
         pname = '__tmp_to_csv_deprecated_options__'