ERR: Boolean comparisons of a Series vs None will now be equivalent to null comparisons
State: closed

xref, #1079

numpy 1.10 shows a deprecation warning for this. this should eliminate a bunch of these.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @njsmith 
cc @seberg

much more tricky than I thought

Sorry, what's the question?

so this is the reality check

cc @njsmith 
@jorisvandenbossche @shoyer 

scalar comparisons vs null are always False, but in an array context we have
always had them as True. Is this right?
Obviously the user should use `.isnull` or even `np.isnan`.
Any reason NOT to change this?

this PR

```
In [6]:  s = Series([1,np.nan,'b'])

In [7]: s == s
Out[7]: 
0     True
1    False
2     True
dtype: bool

In [8]: s == None
Out[8]: 
0    False
1     True
2    False
dtype: bool

In [9]: s == np.nan
Out[9]: 
0    False
1     True
2    False
dtype: bool
```

0.16.2

```
In [1]: s = Series([1,np.nan,'b'])

In [2]: s == s
Out[2]: 
0     True
1    False
2     True
dtype: bool

In [3]: s == None
Out[3]: 
0    False
1    False
2    False
dtype: bool

In [4]: s == np.nan
Out[4]: 
0    False
1    False
2    False
dtype: bool
```

and numpy (both current and 1.10/master)

```
In [9]: s.values
Out[9]: array([1, nan, 'b'], dtype=object)

In [10]: s.values == None
/Users/jreback/anaconda/bin/ipython:1: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  #!/bin/bash /Users/jreback/anaconda/bin/python.app
Out[10]: False

In [11]: s.values == np.nan
Out[11]: array([False, False, False], dtype=bool)

In [14]: s.values==s.values
/Users/jreback/anaconda/bin/ipython:1: FutureWarning: numpy equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.
  #!/bin/bash /Users/jreback/anaconda/bin/python.app
Out[14]: array([ True,  True,  True], dtype=bool)
```

Here is what I think we want (which is none of the above, but IMO also what NumPy should be doing):

```
In [6]:  s = Series([1,np.nan,'b'])

In [7]: s == s
Out[7]: 
0     True
1    False
2     True
dtype: bool

In [8]: s == None
Out[8]: 
0    False
1    False
2    False
dtype: bool

In [9]: s == np.nan
Out[9]: 
0    False
1    False
2    False
dtype: bool
``

I don't think it's a good idea to pretend that `None` is the same thing as `np.nan`. Likewise, we should not change the semantics of nan, which is that it compares False to everything.
```

Was going to say exactly the same thing but @shoyer types faster than me.

On Wed, Jul 15, 2015 at 3:03 PM, Stephan Hoyer notifications@github.com
wrote:

> Here is what I think we want:
> 
> In [6]:  s = Series([1,np.nan,'b'])
> 
> In [7]: s == s
> Out[7]:
> 0     True
> 1    False
> 2     True
> dtype: bool
> 
> In [8]: s == None
> Out[8]:
> 0    False
> 1    False
> 2    False
> dtype: bool
> 
> In [9]: s == np.nan
> Out[9]:
> 0    False
> 1    False
> 2    False
> dtype: bool
> ``
> 
> I don't think it's a good idea to pretend that `None` is the same thing as `np.nan`. Likewise, we should not change the semantics of nan, which is that it compares False to everything.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10569#issuecomment-121730296.

## 

Nathaniel J. Smith -- http://vorpus.org

> > I don't think it's a good idea to pretend that `None` is the same thing as `np.nan`. Likewise, we should not change the semantics of nan, which is that it compares False to everything.

This IS possible but ONLY for object dtypes

`None` -> `np.nan` for other dtypes and comparison

and `None` -> `NaT` for `m|M` comparisons

But nan != nan, so even if None is coerced to nan (ugh, why does numpy do
that), then the comparisons should still return False.

On Wed, Jul 15, 2015 at 3:05 PM, jreback notifications@github.com wrote:

>  I don't think it's a good idea to pretend that None is the same thing as
> np.nan. Likewise, we should not change the semantics of nan, which is
> that it compares False to everything.
> 
>  This IS possible but ONLY for object dtypes
> 
> None -> np.nan for other dtypes and comparison
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10569#issuecomment-121730659.

## 

Nathaniel J. Smith -- http://vorpus.org

ok all fixed up. So at the end of the day I removed the DeprecationWarning that numpy is now showing (well one of them anyhow....still some others).

now allow comparison vs `None` for non-object dtypes (prob not used very much anyhow as it used to raise).

cc @kshedden  
cc @bashtage

this PR removes some numpy warnings that happen in 1.10; long story short is that a lot are coming from some of the stata test code (basically you can't compare M8[ns] dtypes with integer/floats anymore). So in fixing this, I think the actual tests might be broken as in return wrong values.

(you don't need to use numpy master, this is with 1.9.2)
e.g.

```
[jreback-~/pandas] nosetests  pandas/io/tests/test_stata.py
/Users/jreback/anaconda/lib/python2.7/site-packages/numpy/core/numeric.py:2367: FutureWarning: numpy equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.
  return bool(asarray(a1 == a2).all())
.......................FF.....................
======================================================================
FAIL: test_read_chunks_115 (pandas.io.tests.test_stata.TestStata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jreback/pandas/pandas/io/tests/test_stata.py", line 986, in test_read_chunks_115
    check_datetimelike_compat=True)
  File "/Users/jreback/pandas/pandas/util/testing.py", line 768, in assert_frame_equal
    check_datetimelike_compat=check_datetimelike_compat)
  File "/Users/jreback/pandas/pandas/util/testing.py", line 697, in assert_series_equal
    right.values))
AssertionError: [datetimelike_compat=True] [  1.47959720e+12] is not equal to ['2006-11-19T18:13:20.000000000-0500'].

======================================================================
FAIL: test_read_chunks_117 (pandas.io.tests.test_stata.TestStata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jreback/pandas/pandas/io/tests/test_stata.py", line 929, in test_read_chunks_117
    check_datetimelike_compat=True)
  File "/Users/jreback/pandas/pandas/util/testing.py", line 768, in assert_frame_equal
    check_datetimelike_compat=check_datetimelike_compat)
  File "/Users/jreback/pandas/pandas/util/testing.py", line 697, in assert_series_equal
    right.values))
AssertionError: [datetimelike_compat=True] [  1.47959720e+12] is not equal to ['2006-11-19T18:13:20.000000000-0500'].

----------------------------------------------------------------------
Ran 46 tests in 1.108s

```

can you have a look. thxs.

Did you take out the `try except` to get those?

I am doing what was inside the  except inside assert_series_equal

but I think it was completley failing before and just covered up

I just pushed a change to #9882 that should fix the AssertionErrors above.  It was due to inconsistent options set in the test when reading by chunk and when reading the full file.

However you will still get the numpy warnings I think.  It is complaining about comparing pandas timestamps to numpy datetime64 or Python datetimes.  I'm quite sure that the dates/times are the same, but they sometimes have different types.  To get around this, if assert_frame_equal fails we try again comparing elementwise.  Apparently the three date/time representations have elementwise == defined in a way that works consistently across the types.

The root issue is that convert_year_month_safe and related methods in the StataReader check to see if the date falls in the valid pandas range, and uses a timestamp if they do, and a Python datetime if they do not.  When reading the file by chunk, this decision must be made based on the data in the chunk, so may differ from the decision used when reading the entire file.  Many of the example dta files have dates that fall outside of the pandas date range (e.g. year=2).  

I don't see a solution that both checks the values and avoids the warnings, except converting everything to Python datetimes.  Our current solution correctly checks the values but emits warnings.

@kshedden thanks I incorporated your fix and works! thanks!

@jorisvandenbossche I made those changes. Also cleaned the whatsnew a bit.

https://github.com/pydata/pandas/commit/2da060ce4a3635ece00ed947d26510be1a13cb08 should do it

@jreback Nice!

Some comments:
- `s = Series(range(3.))` will raise, `range` only takes integers
- the note on `None==None` returning None is interesting, but maybe we need to explicitely note that we do not follow that behaviour?

oops that was a typo. Yeh I though I made clear that `None` is equiv to `np.nan`. 

BTW, that was something I wanted to note previously on the actual PR. The coercion of None to np.nan leads to some tricky behaviour where somebody would discriminate between both in object serieses, and introduces differences with numpy:

```
In [21]: pd.Series(['a', None]) == pd.Series(['a', None])
Out[21]:
0     True
1    False
dtype: bool

In [22]: pd.Series(['a', None]).values == pd.Series(['a', None]).values
Out[22]: array([ True,  True], dtype=bool)
```

(Not saying that this is a good idea (the discriminating between None and np.nan), but technically for object dtyped series it is possible, so people will do) So I wonder if we shouldn't do this as well when comparing with `==` (and treat None and np.nan differently for object dtypes) 

@jorisvandenbossche yes you are absoutely right. You would have to do that in an `object` dtype. People will abuse even this!. I am not sure that's worth a caveat/gotcha though (prob read even less than that the actual docs :)

@jorisvandenbossche all that said if you find things unclear, pls update!

https://github.com/pydata/pandas/commit/b06105595868cf35bd60cc1b131de94ea3577176

But, with the current behaviour it is a caveat as well as it deviates from standard python and numpy behaviour:

```
In [27]: None == None
Out[27]: True

In [28]: pd.Series([None]) == pd.Series([None])
Out[28]:
0    False
dtype: bool
```

So it is a bit the question which caveat your prefer ...

hah!

yeh I think we just discourage `== None` and `== np.nan` (which has been longstanding) and promote `isnull/notnull`. @wesm did for prob this exact reason/confusion.