API: Add equals method to NDFrames.
State: closed

Also adds `array_equivalent`, which
is similar to `np.array_equal` except that it handles object arrays and
treats NaNs in corresponding locations as equal.

closes https://github.com/pydata/pandas/issues/5183


Labels:

Comments:
pls run a perf check on this (test_perf.sh)

these comparisons are used everywhere

do u need the shape check?
the null check might kill perf on this
why are u not doing == and != ?

@jreback - seems like it doesn't work for this example, but we could be missing something

``` python
left = pd.Float64Index([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, nan], dtype='object')
right = pd.Float64Index([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, nan], dtype='object')

# OR 
left = np.array([1.0, 2.0, nan], dtype=object)
right= np.array([1.0, 2.0, nan], dtype=object)
```

(fully enumerated here - https://gist.github.com/unutbu/7070565)

to be explicit:

``` python
left = np.array([1.0, 2.0, nan], dtype=object)
    ...: right= np.array([1.0, 2.0, nan], dtype=object)
    ...: 

left != right
Out[16]: array([False, False, False], dtype=bool)

left != left
Out[17]: array([False, False, False], dtype=bool)

right != right
Out[18]: array([False, False, False], dtype=bool)

nan != nan
Out[19]: True
```

Though I guess they compare true with `==` so not a real issue - we were going back and forth on another PR b/c sometimes nan handling can be confusing :P

you have to astype to float! before you can do the comparison (not sure exactly why) only works if they r all float values (so you need to do it in a try except)

@jreback: I'm working on installing vbench and figuring out how to run test_perf.sh...

@jreback: When I run

```
time ./test_perf.sh -b array-equivalent -t array-equivalent^ 
```

I get 

```
sqlalchemy.exc.IntegrityError: (IntegrityError) column checksum is not unique u'INSERT INTO benchmarks (checksum, name, description) VALUES (?, ?, ?)' ('ea1993ef61c3cc4e871d2cce3c5d983c', 'eval_frame_chained_cmp_python', None)
```

I see I can limit `test_perf.sh` to one test, such as 

```
time ./test_perf.sh -b array-equivalent -t array-equivalent^ -r reindex
```

which yielded

```
    Invoked with :
    --ncalls: 3
    --repeats: 3


    -------------------------------------------------------------------------------
    Test name                                    | head[ms] | base[ms] |  ratio   |
    -------------------------------------------------------------------------------
    reindex_frame_level_align                    |   2.6046 |  10.1856 |   0.2557 |
    dataframe_reindex                            |   0.4900 |   0.6377 |   0.7684 |
    frame_reindex_axis0                          | 110.6919 | 126.7160 |   0.8735 |
    frame_reindex_columns                        |   0.4164 |   0.4683 |   0.8890 |
    frame_reindex_both_axes_ix                   |  43.5000 |  46.9437 |   0.9266 |
    reindex_frame_level_reindex                  |   2.3306 |   2.3570 |   0.9888 |
    frame_reindex_upcast                         |  16.1486 |  16.2884 |   0.9914 |
    reindex_fillna_pad_float32                   |   0.5860 |   0.5894 |   0.9942 |
    reindex_fillna_backfill_float32              |   0.5997 |   0.6014 |   0.9972 |
    frame_reindex_both_axes                      |  46.7057 |  46.7397 |   0.9993 |
    reindex_daterange_pad                        |   2.9510 |   2.9523 |   0.9995 |
    reindex_fillna_backfill                      |   1.0234 |   1.0213 |   1.0020 |
    reindex_fillna_pad                           |   0.8663 |   0.8514 |   1.0175 |
    reindex_multiindex                           |   1.5457 |   1.5034 |   1.0281 |
    frame_reindex_axis1                          | 558.3910 | 510.9200 |   1.0929 |
    reindex_daterange_backfill                   |   3.4040 |   2.9933 |   1.1372 |
    -------------------------------------------------------------------------------
    Test name                                    | head[ms] | base[ms] |  ratio   |
    -------------------------------------------------------------------------------

    Ratio < 1.0 means the target commit is faster then the baseline.
    Seed used: 1234

    Target [5c6116c] : Merge pull request #5281 from cancan101/index_meta_data_doc

    DOC: Added versionadded for "Setting index metadata"
    Base   [8c8ef7d] : ENH: Add array_equivalent, to address the handling of NaNs when comparing arrays for equality.

    Added NDFrame.equals

    Index, Float64Index, and MultiIndex's equal method now uses array_equivalent
    instead of np.array_equal.
```

Clearly I don't know what I'm doing. What is the right `test_perf.sh` command?
I see there are other choices for `-r` in `pandas/vb_suite`. But which is the right/relevant one(s)?

b should be the commit before 1st of yours and t should be the last commit of yours

generally I rebase to master before this

With array-equivalent  rebased to master,

```
time ./test_perf.sh -b master -t array-equivalent 
```

yields [vb_suite.log](https://gist.github.com/unutbu/7075764)

```
concat_series_axis1                          | 204.8774 |  83.7650 |   2.4459 |
reindex_frame_level_align                    |   8.9770 |   1.2484 |   7.1910 |
```

so look at these in master and in your PR using %prun...and see if you can figure out what's up...

With the current commit, `test_perf.sh` yields

```
groupby_simple_compress_timing               |  54.9030 |  47.4270 |   1.1576 |
frame_iloc_dups                              |   0.3117 |   0.2663 |   1.1704 |
index_int64_intersection                     |  41.4550 |  33.6334 |   1.2326 |
groupby_series_simple_cython                 |   7.6556 |   5.9413 |   1.2885 |
series_align_left_monotonic                  |  30.3703 |  22.4893 |   1.3504 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

I'm going to try adding a check for Int64Index arrays next...

also try doing test_perf again....these could be 'random'.....(e.g. if they are not similar on subsequent runs) then its just an artifact of the data....you can also try with a bigger n (numcalls)

@unutbu have a look at #5219 I believe your replacement will simply be called by that, yes?

question here - why do you need to cast it to float first? I thought it worked with just ==? I'm sure I'm missing something but just wanted to make sure we had an example that fails using ==. (or maybe it's just float dtype that fails)

I think object dtype that has floats in it (iow float64index) fails ; not sure why though 

@jreback Regarding https://github.com/pydata/pandas/pull/5219, yes, I am striving to make `array_equivalent` a drop-in replacement for `np.array_equal`. It should behave exactly like `np.array_equal` except that `NaN`s in corresponding locations should be treated as equal. 

The tests in `tests/test_common.py` in the `test_array_equivalent` show the behavior I'm currently testing for. 

Perhaps related: weird bug in numpy assert_array_equal I came across in a test a while ago (that I can't repo outside of the test suite) #4458.

(@unutbu: your doing pandas' pull requests now, awesome!)

@jtratner: I did try coercing to float (https://github.com/pydata/pandas/pull/5283#discussion-diff-7092284), but found there were problems. (See the link for more details.)

(Fixed incorrect link.)

Currently, `array_equivalent` uses `np.isnan` instead of `pd.isnull` because it is faster, but  since it raises TypeError or NotImplementedError (Python2.6 or 3.2) on object arrays (unlike `pd.isnull`), I'm using `np.array_equal` as a fallback.

Again, can we take a quick step back here: what's an example where it
_doesn't_ work to compare ndarrays with `==` (let's assume that
array_equivalent always gets ndarrays for now). So you don't have to deal
with Index subclasses - will always get actual ndarrays.

if you pass array of floats with dtype object and some are nan, it compares
incorrectly with ==, right?

So if you're thinking of Float64Index - just do '.view(ndarray)' so you're
not dealing with anything on pandas level.

Once we get it to work for ndarray, then can consider what to do for
NDFrame and friends. (trivial to view Index as ndarray for now)
On Oct 21, 2013 5:51 PM, "Jeffrey Tratner" jeffrey.tratner@gmail.com
wrote:

> Again, can we take a quick step back here: what's an example where it
> _doesn't_ work to compare ndarrays with `==` (let's assume that
> array_equivalent always gets ndarrays for now). So you don't have to deal
> with Index subclasses - will always get actual ndarrays.
> 
> if you pass array of floats with dtype object and some are nan, it
> compares incorrectly with ==, right?

@jtratner: I don't quite understand. What is the "it" in the phrase "where it _doesn't_ work..."?

Currently the test

```
assert array_equivalent(np.array([nan, None], dtype='object'),
                        np.array([nan, None], dtype='object')) 
```

pass.s

Finally have a computer - just need to look at something for myself. I
mean, where `a == b | ((a != a) & ( b != b))` doesn't work, since that's
what I'd expect to work everywhere with a check for matching dtypes.

I just used this:

``` python
def array_equiv(n1, n2):
    return n1.shape == n2.shape and n1.dtype == n2.dtype and ((n1 == n2) | ((n1 != n1) & (n2 != n2))).all()
```

And it worked for all of these - am I missing why this is complicated? Is there a numpy version issue?

```
import numpy as np
nan = np.nan
for func in [
             lambda : np.array([0.1, 0.2, np.nan, 0.3], dtype=object),
             lambda : np.array([0.1, 0.2, np.nan, 0.3, np.nan], dtype=float),
             lambda : np.array([None, None, np.nan, None], dtype=object),
             lambda : np.array([], dtype=object)]:
    assert array_equiv(func(), func())
```

Then callers should be responsible for checking anything at pandas-level.

How about:

```
import numpy as np
import pandas as pd
import pandas.core.common as com

def array_equiv(n1, n2):
    return n1.shape == n2.shape and n1.dtype == n2.dtype and ((n1 == n2) | ((n1 != n1) & (n2 != n2))).all()

index = np.random.random(10)
df1 = pd.DataFrame(np.random.random(10,), index=index, columns=['floats'])
df1['dates'] = pd.date_range('2000-1-1', periods=10, freq='T')
df1.ix[::2] = np.nan

print(array_equiv(df1.values, df1.values))
# False
```

However, my `array_equivalent` does not handle object arrays correctly either. To work around the above problem, I had to add code to NDFrame.equals to test each column separately. 

okay, thanks - just wanted to make sure we had something that explicitly didn't work for the simpler version.

actually...why don't we do both...

use the simpler version...if its True (then we are done as we don't have false positives), however a False can fall back to the slower version

My original PR used `pd.isnull`. This version handled anything thrown at it correctly -- scalars, lists, DataFrames, etc.. The only problem was the test_perf.sh showed one benchmark was 7x slower, another 2-3x slower.

As a compromise between usefulness and speed, I changed `pd.isnull` to `np.isnan` because %timeit showed `np.isnan` to be about 7x faster when it did not raise a TypeError (which it does on object arrays). 

This compromise has better `test_perf.sh` results. The worst benchmark is now ~1.15 worse than upstream/master. But `array_equivalent` currently does not handle NaNs in `object` arrays correctly. An example where it fails is on `df1.values` above.

@unutbu per my comment - nan can only be in object or floating, right? so you can use `np.array_equal` for everything else. Not sure if you need to special case NaT too...

`NaT` should be ok its a number; so just special case object/float then

Okay! This sounds promising. Will work on this, but I have to go now.