BUG: Suppress FutureWarning when comparing object arrays
State: closed

Fixes #7065


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
This should use a context manager instead of filtering the warning in all cases. e.g., see https://docs.python.org/2/library/warnings.html#temporarily-suppressing-warnings

Thanks, @shoyer, I've made the change.

can u run the vbench suite - this is called a lot (though maybe not so much on object arrays)

@jreback: With the context manager:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_nonunique_unequal                      |   0.0200 |   0.0103 |   1.9385 |
frame_object_unequal                         |   0.0190 |   0.0097 |   1.9590 |
frame_float_unequal                          |   0.0193 |   0.0096 |   2.0083 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

With module-level warnings filter:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_nonunique_unequal                      |   0.0106 |   0.0106 |   1.0000 |
frame_float_unequal                          |   0.0100 |   0.0097 |   1.0328 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

that's what I thought

do u want to attempt a cython routine?
should be pretty straightforward ?

look in pandas/lib.pyx

you can start with something like isnullobj but takes an object array of 1-d and returns a bool

untested, but I think something like this should work (you can put in lib.pyx)

```
# return ((left == right) | (pd.isnull(left) & pd.isnull(right))).all()

# these are already object so no need to make sure
lib.array_equivalent_object(left.ravel(), right.ravel())

@cython.wraparound(False)
@cython.boundscheck(False)
def array_equivalent_object(ndarray left, ndarray right):
    cdef Py_ssize_t i, n
    cdef object lobj, robj

    n = len(left)
    for i from 0 <= i < n:
        lobj = left[i]
        robj = right[i]

        # we are either not equal or both nan
        # I think None == None will be true here
        if lobj != right:
            if _checknull(lobj) and _checknull(right):
                continue
            return False
    return True
```

With Cython array_equal_object:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_nonunique_unequal                      |   0.0106 |   0.0103 |   1.0308 |
frame_float_unequal                          |   0.0100 |   0.0096 |   1.0413 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

I'm using `test_perf.sh -b master -t equal-warning -r unequal`; I'm not sure why `frame_object_unequal` ran the first time (when using a contextmanager), but not now...

@unutbu so my routine worked as is? I didn't even test it...hahah

ok this looks good (need any more tests?)

_is this always an object array?_: `lib.array_equivalent_object` will get called when `left` is an object array. `right` might not be.

@unutbu hmm, so wrap with `com._ensure_object(...)` (which will make it one if necessary
may need some tests

@jreback: Sorry, I mispoke. It is documented in `array_equivalent`'s doc string that `left` and `right` are assumed to have the same dtype. We check the dtype in the `equals` method(s) before calling `array_equivalent`. So if `left` is an object array, I think we can assume `right` is an object array without having to call `com._ensure_object`.

@unutbu ok, that's what I thought originally.

Ok, see once this passes, looks ok to me (not sure why those tests are fialing).
It IS possible something is not being picked up in the cython comparisons correctly.

you need to wrap both left and right (before passing) with `_values_from_object` (which makes them an ndarray if necessary), which is why the timeseries types fail

and maybe add a check if `type(left) != type(right)` somewhere? (is it needed?)

e.g. `Index([1,2,3]).equals(np.array([1,2,3])` should be False?

The test was failing because I was calling `lib.array_equivalent_object` in two places; the second one on float arrays.  Current commit fixes this.

Note I've also modified v.0.13.1 to fix the type Joris [pointed out](GH 8509). The new example shows DataFrames must have indexes in the same order for them to compare equally.

Can you explain what `_values_from_object` is for? When do these objects have a `get_values` attribute?

_is their an example in basics.rst as well?_: Yes, there we show `(df+df).equals(df*2)`. Sorry I can't seem to add line notes; my browser is too old and github seems to no longer cooperate with it. Will upgrade my OS when I build up enough courage.

`get_values` will do dense conversions. All `_values_from_object` does is `getattr(v,'values',v)` iow if it has a values attribute (e.g. its a pandas object), then get the underlying ndarray.

`.values` is 'guaranteed' to not do much (and simply be an attribute access).

`.get_values()` might do more (for certain types), e.g. `Categorical/SparseArray`

@unutbu hahah, about the example, I only mentioned the example because you added to some stuff in v0.13.0. If you think its appropriate add to basics.rst as well.

Hold on then -- It's also part of the doc string contract that `left` and `right` are NumPy arrays. So we shouldn't need to call `_values_from_object`.

Something else wonky going on then: see this test https://github.com/pydata/pandas/blob/master/pandas/tseries/tests/test_base.py#L81

This: https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1050

should be: `array_equivalent(self.values,other.values)`

actually `core/index.py` needs a bunch of updates for this same thing (they should ALL be sending in their underlying values). my bad.

`nosetests pandas/tests/test_base.py` is passing for me locally. Maybe my local changes haven't been pushed properly.

`pandas/tseries/tests/test_base.py` (its a DatetimeIndex as an object) (this is somewhat new, I moved it when I added `TimedeltaIndex`)

their are 4`def equals` in `core/index.py` all need changing as above (and `FloatIndex` (though `Float64Index` can use its existing because it cannot by definition have anything but `NaN`).

you need to change `_checkull` and use something like this: https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L573

I don't think this is exported. BUT, I think you can simply move it `lib.pyx` (then put a ref back in `tslib.pyx`)

@jreback: I think I need some help with the Cython. I can get `nosetests pandas/tseries/tests/test_base.py` to pass if I change `_checknull` to `checknull_old` in lib.pyx/array_equivalent_object, but I'm not sure what is the correct way to fix `_checknull`.

nvm what I was writing above

_checknull should work

only difference with that and _checknull_old is that old checks for infs