BUG: Timestamp == date match stdlib
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

ATM we have one reasonable behavior `Timestamp("2020-09-04") == date(2020, 9, 4)` and two un-reasonable behaviors: ``Timestamp("2020-09-04").tz_localize("US/Pacific") == date(2020, 9, 4)`, `Timestamp.now() == Timestamp.now().date()`.  Since the stdlib datetime doesnt consider `datetime(2020, 9, 4) == date(2020, 9, 4)`, this follows the stdlib and considers them never equal.


I'm still getting one test failure locally.


cc @mroeschke 

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
Seems reasonable. In general do you think we should reel back `datetime.date` support in ops with datetime64 dtypes?
> In general do you think we should reel back datetime.date support in ops with datetime64 dtypes?

Yes.  I think indexing should behave like comparisons, in which case date objects would not be allowed.  If we were to allow date objects in indexing, I think it would make more sense to treat `date(2020, 9, 5)` like `"2020-09-05"` than like `datetime(2020, 9, 5)`, i.e. like a partial-slice.  But i think thats more hassle than its worth.
Shouldn't we rather see this as an API change than a bug fix? 
I would say yes. I can imagine people relying on this to filter to all timestamps on a given date.
> I can imagine people relying on this to filter to all timestamps on a given date.

That's conceivable, but I think its much more likely that people are assuming that it casts to datetime-at-midnight and so are silently getting incorrect results.
> its much more likely that people are assuming that it casts to datetime-at-midnight and so are silently getting incorrect results.

Well, and they will then also start to see a warning to make them aware of this.

---

Now, it's not only about the "timestamp getting normalized to midnight when compared to date" behaviour, because as I understand it with this PR also Timestamps as midnight will no longer compare equal? I think we should first deprecate that.
Updated to deprecate.
The two places where we are exposing the warning to the user end up getting called through comp_method_OBJECT_ARRAY and array_equivalent_object, respectively.  The call stacks being:

```
pandas/core/groupby/groupby.py:858: in apply
    result = self._python_apply_general(f, self._selected_obj)
pandas/core/groupby/groupby.py:893: in _python_apply_general
    return self._wrap_applied_output(
pandas/core/groupby/generic.py:1196: in _wrap_applied_output
    return self._concat_objects(keys, values, not_indexed_same=not_indexed_same)
pandas/core/groupby/groupby.py:1205: in _concat_objects
    result = concat(
pandas/core/reshape/concat.py:274: in concat
    op = _Concatenator(
pandas/core/reshape/concat.py:454: in __init__
    self.new_axes = self._get_new_axes()
pandas/core/reshape/concat.py:519: in _get_new_axes
    return [
pandas/core/reshape/concat.py:520: in <listcomp>
    self._get_concat_axis() if i == self.bm_axis else self._get_comb_axis(i)
pandas/core/reshape/concat.py:576: in _get_concat_axis
    concat_axis = _make_concat_multiindex(
pandas/core/reshape/concat.py:628: in _make_concat_multiindex
    mask = (isna(level) & isna(key)) | (level == key)
pandas/core/indexes/base.py:139: in cmp_method
    result = ops.comp_method_OBJECT_ARRAY(op, self._values, other)
pandas/core/ops/array_ops.py:55: in comp_method_OBJECT_ARRAY
    result = libops.scalar_compare(x.ravel(), y, op)
pandas/_libs/ops.pyx:91: in pandas._libs.ops.scalar_compare
    result[i] = PyObject_RichCompareBool(x, val, flag)
```

```
pandas/core/frame.py:463: in __init__
    mgr = init_dict(data, index, columns, dtype=dtype)
pandas/core/internals/construction.py:247: in init_dict
    arrays = Series(data, index=columns, dtype=object)
pandas/core/series.py:282: in __init__
    data, index = self._init_dict(data, index, dtype)
pandas/core/series.py:383: in _init_dict
    s = s.reindex(index, copy=False)
pandas/core/series.py:4423: in reindex
    return super().reindex(index=index, **kwargs)
pandas/core/generic.py:4645: in reindex
    if all(
pandas/core/generic.py:4646: in <genexpr>
    self._get_axis(axis).identical(ax)
pandas/core/indexes/base.py:4333: in identical
    self.equals(other)
pandas/core/indexes/multi.py:3293: in equals
    return array_equivalent(self._values, other._values)
pandas/core/dtypes/missing.py:429: in array_equivalent
    return _array_equivalent_object(left, right, strict_nan)
pandas/core/dtypes/missing.py:468: in _array_equivalent_object
    return lib.array_equivalent_object(
pandas/_libs/lib.pyx:584: in pandas._libs.lib.array_equivalent_object
    elif not (PyObject_RichCompareBool(x, y, Py_EQ) or
```

I don't see anywhere in there where it makes sense to suppress this warning.

> And will the behaviour change here after the deprecationg has been enforced?


The constructor test you commented on will not, but the groupby.apply test raises a KeyError ATM if the deprecation is enforced immediately.

But for example the second case:

```
In [7]: import datetime  

In [8]: v = datetime.date.today()  

In [9]: tup = v, v 

In [10]: pd.DataFrame({tup: pd.Series(range(3), index=range(3))}, columns=[tup])  
Out[10]: 
   (2020-09-14, 2020-09-14)
0                         0
1                         1
2                         2
```

This seems to only involve date objects, not Timestamps. Where is the Timestamp comparison coming from? (I can't directly get this from the stacktrace)
> This seems to only involve date objects, not Timestamps. Where is the Timestamp comparison coming from? (I can't directly get this from the stacktrace)


```
> pandas/core/dtypes/missing.py(429)array_equivalent()
-> return _array_equivalent_object(left, right, strict_nan)
(Pdb) u
> pandas/core/indexes/multi.py(3293)equals()
-> return array_equivalent(self._values, other._values)
(Pdb) u
> pandas/core/indexes/base.py(4329)identical()
-> self.equals(other)
(Pdb) u
> pandas/core/generic.py(4714)<genexpr>()
-> self._get_axis(axis).identical(ax)
(Pdb) u
> pandas/core/generic.py(4713)reindex()
-> if all(
(Pdb) u
> pandas/core/series.py(4377)reindex()
-> return super().reindex(index=index, **kwargs)
(Pdb) u
> pandas/core/series.py(379)_init_dict()
-> s = s.reindex(index, copy=False)
(Pdb) u
> pandas/core/series.py(280)__init__()
-> data, index = self._init_dict(data, index, dtype)

```
@jbrockmendel posting basically the same stacktrace again without any explanation is not going to help much. You will need to understand or explain why it happens, or whether it can be avoided, as *in theory* we shouldn't be warning here (the user shouldn't see a warning, as it cannot do anything about it)
@jorisvandenbossche if the stacktrace is clear, then i dont understand what you're looking for [with](https://github.com/pandas-dev/pandas/pull/36131#issuecomment-691842472) "Where is the Timestamp comparison coming from? (I can't directly get this from the stacktrace)"

ATM in array_equivalent_object we catch TypeErrors that result from trying to compare tznaive vs tzaware and treat those as not-equal (xref #28507).  We could do warning-suppression there if we felt like it.

There are also a bunch of issues regarding the desired behavior of the `.equals` method.  Resolving those could conceivably end up getting rid of the calls that get here.
> if the stacktrace is clear,

Maybe for you the stacktrace is clear, but for me it is not. So I am looking for a human explaining it in words / with concrete examples.

The example is using dates, not Timestamps, so why do we do any comparison involving a Timestamp?
Within `Series._init_dict`, we call

```
s = create_series_with_explicit_dtype(values, index=keys, dtype=dtype, dtype_if_empty=np.float64)
```

where `keys = ((datetime.date(2020, 9, 26), datetime.date(2020, 9, 26)),)`.  This comes back with:

```
>>> s.index
MultiIndex([('2020-09-26', '2020-09-26')],
           )
>>> s.index[0]
(Timestamp('2020-09-26 00:00:00'), Timestamp('2020-09-26 00:00:00'))
```

Looks like the date->Timestamp cast occurs within `factorize_from_iterables` called from `MultiIndex.from_arrays`.

Just after the `create_series_with_explicit_dtype` is the `s.reindex(index)` call in the stacktrace above, which does the comparison to see if it can short-circuit.
The sticking point here is what to do about the unwanted warnings in two tests.  We _could_ suppress them at any point in the call stack, but I don't see a way to do that without also suppressing warnings that users _should_ see.

The warning message says `"Use ts == pd.Timestamp(dt) instead"` which will get rid of the warning.

The only other option I see is to just fix the broken behavior instead of deprecating it.
any outstanding issues left here?
btw do we have an actual issue for this? (that this PR would close)
There are still some warnings raised by the tests. For example in the github actions build I see:

```
pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target0-positions0]
pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target1-positions1]
pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target2-positions2]
  /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:3297: FutureWarning: Comparison of Timestamp with datetime.date is deprecated in order to match the standard library behavior.  In a future version these will be considered non-comparable.Use ts == pd.Timestamp(dt) instead.
    indexer = self._engine.get_indexer(target._get_engine_target())
```
cc @jorisvandenbossche 
> There are still some warnings raised by the tests. For example in the github actions build I see:

Huh, why isnt that failing the npdev build?
Updated test_get_indexer_out_of_bounds_date to catch the warning.  Weird thing is that locally every-other-run it fails saying it the expected warning was not emitted.
hmm https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=51340&view=logs&j=f39ddbca-85f6-5283-3bfe-05832bcf5208&t=60a5c9af-5075-529a-e870-f4d886d8e7ce

now builds are failing because of warnings
> now builds are failing because of warnings

harumph, yah that looks like the same thing where that was happening every-other-run locally
https://github.com/pandas-dev/pandas/pull/36131/checks?check_run_id=1631224554 looks legit
suppressed the warning, green
Why can the warning be suppressed in the tests? It's not a valid user case?
> Why can the warning be suppressed in the tests? It's not a valid user case?

its flaky and i haven't figured out why
thanks @jbrockmendel