Repository: pandas-dev/pandas
Pull request: BUG: Fix mixed datetime dtype inference
- [x] closes #33741
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: DatetimeIndex.get_indexer wrong result on mixed-type argument
- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandas.
- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

#### Code Sample, a copy-pastable example

```python
import datetime
import pandas as pd

dates = pd.bdate_range('2020-04-20', '2020-04-25')
print(dates.get_indexer([datetime.date(2020, 4, 20), datetime.date(2020, 4, 21)]))
# [0, 1]
print(dates.get_indexer([pd.Timestamp('2020-04-20'), datetime.date(2020, 4, 21)]))
# [0, -1]
```

#### Problem description

As a general rule I would expect that `index.get_indexer(as + bs) == np.concat([index.get_indexer(as), index.get_indexer(bs)])` for any lists `as` and `bs`, i.e. `index.get_indexer` operates elementwise. However in this case the type of a previous argument is surprisingly affecting the index that is assigned to a later one.

#### Expected Output

`[0, 1]` in both cases

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.2.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_United Kingdom.936

pandas           : 1.0.3
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.1.3.post20200330
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
@batterseapower Thanks, the issue is on master as well and appears to be caused by a bug in the type inference for Index itself:

```python
[ins] In [21]: pd.Index([datetime.date(2020, 4, 20), pd.Timestamp("2020-04-21")]).inferred_type                                                                                                              
Out[21]: 'date'

[ins] In [22]: pd.Index([pd.Timestamp("2020-04-21"), datetime.date(2020, 4, 20)]).inferred_type                                                                                                              
Out[22]: 'mixed'
```

I think fixing that would make get_indexer work as expected (if you try just changing the order of the list in the original example it should give the right output).
This was giving the expected result in 0.25.3 so will mark as regression.

```
>>> import numpy as np
>>> import pandas as pd
>>>
>>> print(pd.__version__)
0.25.3
>>>
>>> import datetime
>>>
>>> dates = pd.bdate_range("2020-04-20", "2020-04-25")
>>> dates
DatetimeIndex(['2020-04-20', '2020-04-21', '2020-04-22', '2020-04-23',
               '2020-04-24'],
              dtype='datetime64[ns]', freq='B')
>>>
>>> print(dates.get_indexer([datetime.date(2020, 4, 20), datetime.date(2020, 4, 21)]))
[0 1]
>>>
>>> print(dates.get_indexer([pd.Timestamp("2020-04-20"), datetime.date(2020, 4, 21)]))
[0 1]
>>>
>>>
```
> This was giving the expected result in 0.25.3 so will mark as regression.

not always. and running code sample on master many times sometimes produces the correct answer.

had a quick go at creating a test and using pytest-repeat, but either always passes or always fails.

unable to bisect, so will remove the regression tag.

Commit message: Merge remote-tracking branch 'upstream/master' into datetime-type-inference

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index e8fdaf0ae5d49..085da91190ce4 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -462,6 +462,7 @@ Datetimelike
 - Bug in :meth:`DatetimeIndex.to_period` not infering the frequency when called with no arguments (:issue:`33358`)
 - Bug in :meth:`DatetimeIndex.tz_localize` incorrectly retaining ``freq`` in some cases where the original freq is no longer valid (:issue:`30511`)
 - Bug in :meth:`DatetimeIndex.intersection` losing ``freq`` and timezone in some cases (:issue:`33604`)
+- Bug in :meth:`DatetimeIndex.get_indexer` where incorrect output would be returned for mixed datetime-like targets (:issue:`33741`)
 - Bug in :class:`DatetimeIndex` addition and subtraction with some types of :class:`DateOffset` objects incorrectly retaining an invalid ``freq`` attribute (:issue:`33779`)
 
 Timedelta
diff --git a/pandas/_libs/lib.pyx b/pandas/_libs/lib.pyx
index a312fdc6cda22..aee79864423ab 100644
--- a/pandas/_libs/lib.pyx
+++ b/pandas/_libs/lib.pyx
@@ -1376,6 +1376,8 @@ def infer_dtype(value: object, skipna: bool = True) -> str:
     elif PyDateTime_Check(val):
         if is_datetime_array(values):
             return "datetime"
+        if is_date_array(values):
+            return "date"
 
     elif PyDate_Check(val):
         if is_date_array(values, skipna=skipna):
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index 649d4e6dfc384..436f40f99ea24 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -7,6 +7,7 @@
 
 from pandas._libs import NaT, Period, Timestamp, index as libindex, lib, tslib as libts
 from pandas._libs.tslibs import fields, parsing, timezones
+from pandas._libs.tslibs.np_datetime import OutOfBoundsDatetime
 from pandas._typing import DtypeObj, Label
 from pandas.util._decorators import cache_readonly
 
@@ -540,7 +541,10 @@ def _validate_partial_date_slice(self, reso: str):
 
     def _maybe_promote(self, other):
         if other.inferred_type == "date":
-            other = DatetimeIndex(other)
+            try:
+                other = DatetimeIndex(other)
+            except OutOfBoundsDatetime:
+                pass
         return self, other
 
     def get_loc(self, key, method=None, tolerance=None):
diff --git a/pandas/tests/dtypes/test_inference.py b/pandas/tests/dtypes/test_inference.py
index 8b20f9ada8ff7..1c3b36f4c4fd3 100644
--- a/pandas/tests/dtypes/test_inference.py
+++ b/pandas/tests/dtypes/test_inference.py
@@ -1095,6 +1095,18 @@ def test_date(self):
         result = lib.infer_dtype(dates, skipna=True)
         assert result == "date"
 
+    @pytest.mark.parametrize(
+        "values",
+        [
+            [date(2020, 1, 1), pd.Timestamp("2020-01-01")],
+            [pd.Timestamp("2020-01-01"), date(2020, 1, 1)],
+        ],
+    )
+    def test_infer_dtype_date_order_invariant(self, values):
+        # https://github.com/pandas-dev/pandas/issues/33741
+        result = lib.infer_dtype(values, skipna=True)
+        assert result == "date"
+
     def test_is_numeric_array(self):
 
         assert lib.is_float_array(np.array([1, 2.0]))
diff --git a/pandas/tests/indexes/datetimes/test_indexing.py b/pandas/tests/indexes/datetimes/test_indexing.py
index 08b8e710237c5..95d2452e21adc 100644
--- a/pandas/tests/indexes/datetimes/test_indexing.py
+++ b/pandas/tests/indexes/datetimes/test_indexing.py
@@ -1,4 +1,4 @@
-from datetime import datetime, time, timedelta
+from datetime import date, datetime, time, timedelta
 
 import numpy as np
 import pytest
@@ -547,6 +547,38 @@ def test_get_indexer(self):
         with pytest.raises(ValueError, match="abbreviation w/o a number"):
             idx.get_indexer(idx[[0]], method="nearest", tolerance="foo")
 
+    @pytest.mark.parametrize(
+        "target",
+        [
+            [date(2020, 1, 1), pd.Timestamp("2020-01-02")],
+            [pd.Timestamp("2020-01-01"), date(2020, 1, 2)],
+        ],
+    )
+    def test_get_indexer_mixed_dtypes(self, target):
+        # https://github.com/pandas-dev/pandas/issues/33741
+        values = pd.DatetimeIndex(
+            [pd.Timestamp("2020-01-01"), pd.Timestamp("2020-01-02")]
+        )
+        result = values.get_indexer(target)
+        expected = np.array([0, 1], dtype=np.intp)
+        tm.assert_numpy_array_equal(result, expected)
+
+    @pytest.mark.parametrize(
+        "target, positions",
+        [
+            ([date(9999, 1, 1), pd.Timestamp("2020-01-01")], [-1, 0]),
+            ([pd.Timestamp("2020-01-01"), date(9999, 1, 1)], [0, -1]),
+            ([date(9999, 1, 1), date(9999, 1, 1)], [-1, -1]),
+        ],
+    )
+    def test_get_indexer_out_of_bounds_date(self, target, positions):
+        values = pd.DatetimeIndex(
+            [pd.Timestamp("2020-01-01"), pd.Timestamp("2020-01-02")]
+        )
+        result = values.get_indexer(target)
+        expected = np.array(positions, dtype=np.intp)
+        tm.assert_numpy_array_equal(result, expected)
+
 
 class TestMaybeCastSliceBound:
     def test_maybe_cast_slice_bounds_empty(self):


Code review comments:
diff_hunk: @@ -540,7 +541,10 @@ def _validate_partial_date_slice(self, reso: str):
 
     def _maybe_promote(self, other):
         if other.inferred_type == "date":
-            other = DatetimeIndex(other)
+            try:
+                other = DatetimeIndex(other)
+            except OutOfBoundsDatetime:
path: pandas/core/indexes/datetimes.py
text: @user1:
what is hitting this? why are we not bubbling this up?

@author:
Cases where we have a datetime.date that's out of bounds for DatetimeIndex (e.g. https://github.com/pandas-dev/pandas/pull/33749/files#diff-0c5fadf83f91d24a931a051d82801917R569) are raising here instead of returning -1  which I believe is a bug on master (that's what @user2 and I were talking about above). I don't think we'd want this to actually raise since it doesn't seem to be an essential step.

diff_hunk: @@ -1095,6 +1095,18 @@ def test_date(self):
         result = lib.infer_dtype(dates, skipna=True)
         assert result == "date"
 
+    @pytest.mark.parametrize(
+        "values",
+        [
+            [date(2020, 1, 1), pd.Timestamp("2020-01-01")],
+            [pd.Timestamp("2020-01-01"), date(2020, 1, 1)],
+        ],
+    )
+    def test_infer_dtype_date_order_invariant(self, values):
+        # https://github.com/pandas-dev/pandas/issues/33741
+        result = lib.infer_dtype(values, skipna=True)
path: pandas/tests/dtypes/test_inference.py
text: @user1:
add NaT and parameterize on skipna as well

diff_hunk: @@ -1376,6 +1376,8 @@ def infer_dtype(value: object, skipna: bool = True) -> str:
     elif PyDateTime_Check(val):
         if is_datetime_array(values):
             return "datetime"
+        if is_date_array(values):
path: pandas/_libs/lib.pyx
text: @user1:
you need to pass skipna=skipna (also is_datetime_array needs to do that as well