Repository: pandas-dev/pandas
Pull request: BUG: fix to_datetime failing to raise on mixed tznaive/tzaware datetimes
- [x] closes #24569
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
DatetimeArray._from_sequence with mixture of tz-naive and tz-aware data.
This should probably raise

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')])

In [3]: pd.arrays.DatetimeArray._from_sequence(arr)
Out[3]:
<DatetimeArrayMixin>
['2000-01-01 01:00:00+01:00', '2000-01-01 00:00:00+01:00']
Length: 2, dtype: datetime64[ns, CET]
```

For reference, a mix of different timezones does raise

```python
In [4]: mix = np.array([pd.Timestamp('2000', tz="US/Central"), pd.Timestamp('2000', tz='CET')])

In [5]: pd.arrays.DatetimeArray._from_sequence(mix)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/sandbox/pandas/pandas/core/arrays/datetimes.py in objects_to_datetime64ns(data, dayfirst, yearfirst, utc, errors, require_iso8601, allow_object)
   1753         try:
-> 1754             values, tz_parsed = conversion.datetime_to_datetime64(data)
   1755             # If tzaware, these values represent unix timestamps, so we

~/sandbox/pandas/pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()
    178                     if not tz_compare(val.tzinfo, inferred_tz):
--> 179                         raise ValueError('Array must be all same time zone')
    180                 else:

ValueError: Array must be all same time zone

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-5-b2c990c41b55> in <module>
----> 1 pd.arrays.DatetimeArray._from_sequence(mix)

~/sandbox/pandas/pandas/core/arrays/datetimes.py in _from_sequence(cls, data, dtype, copy, tz, freq, dayfirst, yearfirst, ambiguous)
    326         subarr, tz, inferred_freq = sequence_to_dt64ns(
    327             data, dtype=dtype, copy=copy, tz=tz,
--> 328             dayfirst=dayfirst, yearfirst=yearfirst, ambiguous=ambiguous)
    329
    330         freq, freq_infer = dtl.validate_inferred_freq(freq, inferred_freq,

~/sandbox/pandas/pandas/core/arrays/datetimes.py in sequence_to_dt64ns(data, dtype, copy, tz, dayfirst, yearfirst, ambiguous)
   1660             #  or M8[ns] to denote wall times
   1661             data, inferred_tz = objects_to_datetime64ns(
-> 1662                 data, dayfirst=dayfirst, yearfirst=yearfirst)
   1663             tz = maybe_infer_tz(tz, inferred_tz)
   1664

~/sandbox/pandas/pandas/core/arrays/datetimes.py in objects_to_datetime64ns(data, dayfirst, yearfirst, utc, errors, require_iso8601, allow_object)
   1757             return values.view('i8'), tz_parsed
   1758         except (ValueError, TypeError):
-> 1759             raise e
   1760
   1761     if tz_parsed is not None:

~/sandbox/pandas/pandas/core/arrays/datetimes.py in objects_to_datetime64ns(data, dayfirst, yearfirst, utc, errors, require_iso8601, allow_object)
   1748             dayfirst=dayfirst,
   1749             yearfirst=yearfirst,
-> 1750             require_iso8601=require_iso8601
   1751         )
   1752     except ValueError as e:

~/sandbox/pandas/pandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()
    458 @cython.wraparound(False)
    459 @cython.boundscheck(False)
--> 460 cpdef array_to_datetime(ndarray[object] values, str errors='raise',
    461                         bint dayfirst=False, bint yearfirst=False,
    462                         object utc=None, bint require_iso8601=False):

~/sandbox/pandas/pandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()
    535                             iresult[i] = _ts.value
    536                         else:
--> 537                             raise ValueError('Tz-aware datetime.datetime '
    538                                              'cannot be converted to '
    539                                              'datetime64 unless utc=True')

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

```

Hints from issue discussion:
I @jbrockmendel already noted this, its in ``to_datetime``, but really because these hit different paths in array_to_datetime

```
In [1]: arr = np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')])
In [2]: pd.to_datetime(arr)
Out[2]: DatetimeIndex(['2000-01-01 01:00:00+01:00', '2000-01-01 00:00:00+01:00'], dtype='datetime64[ns, CET]', freq=None)

In [3]: mix = np.array([pd.Timestamp('2000', tz="US/Central"), pd.Timestamp('2000', tz='CET')])

In [4]: pd.to_datetime(mix)
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True
```
Yah, #24006 would handle these correctly, needs to come off the back-burner.

IIRC there was a "M8ns/i8 are treated inconsistently" issue in there too.
This isn't a blocker for 0.24.0 I think.

Commit message: fix to_datetime failing to raise on mixed tznaive/tzaware datetimes

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index 4bc50695e1ecd..797c3faeb1fd0 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -1549,6 +1549,7 @@ Datetimelike
 - Bug in :class:`PeriodIndex` where comparisons against an array-like object with length 1 failed to raise ``ValueError`` (:issue:`23078`)
 - Bug in :meth:`DatetimeIndex.astype`, :meth:`PeriodIndex.astype` and :meth:`TimedeltaIndex.astype` ignoring the sign of the ``dtype`` for unsigned integer dtypes (:issue:`24405`).
 - Fixed bug in :meth:`Series.max` with ``datetime64[ns]``-dtype failing to return ``NaT`` when nulls are present and ``skipna=False`` is passed (:issue:`24265`)
+- Bug in :func:`to_datetime` where arrays of ``datetime`` objects containing both timezone-aware and timezone-naive ``datetimes`` would fail to raise ``ValueError`` (:issue:`24569`)
 
 Timedelta
 ^^^^^^^^^
diff --git a/pandas/_libs/tslibs/conversion.pyx b/pandas/_libs/tslibs/conversion.pyx
index 6aa02ca1e5421..6c8b732928bc3 100644
--- a/pandas/_libs/tslibs/conversion.pyx
+++ b/pandas/_libs/tslibs/conversion.pyx
@@ -167,6 +167,7 @@ def datetime_to_datetime64(values: object[:]):
         int64_t[:] iresult
         npy_datetimestruct dts
         _TSObject _ts
+        bint found_naive = False
 
     result = np.empty(n, dtype='M8[ns]')
     iresult = result.view('i8')
@@ -176,6 +177,9 @@ def datetime_to_datetime64(values: object[:]):
             iresult[i] = NPY_NAT
         elif PyDateTime_Check(val):
             if val.tzinfo is not None:
+                if found_naive:
+                    raise ValueError('Cannot mix tz-aware with '
+                                     'tz-naive values')
                 if inferred_tz is not None:
                     if not tz_compare(val.tzinfo, inferred_tz):
                         raise ValueError('Array must be all same time zone')
@@ -186,6 +190,7 @@ def datetime_to_datetime64(values: object[:]):
                 iresult[i] = _ts.value
                 check_dts_bounds(&_ts.dts)
             else:
+                found_naive = True
                 if inferred_tz is not None:
                     raise ValueError('Cannot mix tz-aware with '
                                      'tz-naive values')
diff --git a/pandas/tests/arrays/test_array.py b/pandas/tests/arrays/test_array.py
index 1d09a1f65e43f..0698b43b16c05 100644
--- a/pandas/tests/arrays/test_array.py
+++ b/pandas/tests/arrays/test_array.py
@@ -151,11 +151,7 @@ def test_array_inference(data, expected):
     [pd.Timestamp("2000", tz="CET"), pd.Timestamp("2000", tz="UTC")],
     # Mix of tz-aware and tz-naive
     [pd.Timestamp("2000", tz="CET"), pd.Timestamp("2000")],
-    # GH-24569
-    pytest.param(
-        np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')]),
-        marks=pytest.mark.xfail(reason="bug in DTA._from_sequence")
-    ),
+    np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')]),
 ])
 def test_array_inference_fails(data):
     result = pd.array(data)
diff --git a/pandas/tests/arrays/test_datetimes.py b/pandas/tests/arrays/test_datetimes.py
index 8d0e4f5a90557..ae267ad8f9ea4 100644
--- a/pandas/tests/arrays/test_datetimes.py
+++ b/pandas/tests/arrays/test_datetimes.py
@@ -16,6 +16,25 @@
 
 
 class TestDatetimeArrayConstructor(object):
+    def test_mixing_naive_tzaware_raises(self):
+        # GH#24569
+        arr = np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')])
+
+        msg = ('Cannot mix tz-aware with tz-naive values|'
+               'Tz-aware datetime.datetime cannot be converted '
+               'to datetime64 unless utc=True')
+
+        for obj in [arr, arr[::-1]]:
+            # check that we raise regardless of whether naive is found
+            #  before aware or vice-versa
+            for meth in [DatetimeArray._from_sequence,
+                         sequence_to_dt64ns,
+                         pd.to_datetime,
+                         pd.DatetimeIndex]:
+
+                with pytest.raises(ValueError, match=msg):
+                    meth(obj)
+
     def test_from_pandas_array(self):
         arr = pd.array(np.arange(5, dtype=np.int64)) * 3600 * 10**9
 
diff --git a/pandas/tests/indexes/datetimes/test_construction.py b/pandas/tests/indexes/datetimes/test_construction.py
index 97de4cd98dedf..07c42afe44b33 100644
--- a/pandas/tests/indexes/datetimes/test_construction.py
+++ b/pandas/tests/indexes/datetimes/test_construction.py
@@ -306,16 +306,6 @@ def test_construction_dti_with_mixed_timezones(self):
         tm.assert_index_equal(result, exp, exact=True)
         assert isinstance(result, DatetimeIndex)
 
-        # different tz coerces tz-naive to tz-awareIndex(dtype=object)
-        result = DatetimeIndex([Timestamp('2011-01-01 10:00'),
-                                Timestamp('2011-01-02 10:00',
-                                          tz='US/Eastern')], name='idx')
-        exp = DatetimeIndex([Timestamp('2011-01-01 05:00'),
-                             Timestamp('2011-01-02 10:00')],
-                            tz='US/Eastern', name='idx')
-        tm.assert_index_equal(result, exp, exact=True)
-        assert isinstance(result, DatetimeIndex)
-
         # tz mismatch affecting to tz-aware raises TypeError/ValueError
 
         with pytest.raises(ValueError):
@@ -323,7 +313,8 @@ def test_construction_dti_with_mixed_timezones(self):
                            Timestamp('2011-01-02 10:00', tz='US/Eastern')],
                           name='idx')
 
-        with pytest.raises(TypeError, match='data is already tz-aware'):
+        msg = 'cannot be converted to datetime64'
+        with pytest.raises(ValueError, match=msg):
             DatetimeIndex([Timestamp('2011-01-01 10:00'),
                            Timestamp('2011-01-02 10:00', tz='US/Eastern')],
                           tz='Asia/Tokyo', name='idx')
@@ -333,7 +324,7 @@ def test_construction_dti_with_mixed_timezones(self):
                            Timestamp('2011-01-02 10:00', tz='US/Eastern')],
                           tz='US/Eastern', name='idx')
 
-        with pytest.raises(TypeError, match='data is already tz-aware'):
+        with pytest.raises(ValueError, match=msg):
             # passing tz should results in DatetimeIndex, then mismatch raises
             # TypeError
             Index([pd.NaT, Timestamp('2011-01-01 10:00'),


Code review comments:
diff_hunk: @@ -16,6 +16,25 @@
 
 
 class TestDatetimeArrayConstructor(object):
+    def test_mixing_naive_tzaware_raises(self):
+        # GH#24569
+        arr = np.array([pd.Timestamp('2000'), pd.Timestamp('2000', tz='CET')])
+
+        msg = ('Cannot mix tz-aware with tz-naive values|'
+               'Tz-aware datetime.datetime cannot be converted '
+               'to datetime64 unless utc=True')
+
+        for obj in [arr, arr[::-1]]:
+            # check that we raise regardless of whether naive is found
+            #  before aware or vice-versa
+            for meth in [DatetimeArray._from_sequence,
+                         sequence_to_dt64ns,
+                         pd.to_datetime,
+                         pd.DatetimeIndex]:
path: pandas/tests/arrays/test_datetimes.py
text: @user1:
Can you parametrize over `meth` instead of iterating?

@author:
I can if you have a strong preference.  I'm trying to push back against over-parametrizing, since pytest overhead is becoming noticeable.

@user2:
i think it’s worth it here for ease of debuggjng

@author:
done