Repository: pandas-dev/pandas

Issue:
BUG: ensure_index inconsistently coerces np.nan to pd.NaT depending if timezones are present
#### Code Sample, a copy-pastable example if possible
On master:
```python
In [1]: import numpy as np; import pandas as pd; pd.__version__
Out[1]: '0.25.0.dev0+783.g2b9b58dad'

In [2]: ts_list = [pd.Timestamp('2018-01-01'), np.nan] 
   ...: tstz_list = [pd.Timestamp('2018-01-01', tz='UTC'), np.nan]

In [3]: pd.core.indexes.base.ensure_index(ts_list)
Out[3]: DatetimeIndex(['2018-01-01', 'NaT'], dtype='datetime64[ns]', freq=None)

In [4]: pd.core.indexes.base.ensure_index(tstz_list)
Out[4]: Index([2018-01-01 00:00:00+00:00, nan], dtype='object')
```
#### Problem description
`Out[4]` does not coerce `np.nan` to `pd.NaT` and results in an `Index` with `object` dtype instead of a `DatetimeIndex`.

This causes downstream issues with `IntervalIndex`/`IntervalArray` as it can cause a valid `IntervalIndex`/`IntervalArray` to not be roundtripable from it's equivalent `list`/`np.array` representation:
```python
In [5]: left = pd.DatetimeIndex(['2018-01-01', pd.NaT], tz='UTC') 
   ...: right = pd.DatetimeIndex(['2018-01-02', pd.NaT], tz='UTC') 
   ...: ii = pd.IntervalIndex.from_arrays(left, right) 
   ...: ii
Out[5]: 
IntervalIndex([(2018-01-01, 2018-01-02], nan],
              closed='right',
              dtype='interval[datetime64[ns, UTC]]')

In [6]: pd.IntervalIndex(ii.tolist())
---------------------------------------------------------------------------
TypeError: category, object, and string subtypes are not supported for IntervalIndex

In [7]: pd.IntervalIndex(ii.to_numpy())
---------------------------------------------------------------------------
TypeError: category, object, and string subtypes are not supported for IntervalIndex
```
Under the hood the `list`/`np.array` is being converted to `left`/`right` components, which are then passed to `ensure_index`, resulting in an `Index` with `object` dtype, hence the error message.

Note that the equivalent roundtrip without a tz works fine, as expected based on the inconsistency noted in the `ensure_index` example:
```python
In [8]: left = pd.DatetimeIndex(['2018-01-01', pd.NaT]) 
    ...: right = pd.DatetimeIndex(['2018-01-02', pd.NaT]) 
    ...: ii = pd.IntervalIndex.from_arrays(left, right) 
    ...: ii
Out[8]: 
IntervalIndex([(2018-01-01, 2018-01-02], nan],
              closed='right',
              dtype='interval[datetime64[ns]]')

In [9]: pd.IntervalIndex(ii.tolist())
Out[9]: 
IntervalIndex([(2018-01-01, 2018-01-02], nan],
              closed='right',
              dtype='interval[datetime64[ns]]')

In [10]: pd.IntervalIndex(ii.to_numpy())
Out[10]: 
IntervalIndex([(2018-01-01, 2018-01-02], nan],
              closed='right',
              dtype='interval[datetime64[ns]]')
```

#### Expected Output
I'd expect `Out[4]` to be coerced to a `DatetimeIndex` with `pd.NaT` and the appropriate tz:
 ```python
Out[4]: DatetimeIndex(['2018-01-01 00:00:00+00:00', 'NaT'], dtype='datetime64[ns, UTC]', freq=None)
```
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : 2b9b58dadf8a4e02b94747b6c8b22bec4b6eeefd
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.19.14-041914-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0.dev0+783.g2b9b58dad
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 40.8.0
Cython           : 0.29.10
pytest           : 4.6.2
hypothesis       : 4.23.6
sphinx           : 1.8.5
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.3.3
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.5.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : 1.2.1
fastparquet      : 0.3.0
gcsfs            : None
matplotlib       : 3.1.0
numexpr          : 2.6.9
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : 0.11.1
pytables         : None
s3fs             : 0.2.1
scipy            : 1.2.1
sqlalchemy       : 1.3.4
tables           : 3.5.2
xarray           : 0.12.1
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8

</details>



Discussion:
This hits the cython layer that tends not to play nicely with datetime with timezones (and I suspect a other Extension Arrays too).

https://github.com/pandas-dev/pandas/blob/8ea2d087cda0f40a4e41ce108a32859d51b4d69f/pandas/_libs/lib.pyx#L585

But there may possibly be a fix in the inference layer of `Index`

Resolving patch:
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -469,19 +469,15 @@ def __new__(
                     pass
                 elif inferred != "string":
                     if inferred.startswith("datetime"):
-                        if (
-                            lib.is_datetime_with_singletz_array(subarr)
-                            or "tz" in kwargs
-                        ):
-                            # only when subarr has the same tz
-                            from pandas import DatetimeIndex
+                        from pandas import DatetimeIndex
 
-                            try:
-                                return DatetimeIndex(
-                                    subarr, copy=copy, name=name, **kwargs
-                                )
-                            except OutOfBoundsDatetime:
-                                pass
+                        try:
+                            return DatetimeIndex(subarr, copy=copy, name=name, **kwargs)
+                        except (ValueError, OutOfBoundsDatetime):
+                            # GH 27011
+                            # If we have mixed timezones, just send it
+                            # down the base constructor
+                            pass
 
                     elif inferred.startswith("timedelta"):
                         from pandas import TimedeltaIndex


Test patch:
diff --git a/pandas/tests/arrays/interval/test_interval.py b/pandas/tests/arrays/interval/test_interval.py
--- a/pandas/tests/arrays/interval/test_interval.py
+++ b/pandas/tests/arrays/interval/test_interval.py
@@ -42,10 +42,9 @@ class TestAttributes:
             (0, 1),
             (Timedelta("0 days"), Timedelta("1 day")),
             (Timestamp("2018-01-01"), Timestamp("2018-01-02")),
-            pytest.param(
+            (
                 Timestamp("2018-01-01", tz="US/Eastern"),
                 Timestamp("2018-01-02", tz="US/Eastern"),
-                marks=pytest.mark.xfail(strict=True, reason="GH 27011"),
             ),
         ],
     )
diff --git a/pandas/tests/indexes/datetimes/test_construction.py b/pandas/tests/indexes/datetimes/test_construction.py
--- a/pandas/tests/indexes/datetimes/test_construction.py
+++ b/pandas/tests/indexes/datetimes/test_construction.py
@@ -822,6 +822,12 @@ def test_constructor_wrong_precision_raises(self):
         with pytest.raises(ValueError):
             pd.DatetimeIndex(["2000"], dtype="datetime64[us]")
 
+    def test_index_constructor_with_numpy_object_array_and_timestamp_tz_with_nan(self):
+        # GH 27011
+        result = Index(np.array([Timestamp("2019", tz="UTC"), np.nan], dtype=object))
+        expected = DatetimeIndex([Timestamp("2019", tz="UTC"), pd.NaT])
+        tm.assert_index_equal(result, expected)
+
 
 class TestTimeSeries:
     def test_dti_constructor_preserve_dti_freq(self):