Repository: pandas-dev/pandas
Pull request: Fix DatetimeIndex.insert(pd.NaT) for tz-aware index
- [x] closes #16357
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: insert NaT into tz-aware DTI.
this should work (also should test for other index reshaping ops) & for TDI and PI
```
In [7]: pd.DatetimeIndex(['2017-01-01']).insert(0, pd.NaT)
Out[7]: DatetimeIndex(['NaT', '2017-01-01'], dtype='datetime64[ns]', freq=None)

In [8]: pd.DatetimeIndex(['2017-01-01'], tz='UTC').insert(0, pd.NaT)
ValueError: Passed item and index have different timezone
```

Hints from issue discussion:
Taking a look at this

Commit message: Whatsnew note

Patch to review:
diff --git a/doc/source/whatsnew/v0.22.0.txt b/doc/source/whatsnew/v0.22.0.txt
index 0579a80aad28e..8933c2de0f169 100644
--- a/doc/source/whatsnew/v0.22.0.txt
+++ b/doc/source/whatsnew/v0.22.0.txt
@@ -294,6 +294,7 @@ Indexing
 - Bug in tz-aware :class:`DatetimeIndex` where addition/subtraction with a :class:`TimedeltaIndex` or array with ``dtype='timedelta64[ns]'`` was incorrect (:issue:`17558`)
 - :func:`Index.to_series` now accepts ``index`` and ``name`` kwargs (:issue:`18699`)
 - :func:`DatetimeIndex.to_series` now accepts ``index`` and ``name`` kwargs (:issue:`18699`)
+- Bug in :class:`DatetimeIndex.insert` where inserting ``pd.NaT`` into a timezone-aware index incorrectly raised (:issue:`16357`)
 
 I/O
 ^^^
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index ec5c20d341b50..a6061cf0f1a48 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -1776,7 +1776,8 @@ def insert(self, loc, item):
 
         if isinstance(item, (datetime, np.datetime64)):
             self._assert_can_do_op(item)
-            if not self._has_same_tz(item):
+            if not self._has_same_tz(item) and item is not self._na_value:
+                # GH#16537 allow pd.NaT through
                 raise ValueError(
                     'Passed item and index have different timezone')
             # check freq can be preserved on edge cases
diff --git a/pandas/tests/indexes/datetimes/test_indexing.py b/pandas/tests/indexes/datetimes/test_indexing.py
index b3ce22962d5d4..485abf85003fc 100644
--- a/pandas/tests/indexes/datetimes/test_indexing.py
+++ b/pandas/tests/indexes/datetimes/test_indexing.py
@@ -46,6 +46,14 @@ def test_where_tz(self):
         expected = i2
         tm.assert_index_equal(result, expected)
 
+    @pytest.mark.parametrize('tz', [None, 'UTC', 'US/Eastern'])
+    def test_insert_nat(self, tz):
+        # GH#16537
+        idx = pd.DatetimeIndex(['2017-01-01'], tz=tz)
+        res = idx.insert(0, pd.NaT)
+        expected = pd.DatetimeIndex(['NaT', '2017-01-01'], tz=tz)
+        tm.assert_index_equal(res, expected)
+
     def test_insert(self):
         idx = DatetimeIndex(
             ['2000-01-04', '2000-01-01', '2000-01-02'], name='idx')


Code review comments:
diff_hunk: @@ -46,6 +46,14 @@ def test_where_tz(self):
         expected = i2
         tm.assert_index_equal(result, expected)
 
+    @pytest.mark.parametrize('tz', [None, 'UTC', 'US/Eastern'])
+    def test_insert_nat(self, tz):
path: pandas/tests/indexes/datetimes/test_indexing.py
text: @user1:
You could maybe parametrize over different NA values here, since this should work for inserting `np.nan` and `None` as well (shouldn't require any additional code changes).  For example, on a tz-naive DTI:
```python
In [2]: pd.date_range('20170101', periods=2).insert(0, np.nan)
Out[2]: DatetimeIndex(['NaT', '2017-01-01', '2017-01-02'], dtype='datetime64[ns]', freq=None)

In [3]: pd.date_range('20170101', periods=2).insert(0, None)
Out[3]: DatetimeIndex(['NaT', '2017-01-01', '2017-01-02'], dtype='datetime64[ns]', freq=None)
```

@user1:
If you do that, I suppose you could delete these lines in `test_insert` below, as to not duplicate things:

https://github.com/pandas-dev/pandas/blob/ac8ac158377d60c8c8e10e6905d49d48bbe80855/pandas/tests/indexes/datetimes/test_indexing.py#L148-L153

diff_hunk: @@ -294,6 +294,7 @@ Indexing
 - Bug in tz-aware :class:`DatetimeIndex` where addition/subtraction with a :class:`TimedeltaIndex` or array with ``dtype='timedelta64[ns]'`` was incorrect (:issue:`17558`)
 - :func:`Index.to_series` now accepts ``index`` and ``name`` kwargs (:issue:`18699`)
 - :func:`DatetimeIndex.to_series` now accepts ``index`` and ``name`` kwargs (:issue:`18699`)
+- Bug in :class:`DatetimeIndex.insert` where inserting ``pd.NaT`` into a timezone-aware index incorrectly raised (:issue:`16357`)
path: doc/source/whatsnew/v0.22.0.txt
text: @user1:
I think `:class:` should be `:func:` or `:meth:` instead, and it looks like previous convention has been to just use <code>\`\`NaT\`\`</code> with the `pd.` omitted.