Revert handling of i8values to DatetimeIndex
State: closed

TODO:

- [x] decide if we're deprecating anything.
- [x] update the message (and update filterwarnings)
- [x] whatsnew

Questions

1. What's the expected behavior of `pd.Index(i8).dtype(DatetimeTZDtype)`?
2. Do we prefer this `int_as_wall_time` parameter, or @jbrockmendel's `_from_sequence_dti` from https://github.com/jbrockmendel/pandas/commit/635b267cbb7ad63edb86157b35f027898d99542a
3. 


cc @jbrockmendel @jorisvandenbossche @jreback @mroeschke 

xref https://github.com/pandas-dev/pandas/issues/24559

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
I'll take a look at Tom's edits in a little bit.  #24686 should be good to go and handles the only part of `DTA.__init__` that I consider mandatory
Travis looks like just a complaint about stacklevel
Does this correspond to one of the options in the table in #24559?
The options in those tables represent the desired *future* behavior.

This is changing the current behavior of DTI to match 0.23.4's behavior (i8 values are treated as wall times). We need to figure out

1. If we want to deprecate the current behavior in DTI, and what the future desired behavior will be
2. Update all of DatetimeIndex, DatetimeArray, and Timestamp to implement our desired behavior.

For the RC I only care about

1. Reverting the handing of i8 back to 0.23.4 behavior.
2. Implementing the deprecation warning for the future behavior in DTI only (if any change from 0.23.4).
> Does this correspond to one of the options in the table in #24559?

Not directly, as this is mainly about reverting the behaviour to 0.23.4. 
But, whether to add a deprecation warning or not, and what to do for `astype`, does actually depend on the outcome of the discussion in  #24559 though. 

So if we add a deprecation warning, that means that we choose one of the options where DatetimeIndex(int) is unix-epoch (that are still multiple options though).

(I think that is one of the aspects we are most sure about / most agreement, but if needed, we can also only here do the revert, and defer adding a deprecation warning until we made a final discussion on which option to choose)
Double post :-)

Yes, agreed on the priorities for the RC. But so the deprecation warning still depends somewhat on which desired future behaviour we want.
> Yes, agreed on the priorities for the RC. But so the deprecation warning still depends somewhat on which desired future behaviour we want.

Agreed. We'll need to sort that out. I'm off for ~5 hours. Hopefully you all figure it out in the meantime :)
Actually, I think the relevant decisions have been made for this PR, right? Does everyone agree with the following axiom?

* `cls(i8, tz=tz) <=> cls(i8).tz_loacalize("UTC").tz_convert(tz)`

In words, passing integer values and a timezone is equivalent to passing interger values, localizing to UTC, and converting to the desired tz? In other words, integer values are treated like UTC (unix epochs). All the options in 
https://github.com/pandas-dev/pandas/issues/24559#issuecomment-452779835, aside from Option 1 (0.23.4 IIRC), agree that ints should be unix epoch.

On `Index.astype`, in 0.23.4 that errored.

```python
In [2]: pd.Index([946684800000000000]).astype('datetime64[ns, US/Central]')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/indexes/base.py in astype(self, dtype, copy)
   1291         try:
-> 1292             return Index(self.values.astype(dtype, copy=copy), name=self.name,
   1293                          dtype=dtype)

TypeError: Invalid datetime unit in metadata string "[ns, US/Central]"

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-2-e1a69a3f7ce5> in <module>
----> 1 pd.Index([946684800000000000]).astype('datetime64[ns, US/Central]')

~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/indexes/base.py in astype(self, dtype, copy)
   1294         except (TypeError, ValueError):
   1295             msg = 'Cannot cast {name} to dtype {dtype}'
-> 1296             raise TypeError(msg.format(name=type(self).__name__, dtype=dtype))
   1297
   1298     def _to_safe_for_reshape(self):

TypeError: Cannot cast Int64Index to dtype datetime64[ns, US/Central]
```

Does anyone object to doing essentially

```python
In [8]: pd.Index([946684800000000000]).astype('datetime64[ns]').tz_localize("UTC").tz_convert("US/Central")
Out[8]: DatetimeIndex(['1999-12-31 18:00:00-06:00'], dtype='datetime64[ns, US/Central]', freq=None)

```

which is equivalent to the future behavior?

No objections on my end.

So will the `Index.astype` behavior be fixed/introduced in this PR or after the warning is introduced?
> So will the Index.astype behavior be fixed/introduced in this PR or after the warning is introduced?

`Index.astype(DatetimeTZDtype)` works on master. And AFAICT, does the right thing.
master, before this PR:

```python
In [2]: pd.Index([946684800000000000], dtype='datetime64[ns, US/Central]')
Out[2]: DatetimeIndex(['1999-12-31 18:00:00-06:00'], dtype='datetime64[ns, US/Central]', freq=None)

In [3]: pd.Index([946684800000000000]).astype('datetime64[ns]').tz_localize("UTC").tz_convert("US/Central")
Out[3]: DatetimeIndex(['1999-12-31 18:00:00-06:00'], dtype='datetime64[ns, US/Central]', freq=None)

```

I just need to change the implementation to avoid the warning.

This will introduce a *temporary* discrepancy between `Index(i8).astype(DatetimeTZDtype)` and `Index(i8, dtype=DatetimeDTZDtype)`, as the `.astype` version will already be doing the future behavior. I think that's preferred.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=h1) Report
> Merging [#24708](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/021cbae52061b660b2251d6c4dbbfa6d2a246325?src=pr&el=desc) will **decrease** coverage by `49.31%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24708/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24708       +/-   ##
===========================================
- Coverage   92.38%   43.06%   -49.32%     
===========================================
  Files         166      166               
  Lines       52310    52318        +8     
===========================================
- Hits        48327    22532    -25795     
- Misses       3983    29786    +25803
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `43.06% <66.66%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `48.46% <ø> (-47.82%)` | :arrow_down: |
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `11.23% <0%> (-83.66%)` | :arrow_down: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `66.07% <100%> (-31.65%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| ... and [126 more](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=footer). Last update [021cbae...1456699](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=h1) Report
> Merging [#24708](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e2b20686464b5990594b079e15449766f85dd85a?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24708/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24708      +/-   ##
==========================================
+ Coverage   92.38%   92.38%   +<.01%     
==========================================
  Files         166      166              
  Lines       52321    52335      +14     
==========================================
+ Hits        48338    48352      +14     
  Misses       3983     3983
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <100%> (ø)` | :arrow_up: |
| #single | `43.06% <40%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.27% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `94.94% <100%> (+0.05%)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `97.78% <100%> (+0.02%)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.3% <100%> (ø)` | :arrow_up: |
| [pandas/io/packers.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhY2tlcnMucHk=) | `88.21% <0%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.16% <0%> (ø)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24708/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.31% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=footer). Last update [e2b2068...8332c40](https://codecov.io/gh/pandas-dev/pandas/pull/24708?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

See 1456699d234e2d1cd74d29e12f2a366189184280 for the astype changes.

I'll push a whatsnew later tonight, unless anyone beats me to it.

We have an outstanding TODO: Should `DatetimeIndex(i8, tz="UTC")` warn?
Also need the filterwarnings to match our message, so we aren't accidentally silencing errors.
> Should DatetimeIndex(i8, tz="UTC") warn?

I think there's a slight preference not to warn in this case.
Updating UTC handling and writing a release note now.
May still have some py2 failures. Haven't looked closely yet.
Can't reproduce the py2 failure on my Mac. Trying in a docker container. 
This should be ready. I hope 2e00630 fixes the py2 failures. It's possible they were warnings leaking through tests.
Still struggling with the py2 failures.

Edit: Hoping https://github.com/pandas-dev/pandas/issues/24716 is it...
I don't think 90afdee22835374128e90e3a741817f3a359b6f3 will do this trick, but trying anyway.
Just cloned, will take a look
I'm guessing it's just strangeness with python 2's warning handling. The tests are fine on their own AFAICT.
Got it locally. Just narrowing down now. `pytest pandas/tests/indexes pandas/tests/indexes/test_base.py::TestIndex::test_constructor_dtypes_datetime -x --pdb`
`pytest -x --pdb -v 'pandas/tests/indexes/datetimes/test_construction.py::TestDatetimeIndex::test_constructor_with_int_tz[US/Pacific-datetime64[ns, US/Pacific]-box1-Index]' pandas/tests/indexes/test_base.py::TestIndex::test_constructor_dtypes_datetime
`
6633344eced0395e7662311919a3b5b3857c2fa1 may do it, not sure *why* we would need to skip that test on py2...
Not fixed yet :/
`pytest -x --pdb pandas/tests/test_base.py::TestToIterable::test_iterable pandas/tests/indexes/test_base.py::TestIndex::test_constructor_dtypes_datetime[US/Eastern-Index-asi8-True]
` also fails
I can reproduce all 7 failures on Linux with `python -m pytest pandas/tests/test_[abcde]*.py pandas/tests/indexes/test_base.py --skip-slow`.  Currently trying to narrow down the culprits...