WIP: multi-timezone handling for array_to_datetime
State: closed

cc @mroeschke 

ATM `array_to_datetime` handles strings and datetime objects very differently, with `conversion.datetime_to_datetime64` picking up (some of) the slack.  This unifies the treatment of strings/datetimes within array_to_datetime, rendering `conversion_to_datetime64` (and some ugly try/excepts in `pd.to_datetime`) unnecessary.

As of now there are still 5 tests failing locally; resolving them will involve some design decisions.

1) This PR introduces cases where dateutil's UTC or dateutil tzoffsets are returned while the test expects the equivalent pytz object.  We can either a) try to convert them within `array_to_datetime` to more consistently return pytz objects, b) change the tests to expect the dateutil versions, or c) change the tests to not care.

2) In the status quo we do (and test) something weird:

```
vals = [pd.Timestamp('2011-01-01 10:00'), pd.Timestamp('2011-01-02 10:00', tz='US/Eastern')]
# i.e. one tz-naive, one tz-aware

>>> pd.to_datetime(vals)
DatetimeIndex(['2011-01-01 05:00:00-05:00', '2011-01-02 10:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

# for strings we do something more reasonable

>>> pd.to_datetime([str(x) for x in vals])
Index([2011-01-01 10:00:00, 2011-01-02 10:00:00-05:00], dtype='object')
```
This PR changes the first call to behave like the second.

2b) We also need to decide whether datetime64 are considered naive or UTC within `array_to_datetime`

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
Hello @jbrockmendel! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/dtypes/cast.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/core/dtypes/cast.py) !

 - There are no PEP8 issues in the file [`pandas/core/series.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/core/series.py) !

 - In the file [`pandas/core/tools/datetimes.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/core/tools/datetimes.py), following                are the PEP8 issues :
> [Line 554:80](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/core/tools/datetimes.py#L554): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (102 > 79 characters)


 - In the file [`pandas/tests/indexes/datetimes/test_construction.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/tests/indexes/datetimes/test_construction.py), following                are the PEP8 issues :
> [Line 286:28](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/tests/indexes/datetimes/test_construction.py#L286): [E127](https://duckduckgo.com/?q=pep8%20E127) continuation line over-indented for visual indent


 - There are no PEP8 issues in the file [`pandas/tests/indexes/datetimes/test_tools.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/tests/indexes/datetimes/test_tools.py) !

 - There are no PEP8 issues in the file [`pandas/tests/tslibs/test_array_to_datetime.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/tests/tslibs/test_array_to_datetime.py) !

 - In the file [`pandas/util/testing.py`](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/util/testing.py), following                are the PEP8 issues :
> [Line 969:80](https://github.com/jbrockmendel/pandas/blob/ec3500291237347ebbffa7745c54b3c1c58087cf/pandas/util/testing.py#L969): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)




##### Comment last updated on December 09, 2018 at 00:46 Hours UTC
Response to your questions:

1) During my `is_utc` overall, I added a test for a case where a dateutil UTC was getting coerced to a pytz UTC. IMO I think it's a bug if we coerce dateutil objects to pytz objects or vice versa so those test should be fixed. https://github.com/pandas-dev/pandas/commit/94ce05d1bcc3c99e992c48cc99d0fd2726f43102#diff-ad26614f192ef4be1906047f739c3644R592

1-sidenote) The interesting case becomes what happens if you mix dateutil and pytz objects (same timezone or not). My gut reaction is to cast to object, but I could see how raising could also be appropriate.

2) I agree that the second behavior is better than the first.

2b) Well the datetime64 will _technically_ always be naive since its the numpy type? But I think more to your point, I think it should represent UTC inside  `array_to_datetime` and should return UTC if timezone info was parsed. 
 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=h1) Report
> Merging [#24006](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/defa8a8ed6f627310500388a477186bed995fef1?src=pr&el=desc) will **decrease** coverage by `0.57%`.
> The diff coverage is `55.81%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24006/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24006      +/-   ##
==========================================
- Coverage   43.02%   42.44%   -0.58%     
==========================================
  Files         162      161       -1     
  Lines       51700    51563     -137     
==========================================
- Hits        22245    21887     -358     
- Misses      29455    29676     +221
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.44% <55.81%> (-0.58%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `50.74% <0%> (+1.41%)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `48.41% <100%> (-0.18%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `34.68% <41.66%> (+0.63%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `51.88% <66.66%> (+0.26%)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `16.66% <0%> (-46.2%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `38.75% <0%> (-4.66%)` | :arrow_down: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `40.42% <0%> (-4.43%)` | :arrow_down: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `69.9% <0%> (-2.86%)` | :arrow_down: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `50.35% <0%> (-2.83%)` | :arrow_down: |
| [pandas/core/sparse/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3Nlcmllcy5weQ==) | `43.75% <0%> (-2.68%)` | :arrow_down: |
| ... and [45 more](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=footer). Last update [defa8a8...ec35002](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=h1) Report
> Merging [#24006](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5b0610b875476a6f3727d7e9bedb90d370c669b5?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `55.81%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24006/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24006      +/-   ##
==========================================
+ Coverage   42.46%   42.46%   +<.01%     
==========================================
  Files         161      161              
  Lines       51557    51558       +1     
==========================================
+ Hits        21892    21893       +1     
  Misses      29665    29665
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.46% <55.81%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `50.74% <0%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `48.41% <100%> (-0.02%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `34.68% <41.66%> (-0.39%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24006/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `51.88% <66.66%> (+0.07%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=footer). Last update [5b0610b...a2a01c9](https://codecov.io/gh/pandas-dev/pandas/pull/24006?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jbrockmendel still relevant?
> still relevant?

Yes.  I'm in a less-active phase at the moment, but this is still a problem that needs to be solved.  I'll return to it before too long.
closing as stale
Closing per above comment. Can always be reopened