BUG: fix read_csv to parse timezone correctly
State: closed

Use `box=True` for `to_datetime()`, and adjust downstream processing to the change.

- [x] closes #22256
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: Timezones
description: Timezone data dtype

Comments:
cc @gfyoung @mroeschke 
[pandas.io.parsers._infer_types](https://github.com/pandas-dev/pandas/blob/b5d81cfe43eeccfc3641aa9578097f726da9ce9d/pandas/io/parsers.py#L1607) breaks. Specifically, [pandas._libs.parsers.sanitize_objects](https://github.com/pandas-dev/pandas/blob/b5d81cfe43eeccfc3641aa9578097f726da9ce9d/pandas/_libs/parsers.pyx#L2230) only accepts ndarray as the input. Otherwise, it raises an exception.
right so just put a
np.asarray right before sanitize (this doesn’t copy it already an ndarray)

and see what happens
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=h1) Report
> Merging [#22380](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4f11d1a9a9b02a37dbe109d8413cc75d73b92853?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22380/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #22380   +/-   ##
=======================================
  Coverage   92.05%   92.05%           
=======================================
  Files         169      169           
  Lines       50733    50733           
=======================================
  Hits        46702    46702           
  Misses       4031     4031
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <100%> (ø)` | :arrow_up: |
| #single | `42.24% <75%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/22380/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.48% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=footer). Last update [4f11d1a...339f836](https://codecov.io/gh/pandas-dev/pandas/pull/22380?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

We if cast data into np.ndarray, it will lost the timezone information. ndarray can't contain timezone information. [link](https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.datetime.html#changes-with-numpy-1-11) That's why I had to bypass it. 
So this is a case I didn't account for when I recently fixed `to_datetime` parsings of offsets.

```
In [1]: data = ['2018-01-04 09:01:00+09:00', '2018-01-04 09:02:00+09:00']

# This output should probably behave like In[5]
In [3]: pd.to_datetime(data, box=False)
Out[3]:
array(['2018-01-04T00:01:00.000000000', '2018-01-04T00:02:00.000000000'],
      dtype='datetime64[ns]')

In [4]: data = ['2018-01-04 09:01:00+09:00', '2018-01-04 09:02:00+08:00']

In [5]: pd.to_datetime(data, box=False)
Out[5]:
array([datetime.datetime(2018, 1, 4, 9, 1, tzinfo=tzoffset(None, 32400)),
       datetime.datetime(2018, 1, 4, 9, 2, tzinfo=tzoffset(None, 28800))],
      dtype=object)
```

I don't have the patch off the top of my head, but what would happen if instead if the `to_datetime` call returned an array of datetime objects? If it returns the expected result, the fix would probably be in `to_datetime` instead of the parsing code.
@mroeschke Thinks for the idea! I don't want to modify `to_datetime`. Since the `box=True` means the return of `to_datetime` should be an index, so I think `to_datetime` should remain as is. I just want to update CSV parsing process.

I think I have come up with a solution, inspired by @mroeschke .
```
In [1]: data = ['2018-01-04 09:01:00+09:00', '2018-01-04 09:02:00+09:00']
In [2]: dti = pd.to_datetime(data, box=True)
In [3]: np.array(dti.tolist())
Out[3]:
array([Timestamp('2018-01-04 09:01:00+0900', tz='pytz.FixedOffset(540)'),
       Timestamp('2018-01-04 09:02:00+0900', tz='pytz.FixedOffset(540)')],
      dtype=object)

```
The whole problem was caused by `numpy.ndarray` with `datetime64` dtype not being able to contain timezone information. However, `numpy.ndarray` with `object` dtype can contain pandas `Timestamp`s. This makes the PR much simpler.
@swyoon object arrays of Timestamps are quite non performant
this is why we use DTI in the first place 

@jreback I have applied `np.asarray` as you requested. It works well :) All checks are green.
@swyoon : Can you make the changes in #22422 (as a separate commit)?  They should work with your PR.
@gfyoung Added a commit to fix #22422. It worked fine. 
Travis seems to fail occasionally, and it doesn't seem to be related to my PR. Could you take a look?
Yeah...Travis has been acting up a bit.  Restarted the failing builds, as they didn't look related to this PR.
@gfyoung Thank, but it failed again... [failed job](https://travis-ci.org/pandas-dev/pandas/jobs/418065455)  Could you run it one more time, please?
@gfyoung Sorry, but travis still doesn't work
I *think* we also might have another open issue about this (aside from the referenced one), if you'd do a search and see.
@jreback @gfyoung Finally, it's green. This build took so long. I will check out open issues that might be relevant to this PR.
@jreback @gfyoung could you please take a look?
@jreback is there anything needs to be done before merging?
thanks @swyoon 
@jreback @gfyoung @mroeschke Thank you all