BUG: Correctly localize naive datetime strings with Series and datetimetztype (#17415)
State: closed

- [x] closes #17415
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Timezones
description: Timezone data dtype

Comments:
Hello @mroeschke! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on January 12, 2018 at 05:11 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=h1) Report
> Merging [#17603](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fedf92287d0216d54fe756829362e1d10a3fdc58?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17603/graphs/tree.svg?token=eZ4WkYLtcO&width=650&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17603      +/-   ##
==========================================
- Coverage   91.19%   91.18%   -0.02%     
==========================================
  Files         163      163              
  Lines       49625    49629       +4     
==========================================
- Hits        45257    45252       -5     
- Misses       4368     4377       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.96% <100%> (ø)` | :arrow_up: |
| #single | `40.19% <0%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `87.9% <100%> (+0.08%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=footer). Last update [fedf922...8910a54](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=h1) Report
> Merging [#17603](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/982e11233698bc54e59be87eb257da2f33b8eb10?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17603/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17603      +/-   ##
==========================================
+ Coverage   91.52%   91.54%   +0.02%     
==========================================
  Files         148      147       -1     
  Lines       48778    48777       -1     
==========================================
+ Hits        44642    44655      +13     
+ Misses       4136     4122      -14
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.92% <87.5%> (+0.02%)` | :arrow_up: |
| #single | `41.62% <87.5%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `88.66% <87.5%> (+0.23%)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.62% <0%> (ø)` | :arrow_up: |
| [pandas/tools/hashing.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603/diff?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL2hhc2hpbmcucHk=) | | |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.49% <0%> (+0.06%)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17603/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `66.95% <0%> (+1.73%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=footer). Last update [982e112...cee514f](https://codecov.io/gh/pandas-dev/pandas/pull/17603?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

pls rebase and move note to 0.21.1
let me have a look
can you rebase
if you can change this to 0.22 for the whatsnew and rebase can have a look
Rebased and moved the whatsnew to v0.22.

Still curious how to handle the following situation:

```
In [3]: Series([pd.NaT], dtype='datetime64[ns, CET]') # same for np.nan and None
Out[3]: 
0   NaT
dtype: datetime64[ns, CET]

In [4]: Series(pd.Timestamp(pd.NaT, tz='CET')) # same for np.nan and None
Out[4]: 
0   NaT
dtype: datetime64[ns]
```
`Out[4]` should be the correct behavior for both I assume. 
@mroeschke a way you can handle is like this

```
In [1]: from pandas.api.types import DatetimeTZDtype

In [2]: DatetimeTZDtype('datetime64[ns, CET]')
Out[2]: datetime64[ns, CET]

In [3]: DatetimeTZDtype('CET')
ValueError: DatetimeTZDtype constructor must have a tz supplied
```

so in a try/except if you succeed you have a valid dtype, otherwise you can pass it directly to the constructor.
@jreback  After some digging, the reason why:
```
In [3]: Series([pd.NaT], dtype='datetime64[ns, CET]') # same for np.nan and None
Out[3]: 
0   NaT
dtype: datetime64[ns, CET]
```

occurs is that this happens up the stack:

```
In [3]: pd.to_datetime([pd.NaT]).tz_localize('CET')
Out[3]: DatetimeIndex(['NaT'], dtype='datetime64[ns, CET]', freq=None)
```

Curious to get your opinion if this should raise instead.
```
In [1]: pd.Timestamp('NaT')
Out[1]: NaT

In [2]: pd.Timestamp('NaT').tz_localize('CET')
Out[2]: NaT
```

```
In [4]: pd.to_datetime(['nat']).tz_localize('CET')
Out[4]: DatetimeIndex(['NaT'], dtype='datetime64[ns, CET]', freq=None)

In [5]: pd.to_datetime(['nat', '20170101']).tz_localize('CET')
Out[5]: DatetimeIndex(['NaT', '2017-01-01 00:00:00+01:00'], dtype='datetime64[ns, CET]', freq=None)

In [6]: pd.to_datetime(['nat']).tz_localize('CET')[0]
Out[6]: NaT
```
@mroeschke this is pretty consistent. NaT can be part of a tz-aware Index/Series. It doesn't have a tz itself though.
Makes sense, thanks for confirming. 
I realized I was concerned about a non-issue since:

```
In [4]: pd.to_datetime(['nat']).tz_localize('CET')
Out[4]: DatetimeIndex(['NaT'], dtype='datetime64[ns, CET]', freq=None)
```

makes sense then

```
In [3]: Series([pd.NaT], dtype='datetime64[ns, CET]') # same for np.nan and None
Out[3]: 
0   NaT
dtype: datetime64[ns, CET]
```

makes sense as well.

All green. 
thanks!