BUG: Fix concat series loss of timezone
State: closed

Closes #23816

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @evangelineliu! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/dtypes/concat.py`](https://github.com/evangelineliu/pandas/blob/1867b3ad6514a1de18cb5ce7fdfe2cbbb95f57c6/pandas/core/dtypes/concat.py) !

 - There are no PEP8 issues in the file [`pandas/tests/reshape/test_concat.py`](https://github.com/evangelineliu/pandas/blob/1867b3ad6514a1de18cb5ce7fdfe2cbbb95f57c6/pandas/tests/reshape/test_concat.py) !



##### Comment last updated on December 05, 2018 at 22:33 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=h1) Report
> Merging [#24027](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5b6b346b9db57b610df2965f60753473ef16e16f?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24027/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24027      +/-   ##
==========================================
+ Coverage   42.46%   42.46%   +<.01%     
==========================================
  Files         161      161              
  Lines       51557    51556       -1     
==========================================
  Hits        21892    21892              
+ Misses      29665    29664       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.46% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `56.25% <0%> (ø)` | :arrow_up: |
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `11.69% <0%> (+0.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=footer). Last update [5b6b346...1804d7d](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=h1) Report
> Merging [#24027](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1d3ed917638b9890e0410fa996a447c4a780429e?src=pr&el=desc) will **increase** coverage by `49.68%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24027/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24027       +/-   ##
===========================================
+ Coverage   42.52%    92.2%   +49.68%     
===========================================
  Files         161      162        +1     
  Lines       51697    51727       +30     
===========================================
+ Hits        21982    47695    +25713     
+ Misses      29715     4032    -25683
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.6% <100%> (?)` | |
| #single | `43.02% <0%> (+0.5%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `96.6% <100%> (+40.35%)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `96.64% <0%> (ø)` | |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `92.37% <0%> (+0.3%)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.3% <0%> (+0.92%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.24% <0%> (+4.05%)` | :arrow_up: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.3% <0%> (+8.1%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `99.24% <0%> (+9.84%)` | :arrow_up: |
| [pandas/core/reshape/util.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS91dGlsLnB5) | `100% <0%> (+11.53%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `92.85% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `85.71% <0%> (+14.28%)` | :arrow_up: |
| ... and [120 more](https://codecov.io/gh/pandas-dev/pandas/pull/24027/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=footer). Last update [1d3ed91...1867b3a](https://codecov.io/gh/pandas-dev/pandas/pull/24027?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can you add “closes 
https://github.com/pandas-dev/pandas/issues/23816” to the original post?

We also need a test (the one from the issue is fine) and a release note. 
_concat_compat isn’t called from categorical. Maybe it could be.


________________________________
From: Jeff Reback <notifications@github.com>
Sent: Saturday, December 1, 2018 10:55 AM
To: pandas-dev/pandas
Cc: Tom Augspurger; Comment
Subject: Re: [pandas-dev/pandas] BUG: Fix concat series loss of timezone (#24027)


@jreback requested changes on this pull request.

________________________________

In pandas/core/dtypes/concat.py<https://github.com/pandas-dev/pandas/pull/24027#discussion_r238069451>:

> @@ -193,7 +193,8 @@ def _concat_categorical(to_concat, axis=0):

     def _concat_asobject(to_concat):
         to_concat = [x.get_values() if is_categorical_dtype(x.dtype)
-                     else np.asarray(x).ravel() for x in to_concat]
+                     else np.asarray(x).ravel() if not is_datetime64tz_dtype(x)


this is not the way
_concat_conpat already handles this

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/24027#pullrequestreview-180530317>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHInqCusNq4uKR4J1EEmqEonkDD7M1ks5u0rR-gaJpZM4Y8iIc>.

Hi, so should we be calling from concat_compat? Why is calling from concat_compat better? Does the current way of fixing the issue conflict with something else in the codebase? @jreback 
Just wanted to test something here, I'm decently sure that these tests will fail.
Sorry, still trying to figure out why the dimensions would be different for the arrays passed to np.concatenate. In your experience why does this usually happen? 

> Sorry, still trying to figure out why the dimensions would be different for the arrays passed to np.concatenate. In your experience why does this usually happen?

datetimes with timezones are backed by a DatetimeIndex itself which is only 1-D object. datetimes that are naive are 2-D blocks internaly.
Taking a look at concat_compat, I agree with @jakezimmer that I'm not super sure if it actually takes care of the if checks that we tried to delete. 
FYI, here's a test

```diff
diff --git a/pandas/tests/reshape/test_concat.py b/pandas/tests/reshape/test_concat.py
index 4113fb7f0..bb002f151 100644
--- a/pandas/tests/reshape/test_concat.py
+++ b/pandas/tests/reshape/test_concat.py
@@ -2552,3 +2552,16 @@ def test_concat_series_name_npscalar_tuple(s1name, s2name):
     result = pd.concat([s1, s2])
     expected = pd.Series({'a': 1, 'b': 2, 'c': 5, 'd': 6})
     tm.assert_series_equal(result, expected)
+
+
+def test_concat_categorical_tz():
+    # GH-23816
+    a = pd.Series(pd.date_range('2017-01-01', periods=2, tz='US/Pacific'))
+    b = pd.Series(['a', 'b'], dtype='category')
+    result = pd.concat([a, b], ignore_index=True)
+    expected = pd.Series([
+        pd.Timestamp('2017-01-01', tz="US/Pacific"),
+        pd.Timestamp('2017-01-02', tz="US/Pacific"),
+        'a', 'b'
+    ])
+    tm.assert_series_equal(result, expected)
```
Is this a test for the new output? Because I tried what you said earlier already but it didn't build
That's the expected output from whatever ends up fixing the original issue.
Should we add in the test case? I'm a little bit confused about what's happening exactly with the tests
Yes. That test is asserting the correct behavior when concatenating a Categorical and timezone-aware series. The expected output is an object-dtype series.
@evangelineliu pushed a small cleanup. pls add a whatsnew note (reshaping bug fixes). ping on green.
Hi @jreback, I noticed that the fix up [commit](https://github.com/pandas-dev/pandas/pull/24027/commits/c7dcdb445506a0a00fbd9363d6551b0a44d5c259) broke some linting as you can see [here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=4657). Do you know what could be causing this?

Also I added the whatsnew change. This is my first time doing this so would you be able to take a look at the [proposed change](https://github.com/pandas-dev/pandas/pull/24027/commits/165689e863a0f1700796da8265209745904cb51e) and see if it is acceptable.
I'm working soon that in https://github.com/pandas-dev/pandas/pull/24092

On Tue, Dec 4, 2018 at 1:41 PM jakezimmer <notifications@github.com> wrote:

> Hi @jreback <https://github.com/jreback>, I noticed that the fix up commit
> <https://github.com/pandas-dev/pandas/pull/24027/commits/c7dcdb445506a0a00fbd9363d6551b0a44d5c259>
> broke some linting as you can see here
> <https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=4657>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24027#issuecomment-444231192>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjYIy1FN-fq1YfdDip9kBjrH-Tpbks5u1tAHgaJpZM4Y8iIc>
> .
>

@jreback all tests passed!
also pls merge master
thanks ping on green.
@jreback do you know what would cause pandas azure pipelines job to fail like this on the "before install" task. It looks like it couldn't download anaconda. The only change was removing that whitespace and it has never happened in previous commits. Is there a way to try that test?

You can see the error here under Linux py36_locale_slow:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=4780
thanks @jakezimmer 

that pipeline failed on some network timeout.