Repository: pandas-dev/pandas

Issue:
pd.concat reset the tz-aware index to UTC
Hi, 
I reopen https://github.com/pandas-dev/pandas/issues/18422 with  copy-pastable example

#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

idx1 = pd.date_range('2011-01-01', periods=3, freq='H', tz='Europe/Paris')
idx2 = pd.date_range(start=idx1[0], end=idx1[-1], freq='H')

df1 = pd.DataFrame({'a': [1, 2, 3]}, index=idx1)
df2 = pd.DataFrame({'b': [1, 2, 3]}, index=idx2)

res = pd.concat([df1, df2], axis=1)

print(df1.index.tzinfo)
print(df2.index.tzinfo)
print(res.index.tzinfo)
```

#### Output
```
Europe/Paris
Europe/Paris
UTC
```

#### Problem description

pd.concat reset the tz-aware index to UTC

This seems to come from the two different representation of time zone 'Europe/Paris' as this ipython output shows :

```
In [23]: df1.index.tzinfo
Out[23]: <DstTzInfo 'Europe/Paris' LMT+0:09:00 STD>

In [24]: df2.index.tzinfo
Out[24]: <DstTzInfo 'Europe/Paris' CET+1:00:00 STD>

In [26]: res.index.tzinfo
Out[26]: <UTC>
```

#### Expected Output
```
Europe/Paris
Europe/Paris
Europe/Paris
```
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.13.1
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
How odd indeed!  This example is indeed reproducible on my end this time around.

Have a look and see why we're getting this discrepancy in timezone representation.  PR is most welcome!
The issue is from DatetimeIndex : when creating a localized DatetimeIndex, the DatetimeIndex and Timestamps in the DatetimeIndex have different tzinfo ...

```
In [9]: idx = pd.DatetimeIndex(start='2011-01-01', periods=3, freq='H', tz='Europe/Paris')

In [10]: idx.tzinfo
Out[10]: <DstTzInfo 'Europe/Paris' LMT+0:09:00 STD>

In [11]: idx[0].tzinfo
Out[11]: <DstTzInfo 'Europe/Paris' CET+1:00:00 STD>
```
@antvig this is a bug, probably a duplicate of something in : https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3AReshaping+label%3ATimezones

but not because of what you show: https://github.com/pandas-dev/pandas/issues/18523#issuecomment-347583582

rather the combined index is reset to UTC rather than preserve the incoming tz. 

i'll mark this.
The combined index preserve the incoming tz if the two dataframe indexes have exactly the same timezone

```
idx1 = pd.date_range('2011-01-01', periods=3, freq='H', tz='Europe/Paris')
idx2 = pd.date_range('2011-01-01', periods=3, freq='H', tz='Europe/Paris')

df1 = pd.DataFrame({'a': [1, 2, 3]}, index=idx1)
df2 = pd.DataFrame({'b': [1, 2, 3]}, index=idx2)
res = pd.concat([df1, df2], axis=1)

print(df1.index.tzinfo)
print(df2.index.tzinfo)
print(res.index.tzinfo)
``` 
**Output**
``` 
Europe/Paris
Europe/Paris
Europe/Paris
``` 
Seem related to https://github.com/pandas-dev/pandas/issues/17572
@jorisvandenbossche see my comment above, #17572 is irrelevant
And how do you explain https://github.com/pandas-dev/pandas/issues/18523#issuecomment-347837147?
This is due to `_maybe_utc_convert` which checks the timezones by equality:

https://github.com/pandas-dev/pandas/blob/d3c3c2b092b17aa720b489101d59d60aff8799da/pandas/core/indexes/datetimes.py#L1141-L1144

As explained in #17572, such equality check does not always give True, depending on how the `tzinfo` object is constructed, although they are in the same timezone (#17572 links to this [SO question](https://stackoverflow.com/questions/24359540/why-doesnt-pytz-localize-produce-a-datetime-object-with-tzinfo-matching-the-t))
here is a patch to fix:

```
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index 196c881f9..01229e042 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -1138,7 +1138,7 @@ class DatetimeIndex(DatelikeOps, TimelikeOps, DatetimeIndexOpsMixin,
                 raise TypeError('Cannot join tz-naive with tz-aware '
                                 'DatetimeIndex')
 
-            if self.tz != other.tz:
+            if str(self.tz) != str(other.tz):
                 this = self.tz_convert('UTC')
                 other = other.tz_convert('UTC')
         return this, other
```


thanks to @pganssle for the advice!

Resolving patch:
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -521,8 +521,7 @@ def _generate(cls, start, end, periods, name, offset,
                 tz = tz.localize(date.replace(tzinfo=None)).tzinfo
 
         if tz is not None and inferred_tz is not None:
-            if not (timezones.get_timezone(inferred_tz) ==
-                    timezones.get_timezone(tz)):
+            if not timezones.tz_compare(inferred_tz, tz):
                 raise AssertionError("Inferred time zone not equal to passed "
                                      "time zone")
 
@@ -1192,7 +1191,7 @@ def _maybe_utc_convert(self, other):
                 raise TypeError('Cannot join tz-naive with tz-aware '
                                 'DatetimeIndex')
 
-            if self.tz != other.tz:
+            if not timezones.tz_compare(self.tz, other.tz):
                 this = self.tz_convert('UTC')
                 other = other.tz_convert('UTC')
         return this, other
@@ -1296,7 +1295,7 @@ def __iter__(self):
 
     def _wrap_union_result(self, other, result):
         name = self.name if self.name == other.name else None
-        if self.tz != other.tz:
+        if not timezones.tz_compare(self.tz, other.tz):
             raise ValueError('Passed item and index have different timezone')
         return self._simple_new(result, name=name, freq=None, tz=self.tz)
 


Test patch:
diff --git a/pandas/tests/reshape/test_concat.py b/pandas/tests/reshape/test_concat.py
--- a/pandas/tests/reshape/test_concat.py
+++ b/pandas/tests/reshape/test_concat.py
@@ -2074,6 +2074,45 @@ def test_concat_order(self):
             expected = expected.sort_values()
         tm.assert_index_equal(result, expected)
 
+    def test_concat_datetime_timezone(self):
+        # GH 18523
+        idx1 = pd.date_range('2011-01-01', periods=3, freq='H',
+                             tz='Europe/Paris')
+        idx2 = pd.date_range(start=idx1[0], end=idx1[-1], freq='H')
+        df1 = pd.DataFrame({'a': [1, 2, 3]}, index=idx1)
+        df2 = pd.DataFrame({'b': [1, 2, 3]}, index=idx2)
+        result = pd.concat([df1, df2], axis=1)
+
+        exp_idx = DatetimeIndex(['2011-01-01 00:00:00+01:00',
+                                 '2011-01-01 01:00:00+01:00',
+                                 '2011-01-01 02:00:00+01:00'],
+                                freq='H'
+                                ).tz_localize('UTC').tz_convert('Europe/Paris')
+
+        expected = pd.DataFrame([[1, 1], [2, 2], [3, 3]],
+                                index=exp_idx, columns=['a', 'b'])
+
+        tm.assert_frame_equal(result, expected)
+
+        idx3 = pd.date_range('2011-01-01', periods=3,
+                             freq='H', tz='Asia/Tokyo')
+        df3 = pd.DataFrame({'b': [1, 2, 3]}, index=idx3)
+        result = pd.concat([df1, df3], axis=1)
+
+        exp_idx = DatetimeIndex(['2010-12-31 15:00:00+00:00',
+                                 '2010-12-31 16:00:00+00:00',
+                                 '2010-12-31 17:00:00+00:00',
+                                 '2010-12-31 23:00:00+00:00',
+                                 '2011-01-01 00:00:00+00:00',
+                                 '2011-01-01 01:00:00+00:00']
+                                ).tz_localize('UTC')
+
+        expected = pd.DataFrame([[np.nan, 1], [np.nan, 2], [np.nan, 3],
+                                 [1, np.nan], [2, np.nan], [3, np.nan]],
+                                index=exp_idx, columns=['a', 'b'])
+
+        tm.assert_frame_equal(result, expected)
+
 
 @pytest.mark.parametrize('pdt', [pd.Series, pd.DataFrame, pd.Panel])
 @pytest.mark.parametrize('dt', np.sctypes['float'])