Repository: pandas-dev/pandas

Issue:
Timezone info is lost when pickling datetime indices
``` python
>>> c.head(3)
                                                           end_time
location_id sensor_id view_id start_time
54          1000305   0       2014-07-21 07:00:00+00:00    2014-07-21 07:15:00+00:00
                              2014-07-21 07:15:00+00:00    2014-07-21 07:30:00+00:00
                              2014-07-21 07:30:00+00:00    2014-07-21 07:45:00+00:00

>>> c.to_pickle(r"C:\temp\fff")
>>> pd.read_pickle(r"C:\temp\fff").head(3)
                                                   end_time
location_id sensor_id view_id start_time
54          1000305   0       2014-07-21 07:00:00  2014-07-21 07:15:00+00:00
                              2014-07-21 07:15:00  2014-07-21 07:30:00+00:00
                              2014-07-21 07:30:00  2014-07-21 07:45:00+00:00
```

Please note that the timezone is lost from `start_time`.



Discussion:
I forgot my versions information:

``` python
>>> pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Windows
OS-release: 8
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.14.0
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2014.3
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None
```

can u show how this was created in the first place? also df.info()

The dataframe was created with:

``` python
import pytz
from dateutil.parser import parse
def parse_time(v):
    return parse(v).replace(tzinfo=pytz.utc)

df = pd.read_csv(r"C:\counts\raw.data", index_col=[2, 3, 4, 0], date_parser=parse_time,
                 parse_dates=[0, 1], header=0)
```

Here's the result of `df.info()` before serializing:

```
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 1931 entries, (54, 1000320, 0, 2014-07-21 08:00:00+00:00) to (54, 1000317, 0, 2014-07-27 19:15:00+00:00)
Data columns (total 3 columns):
end_time       1931 non-null object
plus_count     1931 non-null int64
minus_count    1931 non-null int64
dtypes: int64(2), object(1)
```

And after deserializing:

```
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 1931 entries, (54, 1000320, 0, 2014-07-21 08:00:00) to (54, 1000317, 0, 2014-07-27 19:15:00)
Data columns (total 3 columns):
end_time       1931 non-null object
plus_count     1931 non-null int64
minus_count    1931 non-null int64
dtypes: int64(2), object(1)

```

try with a pre-release of 0.15.0 and see if the error persists: https://github.com/pydata/pandas/releases

Hmm, this does seem buggy.

```
In [16]: df = DataFrame(np.random.randn(12,2),index=pd.MultiIndex.from_product([[1,2],['a','b'],date_range('20130101',periods=3,tz='US/Eastern')]))

In [17]: df
Out[17]: 
                                      0         1
1 a 2013-01-01 00:00:00-05:00 -0.093854  0.181606
    2013-01-02 00:00:00-05:00  1.950901 -1.415333
    2013-01-03 00:00:00-05:00 -1.631278  1.487381
  b 2013-01-01 00:00:00-05:00  0.040334  2.698802
    2013-01-02 00:00:00-05:00 -0.249099  0.212568
    2013-01-03 00:00:00-05:00  0.003727 -0.012983
2 a 2013-01-01 00:00:00-05:00  0.268326 -0.670518
    2013-01-02 00:00:00-05:00 -0.776091 -1.688146
    2013-01-03 00:00:00-05:00 -0.371485 -0.963880
  b 2013-01-01 00:00:00-05:00  0.430981 -1.735434
    2013-01-02 00:00:00-05:00  0.974573 -0.226431
    2013-01-03 00:00:00-05:00 -0.294460 -0.627837

In [18]: df.to_pickle('test.pkl')

In [19]: pd.read_pickle('test.pkl')
Out[19]: 
                                0         1
1 a 2013-01-01 05:00:00 -0.093854  0.181606
    2013-01-02 05:00:00  1.950901 -1.415333
    2013-01-03 05:00:00 -1.631278  1.487381
  b 2013-01-01 05:00:00  0.040334  2.698802
    2013-01-02 05:00:00 -0.249099  0.212568
    2013-01-03 05:00:00  0.003727 -0.012983
2 a 2013-01-01 05:00:00  0.268326 -0.670518
    2013-01-02 05:00:00 -0.776091 -1.688146
    2013-01-03 05:00:00 -0.371485 -0.963880
  b 2013-01-01 05:00:00  0.430981 -1.735434
    2013-01-02 05:00:00  0.974573 -0.226431
    2013-01-03 05:00:00 -0.294460 -0.627837

In [20]: pd.read_pickle('test.pkl').index
Out[20]: 
MultiIndex(levels=[[1, 2], [u'a', u'b'], [2013-01-01 05:00:00, 2013-01-02 05:00:00, 2013-01-03 05:00:00]],
           labels=[[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1], [0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1], [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]])

In [21]: pd.read_pickle('test.pkl').index.levels[2]
Out[21]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 05:00:00, ..., 2013-01-03 05:00:00]
Length: 3, Freq: None, Timezone: None

```


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -3323,8 +3323,8 @@ def __contains__(self, key):
 
     def __reduce__(self):
         """Necessary for making this object picklable"""
-        d = dict(levels = [lev.view(np.ndarray) for lev in self.levels],
-                 labels = [label.view(np.ndarray) for label in self.labels],
+        d = dict(levels = [lev for lev in self.levels],
+                 labels = [label for label in self.labels],
                  sortorder = self.sortorder,
                  names = list(self.names))
         return _new_Index, (self.__class__, d), None
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -105,6 +105,17 @@ def _ensure_datetime64(other):
 
 _midnight = time(0, 0)
 
+def _new_DatetimeIndex(cls, d):
+    """ This is called upon unpickling, rather than the default which doesn't have arguments
+        and breaks __new__ """
+
+    # simply set the tz
+    # data are already in UTC
+    tz = d.pop('tz',None)
+    result = cls.__new__(cls, **d)
+    result.tz = tz
+    return result
+
 class DatetimeIndex(DatetimeIndexOpsMixin, Int64Index):
     """
     Immutable ndarray of datetime64 data, represented internally as int64, and
@@ -583,6 +594,15 @@ def _formatter_func(self):
         formatter = _get_format_datetime64(is_dates_only=self._is_dates_only)
         return lambda x: formatter(x, tz=self.tz)
 
+    def __reduce__(self):
+
+        # we use a special reudce here because we need
+        # to simply set the .tz (and not reinterpret it)
+
+        d = dict(data=self._data)
+        d.update(self._get_attributes_dict())
+        return _new_DatetimeIndex, (self.__class__, d), None
+
     def __setstate__(self, state):
         """Necessary for making this object picklable"""
         if isinstance(state, dict):


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -1645,6 +1645,14 @@ def test_numeric_compat(self):
                       lambda : pd.date_range('2000-01-01', periods=3) * np.timedelta64(1, 'D').astype('m8[ns]') ]:
                 self.assertRaises(TypeError, f)
 
+    def test_roundtrip_pickle_with_tz(self):
+
+        # GH 8367
+        # round-trip of timezone
+        index=date_range('20130101',periods=3,tz='US/Eastern',name='foo')
+        unpickled = self.round_trip_pickle(index)
+        self.assertTrue(index.equals(unpickled))
+
 class TestPeriodIndex(Base, tm.TestCase):
     _holder = PeriodIndex
     _multiprocess_can_split_ = True
@@ -2347,6 +2355,14 @@ def test_legacy_v2_unpickle(self):
         assert_almost_equal(res, exp)
         assert_almost_equal(exp, exp2)
 
+    def test_roundtrip_pickle_with_tz(self):
+
+        # GH 8367
+        # round-trip of timezone
+        index=MultiIndex.from_product([[1,2],['a','b'],date_range('20130101',periods=3,tz='US/Eastern')],names=['one','two','three'])
+        unpickled = self.round_trip_pickle(index)
+        self.assertTrue(index.equal_levels(unpickled))
+
     def test_from_tuples_index_values(self):
         result = MultiIndex.from_tuples(self.index)
         self.assertTrue((result.values == self.index.values).all())