Repository: pandas-dev/pandas

Issue:
tz info lost when creating multiindex
see http://stackoverflow.com/questions/17159207/change-timezone-of-date-time-column-in-pandas-and-add-as-hierarchical-index/17159276#17159276

```
dat = pd.DataFrame({'label':['a', 'a', 'a', 'b', 'b', 'b'], 'datetime':['2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00', '2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00'], 'value':range(6)})
dat.index = pd.to_datetime(dat.pop('datetime'), utc=True)
dat.index = dat.index.tz_localize('UTC').tz_convert('US/Pacific')

dat
                          label  value
datetime
2011-07-19 00:00:00-07:00     a      0
2011-07-19 01:00:00-07:00     a      1
2011-07-19 02:00:00-07:00     a      2
2011-07-19 00:00:00-07:00     b      3
2011-07-19 01:00:00-07:00     b      4
2011-07-19 02:00:00-07:00     b      5
```

If we add another index (we lose the tz):

```
In [14]: dat.set_index('label', append=True).swaplevel(0, 1)
Out[14]:
                           value
label datetime
a     2011-07-19 07:00:00      0
      2011-07-19 08:00:00      1
      2011-07-19 09:00:00      2
b     2011-07-19 07:00:00      3
      2011-07-19 08:00:00      4
      2011-07-19 09:00:00      5
```



Discussion:
that is a gnarly one-liner there. i get an exception on the index assignment

Ah, you're right. Tried to be too clever there. (fixed)

This problem also occurs with groupby operations that create multiple indexes with time zones. For example:

```
dat = pd.DataFrame({'label':['a', 'a', 'a', 'b', 'b', 'b'], 'datetime':['2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00', '2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00'], 'value':range(6)})
dat['datetime'] = dat['datetime'].apply(lambda d: pandas.Timestamp(d, tz='US/Pacific'))
```

We start out with timezones:

```
In [386]: dat.head()
Out[386]: 
                    datetime label  value
0  2011-07-19 07:00:00-07:00     a      0
1  2011-07-19 08:00:00-07:00     a      1
2  2011-07-19 09:00:00-07:00     a      2
3  2011-07-19 07:00:00-07:00     b      3
4  2011-07-19 08:00:00-07:00     b      4
```

However, if we do a groupby on multiple columns, we will lose the timezones:

```
In [387]: dat.groupby(['datetime', 'label'])['value'].sum()
Out[387]: 
datetime             label
2011-07-19 14:00:00  a        0
                     b        3
2011-07-19 15:00:00  a        1
                     b        4
2011-07-19 16:00:00  a        2
                     b        5
```

We just upgraded to pandas 0.14, and the groupby/pivot bugs related to losing tz-info have been fixed.  However, reset_index of a MultiIndex still loses tz-info.

```
In [7]: ts = pd.date_range('1/1/2011', periods=5, freq='10s', tz = 'US/Eastern')

In [8]: foo = pd.DataFrame({'c' : range(5)}, index = pd.MultiIndex([range(5), ts], [range(5), range(5)], names = [ 'a' , 'b' ]))

In [9]: print foo                                                                                                
                             c
a b                           
0 2011-01-01 00:00:00-05:00  0
1 2011-01-01 00:00:10-05:00  1
2 2011-01-01 00:00:20-05:00  2
3 2011-01-01 00:00:30-05:00  3
4 2011-01-01 00:00:40-05:00  4

In [10]: foo2 = foo.reset_index()                                                                                                

In [11]: print foo2                                                                                                              
   a                   b  c
0  0 2011-01-01 05:00:00  0
1  1 2011-01-01 05:00:10  1
2  2 2011-01-01 05:00:20  2
3  3 2011-01-01 05:00:30  3
4  4 2011-01-01 05:00:40  4
```

And, column assignment still loses tz-info.

```
In [13]: bar = pd.DataFrame( {'a' : ts, 'b':range(5)} )

In [14]: print bar
                    a  b
0 2011-01-01 05:00:00  0
1 2011-01-01 05:00:10  1
2 2011-01-01 05:00:20  2
3 2011-01-01 05:00:30  3
4 2011-01-01 05:00:40  4
```

cc @sinhrks take a look?


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2326,19 +2326,24 @@ def reset_index(self, level=None, drop=False, inplace=False, col_level=0,
         else:
             new_obj = self.copy()
 
-        def _maybe_cast(values, labels=None):
-
-            if values.dtype == np.object_:
-                values = lib.maybe_convert_objects(values)
-
-            # if we have the labels, extract the values with a mask
-            if labels is not None:
-                mask = labels == -1
-                values = values.take(labels)
-                if mask.any():
-                    values, changed = com._maybe_upcast_putmask(
-                        values, mask, np.nan)
-
+        def _maybe_casted_values(index, labels=None):
+            if isinstance(index, PeriodIndex):
+                values = index.asobject
+            elif (isinstance(index, DatetimeIndex) and
+                  index.tz is not None):
+                values = index.asobject
+            else:
+                values = index.values
+                if values.dtype == np.object_:
+                    values = lib.maybe_convert_objects(values)
+
+                # if we have the labels, extract the values with a mask
+                if labels is not None:
+                    mask = labels == -1
+                    values = values.take(labels)
+                    if mask.any():
+                        values, changed = com._maybe_upcast_putmask(values,
+                                                                    mask, np.nan)
             return values
 
         new_index = np.arange(len(new_obj))
@@ -2371,7 +2376,7 @@ def _maybe_cast(values, labels=None):
                             col_name = tuple(name_lst)
 
                     # to ndarray and maybe infer different dtype
-                    level_values = _maybe_cast(lev.values, lab)
+                    level_values = _maybe_casted_values(lev, lab)
                     if level is None or i in level:
                         new_obj.insert(0, col_name, level_values)
 
@@ -2387,13 +2392,7 @@ def _maybe_cast(values, labels=None):
                     lev_num = self.columns._get_level_number(col_level)
                     name_lst[lev_num] = name
                     name = tuple(name_lst)
-            if isinstance(self.index, PeriodIndex):
-                values = self.index.asobject
-            elif (isinstance(self.index, DatetimeIndex) and
-                  self.index.tz is not None):
-                values = self.index.asobject
-            else:
-                values = _maybe_cast(self.index.values)
+            values = _maybe_casted_values(self.index)
             new_obj.insert(0, name, values)
 
         new_obj.index = new_index


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -2182,6 +2182,20 @@ def test_set_index_cast_datetimeindex(self):
         df.pop('ts')
         assert_frame_equal(df, expected)
 
+        # GH 3950
+        # reset_index with single level
+        for tz in ['UTC', 'Asia/Tokyo', 'US/Eastern']:
+            idx = pd.date_range('1/1/2011', periods=5, freq='D', tz=tz, name='idx')
+            df = pd.DataFrame({'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']}, index=idx) 
+
+            expected = pd.DataFrame({'idx': [datetime(2011, 1, 1), datetime(2011, 1, 2),
+                                             datetime(2011, 1, 3), datetime(2011, 1, 4),
+                                             datetime(2011, 1, 5)], 
+                                     'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']},
+                                     columns=['idx', 'a', 'b'])
+            expected['idx'] = expected['idx'].apply(lambda d: pd.Timestamp(d, tz=tz))
+            assert_frame_equal(df.reset_index(), expected)
+
     def test_set_index_multiindexcolumns(self):
         columns = MultiIndex.from_tuples([('foo', 1), ('foo', 2), ('bar', 1)])
         df = DataFrame(np.random.randn(3, 3), columns=columns)
diff --git a/pandas/tests/test_multilevel.py b/pandas/tests/test_multilevel.py
--- a/pandas/tests/test_multilevel.py
+++ b/pandas/tests/test_multilevel.py
@@ -2078,6 +2078,46 @@ def test_set_index_datetime(self):
         self.assertTrue(df.index.get_level_values(1).equals(idx2))
         self.assertTrue(df.index.get_level_values(2).equals(idx3))
 
+    def test_reset_index_datetime(self):
+        # GH 3950
+        for tz in ['UTC', 'Asia/Tokyo', 'US/Eastern']:
+            idx1 = pd.date_range('1/1/2011', periods=5, freq='D', tz=tz, name='idx1')
+            idx2 = pd.Index(range(5), name='idx2')
+            idx = pd.MultiIndex.from_arrays([idx1, idx2])
+            df = pd.DataFrame({'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']}, index=idx) 
+
+            expected = pd.DataFrame({'idx1': [datetime.datetime(2011, 1, 1),
+                                              datetime.datetime(2011, 1, 2),
+                                              datetime.datetime(2011, 1, 3),
+                                              datetime.datetime(2011, 1, 4),
+                                              datetime.datetime(2011, 1, 5)], 
+                                     'idx2': range(5),
+                                     'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']},
+                                     columns=['idx1', 'idx2', 'a', 'b'])
+            expected['idx1'] = expected['idx1'].apply(lambda d: pd.Timestamp(d, tz=tz))
+            assert_frame_equal(df.reset_index(), expected)
+
+            idx3 = pd.date_range('1/1/2012', periods=5, freq='MS', tz='Europe/Paris', name='idx3')
+            idx = pd.MultiIndex.from_arrays([idx1, idx2, idx3])
+            df = pd.DataFrame({'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']}, index=idx) 
+
+            expected = pd.DataFrame({'idx1': [datetime.datetime(2011, 1, 1),
+                                              datetime.datetime(2011, 1, 2),
+                                              datetime.datetime(2011, 1, 3),
+                                              datetime.datetime(2011, 1, 4),
+                                              datetime.datetime(2011, 1, 5)], 
+                                     'idx2': range(5),
+                                     'idx3': [datetime.datetime(2012, 1, 1),
+                                              datetime.datetime(2012, 2, 1),
+                                              datetime.datetime(2012, 3, 1),
+                                              datetime.datetime(2012, 4, 1),
+                                              datetime.datetime(2012, 5, 1)], 
+                                     'a': range(5), 'b': ['A', 'B', 'C', 'D', 'E']},
+                                     columns=['idx1', 'idx2', 'idx3', 'a', 'b'])
+            expected['idx1'] = expected['idx1'].apply(lambda d: pd.Timestamp(d, tz=tz))
+            expected['idx3'] = expected['idx3'].apply(lambda d: pd.Timestamp(d, tz='Europe/Paris'))
+            assert_frame_equal(df.reset_index(), expected)
+
     def test_set_index_period(self):
         # GH 6631
         df = DataFrame(np.random.random(6))