Repository: pandas-dev/pandas

Issue:
BUG: select_column not preserving a UTC timezone
I was having issues with lost tz-info when retrieving a DatetimeIndex from an HDF store using `store.select_column('data', 'index')`.  I was able to track down the issue to `tseries/index.py` in the `Index._to_embed` method. The issue is 

``` python
    def _to_embed(self, keep_tz=False):
        """ return an array repr of this object, potentially casting to object """
        if keep_tz and self.tz is not None and str(self.tz) != 'UTC':                                                                                                                                                                       
             return self.asobject.values
        return self.values
```

It looks like it explicitly rejects UTC timezones.  Is there a good reason for this?

The below code reproduces the problem for me.

``` python
import pandas as pd

drange = pd.date_range('2014-07-07 00:00:00', '2014-07-07 03:00:00', freq='1h')
drange_utc = drange.tz_localize('UTC')
drange_mst = drange.tz_localize('MST')

print drange._to_embed(keep_tz=True)
print drange_utc._to_embed(keep_tz=True)
print drange_mst._to_embed(keep_tz=True)
```

I'm using python 2.7.6 with the following packages:

```
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-431.17.1.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.14.1
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.0.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.2
bottleneck: None
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.3.1
openpyxl: 1.8.6
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.3.5
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None
```



Discussion:
this is an internal method

show a complete example of what u r actually doing 

Making a Series of a DatetimeIndex also illustrates the problem.

``` python
import pandas as pd

drange = pd.date_range('2014-07-07 00:00:00', '2014-07-07 03:00:00', freq='1h')
drange_utc = drange.tz_localize('UTC')
drange_mst = drange.tz_localize('MST')

print pd.Series(drange)
print pd.Series(drange_utc)
print pd.Series(drange_mst)
```

And an example of my original problem getting the index from an HDF store

``` python
import pandas as pd
import numpy as np

drange = pd.date_range('2014-07-07 00:00:00', '2014-07-07 03:00:00', freq='1h')
drange_utc = drange.tz_localize('UTC')
drange_mst = drange.tz_localize('MST')

data = np.ones((drange.size, 3))
df_utc = pd.DataFrame(data, index=drange_utc)
df_mst = pd.DataFrame(data, index=drange_mst)

store_path = 'timezone_test.h5'
with pd.get_store(store_path) as store:
    store.put('utc', df_utc ,'table')
    store.put('mst', df_mst, 'table')

with pd.get_store(store_path) as store:
    print store.select_column('utc', 'index')
    print store.select_column('mst', 'index')
```

@alorenzo175  By definition a `datetime64[ns]` is UTC (though it technically doesn't have a tz attached). I guess technically you want to keep a tz-attached (but with UTC) series. You _can_ do this, but will pretty much have to specify a `dtype=object` a lot.

Because a UTC series is de-facto equivalent to a plain-old `datetime64[ns]` series and is indistiguishable. (However, the index itself IS distriguishable).

I guess this _could_ be a bit confusing. A possible work-around is to store the 'UTC' data as 'GMT', which will be treated as a regular timezone.

selecting this as a full table DOES seem to work though (e.g. `store.select('utc')`), so maybe their is a bug in `select_column` as its not doing the proper conversion (when its shoved into the Series).

ok, will mark that as a bug.

interested in doing a pull-request to fix (it will be in `io/pytables/Table/read_column`)?

After messing around a little with the `Index._to_embed` function, I agree that this should just be a fix for pytables.

I'll try to make a fix but this will be my virgin PR.


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -62,6 +62,18 @@ def _ensure_encoding(encoding):
             encoding = _default_encoding
     return encoding
 
+def _set_tz(values, tz, preserve_UTC=False):
+    """ set the timezone if values are an Index """
+    if tz is not None and isinstance(values, Index):
+        tz = _ensure_decoded(tz)
+        if values.tz is None:
+            values = values.tz_localize('UTC').tz_convert(tz)
+        if preserve_UTC:
+            if tslib.get_timezone(tz) == 'UTC':
+                values = list(values)
+
+    return values
+
 
 Term = Expr
 
@@ -1464,11 +1476,7 @@ def convert(self, values, nan_rep, encoding):
                 kwargs['freq'] = None
             self.values = Index(values, **kwargs)
 
-        # set the timezone if indicated
-        # we stored in utc, so reverse to local timezone
-        if self.tz is not None:
-            self.values = self.values.tz_localize(
-                'UTC').tz_convert(_ensure_decoded(self.tz))
+        self.values = _set_tz(self.values, self.tz)
 
         return self
 
@@ -3443,8 +3451,11 @@ def read_column(self, column, where=None, start=None, stop=None, **kwargs):
                 # column must be an indexable or a data column
                 c = getattr(self.table.cols, column)
                 a.set_info(self.info)
-                return Series(a.convert(c[start:stop], nan_rep=self.nan_rep,
-                                        encoding=self.encoding).take_data())
+                return Series(_set_tz(a.convert(c[start:stop], 
+                                                nan_rep=self.nan_rep,
+                                                encoding=self.encoding
+                                                ).take_data(), 
+                                      a.tz, True))
 
         raise KeyError("column [%s] not found in the table" % column)
 


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -4299,6 +4299,38 @@ def test_tseries_indices_frame(self):
             self.assertEqual(type(result.index), type(df.index))
             self.assertEqual(result.index.freq, df.index.freq)
 
+    def test_tseries_select_index_column(self):
+        # GH7777
+        # selecting a UTC datetimeindex column did
+        # not preserve UTC tzinfo set before storing
+
+        # check that no tz still works
+        rng = date_range('1/1/2000', '1/30/2000')
+        frame = DataFrame(np.random.randn(len(rng), 4), index=rng)
+        
+        with ensure_clean_store(self.path) as store:
+            store.append('frame', frame)
+            result = store.select_column('frame', 'index')
+            self.assertEqual(rng.tz, DatetimeIndex(result.values).tz)
+
+        # check utc
+        rng = date_range('1/1/2000', '1/30/2000', tz='UTC')
+        frame = DataFrame(np.random.randn(len(rng), 4), index=rng)
+        
+        with ensure_clean_store(self.path) as store:
+            store.append('frame', frame)
+            result = store.select_column('frame', 'index')
+            self.assertEqual(rng.tz, DatetimeIndex(result.values).tz)
+
+        # double check non-utc
+        rng = date_range('1/1/2000', '1/30/2000', tz='US/Eastern')
+        frame = DataFrame(np.random.randn(len(rng), 4), index=rng)
+
+        with ensure_clean_store(self.path) as store:
+            store.append('frame', frame)
+            result = store.select_column('frame', 'index')
+            self.assertEqual(rng.tz, DatetimeIndex(result.values).tz)
+
     def test_unicode_index(self):
 
         unicode_values = [u('\u03c3'), u('\u03c3\u03c3')]