Repository: pandas-dev/pandas

Issue:
PyTables dates don't work when you switch to a different time zone
Create a dataframe with a date (not datetime) column as the index. Save that dataframe to the HDFStore. Now open that same file on a computer in a time zone that's behind you. (IE, write the file in New York and then read the file from Dallas). Result: all the dates are off by one.
Here is a code snippet to demonstrate the problem, and also a proposed fix to the pandas code. (But someone more qualified than me can review this fix).

To repro the problem:
# write a file in New York time zone

import pandas
import datetime

def writeToFile(fileName) :
    dates = [ datetime.date.today() , datetime.date.today(), datetime.date.today() ]
    numbers = [ 1, 2, 3 ]
    p = pandas.DataFrame({ "date" : dates, "number": numbers}, columns = ['date', 'number'])
    p.set_index('date', inplace=True)
    store = pandas.HDFStore(fileName)
    store['obj1'] = p
    store.close()
# Then go to a dallas time zone and read that file:

def readFromFile(fileName) :
    store = pandas.HDFStore(fileName)
    return store['obj1']
# Notice that the dates on the dataframe now show YESTERDAY when they ought to show today.

Proposed fix in pandas.io.pytables.py:

def _convert_index(index):
   ...
   elif inferred_type == 'date':
        #OLD LINE: CHANGE THIS LINE
        #converted = np.array([time.mktime(v.timetuple()) for v in values],
        #                     dtype=np.int32)
        #NEW LINE:
        converted = np.array([v.toordinal() for v in values],  dtype=np.int32)        

and then also change:

def _unconvert_index(data, kind):
    ...
    elif kind == 'date':
        # here we'll try from ordinal first.  If the date was saved with the old
        # mktime mechanism it'll throw an exception as it'll be out of bounds.
        # in those cases we'll convert using the old method (with the bug!)  
        try: 
            index = np.array([date.fromordinal(v) for v in data], dtype=object)
        except ValueError:
            index = np.array([date.fromtimestamp(v) for v in data], dtype=object)



Discussion:
this is not well tested
nor are dates really well supported (datetimes are really the way to store as can represent as int64 internally)

but will take a look

thanks

On Feb 12, 2013, at 1:29 PM, tavistmorph notifications@github.com wrote:

> Create a dataframe with a date (not datetime) column as the index. Save that dataframe to the HDFStore. Now open that same file on a computer in a time zone that's behind you. (IE, write the file in New York and then read the file from Dallas). Result: all the dates are off by one.
> Here is a code snippet to demonstrate the problem, and also a proposed fix to the pandas code. (But someone more qualified than me can review this fix).
> 
> To repro the problem:
> 
> #write a file in New York time zone
> import pandas
> import datetime
> 
> def writeToFile(fileName) :
> dates = [ datetime.date.today() , datetime.date.today(), datetime.date.today() ]
> numbers = [ 1, 2, 3 ]
> p = pandas.DataFrame({ "date" : dates, "number": numbers}, columns = ['date', 'number'])
> p.set_index('date', inplace=True)
> store = pandas.HDFStore(fileName)
> store['obj1'] = p
> store.close()
> 
> #Then go to a dallas time zone and read that file:
> 
> def readFromFile(fileName) :
> store = pandas.HDFStore(fileName)
> return store['obj1']
> 
> #Notice that the dates on the dataframe now show YESTERDAY when they ought to show today.
> 
> Proposed fix in pandas.io.pytables.py:
> 
> def _convert_index(index):
> ...
> elif inferred_type == 'date':
> #OLD LINE: CHANGE THIS LINE
> #converted = np.array([time.mktime(v.timetuple()) for v in values],
> 
> # dtype=np.int32)
> 
> #NEW LINE:
> converted = np.array([v.toordinal() for v in values], dtype=np.int32)
> 
> and then also change:
> 
> def _unconvert_index(data, kind):
> ...
> elif kind == 'date':
> 
> # here we'll try from ordinal first. If the date was saved with the old
> 
> # mktime mechanism it'll throw an exception as it'll be out of bounds.
> 
> # in those cases we'll convert using the old method (with the bug!)
> 
> try: 
> index = np.array([date.fromordinal(v) for v in data], dtype=object)
> except ValueError:
> index = np.array([date.fromtimestamp(v) for v in data], dtype=object)
> 
> —
> Reply to this email directly or view it on GitHub.

this solution looks reasonable

can you do a PR with this (and some tests?)

not sure how to 'fake' the timezone....prob have to do it explicity..

Also if the toordinal returns an int? then the column should be Int32 
can they live in 32?

and then don't actually need the try: except: on the unconvert, instead you can check
the column type (you already now its a 'date', so if its a Time32 then
old method, Int32 new method)

Not sure what a PR is (I assume that's a Git thing, and I'm new to Git). But yes, we have tested the fix I described above significantly in our environment with our real-world dataframes and it's working fine for us. 

a pull-request
you can essentially contribute a patch to thhe main code-base and  see that nothing else breaks
https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md

Action here?

pushing till 0.12, have to store extra data for the column, so need to thing how general this needs to be (see #2391, @scottkidder wanted to work on this)

this is closed by #3531

This issue is not fixed in pandas 0.12.0.  I just reproduced it by following the test case outlined in the description.

this particular issue is untestable so closed. The store correctly stores/retrieves timezone aware datetimes in 0.12. What exactly are you reproducing?

I am using dates (not datetimes).  I stored dates in a dataframe index and then saved to an hdf store in Eastern timezone, and then retrieved it in Central timezone and got (date - 1) in the index.  What exactly do you mean by this issue being untestable?

`datetime.dates` are not supported for this issue, only `datetime.datetimes`. and in general `datetime.dates` are not really supported by pandas for many reasons.

what's untestable is resetting the default time zone on the computer (well it IS testable, but the datetimes work fine) so its not an issue. you are having a different issue, in that `datetime.date` are only really supported somewhat for historical reasons

Well, at the end of the day, datetime.date is "supported" because there is an "if" statement checking for it and doing something based on that (something that's causing a problem when using the hdf store on two different timezones).     Is there any harm in making the change suggested by the original submitter?

can you create a self-contained example for testing? (if so, then no problem)

https://www.wakari.io/sharing/bundle/johnv/pandas_issue_2852 -  shared an ipython notebook on wakari. Let me know if you have problems accessing it (code won't work on Windows because of missing tzset).  I'll repeat the code below just in case:

``` python
import datetime
import os
import time
import pandas as pd

def setTZ(tz):
    os.environ['TZ']=tz
    time.tzset()

setTZ('EST5EDT')
today = datetime.date(2013,9,10)
df = pd.DataFrame([1,2,3], index = [today, today, today])
print df
filename = 'test.hdf5'
store = pd.HDFStore(filename)
store['obj1'] = df
store.close()

setTZ('CST6CDT')
store = pd.HDFStore(filename)
read_df = store['obj1']
store.close()
print read_df

df.index[0]==read_df.index[0]
```


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -1740,8 +1740,12 @@ def convert(self, values, nan_rep, encoding):
             elif dtype == u('timedelta64'):
                 self.data = np.asarray(self.data, dtype='m8[ns]')
             elif dtype == u('date'):
-                self.data = np.array(
-                    [date.fromtimestamp(v) for v in self.data], dtype=object)
+                try:
+                    self.data = np.array(
+                        [date.fromordinal(v) for v in data], dtype=object)
+                except (ValueError):
+                    self.data = np.array(
+                        [date.fromtimestamp(v) for v in self.data], dtype=object)
             elif dtype == u('datetime'):
                 self.data = np.array(
                     [datetime.fromtimestamp(v) for v in self.data],
@@ -3769,7 +3773,7 @@ def _convert_index(index, encoding=None):
         return IndexCol(converted, 'datetime', _tables().Time64Col(),
                         index_name=index_name)
     elif inferred_type == 'date':
-        converted = np.array([time.mktime(v.timetuple()) for v in values],
+        converted = np.array([v.toordinal() for v in values],
                              dtype=np.int32)
         return IndexCol(converted, 'date', _tables().Time32Col(),
                         index_name=index_name)
@@ -3809,7 +3813,12 @@ def _unconvert_index(data, kind, encoding=None):
         index = np.array([datetime.fromtimestamp(v) for v in data],
                          dtype=object)
     elif kind == u('date'):
-        index = np.array([date.fromtimestamp(v) for v in data], dtype=object)
+        try:
+            index = np.array(
+                [date.fromordinal(v) for v in data], dtype=object)
+        except (ValueError):
+            index = np.array(
+                [date.fromtimestamp(v) for v in self.data], dtype=object)
     elif kind in (u('integer'), u('float')):
         index = np.array(data)
     elif kind in (u('string')):
@@ -4096,10 +4105,12 @@ def stringify(value):
         elif kind == u('timedelta64') or kind == u('timedelta'):
             v = _coerce_scalar_to_timedelta_type(v,unit='s').item()
             return TermValue(int(v), v, kind)
-        elif (isinstance(v, datetime) or hasattr(v, 'timetuple')
-              or kind == u('date')):
+        elif (isinstance(v, datetime) or hasattr(v, 'timetuple')):
             v = time.mktime(v.timetuple())
             return TermValue(v, Timestamp(v), kind)
+        elif kind == u('date'):
+            v = v.toordinal()
+            return TermValue(v, Timestamp.fromordinal(v), kind)
         elif kind == u('integer'):
             v = int(float(v))
             return TermValue(v, v, kind)


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -1799,6 +1799,47 @@ def compare(a,b):
             result = store.select('df')
             assert_frame_equal(result,df)
 
+    def test_store_timezone(self):
+        # GH2852
+        # issue storing datetime.date with a timezone as it resets when read back in a new timezone
+
+        import platform
+        if platform.system() == "Windows":
+            raise nose.SkipTest("timezone setting not supported on windows")
+
+        import datetime
+        import time
+        import os
+
+        orig_tz = os.environ.get('TZ')
+
+        def setTZ(tz):
+            if tz is None:
+                try:
+                    del os.environ['TZ']
+                except:
+                    pass
+            else:
+                os.environ['TZ']=tz
+                time.tzset()
+
+        try:
+
+            with ensure_clean(self.path) as store:
+
+                setTZ('EST5EDT')
+                today = datetime.date(2013,9,10)
+                df = DataFrame([1,2,3], index = [today, today, today])
+                store['obj1'] = df
+
+                setTZ('CST6CDT')
+                result = store['obj1']
+
+                assert_frame_equal(result, df)
+
+        finally:
+            setTZ(orig_tz)
+
     def test_append_with_timedelta(self):
         if _np_version_under1p7:
             raise nose.SkipTest("requires numpy >= 1.7")