Repository: pandas-dev/pandas

Issue:
AttributeError: 'Hour' object has no attribute 'normalize' when masking a time series
Hi All,

I have a strange bug when I want to mask values of a time series (that I get from a pickled Panel).

I have the following Series:

``` python

In[30]: ts=pd.read_pickle('df_issue.pkl')['wind_speed']

In [31]:ts
Out[31]: 
2013-12-31 16:00:00         NaN
2013-12-31 17:00:00         NaN
2013-12-31 18:00:00    9.845031
2013-12-31 19:00:00         NaN
2013-12-31 20:00:00         NaN
2013-12-31 21:00:00         NaN
2013-12-31 22:00:00         NaN
2013-12-31 23:00:00         NaN
Freq: H, Name: wind_speed, dtype: float64
```

And I have an exception when I try to mask it:

``` python
In [32]: ts<0.
Traceback (most recent call last):

  File "<ipython-input-32-534147d368f7>", line 1, in <module>
    ts<0.

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 585, in wrapper
    res[mask] = masker

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 637, in __setitem__
    self.where(~key, value, inplace=True)

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 3238, in where
    inplace=True)

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 2219, in putmask
    return self.apply('putmask', **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 2185, in apply
    b_items = self.items[b.mgr_locs.indexer]

  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.py", line 1387, in __getitem__
    new_offset = key.step * self.offset

  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/offsets.py", line 265, in __rmul__
    return self.__mul__(someInt)

  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/offsets.py", line 262, in __mul__
    return self.__class__(n=someInt * self.n, normalize=self.normalize, **self.kwds)

AttributeError: 'Hour' object has no attribute 'normalize'
```

If I make the same Series manually, it is working:

``` python

ts1=pd.Series(data=[np.nan,np.nan,9.845031]+[np.nan]*5,index=pd.date_range('2013-12-31 16:00:00',periods=8,freq='H'))

In [36]: ts1
Out[36]: 
2013-12-31 16:00:00         NaN
2013-12-31 17:00:00         NaN
2013-12-31 18:00:00    9.845031
2013-12-31 19:00:00         NaN
2013-12-31 20:00:00         NaN
2013-12-31 21:00:00         NaN
2013-12-31 22:00:00         NaN
2013-12-31 23:00:00         NaN
Freq: H, dtype: float64

In [37]: ts1<0.
Out[37]: 
2013-12-31 16:00:00    False
2013-12-31 17:00:00    False
2013-12-31 18:00:00    False
2013-12-31 19:00:00    False
2013-12-31 20:00:00    False
2013-12-31 21:00:00    False
2013-12-31 22:00:00    False
2013-12-31 23:00:00    False
Freq: H, dtype: bool
```

I really don't understand why...
You can find the pickle file here: http://we.tl/lrsFvmanVl

Thanks,
Greg

Here is the show_version output:
## INSTALLED VERSIONS

commit: None
python: 2.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.8.0-37-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.14.1
nose: 1.3.0
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.13.3
statsmodels: 0.6.0.dev-Unknown
IPython: 1.1.0
sphinx: 1.1.3
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.4
bottleneck: 0.6.0
tables: 3.1.1
numexpr: 2.0.1
matplotlib: 1.3.1
openpyxl: 1.7.0
xlrd: 0.9.2
xlwt: 0.7.2
xlsxwriter: None
lxml: 3.0.0
bs4: 4.3.2
html5lib: None
httplib2: 0.7.2
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None



Discussion:
when/how did you create the pickle? e.g. in a prior version, which one?

df_pickle.pkl is created using pandas 0.14.1. However, the raw data in the pickle file are results of calculations done on data originally stored in HDF5 (created with pandas 0.12).

can you show the original creation of the `df`?

Not really because this is an output of several steps impossible to show here...

FYI, I just downgraded to pandas 0.13.1 and the issue above doesn't appear anymore.

Do you think all this could be due to the raw data loaded from HDF5 generated with pandas 0.12?

prob has to do with the pickling of the `Hour` object from the HDF5. Its an older version. I think you could just do something like

`df.index = df.index.copy()` in 0.14.1 and then it should work.

`df.index = df.index.copy()` doesn't work but doing the following works

```
new_df = pd.DataFrame(df.values,columns=df.columns, index=[i for i in df.index])

new_df['variable'] <0.
```

It is a bit slow due to `[i for i in df.index]` though (the df.index can be quite large). Do you have a faster alternative that may work as well?

you can try this:

`df.index = Index(df.index,freq=df.index.freqstr` to recreate a new index, and the key is to create  _new_ frequency (rather than using the existing)

It worked! Thanks a lot!

gr8. this is not a bug per-se, more of a slightly change in the compat of frequencies in terms of backward-compat for pickle. Not a whole lot I think can do about this.

cc @sinhrks, maybe change these to `normalize=getattr(self,'normalize',None)` ?


Resolving patch:
diff --git a/pandas/tseries/offsets.py b/pandas/tseries/offsets.py
--- a/pandas/tseries/offsets.py
+++ b/pandas/tseries/offsets.py
@@ -130,6 +130,9 @@ def __add__(date):
     _cacheable = False
     _normalize_cache = True
 
+    # default for prior pickles
+    normalize = False
+
     def __init__(self, n=1, normalize=False, **kwds):
         self.n = int(n)
         self.normalize = normalize
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -578,6 +578,7 @@ def pxd(name):
                                   'tests/data/legacy_pickle/0.11.0/*.pickle',
                                   'tests/data/legacy_pickle/0.12.0/*.pickle',
                                   'tests/data/legacy_pickle/0.13.0/*.pickle',
+                                  'tests/data/legacy_pickle/0.14.0/*.pickle',
                                   'tests/data/*.csv',
                                   'tests/data/*.dta',
                                   'tests/data/*.txt',


Test patch:
diff --git a/pandas/io/tests/test_pickle.py b/pandas/io/tests/test_pickle.py
--- a/pandas/io/tests/test_pickle.py
+++ b/pandas/io/tests/test_pickle.py
@@ -48,9 +48,12 @@ def compare(self, vf):
         # py3 compat when reading py2 pickle
         try:
             data = pandas.read_pickle(vf)
-        except (ValueError) as detail:
-            # trying to read a py3 pickle in py2
-            return
+        except (ValueError) as e:
+            if 'unsupported pickle protocol:' in str(e):
+                # trying to read a py3 pickle in py2
+                return
+            else:
+                raise
 
         for typ, dv in data.items():
             for dt, result in dv.items():
@@ -60,6 +63,7 @@ def compare(self, vf):
                     continue
 
                 self.compare_element(typ, result, expected)
+        return data
 
     def read_pickles(self, version):
         if not is_little_endian():
@@ -68,7 +72,14 @@ def read_pickles(self, version):
         pth = tm.get_data_path('legacy_pickle/{0}'.format(str(version)))
         for f in os.listdir(pth):
             vf = os.path.join(pth,f)
-            self.compare(vf)
+            data = self.compare(vf)
+
+            if data is None:
+                continue
+
+            if 'series' in data:
+                if 'ts' in data['series']:
+                    self._validate_timeseries(data['series']['ts'], self.data['series']['ts'])
 
     def test_read_pickles_0_10_1(self):
         self.read_pickles('0.10.1')
@@ -82,6 +93,9 @@ def test_read_pickles_0_12_0(self):
     def test_read_pickles_0_13_0(self):
         self.read_pickles('0.13.0')
 
+    def test_read_pickles_0_14_0(self):
+        self.read_pickles('0.14.0')
+
     def test_round_trip_current(self):
         for typ, dv in self.data.items():
 
@@ -94,6 +108,14 @@ def test_round_trip_current(self):
                     result = pd.read_pickle(path)
                     self.compare_element(typ, result, expected)
 
+    def _validate_timeseries(self, pickled, current):
+        # GH 7748
+        tm.assert_series_equal(pickled, current)
+        self.assertEqual(pickled.index.freq, current.index.freq)
+        self.assertEqual(pickled.index.freq.normalize, False)
+        self.assert_numpy_array_equal(pickled > 0, current > 0)
+
+
 if __name__ == '__main__':
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],