Repository: pandas-dev/pandas

Issue:
compat pickles don't have '_id'
So I'm running into an issue with creating a `Panel` out of legacy `DataFrame` pickles. The issue is that `_id` is set to `None` and the current `Index.is_` logic treats any two Indexes with unset `_id` as the same.

I thought the issue was with the unpickling. But it looks like there was an attempt to address this

``` python
def is_(self, other):
    return self._id is getattr(other, '_id', Ellipsis)
```

I'm guessing whoever thought that `getattr(other, '_id', Ellipsis)` would return `Ellipsis` if the `_id` was not set.

So I'm seeing:
1. Make unpickling make a new `_id` if it's not found in state
2. Make any `Index.is_` always return `False` if either one has an unset id.

Maybe both?



Discussion:
you will have 2 provide a concrete example here of where this breaks and sample pickles

@dalejung ???? releasing 0.15.0 rc soon, If this is a problem would like to address before then.

can you should some code to reproduce?

A self contained example is here:
http://nbviewer.ipython.org/gist/dalejung/e6d7fd95e2bef2cda81b
I attached the pickle files even though you can generate them yourself via `dump.py`. 

The important part is pickling a legacy pandas object, the pickle api doesn't matter as `_id` isn't explicitly pickled either way and just comes along for the ride. 

Man, it'd be nice if github could show ipython notebooks inline.

@dalejung your 0.12 unpickle just fine (for me)
the 0.14.1. have some weird error (unpickle protocol 4), so not sure what you are doing
I can unpickle 0.14.1 just fine.

So still unclear what actually doesn't work

these all read for me in py3.4 with the lastest version of master.

so if you have more info pls post.

@jreback not sure we're on the same page. The issue isn't reading the pickle. The `0.12.0` pickles don't have the `_id` on their Indexes because that did not exist back then. 

This is an issue because if

``` python
assert s1.index._id is None
assert s2.index._id is None

# then
s1.index.is_(s2.index) is True
```

So for something like this:

``` python
# indexes are not identical
assert s1.index.size != s2.index.size
df = pd.DataFrame({'s1' : s1, 's2' : s2}) # erros

/python/externals/pandas/pandas/core/internals.py in _stack_arrays(tuples, dtype)
   3697     stacked = np.empty(shape, dtype=dtype)
   3698     for i, arr in enumerate(arrays):
-> 3699         stacked[i] = _asarray_compat(arr)
   3700 
   3701     return stacked, placement

ValueError: could not broadcast input array from shape (27) into shape (30)
```

You get a shape mismatch because pandas doesn't union the two indexes because it already thinks they are equal. Normally it would've reindexed the data to the same shape. 


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -843,6 +843,7 @@ def __setstate__(self, state):
                 np.ndarray.__setstate__(data, state)
 
             self._data = data
+            self._reset_identity()
         else:
             raise Exception("invalid pickle state")
     _unpickle_compat = __setstate__
@@ -3349,6 +3350,7 @@ def __setstate__(self, state):
         self._set_names(names)
         self.sortorder = sortorder
         self._verify_integrity()
+        self._reset_identity()
 
     def __getitem__(self, key):
         if np.isscalar(key):
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -630,6 +630,7 @@ def __setstate__(self, state):
                 np.ndarray.__setstate__(data, state)
 
             self._data = data
+            self._reset_identity()
 
         else:
             raise Exception("invalid pickle state")


Test patch:
diff --git a/pandas/tests/data/s1-0.12.0.pkl b/pandas/tests/data/s1-0.12.0.pkl
new file mode 100644
Binary files /dev/null and b/pandas/tests/data/s1-0.12.0.pkl differ
diff --git a/pandas/tests/data/s2-0.12.0.pkl b/pandas/tests/data/s2-0.12.0.pkl
new file mode 100644
Binary files /dev/null and b/pandas/tests/data/s2-0.12.0.pkl differ
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -270,6 +270,15 @@ def test_view(self):
         i_view = i.view()
         self.assertEqual(i_view.name, 'Foo')
 
+    def test_legacy_pickle_identity(self):
+
+        # GH 8431
+        pth = tm.get_data_path()
+        s1 = pd.read_pickle(os.path.join(pth,'s1-0.12.0.pkl'))
+        s2 = pd.read_pickle(os.path.join(pth,'s2-0.12.0.pkl'))
+        self.assertFalse(s1.index.identical(s2.index))
+        self.assertFalse(s1.index.equals(s2.index))
+
     def test_astype(self):
         casted = self.intIndex.astype('i8')
 
@@ -532,7 +541,7 @@ def test_intersection(self):
         result3 = idx1.intersection(idx3)
         self.assertTrue(tm.equalContents(result3, expected3))
         self.assertEqual(result3.name, expected3.name)
-        
+
         # non-monotonic non-unique
         idx1 = Index(['A','B','A','C'])
         idx2 = Index(['B','D'])