BUG: pickle failing on FrozenList, when using MultiIndex (GH4788)
State: closed

closes #4788


Labels:

Comments:
@jreback did you see my comment? `FrozenList` never made it into a release, so we can go _way_ simpler and just define a `__reduce__` method...

yep...I just fixed it...much simpler that way....already done...now just going to regenerate the test pickles again....

great :)

and maybe we should leave a note to self to regenerate the legacy pickle right before we actually make the next release?

maybe....I only need to do it if I am actually changing the generate_legacy_pickles.py (which I did :)

in theory it should be stable (but in this case the writing of the pickle actually is different)

hmm....py3 still give me an issue even with reduce.....let me see

@jreback 

Might be easier to just change this:

``` python
    def __reduce__(self):
        """Necessary for making this object picklable"""
        object_state = list(np.ndarray.__reduce__(self))
        subclass_state = (self.levels, self.labels, self.sortorder, self.names)
        object_state[2] = (object_state[2], subclass_state)
        return tuple(object_state)
```

to:

``` python
    def __reduce__(self):
        """Necessary for making this object picklable"""
        object_state = list(np.ndarray.__reduce__(self))
        subclass_state = (list(self.levels), list(self.labels), self.sortorder, list(self.names))
        object_state[2] = (object_state[2], subclass_state)
        return tuple(object_state)
```

and just not deal with having `FrozenList` in the pickle at all...

ok...will try that...set_state looks good then? (in core/index.py)

let me look. I'm having trouble testing this locally... I get this error: `ValueError: unsupported pickle protocol: 3`

actually, it's not quite right...one sec.

@jtratner ok...I updated with my hack it works now (I put in the changes to reduce by no dice)...if you can get it to work w/o the hacks great.....lmk.....I will add 32-bit stuff test pickles as welll

will do -- the pickle error was just a weird thing on my end...

@jtratner alright current version passes travis!

I have a good selection of files....0.12: 2.7/3 for 64-bit linux/AMD, 0.12-dev: 2.7/3.2 for 32-bit/64-bit linux, 0.11 only 3.3

if you have any more arch..pls add, just run: `pandas/io/tests/generate_legacy.py`, but you have to copy it to the location or it will pick up the default system python (e.g. a build location)...then put the file in the appropriate folder and push...

@jreback when I reverted just the `compat/pickle_compat.py` part of this commit, it still passed Travis... do we need the hack? (It worked locally for me too...) https://travis-ci.org/jtratner/pandas/builds/11175350

hmm
if that works great
maybe I had a test file that had a pickles frozen list but cleaned it up

let me try that 

this still does leave `FrozenNDArray` in there for the labels. Is that problematic? Otherwise we could have `__reduce__()` generate list of lists for labels and levels...

when I take out the hack, py3 fails on the 0.12-dev files (e.g. the current ones)....

@jreback you know - I guess I thought that it just ran through pickles for the current version no matter what. Do I need to generate the data first?

@jtratner no...all the files are there. it runs thru all files in each of the directories, then compares to the data that is in the generate_legacy_pickles (which in theory should work as far back as one would want to go, e.g. its not dependent on compat type stuff for example)

@jreback I mean, does it test that round-tripping with a pickle works in the current version? (i.e., it generates pickles for current version into a temporary file, runs them and then deletes them)

i.e.,checks that no one checks in changes like the `FrozenList` change that causes pickles to stop working...

@jtratner no doesn't round trip test per se (maybe could add that)....maybe that is better for the current version I suppose

It checks all of the static files (which  I had generated for the current version)

@jtratner I pushed a round-trip test (which passes), but fails if I remove the hack in pickle_compat....so something is still happenign with frozen list.....maybe it needs a **setstate** ?

@jreback yeah, maybe. Sorry about the confusion there - i'll go figure it out now.

cool...lmk

@jreback I can't reproduce your issue on 3.2, 3.3 or 2.7. In all of them, I can roundtrip a dataframe with a MultiIndex using `to_pickle` and `read_pickle` without any issue. For that `__reduce__()` to work, it means breaking any pickles from the time 033a9329 was committed through to today (about the past month). But I don't think any of that was in an official release so that should be fine.  Are you sure that you are removing all of the 0.12-dev pickles?

For example, if you make a pickle off of current master, then tried to use this, it would fail.

all is the files in 0.11/0.12/0.13 are brand new I regenerated them as part of this pr

I just call the geneate_legacy_pickles.py script with whatever version 

here's the passing Travis - https://travis-ci.org/jtratner/pandas/builds/11177267

hmm maybe I have an old pyc hanging out (that has the frozen list inside)

ok
I'll take it out
or u can

so u need set state in frozen list?

nope, you don't need set_state. it just calls the first argument with the second argument. (I think).

So I'm pretty sure it basically it does (when reduce returns a 2-tuple):

``` python
f = FrozenList([1, 2, 3, 4])
reduced = f.__reduce__() # (FrozenList, ([1, 2, 3, 4],))
reduced[0](*reduced[1])
```