lisa-lab/pylearn — Issue #31: More efficient/fault-tolerant single-file serialization
Author: dwf
Created: 2012-02-10 15:21:17+00:00
State: open

joblib.save is much faster than normal pickle and lets you recover your parameters in the case that your pickles go stale, but has the side effect of creating several files on disk instead of one, which may be undesirable for some.

Looking at a way to remedy this with a custom pickler, it seems possible but a bit trickier. From an email exchange with Ian:

> The trouble is that joblib currently relies on the ability to write out
> arrays to separate files asynchronously, during the (recursive) pickling
> procedure. If you specify compression parameters then it will do this,
>
> After some thought, the simplest thing to do to achieve the same speed and
> fault tolerance with a single file would be to have the Pickler object store
> a list of arrays, replacing them with dummy object as it goes, then use the
> pickler to pickle this munged representation to a StringIO/BytesIO object.
>
> The wrapping "save" function
> would then open a file for writing, loop through
> this list of arrays and repeatedly numpy.save on the same open file
> descriptor. When it was done, it would call cPickle.dump(f, False) or
> something to act as a sentinel, indicating that there are no more arrays.
>
> np.load() can handle pickles as well as npy data, so unarchiving would look
> like this:
>
```python
# f is a pre-existing open file descriptor
arrays = []
while True:
arr = numpy.load(f)
if arr is False:
break
arrays.append(arr)
```
> my_custom_pickler.load(f, arrays)
>
> Now my_custom_pickler has all the information it needs to unpickle while
> subbing in the arrays where appropriate (the dummy objects can also keep
> track of the index in the list of arrays for which they are a stand-in).
>
> A two-file solution is also a possibility, using only one separate file.
>
> Alas, this is a bigger project than I thought, so I'm going to defer actually
> implementing this until after this round of deadlines. For now, if you want
> to take advantage of joblib's fast saves/fault tolerance and don't mind
> multiple files, you can use the file suffix ".joblib". An eventual
> single-file format will probably have its own custom suffix as well, ".pl2"
> or something.