lisa-lab/pylearn — Issue #57: three-pronged dataset serialization proposal
Author: dwf
Created: 2012-03-01 19:08:22+00:00
State: open

This proposal aims to address the problem of serializing datasets that are referenced by monitors or other objects. It is often desirable to not serialize e.g. a monitoring dataset when you serialize a model.

The proposal addresses datasets that have had preprocessing applied to them that destroys the original data, and allow *either*

* serializing the end state of a preprocessed dataset pipeline (for maximal reproducibility, saving computational time, limited resources on the machine performing the deserialization; i.e. the dataset has been preprocessed on a machine with more resources than the one on which the model learning will take place)
* serializing *just a description* of how to reconstruct the entire preprocessed data (and apply appropriate preprocessing) either
* at deserialization time
* at first access

Deserializing at first access (a.k.a. "lazy-loading") has the following advantages:

* Speed of access: You don't need to reload and re-preprocess the actual data if all you need is the dataset's view converter (i.e. for visualizing weights)
* Deserialization of an entire hierarchy of objects won't fail due to missing external disk file dependencies (NPY files, HDF5 files, etc.). This could also be accomplished by simply making a "dataless" instance of the dataset class -- lazy-loading implies that this "dataless" state is the default when deserializing-from-description, and the object is mutated to a "data-ful" state the first time actual data is actually needed.

Implementation
---------------------

First, `apply_preprocessor` should keep a record of preprocessor objects applied, and in what order. This will be serialized along with the dataset object.

The proposal further augments two boolean flags:

* `serialize_final_state` -- A flag, by default set to False, which determines the behaviour at serialization time. If False, serialize on
* `initialized` - Is False if the data has not been loaded from disk or preprocessed, True if the data is usable.

Methods that access the data, such as `iterator()`, `get_design_matrix()`, etc. will call a method like

```python
if not self.initiaized:
self.initialize()
```

At serialization time, if `serialize_final_state` is False, `__setstate__` removes the necessary objects and sets the `initialized` flag to False.

Advanced pickle features like `__getnewargs__` may make things simpler on the deserialization side.