NeuralEnsemble/python-neo — pull request #27 by rproepp (2013-07-15 14:12:35+00:00)
Title: Lazy cascade

Note: I have created this as a pull request to start a discussion if we want to include these features in the way I propose. I believe that these details are not of interest to most of the mailing list members, but we could also discuss it there.

This pull request includes an implementation of two improvements to lazy loading in Neo which enable comfortable use of files larger than the main memory and shorter initial load times. The implementations are currently only for the Hdf5 IO, but they are not API breaking and I propose a standard way in which they could be implemented in other IO classes as well.

Details on the two enhancements:

1. A load_lazy_object() method, as I had proposed on the mailing list (https://groups.google.com/d/msg/neuralensemble/ZRdssJnC43g/qFply2qJMpMJ). In case of the HDF5 IO, it is just one line. The method takes a lazily loaded object and returns the corresponding fully loaded object. It does not set any links of the newly loaded object (e.g. the segment attribute).

Usage example:
```
io = neo.io.Hdf5IO('test.h5')
block = io.read(lazy=True)[0]
lazy_signal = block.segments[0].analogsignals[0]
loaded_signal = io.load_lazy_object(lazy_signal)
```

The information needed to fully load the lazy object should usually be stored in the IO object (e.g. in a dictionary with lazily loaded objects as keys and the offset in the file as values). The NeoHdf5IO attaches an attribute "hdf5_path" to each loaded object, so I've used that attribute here.

2. The second enhancement is more involved, but also more powerful - I call it lazy cascading. In this mode, all objects (including containers such as units or recording channels) are only loaded when they are accessed and automatically linked into the hierarchy and kept in memory afterwards. This mode is completely transparent, i.e. the user can work with the object hierarchy in the same way as usual.

The implementation works by using an object that mimics the behavior of a list but calls the IO to load objects on access (implemented as neo.io.tools.LazyList). This object is used in place of all lists of downlinks, e.g. Block.segments, Unit.spiketrains. The IO is responsible for creating these lists and populating them with the information needed to load the respective objects.

My proposal for the architecture to implement this scheme: An IO that supports lazy cascading needs to define a method "load_lazy_cascade()" with two parameters: First is the information used to load the object (e.g. an Hdf5 path for the Hdf5 IO), second a bool "lazy" that controls the regular lazy mode just like the parameter "lazy" for e.g. the existing "read" method. Therefore, cascading lazy and regular lazy mode can work together in any combination. This method needs to set uplinks, possibly to previously loaded objects! This means that the IO needs to maintain access to already loaded container objects (but not necessarily to data objects, as they cannot be the target of uplinks). It could also mean that the IO needs to recursively load parent objects: If an AnalogSignal is accessed through a Segment, the "segment" and "recordingchannel" attributes need to be set. If the RecordingChannel has not been accessed before, it needs to be loaded at this moment. For the RecordingChannel, the "recordingchannelgroups" attribute needs to be set as well. This could mean loading one or more RecordingChannelGroup objects that haven't been accessed before.

The read methods of the IO then need to accept an additional value for the "cascade" parameter: In addition to False/0 (no cascading), True/1 (regular cascading - load everything) they can be passed 2 to use lazy cascading. When the mode is active, they replace all link lists in container objects by LazyList objects and add an entry for each object that can be loaded.

The usage example is simple, as nothing changes for the user except the cascade parameter to the read method:
```
io = neo.io.Hdf5IO('test.h5')
block = io.read(cascade=2)[0]
# Just works:
signal = block.recordingchannelgroups[0].recordingchannels[0].analogsignals[0]
```

One could even use LazyList objects for the list returned by io.read and RecordingChannel.recordingchannelgroups, although I currently do not do that in the NeoHdf5IO.

The second method in particular opens the door for IO classes using huge data stores where access can be expensive - e.g. remote databases. The main reason for two different methods is that the second method is much more difficult to implement: Many IOs can enable useful lazy loading with little effort using only the first method. The second method solves most issues Neo currently has with large datasets and is invisible to the user, but requires more effort by IO authors.

Please tell me what you think! Should we include these mechanisms? Do you have suggestions for modifications? When we decide to go ahead, I will add some unit tests and documentation as well.