lisa-lab/pylearn2 — pull request #765 by skearnes (2014-03-28 19:54:38+00:00)
Title: Cross-validation and clean handling of multiple independent models

There's been talk about cross-validation in the past (#49), and some basic dataset splitting functionality was added to `DenseDesignMatrix` in #73. However, true cross-validation requires training independent models on different subsets of the data, and it's not obvious how this would be done using the current setup since training algorithms typically rely on the `iterator` method of a dataset.

I have written a cross-validation module that implements three major tools:

1. A `DatasetIterator` object that takes an index iterator (such as an `sklearn.cross_validation` object) and yields new instances of `DenseDesignMatrix` for each subset in each fold.
2. Subclasses of `DatasetIterator` that take a standard pylearn2 dataset and wrap several `sklearn.cross_validation` objects.
3. A `TrainCV` object that acts as a container for a set of `Train` objects, each containing different training, validation and/or test datasets. The trained models are saved in a single pickle, which can be used as a pretrained layer (via `PretrainedLayers`) in another model.

There is a simple working example test_cross_validation.py that trains an MLP with a single hidden layer in two steps, with minimal changes to the YAML required for building the same model without cross validation.

Please let me know if there's a more straightforward way to do this using the existing code.