SimTk/pylearn2 — pull request #1 by skearnes (2014-04-21 18:36:49+00:00)
Title: Cross validation

There's been talk about cross-validation in the past (#49), and some basic dataset splitting functionality was added to `DenseDesignMatrix` in #73. However, true cross-validation requires training independent models on different subsets of the data, and it's not obvious how this would be done using the current setup since training algorithms typically rely on the `iterator` method of a dataset.

I have written a cross-validation module that implements three major tools:

1. A dataset iterator object that takes an index iterator (such as an `sklearn.cross_validation` object) and yields new instances of `DenseDesignMatrix` for each subset in each fold.
2. Subclasses of `DatasetIterator` that take a standard pylearn2 dataset and wrap several `sklearn.cross_validation` objects.
3. A `TrainCV` object that acts as a container for a set of `Train` objects, each containing different training, validation and/or test datasets. The trained models are saved in a single pickle, which can be used as a pretrained layer (via `PretrainedLayers`) in another model.

Additionally, I have written print_monitor_average.py, which applies the same logic from print_monitor.py to print average values for monitor channels across the cross-validation models.