bartvm/blocks — pull request #61 by bartvm (2014-12-07 02:20:25+00:00)
Title: Training algorithms, datasets, iterators and monitoring interface

Still a bit WIP, but ready for some initial feedback perhaps! @rizar @pbrakel

Feel free to ask about design decisions. If things turn out to work okay, I'll spend a bit more effort on cleaning things up, documentation and unit tests before merging this.

Some justifications for design decisions:

* Datasets are nothing more but iterators. The support for subset iterators and such is optional (unlike in Pylearn2). This means that you can implement all sorts of crazy datasets, as long as it returns a minibatch each time `next()` is called.
* Support for infinite datasets such as probability distributions.
* There are a two different types of monitoring:
** Update monitoring channels are compiled together with the update function, and allow monitoring of e.g. gradient norm and likelihood for each batch
** Interval monitoring channels are monitored after each epoch, and can optionally receive data from a validation/test set (values are averaged over batches)
* Modularity is quite strictly maintained, with most classes not relying on the usage of other classes in the framework (except for the models and training algorithms creating `MonitorChannel` instances)
* I'm using Pandas for the monitoring data. I think this is a lot more robust than just having lists lying around. Pandas is great for performing data analysis as well, and automatically fills in `NaN` when e.g. certain channels have no data for a given point of time. Downside is that it might be slow to append constantly to a `DataFrame`; I'll have to profile and check this.

There's a whole bunch of things to do still:

[ ] Implement an interface that allows datasets to specify sources like in Pylearn2, and choose which ones should be returned on iteration (e.g. when training an autoencoder we don't want the labels of MNIST)
[ ] Make datasets pickleable by removing and re-loading their data through `__getstate__` and `__setstate__`, potentially making use of checksums (but pickle the rest, which means that the iterator state should automatically be kept).
[ ] Give though about how pre-processing is performed (there's a document from DWF about this for Pylearn2 which I need to give a thorough read)
[ ] Allow for pickling of the monitor. I'm thinking it would be great if we could actually pickle the `DataFrame` to a separate file (very easy through Pandas' `to_pickle` function) so that you don't have to reload the entire model in order to view monitoring results.
[ ] SGD needs a whole lot more features such as learning rate scheduling, momentum, etc. A lot of this can basically be copied from Pylearn2.

Last but not least, a simple example of how to train a blocks model using all of this. Messy, but functional!

```python
from blocks.bricks import Sigmoid, Softmax, MLP
from blocks.bricks.cost import CategoricalCrossEntropy
from blocks.datasets.mnist import MNIST
from blocks.datasets.iteration import SequentialSubsetIterator
from blocks.initialization import IsotropicGaussian, Constant
from blocks.monitor import Monitor, UpdateMonitorChannel, IntervalMonitorChannel
from blocks.select import Selector
from blocks.training_algorithms import SGD

# Construct the model and calculate the cost
mlp = MLP(activations=[Sigmoid(), Softmax()],
dims=[784, 100, 10],
weights_init=IsotropicGaussian(),
biases_init=Constant(0.01))

x, y = tensor.matrix(), tensor.lmatrix()
y_hat = mlp.apply(x)
c = CategoricalCrossEntropy().apply(y.flatten(), y_hat)

# Get the parameters and ask the SGD algorithm for the updates to perform
params = Selector(mlp).get_params().values()
sgd = SGD(c, params, 0.1)
sgd_update = sgd.get_train_updates()

# Ask the algorithm what it can monitor, also add the cost as a monitoring channel
monitoring_channels = sgd.get_monitoring_channels()
monitoring_channels.append(IntervalMonitorChannel('cost', c, needs_data=True))
update_monitoring_channels = [channel for channel in monitoring_channels
if isinstance(channel, UpdateMonitorChannel)]

# Initialize the monitor and add the channels to it
monitor = Monitor()
monitor.channels.extend(monitoring_channels)
monitor.setup([x, y])

# Load the datasets and initialize the minibatch iterator
mnist = MNIST('train')
mnist_test = MNIST('test')
mnist.set_subset_iterator(
SequentialSubsetIterator(mnist.num_examples, 512))
mnist_test.set_subset_iterator(
SequentialSubsetIterator(mnist_test.num_examples, 2048))

# Compile the training function and also return the monitoring values
train_func = theano.function(
[x, y],
[channel.value for channel in update_monitoring_channels],
updates=sgd_update,
allow_input_downcast=True)

# Perform monitoring before training ()
monitor.monitor(monitoring_datasets=[mnist_test])

# Train for 10 epochs, monitoring in between each epoch
for _ in range(10):
for batch in mnist:
update_monitor_vals = train_func(*batch)
monitor.num_examples_seen += len(batch[0])
monitor.num_updates_seen += 1
monitor.num_epochs_seen += 1
monitor.monitor(dict(zip(update_monitoring_channels,
update_monitor_vals)),
monitoring_datasets=[mnist_test])

# Show the results
monitor.data_frame
```
```
gradient_norm cost	learning_rate batch_gradient_norm
num_epochs_seen num_updates_seen num_examples_seen
0 0 0 0.115426 2.302585 0.1 NaN
1 118 60000 0.099438 2.301734 0.1 0.045990
2 236 120000 0.098677 2.301465 0.1 0.045904
3 354 180000 0.097834 2.300954 0.1 0.045825
4 472 240000 0.096943 2.299892 0.1 0.045771
5 590 300000 0.096225 2.297568 0.1 0.045788
6 708 360000 0.096419 2.292279 0.1 0.046005
7 826 420000 0.099381 2.279888 0.1 0.046745
8 944 480000 0.107754 2.251283 0.1 0.048524
9 1062 540000 0.119336 2.194009 0.1 0.050849
10 1180 600000 0.122696 2.112687 0.1 0.050536
```