Training algorithms, datasets, iterators and monitoring interface

Still a bit WIP, but ready for some initial feedback perhaps! @rizar @pbrakel 

Feel free to ask about design decisions. If things turn out to work okay, I'll spend a bit more effort on cleaning things up, documentation and unit tests before merging this.

Some justifications for design decisions:

* Datasets are nothing more but iterators. The support for subset iterators and such is optional (unlike in Pylearn2). This means that you can implement all sorts of crazy datasets, as long as it returns a minibatch each time `next()` is called.
* Support for infinite datasets such as probability distributions.
* There are a two different types of monitoring:
** Update monitoring channels are compiled together with the update function, and allow monitoring of e.g. gradient norm and likelihood for each batch
** Interval monitoring channels are monitored after each epoch, and can optionally receive data from a validation/test set (values are averaged over batches)
* Modularity is quite strictly maintained, with most classes not relying on the usage of other classes in the framework (except for the models and training algorithms creating `MonitorChannel` instances)
* I'm using Pandas for the monitoring data. I think this is a lot more robust than just having lists lying around. Pandas is great for performing data analysis as well, and automatically fills in `NaN` when e.g. certain channels have no data for a given point of time. Downside is that it might be slow to append constantly to a `DataFrame`; I'll have to profile and check this.

There's a whole bunch of things to do still:

[ ] Implement an interface that allows datasets to specify sources like in Pylearn2, and choose which ones should be returned on iteration (e.g. when training an autoencoder we don't want the labels of MNIST)
[ ] Make datasets pickleable by removing and re-loading their data through `__getstate__` and `__setstate__`, potentially making use of checksums (but pickle the rest, which means that the iterator state should automatically be kept).
[ ] Give though about how pre-processing is performed (there's a document from DWF about this for Pylearn2 which I need to give a thorough read)
[ ] Allow for pickling of the monitor. I'm thinking it would be great if we could actually pickle the `DataFrame` to a separate file (very easy through Pandas' `to_pickle` function) so that you don't have to reload the entire model in order to view monitoring results.
[ ] SGD needs a whole lot more features such as learning rate scheduling, momentum, etc. A lot of this can basically be copied from Pylearn2.

Last but not least, a simple example of how to train a blocks model using all of this. Messy, but functional!

```python
from blocks.bricks import Sigmoid, Softmax, MLP
from blocks.bricks.cost import CategoricalCrossEntropy
from blocks.datasets.mnist import MNIST
from blocks.datasets.iteration import SequentialSubsetIterator
from blocks.initialization import IsotropicGaussian, Constant
from blocks.monitor import Monitor, UpdateMonitorChannel, IntervalMonitorChannel
from blocks.select import Selector
from blocks.training_algorithms import SGD


# Construct the model and calculate the cost
mlp = MLP(activations=[Sigmoid(), Softmax()],
		  dims=[784, 100, 10],
		  weights_init=IsotropicGaussian(),
		  biases_init=Constant(0.01))

x, y = tensor.matrix(), tensor.lmatrix()
y_hat = mlp.apply(x)
c = CategoricalCrossEntropy().apply(y.flatten(), y_hat)

# Get the parameters and ask the SGD algorithm for the updates to perform
params = Selector(mlp).get_params().values()
sgd = SGD(c, params, 0.1)
sgd_update = sgd.get_train_updates()

# Ask the algorithm what it can monitor, also add the cost as a monitoring channel
monitoring_channels = sgd.get_monitoring_channels()
monitoring_channels.append(IntervalMonitorChannel('cost', c, needs_data=True))
update_monitoring_channels = [channel for channel in monitoring_channels
 							  if isinstance(channel, UpdateMonitorChannel)]

# Initialize the monitor and add the channels to it
monitor = Monitor()
monitor.channels.extend(monitoring_channels)
monitor.setup([x, y])

# Load the datasets and initialize the minibatch iterator
mnist = MNIST('train')
mnist_test = MNIST('test')
mnist.set_subset_iterator(
	SequentialSubsetIterator(mnist.num_examples, 512))
mnist_test.set_subset_iterator(
	SequentialSubsetIterator(mnist_test.num_examples, 2048))

# Compile the training function and also return the monitoring values
train_func = theano.function(
	[x, y],
	[channel.value for channel in update_monitoring_channels],
	updates=sgd_update,
	allow_input_downcast=True)

# Perform monitoring before training ()
monitor.monitor(monitoring_datasets=[mnist_test])

# Train for 10 epochs, monitoring in between each epoch
for _ in range(10):
	for batch in mnist:
		update_monitor_vals = train_func(*batch)
		monitor.num_examples_seen += len(batch[0])
		monitor.num_updates_seen += 1
	monitor.num_epochs_seen += 1
	monitor.monitor(dict(zip(update_monitoring_channels,
							 update_monitor_vals)),
					monitoring_datasets=[mnist_test])

# Show the results
monitor.data_frame
```
```
                                                    gradient_norm      cost	 learning_rate    batch_gradient_norm
num_epochs_seen num_updates_seen num_examples_seen  
0               0                0                       0.115426  2.302585            0.1                    NaN
1               118              60000                   0.099438  2.301734            0.1               0.045990
2               236              120000                  0.098677  2.301465            0.1               0.045904
3               354              180000                  0.097834  2.300954            0.1               0.045825
4               472              240000                  0.096943  2.299892            0.1               0.045771
5               590              300000                  0.096225  2.297568            0.1               0.045788
6               708              360000                  0.096419  2.292279            0.1               0.046005
7               826              420000                  0.099381  2.279888            0.1               0.046745
8               944              480000                  0.107754  2.251283            0.1               0.048524
9               1062             540000                  0.119336  2.194009            0.1               0.050849
10              1180             600000                  0.122696  2.112687            0.1               0.050536
```

On 07.12.2014 21:21, Bart van Merriënboer wrote:
>
> In blocks/model.py:
>
> >
> >  logger = logging.getLogger(__name__)
> >
> >
> > -class ComputationGraph(object):
> > +class Model(object):
>
> The reason I ran into this is because I wanted to calculate the 
> Hessian, for which I need to replace all the parameters by (reshaped) 
> slices of a single large vector which contains the parameters of the 
> entire model. This means I had to find all the parameters using 
> selectors and then replace them in the computational graph.
>
> Anyway, I'd propose combining the |ComputationGraph| and |Cost| 
> classes into one, because the distinction doesn't make too much sense 
> to me. We could have a signature like |ComputationGraph(outputs, 
> cost=None)| so that you can still perform operations specific to the 
> cost if you want to.
>
Agreed, the distinction is not significant. And the modifiers like 
`add_noise` should not be part of the ComputationGraph anyways, since 
there can be multitude of those and we do not want the 
`ComputationGraph` to be super big. In this way the `ComputationGraph` 
is an API for writing regularization routines.
>
> I understand the underlying distinction between the |Selector| and 
> |ComputationGraph|, but in practice it can be a bit confusing (e.g. I 
> looked for |get_params| as part of |ComputationGraph| first, although 
> it makes more sense as part of the |Selector| I guess, because there 
> could theoretically be shared variables which aren't parameters; maybe 
> we should deal with parameters which aren't shared variables though).
>
They are very different: one is an interface to the brick hierarchy and 
another is an interface to the computation graph.
>
> So from what I understand a typical workflow would look something like
>
>   * Build a model with bricks
>   * Use |Selector| to e.g. find bricks whose configuration can be
>     changed, change some things around
>   * Allocate and initialise the parameters
>   * Load the outputs and cost of the model in a |ComputationalGraph|
>   * Use |Selector| to e.g. find subsets of bricks or their parameters
>   * Then use |ComputationalGraph| to e.g. apply dropout (to the bricks
>     found using the |Selector|), replace parameters/add weight noise
>     (from this point onwards the bricks can be useless, because their
>     parameters have been replaced and they are not aware of this).
>
Pretty much the same as I see it. The only note so far: `apply_dropout`, 
`apply_weight_noise`, etc. should be functions outside of the 
`ComputationGraph`, `ComputationGraph` should remain just an API.

In fact we have refused from the idea of hacking some 
regularization-related values into tags of the shared variables, which I 
think is good.

> The fact that e.g. bricks can be mistaken about what their parameters 
> actually are after making changes using |ComputationalGraph| would be 
> an argument for making sure that the no |TrainingAlgorithm| should 
> have access to bricks.
>
Very good point. In fact I always assumed that.
>
> Instead they should only take raw Theano variables as input. This 
> should make them more general, and prevents errors.
>
> —
> Reply to this email directly or view it on GitHub 
> <https://github.com/bartvm/blocks/pull/61/files#r21432413>.
>