lisa-lab/pylearn2 — pull request #799 by bartvm (2014-04-07 14:50:54+00:00)
Title: Projection layer and Penn Treebank

These additions are mostly for working with language models:

A `projection` (table lookup) operation is added to MatrixMul in `pylearn2.linear`, and an accompanying `ProjectionLayer` is added to the MLP class. This layer takes a list of indices and embeds them into a continuous space.

I extended the `DenseDesignMatrix` class to allow indices in both the sources and targets. DenseDesignMatrix's `max_labels` argument is replaced with a `y_labels` argument, and a new `X_labels` argument is introduced. When these are set, then the input and output space are respectively set to an `IndexSpace`. I made `X_labels` incompatible with passing data as a `topo_view`, because a topological view of a list of integers didn't make much sense to me.

I also wrote a dataset wrapper for the Penn Treebank dataset.

Lastly, it seems that with the recent introduction of dtypes to space, `IndexSpace` now requires a 2D array. MNIST was giving an error when not using `one_hot=True ` because of this, which I fixed.

The following YAML file is an example of a simple language model using the `ProjectionLayer` and `PennTreebank` dataset:

```
!obj:pylearn2.train.Train {
dataset: &train !obj:pylearn2.datasets.penntree.PennTreebank {
which_set: 'train',
ngram_size: &ngram_size 7
},
model: !obj:pylearn2.models.mlp.MLP {
layers: [
!obj:pylearn2.models.mlp.ProjectionLayer {
layer_name: 'projection',
dim: 192,
irange: 0.01
}, !obj:pylearn2.models.mlp.Tanh {
layer_name: 'rectified_linear',
dim: 512,
irange: 0.01
}, !obj:pylearn2.models.mlp.Softmax {
n_classes: 10000,
layer_name: 'softmax',
irange: 0.01
}
],
input_space: !obj:pylearn2.space.IndexSpace {
dim: 6,
max_labels: 10000
}
},
algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {
batch_size: 256,
learning_rate: .1,
monitoring_dataset: {
'valid' : !obj:pylearn2.datasets.penntree.PennTreebank {
which_set: 'valid',
ngram_size: *ngram_size
}
},
termination_criterion: !obj:pylearn2.termination_criteria.MonitorBased {
channel_name: 'valid_softmax_nll',
prop_decrease: 0.,
N: 10
},
},
}
```