lisa-lab/pylearn2 — pull request #157 by nicholas-leonard (2013-02-18 16:18:20+00:00)
Title: Changes to .models.mlp, .training_algorithms.sgd and .costs.supervised_cost

Hi,

I changed a couple of things, mainly in .models.mlp:
* Made the dropout interface to the MLP class more intuitive:
* Knowing that dropout is never applied to the output of a network, dropout_probs and dropout_scales map to a layer's input units. This removes the need for extra __init__ parameters input_dropout_prob and input_dropout_scale. This also simplifies the code.
* Instead of requiring the probability of including a unit, we require the probability of dropping out the unit, i.e. dropout_prob. This is more intuitive and uses the same language as the original paper.
* Made Linear a base class of many layers, i.e. those that share a common __init__ interface (at least in part) and 2-3 of its methods. Softmax layer does not inherit Linear because it doesn't use a MatrixMul or the variable self.transformer for its weight matrix, while the others do. But I believe it could also inherit Linear with a little more changes.
* Changed the weight initialization interface. Made a bunch of callable classes that initialize weights when called. Then the Layer constructors take such an object to initialize its weights. They are well documented and are basically just a refactorization of the initialization interface that Ian provided. These WeightInit classes can also be subclassed to create all kinds of weight initialization strategies, including copies of weights trained with dbm or what not (TODO).
* Changed the bias initialization interface, but small changes really. Basically, I removed the use_bias parameter from the Layer constructors. Now when init_bias is None, no bias is used.
* Added Tanh and Sigmoid layer. The sigmoid layer should actually be called logistic (since both tanh and logistic are sigmoid), but theano.nnet uses the term sigmoid so called it Sigmoid.

And the .training_alogirithms.sgd:
* Just added an ExponentialDecayOverEpoch extension which combines the interfaces of ExponentialDecay and other OverEpoch classes.

And finally the .costs.supervised_cost:
* Just added MSE and SSE.
* Also, MSE, SSE and Cross-Entropy call model.fprop as discussed with Ian.