rasmusbergpalm/DeepLearnToolbox — pull request #128 by oxinabox (2014-12-16 12:23:34+00:00)
Title: =Made changed to RBMs (and DBNs) to allow for gaussian units

#Motivation
Normalising is a common way to pre-process continuously valued (unbounded) inputs.
'Natural' Guassian process also exist.
Both these input types will not work well with the traditional Bernoulli input layers of an RBM.

The Gaussian Bernoulli RBM was made to deal with this.
Numerous papers talk about it, including: ["A Practical Guide to Training Restricted Boltzmann Machines"](http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf) Geoffrey Hinton 2010, and ["Greedy layer-wise training of deep networks."](http://oldbooks.nips.cc/papers/files/nips19/NIPS2006_0739.pdf) Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle 2007 - Further details of DBNs, including Gaussian Bernoulli unit

##Changes:

RBMs now have a type,
either 'gb' for Gaussian Bernoulli (AKA Gaussian Binary),
or 'bb' for Bernoulli-Bernoulli (AKA Binary-Binary).
If not specified defaults to 'bb'

- Training and encoding algorithm was added.
- Test case was added (it performs normally with ~0.05% error)
- RBM Setup was moved to its own file to clean the code and facilitate the changes
- Batch division was made into a warning instead of an error.

#Limitations
Support was only added for inputs with Unit Variance. NonUnit variance Gasssian inputs are possible but annoying, and would require more restructuring.

(Credit to the Signals and Information Processing Lab of the School of Electrical, Electronic and Computer Engineering, with-in the University of Western Australia, who are (almost directly) paying me to make these changes (and who are happy to have them pushed back upstream).)