mihaelacr/pydeeplearn — pull request #2 by Warvito (2014-10-10 14:30:26+00:00)
Title: Created new file with an example of hyperparameter optimization using hyperopt package

Hi!

In this fork I created an example of using the library Hyperopt for Hyperparameter optimization of a Deep Belief Network. In this example the MNIST database is used, and it optimizes the following parameters: number of hidden layers, size of layers, unsupervised learning rate, supervised learning rate, maximum momentum, and visible dropout rate.

Currently two algorithms are implemented in hyperopt: Random Search and Tree of Parzen Estimators (TPE - which is implemented in the example). More information about the hyperopt in http://hyperopt.github.io/hyperopt/.

I chose TPE over others methods (like Spearmint and SMAC) because this was the only library that I knew. However, searching the literature I found a study in which these algorithms are compared (http://www.cs.ubc.ca/~kevinlb/papers/2013-BayesOpt-Hyperparameters.pdf). In this study it is shown that the TPE algorithm has a better performance in some cases.

PS: Congratulations for the excellent library and report.
PS2: Sorry for the last pull request, I'm still newbie on github. Please ignore and close the old pull request.