GSoC 2014: Standard Extreme Learning Machines

Finished implementing the standard extreme learning machines (ELMs). I am getting the following results with 550 hidden neurons against the digits datasets,

Training accuracy using the logistic activation function: 0.999444
Training accuracy using the tanh activation function: 1.000000

Fortunately, this algorithm is much easier to implement and debug than multi-layer perceptron :).
I will push a test file soon.

@ogrisel , @larsmans
This find it a confusing description. I would rather say:

ELM is a supervised nonlinear learning algorithm that results from the sequential application of the following steps:

- a random projection, possibly to a larger dimensional space than the input space,
- an element-wise non-linear activation function such as a tanh sigmoid function,
- a linear one versus all classifier or a multi-output ridge regression model.

Therefore ELM can be interpreted as a single layer feed forward neural network (see figure 1) where:

- only the hidden to output connection weights are trained,
- the connection weights input to hidden are randomly initialized and left random,
- the (default) output loss function is least squares.

ELM have been empirically found to generalize similarly to 1 hidden layer MLPs or RBF kernel support vector machines on a variety of problems while being significantly faster to train.

The main hyper-parameters of the ELM are:

- the variance of the random projection weights,
- the number of nodes on the hidden layer,
- the regularization strength of the output linear model.