Extreme Learning Machines
By: Prasad Kumkar
Tags: Machine Learning, Data Science, Artificial Intelligence, Research, Computer Science

Extreme Learning Machines I

Part I: Introduction: Why do we need ELM?

Around 2005, A novel machine learning approach was introduced by Guang-Bin Huang and a team of researchers at Nanyang Technological University, Singapore.

This new proposed learning algorithm tends to reach the smallest training error, obtain the smallest norm of weights and the best generalization performance, and runs extremely fast, in order to differentiate it from the other popular SLFN learning algorithms, it is called the Extreme Learning Machine (ELM).

This method mainly addresses the issue of far slower training time of neural networks than required, the main reasons for which is that all the parameters of the networks are tuned iteratively by using such learning algorithms. These slow-gradient based learning algorithms are extensively used to train neural networks.

Important features of ELM Source

Before going into how ELM works and how is it so good, let’s see how gradient-based neural networks based off.

Demonstration of Gradient-based Neural networks

Below are the steps followed in a single-layered feedforward neural network in brief:

Step 1: Evaluate Wx + B

Step 2: Apply activation function g(Wx + B) and Compute output

Step 3: Calculate Loss

Step 4: Compute gradients (using delta rule)

Step 5: Repeat

This method of propagating forward and back involves a hefty number of calculations Also if the input size is large or if there are more layers/nodes, the training takes up a significant amount of time.

fig.1. 3-layered Neural Network

In the above example, we can see for a 4 node input we require W1 (20 parameters), W2 (53 parameters), and W3 (21 parameters), i.e. 94 parameters in total. And the parameters increase rapidly with the increasing input nodes.

Let’s take a real-life example of image classification of numbers with MNIST Dataset:

MNIST Example

This has a 28x28 input size i.e. 784 input nodes. For its architecture, let’s consider two layers with 128 nodes and 64 nodes, which are then classified into 10 classes. Then parameters will be:

First Layer (784, 128) = 100352 parameters

Second Layer (128, 64) = 8192 parameters

Output Layer (64, 10) = 640 parameters

This will give us a total of 109184 parameters. And the repeated adjustment of weights by backpropagation increases the training time by a lot.

And this just for a 28x28 image, consider training it for bigger input size with 10000’s of features. The training time just gets out of hand.

Conclusion:

In almost all practical learning algorithms of feedforward neural networks, the conventional backpropagation method requires all these weights to be adjusted at every back-prop step.

For most of the time, gradient-descent based strategies have been employed in varied learning algorithms of feedforward neural networks. However, it’s clear that gradient descent-based learning strategies square measure usually terribly slow because of improper learning steps or could simply converge to local minimums. And many iterative learning steps are needed by such learning algorithms so as to get higher learning performance.

This makes the training far slower than required, which has been a major bottleneck for various applications.