Build an Extreme Learning Machine in Python | by Glenn Paul Gara | Towards Data Science

Extreme Learning Machines (ELMs) are single-hidden layer feedforward neural networks (SLFNs) capable to learn faster compared to gradient-based learning techniques. It’s like a classical one hidden layer neural network without a learning process. This kind of neural network does not perform iterative tuning, making it faster with better generalization performance than networks trained using backpropagation method.
ELMs are based on the Universal Approximation Theorem which states that:
“A feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of R^n, under mild assumptions on the activation function.”
This simply means that ELMs can solve classification and regression tasks with significant accuracy if it has sufficient hidden neurons and training data to learn for all hidden neurons.
To understand how ELM works, let me show to you an illustration and the steps in building the model.
So, given the following:
Training set
Hidden node output function H(w, b, x)
Number of hidden nodes L
We can implement ELM in three simple steps:
Randomly assign the parameters of the hidden nodes (w, b)Compute the hidden layer output matrix HCompute the output weights β
Randomly assign the parameters of the hidden nodes (w, b)
Compute the hidden layer output matrix H
Compute the output weights β
Now, let’s proceed to the programming part. I am expecting that you know how to program in python and familiar already using packages in machine learning such as scikit-learn, numpy, and pandas.
We will train the network to classify handwritten digits using MNIST dataset.
First, we need to import necessary packages to build the model.
import numpy as npimport pandas as pdfrom sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import OneHotEncoderfrom sklearn.preprocessing import MinMaxScaler
Next, we need to load our dataset to train the network and test the model.
train = pd.read_csv('mnist_train.csv')test = pd.read_csv('mnist_test.csv')
As a preprocessing technique, we will use MinMaxScaler and OneHotEncoder from scikit-learn package to normalize our features within the range of (0,1), and to transform our targets into a one-hot encoding format.
onehotencoder = OneHotEncoder(categories='auto')scaler = MinMaxScaler()X_train = scaler.fit_transform(train.values[:,1:])y_train = onehotencoder.fit_transform(train.values[:,:1]).toarray()X_test = scaler.fit_transform(test.values[:,1:])y_test = onehotencoder.fit_transform(test.values[:,:1]).toarray()
To initialize our network, we need to identify the following:1. The size of the input layer, which is the number of input features2. Number of hidden neurons3. Input to hidden weights4. Hidden layer activation function
The size of the input layer refers to the number of input features of the dataset.
input_size = X_train.shape[1]
Let’s initialize the number of hidden neurons to 1000.
hidden_size = 1000
Next, we need to initialize our input weights and biases randomly, drawn from a Gaussian distribution.
input_weights = np.random.normal(size=[input_size,hidden_size])biases = np.random.normal(size=[hidden_size])
We will use a Rectified Linear Unit (ReLU) as our hidden layer activation function.
Note: you can use different activation functions.
def relu(x):   return np.maximum(x, 0, x)
We are done initializing our network!
The next thing to do is to create a function in computing the output weights which is our β. The goal is to minimize the least square error between the target (training labels) and the output (predicted labels) using the norm least-squares solution:
Where H (dagger) is the Moore–Penrose generalized inverse of matrix H, and T is our target.
Our H here is the hidden layer of our network. Let’s create a function to compute for our H vector.
def hidden_nodes(X):    G = np.dot(X, input_weights)    G = G + biases    H = relu(G)    return H
Now, we can compute our β in python in just one line of code. Let’s assign the output matrix to a variable name output_weights.
output_weights = np.dot(pinv2(hidden_nodes(X_train)), y_train)
The code computes the Moore-Penrose Pseudoinverse of H using the function pinv and we get the dot product of H (dagger) and T. The result is the computed β (hidden to output weights).
Finally, we have a model. We did not apply any techniques to tune our weights, we just simply compute them.
To ensure that our model produces a good result, we must first test it. Let’s create a function to handle the testing.
def predict(X):    out = hidden_nodes(X)    out = np.dot(out, output_weights)    return out
We can start making predictions already. To do this, let’s write the following code:
prediction = predict(X_test)correct = 0total = X_test.shape[0for i in range(total):    predicted = np.argmax(prediction[i])    actual = np.argmax(y_test[i])    correct += 1 if predicted == actual else 0accuracy = correct/totalprint('Accuracy for ', hidden_size, ' hidden nodes: ', accuracy)
The accuracy of the model is 0.9439, which is a good result already considering that we only have one hidden layer with 1000 hidden nodes, and a non-iterative tuning for learning, making it faster than any gradient-based techniques.
Referring to the graph above, ELM’s accuracy is increasing significantly when extending the number of hidden nodes to 1000 (0.9439 to 0.977). This means that ELM can generalize even better whenever we set the right number of hidden nodes of the network.
The jupyter notebook of this implementation is available in my GitHub tutorials repository below.
github.com
[1] Guang-Bin Huang, Qin-Yu Zhu and Chee-Kheong Siew, Extreme learning machine: Theory and applications (2006), Neurocomputing
[2] Guang-Bin Huang, Extreme Learning Machine, Nanyang Technological University
[3] Guang-Bin Huang, Qin-Yu Zhu and Chee-Kheong Siew, Extreme learning machine: a new learning scheme of feedforward neural networks (2004), IEEE International Joint Conference on Neural Networks