Train Neural Network (Numpy)— Particle Swarm Optimization(PSO)
By: Zeeshan Ahmad
Tags: Pso, Python, Numpy, Neural Networks, Evolutionary Algorithms

Neural Network

Import Packages

Import sklearn to load Iris flower dataset, pso_numpy to use PSO algorithm and numpy to perform neural network’s forward pass.

Load Dataset

Load Iris data-set from sklearn and assign input data to X and target labels to Y.

Define Architecture

Define no of input, hidden and output nodes in your neural network model.

One-hot Encoding

One-hot encoding is used in case we want to calculate Categorical Cross Entropy Loss. One-hot encoding is a technique that creates a sparse vector zero everywhere instead of the one position. This way a unique vector is assigned to each target label(class). The function takes Y as input and returns one hot encoded vector for each class.

Image by Kaggle

Softmax Activation

Softmax function is used to calculate probability of each class from logits(output of last layer without any activation applied). The some of these probabilities of each class is always equal to 1.

Image by mc.ai

Loss Functions

We can use one of these two loss functions depending upon our input. If we don’t encode our target labels to one hot vector then we use Negative Log Likelihood but if we encode our labels then we use Categorical Cross Entropy. We’re not going to discuss these loss functions in detail so if you have any questions related to these you can leave a response I’ll try to respond back soon. Here probs is the calculated probability and Y is the actual labels of target class.

Training Forward Pass

The function performs the forward pass of neural network and calculates the error using predicted and actual labels and then return that error to PSO which optimizes the error and update the weights. It takes X(input data), Y(target labels) and W(weights) of the connections between the neurons in the layers of neural network.

Prediction Forward Pass

It performs the forward pass only and returns the predicted labels. It takes X(input) and W(trained weights after the PSO training is complete).

Accuracy

It calculates the accuracy of test data using actual and predicted labels as input. It takes Y(actual labels) and Y_pred(predicted labels) and then count the number of true predictions and then mean it.

Run

Here we just define the no_solution(no of particles in PSO), no_dim(no of dimensions of each particle in PSO), w_range(weights range), lr_range(learning rate range), iw_range(inertia weight range) and a tuple c for cognitive and social parameter. Then initialize the Swarm and call the optimize function with forward_pass(forward pass function), X(input), Y(labels), print_step(after how many iterations you want to see loss) and the number of iterations. After optimization call the method get_best_solution() with Swarm object to get best pair of weights. Then pass that pair of weights to predict and get the output and finally calculate the accuracy of model.

Results

Accuracy of 98.7

Accuracy of 99.3

Accuracy of 100

As you can see the results are very reliable for the training set. So it would also provide good results for other applications if the values of parameters of PSO are properly assigned and trained enough number of iterations.