Understanding and Implementing Neural Networks in Java from Scratch 💻 | by Suyash Sonawane | Towards Data Science

As intimidating as it may sounds we will be creating an Artificial Intelligence program in JAVA from scratch and I will explain all the concepts as they come, also you can code alongside reading, it is advised to code the program rather than Ctrl+C and Ctrl+V.
A Neural Network is a type of computational system which represents the human brain in a smaller manner. The yellow dots are called “Neurons ”and the lines joining them are called “Synapses”, these concepts were taken from the 🧠. The system simulates the learning process of the brain by adjusting the “Weights ” of the synapses by the process of “Backpropagation”.
The very first step in creating this system is to make our very own Matrix library as matrices are the fundamental Data Structure when it comes to Neural Networks.
I have created an opensource library with the same concepts explained in this article, if interested you can contribute to it 🙌
github.com
We will create a Matrix class and add the necessary functionalities to it, such as Addition, Subtraction, Transpose, and Multiplication of Matrices
class Matrix{   double [][]data;   int rows,cols;}
Our matrix class holds 3 object variables 1. Data which is a 2d double array to hold the contents of the matrix2. Rows that hold the number of rows of the matrix3.Cols that hold, as you might have already noticed, Columns!!
public Matrix(int rows,int cols) {        data= new double[rows][cols];        this.rows=rows;        this.cols=cols;        for(int i=0;i<rows;i++)        {            for(int j=0;j<cols;j++)            {                data[i][j]=Math.random()*2-1;            }        }    }
here we have created a constructor for initializing our matrix object with random values between -1 and 1 by passing rows and cols as parameters.
public void add(double scaler){    for(int i=0;i<rows;i++)    {        for(int j=0;j<cols;j++)        {            this.data[i][j]+=scaler;        }            }}public void add(Matrix m){    if(cols!=m.cols || rows!=m.rows) {        System.out.println("Shape Mismatch");        return;    }        for(int i=0;i<rows;i++)    {        for(int j=0;j<cols;j++)        {            this.data[i][j]+=m.data[i][j];        }    }}
Here we created an “add” function which is overloaded with 2 parameters, a double and a Matrix object which does element-wise addition.
public static Matrix subtract(Matrix a, Matrix b) {        Matrix temp=new Matrix(a.rows,a.cols);        for(int i=0;i<a.rows;i++)        {            for(int j=0;j<a.cols;j++)            {                temp.data[i][j]=a.data[i][j]-b.data[i][j];            }        }        return temp;    }public static Matrix transpose(Matrix a) {        Matrix temp=new Matrix(a.cols,a.rows);        for(int i=0;i<a.rows;i++)        {            for(int j=0;j<a.cols;j++)            {                temp.data[j][i]=a.data[i][j];            }        }        return temp;    }
Here are the 2 functions for calculating the subtraction and transpose of the matrices sent as parameters to the class static functions. These functions return the new Matrix object.
public static Matrix multiply(Matrix a, Matrix b) {        Matrix temp=new Matrix(a.rows,b.cols);        for(int i=0;i<temp.rows;i++)        {            for(int j=0;j<temp.cols;j++)            {                double sum=0;                for(int k=0;k<a.cols;k++)                {                    sum+=a.data[i][k]*b.data[k][j];                }                temp.data[i][j]=sum;            }        }        return temp;    }        public void multiply(Matrix a) {        for(int i=0;i<a.rows;i++)        {            for(int j=0;j<a.cols;j++)            {                this.data[i][j]*=a.data[i][j];            }        }            }        public void multiply(double a) {        for(int i=0;i<rows;i++)        {            for(int j=0;j<cols;j++)            {                this.data[i][j]*=a;            }        }            }
Ok! a lot is going on here 🤯 the first multiply function takes 2 Matrix objects and does the dot product operation on respective matrices and returns a new Matrix object, the second function does the element-wise multiplication of the matrices, and the last one scales or multiplies the whole matrix by a scaler.
public void sigmoid() {        for(int i=0;i<rows;i++)        {            for(int j=0;j<cols;j++)                this.data[i][j] = 1/(1+Math.exp(-this.data[i][j]));         }            }        public Matrix dsigmoid() {        Matrix temp=new Matrix(rows,cols);        for(int i=0;i<rows;i++)        {            for(int j=0;j<cols;j++)                temp.data[i][j] = this.data[i][j] * (1-this.data[i][j]);        }        return temp;            }
For this example, we are using the Sigmoid activation function for our AI 🧠. The 2 functions apply sigmoid and derivative of sigmoid to the elements of the matrix. The derivative of Sigmoid is required while calculating gradients for backpropagation.
public static Matrix fromArray(double[]x)    {        Matrix temp = new Matrix(x.length,1);        for(int i =0;i<x.length;i++)            temp.data[i][0]=x[i];        return temp;            }        public List<Double> toArray() {        List<Double> temp= new ArrayList<Double>()  ;                for(int i=0;i<rows;i++)        {            for(int j=0;j<cols;j++)            {                temp.add(data[i][j]);            }        }        return temp;   }
These are the helper functions for converting the matrix object to and from the array.
Now ✌ we have a functioning and easy to debug, our very own — Matrix Library!!
Now here comes the real fun and intimidating concepts of Neural Networks, but fear not, we will learn these concepts through coding and you will surely find them very intuitive.
public class NeuralNetwork {        Matrix weights_ih , weights_ho , bias_h , bias_o;        double l_rate=0.01;}
Here we have our NeuralNetwork class with many variables, lets define them one by one.1. weights_ih 👉 The weights matrix for the input and hidden layer.2. weights_ho 👉 The weights matrix for the hidden and output layer.3. bias_h 👉 The bias matrix for the hidden layer.4. bias_o 👉 The bias matrix for the output layer.5. l_rate 👉 The learning rate, a hyper-parameter used to control the learning steps during optimization of weights.
public NeuralNetwork(int i,int h,int o) {        weights_ih = new Matrix(h,i);        weights_ho = new Matrix(o,h);                bias_h= new Matrix(h,1);        bias_o= new Matrix(o,1);            }
Here we have our constructor initializing all the variables with the help of input, hidden and output shape passed to the constructor.The shapes of the matrices are important to check as the dot product only works on compatible matrices for more info on dot product, please follow this
public List<Double> predict(double[] X)    {        Matrix input = Matrix.fromArray(X);        Matrix hidden = Matrix.multiply(weights_ih, input);        hidden.add(bias_h);        hidden.sigmoid();                Matrix output = Matrix.multiply(weights_ho,hidden);        output.add(bias_o);        output.sigmoid();                return output.toArray();    }
Here we have our predict function which does a forward pass or forward propagation on the Neural Network
Forward Propagation is a computational step in Neural Networks where the input (Matrix) is multiplied with the weights (Matrix) of the hidden layer, then bias (column Matrix) is added to the dot product of the previous step finally, the result is then processed element-wise by the activation function. This is preformed on each layer with the result of preceding layer as the input for the next layer.
This step is also called as Forward Pass and is used for generating prediction from the network.
The function accepts a double array of inputs and then converts them to a column matrix by our helper function the forward pass is then calculated on both the layers and the output is then flattened into a list by another helper function.
public void train(double [] X,double [] Y)    {        Matrix input = Matrix.fromArray(X);        Matrix hidden = Matrix.multiply(weights_ih, input);        hidden.add(bias_h);        hidden.sigmoid();                Matrix output = Matrix.multiply(weights_ho,hidden);        output.add(bias_o);        output.sigmoid();                Matrix target = Matrix.fromArray(Y);                Matrix error = Matrix.subtract(target, output);        Matrix gradient = output.dsigmoid();        gradient.multiply(error);        gradient.multiply(l_rate);                Matrix hidden_T = Matrix.transpose(hidden);        Matrix who_delta =  Matrix.multiply(gradient, hidden_T);                weights_ho.add(who_delta);        bias_o.add(gradient);                Matrix who_T = Matrix.transpose(weights_ho);        Matrix hidden_errors = Matrix.multiply(who_T, error);                Matrix h_gradient = hidden.dsigmoid();        h_gradient.multiply(hidden_errors);        h_gradient.multiply(l_rate);                Matrix i_T = Matrix.transpose(input);        Matrix wih_delta = Matrix.multiply(h_gradient, i_T);                weights_ih.add(wih_delta);        bias_h.add(h_gradient);            }
Now in the above train function, we get X and Y as double arrays, we convert them to matrices, the output from the forward pass is subtracted from the target that is Y in matrix form. The result of the subtraction is the error of the current sample passed. This error is used to calculate gradients for the backpropagation. The derivative of the sigmoid function is applied element-wise on the output matrix, which returns the gradient matrix then it is multiplied by the output errors and the learning rate which decides the learning step.
Backpropagation is just the reverse of the forward pass in which we take the transpose of the weight matrices and then multiply them with the gradient calculated from the errors, which in turn return the deltas that are used to adjust the weights in the current layer. The bias are updated using the gradients.
We repeat the same steps for the previous layers in the network as a back prop runs from the output layers to the input layers.Finally, we have updated the weights of all the layers for the current sample which completes 1 step on the dataset.
public void fit(double[][]X,double[][]Y,int epochs)    {        for(int i=0;i<epochs;i++)        {                int sampleN =  (int)(Math.random() * X.length );            this.train(X[sampleN], Y[sampleN]);        }    }
The fit function takes in 2 2d arrays namely X and Y, also the number of epochs .ie. how many times do we have to iterate over the dataset. Here we repeatedly call the train function with random data points from the dataset so that we can generalize the network over the dataset.This completes our Neural Network class 🥳 , now we can test our program on a simple dataset.
suyashsonawane.me
We will make a simple 2d double array from the data of XOR Logic Gate.
static double [][] X= {            {0,0},            {1,0},            {0,1},            {1,1}    };static double [][] Y= {            {0},{1},{1},{0}    };
Now we will create a Neural Network object that we will train on the dataset
NeuralNetwork nn = new NeuralNetwork(2,10,1);nn.fit(X, Y, 50000);
We used 50000 epochs as the dataset is very small and contains only 4 samples.
double [][] input ={{0,0},{0,1},{1,0},{1,1}};for(double d[]:input){    output = nn.predict(d);    System.out.println(output.toString());}//Output[0.09822298990353093][0.8757877124658147][0.8621529792837699][0.16860984858200806]
Now we test the Neural Network with our input in the form of a double array. The output looks great as the value greater than 0.5 resembles 1 output and the other resembles 0 output.
All the source code is given below
So, we have successfully created a Neural Network from scratch, also we created our very own Matrix Library which handles all the matrix operations seamlessly. We also tested it against a small XOR dataset which gave us pretty good results.
Being a fellow Python programmer I was not very comfortable with Java programming as it requires too much attention towards data types and typecasting and AH!😓 the semicolon; If you spotted any unnecessary or redundant code then please share it in the comments also if you have any suggestions please comment below.🤗
Follow me on:
Portfolio: https://suyashsonawane.me/Twitter: Suyash Sonawane (@SuyashYSonawane) / TwitterLinkedIn: Suyash Sonawane | LinkedInGithub: SuyashSonawane (Suyash Sonawane)