Implementing the XOR Gate using Backpropagation in Neural Networks | by Siddhartha Dutta | Towards Data Science

Implementing logic gates using neural networks help understand the mathematical computation by which a neural network processes its inputs to arrive at a certain output. This neural network will deal with the XOR logic problem. An XOR (exclusive OR gate) is a digital logic gate that gives a true output only when both its inputs differ from each other. The truth table for an XOR gate is shown below:
The goal of the neural network is to classify the input patterns according to the above truth table. If the input patterns are plotted according to their outputs, it is seen that these points are not linearly separable. Hence the neural network has to be modeled to separate these input patterns using decision planes.
As mentioned before, the neural network needs to produce two different decision planes to linearly separate the input data based on the output patterns. This is achieved by using the concept of hidden layers. The neural network will consist of one input layer with two nodes (X1,X2); one hidden layer with two nodes (since two decision planes are needed); and one output layer with one node (Y). Hence, the neural network looks like this:
To implement an XOR gate, I will be using a Sigmoid Neuron as nodes in the neural network. The characteristics of a Sigmoid Neuron are:
1. Can accept real values as input.
2. The value of the activation is equal to the weighted sum of its inputs i.e. ∑wi xi
3. The output of the sigmoid neuron is a function of the sigmoid function, which is also known as a logistic regression function. The sigmoid function is a continuous function which outputs values between 0 and 1:
The information of a neural network is stored in the interconnections between the neurons i.e. the weights. A neural network learns by updating its weights according to a learning algorithm that helps it converge to the expected output. The learning algorithm is a principled way of changing the weights and biases based on the loss function.
Initialize the weights and biases randomly.Iterate over the data i. Compute the predicted output using the sigmoid function ii. Compute the loss using the square error loss functioniii. W(new) = W(old) — α ∆W iv. B(new) = B(old) — α ∆BRepeat until the error is minimal
Initialize the weights and biases randomly.
Iterate over the data i. Compute the predicted output using the sigmoid function ii. Compute the loss using the square error loss functioniii. W(new) = W(old) — α ∆W iv. B(new) = B(old) — α ∆B
Repeat until the error is minimal
This is a fairly simple learning algorithm consisting of only arithmetic operations to update the weights and biases. The algorithm can be divided into two parts: the forward pass and the backward pass also known as “backpropagation.”
Let’s implement the first part of the algorithm. We’ll initialize our weights and expected outputs as per the truth table of XOR.
inputs = np.array([[0,0],[0,1],[1,0],[1,1]])expected_output = np.array([[0],[1],[1],[0]])
Step 1: To initialize the weights and biases with random values
import numpy as npinputLayerNeurons, hiddenLayerNeurons, outputLayerNeurons = 2,2,1hidden_weights = np.random.uniform(size=(inputLayerNeurons,hiddenLayerNeurons))hidden_bias =np.random.uniform(size=(1,hiddenLayerNeurons))output_weights = np.random.uniform(size=(hiddenLayerNeurons,outputLayerNeurons))output_bias = np.random.uniform(size=(1,outputLayerNeurons))
The forward pass involves compute the predicted output, which is a function of the weighted sum of the inputs given to the neurons:
Where Σwx + b is known as the activation.
def sigmoid (x):    return 1/(1 + np.exp(-x))hidden_layer_activation = np.dot(inputs,hidden_weights)hidden_layer_activation += hidden_biashidden_layer_output = sigmoid(hidden_layer_activation)output_layer_activation = np.dot(hidden_layer_output,output_weights)output_layer_activation += output_biaspredicted_output = sigmoid(output_layer_activation)
This completes a single forward pass, where our predicted_output needs to be compared with the expected_output. Based on this comparison, the weights for both the hidden layers and the output layers are changed using backpropagation. Backpropagation is done using the Gradient Descent algorithm.
The loss function of the sigmoid neuron is the squared error loss. If we plot the loss/error against the weights we get something like this:
Our goal is to find the weight vector corresponding to the point where the error is minimum i.e. the minima of the error gradient. And here is where calculus comes into play.
Error can be simply written as the difference between the predicted outcome and the actual outcome. Mathematically:
where t is the targeted/expected output & y is the predicted output
However, is it fair to assign different error values for the same amount of error? For example, the absolute difference between -1 and 0 & 1 and 0 is the same, however the above formula would sway things negatively for the outcome that predicted -1. To solve this problem, we use square error loss.(Note modulus is not used, as it makes it harder to differentiate). Further, this error is divided by 2, to make it easier to differentiate, as we’ll see in the following steps.
Since, there may be many weights contributing to this error, we take the partial derivative, to find the minimum error, with respect to each weight at a time. The change in weights are different for the output layer weights (W31 & W32) and different for the hidden layer weights (W11, W12, W21, W22).Let the outer layer weights be wo while the hidden layer weights be wh.
We’ll first find ∆W for the outer layer weights. Since the outcome is a function of activation and further activation is a function of weights, by chain rule:
On solving,
Note that for Xo is nothing but the output from the hidden layer nodes.This output from the hidden layer node is again a function of the activation and correspondingly a function of weights. Hence, the chain rule expands for the hidden layer weights:
Which comes to,
NOTE: Xo can also be considered to be Yh i.e. the output from the hidden layer is the input to the output layer. Xh is the input to the hidden layer, which are the actual input patterns from the truth table.
Let’s then implement the backward pass.
def sigmoid_derivative(x):    return x * (1 - x)#Backpropagationerror = expected_output - predicted_outputd_predicted_output = error * sigmoid_derivative(predicted_output) error_hidden_layer = d_predicted_output.dot(output_weights.T)d_hidden_layer = error_hidden_layer * sigmoid_derivative(hidden_layer_output)#Updating Weights and Biasesoutput_weights += hidden_layer_output.T.dot(d_predicted_output) * lroutput_bias += np.sum(d_predicted_output,axis=0,keepdims=True) * lrhidden_weights += inputs.T.dot(d_hidden_layer) * lrhidden_bias += np.sum(d_hidden_layer,axis=0,keepdims=True) * lr
This process is repeated until the predicted_output converges to the expected_output. It is easier to repeat this process a certain number of times (iterations/epochs) rather than setting a threshold for how much convergence should be expected.
Choosing the number of epochs and the value of the learning rate decides two things: how accurate the model is, and how fast did the model take to compute the final output. The concept of hyperparameter tuning is a whole subject by itself. The output with epochs = 10,000 and learning rate = 0.1 is:
Output from neural network after 10,000 epochs: [0.05770383] [0.9470198] [0.9469948] [0.05712647]
Hence, the neural network has converged to the expected output:[0] [1] [1] [0]. The epoch vs error graph shows how the error is minimized.
An Introduction to Neural Networks by Kevin Gurneyhttps://www.analyticsvidhya.com/blog/2017/05/neural-network-from-scratch-in-python-and-r/Thanks to https://www.codecogs.com/latex/eqneditor.php for converting LaTeX Equations to PNGs used in this article.
An Introduction to Neural Networks by Kevin Gurney
https://www.analyticsvidhya.com/blog/2017/05/neural-network-from-scratch-in-python-and-r/
Thanks to https://www.codecogs.com/latex/eqneditor.php for converting LaTeX Equations to PNGs used in this article.
I would love to hear about any feedback/suggestions! Connect with me here.