Neural Network from Scratch With Python | by Angela Shi | Towards Data Science

Neural Network is often considered a black-box algorithm. Data visualization can help us better understand the principles of this algorithm. Since standard packages don’t give all details of how the parameters are found, we will code a neural network from scratch. And in order to visualize simply the results, we have chosen a simple dataset.
Let’s use this simple dataset with only one feature X.
import numpy as npX=np.array([[-1.51], [-1.29], [-1.18], [-0.64],[-0.53], [-0.09], [0.13], [0.35],[0.89], [1.11], [1.33], [1.44]])y=np.array([[0], [0], [0], [0],[1], [1], [1], [1],[0], [0], [0], [0]])
X is a single column vector with 12 observations, and y is also a column vector with 12 values that represent the target. We can visualize this dataset.
import matplotlib.pyplot as pltplt.scatter(X,y)
For those of you who already know how a neural network works, you should be able to find a simple structure by seeing this graph. In the next part, the activation function will be the sigmoid function.
So the question is: how many layers and neurons do we need in order to build a neural network that would fit the dataset above?
If we use only one neuron it will be the same thing as doing a Logistic Regression because the activation function is the sigmoid function. And we know that it won’t work because the dataset is not linearly separable, and Simple Logistic Regression doesn’t work with not linearly separable data. So we have to add a hidden layer. Each neuron in the hidden layer will result in a linear decision boundary.
In general, a Logistic Regression creates a hyperplane as a decision boundary. Since here we only have one feature, then this hyperplane is only a dot. Visually, we can see that we would need two dots to separate one class from the other. And their values would be around for one -0.5 and the other 0.5.
So a neural network with the following structure would be a good classifier for our dataset.
If it is not clear for you, you can read this article.
towardsdatascience.com
Before building the neural network from scratch, let’s first use algorithms already built to confirm that such a neural network is suitable, and visualize the results.
We can use the MLPClassifier in scikit learn. In the following code, we specify the number of hidden layers and the number of neurons with the argument hidden_layer_sizes.
from sklearn.neural_network import MLPClassifierclf = MLPClassifier(solver=’lbfgs’,hidden_layer_sizes=(2,), activation=”logistic”,max_iter=1000)clf.fit(X, y)
Then we can calculate the score. (You should get 1.0, otherwise, you may have to run the code again because of local minimums).
clf.score(X,y)
So great, how can we visualize the results of the algorithm? Since we know that this neural network is made by 2+1 Logistic Regression, we can get the parameters with the following code.
clf.coefs_clf.intercepts_
How do we interpret the results?
For clf.coefs_, you will get (for example):
[array([[-20.89123833, -8.09121263]]), array([[-20.19430919], [ 17.74430684]])]
And for clf.intercepts_
[array([-12.35004862, 4.62846821]), array([-8.19425129])]
The first items of the lists contain the parameters for the hidden layer, and the second items contain the parameters for the output layer.
With these parameters, we can plot the curves:
def sigmoid(x):    return 1.0/(1+ np.exp(-x))plt.scatter(X,y)a1_1=sigmoid(xseq*clf.coefs_[0][0,0]+clf.intercepts_[0][0])a1_2=sigmoid(xseq*clf.coefs_[0][0,1]+clf.intercepts_[0][1])output=sigmoid(a1_1*clf.coefs_[1][0]+a1_2*clf.coefs_[1][1]+clf.intercepts_[1])plt.plot(xseq,a1_1,c=”red”)plt.plot(xseq,a1_2,c=”blue”)plt.plot(xseq,output,c=”black”)
And we can get the following graph:
The red one is the result of the neuron 1 of the hidden layer
The blue one is the result of the neuron 2 of the hidden layer
The black one is the output
If you run the code, you may get another result, because the loss function has several global minimums.
In keras, it is of course also possible to create the same structure:
from keras.models import Sequentialfrom keras.layers import Densemodel = Sequential()model.add(Dense(2, activation=’sigmoid’))model.add(Dense(1, activation=’sigmoid’))model.compile(loss=’binary_crossentropy’, optimizer=’adam’, metrics=[‘accuracy’])model.fit(X_train, y_train, epochs=300)
Now the question is how are all the seven parameters found? One method is to use gradient descent.
First, let’s do the forward propagation.
For each neuron, we have to find the weight w and the bias b. Let’s try some random values.
plt.scatter(X,y)plt.plot(xseq,sigmoid(xseq*(-11)-11),c="red")
Since there are two neurons, we can do a matrice multiplication by creating matrices for the parameters:
the weight matrix should have two columns. (Since here the input data has one column, then the weight matrix should have one row and two columns). We can do a random initialization, and choose some values.
the bias should have the same structure.
w1=np.random.rand(X.shape[1],2) # random initializationw1=np.array([[-1,9]])b1=np.array([[-1,5]])z1=np.dot(X, w1)+b1a1=sigmoid(z1)
If you are just reading and not running a notebook at the same time, one exercise to do is to answer the following questions:
What is the dimension of np.dot(X, w1)?
What is the dimension of z1?
Why is it OK to do addition? What if b1 is a simple 1d-array?
What is the dimension of a1?
If we replace the input X by xseq created before, we can plot the curves:
a1_seq=sigmoid(np.dot(xseq.reshape(-1,1), w1)+b1)plt.scatter(X,y)plt.plot(xseq,a1_seq[:,0],c="red")plt.plot(xseq,a1_seq[:,1],c="blue")
Now for the output, it is a very similar calculation.
The weight matrix has now the rows because the hidden layer results in a matrix of two columns.
The bias matrix is a scalar
w2 = np.random.rand(2,1)b2=0output=sigmoid(np.dot(a1,w2)+b2)
Then we can plot the output, with the other curves
output_seq=sigmoid(np.dot(a1_seq,w2)+b2)plt.scatter(X,y)plt.plot(xseq,a1_seq[:,0],c=”red”)plt.plot(xseq,a1_seq[:,1],c=”blue”)plt.plot(xseq,output_seq,c=”black”)
As you can see, the randomly chosen parameters are not good.
Just to sum up the forward propagation:
def feedforward(input,w1,w2,b1,b2):    a1 = sigmoid(np.dot(input, w1)+b1)    output = sigmoid(np.dot(a1, w2)+b2)    return output
The suitable parameters are those which minimize the cost function. We can use the cross-entropy:
The function can coded as below:
def cost(y,output):    return -np.sum(y*np.log(output)+(1-y)*np.log(1-output))/12
Since there are 7 parameters, visualizing the cost function is not easy. Let’s just choose one of them to vary. For example the first weight in w1.
b1=np.array([[16.81,-23.41]])w2= np.array([28.8,-52.89])b2=-17.53p = np.linspace(-100,100,10000)for i in range(len(p)):    w1=np.array([[p[i],-37.94]])    output=feedforward(X,w1,w2,b1,b2)    cost_seq[i]=cost(y,output)
And you can see that it is not convex at all.
And it is also possible to vary two parameters.
In order to better visualize the cost function, we can also make an animation.
Now let find some suitable global minimums of this cost function using gradient descent which is call backpropagation.
Partial derivatives can be ugly, but fortunately, with the cross-entropy as a loss function, there are some simplifications in the final results.
Here is the cost function again:
Please note that when you use the cost function to calculate the cost of a model, then the input variables for this function are the output of the model and the real value of the target variable.
If we try to find the optimum parameters of the model, then we consider that the input variables for this cost function are these parameters. And we are going to calculate the partial derivatives of the cost function with respect to every parameter.
For the partial derivative with respect to w1, using the chain rule, we have:
First, for the sigmoid function, the derivative can be written as:
(Please note that the cost function is a sum of functions, and the partial derivative of a sum of functions is the sum the partial derivates of the functions, so in order to simplify the notation, we will drop the sum symbol, or to be exact, the mean calculation).
Let’s first calculate the first two items as below:
And we can notice that they can be simplified to (yhat-y).
Then we have the final result for the w1:
For b1, the expression is quite similar, as the only difference is the last partial derivative:
We are going to code the computation with matrix multiplication. Before coding, we can ask ourselves some questions (and answer them):
What is the dimension of the residuals (yhat — y)? It is a column vector and the number of rows is equal to the number of total observations.
What is the dimension of w2? It is a matrix of two rows and one column. Remember, it is the weight matrix for the two hidden neurons, to compute the output.
What is the dimension of (yhat — y)*w2? Since the dimension w2 is (2,1), we can not do a simple multiplication. What should we do? We can transpose the w1 matrix. Then (yhat — y)*w2 will give us a matrix of two columns and 12 observations. It is perfect because we want to do the computation for each of the weights.
What is the dimension of a1? It is the results of the hidden layer. Since we have two neurons, a1 has two columns and 12 observations. And the multiplication with the previous matrix will be element-wise multiplication.
All this is very consistent because, in the end, we will get a matrix of 2 columns and 12 observations. And the first column is associated with the weight of the first neuron, and the second column is associated with the weight of the second neuron in the hidden layer.
Let’s do some coding:
d_b1_v=np.dot((output-y), w2.T) * a1*(1-a1)
What does this matrix represent? We can show the partial derivative with respect to b1 again.
The matrix d_b1_v is the partial derivative for all observations. And to get the final derivatives, we have to calculate the sum of those associated with all observations (remember, L is a sum of functions), and calculate the average value.
d_b1=np.mean(d_b1_v,axis=0)
For w1, we have to take x into account. For each observation, we have to multiply the value obtained by previous partial derivatives by the value of x, then sum them all. This is exactly a dot product. And to obtain the average value, we have to divide by the number of observations.
d_w1 = np.dot(X.T, d_b1_v)/12
Now, let continue for the parameters of the output layer. It will be much easier.
We already have:
So the derivatives with respect to w2 and b2 are:
For b2, we just have to sum the residuals
np.sum(output-y)/12
For w2, it is the dot product between a1 (results of layer 1) and the residuals.
np.dot(a1.T, (output-y))/12
Now we can create a class to include the two steps of forward propagation and backward propagation.
I used the python code based on this very popular article. You may already have read it. The differences are
the loss function (cross-entropy instead of MSE)
adding a learning rate
class NeuralNetwork:    def __init__(self, x, y):        self.input = x        self.w1 = np.random.rand(self.input.shape[1],2)        self.w2 = np.random.rand(2,1)        self.b1 = np.zeros(2)        self.b2 = 0.0        self.y = y        self.output = np.zeros(self.y.shape)    def feedforward(self):        self.a1 = sigmoid(np.dot(self.input, self.w1)+self.b1)        self.output = sigmoid(np.dot(self.a1, self.w2)+self.b2)    def backprop(self):        lr=0.1        res=self.output-self.y        d_w2 = np.dot(self.a1.T, res)        d_b2 = np.sum(res)        d_b1_v=np.dot(res, self.w2.T) * self.a1*(1-self.a1)        d_b1 = np.sum(d_b1_v,axis=0)        d_w1 = np.dot(self.input.T, d_b1_v)        self.w1 -= d_w1*lr        self.w2 -= d_w2*lr        self.b1 -= d_b1*lr        self.b2 -= d_b2*lr
It is then possible to store the intermediate values of the 7 parameters during the gradient descent and plot the curves.
The animation is made with graphs from R code. So if you are interested in R code of the neural network from scratch, please comment.
If you the code is difficult for you to understand, I also created an Excel (Google Sheet) file to do the gradient descent. If you are interested, please let me know in the comments. Yes, you may think that is it crazy to do machine learning in Excel, and I agree with you, especially after having done all the steps of gradient descent for all the seven parameters. But the objective is to better understand. And for that, Excel is an excellent tool.