7 Steps to Design a Basic Neural Network (part 1 of 2) | by Gabe Verzino | Towards Data Science

This two-part article takes a more holistic, overarching (and yes, less math-y) approach to building a neural network from scratch. Python for completing the network is also included in each of the 7 steps.
Part One: (1) Define the network structure, (2) Initialize parameters, and (3) Implement forward propagation. Part Two: (4) Estimate cost, (5) Implement backward propagation, (6) Update parameters, and (7) Make predictions.
Data is often non-linearly distributed, or contains unusual boundaries for which traditional classification models can’t differentiate very well. For example, let’s say we want to correctly classify the red and blue dots in the dataset below:
We can start with a logistic regression — the go-to model for binary classification problems. As usual, we’ll import the necessary libraries and fit the logistic model to our X variable (i.e. the x-axis and y-axis coordinates of the dots above) and our Y outcome variable (i.e. red or blue dots).
import pandas as pdfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import classification_report, confusion_matrixlogmodel = LogisticRegression()logmodel.fit(X.T, Y.T)predictions = logmodel.predict(X.T)
Applying the boundary predictions for the logistic regression model, we can immediately see some misclassification occurring.
plot_decision_boundary(lambda x: logmodel.predict(x), X, Y)plt.title("Logistic Regression")
While most dots got predicted accurately, some red dots are getting classified as blue, and vice-versa. This model is a bit too general and under-fits the data, which will result in high bias. Running a quick classification report and confusion matrix shows the accuracy of our logistic model is about 87%.
print(classification_report(predictions, Y.T))
conf_matrix = np.array(confusion_matrix(Y.T, predictions))pd.DataFrame(conf_matrix, index=['Purple','Red'],columns=['Predicted Purple','Predicted Red'])
While this might suffice for some data tasks — a basic prediction model or a quick gut-check — neural networks can do a much better job at recognizing complex patterns (like the above) and optimizing your prediction.
Alrighty, so let’s get to it!
People compare neural networks to neurons in the brain because they look similar and are both involved in learning. Truth is, unlike brain neurons, neural networks “fire messages” both ways, and this unique bi-directionality is central to learning that occurs in backward propagation (covered in Part 2).
With that visual in mind, we first have to structure our neural network. As shown below, networks contain nodes and layers (columns of nodes).
Input Layer: From left to right, we begin with the input layer and 2 nodes, each node representing a variable from our dataset (our X variable contains both X- and Y-axis coordinate data, so it becomes X1 and X2).
Hidden Layer: In a process known as forward propagation, input layer data is transmitted to each of 4 nodes in the hidden layer. The number of nodes you choose in the hidden later is important; too few nodes and your model may underfit, and too many nodes may cause your model to overfit and run slowly. Given the information explained in this StackExchange post, we’ll choose 4 nodes for our hidden layer.
Output Layer: Information from the hidden layer is then transferred to the output layer to make our final predictions (i.e. is the dot blue or red?). This is made possible by completing some steps mentioned later on in this article (estimating cost, backward propagation, and updating our parameters).
So let’s create these 3 layers now with their corresponding number of nodes.
def layer_sizes(X, Y):    """Takes the arguments:        X = shape of feature variable data        Y = shape of the output data                Returns:        n_x = input layer size        n_h = hidden layer size        n_y = output layer size"""    n_x = X.shape[0]    n_h = 4    n_y = Y.shape[0]    return(n_x, n_h, n_y)(n_x, n_h, n_y) = layer_sizes(X, Y)print("The size of the input layer is: n_x = " + str(n_x))print("The size of the hidden layer is: n_h = " + str(n_h))print("The size of the output layer is: n_y = " + str(n_y))
The output of which is:
Now that we have defined our network structure, we need to think about building functions that will transfer input-layer data into our hidden layer to kick-start the neural network process. However, these functions contain terms (called parameters) for which we must initialize (i.e. set values).
Here’s our first function that will be used (in Step 3. Forward Propagation):
In this function, we already have x our input data, w the weight (significance) to that data, and b our bias term. The w and b are the parameters. And because we don’t have values for the parameters yet we must initialize them.
How and why we initialize these parameters has a great deal to do with gradient descent — a central component to neural network behavior. So let’s chat about that briefly.
Like many nondeterministic models, neural networks operate on the premise of stochastic gradient descent. During training, these algorithms take incremental steps to minimize cost — a single value that reflects how well the model performed as a whole (i.e. error between predicted and actual values). And the “learning steps” they take to achieve that depends on our w (weights) and b (biases).
For our model, we want the lowest cost possible. Imagine the gradient descent process as a ball rolling down hills, attempting to find the lowest point (cost).
In good (not great) models, the cost can get stuck at a lower point from where it began (a local minimum), but not at an absolute low (the global minimum). So, because we (1) can’t visualize our search space in advance, and (2) don’t want to “trap” our model in a suboptimal cost, a good approach is to initialize our parameters w (weights) and b (biases) with random values. This ensures gradient descent begins anywhere without injecting our own bias.
Secondly, we want small values for our parameters — for example, something on the order of 0.01. It’s a bit of Goldilocks here with our choice; a value too large and the cost will oscillate too much around a minimum value (i.e. an “exploding gradient problem”), and a value of zero basically prevents our gradient from updating any weights. This is called “failing to break symmetry” and results from neurons computing similar outputs, preventing independent learning, and essentially nullifying the advantages of neural networks.
The parameters we need to initialize will span the input layer to the hidden layer (call them W1 and b1), and then the hidden layer to the output layer (call them W2 and b2). And as we mentioned just above, we’ll initialize them as random, small values. Note that the bias terms b1 and b2 are vectors, and because they are added to the w*x products through broadcasting, they can just be initialized as zeros with columns of size 1.
def initialize_parameters(n_x, n_h, n_y):     """Initialize the parameters for weight matrix w1, w2 and bias vectors b1, b2.              Where:        n_x -- size of the input layer       n_h -- size of the hidden layer       n_y -- size of the output layer           Returns:       params -- python dictionary containing your parameters:                        W1 -- weight matrix of shape (n_h, n_x)                        b1 -- bias vector of shape (n_h, 1)                        W2 -- weight matrix of shape (n_y, n_h)                        b2 -- bias vector of shape (n_y, 1)    """    np.random.seed(2)        W1 = np.random.randn(n_h, n_x) * 0.01    b1 = np.zeros((n_h, 1))    W2 = np.random.randn(n_y, n_h) * 0.01    b2 = np.zeros((n_y, 1))        assert (W1.shape == (n_h, n_x))    assert (b1.shape == (n_h, 1))    assert (W2.shape == (n_y, n_h))    assert (b2.shape == (n_y, 1))        parameters = {'W1': W1,                  'b1': b1,                  'W2': W2,                  'b2': b2}        return parameters
Let’s take a look at what these parameters will look like:
parameters = initialize_parameters(n_x, n_h, n_y)print("W1 = " + str(parameters["W1"]))print("b1 = " + str(parameters["b1"]))print("W2 = " + str(parameters["W2"]))print("b2 = " + str(parameters["b2"]))
Returns:
With our parameters initialized, we can now implement our linear functions (z = w * x + b) as described above. However, linear functions alone would just result in a neural network that performs similar to logistic or linear regression. So, we must also pass our z outputs into non-linear functions called activation functions.
The intuition here is that we have a noisy dataset with non-linear decision boundaries — so why apply linear models? It’s these non-linear activation functions that will take an additional step to transform outputs from our linear function (z = w * x + b) to better classify our non-linear data.
There are a number of activation functions we can use in neural networks, each with its own pros and cons. For our neural network, we’ll choose a tanh activation function in the hidden layer and a sigmoid activation function in the output layer. Tanh computes outputs between -1 and 1 with the mean of activations at zero. This has the effect of naturally centering our data, which allows learning between hidden layers to compute faster. Sigmoid is ideal in the output layer, especially for binary classification problems, because it computes outputs between 0 and 1.
There are slight drawbacks to both sigmoid and tanh if our z inputs are very large or very small; the gradient (or slope) becomes very small (sometimes close to zero) and can slow down gradient descent.
Let’s unpack details from the 4 functions shown above.
The first linear z[1] function computes the product of all input layer data a[0] (a way to represent all our X feature data) and input layer weights W[1], and adds all input layer bias terms b[1].
The activation function a[1] is the tanh function of the z[1] output.
The second linear z[2] function computes the product of all hidden layer data a[1] (a way to represent all our initial activation functions) and hidden layer weights W[2], and adds all hidden layer bias terms b[2].
The activation function a[2] is the sigmoid function of the z[2] output.
We can code the described functions above for z[1], a[1], z[2] and a[2]. Then we can store these values into a dictionary called “cache” for later usage.
def forward_propagation(X, parameters):    """Forward propagation will compute activations functions for     the hidden and output layer, producting outputs A1 and A2.    Arguments: Takes X data and parameter variables (W1, W2, b1, b2)        """        #retrieve each of the paramters from the dictionary    W1 = parameters['W1']    b1 = parameters['b1']    W2 = parameters['W2']    b2 = parameters['b2']        Z1 = np.dot(W1,X) + b1    A1 = np.tanh(Z1)    Z2 = np.dot(W2, A1) + b2    A2 = sigmoid(Z2)        cache = {"Z1": Z1,             "A1": A1,             "Z2": Z2,             "A2": A2}        return A2, cache
Now we have our linear Z functions (z[1], z[2]) and activation functions (a[1], a[2]) defined in a cache for later usage.
Great job getting this far! We have learned how to create a neural network structure, initialize parameters, implement some activation functions, and yes, even survived a little math.
In Part 2 of this post, we’ll cover the remaining steps of building our neural network model: (4) defining cost, (5) implementing backward propagation, (6) updating parameters, and (7) making predictions.