Neural Networks, Demystified
By: Ben Rohald
Tags: Machine Learning, Education, Neural Networks, Artificial Intelligence

You’ve no doubt heard about neural networks — the mysterious and sci-fi-like technology that makes for a great buzzword. But being someone who isn’t technical, you’ve written them off as an enigma left only for computer science nerds (like myself). That changes today with this primer on how they work, designed for people who know nothing about computer science, coding, or mathematics.

What is a neural network?

A neural network can be thought of as an artificial information processor. It takes input, processes it in some way, and then produces some output. The structure of the network defines how it does the processing, with different structures producing different output. The result of which are networks that can classify images, translate languages and much, much more.

As we’ll soon see, some parts of the network are fixed while others, known as parameters, can change. Our goal is to adjust these parameters in such a way that our network learns to solve a problem. Initially, our network will be very bad at its task, like a child doing calculus, because these parameters are set randomly. But as we iterate through many cycles of testing the network, and updating the parameters based on its response, it will get better over time. Unsurprisingly, this repeated process of testing and updating means that training data is a big part of neural networks.

Let's take a look at what a neural network looks like.

Network Architecture

The original motivation for neural networks are neurons in the human brain, which have a few important features:

Neurons in our brain are connected to each other by a massive network where the output of some neurons can serve as input to others. The strength of connections between neurons can change based on the frequency of use, leading to the popular phrase by Donald Hebb “neurons that fire together, wire together”. The electrochemical potential in a neuron can build up, but the neuron won’t ‘fire’ until the potential passes some threshold.

Let’s see if we can artificially replicate some of this functionality by looking at the building blocks of neural networks, perceptrons.

In the above diagram, we’ve represented two connected neurons, A and C, where the output of neuron A, x, is equal to the input of neuron C. We’ll represent neurons with nodes (the circles) and the connections between neurons with edges (the lines). You can think of a neuron like this: it takes some input, it holds a value (some combination of its input) and then passes that value on as output. So far this model satisfies the first feature listed above. Let’s introduce connection strengths.

We can vary the strength of connections by introducing a connection weight, w. The input to neuron C will now be the output of neuron A, x, multiplied by the weight w. Intuitively, the greater (smaller) the value of w, the stronger (weaker) the connection between the two neurons. This satisfies the second feature. Finally, let’s introduce potential thresholds.

We’ve now introduced another neuron B which has a value of b and a connection weight of -1. B is known as a bias, and we’ll see why soon. The input to C becomes the weighted sum of A and B, that is, w*x + (-1)*b. Next, we apply the step function to the input at C which is defined as f(x) = 1 if x > 0, 0 otherwise.

The step function

To summarise, the value at C becomes 1 if w*x -b > 0, 0 otherwise. Why on earth would we do that? Well, the value at C will equal 0 if w*x < b. In other words, the bias b acts as a threshold that we need to pass in order for the value at C to not be 0. This is exactly like the third feature of neurons discussed earlier! Because of this, we call the step function an ‘activation function’.

There’s only one problem. The vertical part of the step graph at x = 0 means that it's not differentiable. If you don’t know what that means, don’t worry (see the conclusion if you’re not satisfied). All you need to know is that we can approximate the step function with the sigmoid function.

The Sigmoid function

You can think of the sigmoid function as a ‘squishification’ of all possible inputs to fit between 0 and 1. The larger (smaller) x is, the closer sigmoid(x) is to 1 (0).

We can extend our current model to have many neurons feeding input, each of which will have its own weight. Note that only one of these will be a bias. Again, the input becomes the weighted sum (the product of the output of each node by its connection weight) of the neurons before it.

And while we’re at it, why not add a couple more nodes to each layer, and a couple more layers of connectivity? We call the layers between the first layer and the last ‘hidden layers’. Here, each layer will have only one bias.

We generally start by populating the leftmost layer of neurons and move ‘forward’ through the network by calculating the value of each neuron in the next layer, and so on. Finally, we can compute the value of the neurons in the output layer.

As we said before, there are some fixed features and some parameters in our network. The general structure, that is, the number of layers, the number of nodes in each layer and the activation function are fixed. Based on how we move forward through the network, the value of each neuron is deterministic given the neurons and the weights that precede it. Therefore the only thing we can change, our parameters, become the weights of the connections between neurons.

Now that we understand what a network is, let’s look at how we can use it to solve a problem.

How it ‘learns’

We’ll take a look at one of the most famous machine learning tasks, identifying handwritten images.

The general process for learning is this:

Define a network Pass an image into the network (input) The network will predict the label of the image (output) Use the prediction to update the network in such a way that it ‘learns’ Return to step two and repeat

Let’s assume the images are each 28x28 (784) pixels, and since they are grey-scaled, the value of each pixel ranges from 0 (black) to 1 (white). In order to train the network, we need training data in the form of images and their associated label.

The first layer of our network will represent the data; it’s how we feed a data point (an image) into our network. There will be 784 neurons in the first layer (plus a bias) and the value of each neuron will be the value of a pixel from the training image. The last layer in the network will represent the output; the model’s prediction for the label of the image. There will be 10 neurons in this layer, and the closer the value in neuron i is to 1, the more the model thinks the image has label i.

Initially, we set the weights of the graph to random values, which is why the initial predictions won’t be very good. The choice for the number of hidden layers and the number of neurons in each is a challenging problem to solve, which we’ll skip over. For educational purposes let’s just assume there is 1 hidden layer with 10 nodes, and look at an example.