How Do Neural Networks Learn?
By: Moral Robots
Tags: Neural Networks, Artificial Intelligence, Artificial Neural Network, AI, Programming

How Do Neural Networks Learn?

Understand AI… without the math!

Photo by Element5 Digital on Unsplash

In the previous two posts, we saw what a neural network is and even how we can simulate one inside a spreadsheet. Feel free to go back to have a refresher, if you like! You can also get the whole series as a book!

Let us now have a closer look at how a neural network can learn by itself. After all, this is the main attraction of neural networks over conventional computer programs.

Synaptic weights

Image by the author

Imagine again that you have a neural network like this one.

This is the same that we used in our examples before. But now imagine that it is not trained at all. So, in the beginning, its synaptic weights will be initialised to random values. Clearly, then, the behaviour of the network will be random too. When you apply some input pattern, you will get some random output pattern.

Now, how do we make this neural network learn? Let’s say we wanted to teach it the or function. What we would do is take the first line of our truth table. The first line says that if the inputs are 0 and 0, then the output should be 0 too:

Input A Input B Output

------- ------- ------

0 0 0

0 1 1

1 0 1

1 1 1

----------------------

Now we set both inputs to 0, and we tell the network to start changing the weights until the output is also 0. In the beginning, as we saw, the output will be a random number. Now the network will change one of the weights by a little bit and look at what happens to the output. Will the output change, or will it not change? If the output does not change, then the network will try to increase or decrease the synaptic weight a little more. Then it will try the same with the other synaptic weights until finally, it reaches a combination of synaptic weights that brings the output to the desired value of 0.

The activation function

You can already see now why our threshold activation function was not a particularly good idea. If you have a binary activation function like this, which can only jump from 0 to 1 and back, then there is only one particular change to the synaptic weights that will completely flip the output from 0 to 1 or from 1 to 0. Every other change will be ignored as long as the sum of the inputs does not cross the threshold value. This makes it difficult for the neural network to guess whether particular small changes in the synaptic weights represent any progress towards reaching the desired output. It is a little like someone being blindfolded and then trying to grab a particular object in the room. Being blindfolded, the player of this game does not have any idea whether a step in one direction brings him closer to the object he wants to grab or not. Every step he’ll take in that room is just a random step that either succeeds or fails in terms of grabbing the object. It would be much better if the player had some feedback. For example, a second player telling him how close he is to the object. Then the player could make successive steps that bring him closer to the object, and he would notice when particular steps are not useful in approaching the object.

In order to model this with our neural networks, we can change the so-called activation function to be not a simple threshold but a function that continuously maps its inputs onto an output value. We could then calculate the difference between the actual output of the neuron and the desired output, and we would always know whether particular changes in the synaptic weights are bringing us closer to the desired value or not, instead of having the output suddenly jump to the desired output value or do nothing.

Training a network

So, let us now assume that we have such an activation function, for example, this one:

Sigmoid function. Source: Wikipedia

This is called the sigmoid function because it looks like the letter S, which in Greek is called sigma. Think of it like this: the horizontal axis is what comes into the neuron as the sum of its weighted inputs. The vertical axis determines what the neuron will output to its axon, and what, therefore, the neurons that come after it will see as an input.

The sigmoid function (or any other activation function) has the important property of normalising all output values of neurons to a range of 0 to 1. Since every neuron can have multiple inputs, if we didn’t do that and just added up the inputs, the results would possibly be greater than 1, and they would grow and grow as they walk through the layers of the network, and neurons keep adding up their inputs. This would make it difficult to handle the synaptic weights because some synaptic weights would be multiplied by very big numbers, and others by very small numbers. Mapping all outputs to a range between 0 and 1 solves this problem.

The activation function has the important property of normalising all output values of neurons to a range of 0 to 1.

Now let us again apply the input values 0 and 0 (as in the first line of our logical or truth table) and let the network change the random weights just a little bit. Now the network can observe the output value, which it knows should become 0. With every change of the synaptic weights, the network would know whether it is making progress in the right direction or not. After a while of changing the weights this way and that, the network would manage to approach a 0 output value.

Training more patterns

So now we have trained the network to correctly classify the first of our patterns: the first line in the logical or truth table. We can now go for the second line. Again, we apply the input values, in this case, 0 and 1, to the input side of the neuron, and we tell the neural network that the desired output value, in this case, is 1.

The neural network will again start changing the weights slightly so that it reaches the desired output value. Of course, this will now probably destroy the result we had before. By training the network with a second pattern, we might erase the previous training of the first pattern in the truth table. So now the network has a more difficult task: it has to try all of the patterns we have shown it up to this point and try to adjust the weights in such a way that they can all be satisfied at the same time. You can see that, as you add more and more patterns, this process will take more and more time because it will be more difficult to set the weights in such a way that previous knowledge is not erased when the network learns a new line of our truth table (i.e. a new pair of input-output patterns).

This is precisely why training a neural network takes a long time. The network has to go again and again through the patterns and adjust all the synaptic weights one after the other until it can produce all the desired output patterns. Eventually, the network will be able to adjust the weights in the right way, and all the patterns will have been learnt.

Since our activation function is not binary any more, but a continuous function, it is probable that the network will not reach exactly an output of 1 or 0 for each pattern. It might only get an output of 0.9 for a 1, or 0.1 for a 0.

This is still fine with us because we know that we only expect 0 and 1 as output values, so it is clear that the value of 0.1 has to mean 0 and that a value of 0.9 has to mean 1.

The goal of training a neural network is to minimise the error.

The difference between the value that comes out of an untrained neural network and the desired output value of a well-trained network is called the error of the network. So we can say that the goal of training a neural network is to minimise the error. The trainer of the network can decide which error is considered as successful learning. We might be happy with a value of 0.9 instead of a 1. Or we might want a value of 0.99, perhaps because we have to distinguish not only between 0 and 1 but also between 0.8, 0.9 and 1.0 in our outputs, where each of these values means a different thing to our application. If we train the network long enough, we can get smaller and smaller error values. It depends entirely on the application, what error value we consider acceptable. What we are doing is essentially a trade-off between training time and accuracy. If we want the network to be more accurate, we have to train it longer. If we want to cut down on the training time, we can do it by accepting greater error values.