How Neural Networks Work
By: Moral Robots
Tags: Neural Networks, Artificial Intelligence, AI, Computer Science, Programming

How Neural Networks Work

Understand what’s happening inside a neural network

Image by the author

This article is part of a series that explains neural networks without the math. The first part is here. The next part is here. You can also get the whole series as a book.

The basic structure of a simulated neuron

In the first part of this introduction, we talked about what an artificial neuron is. Artificial neurons are inspired by biological nerve cells, and transmit a signal from their “input side” (dendrites) to their “output side” (the axon):

Image by the author

The axon at the end divides into terminal branches, which connect to other dendrites (of many other such neurons), creating a network of billions of connected neurons. What makes an artificial neuron more than just a connecting cable is the ability of the neuron to decide whether it should actually propagate a signal down its axon or not.

In artificial neurons, the “cell body” (usually just a function in a programming language) will first weigh and then add up its inputs and, if they add up to more than a set threshold value, fire a signal down its axon. If the sum of the weighted inputs doesn’t exceed the threshold value, the neuron will stay silent and not fire a signal to the neurons that are connected at its terminal branches:

Image by the author

By connecting many such units in multiple layers with each other, we get an artificial neural network:

Image by the author

The weights between the inputs and the summing function can change, and represent the actual “learned” content of the network. All the information that the network has is stored in these weights. We will see in a moment how this works in detail. Read on!

A neuron for a logical ‘or’

Let us have a look at a few very simple neural networks so that we can see what happens inside them. These networks don’t do anything overly exciting, but they are easy to understand, and they do demonstrate the basic principles behind neural networks.

First, we want to have a look at a neuron that works as a logical or. The logical or operation has two inputs, A and B, and one output.

Consider a sentence like “If it is Saturday or very hot, I will go to the beach.” The two inputs are “it is Saturday” and “it is very hot”. The output is “I will go to the beach”. Each of the two input conditions can be true or false independently of the other. So it might be Saturday and very hot, or Saturday and not very hot, or not Saturday and not very hot. All in all, we have four possible combinations of input patterns, and for each, we have one desired output.

If it is not Saturday and not very hot, I will not go to the beach. If it is not Saturday but very hot, I will go to the beach. If it is Saturday but not very hot, I will go to the beach. If is Saturday and very hot, I will go to the beach.

So in three of the four cases, I will go to the beach (so my output will be true). In one of the four, namely, if both conditions are false, then I will not go to the beach. We can express the truth of the conditions either with the symbols T and F (for “true” and “false”) or just with 1 (true) and 0 (false).

To summarise: if either A or B is true, then the output of this logical operation should also be true. The output will also be true if only one of A and B is true. The output will only be false if both A and B are false. The following table shows what we want to achieve.

Input A Input B Output

------- ------- ------

0 0 0

0 1 1

1 0 1

1 1 1

----------------------

In order to build this as a neural network, we will need just one neuron. This one neuron has two inputs: one for the value of A and one for the value of B. Remember that between each input and the neuron is also a synaptic weight, which is shown in the diagram below as a little red circle with a number in it. This number is the factor by which the synapse will multiply its input before it passes it on to the neuron.

The neuron itself will add up its two inputs, and it will fire if the sum of the inputs is equal to or greater than 1. We say that the neuron has a threshold of 1.0. You can see this on the right side of the neuron below, right under the arrow that represents the neuron’s “axon”.

Now, the question is: how can we set the values of the synaptic weights so that this neuron fulfils the function of a logical or as just described?

Image by the author

Obviously, if each synaptic weight has a value that is equal to or greater than 1, then each one of the inputs A and B will be able to make the neuron fire. Let’s say A is logically true, which we will express as an input of 1. B is false, which means that the input is 0. Now we have to multiply A (which is 1, because A is true) with the synaptic weight 1.1:

1 × 1.1 = 1.1

We have to multiply B (which is false, i.e. 0) with its synaptic weight (1.1) too, but because B is 0, the result of the operation will be 0. Now the neuron gets one input with the value 1.1, and one input with the value 0. Its threshold is 1; therefore, since one of the inputs is already greater than 1, the neuron will fire and will set its output to 1.

This behaviour is exactly what we wanted! This neuron behaves like a logical or. You can check the other combinations of input values yourself to verify that this neuron would indeed work correctly in all four cases.

By the way, the scary Greek ‘Σ’ letter (pronounced “sigma”) inside the neuron’s body just means “sum”. So the neuron is summing up its inputs at this point and checking whether the sum is greater than the threshold value or not.

Calculating a logical ‘and’

Now let us consider another logical operation, the and. Take an example sentence like “If it is Saturday and very hot, then I will go to the beach.” This is different from before because now both conditions need to be fulfilled at the same time in order for me to go to the beach.

Here is a truth table for an and:

Input A Input B Output

------- ------- ------

0 0 0

0 1 0

1 0 0

1 1 1

----------------------

How do I have to set the synaptic weights so that our neuron now behaves as a logical and?

Clearly, I will need to set the weights so that each one by itself is unable to make the neuron fire. That is, each synaptic weight should be less than the threshold; but the two synaptic weights, when added together, should give us a greater value than the threshold value 1. This means that I can take any value between 0.5 and 0.9 for the synaptic weights. I get the following diagram:

Image by the author

You can easily verify that this will behave like a logical and.

Exclusive or not?

Things become slightly more complicated if I want to create a neuron that encodes a logical exclusive or, or xor. An exclusive or is true only if either A or B is true, but not both. Here is a truth table for the xor operation:

Input A Input B Output

------- ------- ------

0 0 0

0 1 1

1 0 1

1 1 0

----------------------

If you think about it for a moment, you will see that we cannot possibly achieve this result with a single neuron, because the weights would have to be greater than 1 so that each input can trigger the neuron alone and make it behave like an or. But if this was the case, then we could not get the right result for the last line where both inputs are 1, but the result is supposed to be 0.

So here we really need three neurons. One will act as an or and will fire if either one of its inputs is true. The second neuron will only have the job of stopping the output from becoming 1 in the case that both inputs are true. Therefore, one neuron needs to have both synapses set to 1 or 1.1, so that it will fire like an or. The second neuron is actually encoding an and, and if it fires, it will produce a negative output of −2. In this way, in the last line of the truth table, the sum of these two outputs will be less than 1. So we need a third neuron, which will be the output neuron, and this will fire only if the sum of its inputs is greater or equal to 1:

Image by the author

You can imagine that we can make the processing that happens between input and output as complex as we like by adding more and more neurons. Each new layer of neurons enables us to model more complex mappings between input and output.

Thanks for reading! In the next part, we will simulate neurons inside a spreadsheet, to see how exactly they work. Stay tuned.