Neural Networks Demystified
By: Joshua Payne
Tags: Machine Learning, Artificial Intelligence

Neural Networks Demystified

Computers are now simulating how the brain works- how?

Introduction

Imagine you were in an exam for a computer science class. You’re soon to apply for universities,, and therefore are determined to get excellent grades on all of your work. You’re completely fluent in three programming languages and know all the course material, so you’re pretty sure you’ll do fine.

Exams — Often a Tale of Dread and Procrastination

You’re halfway through the coding problems, and you then read:

*^@%!???? The entire test is out of 40, and half of those marks come from this beyond difficult question. I mean, how would you even approach whipping up something capable of sorting thousands of images in a couple hours?

This is where machine learning comes into play, an emerging technology capable of tasks like recognizing butterflies on their own. In fact, way more, like even detecting cancer or driving cars.

Uh, ok? So how does any of that work, or for that matter, make sense?

Background

Well, the basis for machine learning, a subset of artificial intelligence, is neural networks. As the name suggests, they take inspiration from how the human brain operates.

A Model of a Neural Network

Artificial neurons simulate real neurons, and the entire network is capable of making predictions and drawing conclusions through processing inputted data. Today, I’ll be discussing feedforward neural networks — where inputs and outputs strictly move to the right. This’ll make more sense in a bit.

Neurons

Neurons are the fundamental building block of neural networks. Think cells for organisms or atoms for matter. They are necessary for your model. They produce values that represent predictions/decisions through computing inputs.

Inputs, Weights and Biases

Before talking about the types of neurons, it’s important to discuss the important concepts that accompany them. Inputs, weights and biases are what make a group of neurons an actual network. They are all necessary for a neuron to compute and then output something.

Inputs (x)

Inputs are simply data points fed into a neuron. Because neural networks are mathematical models at their core, inputs are represented by a numerical value. For example, the pixel values of a greyscale butterfly photo.

Weights (w)

Then, weights come into play. Essentially, they represent the importance of an input, and multiply the input by a select number. When a data point fed into a neuron is very important, they’re in a sense heavily weighted (a high weight value).

Thresholds and Bias (b)

So far, we have the data we’re inputting into the neuron and weight, what we multiply the former by. However, there is one more thing to learn. Assuming we’re using perceptron neurons (more on that later), our neurons look something like this.

Output 0 if wx < threshold Output 1 if wx ≥ threshold

The threshold refers to the number necessary for our weighted input(s) to be equal or over to for the perceptron to output a 1. For clarification, perceptrons output 1’s when confident in a prediction and 0’s when not.

If we were working with watching a football game, our input could be how much the user likes the teams playing from 1–5. This seems like an obvious indicator of whether they should watch or not, so we’d give it a high weight of 2. And, the perceptron should only output “1”, AKA you should watch, if their favourite teams are playing. As such, the threshold is 6 (need to like the team > 3). This would look like this:

Output 0 if 2x < 6 Output 1 if 2x ≥ 6

However, this inequality can be simplified. Introducing biases! By moving thresholds to the other side and switching them to positive/negative (rule of simplifying equations) values, we now have a value called bias. This can be shortly explained as a value representing how easy it is to get above zero. Biases adjust the weighted sum of the inputs as an added parameter.

So in our example, we’d now have:

Output 0 if 2x — 6 < 0 Output 1 if 2x — 6 ≥ 0

With our weight being 2 and the bias -6. And as you may see, the inputs, weights and biases can be modelled with the equation wx + b. This is the same form as a linear equation!

Once you get more than one input, you get a new weight as well. Multiple inputs are often used in neural networks.

The output of a neural network before it’s been activated… keep reading!

Activation Functions

However, outputs of a neuron can get more complicated than a 1 or a 0. This happens when an activation is applied. Activations alter the output of a neuron.

We’ll go into two common types of activation functions, and each has their own type of neuron.

Perceptrons

An outdated but important type of neuron is the perceptron. As aforementioned, after weighing and computing inputted data, it’s able to produce a 0 or 1 value.

The function used here is the step function. There’s little curvature, and if the value of x after applied weight and bias is under 0, as you can see on the graph, the output will be 0. Whereas if the value of x after the bias and weight is 0 or above, you can see that the output will be a 1. We consider x = 0 as 1 for simplicity.

Sigmoid Neurons

Sigmoid neurons work a little differently, and provide a value between 0 and 1 inclusive.

As the name suggests, the function used is the sigmoid function (however, also called the logistic regression function). As presented in the photo, the graph is much more smoothed out than the step function. This is how we’ll get values in between 0 and 1.

Sigmoid neurons are used because we can get more descriptive outputs than 0 or 1. Perceptrons are harsh by nature — despite how close you may be, you could get a 0 outputted because the weighted input was 0.00002 less than 0. Because of this binary system, where everything is strictly constrained with 0 and 1, extremely close equations would be bundled with equations that are massively off due to the fact that all of their perceptrons would produce 0. Sigmoid gives us more accuracy, so we can see just from the number output whether a weighted input was close or not (0.213, 0.967, etc.).

Basic Architecture

The structure of a neural network is quite simple. They are comprised of three different types of layers — input, hidden and output.

Input Layer

The input layer refers to the leftmost column or group of neurons (circles) in the photo. They’re the first layer, in that data first starts here. This is because the data are “inputs”, and the input layer is responsible for feeding it to other neurons in the first hidden layer.

Hidden Layers

That brings us to the next type — hidden layers. Hidden layers sound mysterious, but really are just layers between the input and output layers. They don’t take in the initial inputted data/produce the final output.

Output Layers

And finally, the output layers. Here, the official outputs of the neural networks are released, and their numerical value represents something. These outputs are all stored in neurons.

Typically, each neuron in the output layer is used as an individual component, representing something.

So back to Problem #20, if you were classifying butterflies with perceptrons, each output neuron would correspond to the confidence the network has that the initial photo is a certain species. Let’s say Monarch, Painted Lady, Cabbage White and Red Admiral. If the neural net was saying “we think this image is a Monarch butterfly”, the output neuron for the monarch species would have a value of 1 and the other species’ 0.

Species found through a quick google search.. I’m no expert!

This would be a feedforward neural network — photos are initially fed into the input layer on the left, and then the output layer on the right produces final values.