Activation Functions in Artificial Neural Networks
By: Joshua Payne
Tags: Machine Learning, Activation Functions, Neural Networks, Artificial Intelligence

This article assumes a decent understanding of neural networks and backpropagation— for background, check this out.

In all neural networks, the core architecture consists of neurons within several layers; input, hidden, and output. As data is propagated throughout the network, weights adjust their values and help the net make a final prediction of some kind.

An example of a feedforward neural network

Weights and biases are commonly known for nets, but less popular is the concept of activation functions.

In this article, I’ll discuss what these functions do, different types of them, and their use cases. Let’s go!

Firstly… what are activation functions?

Activation functions are functions we apply to the input of our node — our ‘input’ being the weighted sum of all of the neurons’ inputs with the bias.

That may sound a little confusing, so let’s visually break it down.

Each input node, x, have respective weights that’re multiplied to their values. They’re all summed together along with the bias (b) value. Then, our activation function, f, is applied to that value.

Our weighted sum and bias our considered the neuron’s input, and are passed into our node where it undergoes the activation function. This produced value is considered the output of our neuron.

Ok, makes sense … why do we need them?

The main benefit and core reason for activation function usage is that it introduces non-linearity to our neuron’s output, allowing us to work with complex and large data. This means we can learn and make sense of complicated data.

Without activation functions, our outputs would entirely be linear — and linear equations significantly limit what our neural network is capable of.

By enabling non-linearities, we’re able to map and work with non-linear functions. Through being able to map inputs to outputs non-linearly, we can learn more complex things from our data.

The sigmoid activation function — clearly nonlinear

Activation functions make our neural networks more powerful!

Let’s now dive into the different types of activation functions!

There are several different kinds of activation functions, some applied in different areas or layers than others. Let’s unpack a few!

Sigmoid

When a sigmoid function is used, a value between 0 and 1 inclusive is outputted. The input, x, is passed through this equation to produce the output.

Here, x is the negative exponent for base e, representing Euler’s number (2.718…). This value is then added to 1 and becomes the denominator of the fraction, with 1 as the numerator.

Because this equation is a fraction, we receive our decimal output.

When our input is really big, e^-x becomes 0 due to how small 1/e^x would be. For example, if x was 20000, e^-x is equivalent to 1/e²⁰⁰⁰⁰. This solves to a number so small, a calculator will say zero. But remember, our denominator adds one to this variable. This means that we have 1/(1 + 0), aka 1/1, aka 1.

That’s why large inputs (x) produce one!

For super low negative values, like -20000, e^-x becomes e²⁰⁰⁰⁰. This value is the denominator, so the fraction would look like 1/(1 + e²⁰⁰⁰⁰). This extremely small number resolves to 0 on a calculator.

That’s why very low inputs (x) produce zero!

There is, however, a downside to this function that’s largely responsible for its low usage. Because the Sigmoid function outputs a number between 0 and 1, it isn’t zero-centered — it’s 0.5 centered. This makes optimization of a neural network harder than necessary. Additionally, it falls victim to the vanishing gradient problem, more of which will be discussed later.

Tanh (Hyperbolic tangent)

As you can see, this function and the sigmoid are very visually similar. However, differently, tanh goes from -1 to 1 inclusive.

As implied by the name, we use trigonometry with this function. The math is more complex, so just remember that using tanh we get a number between -1 and 1.

With the hyperbolic tangent function, we avoid the issue of not being zero-centered through having an output within the range of -1 and 1. Now, we are zero-centered, so we avoid issues with optimization. However, tanh’s exponential operations make it computationally expensive, and it still doesn’t overcome the vanishing gradient problem. This is why we now rarely use this function.

An issue with both of these functions as well is their saturation — high numbers are automatically snapped to 1.0 and very low numbers are snapped to 0.0. This means we can’t learn as much from our data when high inputs of different values all become the same thing — 1.0. And vice versa. These functions are only sensitive to changes around their midpoint.

Instead, we use ReLU, which avoids these issues!

ReLU (Rectified Linearity Units)

Visually, it’s easy to understand what’s happening here. Any input values under 0 are activated and become zeroes, whereas nothing is changed for input values over 0.

In a sense, it’s “half linear” — positive values remain the same whereas negative ones are snapped to zero.

How does this work mathematically? Here’s the formula for the function.

If x is our input and is negative, the output will be 0. This is because 0 is more than any negative value, and our max function looks for the highest value and outputs it. Therefore if x is negative, ReLU will output 0 as it’s the maximum value. If x is positive, this will be the output.

The issues with tanh and sigmoid are avoided with the ReLU function, which is why it’s so commonly used today. For instance, the vanishing gradient problem. This is when gradients shrink as the network backpropagates to values so small that weights are barely updated in the initial layers. As an example, sigmoids have a maximum slope of 0.25, so gradients get smaller as they’re multiplied with these values. However, with ReLU, the slope is always 1 if above 0, so gradients don’t get smaller. This means ReLU avoids the problem of vanishing gradients!

Additionally, because the function simply looks for a maximum number between the input and 0, it’s very computationally inexpensive.

However, the limitation with this activation function is that it should only be used in the hidden layers of neural networks. And, they’re often ineffective in RNNs.

For our output layers, we often use a sigmoid or softmax function for classification problems. Softmax functions produce a probability distribution of the input belonging to each class, whereas sigmoids are used more with binary classification. This is for when we have two classes, like Cat or Dog. Say we represented our classes numerically — with ‘cat’ as 0 and ‘dog’ as 1. Because sigmoid returns a value between 0 and 1 inclusive, we know whichever number the output is closer to is what the network predicts. For example, if the sigmoid in the output layer of this net outputted a 0.4, it predicted the image was a cat. Whereas if it outputted 0.79, it thinks the image is a dog. I go more into how this network would be technically built here.

For regression applications, we’d stick with linear functions.

While there are tons of different activation functions, today we covered the three most common. I encourage you to do more reading into the vanishing gradient problem, and other activation functions! Thanks for reading!