What is an Activation Function?
By: Apoorv Yadav
Tags: Softmax, Sigmoid, Relu, Activation Functions, Data Science

Many more activation functions…

I. Linear Function y = ax :

A function that gives output proportional to x. This way we are not constrained to binary classifications.

If you are familiar with the concept of differentiation, you can observe that the gradient here is constant and if your algorithm depends on gradient, this function might trouble you.

Also, when you have a lot of layers stacked over each other, the input might get very large at one point.

Linear Function

II. Sigmoid Function:

It looks like a smoothed step function.

Sigmoid Function | source: https://commons.wikimedia.org/wiki/File:Logistic-curve.svg

Mathematically, It can be represented as:

This function takes input of range [-inf. +inf ] and outputs Y in range [ 0, 1]

If you carefully observe the graph, you will notice at least two things:

It tries it’s best to bring activation on either side for values between -2 to 2 On the other hand, Y closer to the end responds very less to X. In mathematical terms, we can say, the gradient is very less and this gives rise to the problem of vanishing gradients (gradient close to 0) when you have a deep network.

Sigmoid Function

III. Tanh Function:

It looks similar to the sigmoid function we discussed above. It is zero-centric and very popular to be used for binary classifications. This function maps a real valued number to the range [ -1, 1 ].

Tanh vs Sigmoid | source: https://stats.stackexchange.com/questions/330559/why-is-tanh-almost-always-better-than-sigmoid-as-an-activation-function

It can be observed that it’s steeper than sigmoid. However, It still possesses the issue of vanishing gradients. We can see how Tanh and Sigmoid are related to each other below:

f(x) here is tanh function in terms of sigmoid function

Tanh Function

IV. ReLU :

With time, we got a new and now very popular activation function, Relu (Rectified Linear Unit). It gives the output if input is greater than zero.

ReLu | Despite the name and the graph given, This activation is non-linear in nature | source : https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#/media/File:Rectifier_and_softplus_functions.svg

It eradicates the problem of vanishing gradients we talked about before.But is it like a magic lamp which can have no disadvantages? Since Y bounds to the range[ 0, -inf ), It can blow up the activation, just like the linear function.

Now we may face an issue for X<0. For such x, we get our gradient 0 and weights would not be adjusted for such nodes. This might lead to the dying ReLU problem.We have the next one for this issue.

Rectified Linear Unit (ReLU)

V. Leaky ReLU:

Its like a ReLU, with some changes for x<0. Here we do not completely cut off the input less than zero. It helps in case of dying ReLU problem.

Leaky ReLU | source: https://mc.ai/activation-functions-in-deep-neural-networks/

You can vary the component for x<0 i.e. 0.01 according to your needs as long as it keeps the gradient non zero.

Both of the ReLu are fairly less cheap in computation than sigmoid and tanh. Thus you can use them in deeper and larger networks.

Leaky ReLU

VII. Okay this is last :), Softmax Function:

This function is widely used at the output layer. It is used to convert the output values to a range which can be comprehended as probabilities for each class/type. It is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponential of the input numbers.

Softmax Function | It helps us to convert the output values into a normalized one so as treat them as Probabilities.