Dive into Deep Learning
By: Shanika Perera
Tags: Artificial Intelligence, Artificial Neural Network, Machine Learning, Deep Learning, Neural Networks

What is a Neural Network(NN)?

Neural networks are a set of algorithms, modeled after the human brain that is designed to recognize patterns. NN can interpret sensory data through a machine perception, labeling or clustering. The patterns that are recognized are either numerical contained in vectors into which all real-world data images, sound, text or time series must be translated.

Neural networks can classify and cluster data. It works as a clustering and classification layer on top of data that needs to be managed and stored. Neural networks help to group unlabeled data according to similarities among the example inputs, and they classify data when they have a labeled dataset to train on.

Neural networks elements

Node — Just a place where computation happens, patterned on a neuron in the human brain. A node combines input from the data with a set coefficients or weights

Node layer — A row of switches that turn on and off as the input is fed through the network. Each layer’s output is simultaneously the subsequent layer’s input, starting from an initial input layer receiving your data.

Depth — Number of node layers

Neural networks are composed of one input layer and one output layer. It can contain more than one hidden layers. If it contains more than three layers, (including the input and output layer) it qualifies as “Deep” learning.

Activation function

Activation functions are really important for a Neural network to learn and make sense of something really complicated and non-linear complex functional mappings between the inputs and response variable. The main purpose of an activation function is that it converts an input signal of a node in an ANN to an output signal. That output signal is then used as the input of the next layer in the stack.

If an activation function is not available then the output signal would simply be a simple linear function. It decides whether a neuron/ node should be activated or not by calculating the weighted sum and further adding bias with it.

Activation functions available —

Sigmoid — ranges from 0 to 1. Also known as Logistic activation function. Used for binary classification problems.

— ranges from 0 to 1. Also known as Logistic activation function. Used for binary classification problems. TanH — The output ranges from -1 to 1.

— The output ranges from -1 to 1. Softmax — Outputs a vector that represents the probability distributions of a list of potential outcomes. Mainly used for multiclass classification problems.

— Outputs a vector that represents the probability distributions of a list of potential outcomes. Mainly used for multiclass classification problems. ReLu — If the input is a positive number the function returns the number itself and if the input is a negative number then the function returns 0. Used in hidden layers.

Epoch

One epoch is when an entire dataset is passed forward and backward through the neural network only once. The number of epochs is traditionally large, often hundreds or thousands, allowing the learning algorithm to run until the error from the model has been sufficiently minimized. You may see examples of the number of epochs in the literature and in tutorials set to 10, 100, 500, 1000, and larger.

Learning rate

The learning rate (α) is a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are uploaded. It affects how quickly the model can converge to a local minimum.

Types of Neural networks

Feedforward NN

This is one of the simplest types of artificial NN. In a feedforward NN, the data passes through different input nodes until it reaches the output node. The data moves only in one direction in FNN. There are loops or backward propagating in FNN. Feedforward NN is used in face recognition and computer vision tasks.

2. Recurrent NN

This is also known as Long Short Term Memory. In this neural network, the output of a particular layer is saved and fed back to the input. This is used to predict the outcome of the layer. If the prediction is wrong, the system self-learns and works towards making the right prediction during the backpropagation. These type of NN are used in text-to-speech conversion technology.

3. Multilayer Perceptron

A multilayer perceptron has three or more layers. It is used to classify data that cannot be separated linearly. It is a type of artificial neural network that is fully connected. MLP is mainly used in speech recognition and machine translation technologies.

4. Convolutional NN

A convolutional NN uses a variation of the MLP. It contains more than one convolutional layer. Before passing the result to the next layer, the convolutional layer uses a convolutional operation on the input. Due to this convolutional operation, the network can be much deeper but with much fewer parameters. Because of this feature, CNN is used in image and video recognition, natural processing and paraphrase detection.

5. Modular NN

A modular neural network has a number of different networks that function independently and perform sub-tasks. The different networks do not really interact with or signal each other during the computation process. They work independently towards achieving the output.