Title: Introduction to Neural Networks - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/06/introduction-to-neural-networks/

Published Time: 2021-05-06T14:00:00+00:00

Markdown Content:
We’ll study the fundamentals of neural networks in depth. We’ll start with a discussion of artificial neural networks and how they are inspired by the real-life biological neural networks in our own bodies. From there, we’ll review the classic Perceptron algorithm and the role it has played in neural network history.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20348'%3E%3C/svg%3E)

Building on the Perceptron, we’ll also study the _backpropagation algorithm_, the cornerstone of modern neural learning — without backpropagation, we would be unable to efficiently train our networks. We’ll also implement backpropagation with Python from scratch, ensuring we understand this important algorithm.

Of course, modern neural network libraries such as Keras already have (highly optimized) backpropagation algorithms built-in. Implementing backpropagation by hand each time we wished to train a neural network would be like coding a linked list or hash table data structure from scratch each time we worked on a general purpose programming problem — not only is it unrealistic, but it’s also a waste of our time and resources. In order to streamline the process, I’ll demonstrate how to create standard feedforward neural networks using the Keras library.

Finally, we’ll discuss the four ingredients you’ll need when building _any_ neural network.

**Neural Network Basics**
-------------------------

Before we can work with Convolutional Neural Networks, we first need to understand the basics of neural networks. We’ll review:

• Artificial Neural Networks and their relation to biology.  
• The seminal Perceptron algorithm.  
• The backpropagation algorithm and how it can be used to train _multi-layer_ neural networks efficiently.  
• How to train neural networks using the Keras library.

By the time you finish, you’ll have a strong understanding of neural networks and be able to move on to the more advanced Convolutional Neural Networks.

Neural networks are the building blocks of deep learning systems. In order to be successful at deep learning, we need to start by reviewing the basics of neural networks, including _architecture_, _node_ _types_, and _algorithms for “teaching”_ our networks.

We’ll start with a high-level overview of neural networks and the motivation behind them, including their relation to biology in the human mind. From there we’ll discuss the most common type of architecture, **feedforward neural networks**. We’ll also briefly discuss the concept of _neural learning_ and how it will later relate to the algorithms we use to train neural networks.

### **What are Neural Networks?**

Many tasks that involve intelligence, pattern recognition, and object detection are _extremely difficult_ _to automate_, yet _seem to be performed easily and naturally_ by animals and young children. For example, how does your family dog recognize you, the owner, versus a complete and total stranger? How does a small child learn to recognize the difference between a _school bus_ and a _transit bus_? And how do our own brains subconsciously perform complex pattern recognition tasks each and every day without us even noticing?

**The answer lies within our own bodies.** Each of us contains a real-life biological neural network that is connected to our nervous systems — this network is made up of a large number of interconnected _neurons_ (nerve cells).

The word _“neural”_ is the adjective form of _“neuron,”_ and _“network”_ denotes a graph-like structure; therefore, an _“Artificial Neural Network”_ is a computation system that attempts to mimic (or at least, is inspired by) the neural connections in our nervous system. Artificial neural networks are also referred to as _“neural networks”_ or _“artificial neural systems.”_ It is common to abbreviate Artificial Neural Network and refer to them as _“ANN”_ or simply _“NN” —_ I will be using both of the abbreviations.

For a system to be considered an NN, it must contain a labeled, directed graph structure where each node in the graph performs some _simple computation_. From graph theory, we know that a directed graph consists of a set of nodes (i.e., vertices) and a set of connections (i.e., edges) that link together pairs of nodes. In **Figure 1,** we can see an example of such an NN graph.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20496'%3E%3C/svg%3E)

**Figure 1:** A simple neural network architecture. Inputs are presented to the network. Each connection carries a signal through the two hidden layers in the network. A final function computes the output class label.

Each node performs a simple computation. Each connection then carries a _signal_ (i.e., the output of the computation) from one node to another, labeled by a _weight_ indicating the extent to which the signal is amplified or diminished. Some connections have large, _positive weights_ that amplify the signal, indicating that the signal is very important when making a classification. Others have _negative weights,_ diminishing the strength of the signal, thus specifying that the output of the node is less important in the final classification. We call such a system an _Artificial Neural Network_ if it consists of a graph structure (like in **Figure 1**) with connection weights that are _modifiable_ using a learning algorithm.

### **Relation to Biology**

Our brains are composed of approximately 10 billion neurons, each connected to about 10,000 other neurons. The cell body of the neuron is called the _soma_, where the inputs (_dendrites_) and outputs (_axons_) connect soma to other soma (**Figure 2**).

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20348'%3E%3C/svg%3E)

**Figure 2:** The structure of a biological neuron. Neurons are connected to other neurons through their dendrites and neurons.

Each neuron receives electrochemical inputs from other neurons at their dendrites. If these electrical inputs are sufficiently powerful to activate the neuron, then the activated neuron transmits the signal along its axon, passing it along to the dendrites of other neurons. These attached neurons may also fire, thus continuing the process of passing the message along.

The key takeaway here is that a neuron firing is a **binary operation** — **the neuron either _fires_ or it _doesn’t fire_**. There are no different “grades” of firing. Simply put, a neuron will only fire if the total signal received at the soma exceeds a given threshold.

However, keep in mind that ANNs are simply _inspired_ by what we know about the brain and how it works. The goal of deep learning is _not_ to mimic how our brains function, but rather take the pieces that we _understand_ and allow us to draw similar parallels in our own work. At the end of the day, we do not know enough about neuroscience and the deeper functions of the brain to be able to correctly model how the brain works — instead, we take our _inspirations_ and move on from there.

### **Artificial Models**

Let’s start by taking a look at a basic NN that performs a simple weighted summation of the inputs in **Figure 3**. The values _x_1_, x_2_, and, x_3 are the **inputs** to our NN and typically correspond to a _single row_ (i.e., data point) from our design matrix. The constant value 1 is our bias that is assumed to be embedded into the design matrix. We can think of these inputs as the input feature vectors to the NN.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20652%20475'%3E%3C/svg%3E)

**Figure 3:** A simple NN that takes the weighted sum of the input _x_ and weights _w_. This weighted sum is then passed through the activation function to determine if the neuron fires.

In practice, these inputs could be vectors used to quantify the contents of an image in a systematic, predefined way (e.g., color histograms, [Histogram of Oriented Gradients](http://dx.doi.org/10.1109/CVPR.2005.177), [Local Binary Patterns](https://doi.org/10.1109/TPAMI.2002.1017623), etc.). In the context of deep learning, these inputs are the _raw pixel intensities_ of the images themselves.

Each _x_ is connected to a neuron via a weight vector **_W_** consists of _w_1_, w_2_, …, wn_, meaning that for each input _x_ we also have an associated weight _w_.

Finally, the _output node_ on the right of **Figure 3** takes the weighted sum, applies an activation function _f_ (used to determine if the neuron “fires” or not), and outputs a value. Expressing the output mathematically, you’ll typically encounter the following three forms:

• _f_(_w_1_x_1 +_w_2_x_2 \+ _···_ +_wnxn_)  
• _f_(∑_ni_\=1_wixi_)  
• Or simply, _f_(_net_), where _net_ \= ∑_ni_\=1_wixi_

Regardless of how the output value is expressed, understand that we are simply taking the weighted sum of inputs, followed by applying an activation function _f_.

### **Activation Functions**

The most simple activation function is the “step function,” used by the Perceptron algorithm.

![Image 5:  f(\textit{net}) \begin{cases} 1 & \textit{if net} > 0 \\ 0 & \textit{otherwise} \end{cases} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20153%2051'%3E%3C/svg%3E)

As we can see from the equation above, this is a very simple threshold function. If the weighted sum ∑_n__i_\=1_w__i__x__i_ _\>_ 0, we output 1, otherwise, we output 0.

Plotting input values along the _x_\-axis and the output of _f_(_net_) along the _y-axis,_ we can see why this activation function received its name (**Figure 4**, _top-left_). The output of _f_ is always zero when _net_ is less than or equal zero. If _net_ is greater than zero, then _f_ will return one. Thus, this function looks like a stair step, not dissimilar to the stairs you walk up and down every day.

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20624%20475'%3E%3C/svg%3E)

**Figure 4:** _Top-left:_ Step function. _Top-right:_ Sigmoid activation function. _Mid-left:_ Hyperbolic tangent. _Mid-right:_ ReLU activation (most used activation function for deep neural networks). _Bottom-left:_ Leaky ReLU, variant of the ReLU that allows for negative values. _Bottom-right:_ ELU, another variant of ReLU that can often perform better than Leaky ReLU.

However, while being intuitive and easy to use, the step function is not differentiable, which can lead to problems when applying gradient descent and training our network.

Instead, a more common activation function used in the history of NN literature is the sigmoid function (**Figure 4**, _top-right_), which follows the equation:

**(1)** ![Image 7: \mbox{ } \qquad t = \sum\limits_{i=1}^{n} w_{i}x_{i} \qquad s(t) = 1 / (1 + e^{-t})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20238%2036'%3E%3C/svg%3E)

The sigmoid function is a better choice for learning than the simple step function since it:

1.  Is continuous and differentiable everywhere.
2.  Is symmetric around the _y_\-axis.
3.  Asymptotically approaches its saturation values.

The primary advantage here is that the smoothness of the sigmoid function makes it easier to devise learning algorithms. However, there are _two big problems_ with the sigmoid function:

1.  The outputs of the sigmoid are not zero centered.
2.  Saturated neurons essentially kill the gradient, since the delta of the gradient will be extremely small.

The hyperbolic tangent, or _tanh_ (with a similar shape of the sigmoid) was also heavily used as an activation function up until the late 1990s (**Figure 4**, _mid-left_): The equation for _tanh_ follows:

**(2)** _f_(_z_) = _tanh_(_z_) = (_ez −e−z_) _/_ (_ez_ +_e−z_)

The _tanh_ function is zero centered, but the gradients are still killed when neurons become saturated.

We now know there are better choices for activation functions than the sigmoid and _tanh_ functions. Specifically, the work of [Hahnloser et al. in their 2000 paper, _Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit_](https://doi.org/10.1038/35016072), introduced the **Rectified Linear Unit (ReLU)**, defined as:

**(3)** _f_(_x_) = _max_(0_, x_)

ReLUs are also called “ramp functions” due to how they look when plotted (**Figure 4**, _mid-right_). Notice how the function is zero for negative inputs but then linearly increases for positive values. The ReLU function is not saturable and is also extremely computationally efficient.

Empirically, the ReLU activation function tends to outperform _both_ the sigmoid and _tanh_ functions in nearly all applications. Combined with the work of Hahnloser and Seung in their follow-up 2003 paper [_Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks_](http://dx.doi.org/10.1162/089976603321192103), it was found that the ReLU activation function has stronger biological motivations than the previous families of activation functions, including more complete mathematical justifications.

As of 2015, ReLU is the _most popular_ activation function used in deep learning [(LeCun, Bengio, and Hinton, 2015)](https://doi.org/10.1038/nature14539). However, a problem arises when we have a value of zero — _the gradient cannot be taken_.

A variant of ReLUs, called _Leaky ReLUs_ [(Maas, Hannun, and Ng, 2013)](https://sites.google.com/site/deeplearningicml2013/relu_hybrid_icml2013_final.pdf?attredirects=0&d=1) allow for a small, non-zero gradient when the unit is not active:

![Image 8: f(\textit{net}) = \begin{cases} \textit{net} & \textit{if net} >= 0 \\ \alpha \times \textit{net} & \textit{otherwise} \end{cases}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20229%2051'%3E%3C/svg%3E)

Plotting this function in **Figure 4** (_bottom-left_), we can see that the function is indeed allowed to take on a negative value, unlike traditional ReLUs which “clamp” the function output at zero.

Parametric ReLUs, or PReLUs for short [(**He, Zhang, Ren, and Sun, 2015**)](https://arxiv.org/abs/1512.03385v1), build on Leaky ReLUs and allow the parameter _α_ to be learned on an activation-by-activation basis, implying that each node in the network can learn a different “coefficient of leakage” separate from the other nodes.

Finally, we also have _Exponential Linear Units (ELUs)_ introduced by [Clevert et al. in their 2015 paper, _Fast and Accurate Deep Learning by Exponential Linear Units (ELUs)_](https://arxiv.org/abs/1511.07289):

![Image 9: f(\textit{net}) = \begin{cases} \textit{net} & \textit{if net} >= 0 \\ \alpha \times (\textit{exp}(\textit{net}) - 1) & \textit{otherwise} \end{cases}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20306%2051'%3E%3C/svg%3E)

The value of _α_ is constant and _set when the network architecture is instantiated_ — this is unlike PReLUs where _α_ is learned. A typical value for _α_ is _α_ \= 1_._0\. **Figure 4** (_bottom-right_) visualizes the ELU activation function.

Through the work of Clevert et al. (and my own anecdotal experiments), ELUs often obtain higher classification accuracy than ReLUs. ELUs rarely, if ever perform worse than your standard ReLU function.

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20672%20475'%3E%3C/svg%3E)

**Figure 5:** An example of a feedforward neural network with 3 input nodes, a hidden layer with 2 nodes, a second hidden layer with 3 nodes, and a final output layer with 2 nodes.

### **Which Activation Function Do I Use?**

Given the popularity of the most recent incarnation of deep learning, there has been an associated explosion in activation functions. Due to the number of choices of activation functions, both modern (ReLU, Leaky ReLU, ELU, etc.) and “classical” ones (step, sigmoid, _tanh_, etc.), it may appear to be a daunting, perhaps even overwhelming task to select an appropriate activation function.

However, in nearly all situations, I recommend starting with a ReLU to obtain a baseline accuracy (as do most papers published in the deep learning literature). From there you can try swapping out your standard ReLU for a Leaky ReLU variant.

My personal preference is to start with a ReLU, tune my network and optimizer parameters (architecture, learning rate, regularization strength, etc.), and note the accuracy. Once I am reasonably satisfied with the accuracy, I swap in an ELU and often notice a 1_−_5% improvement in classification accuracy depending on the dataset. Again, this is only my anecdotal advice. You should run your own experiments and note your findings, but as a general rule of thumb, start with a normal ReLU and tune the other parameters in your network — then swap in some of the more “exotic” ReLU variants.

### **Feedforward Network Architectures**

While there are many, _many_ different NN architectures, the most common architecture is the _f__eedforward network_, as presented in **Figure 5**.

In this type of architecture, a connection between nodes is _only allowed_ from nodes in layer _i_ to nodes in layer _i_+1 (hence the term, _feedforward_). There are no backward or inter-layer connections allowed. When feedforward networks include _feedback connections_ (output connections that feed back into the inputs) they are called **recurrent neural networks**.

We focus on feedforward neural networks as they are the cornerstone of modern deep learning applied to computer vision. Convolutional Neural Networks are simply a special case of feedforward neural networks.

To describe a feedforward network, we normally use a sequence of integers to quickly and concisely denote the number of nodes in each layer. For example, the network in **Figure 5** is a _3-2-3-2_ feedforward network:

**Layer 0** contains 3 inputs, our _x__i_ values. These could be raw pixel intensities of an image or a feature vector extracted from the image.

**Layers 1 and 2** are _hidden layers_ containing 2 and 3 nodes, respectively.

**Layer 3** is the _output layer or the visible layer_ — there is where we obtain the overall output classification from our network. The output layer typically has as many nodes as class labels; one node for each potential output. For example, if we were to build an NN to classify handwritten digits, our output layer would consist of 10 nodes, one for each digit _0-9_.

### **Neural Learning**

Neural learning refers to the method of modifying the weights and connections between nodes in a network. Biologically, we define learning in terms of Hebb’s principle:

> _When an axon of cell A is near enough to excite cell B, and repeatedly or persistently takes place in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased._
> 
> _—_ [Donald Hebb (1949)](https://www.amazon.com/Organization-Behavior-Neuropsychological-Theory/dp/0805843000)

In terms of ANNs, this principle implies that there should be an increase in the strength of connections between nodes that have similar outputs when presented with the same input. We call this _correlation learning_ because the strength of the connections between neurons eventually represents the correlation between outputs.

### **What are Neural Networks Used For?**

Neural Networks can be used in both supervised, unsupervised, and semi-supervised learning tasks, provided the appropriate architecture is utilized, of course. Common applications of NN include classification, regression, clustering, vector quantization, pattern association, and function approximation, just to name a few.

In fact, for nearly every facet of machine learning, NNs have been applied in some form or another. We’ll be using NNs for _computer vision_ and _image classification_.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

### **A Summary of Neural Network Basics**

Today, we reviewed the basics of Artificial Neural Networks (ANNs, or simply NNs). We started by examining the biological motivation behind ANNs and then learned how we can _mathematically define_ a function to mimic the activation of a neuron (i.e., the activation function).

Based on this model of a neuron, we are able to define the _architecture_ of a network consisting of (at a bare minimum), an _input layer_ and an _output layer_. Some network architectures may include _multiple hidden layers_ between the input and output layers. Finally, each layer can have one or more nodes. Nodes in the input layer _do not_ contain an activation function (they are “where” the individual pixel intensities of our image are inputted); however, nodes in both the hidden and output layers _do_ contain an activation function.

We also reviewed three popular activation functions: _sigmoid_, _tanh_, and _ReLU_ (and its variants).

Traditionally the sigmoid and _tanh_ functions have been used to train networks; however, since [Hahnloser et al.’s 2000 paper](https://doi.org/10.1038/35016072), the ReLU function has been used more often.

In 2015, ReLU is _by far_ the most popular activation function used in deep learning architectures [(LeCun, Bengio, and Hinton, 2015)](https://doi.org/10.1038/nature14539). Based on the success of ReLU, we also have _Leaky ReLUs_, a variant of ReLUs that seek to improve network performance by allowing the function to take on a negative value. The Leaky ReLU family of functions consists of your standard leaky ReLU variant, PReLUs, and ELUs.

Finally, it’s important to note that even though we are focusing on deep learning strictly in the context of _image classification_, neural networks have been used in some fashion in nearly all niches of machine learning.

![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.