Title: Convolutional Neural Networks (CNNs) and Layer Types - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/14/convolutional-neural-networks-cnns-and-layer-types/

Published Time: 2021-05-14T14:00:00+00:00

Markdown Content:
**CNN Building Blocks**
-----------------------

Neural networks accept an input image/feature vector (one input node for each entry) and transform it through a series of hidden layers, commonly using nonlinear activation functions. Each hidden layer is also made up of a set of neurons, where each neuron is _fully connected_ to all neurons in the previous layer. The last layer of a neural network (i.e., the “output layer”) is also fully connected and represents the final output classifications of the network.

A diverse dataset is essential for training a CNN and understanding its different layer types. By training the network on the dataset, we can observe how each layer extracts features and contributes to the final prediction.

[Roboflow](https://roboflow.com/?ref=pyimagesearch) has free tools for each stage of the computer vision pipeline that will streamline your workflows and supercharge your productivity.

Sign up or Log in to your [Roboflow account](https://roboflow.com/?ref=pyimagesearch) to access state of the art dataset libaries and revolutionize your computer vision pipeline.

You can start by choosing your own datasets or using our [PyimageSearch’s assorted library of useful datasets](https://universe.roboflow.com/pyimagesearch?ref=pyimagesearch).

Bring data in any of 40+ formats to [Roboflow](https://roboflow.com/?ref=pyimagesearch), train using any state-of-the-art model architectures, deploy across multiple platforms (API, NVIDIA, browser, iOS, etc), and connect to applications or 3rd party tools.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

However, neural networks operating directly on raw pixel intensities:

1.  Do not scale well as the image size increases.
2.  Leaves much accuracy to be desired (i.e., a standard feedforward neural network on CIFAR-10 obtained only 52% accuracy).

To demonstrate how standard neural networks do not scale well as image size increases, let’s again consider the CIFAR-10 dataset. Each image in CIFAR-10 is 32_×_32 with a Red, Green, and Blue channel, yielding a total of 32_×_32_×_3 = 3_,_072 total inputs to our network.

A total of 3_,_072 inputs does not seem to amount to much, but consider if we were using 250_×_250 pixel images — the total number of inputs and weights would jump to 250_×_250_×_3 = 187_,_500 — and this number is only for the input layer alone! Surely, we would want to add multiple hidden layers with a varying number of nodes per layer — these parameters can quickly add up, and given the poor performance of standard neural networks on raw pixel intensities, this bloat is hardly worth it.

Instead, we can use _Convolutional Neural Networks (CNNs)_ that take advantage of the input image structure and define a network architecture in a more sensible way. Unlike a standard neural network, layers of a CNN are arranged in a _3D volume_ in three dimensions: **width**, **height**, and **depth** (where _depth_ refers to the third dimension of the volume, such as the number of channels in an image or the number of filters in a layer).

To make this example more concrete, again consider the CIFAR-10 dataset: the input volume will have dimensions 32_×_32_×_3 (width, height, and depth, respectively). Neurons in subsequent layers will only be connected to a _small region_ of the layer before it (rather than the fully connected structure of a standard neural network) — we call this **_local connectivity_**, which enables us to save a _huge_ amount of parameters in our network. Finally, the output layer will be a 1_×_1_×N_ volume, which represents the image distilled into a single vector of class scores. In the case of CIFAR-10, given ten classes, _N_ \= 10, yielding a 1_×_1_×_10 volume.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

### **Layer Types**

There are many types of layers used to build Convolutional Neural Networks, but the ones you are most likely to encounter include:

*   Convolutional (`CONV`)
*   Activation (`ACT` or `RELU`, where we use the same or the actual activation function)
*   Pooling (`POOL`)
*   Fully connected (`FC`)
*   Batch normalization (`BN`)
*   Dropout (`DO`)

Stacking a series of these layers in a specific manner yields a CNN. We often use simple text diagrams to describe a CNN: `INPUT => CONV => RELU => FC => SOFTMAX`.

Here, we define a simple CNN that accepts an input, applies a convolution layer, then an activation layer, then a fully connected layer, and, finally, a softmax classifier to obtain the output classification probabilities. The `SOFTMAX` activation layer is often omitted from the network diagram as it is assumed it directly follows the final `FC`.

Of these layer types, `CONV` and `FC` (and to a lesser extent, `BN`) are the only layers that contain parameters that are _learned_ during the training process. Activation and dropout layers are not considered true “layers” themselves but are often included in network diagrams to make the architecture _explicitly_ clear. Pooling layers (`POOL`), of equal importance as `CONV` and `FC`, are also included in network diagrams as they have a _substantial impact_ on the spatial dimensions of an image as it moves through a CNN.

**`CONV`, `POOL`, `RELU`, and `FC` are the most important when defining your actual network architecture.** That’s not to say that the other layers are not critical, but take a backseat to this critical set of four as they define the _actual architecture itself_.

**_Remark:_** Activation functions themselves are practically **assumed** to be part of the architecture, When defining CNN architectures we often _omit_ the activation layers from a table/diagram to save space; however, the activation layers are _implicitly_ assumed to be part of the architecture.

In this tutorial, we’ll review each of these layer types in detail and discuss the parameters associated with each layer (and how to set them). In a future tutorial, I’ll discuss in more detail how to stack these layers properly to build your own CNN architectures.

### **Convolutional Layers**

The `CONV` layer is the core building block of a Convolutional Neural Network. The `CONV` layer parameters consist of a set of _K_ learnable filters (i.e., “kernels”), where each filter has a width and a height, and are nearly always square. These filters are small (in terms of their spatial dimensions) but extend throughout the full depth of the volume.

For inputs to the CNN, the depth is the number of channels in the image (i.e., a depth of three when working with RGB images, one for each channel). For volumes deeper in the network, the depth will be the number of filters applied in the _previous_ layer.

To make this concept more clear, let’s consider the forward-pass of a CNN, where we convolve each of the _K_ filters across the width and height of the input volume. More simply, we can think of each of our _K_ kernels sliding across the input region, computing an element-wise multiplication, summing, and then storing the output value in a 2-dimensional _activation map_, such as in **Figure 1**.

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20243'%3E%3C/svg%3E)

**Figure 1:** _Left:_ At each convolutional layer in a CNN, there are _K_ kernels applied to the input volume. _Middle:_ Each of the _K_ kernels is convolved with the input volume. _Right:_ Each kernel produces a 2D output, called an _activation map_.

After applying all _K_ filters to the input volume, we now have _K_, 2-dimensional activation maps. We then stack our _K_ activation maps along the depth dimension of our array to form the final output volume **(Figure 2**).

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20322'%3E%3C/svg%3E)

**Figure 2:** After obtaining the _K_ activation maps, they are stacked together to form the input volume to the next layer in the network.

Every entry in the output volume is thus an output of a neuron that “looks” at only a small region of the input. In this manner, the network “learns” filters that activate when they see a specific type of feature at a _given spatial location_ in the input volume. In lower layers of the network, filters may activate when they see edge-like or corner-like regions.

Then, in the deeper layers of the network, filters may activate in the presence of high-level features, such as parts of the face, the paw of a dog, the hood of a car, etc. This activation concept is as if these neurons are becoming “excited” and “activating” when they see a particular pattern in an input image.

The concept of convolving a small filter with a large(r) input volume has special meaning in Convolutional Neural Networks — specifically, the **local connectivity** and the **receptive field** of a neuron. When working with images, it’s often impractical to connect neurons in the current volume to _all_ neurons in the previous volume — there are simply too many connections and too many weights, making it impossible to train deep networks on images with large spatial dimensions. Instead, when utilizing CNNs, we choose to connect each neuron to only a _local region_ of the input volume — we call the size of this local region the **receptive field** (or simply, the variable _F_) of the neuron.

To make this point clear, let’s return to our CIFAR-10 dataset, where the input volume has an input size of 32_×_32_×_3\. Each image thus has a width of 32 pixels, a height of 32 pixels, and a depth of 3 (one for each RGB channel). If our receptive field is of size 3_×_3, then each neuron in the `CONV` layer will connect to a 3_×_3 local region of the image for a total of 3_×_3_×_3 = 27 weights (remember, the depth of the filters is three because they extend through the full depth of the input image, in this case, three channels).

Now, let’s assume that the spatial dimensions of our input volume have been reduced to a smaller size, but our depth is now larger, due to utilizing more filters deeper in the network, such that the volume size is now 16_×_16_×_94\. Again, if we assume a receptive field of size 3_×_3, then every neuron in the `CONV` layer will have a total of 3_×_3_×_94 = 846 connections to the input volume. Simply put, the receptive field _F_ is the **_size_** of the filter, yielding an _F×F_ kernel that is convolved with the input volume.

At this point, we have explained the connectivity of neurons in the input volume, but not the arrangement or size of the output volume. There are three parameters that control the size of an output volume: the **depth**, **stride**, and **zero-padding** size, each of which we’ll review below.

**Depth**

The _depth_ of an output volume controls the number of neurons (i.e., filters) in the `CONV` layer that connect to a local region of the input volume. Each filter produces an activation map that “activates” in the presence of oriented edges or blobs or color.

For a given `CONV` layer, the depth of the activation map will be _K_, or simply the number of filters we are learning in the current layer. The set of filters that are “looking at” the same (_x, y_) location of the input is called the **depth column**.

**Stride**

Consider where we described a convolution operation as “sliding” a small matrix across a large matrix, stopping at each coordinate, computing an element-wise multiplication and sum, then storing the output. This description is similar to a _sliding window_ ([http://pyimg.co/0yizo](http://pyimg.co/0yizo)) that slides from _left-to-right_ and _top-to-bottom_ across an image.

In the context on the convolution above, we only took a step of one pixel each time. In the context of CNNs, the same principle can be applied — for each step, we create a new depth column around the local region of the image, where we convolve each of the _K_ filters with the region and store the output in a 3D volume. When creating our `CONV` layers we normally use a stride step size _S_ of either _S_ \= 1 or _S_ \= 2.

Smaller strides will lead to overlapping receptive fields and larger output volumes. Conversely, larger strides will result in less overlapping receptive fields and smaller output volumes. To make the concept of convolutional stride more concrete, consider **Table 1**, where we have a 5_×_5 input image (_left_) along with a 3_×_3 Laplacian kernel (right).

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20188'%3E%3C/svg%3E)

**Table 1:** Our input 5×5 image _(left)_ that we are going to convolve with a Laplacian kernel _(right)_.

Using _S_ \= 1, our kernel slides from _left-to-right_ and _top-to-bottom_, one pixel at a time, producing the following output (**Table 2**, _left_). However, if we were to apply the same operation, only this time with a stride of _S_ \= 2, we skip _two pixels at a time_ (two pixels along the _x_\-axis and two pixels along the _y_\-axis), producing a smaller output volume (_right_).

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20400%20121'%3E%3C/svg%3E)

**Table 2:** _Left:_ Output of convolution with 1_×_1 stride. _Right:_ Output of convolution with 2_×_2 stride. Notice how a larger stride can reduce the spatial dimensions of the input.

Thus, we can see how convolution layers can be used to reduce the spatial dimensions of the input volumes simply by changing the stride of the kernel. Convolutional layers and pooling layers are the primary methods to reduce spatial input size.

**Zero-padding**

We need to “pad” the borders of an image to retain the _original image size_ when applying a convolution — the same is true for filters inside of a CNN. Using zero-padding, we can “pad” our input along the borders such that our output volume size matches our input volume size. The amount of padding we apply is controlled by the parameter _P_.

This technique is _especially critical_ when we start looking at deep CNN architectures that apply multiple `CONV` filters on top of each other. To visualize zero-padding, again refer to **Table 1**, where we applied a 3_×_3 Laplacian kernel to a 5_×_5 input image with a stride of _S_ \= 1.

We can see in **Table 3** (_left_) how the output volume is _smaller_ (3_×_3) than the input volume (5×5) due to the nature of the convolution operation. If we instead set _P_ \= 1, we can pad our input volume with zeros (_right_) to create a 7_×_7 volume and then apply the convolution operation, leading to an output volume size that matches the original input volume size of 5_×_5 (_bottom_).

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20374'%3E%3C/svg%3E)

**Table 3:** _Left:_ The output of applying a 3×3 convolution to a 5×5 input (i.e., the spatial dimensions decrease). _Right:_ Applying zero-padding to the original input with _P_ \= 1 increases the spatial dimensions to 7_×_7\. _Bottom:_ After applying the 3_×_3 convolution to the padded input, our output volume matches the _original_ input volume size of 5_×_5, thus zero padding helps us preserve spatial dimensions.

Without zero padding, the spatial dimensions of the input volume would decrease too quickly, and we wouldn’t be able to train deep networks (as the input volumes would be too tiny to learn any useful patterns from).

Putting all these parameters together, we can compute the size of an output volume as a function of the input volume size (_W_, assuming the input images are square, which they nearly always are), the receptive field size _F_, the stride _S_, and the amount of zero-padding _P_. To construct a valid `CONV` layer, we need to ensure the following equation is an integer:

**(1)** ((_W_ − _F_ + 2_P_) / _S_) + 1

If it is _not_ an integer, then the strides are set incorrectly, and the neurons cannot be tiled such that they fit across the input volume in a symmetric way.

As an example, consider the first layer of the AlexNet architecture which won the 2012 ImageNet classification challenge and is _hugely_ responsible for the current boom of deep learning applied to image classification. Inside their paper, [Krizhevsky et al. (2012)](http://papers.nips.cc/paper/4824-imagenet-classification-with-deepconvolutional-%20neural-networks.pdf) documented their CNN architecture according to **Figure 3**.

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20241'%3E%3C/svg%3E)

**Figure 3:** The original AlexNet architecture diagram provided by [Krizhevsky et al. (2012)](http://papers.nips.cc/paper/4824-imagenet-classification-with-deepconvolutional-%20neural-networks.pdf). Notice how the input image is documented to be 224_×_224_×_3, although this cannot be possible due to **Equation (1)**. It’s also worth noting that we are unsure why the _top-half_ of this figure is cutoff from the original publication.

Notice how the first layer claims that the input image size is 224_×_224 pixels. However, this can’t possibly be correct if we apply our equation above using 11_×_11 filters, a stride of four, and no padding:

**(2)** ((224 _−_ 11 + 2(0)) _/_ 4) + 1 = 54_._25

Which is certainly not an integer.

For novice readers just getting started in deep learning and CNNs, this small error in such a seminal paper has caused countless errors of confusion and frustration. It’s unknown why this typo occurred, but it’s likely that Krizhevsky et al. used 227_×_227 input images, since:

**(3)** ((227 _−_ 11 + 2(0)) _/_ 4) + 1 = 55

Errors like these are more common than you might think, so when implementing CNNs from publications, be sure to _check the parameters yourself_ rather than simply assuming the parameters listed are correct. Due to the vast number of parameters in a CNN, it’s quite easy to make a typographical mistake when documenting an architecture (I’ve done it myself many times).

To summarize, the `CONV` layer in the same, elegant manner as [Karpathy](http://cs231n.github.io/convolutional-networks/):

*   Accepts an input volume of size _Winput×Hinput×Dinput_ (the input sizes are normally square, so it’s common to see _Winput_ \= _Hinput_).
*   Requires four parameters:
    *   The number of filters _K_ (which controls the _depth_ of the output volume).
    *   The receptive field size _F_ (the size of the _K_ kernels used for convolution and is early always _square_, yielding an _F×F_ kernel).
    *   The stride _S_.
    *   The amount of zero-padding _P_.

*   The output of the `CONV` layer is then _Woutput×Houtput×Doutput_, where:
    *   _Woutput_ \= ((_Winput −F_ +2_P_) _/S_) +1
    *   _Houtput_ \= ((_Hinput −F_ +2_P_) _/ S_) +1
    *   _Doutput_ \= _K_

### **Activation Layers**

After each `CONV` layer in a CNN, we apply a nonlinear activation function, such as ReLU, ELU, or any of the other Leaky ReLU variants. We typically denote activation layers as `RELU` in network diagrams as since ReLU activations are most commonly used, we may also simply state `ACT` — in either case, we are making it clear that an activation function is being applied inside the network architecture.

Activation layers are not technically “layers” (due to the fact that no parameters/weights are learned inside an activation layer) and are sometimes omitted from network architecture diagrams as it’s _assumed_ that an activation _immediately follows_ a convolution.

In this case, authors of publications will mention which activation function they are using after each `CONV` layer somewhere in their paper. As an example, consider the following network architecture: `INPUT => CONV => RELU => FC`.

To make this diagram more concise, we could simply remove the `RELU` component since it’s assumed that an activation always follows a convolution: `INPUT => CONV => FC`. I personally do not like this and choose to _explicitly_ include the activation layer in a network diagram to make it clear _when_ and _what_ activation function I am applying in the network.

An activation layer accepts an input volume of size _Winput×Hinput×Dinput_ and then applies the given activation function (**Figure 4**). Since the activation function is applied in an element-wise manner, the output of an activation layer is always the same as the input dimension, _Winput_ \= _Woutput_, _Hinput_ \= _Houtput_, _Dinput_ \= _Doutput_.

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20359'%3E%3C/svg%3E)

**Figure 4:** An example of an input volume going through a ReLU activation, max(0_, x_). Activations are done _in-place_ so there is no need to create a separate output volume although it is easy to visualize the flow of the network in this manner.

### **Pooling Layers**

There are two methods to reduce the size of an input volume — `CONV` layers with a stride _\>_ 1 (which we’ve already seen) and `POOL` layers. It is common to insert `POOL` layers in-between consecutive `CONV` layers in a CNN architectures:

`INPUT => CONV => RELU => POOL => CONV => RELU => POOL => FC`

The primary function of the `POOL` layer is to progressively reduce the spatial size (i.e., width and height) of the input volume. Doing this allows us to reduce the amount of parameters and computation in the network — pooling also helps us control overfitting.

`POOL` layers operate on each of the depth slices of an input _independently_ using either the _max_ or _average_ function. Max pooling is typically done in the middle of the CNN architecture to reduce spatial size, whereas average pooling is normally used as the final layer of the network (e.g., GoogLeNet, SqueezeNet, ResNet), where we wish to avoid using `FC` layers entirely. The most common type of `POOL` layer is max pooling, although this trend is changing with the introduction of more exotic micro-architectures.

Typically we’ll use a pool size of 2_×_2, although deeper CNNs that use larger input images (_\>_ 200 pixels) may use a 3_×_3 pool size early in the network architecture. We also commonly set the stride to either _S_ \= 1 or _S_ \= 2. **Figure 5** (heavily inspired by [Karpathy et al.](http://cs231n.github.io/convolutional-networks/)) follows an example of applying max pooling with 2_×_2 pool size and a stride of _S_ \= 1. Notice for every 2_×_2 block, we keep only the largest value, take a single step (like a sliding window), and apply the operation again — thus producing an output volume size of 3_×_3.

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20373'%3E%3C/svg%3E)

**Figure 5:** _Left:_ Our input 4_×_4 volume. _Right:_ Applying 2_×_2 max pooling with a stride of _S_ \= 1. _Bottom:_ Applying 2_×_2 max pooling with _S_ \= 2 — this dramatically reduces the spatial dimensions of our input.

We can further decrease the size of our output volume by increasing the stride — here we apply _S_ \= 2 to the same input (**Figure 5**, _bottom_). For every 2_×_2 block in the input, we keep only the largest value, then take a step of _two pixels_, and apply the operation again. This pooling allows us to reduce the width and height by a factor of two, effectively discarding 75% of activations from the previous layer.

In summary, `POOL` layers Accept an input volume of size _Winput×Hinput×Dinput_. They then require two parameters:

*   The receptive field size _F_ (also called the “pool size”).
*   The stride _S_.

Applying the `POOL` operation yields an output volume of size _Woutput×Houtput×Doutput_, where:

*   _Woutput_ \= ((_Winput −F_) _/ S_) +1
*   _Houtput_ \= ((_Hinput −F_) _/ S_) +1
*   _Doutput_ \= _Dinput_

In practice, we tend to see two types of max pooling variations:

*   **Type #1:** _F_ \= 3_, S_ \= 2, which is called _overlapping pooling_ and normally applied to images/input volumes with large spatial dimensions.
*   **Type #2:** _F_ \= 2_, S_ \= 2, which is called _non-overlapping pooling_. This is the most common type of pooling and is applied to images with smaller spatial dimensions.

For network architectures that accept smaller input images (in the range of 32_−_64 pixels) you may also see _F_ \= 2_, S_ \= 1 as well.

**To POOL or CONV?**

In their [2014 paper, **_Striving for Simplicity: The All Convolutional Net_, Springenberg et al.**](http://arxiv.org/abs/1412.6806) recommend discarding the `POOL` layer _entirely_ and simply relying on `CONV` layers with a larger stride to handle downsampling the spatial dimensions of the volume. Their work demonstrated this approach works very well on a variety of datasets, including CIFAR-10 (small images, low number of classes) and ImageNet (large input images, 1,000 classes). This trend continues with the **[ResNet architecture](http://arxiv.org/abs/1512.03385)**, which uses `CONV` layers for downsampling as well.

It’s becoming increasingly more common to _not_ use `POOL` layers in the middle of the network architecture and _only_ use average pooling at the end of the network if `FC` layers are to be avoided. Perhaps in the future there won’t be pooling layers in Convolutional Neural Networks — but in the meantime, it’s important that we study them, learn how they work, and apply them to our own architectures.

### **Fully connected Layers**

Neurons in `FC` layers are fully connected to all activations in the previous layer, as is the standard for feedforward neural networks. `FC` layers are _always_ placed at the end of the network (i.e., we don’t apply a `CONV` layer, then an `FC` layer, followed by another `CONV`) layer.

It’s common to use one or two `FC` layers prior to applying the softmax classifier, as the following (simplified) architecture demonstrates:

INPUT => CONV => RELU => POOL => CONV => RELU => POOL => FC => FC

Here, we apply two fully connected layers before our (implied) softmax classifier which will compute our final output probabilities for each class.

### **Batch Normalization**

First introduced by [Ioffe and Szegedy in their 2015 paper, _Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift_](http://arxiv.org/abs/1502.03167), batch normalization layers (or `BN` for short), as the name suggests, are used to normalize the activations of a given input volume before passing it into the next layer in the network.

If we consider ![Image 11: x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) to be our mini-batch of activations, then we can compute the normalized ![Image 12: \hat{x}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2013'%3E%3C/svg%3E) via the following equation:

**(4)** ![Image 13: \hat{x_{i}} = \displaystyle\frac{x_{i} - \mu_{\beta}}{\sqrt{\sigma_{\beta}^{2} + \epsilon}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2098%2048'%3E%3C/svg%3E)

During _training_, we compute the _µ__β_ and _σ__β_ over each mini-batch _β_, where:

**(5)** ![Image 14: \mu_{\beta} = \displaystyle\frac{1}{M} \sum_{i=1}^{m} x_{i} \qquad \sigma_{\beta}^{2} = \displaystyle\frac{1}{M} \sum_{i=1}^{m} (x_{i} - \mu_{\beta})^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20293%2046'%3E%3C/svg%3E)

We set _ε_ equal to a small positive value such as 1e-7 to avoid dividing by zero. Applying this equation implies that the activations leaving a batch normalization layer will have approximately zero mean and unit variance (i.e., zero-centered).

At _testing_ time, we replace the mini-batch _µβ_ and _σβ_ with _running averages_ of _µβ_ and _σβ_ computed during the training process. This ensures that we can pass images through our network and still obtain accurate predictions without being biased by the _µβ_ and _σβ_ from the final mini-batch passed through the network at training time.

Batch normalization has been shown to be _extremely effective_ at reducing the number of epochs it takes to train a neural network. Batch normalization also has the added benefit of helping “stabilize” training, allowing for a larger variety of learning rates and regularization strengths. Using batch normalization doesn’t alleviate the need to tune these parameters of course, but it _will_ make your life easier by making learning rate and regularization less volatile and more straightforward to tune. You’ll also tend to notice _lower final loss_ and a _more stable loss curve_ when using batch normalization in your networks.

The biggest drawback of batch normalization is that it can actually slow down the wall time it takes to train your network (even though you’ll need fewer epochs to obtain reasonable accuracy) by 2-3x due to the computation of per-batch statistics and normalization.

That said, I recommend using batch normalization in _nearly every situation_ as it does make a significant difference. Applying batch normalization to our network architectures can help us prevent overfitting and allows us to obtain significantly higher classification accuracy in fewer epochs compared to the same network architecture _without_ batch normalization.

**So, Where Do the Batch Normalization Layers Go?**

You’ve probably noticed in my discussion of batch normalization I’ve left out exactly _where_ in the network architecture we place the batch normalization layer. According to the original paper by **[Ioffe and Szegedy](http://arxiv.org/abs/1502.03167)**, they placed their batch normalization (`BN`) _before_ the activation:

> _We add the BN transform immediately before the nonlinearity, by normalizing x_ \= _Wu_ + _b._

Using this scheme, a network architecture utilizing batch normalization would look like this:

INPUT => CONV => BN => RELU ...

However, this view of batch normalization doesn’t make sense from a statistical point of view. In this context, a `BN` layer is normalizing the distribution of features coming out of a `CONV` layer. Some of these features may be negative, in which they will be clamped (i.e., set to zero) by a nonlinear activation function such as ReLU.

If we normalize _before_ activation, we are essentially including the negative values inside the normalization. Our zero-centered features are then passed through the ReLU where we kill of any activations less than zero (which include features that may have not been negative _before_ the normalization) — this layer ordering entirely defeats the purpose of applying batch normalization in the first place.

Instead, if we place the batch normalization _after_ the ReLU we will normalize the positive valued features without statistically biasing them with features that would have otherwise not made it to the next `CONV` layer. In fact, François Chollet, the creator and maintainer of Keras confirms this point stating that the `BN` should come after the activation:

> _I can guarantee that recent code written by Christian \[Szegedy, from the BN paper\] applies relu before BN. It is still occasionally a topic of debate, though._
> 
> _[Szegedy](https://github.com/keras-team/keras/issues/1802#issuecomment-187966878)_

It is unclear why Ioffe and Szegedy suggested placing the `BN` layer before the activation in their paper, but further [experiments](https://github.com/ducha-aiki/caffenet-benchmark/blob/master/batchnorm.md) as well as anecdotal evidence from other [deep learning researchers](https://www.reddit.com/r/MachineLearning/comments/67gonq/d_batch_normalization_before_or_after_relu/) confirm that placing the batch normalization layer _after_ the nonlinear activation yields higher accuracy and lower loss in nearly all situations.

Placing the `BN` after the activation in a network architecture would look like this:

INPUT => CONV => RELU => BN ...

I can confirm that in nearly all experiments I’ve performed with CNNs, placing the `BN` after the `RELU` yields slightly higher accuracy and lower loss. That said, take note of the word _“nearly” —_ there have been a _very small_ number of situations where placing the `BN` before the activation worked better, which implies that you should default to placing the `BN` after the activation, but may want to dedicate (at most) one experiment to placing the `BN` before the activation and noting the results.

After running a few of these experiments, you’ll quickly realize that `BN` after the activation performs better and there are more important parameters to your network to tune to obtain higher classification accuracy.

### **Dropout**

The last layer type we are going to discuss is dropout. Dropout is actually a form of _regularization_ that aims to help prevent overfitting by increasing testing accuracy, perhaps at the expense of training accuracy. For each mini-batch in our training set, dropout layers, with probability _p_, randomly disconnect inputs from the preceding layer to the next layer in the network architecture.

**Figure 6** visualizes this concept where we randomly disconnect with probability _p_ \= 0_._5 the connections between two `FC` layers for a given mini-batch. Again, notice how half of the connections are severed for this mini-batch. After the forward and backward pass are computed for the mini-batch, we re-connect the dropped connections, and then sample another set of connections to drop.

![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20173'%3E%3C/svg%3E)

**Figure 6:** _Left:_ Two layers of a neural network that are fully connected with no dropout. _Right:_ The same two layers after dropping 50% of the connections.

The reason we apply dropout is to reduce overfitting by _explicitly_ altering the network architecture at training time. Randomly dropping connections ensures that no single node in the network is responsible for “activating” when presented with a given pattern. Instead, dropout ensures there are _multiple, redundant nodes_ that will activate when presented with similar inputs — this, in turn, helps our model to _generalize_.

It is most common to place dropout layers with _p_ \= 0_._5 **_in-between_** `FC` layers of an architecture where the final `FC` layer is assumed to be our softmax classifier:

... CONV => RELU => POOL => FC => DO => FC => DO => FC

However, we may also apply dropout with smaller probabilities (i.e., _p_ \= 0_._10_−_0_._25) in earlier layers of the network as well (normally following a downsampling operation, either via max pooling or convolution).

**Common Architectures and Training Patterns**
----------------------------------------------

As we have seen, Convolutional Neural Networks are made up of four primary layers: `CONV`, `POOL`, `RELU`, and `FC`. Taking these layers and stacking them together in a particular pattern yields a _CNN architecture_.

The `CONV` and `FC` layers (and `BN`) are the only layers of the network that actually learn parameters the other layers are simply responsible for performing a given operation. Activation layers, (`ACT`) such as `RELU` and dropout aren’t technically layers, but are often included in the CNN architecture diagrams to make the operation order _explicitly clear_ — we’ll adopt the same convention.

### **Layer Patterns**

By far, the most common form of CNN architecture is to stack a few `CONV` and `RELU` layers, following them with a `POOL` operation. We repeat this sequence until the volume width and height is small, at which point we apply one or more `FC` layers. Therefore, we can derive the most common CNN architecture using the following [pattern](http://cs231n.github.io/convolutional-networks/):

INPUT => \[\[CONV => RELU\]\*N => POOL?\]\*M => \[FC => RELU\]\*K => FC

Here the `*` operator implies one or more and the `?` indicates an optional operation.

Common choices for each repetition include:

*   0 _<_\= _N <_\= 3
*   _M >_\= 0
*   0 _<_\= _K <_\= 2

Below, we can see some examples of CNN architectures that follow this pattern:

*   `INPUT => FC`
*   `INPUT => [CONV => RELU => POOL] * 2 => FC => RELU => FC`
*   `INPUT => [CONV => RELU => CONV => RELU => POOL] * 3 => [FC => RELU] * 2 => FC`

Here is an example of a very shallow CNN with only one `CONV` layer (_N_ \= _M_ \= _K_ \= 0):

INPUT => CONV => RELU => FC

Below is an example of an [AlexNet-like](http://papers.nips.cc/paper/4824-imagenet-classification-with-deepconvolutional-%20neural-networks.pdf) CNN architecture which has multiple `CONV => RELU => POOL` layer sets, followed by `FC` layers:

INPUT => \[CONV => RELU => POOL\] \* 2 => \[CONV => RELU\] \* 3 => POOL =>
	\[FC => RELU => DO\] \* 2 => SOFTMAX

For deeper network architectures, such as **[VGGNet](http://arxiv.org/%20abs/1409.1556)**, we’ll stack two (or more) layers before every `POOL` layer:

INPUT => \[CONV => RELU\] \* 2 => POOL => \[CONV => RELU\] \* 2 => POOL =>
	\[CONV => RELU\] \* 3 => POOL => \[CONV => RELU\] \* 3 => POOL =>
	\[FC => RELU => DO\] \* 2 => SOFTMAX

Generally, we apply deeper network architectures when we (1) have lots of labeled training data and (2) the classification problem is sufficiently challenging. Stacking multiple `CONV` layers before applying a `POOL` layer allows the `CONV` layers to develop more complex features before the destructive pooling operation is performed.

There are more “exotic” network architectures that deviate from these patterns and, in turn, have created patterns of their own. Some architectures remove the `POOL` operation entirely, relying on `CONV` layers to downsample the volume — then, at the end of the network, average pooling is applied rather than `FC` layers to obtain the input to the softmax classifiers.

Network architectures such as GoogLeNet, ResNet, and SqueezeNet ([He et al.](https://arxiv.org/abs/1512.03385), [Szegedy et al.](http://arxiv.org/abs/1409.4842), [Iandola et al.](https://arxiv.org/abs/1602.07360)) are great examples of this pattern and demonstrate how removing `FC` layers leads to less parameters and faster training time.

These types of network architectures also “stack” and concatenate filters across the channel dimension: GoogLeNet applies 1_×_1, 3_×_3, and 5_×_5 filters and then concatenates them together across the channel dimension to learn multi-level features. Again, these architectures are considered more “exotic” and considered advanced techniques.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

### **Rules of Thumb**

I’ll review common rules of thumb when constructing your own CNNs. To start, the images presented to the **input layer** should be _square_. Using square inputs allows us to take advantage of linear algebra optimization libraries. Common input layer sizes include 32×32, 64×64, 96×96, 224×224, 227×227, and 229×229 (leaving out the number of channels for notational convenience).

Secondly, the input layer should also be _divisible by two multiple times_ after the first `CONV` operation is applied. You can do this by tweaking your filter size and stride. The “divisible by two rule” enables the spatial inputs in our network to be conveniently down sampled via `POOL` operation in an efficient manner.

In general, your `CONV` layers should use smaller filter sizes such as 3_×_3 and 5_×_5\. Tiny 1_×_1 filters are used to learn local features, but only in your more advanced network architectures. Larger filter sizes such as 7_×_7 and 11_×_11 may be used as the _first_ `CONV` layer in the network (to reduce spatial input size, provided your images are sufficiently larger than _\>_ 200_×_200 pixels); however, after this initial `CONV` layer the filter size should drop dramatically, otherwise you will reduce the spatial dimensions of your volume too quickly.

You’ll also commonly use a stride of _S_ \= 1 for `CONV` layers, at least for smaller spatial input volumes (networks that accept larger input volumes use a stride _S >_\= 2 in the first `CONV` layer to help reduce spatial dimensions). Using a stride of _S_ \= 1 enables our `CONV` layers to learn filters while the `POOL` layer is responsible for downsampling. However, keep in mind that not _all_ network architectures follow this pattern — some architectures skip max pooling altogether and rely on the `CONV` stride to reduce volume size.

My personal preference is to apply **zero-padding** to my `CONV` layers to ensure the output dimension size matches the input dimension size — the only exception to this rule is if I want to _purposely_ reduce spatial dimensions via convolution. Applying zero-padding when _stacking_ multiple `CONV` layers on top of each other has also demonstrated to increase classification accuracy in practice. Libraries such as Keras can automatically compute zero-padding for you, making it even easier to build CNN architectures.

A second personal recommendation is to use `POOL` layers (rather than `CONV` layers) to reduce the spatial dimensions of your input, at least until you become more experienced constructing your own CNN architectures. Once you reach that point, you should start experimenting with using `CONV` layers to reduce spatial input size and try removing max pooling layers from your architecture.

Most commonly, you’ll see max pooling applied over a 2_×_2 receptive field size and a stride of _S_ \= 2. You might also see a 3_×_3 receptive field early in the network architecture to help reduce image size. It is **highly uncommon** to see receptive fields larger than three since these operations are very destructive to their inputs.

Batch normalization is an expensive operation that can _double_ or _triple_ the amount of time it takes to train your CNN; however, I recommend using `BN` in **_nearly all situations_**. While BN does indeed slow down the training time, it also tends to “stabilize” training, making it easier to tune other hyperparameters (there are some exceptions, of course).

I also place the batch normalization _after_ the activation, as has become commonplace in the deep learning community even though it goes against the original **[Ioffe and Szegedy paper](http://arxiv.org/abs/1502.03167)**.

Inserting `BN` into the common layer architectures above, they become:

*   `INPUT => CONV => RELU => BN => FC`
*   `INPUT => [CONV => RELU => BN => POOL] * 2 => FC => RELU => BN => FC`
*   `INPUT => [CONV => RELU => BN => CONV => RELU => BN => POOL] * 3 => [FC RELU => BN] * 2 => FC`

You _do not_ apply batch normalization before the softmax classifier as at this point we assume our network has learned its discriminative features earlier in the architecture.

Dropout (`DO`) is typically applied in between `FC` layers with a dropout probability of 50% — you should consider applying dropout in nearly _every_ architecture you build. While not always performed, I also like to include dropout layers (with a very small probability, 10-25%) between `POOL` and `CONV` layers. Due to the local connectivity of `CONV` layers, dropout is less effective here, but I’ve often found it helpful when battling overfitting.

By keeping these rules of thumb in mind, you’ll be able to reduce your headaches when constructing CNN architectures since your `CONV` layers will preserve input sizes while the `POOL` layers take care of reducing spatial dimensions of the volumes, eventually leading to `FC` layers and the final output classifications.

Once you master this “traditional” method of building Convolutional Neural Networks, you should then start exploring leaving max pooling operations out _entirely_ and using _just_ `CONV` layers to reduce spatial dimensions, eventually leading to _average pooling_ rather than an `FC` layer.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!