Title: Deep learning: How OpenCV's blobFromImage works - PyImageSearch

URL Source: https://pyimagesearch.com/2017/11/06/deep-learning-opencvs-blobfromimage-works/

Published Time: 2017-11-06T15:00:11+00:00

Markdown Content:
Today’s blog post is inspired by a number of PyImageSearch readers who have commented on previous deep learning tutorials wanting to understand what exactly OpenCV’s `blobFromImage` function is doing under the hood.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20303'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_header.png)

You see, to obtain (correct) predictions from deep neural networks you first need to preprocess your data.

In the context of deep learning and image classification, these preprocessing tasks normally involve:

1.  Mean subtraction
2.  Scaling by some factor

OpenCV’s new deep neural network (`dnn` ) module contains two functions that can be used for preprocessing images and preparing them for classification via pre-trained deep learning models.

In today’s blog post we are going to take apart OpenCV’s `cv2.dnn.blobFromImage` and `cv2.dnn.blobFromImages` preprocessing functions and understand how they work.

**To learn more about image preprocessing for deep learning via OpenCV, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

OpenCV provides two functions to facilitate image preprocessing for deep learning classification:

*   `cv2.dnn.blobFromImage`
*   `cv2.dnn.blobFromImages`

These two functions perform

1.  Mean subtraction
2.  Scaling
3.  And optionally channel swapping

In the remainder of this tutorial we’ll:

1.  Explore mean subtraction and scaling
2.  Examine the function signature of each deep learning preprocessing function
3.  Study these methods in detail
4.  And finally, apply OpenCV’s deep learning functions to a set of input images

Let’s go ahead and get started.

### Deep learning and mean subtraction

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20139'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_mean_subtraction.jpg)

**Figure 1:** A visual representation of mean subtraction where the RGB mean (_center_) has been calculated from a dataset of images and subtracted from the original image _(left)_ resulting in the output image _(right)_.

Before we dive into an explanation of OpenCV’s deep learning preprocessing functions, we first need to understand mean subtraction. Mean subtraction is used to help combat illumination changes in the input images in our dataset. We can therefore view mean subtraction as a technique used to aid our Convolutional Neural Networks.

Before we even begin training our deep neural network, we first compute the _average pixel intensity_ across all images in the _training set_ for each of the Red, Green, and Blue channels.

This implies that we end up with three variables:

![Image 4: \mu_R](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2019%2010'%3E%3C/svg%3E), ![Image 5: \mu_G](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2019%2010'%3E%3C/svg%3E), and ![Image 6: \mu_B](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2019%2010'%3E%3C/svg%3E)

Typically the resulting values are a 3-tuple consisting of the mean of the Red, Green, and Blue channels, respectively.

For example, the mean values for the ImageNet training set are _R=103.93_, _G=116.77_, and _B=123.68_ (you may have already encountered these values before if you have used a network that was pre-trained on ImageNet).

However, in some cases the mean Red, Green, and Blue values may be computed _channel-wise_ rather than _pixel-wise_, resulting in an _MxN_ matrix. In this case the _MxN_ matrix for each channel is then subtracted from the input image during training/testing.

Both methods are perfectly valid forms of mean subtraction; however, we tend to see the pixel-wise version used more often, _especially_ for larger datasets.

When we are ready to pass an image through our network (whether for _training_ or _testing_), we subtract the _mean_, ![Image 7: \mu](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2010'%3E%3C/svg%3E), from each input channel of the input image:

![Image 8: R = R - \mu_R](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2086%2014'%3E%3C/svg%3E)

![Image 9: G = G - \mu_G](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2087%2014'%3E%3C/svg%3E)

![Image 10: B = B - \mu_B](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2088%2014'%3E%3C/svg%3E)

We may also have a _scaling factor_, ![Image 11: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E), which adds in a normalization:

![Image 12: R = (R - \mu_R) / \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20117%2018'%3E%3C/svg%3E)

![Image 13: G = (G - \mu_G) / \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20118%2018'%3E%3C/svg%3E)

![Image 14: B = (B - \mu_B) / \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20119%2018'%3E%3C/svg%3E)

The value of ![Image 15: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) may be the standard deviation across the training set (thereby turning the preprocessing step into a standard score/z-score). However,![Image 16: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) may also be manually set (versus calculated) to scale the input image space into a particular range — it really depends on the architecture, how the network was trained, and the techniques the implementing author is familiar with.

It’s important to note that **not all deep learning architectures perform mean subtraction and scaling!** Before you preprocess your images, be sure to read the relevant publication/documentation for the deep neural network you are using.

As you’ll find on your deep learning journey, some architectures perform mean subtraction only (thereby setting ![Image 17: \sigma=1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2039%2013'%3E%3C/svg%3E)). Other architectures perform both mean subtraction and scaling. Even other architectures choose to perform _no_ mean subtraction or scaling. Always check the relevant publication you are implementing/using to verify the techniques the author is using.

Mean subtraction, scaling, and normalization are covered in more detail inside _[Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)_.

### OpenCV’s blobFromImage and blobFromImages function

Let’s start off by referring to the [official OpenCV documentation](https://docs.opencv.org/trunk/d6/d0f/group__dnn.html#ga33d1b39b53a891e98a654fdeabba22eb) for `cv2.dnn.blobFromImage` :

> \[blobFromImage\] creates 4-dimensional blob from image. Optionally resizes and crops `image` from center, subtract `mean` values, scales values by `scalefactor`, swap Blue and Red channels.

Informally, a _blob_ is just a (potentially collection) of image(s) with the same spatial dimensions (i.e., width and height), same depth (number of channels), that have all be preprocessed in the same manner.

The `cv2.dnn.blobFromImage` and `cv2.dnn.blobFromImages` functions are near identical.

Let’s start with examining the `cv2.dnn.blobFromImage` function signature below:

`blob = cv2.dnn.blobFromImage(image, scalefactor=1.0, size, mean, swapRB=True)`

I’ve provided a discussion of each parameter below:

1.  `image` : This is the input image we want to preprocess before passing it through our deep neural network for classification.
2.  `scalefactor` : After we perform mean subtraction we can optionally scale our images by some factor. This value defaults to \`1.0\` (i.e., no scaling) but we can supply another value as well. It’s also important to note that `scalefactor` should be ![Image 18: 1 / \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2025%2016'%3E%3C/svg%3E) as we’re actually multiplying the input channels (after mean subtraction) by `scalefactor` .
3.  `size` : Here we supply the spatial size that the Convolutional Neural Network expects. For most current state-of-the-art neural networks this is either _224×224_, _227×227_, or _299×299_.
4.  `mean` : These are our mean subtraction values. They can be a 3-tuple of the RGB means or they can be a single value in which case the supplied value is subtracted from every channel of the image. If you’re performing mean subtraction, ensure you supply the 3-tuple in \`(R, G, B)\` order, especially when utilizing the default behavior of `swapRB=True` .
5.  `swapRB` : OpenCV assumes images are in BGR channel order; however, the \`mean\` value assumes we are using RGB order. To resolve this discrepancy we can swap the R and B channels in `image` by setting this value to \`True\`. By default OpenCV performs this channel swapping for us.

The `cv2.dnn.blobFromImage` function returns a `blob` which is our input image after mean subtraction, normalizing, and channel swapping.

The `cv2.dnn.blobFromImages` function is exactly the same:

`blob = cv2.dnn.blobFromImages(images, scalefactor=1.0, size, mean, swapRB=True)`

The only exception is that we can pass in _multiple images_, enabling us to batch process a set of `images` .

If you’re processing multiple images/frames, be sure to use the `cv2.dnn.blobFromImages` function as there is less function call overhead and you’ll be able to batch process the images/frames faster.

### Deep learning with OpenCV’s blobFromImage function

Now that we’ve studied both the `blobFromImage` and `blobFromImages` functions, let’s apply them to a few example images and then pass them through a Convolutional Neural Network for classification.

As a prerequisite, you need OpenCV version 3.3.0 _at a minimum_. NumPy is a dependency of OpenCV’s Python bindings and [imutils](https://pyimagesearch.com/2015/02/02/just-open-sourced-personal-imutils-package-series-opencv-convenience-functions/) is my package of convenience functions available on [GitHub](https://github.com/jrosebr1/imutils) and in the [Python Package Index](https://pypi.python.org/pypi/imutils).

If you haven’t installed OpenCV, you’ll want to follow the latest tutorials available [here](https://pyimagesearch.com/opencv-tutorials-resources-guides/), and _be sure to specify OpenCV 3.3.0 or higher_ when you clone/download `opencv` and `opencv_contrib` .

The imutils package can be installed via `pip` :

$ pip install imutils

Assuming your image processing environment is ready to go, let’s open up a new file, name it `blob_from_images.py` , and insert the following code:

\# import the necessary packages
from imutils import paths
import numpy as np
import cv2

# load the class labels from disk
rows = open("synset\_words.txt").read().strip().split("\\n")
classes = \[r\[r.find(" ") + 1:\].split(",")\[0\] for r in rows\]

# load our serialized model from disk
net = cv2.dnn.readNetFromCaffe("bvlc\_googlenet.prototxt",
	"bvlc\_googlenet.caffemodel")

# grab the paths to the input images
imagePaths = sorted(list(paths.list\_images("images/")))

First we import `imutils` , `numpy` , and `cv2` (**Lines 2-4**).

Then we read `synset_words.txt` (the ImageNet Class labels) and extract `classes` , our class labels, on **Lines 7 and 8**.

To load our model model from disk we use the DNN function, `cv2.dnn.readNetFromCaffe` , and specify `bvlc_googlenet.prototxt` as the filename parameter and `bvlc_googlenet.caffemodel` as the actual model file (**Lines 11 and 12**).

_**Note:** You can grab the pre-trained Convolutional Neural Network, class labels text file, source code, and example images to this post using the **“Downloads”** section at the bottom of this tutorial._

Finally, we grab the paths to the input images on **Line 15**. If you’re using Windows you should change the path separator here to ensure you can correctly load the image paths.

Next, we’ll load images from disk and pre-process them using `blobFromImage` :

\# (1) load the first image from disk, (2) pre-process it by resizing
# it to 224x224 pixels, and (3) construct a blob that can be passed
# through the pre-trained network
image = cv2.imread(imagePaths\[0\])
resized = cv2.resize(image, (224, 224))
blob = cv2.dnn.blobFromImage(resized, 1, (224, 224), (104, 117, 123))
print("First Blob: {}".format(blob.shape))

In this block, we first load the `image` (**Line 20**) and then resize it to _224×224_ (**Line 21**), the required input image dimensions for GoogLeNet.

**Now we’re to the crux of this post.**

On **Line 22**, we call `cv2.dnn.blobFromImage` which, as stated in the previous section, will create a 4-dimensional `blob` for use in our neural net.

Let’s print the shape of our `blob` so we can analyze it in the terminal later (**Line 23**).

Next, we’ll feed `blob` through GoogLeNet:

\# set the input to the pre-trained deep learning network and obtain
# the output predicted probabilities for each of the 1,000 ImageNet
# classes
net.setInput(blob)
preds = net.forward()

# sort the probabilities (in descending) order, grab the index of the
# top predicted label, and draw it on the input image
idx = np.argsort(preds\[0\])\[::-1\]\[0\]
text = "Label: {}, {:.2f}%".format(classes\[idx\],
	preds\[0\]\[idx\] \* 100)
cv2.putText(image, text, (5, 25),  cv2.FONT\_HERSHEY\_SIMPLEX,
	0.7, (0, 0, 255), 2)

# show the output image
cv2.imshow("Image", image)
cv2.waitKey(0)

If you’re familiar with recent deep learning posts on this blog, the above lines should look familiar.

We feed the `blob` through the network (**Lines 28 and 29**) and grab the predictions, `preds` .

Then we sort `preds` (**Line 33**) with the most confident predictions at the front of the list, and generate a label text to display on the image. The label text consists of the class label and the prediction percentage value for the top prediction (**Lines 34 and 35**).

From there, we write the label `text` at the top of the `image` (**Lines 36 and 37**) followed by displaying the `image` on the screen and waiting for a keypress before moving on (**Lines 40 and 41**).

**Now it’s time to use the _plural form_ of the `blobFromImage` function.**

Here we’ll do (nearly) the same thing, except we’ll instead create and populate a list of `images` followed by passing the list as a parameter to `blobFromImages` :

\# initialize the list of images we'll be passing through the network
images = \[\]

# loop over the input images (excluding the first one since we
# already classified it), pre-process each image, and update the
# \`images\` list
for p in imagePaths\[1:\]:
	image = cv2.imread(p)
	image = cv2.resize(image, (224, 224))
	images.append(image)

# convert the images list into an OpenCV-compatible blob
blob = cv2.dnn.blobFromImages(images, 1, (224, 224), (104, 117, 123))
print("Second Blob: {}".format(blob.shape))

First we initialize our `images` list (**Line 44**), and then, using the `imagePaths` , we read, resize, and append the `image` to the list (**Lines 49-52**).

Using list slicing, we’ve omitted the first image from `imagePaths` on **Line 49**.

From there, we pass the `images` into `cv2.dnn.blobFromImages` as the first parameter on **Line 55**. All other parameters to `cv2.dnn.blobFromImages` are identical to `cv2.dnn.blobFromImage` above.

For analysis later we print `blob.shape` on **Line 56**.

We’ll next pass the `blob` through GoogLeNet and write the class label and prediction at the top of each image:

\# set the input to our pre-trained network and obtain the output
# class label predictions
net.setInput(blob)
preds = net.forward()

# loop over the input images
for (i, p) in enumerate(imagePaths\[1:\]):
	# load the image from disk
	image = cv2.imread(p)

	# find the top class label from the \`preds\` list and draw it on
	# the image
	idx = np.argsort(preds\[i\])\[::-1\]\[0\]
	text = "Label: {}, {:.2f}%".format(classes\[idx\],
		preds\[i\]\[idx\] \* 100)
	cv2.putText(image, text, (5, 25),  cv2.FONT\_HERSHEY\_SIMPLEX,
		0.7, (0, 0, 255), 2)

	# display the output image
	cv2.imshow("Image", image)
	cv2.waitKey(0)

The remaining code is essentially the same as above, only our `for` loop now handles looping through each of the `imagePaths` (again, omitting the first one as we have already classified it).

And that’s it! Let’s see the script in action in the next section.

### OpenCV blobfromImage and blobFromImages results

Now we’ve reached the fun part.

Go ahead and use the _**“Downloads”**_ section of this blog post to download the source code, example images, and pre-trained neural network. You will need the additional files in order to execute the code.

From there, fire up a terminal and run the following command:

$ python blob\_from\_images.py

The first terminal output is with respect to the first image found in the `images` folder where we apply the `cv2.dnn.blobFromImage` function:

First Blob: (1, 3, 224, 224)

The resulting beer glass image is displayed on the screen:

[![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20345'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_result01.jpg)

**Figure 2:** An enticing beer has been labeled and recognized with high confidence by GoogLeNet. The blob dimensions resulting from `blobFromImage` are displayed in the terminal.

That full beer glass makes me thirsty. But before I enjoy a beer myself, I’ll explain why the shape of the blob is `(1, 3, 224, 224)` .

The resulting tuple has the following format:

`(num_images=1, num_channels=3, width=224, height=224)`

Since we’ve only processed _one_ image, we only have _one_ entry in our `blob` . The channel count is three for BGR channels. And finally _224×224_ is the spatial width and height for our input image.

Next, let’s build a `blob` from the remaining four input images.

The second blob’s shape is:

Second Blob: (4, 3, 224, 224)

Since this blob contains 4 images, the `num_images=4` . The remaining dimensions are the same as the first, single image, blob.

I’ve included a sample of correctly classified images below:

[![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20407'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_result02.jpg)

**Figure 3:** My keyboard has been correctly identified by GoogLeNet with a prediction confidence of 81%.

  

[![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20407'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_result03.jpg)

**Figure 4:** I tested the pre-trained network on my computer monitor as well. Here we can see the input image is correctly classified using our Convolutional Neural Network.

  

[![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20428'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2017/11/blob_from_images_result04.jpg)

**Figure 5:** A NASA space shuttle is recognized with a prediction value of over 99% by our deep neural network.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s tutorial we examined OpenCV’s `blobFromImage` and `blobFromImages` deep learning functions.

These methods are used to prepare input images for classification via pre-trained deep learning models.

Both `blobFromImage` and `blobFromImages` perform mean subtraction and scaling. We can also swap the Red and Blue channels of the image depending on channel ordering. Nearly all state-of-the-art deep learning models perform mean subtraction and scaling — the benefit here is that OpenCV makes these preprocessing tasks dead simple.

If you’re interested in studying deep learning in more detail, be sure to take a look at my brand new book, _[Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)_.

Inside the book you’ll discover:

*   **Super practical walkthroughs** that present solutions to actual, real-world image classification problems, challenges, and competitions.
*   Detailed, thorough experiments (with highly documented code) enabling you to **reproduce state-of-the-art results.**
*   **My favorite “best practices”** to improve network accuracy. These techniques alone will save you enough time to pay for the book multiple times over.
*   _..and much more!_

Sound good?

**[Click here to start your journey to deep learning mastery](https://pyimagesearch.com/deep-learning-computer-vision-python-book/).**

Otherwise, be sure to enter your email address in the form below to be notified when future deep learning tutorials are published here on the PyImageSearch blog.

See you next week!

![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!