Title: Image Classification Basics - PyImageSearch

URL Source: https://pyimagesearch.com/2021/04/17/image-classification-basics/

Published Time: 2021-04-17T14:00:00+00:00

Markdown Content:
> _A picture is worth a thousand words_.
> 
> — English idiom

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20650%20488'%3E%3C/svg%3E)

We’ve heard this adage countless times in our lives. It simply means that a complex idea can be conveyed in a single image. Whether examining the line chart of our stock portfolio investments, looking at the spread of an upcoming football game, or simply taking in the art and brush strokes of a painting master, we are constantly ingesting visual content, interpreting the meaning, and storing the knowledge for later use.

However, for computers, interpreting the contents of an image is less trivial — all our computer sees is a big matrix of numbers. It has _no idea_ regarding the thoughts, knowledge, or meaning the image is trying to convey.

In order to understand the contents of an image, we must apply **_image classification_**, which is the task of using computer vision and machine learning algorithms to extract meaning from an image. This action could be as simple as assigning a label to what the image contains, or as advanced as interpreting the contents of an image and returning a human-readable sentence.

Image classification is a very large field of study, encompassing a wide variety of techniques — and with the popularity of deep learning, it is continuing to grow.

**Now is the time to ride the deep learning and image classification wave — those who successfully do so will be handsomely rewarded.**

Image classification and image understanding are currently (and will continue to be) the most popular sub-field of computer vision for the next ten years. In the future, we’ll see companies like Google, Microsoft, Baidu, and others quickly acquire successful image understanding startup companies. We’ll see more and more consumer applications on our smartphones that can understand and interpret the contents of an image. Even wars will likely be fought using unmanned aircrafts that are _automatically_ guided using computer vision algorithms.

Inside this chapter, I’ll provide a high-level overview of what image classification is, along with the many challenges an image classification algorithm has to overcome. We’ll also review the three different types of learning associated with image classification and machine learning.

Finally, we’ll wrap up this chapter by discussing the four steps of training a deep learning network for image classification and how this four-step pipeline compares to the _traditional,_ _hand-engineered_ feature extraction pipeline.

**What Is Image Classification?**
---------------------------------

Image classification, at its very core, is the task of _assigning a label to an image_ from a _predefined_ _set of categories_.

Practically, this means that our task is to analyze an input image and return a label that categorizes the image. The label is always from a predefined set of possible categories.

For example, let’s assume that our set of possible categories includes:

`categories = {cat, dog, panda}`

Then we present the following image (**Figure 1**) to our classification system:

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20697%20446'%3E%3C/svg%3E)

**Figure 1:** The goal of an image classification system is to take an input image and assign a label based on a predefined set of categories.

Our goal here is to take this input image and assign a label to it from our **categories** set — in this case, **dog**.

Our classification system could also assign multiple labels to the image via probabilities, such as **dog: 95%**; **cat: 4%**; **panda: 1%**.

More formally, given our input image of _W×H_ pixels with three channels, Red, Green, and Blue, respectively, our goal is to take the _W×H×3 = N_ pixel image and figure out how to correctly classify the contents of the image.

### **A Note on Terminology**

When performing machine learning and deep learning, we have a **_dataset_** we are trying to extract knowledge from. Each example/item in the dataset (whether it be image data, text data, audio data, etc.) is a **_data point_**. A dataset is therefore a collection of data points (**Figure 2**).

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20730%20618'%3E%3C/svg%3E)

**Figure 2:** A dataset (_outer rectangle_) is a _collection_ of data points (_circles_).

Our goal is to apply machine learning and deep learning algorithms to discover underlying patterns in the dataset, enabling us to correctly classify data points that our algorithm has not encountered yet. Take the time now to familiarize yourself with this terminology:

1.  In the context of image classification, our _dataset_ is a _collection of images_.
2.  Each _image_ is, therefore, a _data point_.

I’ll be using the term _image_ and _data point_ interchangeably throughout the rest of this book, so keep this in mind now.

### **The Semantic Gap**

Take a look at the two photos (_top_) in **Figure 3**. It should be fairly trivial for us to tell the difference between the two photos — there is clearly a _cat_ on the left and a _dog_ on the right. But all a computer sees is two big matrices of pixels (_bottom_).

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20650%20488'%3E%3C/svg%3E)

**Figure 3:** _Top:_ Our brains can clearly see the difference between an image that contains a _cat_ and an image that contains a _dog_. _Bottom:_ However, all a computer “sees” is a big matrix of numbers. The difference between how we perceive an image and how the image is represented (a matrix of numbers) is called the _semantic gap_.

Given that all a computer sees is a big matrix of pixels, we arrive at the problem of the _semantic_ _gap_. The semantic gap is the difference between how a human _perceives_ the contents of an image versus how an image can be _represented_ in a way a computer can understand the process.

Again, a quick visual examination of the two photos above can reveal the difference between the two species of an animal. But in reality, the computer has _no idea_ there are animals in the image to begin with. To make this point clear, take a look at **Figure 4**, containing a photo of a tranquil beach.

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20445'%3E%3C/svg%3E)

**Figure 4:** When describing the contents of this image we may focus on words that convey the _spatial layout_, _color_, and _texture_ — the same is true for computer vision algorithms.

We might describe the image as follows:

*   **Spatial:** The sky is at the top of the image and the sand/ocean are at the bottom.
*   **Color:** The sky is dark blue, the ocean water is a lighter blue than the sky, while the sand is tan.
*   **Texture:** The sky has a relatively uniform pattern, while the sand is very coarse.

How do we go about encoding all this information in a way that a computer can understand it? The answer is to apply _feature extraction_ to quantify the contents of an image. Feature extraction is the process of taking an input image, applying an algorithm, and obtaining a feature vector (i.e., a list of numbers) that quantifies our image.

To accomplish this process, we may consider applying hand-engineered features such as HOG, LBPs, or other “traditional” approaches to image quantifying. Another method, and the one taken by this book, is to apply deep learning to automatically _learn a set of features_ that can be used to quantify and ultimately _label_ the contents of the image itself.

However, it’s not that simple . . . because once we start examining images in the real world, we are faced with many, _many_ challenges.

### **Challenges**

If the semantic gap were not enough of a problem, we also have to handle **[factors of variation](http://www.deeplearningbook.org/)** in how an image or object appears. **Figure 5** displays a visualization of a number of these factors of variation.

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20413%20558'%3E%3C/svg%3E)

**Figure 5:** When developing an image classification system, we need to be cognizant of how an object can appear at varying viewpoints, lighting conditions, occlusions, scale, etc.

To start, we have **_viewpoint variation_**, where an object can be oriented/rotated in multiple dimensions with respect to how the object is photographed and captured. No matter the angle in which we capture this Raspberry Pi, it’s still a Raspberry Pi.

We also have to account for **_scale variation_** as well. Have you ever ordered a tall, grande, or venti cup of coffee from Starbucks? Technically they are all the same thing — a cup of coffee. But they are all different _sizes_ of a cup of coffee. Furthermore, that same venti coffee will look dramatically different when it is photographed up close versus when it is captured from farther away. Our image classification methods must be tolerable to these types of scale variations.

One of the hardest variations to account for is **_deformation_**. For those of you familiar with the television series _Gumby_, we can see the main character in the image above. As the name of the TV show suggests, this character is elastic, stretchable, and capable of contorting his body in many different poses. We can look at these images of _Gumby_ as a type of _object deformation_ — all images contain the _Gumby_ character; however, they are all dramatically different from each other.

Our image classification should also be able to handle **_occlusions_**, where large parts of the object we want to classify are hidden from view in the image (**Figure 5**). On the _left,_ we have to have a picture of a dog. And on the _right,_ we have a photo of the same dog, but notice how the dog is resting underneath the covers, _occluded_ from our view. The dog is still clearly in both images — she’s just more visible in one image than the other. Image classification algorithms should still be able to detect and label the presence of the dog in both images.

Just as challenging as the _deformations_ and _occlusions_ mentioned above, we also need to handle the changes in **_illumination_**. Take a look at the coffee cup captured in standard and low lighting (**Figure 5**). The image on the _left_ was photographed with standard overhead lighting while the image on the _right_ was captured with very little lighting. We are still examining the same cup — but based on the lighting conditions, the cup looks dramatically different (nice how the vertical cardboard seam of the cup is clearly visible in the low lighting conditions, but not the standard lighting).

Continuing on, we must also account for **_background clutter_**. Ever play a game of _Where’s Waldo?_ (Or _Where’s Wally?_ for our international readers.) If so, then you know the goal of the game is to find our favorite red-and-white, striped shirt friend. However, these puzzles are more than just an entertaining children’s game — they are also the perfect representation of _background clutter_. These images are incredibly “noisy” and have a lot going on in them. We are only interested in _one_ particular object in the image; however, due to all the “noise,” it’s not easy to pick out Waldo/Wally. If it’s not easy for us to do, imagine how hard it is for a computer with no semantic understanding of the image!

Finally, we have **_intra-class variation_**. The canonical example of intra-class variation in computer vision is displaying the diversification of chairs. From comfy chairs that we use to curl up and read a book, to chairs that line our kitchen table for family gatherings, to ultra-modern art deco chairs found in prestigious homes, a chair is still a chair — and our image classification algorithms must be able to categorize all these variations correctly.

Are you starting to feel a bit overwhelmed with the complexity of building an image classifier? Unfortunately, it only gets worse — it’s not enough for our image classification system to be robust to these variations _independently_, but our system must also handle _multiple variations combined_ _together!_

So how do we account for such an incredible number of variations in objects/images? In general, we try to frame the problem as best we can. We make assumptions regarding the contents of our images and to which variations we want to be tolerant. We also consider the scope of our project — what is the end goal? And what are we trying to build?

Successful computer vision, image classification, and deep learning systems deployed to the real-world make _careful assumptions and considerations_ before a single line of code is ever written.

If you take too broad of an approach, such as _“I want to classify and detect every single object_ _in my kitchen,”_ (where there could be hundreds of possible objects) then your classification system is unlikely to perform well unless you have years of experience building image classifiers — and even then, there is no guarantee to the success of the project.

But if you **frame your problem** and make it narrow in scope, such as _“I want to recognize just stoves and refrigerators,”_ then your system is **much more likely** to be accurate and functioning, _especially_ if this is your first time working with image classification and deep learning.

The key takeaway here is to **_always consider the scope of your image classifier_**. While deep learning and Convolutional Neural Networks have demonstrated significant robustness and classification power under a variety of challenges, you _still_ should keep the scope of your project as tight and well-defined as possible.

Keep in mind that [ImageNet](https://link.springer.com/article/10.1007/s11263-015-0816-y), the _de facto_ standard benchmark dataset for image classification algorithms, consists of 1,000 objects that we encounter in our everyday lives — and this dataset is _still_ actively used by researchers trying to push the state-of-the-art for deep learning forward.

Deep learning is _not_ magic. Instead, deep learning is like a scroll saw in your garage — powerful and useful when wielded correctly, but hazardous if used without proper consideration. Throughout the rest of this book, I will guide you on your deep learning journey and help point out when you should reach for these power tools and when you should instead refer to a simpler approach (or mention if a problem isn’t reasonable for image classification to solve).

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Types of Learning**
---------------------

There are three types of learning that you are likely to encounter in your machine learning and deep learning career: supervised learning, unsupervised learning, and semi-supervised learning. This book focuses mostly on supervised learning in the context of deep learning. Nonetheless, descriptions of all three types of learning are presented below.

### **Supervised Learning**

Imagine this: you’ve just graduated from college with your Bachelors of Science in Computer Science. You’re young. Broke. And looking for a job in the field — perhaps you even feel lost in your job search.

But before you know it, a Google recruiter finds you on LinkedIn and offers you a position working on their Gmail software. Are you going to take it? Most likely.

A few weeks later, you pull up to Google’s spectacular campus in Mountain View, California, overwhelmed by the breathtaking landscape, the fleet of Teslas in the parking lot, and the almost never-ending rows of gourmet food in the cafeteria.

You finally sit down at your desk in a wide-open workspace among hundreds of other employees . . . _and then you find out your role in the company_. You’ve been hired to create a piece of software to _automatically classify_ email as _spam_ or _not-spam_.

How are you going to accomplish this goal? Would a rule-based approach work? Could you write a series of `if/else` statements that look for certain words and then determine if an email is spam based on these rules? That might work . . . to a degree. But this approach would also be easily defeated and near impossible to maintain.

Instead, what you _really need_ is machine learning. You need a _training set_ consisting of the emails themselves along with their _labels_, in this case, _spam_ or _not-spam_. Given this data, you can analyze the text (i.e., the distributions of words) in the email and utilize the spam/not-spam labels to teach a machine learning classifier what words occur in a spam email and which do not — all without having to manually create a long and complicated series of `if/else` statements.

This example of creating a spam filter system is an example of **supervised learning**. Supervised learning is arguably the most well-known and studied type of machine learning. Given our training data, a model (or “classifier”) is created through a training process where predictions are made on the input data and then corrected when the predictions are wrong. This training process continues until the model achieves some desired stopping criterion, such as a low error rate or a maximum number of training iterations.

Common supervised learning algorithms include Logistic Regression, Support Vector Machines (SVMs) ([Cortes and Vapnik, 1995](http://dx.doi.org/10.1023/A:1022627411411), [Boser et al., 1992](http://doi.acm.org/10.1145/130385.130401)), [Random Forests](http://dx.doi.org/10.1023/A:1010933404324), and Artificial Neural Networks.

In the context of **image classification**, we assume our image dataset consists of the images themselves along with their corresponding _class label_ that we can use to teach our machine learning classifier what each category “looks like.” If our classifier makes an incorrect prediction, we can then apply methods to correct its mistake.

The differences between supervised, unsupervised, and semi-supervised learning can best be understood by looking at the example in **Table 1**. The first column of our table is the label associated with a particular image. The remaining six columns correspond to our feature vector for each data point — here, we have chosen to quantify our image contents by computing the mean and standard deviation for each RGB color channel, respectively.

| **Label** | **_Rµ_** | **_Gµ_** | **_Bµ_** | **_Rσ_** | **_Gσ_** | **_Bσ_** |
| --- | --- | --- | --- | --- | --- | --- |
| Cat | 57.61 | 41.36 | 123.44 | 158.33 | 149.86 | 93.33 |
| Cat | 120.23 | 121.59 | 181.43 | 145.58 | 69.13 | 116.91 |
| Cat | 124.15 | 193.35 | 65.77 | 23.63 | 193.74 | 162.70 |
| Dog | 100.28 | 163.82 | 104.81 | 19.62 | 117.07 | 21.11 |
| Dog | 177.43 | 22.31 | 149.49 | 197.41 | 18.99 | 187.78 |
| Dog | 149.73 | 87.17 | 187.97 | 50.27 | 87.15 | 36.65 |

**Table 1:** A table of data containing both the class labels (either _dog_ or _cat_) and feature vectors for each data point (the mean and standard deviation of each Red, Green, and Blue color channel, respectively). This is an example of a **_supervised classification_** task.

Our supervised learning algorithm will make predictions on each of these feature vectors, and if it makes an incorrect prediction, we’ll attempt to correct it by telling it what the correct label actually is. This process will then continue until the desired stopping criterion has been met, such as accuracy, number of iterations of the learning process, or simply an arbitrary amount of wall time.

**_Remark:_** To explain the differences between supervised, unsupervised, and semi-supervised learning, I have chosen to use a feature-based approach (i.e., the mean and standard deviation of the RGB color channels) to quantify the content of an image. When we start working with Convolutional Neural Networks, we’ll actually **skip** the feature extraction step and use the raw pixel intensities themselves. Since images can be large _MxN_ matrices (and therefore cannot fit nicely into this spreadsheet/table example), I have used the feature-extraction process to help visualize the differences between types of learning.

### **Unsupervised Learning**

In contrast to supervised learning, **unsupervised learning** (sometimes called **self-taught learning**) has no labels associated with the input data and thus we cannot correct our model if it makes an incorrect prediction.

Going back to the spreadsheet example, converting a supervised learning problem to an unsupervised learning one is as simple as removing the “label” column (**Table 2**).

Unsupervised learning is sometimes considered the “holy grail” of machine learning and image classification. When we consider the number of images on Flickr or the number of videos on YouTube, we quickly realize there is a _vast amount_ of unlabeled data available on the internet. If we could get our algorithm to learn patterns from _unlabeled data_, then we wouldn’t have to spend large amounts of time (and money) arduously labeling images for supervised tasks.

| _**Rµ**_ | _**Gµ**_ | _**Bµ**_ | _**Rσ**_ | _**Gσ**_ | _**Bσ**_ |
| --- | --- | --- | --- | --- | --- |
| 57.61 | 41.36 | 123.44 | 158.33 | 149.86 | 93.33 |
| 120.23 | 121.59 | 181.43 | 145.58 | 69.13 | 116.91 |
| 124.15 | 193.35 | 65.77 | 23.63 | 193.74 | 162.70 |
| 100.28 | 163.82 | 104.81 | 19.62 | 117.07 | 21.11 |
| 177.43 | 22.31 | 149.49 | 197.41 | 18.99 | 187.78 |
| 149.73 | 87.17 | 187.97 | 50.27 | 87.15 | 36.65 |

**Table 2:** Unsupervised learning algorithms attempt to learn underlying patterns in a dataset _without_ class labels. In this example we have removed the class label column, thus turning this task into an **_unsupervised learning_** problem.

Most unsupervised learning algorithms are most successful when we can learn the underlying structure of a dataset and then, in turn, apply our learned features to a _supervised_ learning problem where there is too little labeled data to be of use.

Classic machine learning algorithms for unsupervised learning include Principal Component Analysis (PCA) and k-means clustering. Specific to neural networks, we see Autoencoders, Self Organizing Maps (SOMs), and Adaptive Resonance Theory applied to unsupervised learning. Unsupervised learning is an extremely active area of research and one that has yet to be solved. We do not focus on unsupervised learning in this book.

### **Semi-supervised Learning**

So, what happens if we only have _some_ of the labels associated with our data and _no labels_ for the other? Is there a way we can apply some hybrid of supervised and unsupervised learning and still be able to classify each of the data points? It turns out the answer is _yes_ — we just need to apply semi-supervised learning.

Going back to our spreadsheet example, let’s say we only have labels for a small fraction of our input data (**Table 3**). Our semi-supervised learning algorithm would take the known pieces of data, analyze them, and try to label each of the unlabeled data points for use as _additional_ training data. This process can repeat for many iterations as the semi-supervised algorithm learns the “structure” of the data to make more accurate predictions and generate more reliable training data.

| **Label** | **_Rµ_** | **_Gµ_** | **_Bµ_** | **_Rσ_** | **_Gσ_** | **_Bσ_** |
| --- | --- | --- | --- | --- | --- | --- |
| Cat | 57.61 | 41.36 | 123.44 | 158.33 | 149.86 | 93.33 |
| ? | 120.23 | 121.59 | 181.43 | 145.58 | 69.13 | 116.91 |
| ? | 124.15 | 193.35 | 65.77 | 23.63 | 193.74 | 162.70 |
| Dog | 100.28 | 163.82 | 104.81 | 19.62 | 117.07 | 21.11 |
| ? | 177.43 | 22.31 | 149.49 | 197.41 | 18.99 | 187.78 |
| Dog | 149.73 | 87.17 | 187.97 | 50.27 | 87.15 | 36.65 |

**Table 3:** When performing **_semi-supervised learning_** we only have the labels for a subset of the images/feature vectors and must try to label the other data points to utilize them as extra training data.

Semi-supervised learning is especially useful in computer vision where it is often time-consuming, tedious, and expensive (at least in terms of man-hours) to label each and every single image in our training set. In cases where we simply do not have the time or resources to label each individual image, we can label only a tiny fraction of our data and utilize semi-supervised learning to label and classify the rest of the images.

Semi-supervised learning algorithms often trade smaller labeled input datasets for some tolerable reduction in classification accuracy. Normally, the more accurately labeled training a supervised learning algorithm has, the more accurate predictions it can make (this is _especially_ true for deep learning algorithms).

As the amount of training data decreases, accuracy inevitably suffers. Semi-supervised learning takes this relationship between accuracy and amount of data into account and attempts to keep classification accuracy within tolerable limits while dramatically reducing the amount of training data required to build a model — the end result is an accurate classifier (but normally not as accurate as a supervised classifier) with less effort and training data. Popular choices for semi-supervised learning include [label spreading](https://www.google.com/books/edition/Advances_in_Neural_Information_Processin/0F-9C7K8fQ8C?hl=en&gbpv=1&dq=Learning+with+Local+and+Global+Consistency&pg=PA321&printsec=frontcover), [label propagation](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.14.3864&rep=rep1&type=pdf), [ladder networks](http://arxiv.org/abs/1507.02672), and [co-learning/co-training](http://doi.acm.org/10.1145/279943.279962).

Again, we’ll primarily be focusing on supervised learning inside this book, as both unsupervised and semi-supervised learning in the context of deep learning for computer vision are still very active research topics without clear guidelines on which methods to use.

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.