Title: Softmax Classifiers Explained - PyImageSearch

URL Source: https://pyimagesearch.com/2016/09/12/softmax-classifiers-explained/

Published Time: 2016-09-12T14:00:51+00:00

Markdown Content:
![Image 1: Figure 4: Taking the negative log of the probability for the correct ground-truth class yields the final loss for the data point.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20260'%3E%3C/svg%3E)

Last week, we discussed [Multi-class SVM loss](https://pyimagesearch.com/2016/09/05/multi-class-svm-loss/); specifically, the hinge loss and squared hinge loss functions.

A loss function, in the context of Machine Learning and Deep Learning, allows us to quantify how “good” or “bad” a given classification function (also called a “scoring function”) is at correctly classifying data points in our dataset.

However, while hinge loss and squared hinge loss are commonly used when training Machine Learning/Deep Learning classifiers, there is _another_ method more heavily used…

In fact, if you have done previous work in Deep Learning, you have likely heard of this function before — do the terms _**Softmax classifier**_ and _**cross-entropy loss**_ sound familiar?

I’ll go as far to say that if you do _any_ work in Deep Learning (especially Convolutional Neural Networks) that you’ll run into the term “Softmax”: it’s the _final layer_ at the end of the network that _yields your actual probability scores_ for each class label.

To learn more about Softmax classifiers and the cross-entropy loss function, keep reading.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

While hinge loss is quite popular, you’re more likely to run into cross-entropy loss and Softmax classifiers in the context of Deep Learning and Convolutional Neural Networks.

Why is this?

Simply put:

**Softmax classifiers give you _probabilities_ for each class label while hinge loss gives you the _margin__._**

It’s much easier for us as humans to interpret _probabilities_ rather than margin scores (such as in hinge loss and squared hinge loss).

Furthermore, for datasets such as ImageNet, we often look at the rank-5 accuracy of Convolutional Neural Networks (where we check to see if the ground-truth label is in the top-5 predicted labels returned by a network for a given input image).

Seeing (1) if the true class label exists in the top-5 predictions and (2) the _probability_ associated with the predicted label is a nice property.

### Understanding Multinomial Logistic Regression and Softmax Classifiers

The Softmax classifier is a generalization of the binary form of Logistic Regression. Just like in hinge loss or squared hinge loss, our mapping function _f_ is defined such that it takes an input set of data _x_ and maps them to the output class labels via a simple (linear) dot product of the data _x_ and weight matrix _W_:

![Image 3: f(x_{i}, W) = Wx_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20114%2018'%3E%3C/svg%3E)

However, unlike hinge loss, we interpret these scores as _unnormalized log probabilities_ for each class label — this amounts to swapping out our hinge loss function with cross-entropy loss:

![Image 4: L_{i} = -log(e^{s_{y_{i}}} / \sum_{j} e^{s_{j}})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20164%2021'%3E%3C/svg%3E)

So, how did I arrive here? Let’s break the function apart and take a look.

To start, our loss function should minimize the negative log likelihood of the correct class:

![Image 5: L_{i} = -log P(Y=y_{i}|X=x_{i})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20199%2018'%3E%3C/svg%3E)

This probability statement can be interpreted as:

![Image 6: P(Y=k|X=x_{i}) = e^{s_{y_{i}}} / \sum_{j} e^{s_{j}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20225%2021'%3E%3C/svg%3E)

Where we use our standard scoring function form:

![Image 7: s = f(x_{i}, W)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2018'%3E%3C/svg%3E)

As a whole, this yields our final loss function for a _single_ data point, just like above:

![Image 8: L_{i} = -log(e^{s_{y_{i}}} / \sum_{j} e^{s_{j}})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20164%2021'%3E%3C/svg%3E)

_**Note:**_ _Your logarithm here is actually base e (natural logarithm) since we are taking the inverse of the exponentiation over e earlier._

The actual exponentiation and normalization via the sum of exponents is our actual _Softmax function_. The negative log yields our actual _cross-entropy loss._

Just as in hinge loss or squared hinge loss, computing the cross-entropy loss over an _entire dataset_ is done by taking the average:

![Image 9: L = \frac{1}{N} \sum^{N}_{i=1} L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20105%2022'%3E%3C/svg%3E)

If these equations seem scary, don’t worry — I’ll be working an actual numerical example in the next section.

_**Note:**_ _I’m purposely leaving out the regularization term as to not bloat this tutorial or confuse readers. We’ll return to regularization and explain what it is, how to use, and why it’s important for machine learning/deep learning in a future blog post._

### A worked Softmax example

To demonstrate cross-entropy loss in action, consider the following figure:

![Image 10: Figure 1: To compute our cross-entropy loss, let's start with the output of our scoring function (the first column).](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20260'%3E%3C/svg%3E)

**Figure 1:** To compute our cross-entropy loss, let’s start with the output of our scoring function (the first column).

Our goal is to classify whether the image above contains a _dog_, _cat_, _boat_, or _airplane._

Clearly we can see that this image is an “airplane”. _But does our Softmax classifier?_

To find out, I’ve included the output of our scoring function _f_ for each of the four classes, respectively, in **Figure 1** above. These values are our _unnormalized log probabilities_ for the four classes.

_**Note:** I used a random number generator to obtain these values for this particular example. These values are simply used to demonstrate how the calculations of the Softmax classifier/cross-entropy loss function are performed. In reality, these values would not be randomly generated — they would instead be the output of your scoring function_ f.

Let’s exponentiate the output of the scoring function, yielding our _unnormalized probabilities:_

![Image 11: Figure 2: Exponentiating the output values from the scoring function gives us our unnormalized probabilities.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20260'%3E%3C/svg%3E)

**Figure 2:** Exponentiating the output values from the scoring function gives us our unnormalized probabilities.

The next step is to take the denominator, sum the exponents, and divide by the sum, thereby yielding the _actual probabilities associated with each class label:_

![Image 12: Figure 3: To obtain the actual probabilities, we divide each individual unnormalized probability by the sum of unnormalized probabilities.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20260'%3E%3C/svg%3E)

**Figure 3:** To obtain the actual probabilities, we divide each individual unnormalized probability by the sum of unnormalized probabilities.

Finally, we can take the negative log, yielding our final loss:

![Image 13: Figure 4: Taking the negative log of the probability for the correct ground-truth class yields the final loss for the data point.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20260'%3E%3C/svg%3E)

**Figure 4:** Taking the negative log of the probability for the correct ground-truth class yields the final loss for the data point.

In this case, our Softmax classifier would correctly report the image as _airplane_ with 93.15% confidence.

### The Softmax Classifier in Python

In order to demonstrate some of the concepts we have learned thus far with actual Python code, we are going to use a [SGDClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html) with a log loss function.

_**Note:**_ _We’ll learn more about Stochastic Gradient Descent and other optimization methods in future blog posts._

For this example, we’ll once again be using the Kaggle Dogs vs. Cats dataset, so before we get started, make sure you have:

1.  Downloaded the source code to this blog post used the _**“Downloads”**_ form at the bottom of this tutorial.
2.  Downloaded the [Kaggle Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats/data).

In our particular example, the Softmax classifier will actually reduce to a special case — when there are _K=2_ classes, the Softmax classifier reduces to simple Logistic Regression. If we have _\> 2_ classes, then our classification problem would become _Multinomial Logistic Regression_, or more simply, a Softmax classifier.

With that said, open up a new file, name it `softmax.py` , and insert the following code:

\# import the necessary packages
from sklearn.preprocessing import LabelEncoder
from sklearn.linear\_model import SGDClassifier
from sklearn.metrics import classification\_report
from sklearn.cross\_validation import train\_test\_split
from imutils import paths
import numpy as np
import argparse
import imutils
import cv2
import os

If you’ve been following along on the PyImageSearch blog over the past few weeks, then the code above likely looks fairly familiar — all we are doing here is importing our required Python packages.

We’ll be using the [scikit-learn library](http://scikit-learn.org/stable/), so if you don’t already have it installed, be sure to install it now:

$ pip install scikit-learn

We’ll also be using my [imutils package](https://github.com/jrosebr1/imutils), a series of convenience functions used to make performing common image processing operations an easier task. If you do not have `imutils` installed, you’ll want to install it as well:

$ pip install imutils

Next, we define our `extract_color_histogram` function which is used to quantify the color distribution of our input `image` using the supplied number of `bins` :

def extract\_color\_histogram(image, bins=(8, 8, 8)):
	# extract a 3D color histogram from the HSV color space using
	# the supplied number of \`bins\` per channel
	hsv = cv2.cvtColor(image, cv2.COLOR\_BGR2HSV)
	hist = cv2.calcHist(\[hsv\], \[0, 1, 2\], None, bins,
		\[0, 180, 0, 256, 0, 256\])

	# handle normalizing the histogram if we are using OpenCV 2.4.X
	if imutils.is\_cv2():
		hist = cv2.normalize(hist)

	# otherwise, perform "in place" normalization in OpenCV 3 (I
	# personally hate the way this is done
	else:
		cv2.normalize(hist, hist)

	# return the flattened histogram as the feature vector
	return hist.flatten()

I’ve already reviewed this function a few times before, so I’m going to skip the detailed review. For a more thorough discussion of `extract_color_histogram` , why we are using it, and how it works, [please see this blog post](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/).

In the meantime, simply keep in mind that this function **quantifies the contents of an image by constructing a histogram over the pixel intensities.**

Let’s parse our command line arguments and grab the paths to our 25,000 Dogs vs. Cats images from disk:

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-d", "--dataset", required=True,
	help="path to input dataset")
args = vars(ap.parse\_args())

# grab the list of images that we'll be describing
print("\[INFO\] describing images...")
imagePaths = list(paths.list\_images(args\["dataset"\]))

# initialize the data matrix and labels list
data = \[\]
labels = \[\]

We only need a single switch here, `--dataset` , which is the path to our input Dogs vs. Cats images.

Once we have the paths to these images, we can loop over them individually and extract a color histogram for each image:

\# loop over the input images
for (i, imagePath) in enumerate(imagePaths):
	# load the image and extract the class label (assuming that our
	# path as the format: /path/to/dataset/{class}.{image\_num}.jpg
	image = cv2.imread(imagePath)
	label = imagePath.split(os.path.sep)\[-1\].split(".")\[0\]

	# extract a color histogram from the image, then update the
	# data matrix and labels list
	hist = extract\_color\_histogram(image)
	data.append(hist)
	labels.append(label)

	# show an update every 1,000 images
	if i > 0 and i % 1000 == 0:
		print("\[INFO\] processed {}/{}".format(i, len(imagePaths)))

Again, since I have already reviewed this boilerplate code multiple times on the PyImageSearch blog, I’ll refer you to [this blog post](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/) for a more detailed discussion on the feature extraction process.

Our next step is to construct the training and testing split. We’ll use 75% of the data for training our classifier and the remaining 25% for testing and evaluating the model:

\# encode the labels, converting them from strings to integers
le = LabelEncoder()
labels = le.fit\_transform(labels)

# partition the data into training and testing splits, using 75%
# of the data for training and the remaining 25% for testing
print("\[INFO\] constructing training/testing split...")
(trainData, testData, trainLabels, testLabels) = train\_test\_split(
	np.array(data), labels, test\_size=0.25, random\_state=42)

# train a Stochastic Gradient Descent classifier using a softmax
# loss function and 10 epochs
model = SGDClassifier(loss="log", random\_state=967, n\_iter=10)
model.fit(trainData, trainLabels)

# evaluate the classifier
print("\[INFO\] evaluating classifier...")
predictions = model.predict(testData)
print(classification\_report(testLabels, predictions,
	target\_names=le.classes\_))

We also train our `SGDClassifier` using the `log` loss function (**Lines 75 and 76**). Using the `log` loss function ensures that we’ll obtain _probability estimates_ for each class label at testing time.

**Lines 79-82** then display a nicely formatted accuracy report for our classifier.

To examine some actual _probabilities_, let’s loop over a few randomly sampled training examples and examine the output probabilities returned by the classifier:

\# to demonstrate that our classifier actually "learned" from
# our training data, randomly sample a few training images
idxs = np.random.choice(np.arange(0, len(trainData)), size=(5,))

# loop over the training indexes
for i in idxs:
	# predict class probabilities based on the extracted color
	# histogram
	hist = trainData\[i\].reshape(1, -1)
	(catProb, dogProb) = model.predict\_proba(hist)\[0\]

	# show the predicted probabilities along with the actual
	# class label
	print("cat={:.1f}%, dog={:.1f}%, actual={}".format(catProb \* 100,
		dogProb \* 100, le.inverse\_transform(trainLabels\[i\])))

_**Note:** I’m randomly sampling from the_ training data _rather than the_ testing data _to demonstrate that there should be a noticeably large gap in between the probabilities for each class label. Whether or not each classification is_ correct _is a a different story — but even if our prediction is wrong, we should still see some sort of gap that indicates that our classifier is actually learning from the data._

**Line 93** handles computing the probabilities associated with the randomly sampled data point via the `.predict_proba` function.

The predicted probabilities for the cat and dog class are then displayed to our screen on **Lines 97 and 98**.

### Softmax Classifier Results

Once you have:

1.  Downloaded both the source code to this blog using the **_“Downloads”_** form at the bottom of this tutorial.
2.  Downloaded the [Kaggle Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats/data).

You can execute the following command to extract features from our dataset and train our classifier:

$ python softmax.py --dataset kaggle\_dogs\_vs\_cats

After training our `SGDClassifier` , you should see the following classification report:

![Image 14: Figure 5: Computing the accuracy of our SGDClassifier with log loss -- we obtain 65% classification accuracy.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20396'%3E%3C/svg%3E)

**Figure 5:** Computing the accuracy of our SGDClassifier with log loss — we obtain 65% classification accuracy.

Notice that our classifier has obtained **65% accuracy**, an increase from the **64% accuracy** when utilizing a Linear SVM in our [linear classification post](https://pyimagesearch.com/2016/08/22/an-intro-to-linear-classification-with-python/).

To investigate the individual class probabilities for a given data point, take a look at the rest of the `softmax.py`  output:

![Image 15: Figure 6: Investigating the class label probabilities for each prediction.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20514%20287'%3E%3C/svg%3E)

**Figure 6:** Investigating the class label probabilities for each prediction.

For each of the randomly sampled data points, we are given the class label probability for _both_ “dog” and “cat”, along with the _actual ground-truth label._

Based on this sample, we can see that we obtained _4 / 5 = 80%_ accuracy.

But more importantly, notice how there is a _particularly large gap_ in between class label probabilities. If our Softmax classifier predicts “dog”, then the probability associated with “dog” will be high. And conversely, the class label probability associated with “cat” will be low.

Similarly, if our Softmax classifier predicts “cat”, then the probability associated with “cat” will be high, while the probability for “dog” will be “low”.

This behavior implies that there some actual _confidence_ in our predictions and that our algorithm is actually _learning_ from the dataset.

Exactly _how_ the learning takes place involves updating our weight matrix _W_, which boils down to being an _optimization problem_. We’ll be reviewing how to perform gradient decent and other optimization algorithms in future blog posts.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post, we looked at the Softmax classifier, which is simply a generalization of the the binary Logistic Regression classifier.

When constructing Deep Learning and Convolutional Neural Network models, you’ll _undoubtedly_ run in to the Softmax classifier and the cross-entropy loss function.

While both hinge loss and squared hinge loss are popular choices, I can almost guarantee with absolute certainly that you’ll see cross-entropy loss with more frequency — this is mainly due to the fact that the Softmax classifier outputs _probabilities_ rather than _margins_. Probabilities are much easier for us as humans to interpret, so that is a particularly nice quality of Softmax classifiers.

Now that we understand the fundamentals of loss functions, we’re ready to tack on another term to our loss method — _**regularization.**_

The regularization term is appended to our loss function and is used to control how our weight matrix _W_ “looks”. By controlling _W_ and ensuring that it “looks” a certain way, we can actually _increase classification accuracy._

After we discuss regularization, we can then move on to _optimization_ — the process that actually takes the output of our scoring and loss functions and uses this output to tune our weight matrix _W_ to actually “learn”.

Anyway, I hope you enjoyed this blog post!

_**Before you go, be sure to enter your email address in the form below to be notified when new blog posts go live!**_

![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!