Title: An Intro to Linear Classification with Python - PyImageSearch

URL Source: https://pyimagesearch.com/2016/08/22/an-intro-to-linear-classification-with-python/

Published Time: 2016-08-22T14:00:55+00:00

Markdown Content:
We previously learned about the [k-NN classifier](https://pyimagesearch.com/2021/04/17/your-first-image-classifier-using-k-nn-to-classify-images/) — a machine learning model so simple that it doesn’t do any actual “learning” at all. We simply have to store the training data inside the model, and then predictions are made at test time by comparing the testing data points to our training data.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20317'%3E%3C/svg%3E)

We’ve already discussed many of the pros and cons of k-NN, but in the context of large-scale datasets and deep learning, the most prohibitive aspect of k-NN is _the data itself_. While training may be simple, testing is quite slow, with the bottleneck being the distance computation between vectors. Computing the distances between training and testing points scales linearly with the number of points in our dataset, making the method impractical when our datasets become quite large. And while we can apply Approximate Nearest Neighbor methods such as [ANN](https://www.cs.umd.edu/~mount/ANN/), [FLANN](https://ieeexplore.ieee.org/document/6809191), or [Annoy](https://github.com/spotify/annoy), to speed up the search, that still doesn’t alleviate the problem that k-NN cannot function without maintaining a replica of data inside the instantiation (or at least have a pointer to training set on disk, etc.).

To see why storing an exact replica of the training data inside the model is an issue, consider training a k-NN model and then deploying it to a customer base of 100, 1,000, or even 1,000,000 users. If your training set is only a few megabytes, this may not be a problem — but if your training set is measured in _gigabytes_ to _terabytes_ (as is the case for many datasets that we apply deep learning to), you have a real problem on your hands.

Consider the training set of the [ImageNet dataset](https://link.springer.com/article/10.1007%2Fs11263-015-0816-y), which includes over 1.2 million images. If we trained a k-NN model on this dataset and then tried to deploy it to a set of users, we would need these users to download the k-NN model which internally represents _replicas_ of the 1.2 million images. Depending on how you compress and store the data, this model could measure in hundreds of gigabytes to terabytes in storage costs and network overhead. Not only is this a waste of resources, but it’s also not optimal for constructing a machine learning model.

Instead, a more desirable approach would be to define a machine learning model that can _learn_ _patterns_ from our input data during training time (requiring us to spend more time on the training process), but have the benefit of being defined by a _small number of parameters_ that can easily be used to represent the model, _regardless of training size_. This type of machine learning is called _parameterized learning_, which is defined as:

> _A learning model that summarizes data with a set of parameters of fixed size (independent of the number of training examples) is called a parametric model. No matter how much data you throw at the parametric model, it won’t change its mind about how many parameters it needs._
> 
> _—_ [Russell and Norvig (2009)](https://www.amazon.com/dp/0136042597/)

We’ll review the concept of parameterized learning and discuss how to implement a simple linear classifier. As we’ll see later, parameterized learning is the cornerstone of modern day machine learning and deep learning algorithms.

**_Remark:_** Much of this material was inspired by Andrej Karpathy’s excellent Linear Classification notes inside [Stanford’s cs231n class](https://cs231n.github.io/linear-classify/). A big thank you to Karpathy and the rest of the cs231n teaching assistants for putting together such accessible notes.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

I’ve used the word “parameterized” a few times now, but what exactly does it mean?

**Simply put:** **_parameterization_** **is the process of defining the necessary parameters of a given model.** In the task of machine learning, parameterization involves defining a problem in terms of four key components: _data_, a _scoring function_, a _loss function_, and _weights and biases_. We’ll review each of these below.

**Data**

This component is our _input data_ that we are going to learn from. This data includes _both_ the data points (i.e., raw pixel intensities from images, extracted features, etc.) and their associated class labels. Typically we denote our data in terms of a multi-dimensional **[design matrix](https://www.deeplearningbook.org/)**.

Each row in the design matrix represents a data point while each column (which itself could be a multi-dimensional array) of the matrix corresponds to a different feature. For example, consider a dataset of 100 images in the RGB color space, each image sized _32×32_ pixels. The design matrix for this dataset would be _X ⊆ R100×(32×32×3)_, where _Xi_ defines the _i_\-th image in _R_. Using this notation, _X_1 is the first image, _X_2 the second image, and so on.

Along with the design matrix, we also define a vector _y_ where _y__i_ provides the class label for the _i_\-th example in the dataset.

**Scoring Function**

The scoring function accepts our data as an input and maps the data to class labels. For instance, given our set of input images, the scoring function takes these data points, applies some function _f_ (our scoring function), and then returns the predicted class labels, similar to the pseudocode below:

INPUT\_IMAGES => F(INPUT\_IMAGES) => OUTPUT\_CLASS\_LABELS

**Loss Function**

A loss function quantifies how well our _predicted class labels_ agree with our _ground-truth labels_. The higher level of agreement between these two sets of labels, the _lower our loss_ (and higher our classification accuracy, at least on the training set).

Our goal when training a machine learning model is to _minimize the loss function_, thereby increasing our classification accuracy.

**Weights and Biases**

The weight matrix, typically denoted as **_W_** and the bias vector **_b_** are called the **weights** or **parameters** of our classifier that we’ll actually be optimizing. Based on the output of our scoring function and loss function, we’ll be tweaking and fiddling with the values of the weights and biases to increase classification accuracy.

Depending on your model type, there may exist many more parameters, but at the most basic level, these are the four building blocks of parameterized learning that you’ll commonly encounter. Once we’ve defined these four key components, we can then apply optimization methods that allow us to find a set of parameters **_W_** and **_b_** that minimize our loss function with respect to our scoring function (while increasing classification accuracy on our data).

Next, let’s look at how these components can work together to build a linear classifier, transforming the input data into actual predictions.

### **Linear Classification: From Images to Labels**

In this section, we are going to look at a more mathematical motivation of the parameterized model approach to machine learning.

To start, we need our **_data_**. Let’s assume that our training dataset is denoted as _xi_ where each image has an associated class label _yi_. We’ll assume that _i = 1, …, N_ and _yi \= 1, …, K_, implying that we have _N_ data points of dimensionality _D_, separated into _K_ unique categories.

To make this idea more concrete, consider an “Animals” dataset from the lesson on [image classifiers](https://pyimagesearch.com/2021/04/17/your-first-image-classifier-using-k-nn-to-classify-images/). In such a dataset, we may have _N = 3,000_ total images. Each image is _32×32_ pixels, represented in the RGB color space (i.e., three channels per image). We can represent each image as _D = 32×32×3 = 3,072_ distinct values. Finally, we know there are a total of _K = 3_ class labels: one for the dog, cat, and panda classes, respectively.

Given these variables, we must now define a scoring function _f_ that maps the images to the class label scores. One method to accomplish this scoring is via a simple linear mapping:

**(1)** ![Image 3: f(x_i,W,b) = W x_i +b](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20153%2018'%3E%3C/svg%3E)

Let’s assume that each _xi_ is represented as a single column vector with shape \[_D×1_\] (in this example we would flatten the _32×32×3_ image into a list of 3,072 integers). Our weight matrix **_W_** would then have a shape of \[_K×D_\] (the number of class labels by the dimensionality of the input images). Finally **_b_**, the **bias vector** would be of size \[_K×1_\]. The bias vector allows us to shift and translate our scoring function in one direction or another without actually influencing our weight matrix **_W_**. The bias parameter is often critical for successful learning.

Going back to the Animals dataset example, each _xi_ is represented by a list of 3,072 pixel values, so _xi_, therefore, has the shape \[_3,072×1_\]. The weight matrix **_W_** will have a shape of \[_3×3,072_\] and finally the bias vector **_b_** will be of size \[_3×1_\].

**Figure 1** follows an illustration of the linear classification scoring function _f_ . On the _left_, we have our original input image, represented as a _32×32×3_ image. We then _flatten_ this image into a list of 3,072 pixel intensities by taking the 3D array and reshaping it into a 1D list.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20312'%3E%3C/svg%3E)

**Figure 1:** Illustrating the dot product of the weight matrix **_W_** and feature vector **_x_**, followed by the addition of the bias term. Figure inspired by Karpathy’s example in [Stanford University’s CS231n course](http://cs231n.stanford.edu/).

Our weight matrix **_W_** contains three rows (one for each class label) and 3,072 columns (one for each of the pixels in the image). After taking the dot product between **_W_** and _x__i_, we add in the bias vector **_b_** — the result is our actual **scoring function**. Our scoring function yields three values on the _right_: the scores associated with the dog, cat, and panda labels, respectively.

**_Remark:_** Readers who are unfamiliar with taking dot products should read this quick and concise [tutorial](http://pyimg.co/fgcvp). For readers interested in studying linear algebra in-depth, I highly recommend working through [_Coding the Matrix Linear Algebra through Applications to Computer Science_ by Philip N. Klein (2013)](https://books.google.com/books?id=3AA4nwEACAAJ).

Looking at **Figure 1** and **Equation (1)**, you can convince yourself that the input _xi_ and _yi_ are _fixed_ and _not something we can modify_. Sure, we can obtain different _xi_s by applying various transformations to the input image — but once we pass the image into the scoring function, **these values do not change**. In fact, the only parameters that we have any control over (in terms of parameterized learning) are our weight matrix **_W_** and our bias vector **_b_**. Therefore, our goal is to utilize both our scoring function and loss function to _optimize_ (i.e., modify in a systematic way) the weight and bias vectors such that our classification accuracy _increases_.

Exactly _how_ we optimize the weight matrix depends on our loss function, but typically involves some form of gradient descent.

### **Advantages of Parameterized Learning and Linear Classification**

There are two primary advantages to utilizing parameterized learning:

1.  **Once we are done training our model, we can discard the input data and keep only the weight matrix _W_ and the bias vector _b_**. This _substantially_ reduces the size of our model since we need to store two sets of vectors (versus the _entire_ training set).
2.  **Classifying new test data is _fast_.** In order to perform a classification, all we need to do is take the dot product of **_W_** and _xi_, follow by adding in the bias **_b_** (i.e., apply our scoring function). Doing it this way is _significantly faster_ than needing to compare each testing point to _every_ training example, as in the k-NN algorithm.

### **A Simple Linear Classifier With Python**

Now that we’ve reviewed the concept of parameterized learning and linear classification, let’s implement a _very simple_ linear classifier using Python.

The purpose of this example is _not_ to demonstrate how we train a model from start to finish. The purpose of this example is to show how we would initialize a weight matrix **_W_**, bias vector **_b_**, and then use these parameters to classify an image via a simple dot product.

Let’s go ahead and get this example started. Our goal here is to write a Python script that will correctly classify **Figure 2** as _“dog.”_

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20336'%3E%3C/svg%3E)

**Figure 2:** Our example input image that we are going to classifier with a simple linear classifier.

To see how we can accomplish this classification, open a new file, name it `linear_example.py`, and insert the following code:

\# import the necessary packages
import numpy as np
import cv2

# initialize the class labels and set the seed of the pseudorandom
# number generator so we can reproduce our results
labels = \["dog", "cat", "panda"\]
np.random.seed(1)

**Lines 2 and 3** import our required Python packages. We’ll use NumPy for our numerical processing and OpenCV to load our example image from disk.

**Line 7** initializes the list of target class labels for the “Animals” dataset while **Line 8** sets the pseudorandom number generator for NumPy, ensuring that we can reproduce the results of this experiment.

Next, let’s initialize our weight matrix and bias vector:

\# randomly initialize our weight matrix and bias vector -- in a
# \*real\* training and classification task, these parameters would
# be \*learned\* by our model, but for the sake of this example,
# let's use random values
W = np.random.randn(3, 3072)
b = np.random.randn(3)

**Line 14** initializes the weight matrix `W` with random values from a normal distribution, with zero mean and unit variance. Weights inside of a neural network are able to take on negative, positive, and zero values. This weight matrix has `3` rows (one for each of the class labels) and `3072` columns (one for each of the pixels in our _32×32×3_ image).

We then initialize the bias vector on **Line 15** — this vector is also randomly filled with values sampled over a standard normal distribution. Our bias vector has `3` rows (corresponding to the number of class labels) along with one column.

If we were training this linear classifier _from scratch_ we would need to _learn_ the values of `W` and `b` through an optimization process. However, since we have not reached the optimization stage of training a model, I have initialized the pseudorandom number generator with a value `1` to ensure the random values give us the “correct” classification (I tested random initialization values ahead of time to determine which value gives us the correct classification). For the time being, simply treat the weight matrix `W` and the bias vector `b` as “black box arrays” that are optimized in a magical way — we’ll pull back the curtain and reveal how these parameters are learned in future lessons.

Now that our weight matrix and bias vector are initialized, let’s load our example image from disk:

\# load our example image, resize it, and then flatten it into our
# "feature vector" representation
orig = cv2.imread("beagle.png")
image = cv2.resize(orig, (32, 32)).flatten()

**Line 19** loads our image from disk via `cv2.imread`. We then resize the image to _32×32_ pixels (ignoring the aspect ratio) on **Line 20** — our image is now represented as a `(32, 32, 3)` NumPy array, which we flatten into a 3,072-dim vector.

The next step is to compute the output class label scores by applying our scoring function:

\# compute the output scores by taking the dot product between the
# weight matrix and image pixels, followed by adding in the bias
scores = W.dot(image) + b

**Line 24** is the scoring function itself — it’s simply the dot product between the weight matrix `W` and the input image pixel intensities, followed by adding in the bias `b`.

Finally, our last code block handles writing the scoring function values for each of the class labels to our terminal, then displaying the result to our screen:

\# loop over the scores + labels and display them
for (label, score) in zip(labels, scores):
	print("\[INFO\] {}: {:.2f}".format(label, score))

# draw the label with the highest score on the image as our
# prediction
cv2.putText(orig, "Label: {}".format(labels\[np.argmax(scores)\]),
	(10, 30), cv2.FONT\_HERSHEY\_SIMPLEX, 0.9, (0, 255, 0), 2)

# display our input image
cv2.imshow("Image", orig)
cv2.waitKey(0)

To execute our example, just issue the following command:

$ python linear\_example.py 
\[INFO\] dog: 7963.93
\[INFO\] cat: -2930.99
\[INFO\] panda: 3362.47

Notice how the _dog_ class has the _largest scoring function value_, which implies that the “dog” class would be chosen as the prediction by our classifier. In fact, we can see the text `dog` correctly drawn on our input image (**Figure 2**) in **Figure 3**.

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20691%20500'%3E%3C/svg%3E)

**Figure 3:** In this example, our linear classifier was correctly able to label the input image as _dog_; however, keep in mind that this is a worked example. In a later lesson, you’ll learn how to _train_ our weights and biases to automatically make these predictions.

Again, keep in mind that this was a _worked example_. I **_purposely_** set the random state of our Python script to generate `W` and `b` values that would lead to the correct classification (you can change the pseudorandom seed value on **Line 8** to see for yourself how different random initializations will produce different output predictions).

In practice, you would _never_ initialize your `W` and `b` values and _assume_ they would give you the correct classification without some sort of learning process. Instead, when training our own machine learning models from scratch we would need to _optimize_ and _learn_ `W` and `b` via an optimization algorithm, such as gradient descent.

We’ll cover optimization and gradient descent in a future lesson, but in the meantime, take the time to ensure you understand **Line 24** and how a linear classifier makes a classification by taking the dot product between a weight matrix and an input data point, followed by adding in the bias. Our _entire model_ can therefore be defined via two values: the weight matrix and the bias vector. This representation is not only _compact_ but also quite _powerful_ when we train machine learning models from scratch.

**The Role of Loss Functions**
------------------------------

In our last section we discussed the concept of parameterized learning. This type of learning allows us to take sets of input data and class labels, and actually learn a function that _maps_ the input to the output predictions by defining a set of parameters and optimizing over them.

But in order to actually “learn” the mapping from the input data to class labels via our scoring function, we need to discuss two important concepts:

1.  Loss functions
2.  Optimization methods

Let’s explore common loss functions you’ll encounter when building neural networks and deep learning networks.

Our exploration is a _brief review_ of loss functions and their role in parameterized learning. If you wish to complement this lesson with more mathematically rigorous derivations, I would highly recommend [Andrew Ng’s Coursera course](https://www.coursera.org/learn/machine-learning), [Witten et al. (2011)](https://www.amazon.com/Data-Mining-Practical-Techniques-Management/dp/0123748569/ref=sr_1_1?dchild=1&keywords=ISBN%3A+9780123748560&qid=1617456349&s=books&sr=1-1), [Harrington (2012)](https://www.amazon.com/1617290181-9781617290183-Machine-Learning-Paperback/dp/B07L5M2XZ3/), and [Marsland (2009)](https://www.amazon.com/Machine-Learning-Algorithmic-Perspective-Recognition/dp/1420067184/).

### **What Are Loss Functions?**

At the most basic level, a loss function quantifies how “good” or “bad” a given predictor is at classifying the input data points in a dataset. A visualization of loss functions plotted over time for two separate models trained on the CIFAR-10 dataset is shown in **Figure 4**. The smaller the loss, the better a job the classifier is at modeling the relationship between the input data and output class labels (although there is a point where we can _overfit_ our model — by modeling the training data _too closely_, our model loses the ability to generalize. Conversely, the larger our loss, the _more work needs to be done_ to increase classification accuracy.

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20667%20500'%3E%3C/svg%3E)

**Figure 4:** The training losses for two separate models trained on the CIFAR-10 dataset are plotted over time. Our loss function quantifies how “good” or “bad” of a job a given model is doing at classifying data points from the dataset. **Model #1** achieves a considerably lower loss than **Model #2**.

To improve our classification accuracy, we need to tune the parameters of our weight matrix **_W_** or bias vector **_b_**. Exactly _how_ we go about updating these parameters is an _optimization_ _problem_, which we’ll be covering in a future lesson. For the time being, simply understand that a _loss function_ can be used to quantify how well our _scoring function_ is doing at classifying input data points.

Ideally, our loss should decrease over time as we tune our model parameters. As **Figure 4** demonstrates, **Model #1**’s loss starts slightly higher than **Model #2**, but then decreases rapidly and continues to stay low when trained on the CIFAR-10 dataset. Conversely, the loss for **Model #2** decreases initially but quickly stagnates. In this specific example, **Model #1** is achieving lower overall loss and is likely a more desirable model to be used on classifying other images from the CIFAR-10 dataset. I say “likely” because there is a chance that **Model #1** has overfit to the training data. We’ll cover this concept of overfitting and how to spot it in a future lesson.

### **Multi-class SVM Loss**

Multi-class SVM Loss (as the name suggests) is inspired by (Linear) [Support Vector Machines (SVMs)](http://dx.doi.org/10.1023/A:1022627411411), which uses a scoring function _f_ to map our data points to numerical scores for each class labels. This function _f_ is a simple learning mapping:

**(2)** ![Image 8: f(x_i,W,b) = W x_i +b](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20153%2018'%3E%3C/svg%3E)

Now that we have our scoring function, we need to determine how “good” or “bad” this function is (given the weight matrix **_W_** and bias vector **_b_**) at making predictions. To make this determination, we need a _loss function_.

Recall that when creating a machine learning model we have a _design matrix_ **_X_**, where each row in **_X_** contains a data point we wish to classify. In the context of image classification, each row in **_X_** is an image and we seek to correctly label this image. We can access the _i_\-th image inside **_X_** via the syntax _x__i_.

Similarly, we also have a vector **_y_**_,_ which contains our class labels for each **_X_**. These **_y_** values are our _ground-truth labels_ and what we hope our scoring function will correctly predict. Just like we can access a given image as _xi_, we can access the associated class label via _yi_.

As a matter of simplicity, let’s abbreviate our scoring function as _s_:

(**3**) ![Image 9: s = f(x_i,W)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2018'%3E%3C/svg%3E)

which implies that we can obtain the predicted score of the _j_\-th class via the _i_\-th data point:

(**4**) ![Image 10: s_{j} = f(x_{i}, W)_{j}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20102%2019'%3E%3C/svg%3E)

Using this syntax, we can put it all together, obtaining the **_hinge loss function_**:

(**5**) ![Image 11: L_{i} = \sum\limits_{j \neq y_{i}} \max(0, s_{j} - s_{y_{i}} + 1)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20203%2032'%3E%3C/svg%3E)

**_Remark:_** Nearly all loss functions include a regularization term. I am skipping this idea for now as we’ll review regularization in a future lesson when we understand loss functions better.

Looking at the hinge loss equation above, you might be confused at what it’s actually doing. Essentially, the hinge loss function is summing across all _incorrect classes_ (_i ≠_ _j_) and comparing the output of our scoring function _s_ returned for the _j_\-th class label (the incorrect class) and the _yi_\-th class (the correct class). We apply the `max` operation to clamp values at zero, which is important to ensure we do not sum negative values.

A given _xi_ is classified correctly when the loss _Li_ \= 0 (I’ll provide a numerical example in the following section). To derive the loss across our _entire training set_, we simply take the mean over each individual _Li_:

**(6)** ![Image 12:  L = {\displaystyle\frac{1}{N}} \sum\limits^{N}_{i=1} L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2091%2039'%3E%3C/svg%3E)

Another related loss function you may encounter is the **_squared hinge loss_**:

**(7)** ![Image 13: L_{i} = \sum\limits_{j \neq y_i} \max(0, s_{j} - s_{y_{i}} + 1)^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20210%2032'%3E%3C/svg%3E)

The squared term penalizes our loss more heavily by squaring the output, which leads to quadratic growth in loss in a prediction that is incorrect (versus a linear growth).

As for which loss function you should use, that is entirely dependent on your dataset. It’s typical to see the standard hinge loss function used more, but on some datasets the squared variation may obtain better accuracy. Overall, this is a _hyperparameter_ that you should consider tuning.

**A Multi-class SVM Loss Example**

Now that we’ve taken a look at the mathematics behind hinge loss, let’s examine a worked example. We’ll again be using the “Animals” dataset which aims to classify a given image as containing a _cat_, _dog_, or _panda_. To start, take a look at **Figure 5**, where I have included three training examples from the three classes of the “Animals” dataset.

![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20904%20410'%3E%3C/svg%3E)

**Figure 5:** At the top of the figure we have three input images: one for each of the dog, cat, and panda class, respectively. The body of the table contains the scoring function outputs for each of the classes. We will use the scoring function to derive the total loss for each input image.

Given some arbitrary weight matrix **_W_** and bias vector **_b_**, the output scores of _f_(_x,W_) = _W x_+_b_ are displayed in the body of the matrix. The _larger_ the scores are, the _more confident_ our scoring function is regarding the prediction.

Let’s start by computing the loss _Li_ for the “dog” class:

\>>> max(0, 1.33 - 4.26 + 1) + max(0, -1.01 - 4.26 + 1)
0

Notice how our equation here includes two terms — the difference between the predicted dog score and _both_ the cat and panda score. Also, observe how the loss for “dog” is _zero_ — this implies that the dog was correctly predicted. A quick investigation of _Image #1_ from **Figure 5** demonstrates this result to be true: the “dog” score is greater than both the “cat” and “panda” scores.

Similarly, we can compute the hinge loss for _Image #2_, this one containing a cat:

\>>> max(0, 3.76 - (-1.20) + 1) + max(0, -3.81 - (-1.20) + 1)
5.96

In this case, our loss function is greater than zero, indicating that our prediction is _incorrect_. Looking at our scoring function, we see that our model predicts _dog_ as the proposed label with a score of 3_._76 (as this is the label with the highest score). We know that this label is incorrect — and in a future lesson, we’ll learn how to automatically tune our weights to correct these predictions.

Finally, let’s compute the hinge loss for the panda example:

\>>> max(0, -2.37 - (-2.27) + 1) + max(0, 1.03 - (-2.27) + 1)
5.199999999999999

Again, our loss is non-zero, so we know we have an incorrect prediction. Looking at our scoring function, our model has incorrectly labeled this image as “cat” when it should be “panda.”

We can then obtain the _total loss_ over the three examples by taking the average:

\>>> (0.0 + 5.96 + 5.2) / 3.0
3.72

Therefore, given our three training examples our overall hinge loss is 3_._72 for the parameters **_W_** and **_b_**.

Also take note that our loss was zero for only _one_ of the three input images, implying that _two_ of our predictions were incorrect. In a future lesson, we’ll learn how to optimize **_W_** and **_b_** to make better predictions by using the loss function to help drive and steer us in the right direction.

### **Cross-entropy Loss and Softmax Classifiers**

While hinge loss is quite popular, you’re _much_ more likely to run into cross-entropy loss and Softmax classifiers in the context of deep learning and convolutional neural networks.

Why is this? Simply put:

**Softmax classifiers give you** **_probabilities_** **for each class label while hinge loss gives you the** **_margin_****.**

It’s much easier for us as humans to interpret _probabilities_ rather than margin scores. Furthermore, for datasets such as ImageNet, we often look at the rank-5 accuracy of Convolutional Neural Networks (where we check to see if the ground-truth label is in the top-5 predicted labels returned by a network for a given input image). Seeing if (1) the true class label exists in the top-5 predictions and (2) the _probability_ associated with each label is a nice property.

**Understanding Cross-entropy Loss**

The Softmax classifier is a generalization of the binary form of Logistic Regression. Just like in hinge loss or squared hinge loss, our mapping function _f_ is defined such that it takes an input set of data _x__i_ and maps them to output class labels via dot product of the data _x__i_ and weight matrix **_W_** (omitting the bias term for brevity):

**(8)** ![Image 15: f(x_i,W) = W x_i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20114%2018'%3E%3C/svg%3E)

However, unlike hinge loss, we can interpret these scores as _unnormalized log probabilities_ for each class label, which amounts to swapping out the hinge loss function with cross-entropy loss:

**(9)** ![Image 16: L_{i} = -\log(e^{s_{y_{i}}} / \sum\limits_{j} e^{s_{j}})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20160%2031'%3E%3C/svg%3E)

So, how did I arrive here? Let’s break the function apart and take a look. To start, our loss function should minimize the negative log likelihood of the correct class:

**(10)** ![Image 17: L_i = -\log P(Y = y_i|X = x_i) ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20204%2018'%3E%3C/svg%3E)

The probability statement can be interpreted as:

**(11)** ![Image 18: P(Y=y_{i}|X=x_{i}) = e^{s_{y_{i}}} / \sum\limits_{j} e^{s_{j}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20222%2031'%3E%3C/svg%3E)

where we use our standard scoring function form:

**(12)** ![Image 19: s = f(x_i,W)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2018'%3E%3C/svg%3E)

As a whole, this yields our final loss function for a _single_ data point, just like above:

**(13)** ![Image 20:  L_{i} = -\log(e^{s_{y_{i}}} / \sum\limits_{j} e^{s_{j}})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20160%2031'%3E%3C/svg%3E)

Take note that your logarithm here is actually base _e_ (natural logarithm) since we are taking the inverse of the exponentiation over _e_ earlier. The actual exponentiation and normalization via the sum of exponents is our _Softmax function_. The negative log yields our actual _cross-entropy loss_.

Just as in hinge loss and squared hinge loss, computing the cross-entropy loss over an entire dataset is done by taking the average:

**(14)** ![Image 21: L = \displaystyle\frac{1}{N} \sum\limits^{N}_{i=1} L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2095%2049'%3E%3C/svg%3E)

Again, I’m purposely omitting the regularization term from our loss function. We’ll return to regularization, explain what it is, how to use it, and why it’s critical to neural networks and deep learning in a future lesson. If the equations above seem scary, don’t worry — we’re about to work through numerical examples in the next section to ensure you understand how cross-entropy loss works.

**A Worked Softmax Example**

To demonstrate cross-entropy loss in action, consider **Figure 6**. Our goal is to classify whether the image above contains a _dog_, _cat_, or _panda_. Clearly, we can see that the image is a “panda” — _but what does our Softmax classifier think?_ To find out, we’ll need to work through each of the four tables in the figure.

![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20596%20500'%3E%3C/svg%3E)

**Figure 6:** _First Table:_ To compute our cross-entropy loss, let’s start with the output of the scoring function. _Second Table:_ Exponentiating the output values from the scoring function gives us our unnormalized probabilities. _Third Table:_ To obtain the actual probabilities, we divide each individual unnormalized probabilities by the sum of all unnormalized probabilities. _Fourth Table:_ Taking the negative natural logarithm of the probability for the correct ground-truth yields the final loss for the data point.

The _first table_ includes the output of our scoring function _f_ for each of the three classes, respectively. These values are our _unnormalized log probabilities_ for the three classes. Let’s exponentiate the output of the scoring function (_e__s_, where _s_ is our score function value), yielding our _unnormalized probabilities_ (_second table_).

The next step is to take the denominator of **Equation (11)**, sum the exponents, and divide by the sum, thereby yielding the _actual probabilities associated with each class label_ (_third table_). Notice how the probabilities sum to one.

Finally, we can take the negative natural logarithm, _−_ln(_p_), where _p_ is the normalized probability, yielding our final loss (the _fourth and final table_).

In this case, our Softmax classifier would correctly report the image as _panda_ with 93.93% confidence. We can then repeat this process for all images in our training set, take the average, and obtain the overall cross-entropy loss for the training set. This process allows us to quantify how good or bad a set of parameters are performing on our training set.

**_Remark:_** I used a random number generator to obtain the score function values for this particular example. These values are simply used to demonstrate how the calculations of the Softmax classifier/cross-entropy loss function are performed. In reality, these values would _not_ be randomly generated — they would instead be the output of your scoring function _f_ based on your parameters **_W_** _and_ **_b_**.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this lesson, we reviewed four components of parameterized learning:

1.  Data
2.  Scoring function
3.  Loss function
4.  Weights and biases

In the context of image classification, our input data is our dataset of images. The scoring function produces _predictions_ for a given input image. The loss function then _quantifies_ how good or bad a set of predictions are over the dataset. Finally, the weight matrix and bias vectors are what enable us to actually “learn” from the input data — these parameters will be tweaked and tuned via optimization methods in an attempt to obtain higher classification accuracy.

We then reviewed two popular loss functions: _hinge loss_ and _cross-entropy loss_. While hinge loss is used in many machine learning applications (e.g., SVMs), I can almost guarantee with absolute certainty that you’ll see cross-entropy loss with more frequency primarily due to the fact that Softmax classifiers output _probabilities_ rather than _margins_. Probabilities are much easier for us as humans to interpret, so this fact is a particularly nice quality of cross-entropy loss and Softmax classifiers. For more information on hinge loss and cross-entropy loss, please refer to **Stanford University’s CS231n course ([http://cs231n.stanford.edu](http://cs231n.stanford.edu/)**, [https://cs231n.github.io/linear-classify](https://cs231n.github.io/linear-classify/)**)**.

In a future lesson, we’ll review optimization methods that are used to tune our weight matrix and bias vector. Optimization methods allow our algorithms to actually _learn_ from our input data by updating the weight matrix and bias vector based on the output of our scoring and loss functions. Using these techniques we can take _incremental steps_ towards parameter values that obtain lower loss and higher accuracy. Optimization methods are the cornerstone of modern day neural networks and deep learning, and without them, we would be unable to learn patterns from our input data.

![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!