Title: Multi-class SVM Loss - PyImageSearch

URL Source: https://pyimagesearch.com/2016/09/05/multi-class-svm-loss/

Published Time: 2016-09-05T14:00:26+00:00

Markdown Content:
![Image 1: mutliclass_svm_loss](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20510'%3E%3C/svg%3E)

A couple weeks ago,we discussed the concepts of both [_linear classification_ and _parameterized learning_](https://pyimagesearch.com/2016/08/22/an-intro-to-linear-classification-with-python/). This type of learning allows us to take a set of input data and class labels, and actually learn a function that _maps_ the input to the output predictions, simply by defining a set of parameters and optimizing over them.

Our linear classification tutorial focused mainly on the concept of a _scoring function_ and how it can be used to map the input data to class labels. But in order to actually “learn” the mapping from the input data to class labels, we need to discuss two important concepts:

1.  Loss functions
2.  Optimization methods

In today and next week’s blog posts, we’ll be discussing two _loss functions_ that you’ll commonly see when applying Machine Learning, Neural Networks, and Deep Learning algorithms:

1.  Multi-class SVM loss
2.  Cross-entropy (used for Softmax classifiers/Multinomial Logistic Regression)

To learn more about your first loss function, Multi-class SVM loss, just keep reading.

At the most basic level, a loss function is simply used to quantify how “good” or “bad” a given predictor is at classifying the input data points in a dataset.

The smaller the loss, the better a job our classifier is at modeling the relationship between the input data and the output class labels (although there is a point where we can _overfit_ our model — this happens when the training data is modeled _too closely_ and our model loses the ability to generalize).

Conversely, the larger our loss is, the _more work needs to be done_ to increase classification accuracy. In terms of parameterized learning, this involves tuning parameters such as our weight matrix _W_ or bias vector _b_ to improve classification accuracy. Exactly _how_ we go about updating these parameters is an _optimization problem_, which we’ll be covering later in this series of tutorials.

### The mathematics behind Multi-class SVM loss

After reading through the [linear classification with Python tutorial](https://pyimagesearch.com/2016/08/22/an-intro-to-linear-classification-with-python/), you’ll note that we used a [Linear Support Vector machine (SVM)](https://en.wikipedia.org/wiki/Support_vector_machine) as our classifier of choice.

This previous tutorial focused on the concept of a scoring function _f_ that maps our feature vectors to class labels as numerical scores. As the name suggests, a Linear SVM applies a simple linear mapping:

![Image 2: f(x_{i}, W, b) = Wx_{i} + b](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20153%2018'%3E%3C/svg%3E)

Now that we have this scoring/mapping function _f_, we need to determine how “good” or “bad” this function is (given the weight matrix _W_ and bias vector _b_) at making predictions.

To accomplish this, we need a _**loss function**._ Let’s go ahead and start defining that now.

Based on our [previous linear classification tutorial](https://pyimagesearch.com/2016/08/22/an-intro-to-linear-classification-with-python/), we know that we have a matrix of feature vectors _x_ — these feature vectors could be extracted color histograms, Histogram of Oriented Gradients features, or even raw pixel intensities.

Regardless of _how_ we choose to quantify our images, the point is that we have a matrix _x_ of features extracted from our image dataset. We can then access the features associated with a given image via the syntax ![Image 3: x_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E), which will yield the _i_\-th feature vector inside _x_.

Similarly, we also have a vector _y_ which contains our _class labels_ for each _x_. These _y_ values are our _ground-truth labels_ and what we hope our scoring function will correctly predict. Just like we can access a given feature vector via ![Image 4: x_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E), we can access the _i_\-th class label via ![Image 5: y_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E).

As a matter of simplicity, let’s abbreviate our scoring function as _s_:

![Image 6: s = f(x_{i}, W)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2018'%3E%3C/svg%3E)

Which implies that we can obtain the predicted score of the _j_\-th class via for the _i_\-th data point:

![Image 7: s_{j} = f(x_{i}, W)_{j}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20102%2019'%3E%3C/svg%3E).

Using this syntax, we can put it all together, obtaining the _**hinge loss function:**_

![Image 8: L_{i} = \sum_{j \neq y_{i}} max(0, s_{j} - s_{y_{i}} + 1)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20223%2022'%3E%3C/svg%3E)

_**Note:** I’m purposely skipping the regularization parameter for now. We’ll return to regularization in a future post once we better understand loss functions._

So what is the above equation doing exactly?

I’m glad you asked.

Essentially, the hinge loss function is summing across all _incorrect classes_ (![Image 9: i \neq j](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2034%2016'%3E%3C/svg%3E)) and comparing the output of our scoring function _s_ returned for the _j_\-th class label (the incorrect class) and the ![Image 10: y_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E)\-th class (the correct class).

We apply the _max_ operation to clamp values to _0_ — this is important to do so that we do not end up summing negative values.

A given ![Image 11: x_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E) is classified correctly when the loss ![Image 12: L_{i} = 0](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2046%2015'%3E%3C/svg%3E) (I’ll provide a numerical example of this in the following section).

To derive the loss across our _**entire training set**_, we simply take the mean over each individual ![Image 13: L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2014'%3E%3C/svg%3E):

![Image 14: L = \frac{1}{N} \sum^{N}_{i=1} L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20105%2022'%3E%3C/svg%3E)

Another related, common loss function you may come across is the _**squared hinge loss:**_

![Image 15: L_{i} = \sum_{j \neq y_{i}} max(0, s_{j} - s_{y_{i}} + 1)^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20230%2022'%3E%3C/svg%3E)

The squared term penalizes our loss more heavily by squaring the output. This leads to a quadratic growth in loss rather than a linear one.

As for which loss function you should use, that is entirely dependent on your dataset. It’s typical to see the standard hinge loss function used more often, but on some datasets the squared variation might obtain better accuracy — **overall, this is a [hyperparameter that you should cross-validate](https://pyimagesearch.com/2016/08/15/how-to-tune-hyperparameters-with-python-and-scikit-learn/).**

### A Multi-class SVM loss example

Now that we’ve taken a look at the mathematics behind hinge loss and squared hinge loss, let’s take a look at a worked example.

We’ll again assume that we’re working with the [Kaggle Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats/data), which as the name suggests, aims to classify whether a given image contains a _dog_ or a _cat_.

There are only two possible class labels in this dataset and is therefore a 2-class problem which can be solved using a standard, binary SVM loss function. That said, let’s still apply Multi-class SVM loss so we can have a worked example on how to apply it. **From there, I’ll extend the example to handle a 3-class problem as well.**

To start, take a look at the following figure where I have included 2 training examples from the 2 classes of the Dogs vs. Cats dataset:

![Image 16: Figure 1: Let's apply hinge loss to the images in this figure.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20316'%3E%3C/svg%3E)

**Figure 1:** Let’s apply hinge loss to the images in this figure.

Given some (arbitrary) weight matrix _W_ and bias vector _b_, the output scores of ![Image 17: f(x, W) = Wx + b](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20132%2018'%3E%3C/svg%3E) are displayed in the body of the matrix. The _larger_ the scores are, the more _confident_ our scoring function is regarding the prediction.

Let’s start by computing the loss ![Image 18: L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2014'%3E%3C/svg%3E) for the “dog” class. Given a two class problem, this is trivially easy:

\>>> max(0, 1.33 - 4.26 + 1)
0
>>>

Notice how the loss for “dog” is _zero_ — this implies that the dog class was correctly predicted. A quick investigation of **Figure 1** above demonstrates this to be true: the “dog” score is greater than the “cat” score.

Similarly, we can do the same for the second image, this one containing a cat:

\>>> max(0, 3.76 - (-1.2) + 1)
5.96
>>>

In this case, our loss function is greater than zero, indicating that our prediction is not correct.

We then obtain the _total loss_ over the two example images by taking the average:

\>>> (0 + 5.96) / 2
2.98
>>

That was simple enough for a 2-class problem, but what about a 3-class problem? Does the process become more complicated?

In reality, it doesn’t — our summation just expands a bit. You can find an example of a 3-class problem below, were I have added a third class, “horse”:

![Image 19: Figure 2: An example of applying hinge loss to a 3-class image classification problem.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20272'%3E%3C/svg%3E)

**Figure 2:** An example of applying hinge loss to a 3-class image classification problem.

Let’s again compute the loss for the dog class:

\>>> max(0, 1.49 - (-0.39) + 1) + max(0, 4.21 - (-0.39) + 1)
8.48
>>>

Notice how that our summation has expanded to include two terms — the difference between the predicted dog score and **_both_** the cat and horse score.

Similarly, we can compute the loss for the cat class:

\>>> max(0, -4.61 - 3.28 + 1) + max(0, 1.46 - 3.28 + 1)
0
>>>

And finally the loss for the horse class:

\>>> max(0, 1.03 - (-2.27) + 1) + max(0, -2.37 - (-2.27) + 1)
5.199999999999999
>>>

The total loss is therefore:

\>>> (8.48 + 0.0 + 5.2) / 3
4.56
>>>

As you can see, the same general principles apply — just keep in mind that as the number of classes expands, your summation will expand as well.

_**Quiz:** Based on the_ loss _from the three classes above, which one was classified correctly?_

### Do I need to implement Multi-class SVM Loss by hand?

If you want, you could implement hinge loss and squared hinge loss by hand — but this would mainly be for educational purposes.

You’ll see both hinge loss and squared hinge loss implemented in nearly any machine learning/deep learning library, including scikit-learn, Keras, Caffe, etc.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

Today I discussed the concept of Multi-class SVM loss. Given a _scoring function_ (which maps input data to output class labels), our _loss function_ can be used to quantify how “good” or “bad” our scoring function is at predicting the correct class labels in our dataset.

The smaller the loss, the more accurate our predictions are (but we also run the risk of “overfitting”, where we model the mapping of the input data to class labels _too closely_).

Conversely, the larger the loss, the less accurate our predictions are, thus we need to optimize our _W_ and _b_ parameters further — but we’ll save optimization methods for future posts once we better understand loss functions.

After understanding the concept of “loss” and how it applies to machine learning and deep learning algorithms, we then looked at two specific loss functions:

1.  Hinge loss
2.  Squared hinge loss

In general, you’l see hinge loss more often — but it’s still worth attempting to [tune the hyperparameters](https://pyimagesearch.com/2016/08/15/how-to-tune-hyperparameters-with-python-and-scikit-learn/) to your classifier to determine which loss function gives better accuracy on your particular dataset.

Next week I’ll be back to discuss a second loss function — cross-entropy — and the relation it has to Multinomial Logistic Regression. If you have any prior experience in machine learning or deep learning, you may know this function better as the _**Softmax classifier.**_

If you’re interested in applying Deep Learning and Convolutional Neural Networks, then you don’t want to miss this upcoming post — as you’ll find out, **the Softmax classifier is the most used model/loss function in Deep Learning.**

_**To be notified when the next blog post is published, just enter your email address in the form below. See you next week!**_

![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.