A definitive explanation to the Hinge Loss for Support Vector Machines. | by Vagif Aliyev | Towards Data Science

NOTE: This article assumes that you are familiar with how an SVM operates. If this is not the case for you, be sure to check my out previous article which breaks down the SVM algorithm from first principles, and also includes a coded implementation of the algorithm from scratch!
I have seen lots of articles and blog posts on the Hinge Loss and how it works. However, I find most of them to be quite vague and not giving a clear explanation of what exactly the function does and what it is. Instead, most of the time an unclear graph is shown and the reader is left bewildered.
In this article, I hope to explain the function in a simplified manner, both visually and mathematically to help you grasp a solid understanding of the cost function. But before we dive in, let’s refresh your knowledge of cost functions!
Essentially, A cost function is a function that measures the loss, or cost, of a specific model. It is essentially an error rate that tells you how well your model is performing by means of a specific mathematical formula. The main goal in Machine Learning is to tune your model so that the cost of your model is minimised.
If you have done any Kaggle Tournaments, you may have seen them as the metric they use to score your model on the leaderboard. Some examples of cost functions (other than the hinge loss) include:
Root Mean Squared Error(Regression)
Logarithmic Loss(Classification)
Mean Absolute Error(Regression)
Gini Impurity(Classification)
As you might have deducted, Hinge Loss is also a type of cost function that is specifically tailored to Support Vector Machines. Why this loss exactly and not the other losses mentioned above? Well, why don’t we find out with our first introduction to the Hinge Loss!
The hinge loss is a loss function used for training classifiers, most notably the SVM. Here is a really good visualisation of what it looks like.
The x-axis represents the distance from the boundary of any single instance, and the y-axis represents the loss size, or penalty, that the function will incur depending on its distance.
There are a few things to note here:
That dotted line on the x-axis represents the number 1. This means that when an instance’s distance from the boundary is greater than or at 1, our loss size is 0. Keep this in mind, as it will really help in understanding the maths of the function.If the distance from the boundary is 0 (meaning that the instance is literally on the boundary), then we incur a loss size of 1.We see that correctly classified points will have a small(or none) loss size, while incorrectly classified instances will have a high loss size.A negative distance from the boundary incurs a high hinge loss. This essentially means that we are on the wrong side of the boundary, and that the instance will be classified incorrectly.On the flip size, a positive distance from the boundary incurs a low hinge loss, or no hinge loss at all, and the further we are away from the boundary(and on the right side of it), the lower our hinge loss will be.
That dotted line on the x-axis represents the number 1. This means that when an instance’s distance from the boundary is greater than or at 1, our loss size is 0. Keep this in mind, as it will really help in understanding the maths of the function.
If the distance from the boundary is 0 (meaning that the instance is literally on the boundary), then we incur a loss size of 1.
We see that correctly classified points will have a small(or none) loss size, while incorrectly classified instances will have a high loss size.
A negative distance from the boundary incurs a high hinge loss. This essentially means that we are on the wrong side of the boundary, and that the instance will be classified incorrectly.
On the flip size, a positive distance from the boundary incurs a low hinge loss, or no hinge loss at all, and the further we are away from the boundary(and on the right side of it), the lower our hinge loss will be.
Now, Let’s see a more numerical visualisation:
This graph essentially strengthens the observations we made from the previous visualisation. We can see that again, when an instance’s distance is greater or equal to 1, it has a hinge loss of zero.
When the point is at the boundary, the hinge loss is one(denoted by the green box), and when the distance from the boundary is negative(meaning it’s on the wrong side of the boundary) we get an incrementally larger hinge loss.
By now you should have a pretty good idea of what hinge loss is and how it works. Now, before we actually get to the maths of the hinge loss, let’s further strengthen our knowledge of the loss function by understanding it with the use of a table!
Now, let’s examine the hinge loss for a number of predictions made by a hypothetical SVM:
     actual  predicted  hinge loss==================================[0]   +1        0.97       0.03 [1]   +1        1.20       0.00      [2]   +1        0.00       1.00       [3]   +1       -0.25       1.25[4]   -1       -0.88       0.12       [5]   -1       -1.01       0.00[6]   -1       -0.00       1.00[7]   -1        0.40       1.40
One key characteristic of the SVM and the Hinge loss is that the boundary separates negative and positive instances as +1 and -1, with -1 being on the left side of the boundary and +1 being on the right.
Now, we can examine specific examples:
[0]: the actual value of this instance is +1 and the predicted value is 0.97, so the hinge loss is very small as the instance is very far away from the boundary.
[1]: the actual value of this instance is +1 and the predicted value is 1.2, which is greater than 1, thus resulting in no hinge loss
[2]: the actual value of this instance is +1 and the predicted value is 0, which means that the point is on the boundary, thus incurring a cost of 1.
[3]: the actual value of this instance is +1 and the predicted value is -0.25, meaning the point is on the wrong side of the boundary, thus incurring a large hinge loss of 1.25
[4]: the actual value of this instance is -1 and the predicted value is -0.88, which is a correct classification but the point is slightly penalised because it is slightly on the margin
[5]: the actual value of this instance is -1 and the predicted value is -1.01, again perfect classification and the point is not on the margin, resulting in a loss of 0
[6]: the actual value of this instance is -1 and the predicted value is 0, which means that the point is on the boundary, thus incurring a cost of 1.
[7]: the actual value of this instance is -1 and the predicted value is 0.40, meaning the point is on the wrong side of the boundary, thus incurring a large hinge loss of 1.40
Hopefully this intuitive example gave you a better sense of how hinge loss works. By now, you are probably wondering how to compute hinge loss, which leads us to the math behind hinge loss!
Hinge loss is actually quite simple to compute. The formula for hinge loss is given by the following:
With l referring to the loss of any given instance, y[i] and x[i] referring to the ith instance in the training set and b referring to the bias term
This formula can be broken down to the following:
Now, I recommend you to actually make up some points and calculate the hinge loss for those points. Try and verify your findings by looking at the graphs at the beginning of the article and seeing if your predictions seem reasonable.
Seemingly daunting at first, Hinge Loss may seem like a terrifying concept to grasp, but I hope that I have enlightened you on the simple yet effective strategy that the hinge loss formula incorporates.
I hope you have learned something new, and I hope you have benefited positively from this article. I wish you all the best in the future, and implore you to stay tuned for more!