Mathematics behind Supervised Learning-Classification
By: Garima Malik
Tags: Supervised Learning, Machine Learning, Classification, Generalization

This article talks about the basic classification in machine learning or specifically in supervised learning. How a classifier builds the boundary to separate the two classes. It also talks about how binary classification is important and mathematics behind the supervised learning classification task.

We will try to understand the concept of classification with the simplest case of classification i.e. binary classification or learning a classifier that can separate one class from another. This article tries to develop an academic understanding of this concept.

Learning a Class From Examples

Let us assume we have 2 classes namely family cars and non-family cars. for simplicity family cars are positive examples denoted by ‘+’ and non-family cars are negative examples denoted by ‘-’.

Learning a classifier means that is there any boundary or level of separation that exists between given two classes? so, if we are able to find a boundary between positive and negative examples we are good to go…….

To see these classes graphically we basically need some features through which we can look and decide what will be the class for a particular feature instance. In this example, we can take two features such as price and engine power.

Visualizing the Data

Figure: plotting the input points on the X-Y plane and show positive and negative classes

On X-axis It shows the price and Y-axis shows engine power. Positive circles denote the family cars and negative circle shows non-family cars. Now, Our task is to build a classifier that truly classifies the newly input instances to either in the family car or non-family car class.

Building a Boundary

Figure: showing the creation of hypothetical classifier which separates both the classes

When we talk about separating two things the simple thought can come in our minds after seeing the data distribution can be visualized through the above figure. It shows that simply a rectangle or a box can separate or creates the boundary between the given classes.

Equation 1 : (p1 ≤ price ≤ p2 ) AND (e1 ≤ engine power ≤ e2) p1, p2, and e1,e2 are the points corresponding to the boundary on the X-Y plane.

After analyzing the rectangular boundary, we can say that if an input instance follows equation 1 and lie in the condition it can be classified as positive or in the class of family cars otherwise can be classified as negative or non-family cars class.

Concept of Specification and Generalization

The basic goal of machine learning experts is to learn the most general classifiers which can easily classify the new input instances. Building a classifier specific to the training instances gives problems on test data as it fits the training data well however on test data it will follow the structure of a specific classifier which results in lower classification accuracy in testing the classifier on new data.

It is always good to make a generalized classifier that can really fit the test data. Let’s understand the concept of specification and generalization through the example of family cars and non-family cars.

Figure: showing the specific and general classifier

S: most specific classifier which basically tries to separate the two classes by building the tightest rectangle that includes all the positive class.

G: most general classifier which basically draws the largest rectangle possible which can include all the positive class none of the negative examples.

C: Our hypothesis or classifier which is between specific and general classifier.

The difference between S and G is called the margin and we try to maximize this margin so that we can learn nearly generalized or may be close to a general classifier.

Advantages of simple classifiers

We can have numerous kinds of classifiers that can draw a boundary between two classes such as rectangles, circle, ellipse, or some complex shape which can separate the classes. the more complex the classifier is more no of parameters will be needed to train the classifier.

For example, if your classifier is a rectangle, will only need the four points as p1,p2, and e1,e2 to build the rectangular classifier however if you take the ellipse you need the major axis radius, minor axis radius, a center of an ellipse which basically complicates the classification process.

If it is a simple model just like a rectangular classifier it is easy to see whether an input instance falls into the rectangle or outside the rectangle Fewer the number of parameters, it is easy to find out the corner values of a rectangle rather than finding the control points of arbitrary shape. A simple model is also said to have low variance or high bias. It will be relatively easy to explain a classifier that is build of rectangular shape or even simpler boundary such as a line.

Conclusion

In the nutshell, I just want to advise people that you can start with simpler models or classifier and then gradually proceeds towards complex models. Simpler models provide a better understanding of academic concepts and are easier to implement.

Note: In the real-world, the data is not as simple as the problem explained in this article. However, I would encourage people who want to start reading about machine learning or supervised learning, they can give a quick read just to understand how the process of classification works.