One-vs-Rest strategy for Multi-Class Classification

17 Jul, 2020
Prerequisite: Getting Started with Classification/
Classification is perhaps the most common Machine Learning task. Before we jump into what One-vs-Rest (OVR) classifiers are and how they work, you may follow the link below and get a brief overview of what classification is and how it is useful.
In general, there are two types of classification algorithms:
Binary classification algorithms.Multi-class classification algorithms.
Binary classification algorithms.
Multi-class classification algorithms.
Binary classification is when we have to classify objects into two groups. Generally, these two groups consist of ‘True’ and ‘False’. For example, given a certain set of health attributes, a binary classification task may be to determine whether a person has diabetes or not.
On the other hand, in multi-class classification, there are more than two classes. For example, given a set of attributes of fruit, like it’s shape and colour, a multi-class classification task would be to determine the type of fruit.
So, now that you have an idea of how binary and multi-class classification work, let us get on to how the one-vs-rest heuristic method is used.
One-vs-Rest (OVR) Method:Many popular classification algorithms were designed natively for binary classification problems. These algorithms include :
Logistic Regression
Support Vector Machines (SVM)
Perceptron Models
and many more.
So, these popular classification algorithms cannot directly be used for multi-class classification problems. Some heuristic methods are available that can split up multi-class classification problems into many different binary classification problems. To understand how this works, let us consider an example : Say, a classification problem is to classify various fruits into three types of fruits: banana, orange or apple. Now, this is clearly a multi-class classification problem. If you want to use a binary classification algorithm like, say SVM. The way One-vs-Rest method will deal with this is illustrated below : 
Since there are three classes in the classification problem, the One-vs-Rest method will break down this problem into three binary classification problems:
Problem 1 : Banana vs [Orange, Apple]
Problem 2 : Orange vs [Banana, Apple]
Problem 3 : Apple vs [Banana, Orange]
So instead of solving it as (Banana vs Orange vs Apple), it is solved using three binary classification problems as shown above.
A major downside or disadvantage of this method is that many models have to be created. For a multi-class problem with ‘n’ number of classes, ‘n’ number of models have to be created, which may slow down the entire process. However, it is very useful with datasets having a small number of classes, where we want to use a model like SVM or Logistic Regression.
Implementation of One-vs-Rest method using Python3Python’s scikit-learn library offers a method OneVsRestClassifier(estimator, *, n_jobs=None) to implement this method. For this implementation, we will be using the popular ‘Wine dataset’, to determine the origin of wines using chemical attributes. We can direct this dataset using scikit-learn. To know more about this dataset, you can use the link below : Wine Dataset
We will use a Support Vector Machine, which is a binary classification algorithm and use it with the One-vs-Rest heuristic to perform multi-class classification.
To evaluate our model, we will see the accuracy score of the test set and the classification report of the model.
from sklearn.datasets import load_winefrom sklearn.multiclass import OneVsRestClassifierfrom sklearn.svm import SVCfrom sklearn.metrics import accuracy_score, classification_reportfrom sklearn.model_selection import train_test_splitimport warnings   ''' We are ignoring warnings because of a peculiar fact about thisdataset. The 3rd label, 'Label2' is never predicted and so the python interpreter throws a warning. However, this can safely be ignored because we are not concerned if a certain label is predicted or not '''warnings.filterwarnings('ignore')   # Loading the datasetdataset = load_wine()X = dataset.datay = dataset.target   # Splitting the dataset into training and testing setsX_train, X_test, y_train, y_test = train_test_split(    X, y, test_size = 0.1, random_state = 13)   # Creating the SVM modelmodel = OneVsRestClassifier(SVC())   # Fitting the model with training datamodel.fit(X_train, y_train)   # Making a prediction on the test setprediction = model.predict(X_test)   # Evaluating the modelprint(f"Test Set Accuracy : {accuracy_score(    y_test, prediction) * 100} %\n\n")print(f"Classification Report : \n\n{classification_report(    y_test, prediction)}")
Output:
Test Set Accuracy : 66.66666666666666 %

Classification Report : 

              precision    recall  f1-score   support

           0       0.62      1.00      0.77         5
           1       0.70      0.88      0.78         8

   micro avg       0.67      0.92      0.77        13
   macro avg       0.66      0.94      0.77        13
weighted avg       0.67      0.92      0.77        13

We get a test set accuracy of approximately 66.667%. This is not bad for this dataset. This dataset is notorious for being difficult to classify and the benchmark accuracy is 62.4 +- 0.4 %. So, our result is actually quite good.
Conclusion:Now that you know how to use the One-vs-Rest heuristic method for performing multi-class classification with binary classifiers, you can try using it next time you have to perform some multi-class classification task.
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Introduction to Recurrent Neural Network
ML | Monte Carlo Tree Search (MCTS)
Markov Decision Process
Support Vector Machine Algorithm
DBSCAN Clustering in ML | Density based clustering
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
Python Dictionary
How to get column names in Pandas dataframe