Still using Accuracy as a Classification Metric? | by Pranav Kaushik | Towards Data Science

Accuracy is the most common evaluation metric for classification models because of its simplicity and interpretation. But when you have a multiclass classification problem in hand, say, for example, with 15 different target classes, looking at the standard accuracy of the model might be misleading. This is where “top N” accuracies might be of some use, and in this post, I’ll take you through the basic intuition and python implementation of top N accuracies.
Before we get into top N accuracy, a small refresher on standard accuracy metric:
Accuracy is the fraction of total records that are correctly predicted by the model. Accuracy = (Number of correctly predicted records / Total number of records) or Accuracy = (TP+TN) / (TP+TN+FP+FN)
When you have a lot of different classes, the classification model might not be able to predict the right class exactly. This especially becomes a problem in NLP cases like text classification, where you have a vast number of features, and the data is not adequately clustered across classes. So looking at your standard accuracy metric might be misleading at times. As I said earlier, measuring the top N accuracies might help in overcoming this issue.
Top N accuracy is not any different metric, but it’s just standard accuracy of the true class being equal to any of the N most probable classes predicted by the classification model.
Top 1 accuracy is the accuracy where true class matches with the most probable classes predicted by the model, which is the same as our standard accuracy.
Top 2 accuracy is the accuracy where true class matches with any one of the 2 most probable classes predicted by the model.
Top 3 accuracy is the accuracy where true class matches with any one of the 3 most probable classes predicted by the model.
And in the same way, we can measure the top 4, top 5, top 6, and so on.
Let me show you an example to understand better. Consider having to classify records into respective animals (dog, cat, lion, tiger, and elephant). The table shows the true class and predicted class of 6 records.
From this, we’ll be getting an accuracy of 50%, which is not so satisfying. Now let’s find out the top 2 accuracies for this problem. The table shows the true class and 2 most probable classes for the same set of records.
With this, we get an accuracy of 83%, which is a significant increase compared to the previous one. This is the basic intuition behind finding top N accuracies.
I hope you would have understood the significance of measuring the top N accuracies. Even if the classification model cannot predict the exact class, looking at the top 2 or top 3 accuracies might be of use in many situations, especially if there are a lot of different classes like 15 or 20.
Now I’ll show you some code to find top N accuracies using python. Unfortunately, sci-kit learn doesn’t have any inbuilt function for this. So I’ve defined a function here that you can directly copy and use in your problems.
After importing the required libraries and preprocessing the data, run the provided function.
def top_n_accuracy(X,y,n,classifier):  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)  vectorizer = TfidfVectorizer(min_df=2)  X_train_sparse = vectorizer.fit_transform(X_train)  feature_names = vectorizer.get_feature_names()  test = vectorizer.transform(X_test)  clf = classifier  clf.fit(X_train_sparse,y_train)  predictions = clf.predict(test)  probs = clf.predict_proba(test)  topn = np.argsort(probs, axis = 1)[:,-n:]  y_true = np.array(y_test)  return np.mean(np.array([1 if y_true[k] in topn[k] else 0 for k in range(len(topn))]))
You can call the function by providing the following parameters:
X = feature variablesy = target classn = top N that you want to measure (for example n=2 for top 2)Classifier = classification model that you want to use (example LogisticRegression())
X = feature variables
y = target class
n = top N that you want to measure (for example n=2 for top 2)
Classifier = classification model that you want to use (example LogisticRegression())
I’ve defined this function for text classification, but you can simply remove the lines having vectorizer instance and use it for standard classification.