Zero-Shot Text Classification & Evaluation | by Abdullatif Köksal | Towards Data Science

Recently, zero-shot text classification attracted a huge interest due to its simplicity. In this post, we will see how to use zero-shot text classification with any labels and explain the background model. Then, we will evaluate its performance by human annotated datasets in sentiment analysis, news categorization, and emotion classification.
Zero-Shot Text Classification
In zero-shot text classification, the model can classify any text between given labels without any prior data.
With zero-shot text classification, it is possible to perform:
Sentiment analysis
News categorization
Emotion analysis
Actually, the latest implementations of zero-shot text classification born out of a very simple but brilliant idea. There is a field called Natural Language Inference (NLI) in NLP. This field investigates whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) for a given premise.
Now, let’s assume our text is “I love this movie.” and we want to predict the sentiment of the text between candidate labels of positive and negative. We give these two hypothesis-premise pairs to already trained NLI model and check the results.
Premise: I love this movie.Hypothesis-1: This example is positive.
Premise: I love this movie.Hypothesis-2: This example is negative.
Basically, it creates hypothesis template of “this example is ...” for each class to predict the class of the premise. If the inference is entailment, it means that the premise belongs to that class. In this case, it is positive.
Thanks to HuggingFace, it can be easily used through the pipeline module.
#!pip install transformers datasetsfrom transformers import pipelineclassifier = pipeline("zero-shot-classification", device=0) #GPUcandidate_labels = ["positive", "negative"]text = "I don't know why I like this movie so well, but I never get tired of watching it."classifier(text, candidate_labels)> {'labels': ['positive', 'negative'],> 'scores': [0.8987422585487366, 0.10125774145126343],> 'sequence': "I don't know why I like this movie so well, but Inever get tired of watching it."}
In the first example, we initialize the classifier from transformers pipeline and then give an example from IMDB dataset. You can see that the classifier produces scores for each label. In the first example, it predicts the sentiment of the text as positive, correctly.
candidate_labels = ["world", "sports", "business", "sci/tech"]text = "Quality Gets Swept Away Quality Distribution is hammered after reporting a large loss for the second quarter."classifier(text, candidate_labels)> {'labels': ['business', 'world', 'sci/tech', 'sports'],> 'scores': [0.8066419363021851,   0.16538377106189728,   0.018306914716959,   0.009667363949120045],> 'sequence': 'Quality Gets Swept Away Quality Distribution is hammered after reporting a large loss for the second quarter.'}
Our second example is for news categorization from AG News dataset. It correctly predicts the news in business category.
candidate_labels = ["anger", "fear", "joy", "love", "sadness", "surprise"]text = "i didnt feel humiliated"classifier(text, candidate_labels)> {'labels': ['surprise', 'joy', 'love', 'sadness', 'fear', 'anger'],> 'scores': [0.66361004114151,   0.1976112276315689,   0.04634414240717888,   0.03801531344652176,   0.03516925126314163,   0.01925000175833702],> 'sequence': 'i didnt feel humiliated'}
In our last example, we investigated an example from Emotion dataset. Zero-shot classification model predicts emotion of the sentence “i didnt feel humiliated” as surprise, however gold label is sadness.
Zero-shot classification looks promising in these examples. However, its performance should be evaluated with correct measurements by using already labeled examples.
By using the latest dataset library of HuggingFace, we can easily evaluate its performance on several datasets.
IMDB dataset: sentiment analysisClasses: positive, negative
AG-News dataset: news categorizationClasses: world, sports, business, sci/tech
Emotion dataset: emotion classificationClasses: anger, fear, joy, love, sadness, surprise
Let’s compare our zero-shot text classification model with the state-of-the-art models and random pick in micro-average F1.
For more details about initializing zero-shot classification pipeline and evaluation code, check out this well-prepared Colab Notebook:
colab.research.google.com
We can see that zero-shot text classification performs significant results in sentiment analysis and news categorization. The performance in the emotion classification with 6 class is rather poor. I believe that it might be due to the similarity between classes. It is a very hard task to make a distinction between joy, love, and surprise classes without any prior data.
Performance of zero-shot classification is lower than supervised models for each task as expected. Even so, it’s worth trying if you don’t have any data for a specific classification problem!
Originally published at https://akoksal.com.
Follow me on Twitter: https://twitter.com/alkksl