Multi-Label Text Classification. Assign labels to movies based on... | by Zuzanna Deutschman | Towards Data Science

To understand various classification types, let’s consider the picture above.
┌──────────────┬────────────────────┬──────────────────────┐│ Is it a hen? │    Which label?    │    Which labels?     │├──────────────┼────────────────────┼──────────────────────┤│ Yes/No       │ harmony/hen/nature │ harmony, hen, nature ││ BINARY       │ MULTICLASS         │ MULTI-LABEL          │└──────────────┴────────────────────┴──────────────────────┘
In the case of binary classification we just ask a yes/no type of question. If there are multiple possible answers and only one to be chosen, then it’s multiclass classification. In our example we can’t really select only one label, I would say that all of them match the photo. The goal of multi-label classification is to assign a set of relevant labels for a single instance. However, most of widely known algorithms are designed for a single label classification problems. In this article four approaches for multi-label classification available in scikit-multilearn library are described and sample analysis is introduced.
Problem transformation methods divide the multi-label problem into one or more conventional single-label problems.Problem adaptation methods generalize single-label algorithms to cope with multi-labeled data directly.Ensemble methods incorporate the merits of two previous approaches
Problem transformation methods divide the multi-label problem into one or more conventional single-label problems.
Problem adaptation methods generalize single-label algorithms to cope with multi-labeled data directly.
Ensemble methods incorporate the merits of two previous approaches
In the next sections I will describe methods presented in below diagram.
In the case of Binary Relevance, an ensemble of single-label binary classifiers is trained independently on the original dataset to predict a membership to each class, as shown on the fig. 2.
┌─────────────────────┬─────────────────────────────────────────┐│     Advantages      │              Disadvantages              │├─────────────────────┼─────────────────────────────────────────┤│ - linear complexity │ - ignoring correlation between labels   ││ - simplicity        │   by treating each label independently  │└─────────────────────┴─────────────────────────────────────────┘
This approach maps each combination of labels into a single label and trains a single label classifier (fig. 3).
┌────────────────┬───────────────────────────────────┐│   Advantages   │           Disadvantages           │├────────────────┼───────────────────────────────────┤│ - taking label │ - high computational complexity   ││   correlations │ - can lead to imbalanced dataset  ││   into account │  with multiple classes associated ││                │  with few examples                │└────────────────┴───────────────────────────────────┘
For each instance in the test set, its K nearest neighbours in the train set are identified. For each class y in Y the numbers of neighbouring instances belonging to y are used to compute the posterior probabilities that the test instance belongs or does not belong to y. Based on which of these probabilities are greater , we decide to assign or not the class y to a test instance.
For better understanding, assume that samples below are neighbours found out in the training set for test instance X = 1.
Now we have to calculate prior and posteriori probabilities for each class for test instance X = 1
Maximal values were obtained for f1 and f2. Hence, we assign test instance to those two classes.
┌─────────────────────────┬───────────────────────────────────────┐│       Advantages        │             Disadvantages             │├─────────────────────────┼───────────────────────────────────────┤│ - better accuracy than  │ - with the binary relevance strategy, ││   the previous methods  │   the ratio estimation                ││ - correlations between  │   may not be accurate                 ││   labels are considered │ - data distributions for some labels  ││                         │   are imbalanced                      │└─────────────────────────┴───────────────────────────────────────┘
Classifier chain model learns classifiers as in binary relevance method. However, all classifiers are linked in a chain.
Firstly, all the features (X1, X2, ..., Xm) are used to predict y1.Then, all the features (X1, X2, ..., Xm, y1) are used to predict y2Finally, (X1, X2, ..., Xm, y1, y2) are applied to predict y3
Firstly, all the features (X1, X2, ..., Xm) are used to predict y1.
Then, all the features (X1, X2, ..., Xm, y1) are used to predict y2
Finally, (X1, X2, ..., Xm, y1, y2) are applied to predict y3
The order in which labels are predicted has large impact on the results.
┌────────────────────────────┬──────────────────────────┐│         Advantages         │      Disadvantages       │├────────────────────────────┼──────────────────────────┤│ - label correlation taken  │ - accuracy heavily       ││   into consideration       │   depends on the order   ││ - acceptable computational │ - for n labels there     ││   complexity               │   are n! possible orders │└────────────────────────────┴──────────────────────────┘
We will use the CMU Movie Summary Corpus open dataset for our project. You can download the dataset directly from this link.
This dataset contains multiple files, but we’ll need only two of them:
movie.metadata.tsv: Metadata for 81,741 movies, extracted from the November 4, 2012 dump of Freebase. The movie genre tags are available in this file
plot_summaries.txt: Plot summaries of 42,306 movies extracted from the November 2, 2012 dump of English-language Wikipedia. Each line contains the Wikipedia movie ID (which indexes into movie.metadata.tsv) followed by the plot summary
We will create a table containing multiple genres assigned to movies with description. Based on that description we will try to assign proper genres to each movie.
Let’s start with importing necessary libraries. For described above methods we will use skmultilearn module.
Now it’s time for cleaning the text from special signs, removing stopwords, stamming and putting genre variable into separate binary columns.
As the data is ready, we can split it into training and test sets and convert to a matrix of TF-IDF features. We will take top 10000 of them ordered by term frequency across the corpus (due to computational complexity). Please note that N-grams of maximal length 3 are considered here as potential features. As n_gram_range and max_features are essential parameters in terms of accuracy, I recommend you to play with them by your own :).
Let’s apply models described above and compare the results. For selecting the best model we will measure:
Accuracy — accuracy is the most common metric. However, it can be largely contributed by a large number of True Negatives which we have in this case and in the most of businesss cases.F1 score — F1 Score might be a better measure to use if we need to seek a balance between Precision and Recall and there is an uneven class distribution (large number of Actual Negatives) as in our exampleHamming loss — the fraction of labels that are incorrectly predicted. Brings additional information in case of unbalanced datasets.
Accuracy — accuracy is the most common metric. However, it can be largely contributed by a large number of True Negatives which we have in this case and in the most of businesss cases.
F1 score — F1 Score might be a better measure to use if we need to seek a balance between Precision and Recall and there is an uneven class distribution (large number of Actual Negatives) as in our example
Hamming loss — the fraction of labels that are incorrectly predicted. Brings additional information in case of unbalanced datasets.
Accuracy = 0.004, F1 score = 0.222, Hamming loss = 0.033
Accuracy = 0.090, F1 score = 0.261, Hamming loss = 0.009
Accuracy = 0.052, F1 score = 0.235, Hamming loss = 0.01
As you see, accuracy is extremely low in all the cases since the dataset is strongly unbalanced. The best results were obtained for Label Powerset model. However, we have still Classifier Chain to be applied. I will predict here each class probability assignment and then play with threshold parameter a bit (by default it equals 0.5). It says how to distinguish between 0/1 label assignment. You will see that its value is crucial for the model selection.
The plot shows that the most optimal threshold is around 0.2. This is the point where F1 score is near the maximum point 0.43, accuracy increases and Hamming loss is low.
To sum up, popular methods for multilabel classification were described and compared on real-life example. The best results were obtained for Classifier Chain method, by modyfing minimal threshold, based on which we assign 1 in binary classification problems. In future work, the order of labels could be shuffled and tested, which is time consuming process.
In the next article I will use deep learning methods for the same problems. This is the fastest-growing and the most significantt subfield of Natural Language Processing as allows to obtain the best results.
Thanks for reading!