Multi-label Text Classification with Scikit-learn and Tensorflow
By: Rodolfo Saldanha
Tags: Deep Learning, Artificial Intelligence, Neural Networks, Towards Data Science, Machine Learning

Multi-label models

There exists multiple ways how to transform a multi-label classification, but I chose two approaches:

Binary classification transformation — This strategy divides the problem into several independent binary classification tasks. It resembles the one-vs-rest method, but each classifier deals with a single label, which means the algorithm assumes they are mutually exclusive.

— This strategy divides the problem into several independent binary classification tasks. It resembles the one-vs-rest method, but each classifier deals with a single label, which means the algorithm assumes they are mutually exclusive. Multi-class classification transformation — The labels are combined into one big binary classifier called powerset. For instance, having the targets A, B, and C, with 0 or 1 as outputs, we have A B C -> [0 1 0], while the binary classification transformation treats it as A B C -> [0] [1] [0].

The evaluation metric to measure the performance of the models is the AUC measure, which stands for “Area Under the ROC Curve.” A ROC curve is a graph showing the performance of a classification model at all classification thresholds.

Figure 8 — AUC (Area Under the Curve)

This curve plots two parameters:

True Positive Rate

TPR = TP/(TP+FN)

False Positive Rate

FPR = FP/(FP +TN)

TP = True Positive; FP = False Positive; FP = False Positive; FN = False Negative

A model’s performance is assessed after running it with 5 different seeds to try to mitigate any bias.

Scikit-learn

First of all, it is necessary to vectorize the words before training the model, and here we are going to use the tf-idf vectorizer.

Tf-idf stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.

from sklearn.feature_extraction.text import TfidfVectorizer vectorizer = TfidfVectorizer(strip_accents='unicode', analyzer='word', ngram_range=(1,3), norm='l2')

vectorizer.fit(X_train) X_train = vectorizer.transform(X_train)

X_test = vectorizer.transform(X_test)

1. OneVsRestClassifier

The estimator used was RandomForestClassifier, and since the labels are analyzed separately, the result is the average of the AUC score of the categories.

from sklearn.multiclass import OneVsRestClassifier

from sklearn.ensemble import RandomForestClassifier

Figure 9 — AUC score per category

AUC score: 0.517097

2. BinaryRelevanceClassifier

This method is very similar to the OneVsAll, but not the same. If there are x labels, the binary relevance method creates x new datasets, one for each label, and trains single-label classifiers on each new data set. One classifier may answer yes/no, thus the “binary relevance.” This is a simple approach but does not work well when there are dependencies between the labels.

The estimator used is GaussianNB (Gaussian Naive Bayes).

from skmultilearn.problem_transform import BinaryRelevance

from sklearn.naive_bayes import GaussianNB classifier = BinaryRelevance(GaussianNB())

classifier.fit(X_train, y_train)

predictions = classifier.predict(X_test)



print('AUC score: {}'.format(roc_auc_score(y_test,predictions.toarray())))

AUC score: 0.544241

3. ClassifierChain

This approach combines the computational efficiency of the Binary Relevance method while still being able to take the label dependencies into account for classification. On the other hand, that makes this method more expensive computationally speaking.

The estimator used is LogisticRegression.

from skmultilearn.problem_transform import ClassifierChain

from sklearn.linear_model import LogisticRegression classifier = ClassifierChain(LogisticRegression())

classifier.fit(X_train, y_train)

predictions = classifier.predict(X_test) print('AUC score: {}'.format(roc_auc_score(y_test,predictions.toarray())))

AUC score: 0.519823

4. MultiOutputClassifier

This strategy consists of fitting one classifier per target(A B C -> [0 1 0]). This is a simple strategy for extending classifiers that do not natively support multi-target classification.

The estimator used is KNeighborsClassifier.

from sklearn.multioutput import MultiOutputClassifier

from sklearn.neighbors import KNeighborsClassifier clf = MultiOutputClassifier(KNeighborsClassifier()).fit(X_train, y_train)

predictions = clf.predict(X_test) print('AUC score: {}'.format(roc_auc_score(y_test,predictions)))

AUC score: 0.564452

Tensorflow

Text classification has benefited from the deep learning architectures’ trend due to their potential to reach high accuracy. There are different libraries available for deep learning, but we chose to use here Tensorflow because, alongside with PyTorch, they have become the most popular libraries for the topic.

On the other hand, word embeddings are low dimensional as they represent tokens as dense floating-point vectors and thus pack more information into fewer dimensions. This technique normally gives a performance boost in NLP tasks, for instance, syntactic parsing and sentiment analysis. It is possible to either train the WordEmbedding layer or use a pre-trained one through transfer learning, such as word2vec and GloVe.

For the following models, the vectorization used was texts_to_sequences, which transforms the words in numbers, and the pad_sequences ensures all the vectors have the same length.

from keras.preprocessing.text import Tokenizer

from keras.preprocessing.sequence import pad_sequences tokenizer = Tokenizer(num_words=5000, lower=True)

tokenizer.fit_on_texts(data['description'])

sequences = tokenizer.texts_to_sequences(data['description'])

x = pad_sequences(sequences, maxlen=200)

Class weights were calculated to address the imbalance problem in the categories.

most_common_cat['class_weight'] = len(most_common_cat) / most_common_cat['count']

class_weight = {}

for index, label in enumerate(categories):

class_weight[index] = most_common_cat[most_common_cat['cat'] == categories]['class_weight'].values[0]



most_common_cat.head()

Figure 10 — Class weights

1.DNN with WordEmbedding

We started with a simple model which only consists of an embedding layer, a dropout layer to reduce the size and prevent overfitting, a max-pooling layer, and one dense layer with a sigmoid activation to produce probabilities for each of the categories that we want to predict.

from keras.models import Sequential

from keras.layers import Dense, Embedding, GlobalMaxPool1D

from keras.optimizers import Adam

import tensorflow as tf model = Sequential()

model.add(Embedding(max_words, 20, input_length=maxlen))

model.add(GlobalMaxPool1D())

model.add(Dense(num_classes, activation='sigmoid')) model.compile(optimizer=Adam(0.015), loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])

Figure 11 — DNN architecture

AUC score: 0.890245

2. CNN with WordEmbedding

Convolutional Neural Networks recognize local patterns in a sequence by processing multiple words at the same time, and 1D convolutional networks are suitable for text processing tasks. In this case, the convolutional layer uses a window size of 3 and learns word sequences that can later be recognized in any position of a text.

from keras.layers import Dense, Activation, Embedding, Flatten, GlobalMaxPool1D, Dropout, Conv1D filter_length = 300 model = Sequential()

model.add(Embedding(max_words, 20, input_length=maxlen))

model.add(Conv1D(filter_length, 3, padding='valid', activation='relu', strides=1))

model.add(GlobalMaxPool1D())

model.add(Dense(num_classes))

model.add(Activation('sigmoid')) model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])

Figure 12 — CNN architecture

AUC score: 0.886286

3. LSTM with GloVe WordEmbedding

In this model, we will use GloVe word embedding to convert text inputs to their numeric counterparts, which is a different approach because this is a pre-trained layer. The model will have one input layer, one embedding layer, one LSTM layer with 128 neurons, and one output layer with 21 neurons (the number of targets.)

from keras.layers import Flatten, LSTM

from keras.models import Model deep_inputs = Input(shape=(maxlen,))

embedding_layer = Embedding(max_words, 100, weights=[embedding_matrix], trainable=False)(deep_inputs)

LSTM_Layer_1 = LSTM(128)(embedding_layer)

dense_layer_1 = Dense(21, activation='sigmoid')(LSTM_Layer_1)

model = Model(inputs=deep_inputs, outputs=dense_layer_1) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[tf.keras.metrics.AUC()])

Figure 13 — LSTM architecture

AUC score: 0.887574