Detecting Toxic Comment
By: Chengxi Li
Tags: NLP, Deep Learning, Cybercrime, Machine Learning, Big Data

Dive into NLP: using Multi-label Classification techniques

Authored by Chengx Li, Chengxi (Michael) Yang, Haopeng Wang, Hao Zheng

MOTIVATION

In modern society, as we get more and more data, there is a huge need for organizing and classifying data. And the classification problems often involve predicting multiple labels at the same time, which is so-called Multi-Label Classification.

We share the interests on this topic with many data scientists as well as scholars. After some in-depth survey on the potential applications of Multi-Label classification, a project done by a group of young students at Simon Fraser University has presented very promising results in toxic comment detection using multi-label classification techniques.

We had the honour to have one of the members of this distinct group to discuss their project. Here in this blog, we are going to dive right into how multi-label classification is implemented for toxic comment detection.

They made improvements based on a simple baseline model and compared this model to other great solutions to properly evaluate their work.

Firstly, what is Multi-Label Classification? What is the difference between Multi-Label Classification and Multi-Class Classification?

Let’s recall the Binary Classification Problem. Suppose we have two class, and we want to decide whether an item belongs to Class 1 or Class 0. If we build a fully connected Neural Network, and the last two layer of our network might look like this:

Notice that there is only one node in the output layer, and the activate function is Sigmoid, which will return a value in the range of (0, 1). This value can be viewed as the probability that this item belongs to class 1. Then we can set a threshold value, for this example 0.5, if the output of the activation function is greater than the threshold, we classify this item belongs to Class 1, otherwise this item will be assigned as Class 0.

For the Multi-Class Classification problem, the most famous one is the ImageNet problem: classify image to one of the 1000 classes. The last two layers network structure might look like this.

Suppose we have three classes that we like to classify an item to Class 1, Class 2, and Class 3, and one item can only belong to one of those three classes. In the output layer, the number of notes is equal to the number of classes. No matter what activation function is used for the last layer, we just apply the Softmax function to the last layer. Softmax can be viewed as a normalization over the output layer, so that sum of all the output notes add up to 1, in our case, y1 + y2 + y3 =1. Each value represents the probability of an item belongs to a certain class. And we classify the item to the class which has the highest probability, it doesn’t matter if this probability is greater than 0.5 or not.

For Multi-Label Classification tasks, it’s very similar to Multi-Class Classification. The difference is that in Multi-Label tasks, one item can be classified to have multiple labels, while in Multi-Class task we already mentioned, one item can only be classified to one of the classes.

Suppose we have three classes (or labels to be more precise). Here is the last two layers of the Network:

There are three notes in the output layer with Sigmoid activation. In this case, we compute a probability for each class, and set a threshold. If a probability for a class is greater than the threshold, we label that class. In our example, both the top and bottom node in the last layer have probabilities greater than 0.5, so we label this item as both Class1 and Class 3.

In this task for toxic comment classification, it’s a multi-label classification problem with six labels: toxic, severe_toxic, obscene, threat, insult, identity_hate. A sample of training data might look like this:

We need to build a model which inputs a comment, and outputs all the toxic labels it belongs to.

APPROACH

3 different models were implemented by the author to compare performances and results.

Baseline: Logistic Regression

Recurrent Neural Network with ELMo Embeddings

Recurrent Neural Network with Convolution Filter

APPROACH 1: LOGISTIC REGRESSION

The first approach used TF-IDF to vectorize word from the dataset and feed TF-IDF matrix into logistic regression.

TF-IDF is short for term frequency-inverse document frequency which is a method to display how important a word is to a document in a corpus. The tf–idf value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word, which helps to adjust for the fact that some words appear more frequently in general.

APPROACH 2: RECURRENT NEURAL NETWORK WITH ELMo EMBEDDINGS

ELMo embeddings (Embeddings from Language Models) is tailored to learn the semantics and context of the words. ELMo has an internal bi-LSTM which will compute the probability of forwarding and backward language model.

Basic RNN is not very effective, so the Long-short-term-memory (LSTM) was introduced, the repeating module does more operations which enable the LSTM network to remember long-term dependencies.

Bidirectional RNN with LSTM was used to capture information in both forward and backward sequence of words in sentences. The output of the LSTM layer is the concatenation of bidirectional LSTM cells.

Dropout was used during model training. The model will produce an output of 6 probabilities specifying whether the sentence is a toxic comment. Using a fully-connected layer, multiple neurons will be reduced to 6 outputs. A sigmoid activation is used in the layer to form a probability ∈[0,1].

APPROACH 3: RECURRENT NEURAL NETWORK WITH CONVOLUTIONAL FILTER

This neural network setup is similar to the one above. The following is the difference compared to the previous model:

ELMo embedding is replaced by a glove embedding layer constructed from the vocabulary of a dataset.

Convolution filter is applied to the output of the LSTM layer.

RESULTS

As the bar chart shown in Figure 1, we can see that the accuracy score of Logistic Regression, ELMo+Bi-RNN and Bi-RNN+convolution models are 0.9756, 0.9791 and 0.9836 respectively. The baseline logistic regression has the lowest accuracy score among the three models. Both of the neural network models have effective improvement. As shown in Table 1, the ELMo+Bi-RNN model achieves an improvement of 0.35% accuracy score on the test set compared to the baseline. Besides, the Bi-RNN+Convolution model also has an improvement of 0.82% accuracy score. However, based on the experiments compared with the baseline, Bidirectional Recurrent Neural Network with convolutional filter works better than the ELMo word embedding fed into Bi-LSTM layer.

Takeaway from the analysis

ELMo embeddings do not provide much improvement over the baseline.

ELMo is trained on Google news data. The vocabulary of news corpus does not contain impolite words. However, there are many impolite words in toxic comment dataset. Because of the vocabulary does not overlap, ELMo embeddings do not provide informative data for the neural network.

Both neural network models show improvement over the logistic regression baseline.

This is because the neural network produces a high dimensional classification boundary for the dataset. Logistic regression produces a classification function with fewer dimensions compared to neural networks.

Thank you so much for your time, hopefully, you’ve enjoyed reading our blog post. With your help, we can work together to help raise awareness on cyber-bullying, by simply clicking the clap button, you could influence or even inspire more and more people to help to build a safer, healthier online environment. Letting more data scientists know, multi-label classification is armed and will be ready for trolls and their toxic comments.

References

Kaggle Inc (2018) Toxic Comment Classification Challenge, Available at https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

Sujay S Kumar (2nd Oct 2018) ELMo Embeddings in Keras, Available at http://sujayskumar.com/2018/10/02/elmo-embeddings-in-keras/

Edward Ma (30th Oct 2018) ELMo helps to further improve your sentence embeddings, Available at https://towardsdatascience.com/elmo-helps-to-further-improve-your-word-embeddings-c6ed2c9df95f

Michailidis, Marios. (2017). Investigating machine learning methods in recommender systems (Thesis). University College London, Available at https://mlwave.com/kaggle-ensembling-guide/

Amar Budhiraja (15th Dec 2016) Dropout in (Deep) Machine learning, Available at https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5

Rohith Gandhi (Jun 26, 2018) Introduction to Sequence Models — RNN, Bidirectional RNN, LSTM, GRU, Available at https://towardsdatascience.com/introduction-to-sequence-models-rnn-bidirectional-rnn-lstm-gru-73927ec9df15