Introduction
The exponential increase of interactions on the various social media platforms has generated the huge amount of data on social media platforms like Facebook and Twitter, etc. These interactions resulted not only positive effect but also negative effect over billions of people owing to the fact that there are lots of aggressive comments (like hate, anger, and bullying). These cause not only mental and psychological stress but also account deactivation and even suicideBIBREF1. In this paper we concentrate on problems related to aggressiveness.
The fine-grained definition of the aggressiveness/aggression identification is provided by the organizers of TRAC-2018 BIBREF0, BIBREF2. They have classified the aggressiveness into three labels (Overtly aggressive(OAG), Covertly aggressive(CAG), Non-aggressive(NAG)). The detailed description for each of the three labels is described as follows:
Overtly Aggressive(OAG) - This type of aggression shows direct verbal attack pointing to the particular individual or group. For example, "Well said sonu..you have courage to stand against dadagiri of Muslims".
Covertly Aggressive(CAG) - This type of aggression the attack is not direct but hidden, subtle and more indirect while being stated politely most of the times. For example, "Dear India, stop playing with the emotions of your people for votes."
Non-Aggressive(NAG) - Generally these type of text lack any kind of aggression it is basically used to state facts, wishing on occasions and polite and supportive.
The additional discussion on aggressiveness task can be found in Kaggle task , which just divided the task into two classes - i.e., presence or absence of aggression in tweets.
The informal setting/environment of social media often encourage multilingual speakers to switch back and forth between languages when speaking or writing. These all resulted in code-mixing and code-switching. Code-mixing refers to the use of linguistic units from different languages in a single utterance or sentence, whereas code-switching refers to the co-occurrence of speech extracts belonging to two different grammatical systemsBIBREF3. This language interchange makes the grammar more complex and thus it becomes tough to handle it by traditional algorithms. Thus the presence of high percentage of code-mixed content in social media text has increased the complexity of the aggression detection task. For example, the dataset provided by the organizers of TRAC-2018 BIBREF0, BIBREF2 is actually a code-mixed dataset.
The massive increase of the social media data rendered the manual methods of content moderation difficult and costly. Machine Learning and Deep Learning methods to identify such phenomena have attracted more attention to the research community in recent yearsBIBREF4.
Based on the current context, we can divide the problem into three sub-problems: (a) detection of aggression levels, (b) handling code-mixed data and (c) handling styles (due to differences in social media platforms and text entry rules/restrictions).
A lot of the previous approachesBIBREF5 have used an ensemble model for the task. For example, some of them uses ensemble of statistical modelsBIBREF6, BIBREF7, BIBREF8, BIBREF9 some used ensemble of statistical and deep learning modelsBIBREF10, BIBREF11, BIBREF12 some used ensemble of deep learning models BIBREF13. There are approaches which proposed unified architecture based on deep learningBIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18, BIBREF19 while some proposed unified statistical modelBIBREF7. Additionally, there are some approaches uses data augmentation either through translation or labeling external data to make the model generalize across domainsBIBREF14, BIBREF10, BIBREF7.
Most of the above-discussed systems either shows high performance on (a) Twitter dataset or (b) Facebook dataset (given in the TRAC-2018), but not on both English code-mixed datasets. This may be due to the text style or level of complexities of both datasets. So, we concentrated to develop a robust system for English code-mixed texts, and uni-lingual texts, which can also handle different writing styles. Our approach is based on three main ideas:
Deep-Text Learning. The goal is to learn long range associations, dependencies between regions of text, N-grams, key-patterns, topical information, and sequential dependencies.
Exploiting psycho-linguistic features with basic linguistic features as meta-data. The main aim is to minimize the direct dependencies on in-depth grammatical structure of the language (i.e., to support code-mixed data). We have also included emoticons, and punctuation features with it. We use the term "NLP Features" to represent it in the entire paper.
Dual embedding based on FastText and Glove. This dual embedding helps in high vocabulary coverage and to capture the rare and partially incorrect words in the text (specially by FastText BIBREF20).
Our "Deep-text architecture" uses model averaging strategy with three different deep learning architectures. Model averaging belongs to the family of ensemble learning techniques that uses multiple models for the same problem and combines their predictions to produce a more reliable and consistent prediction accuracy BIBREF21. This is the simplest form of weighted average ensemble based predictionBIBREF22 where, each ensemble member contribute equally to predictions. Specifically in our case, three different models have been used. The following contains the intuition behind the selection of these three models:
Deep Pyramid CNN BIBREF23 being deeper helps to learn long range associations between temporal regions of text using two-view embeddings.
Disconnected RNN BIBREF24 is very helpful in encoding the sequential information with temporal key patterns in the text.
Pooled BiLSTM In this architecture the last hidden state of BiLSTM is concatenated with mean and max-pooled representation of the hidden states obtained over all the time steps of Bi-LSTM. The idea of using mean and max pooling layers together is taken from BIBREF25 to avoid the loss of information in longer sequences of texts and max-pooling is taken to capture the topical informationBIBREF26.
NLP Features In each of the individual models, the NLP features are concatenated with last hidden state before the softmax classification layer as meta-data. The main aim is to provide additional information to the deep learning network.
The intuition behind the NLP features are the following:
Emotion Sensor Dataset We have introduced to use of emotion sensor features, as a meta-data information. We have obtained the word sensor dataset from Kaggle. In this dataset each word is statistically classified into 7 distinct classes (Disgust, Surprise, Neutral, Anger, Sad, Happy and Fear) using Naive Bayes, based on sentences collected from twitter and blogs.
Controlled Topical Signals from Empath. Empath can analyse the text across 200 gold standard topics and emotions. Additionally, it uses neural embedding to draw connotation among words across more than 1.8 billion words. We have used only selected categories like violence, hate, anger, aggression, social media and dispute from 200 Empath categories useful for us unlikeBIBREF12 which takes 194 categories.
Emoticons frequently used on social media indicates the sense of sentenceBIBREF17, BIBREF19, BIBREF9.
Normalized frequency of POS tags According to BIBREF12, BIBREF11, BIBREF7, BIBREF15 POS Tags provide the degree of target aggressiveness. LikeBIBREF12, we have used only four tags (a) adjective (JJ, JJR, JJS), (b) adverb (RB, RBR, RBS), (c) verb (VB, VBD, VBG, VBN, VBP, VBZ) and (d) noun (NN, NNS, NNP, NNPS) (See Penn-Treebank POS Tags for abbreviations and the full list). The main reason behind the selection of these four tags is to just identify words related to persons, activities, quality, etc, in the text.
Sentiment polarity obtained from VADER Sentiment Analysis BIBREF27 (positive, negative and neutral) like used in BIBREF15, BIBREF10, BIBREF11, BIBREF7. It helps to demarcate aggressiveness with non-aggressiveness in the text.
The block diagram of the proposed system is shown in Figure FIGREF22. The proposed system does not use any data augmentation techniques like BIBREF14, which is the top performer in TRAC (in English code-mixed Facebook data). This means the performance achieved by our system totally depends on the training dataset provided by TRAC. This also proves the effectiveness of our approach. Our system outperforms all the previous state of the art approaches used for aggression identification on English code-mixed TRAC data, while being trained only from Facebook comments the system outperforms other approaches on the additional Twitter test set. The remaining part of this paper is organized as follows: Section SECREF2 is an overview of related work. Section SECREF3 presents the methodology and algorithmic details. Section SECREF4 discusses the experimental evaluation of the system, and Section SECREF5 concludes this paper.
Related work
There are several works for aggression identification submitted at TRAC 2018 among them some approaches use the ensemble of multiple statistical modelsBIBREF6, BIBREF7, BIBREF8, BIBREF9. Similarly, some of the models likeBIBREF10, BIBREF11, BIBREF12 have used ensemble of statistical and deep learning models. In these models the statistical part of the model uses additional features from text analysis like parts-of-speech tags, punctuation, emotion, emoticon etc. Model like: BIBREF13 has used the ensemble of deep learning models based on majority voting.
Some other models like: BIBREF28, BIBREF12, BIBREF9 have used different models for Facebook and twitter. While approaches like:BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18, BIBREF19 have proposed unified architecture based on deep learning. Systems likeBIBREF14, BIBREF10, BIBREF7 have used data augmentation either through translation or labelling external data to make the model generalize across domains. While BIBREF7 has proposed a unified statistical model.
Among approaches likeBIBREF6 extracted features from TF-IDF of character n-grams whileBIBREF28 uses LSTM with pre-trained embeddings from FastText. BIBREF15 have used the BiLSTM based model and the SVM metaclassifier model for the Facebook and Twitter test sets, respectively. While BIBREF13 tried ensembling of CNN, LSTM, and BILSTM.
Some approaches like:BIBREF12 has used emotions frequency as one of the features, while some others use sentiment emotion as featureBIBREF11. Also,BIBREF17, BIBREF19 have converted emoticons to their description. BIBREF9 have used TF-IDF of emoticons per-class as one of the features. Compared to all these approaches, we have concentrated to capture multiple linguistic/pattern based relations, key-terms and key-patters (with their association in text) through a combination of deep learning architectures with model averaging. We have also used NLP features as additional features with our deep learning architecture, obtained from psycho-linguistic and basic linguistic features.
Methodology
In this section, we describe our system architecture for aggressiveness classifier. In section SECREF23 we describe data preprocessing applied on the input text before feeding it to each of the classification models. Section SECREF26 describes the computation of NLP features. In Sections SECREF30, SECREF34 and SECREF45 we have described the architecture of different deep learning models like Deep Pyramid CNN, Disconnected RNN and Pooled BiLSTM respectively. Finally, in Section SECREF49, we describe model averaging based classification model which combines the prediction probabilities from three deep learninig architectures discussed above. (see Figure FIGREF22. for block diagram of system architecture).
Methodology ::: Data Preprocessing
We consider the text to be well formatted before applying the text to the embedding layer. First, we detect non-English text(which are few) and translate all of them to English using Google Translate. Still, there is some code mixed words like "mc", "bc" and other English abbreviations and spelling errors like "nd" in place of "and", "u" in place of "you" causes deep learning model to confuse with sentences of the same meaning. We follow the strategy of preprocessor as inBIBREF17 to normalize the abbreviations and remove spelling errors, URLs and punctuation marks, converting emojis to their description.
https://spacy.io/usage/linguistic-features#pos-tagging
Methodology ::: NLP Features
We have identified a novel combination of features which are highly effective in aggression classification when applied in addition to the features obtained from the deep learning classifier at the classification layer. We have introduced two new features in addition to the previously available features. The first one is the Emotion Sensor Feature which use a statistical model to classify the words into 7 different classes based on the sentences obtained from twitter and blogs which contain total 1,185,540 words. The second one is the collection of selected topical signal from text collected using Empath (see Table 1.).
Different from previous approachesBIBREF8, BIBREF12 where BIBREF12 have used Emotion features in the form of frequency while BIBREF8 have used emotion feature vector obtained from LIWC 2007BIBREF30. UnlikeBIBREF12 we have used only 6 topical signals from EmapthBIBREF29. We have borrowed the idea of using other features like punctuation features and parts-of-speech tags from BIBREF12. The Table 1. lists and describes features, tools used to obtain them and the number of features resulted from each type.
Methodology ::: Deep Pyramid CNN(DPCNN)
Since it has been proved that CNNs are great feature extractors for text classificationBIBREF31, BIBREF32, BIBREF33, BIBREF34, BIBREF35, BIBREF23 while deeper networks(whether RNNs or CNN's) has been proven for learning long-range association like deeper character level CNN'sBIBREF36, BIBREF37, and complex combination of RNN and CNNBIBREF38, BIBREF39, BIBREF40, BIBREF41, BIBREF42. Deep Pyramid CNN (DPCNN)BIBREF23 has 15 layers of word-level CNN's and contains similar pre-activation as proposed in improved ResnetBIBREF43. DPCNN outperforms the 32-layer character CNNBIBREF37 and Hierarchical attention networksBIBREF42 it has added advantage that due to its pyramid structure it does not require dimension matching in shortcut connections defined as z + h(z) as inBIBREF43 where h(z) represents the skipped layers essentially contains two convolutional layers with pre-activation. It uses enhanced region embedding which consumes pre-trained embeddings (in our case it is FastText+Glove based dual embedding).
Enhanced Region Embedding. The current DPCNNBIBREF23, uses two view type enhanced region embedding. For the text categorization, it defines a region of text as view-1 and its adjacent regions as view-2. Then using unlabeled data, it trains a neural network of one hidden layer with an artificial task of predicting view-2 from view-1. The obtained hidden layer, which is an embedding function that takes view-1 as input, serves as an unsupervised embedding function in the model for text categorization. The detailed architecture has been shown in Figure FIGREF29.
Let each word input $x_j \in R^d$ be the d-dimensional vector for the $j^{th}$ word $w_{j}$ and the sentence $s_i$ contains sequence of $n$ words $\lbrace w_{1},w_{2},w_{3},......,w_{n}\rbrace $ as shown in Figure FIGREF29. In comparision to conventional convolution layer, DPCNN proposes to use pre-activation, thus essentially the convolutional layer of DPCNN is $\textbf {W}\sigma (\textbf {x})+\textbf {b}$, where $\textbf {W}$ and $\textbf {b}$(unique to each layer) are the weights matrix and bias respectively, we use $\sigma $ as PReLUBIBREF44. During implementation we use kernel size of 3(represented by $\textbf {x}$ to denote the small overlapping regions of text.), The number of filters(number of feature maps denoted by the number of rows of $\textbf {W}$) is 128 as depicted in Figure FIGREF29. With the number of filters same in each convolution layer and max-pooling with stride 2 makes the computation time halved, and doubles the net coverage of convolution kernel. Thus the deeper layers cause to learn long-range associations between regions of text. Let's say $h_{dpcnn} \in R^{p_1}$ be the hidden state obtained from DPCNN just before the classification layer and $f_{nlp} \in R^{24}$ be the NLP features computed from the text. Lets $z_1 \in R^{p_1 + 24}$ be another hidden state obtained as
where, $\oplus $ denotes concatenation. The vector $z_1$ obtained, then fed to the fully connected layer with softmax activation. Let $y_{i1}^*$ be the softmax probabilities, specifically for class label $k$ is given as:
where $K$ is the number of classes, $W_{dpcnn}$ and $b_{dpcnn}$ are the weight matrix and bias respectively.
Methodology ::: Disconnected RNN(DRNN)
Given a sequence $s_i = [x_{1}, x_{2}, x_{3},....x_{n}]$ where $x_{j} \in R^d$ represents the d-dimensional word vector for word $w_{j}$ and $n$ is the length of input text applied to a variant of RNN called Long Short-Term Memory (LSTM)BIBREF45 as shown in Figure FIGREF33. It is widely used for sequential modelling with long-term dependencies. For sequence modelling it keeps on updating the memory cell with current input using an adaptive gating mechanism. At time step $t$ the memory $c_t$ and the hidden state $h_t$ are updated as follows:
where $\hat{c}_t$ is the current cell state obtained from current input $x_t$ and previous hidden state $h_{t-1}$, $i_t$, $f_t$ and $o_t$ are the activation corresponding to input gate, forget gate and output gate respectively, $\sigma $ denotes the logistic sigmoid function and $\odot $ denotes the element-wise multiplication. Hence the hidden state representation at time step $t$ depends on all the previous input vectors given as
Specifically we have used Bi-directional LSTM BIBREF45 to capture both past and future context. It provides $h_t$ from both directions(forward & backward). The forward LSTM takes the natural order of words from $x_{1}$ to $x_{n}$ to obtain $\overrightarrow{h_t}$, while backward-LSTM $x_{n}$ to $x_{1}$ to obtain $\overleftarrow{h_t}$. then $h_t$ is calculated as
where $\oplus $ is the concatenation and $L$ is the size for one-directional LSTM. Therefore we denote the hidden state in equation DISPLAY_FORM37 with BiLSTM as
To avoid handling of long sequence and to capture local information for each word we define the window size $k$ for each word such that the BiLSTM only sees the the previous $k-1$ words with the current word, where $k$ is a hyperparameterBIBREF24. We use padding <PAD> to make the slices of fixed size k(as shown in Figure FIGREF33). It provides each hidden state $h_t$ with sequence of $k$ previous words. Since the phrase of $k$ words can lie anywhere in the text it helps to model the position invariant phrase representation due to which the it identifies key phrases important for identifying particular category. In this case, the equation of $h_t$ is given as
The output hidden vectors, $H = [h_1, h_2, h_3, ...... h_n] \in R^{n \times 2L}$ are converted to fixed-length vector $h_{drnn} \in R^{2L}$ with max pooling over time:
Let's say $f_{nlp} \in R^{24}$ be the NLP features computed from the text. Let's $z_2 \in R^{2L + 24}$ be another hidden state obtained as
where $\oplus $ denotes concatenation. The vector $z_2$ obtained, then fed to the fully connected layer with softmax activation. Let $y_{i2}^*$ be the softmax probabilities, specifically for class label $k$ is given as:
where $K$ is the number of classes, $W_{drnn}$ is the weight matrix, and $b_{drnn}$ is the bias.
Methodology ::: Pooled BiLSTM
The architecture has been shown in Figure FIGREF44. Given a sequence $s_i = [x_{1}, x_{2}, x_{3}, ..... x_{j}]$, where $x_j \in R^d$ is the d-dimensional word vector for word $w_j$, the hidden state obtained after BiLSTM is given as
To avoid the loss of information because of modelling the entire sequence, we have concatenated the max-pooled($c_{max}$) and mean-pooled($c_{mean}$) representation of hidden states calculated over all time steps BIBREF25. We have also concatenated the nlp features, $f_{nlp} \in R^{24}$ the final feature vector $z_{3}$ is given as
where $\oplus $ denotes concatenation. The final feature $z_3$ vector is fed to the fully connected layer with softmax activation. Let $y_{i3}^*$ be the softmax probablities, specifically for class label $k$ given as:
where $K$ is the number of classes and $W_{bilstm}$ and $b_{bilstm}$ are the weight matrix and bias respectively.
Methodology ::: Classification Model
According to deep learning literature BIBREF46, BIBREF47, BIBREF48, unweighted averaging might be a reasonable ensemble for similar base learners of comparable performance. Now, similar to the information discussed in BIBREF21, we can compute the model averaging (unweighted) by combining the softmax probabilities of three different classification models obtained from equations DISPLAY_FORM32, DISPLAY_FORM43, DISPLAY_FORM48. The averaged class probabilities are computed as:
where K is the number of classes, and $\hat{y_i}$ is the predicted label for sentence $s_i$.
Experiment and Evaluation ::: Dataset Description
We have used two datasets in our experimental evaluations: (1) TRAC 2018 Dataset and (2) Kaggle Dataset.
TRAC 2018 Dataset: We have used the English code-mixed dataset provided by TRAC 2018. This dataset contains three labels, (a) Non-Aggressive(NAG), (b) Overtly-Aggressive (OAG) and (c) Covertly-Aggressive(CAG). The distribution of training, validation and test sets are described in Table TABREF56.
Kaggle Dataset: This dataset contains 20001 tweets which are manually labeled. The labels are divided into two categories (indicating presence or absence of aggression in tweets) AGG(Aggressive) or NAG(Non-Aggressive). We have used the same test split available in the baseline code. The distribution for each of the training and test is given in Table TABREF56.
Experiment and Evaluation ::: Experimental Setup
We have used Glove EmbeddingsBIBREF49 concatenated with FastText EmbeddingsBIBREF20 in all the three classification models presented in this paper. Specifically, we used Glove pre-trained vectors obtained from Twitter corpus containing 27 billion tokens and 1.2 million vocabulary entries where each word is represented using 100-dimensional vector. In the case of FastText the word is represented using 300-dimensional vector. Also, we have applied spatial dropoutBIBREF50 of 0.3 at embedding layer for DPCNN(in section SECREF30) and Pooled BiLSTM(in section SECREF45). For DPCNN model(in SECREF30) we have learnt 128-dimensional vector representation for unsupervised embeddings implicitly for task specific representation as in BIBREF23. Additionally, for DPCNN all the convolutional layers used 128 filters, kernel size of 3 and max-pooling stride 2. Additionally, in the case of DPCNN we have used kernel and bias regularizer of value 0.00001 for all convolutional kernels. The pre-activation function used in DPCNN is Parametric ReLU (PReLU) proposed in BIBREF44 while the activation at each of the convolutional kernel is linear. For, DRNN(in section SECREF34) we have used the window size of 8 and rest of the parameters related to LSTM units are same as given inBIBREF24. For, Pooled BiLSTM(in section SECREF45) we have used LSTM hidden units size as 256. The maximum sequence length is 200 in all three models. In each of the classification model the classification layer contains the fully connected layer with softmax activation with output size of 3 equal to number of classes in case of TRAC 2018 dataset and its 2 in case of Kaggle dataset. Training has been done using ADAM optimizerBIBREF51 for DPCNN and RMSPROPBIBREF52 for DRNN and Pooled Bi-LSTM models. All the models are trained end-to-end using softmax cross entropy lossBIBREF53 for TRAC 2018 dataset and binary cross entropy lossBIBREF53 for Kaggle dataset.
To train our model for TRAC 2018 dataset, we merged the training and validation dataset and then used 10% split from shuffled dataset to save the best model, for all classifiers. We have used only 20 NLP features (except TF-IDF Emoticon feature and Punctuation feature as given in Table TABREF25) for Kaggle dataset (as these are not present in the Kaggle dataset).
Experiment and Evaluation ::: Evaluation Strategy
To compare our experimental results we have used top-5 systems from the published results of TRAC-2018BIBREF5. To compare our results on Kaggle dataset, we have used the last & the best published result on Kaggle website as a baseline. We have conducted the separate experiments, to properly investigate the performance of (a) each of the classifiers (used in our model averaging based system), (b) impact of the NLP features on each of these classifiers and finally, (c) the performance of our proposed system. In Table TABREF57, TABREF57 and TABREF57, models, named as DPCNN(ref SECREF30), DRNN (ref SECREF34) and Pooled BiLSTM(ref SECREF45) are corresponding models without NLP features. Similarly, DPCNN+NLP Features, DRNN + NLP Features and Pooled BiLSTM + NLP Features are corresponding models with NLP features. The Model Averaging (A+B+C) is the ensemble of three models (i.e., model averaging of DPCNN, DRNN and Pooled BiLSTM) without NLP features. Finally, Our Proposed Method, which represents the model averaging of three models with NLP features.
Experiment and Evaluation ::: Results and Discussion
In this paper, we have evaluated our model using weighted macro-averaged F-score. The measure is defined as in (See BIBREF5, BIBREF2). It weights the F-score computed per class based on the class composition in the test set and then takes the average of these per-class F-score gives the final F-score. Table TABREF57, TABREF57 and TABREF57. presents the comparative experimental results for the proposed method in this paper with respect to the state-of-the-art. The top 5 modelsBIBREF5 given in Table TABREF57 and TABREF57. are the best performing models for Facebook and Twitter test dataset respectively on TRAC 2018. We have followed all the experimental guidelines as discussed in TRAC contest guideline paperBIBREF2, BIBREF5. From the results given in Table TABREF57, TABREF57 and TABREF57 it is clear that our proposed model shows the best performance among all of the approaches. These results also state that all the deep learning architectures with NLP features, perform better than individual corresponding deep learning architectures. This means NLP features, adds some value to the architectures, even if it is not very high.
Conclusion and Future Work
In this paper, we have briefly described the approach we have taken to solve the aggressive identification on online social media texts which is very challenging since the dataset is noisy and code-mixed. We presented an ensemble of deep learning models which outperform previous approaches by sufficient margin while having the ability to generalize across domains.
In future, we will explore other methods to increase the understanding of deep learning models on group targeted text, although the categories are well defined we will look after if we further fine-tune the categories with more data. In the future, we are planning to pay attention on a generalized language model for code-mixed texts which can also handle Hindi-code-mixed and other multi-lingual code-mixed datasets (i.e., trying to reduce the dependencies on language-specific code-mixed resources).

Question: What is English mixed with in the TRAC dataset?
Answer: Hindi

Question: Which psycholinguistic and basic linguistic features are used?
Answer: Emotion Sensor Feature, Part of Speech, Punctuation, Sentiment Analysis, Empath, TF-IDF Emoticon features

Question: How have the differences in communication styles between Twitter and Facebook increase the complexity of the problem?
Answer: Systems do not perform well both in Facebook and Twitter texts

Question: What are the key differences in communication styles between Twitter and Facebook?
Answer: Unanswerable

Question: What data/studies do the authors provide to support the assertion that the majority of aggressive conversations contain code-mixed languages?
Answer: None