A Unified System for Aggression Identification in English Code-Mixed and Uni-Lingual Texts	Wide usage of social media platforms has increased the risk of aggression, which results in mental stress and affects the lives of people negatively like psychological agony, fighting behavior, and disrespect to others. Majority of such conversations contains code-mixed languages[28]. Additionally, the way used to express thought or communication style also changes from one social media plat-form to another platform (e.g., communication styles are different in twitter and Facebook). These all have increased the complexity of the problem. To solve these problems, we have introduced a unified and robust multi-modal deep learning architecture which works for English code-mixed dataset and uni-lingual English dataset both.The devised system, uses psycho-linguistic features and very ba-sic linguistic features. Our multi-modal deep learning architecture contains, Deep Pyramid CNN, Pooled BiLSTM, and Disconnected RNN(with Glove and FastText embedding, both). Finally, the system takes the decision based on model averaging. We evaluated our system on English Code-Mixed TRAC 2018 dataset and uni-lingual English dataset obtained from Kaggle.

The massive increase of the social media data rendered the manual methods of content moderation difficult and costly. Machine Learning and Deep Learning methods to identify such phenomena have attracted more attention to the research community in recent years. Based on the current context, we can divide the problem into three sub-problems: (a) detection of aggression levels, (b) handling code-mixed data and (c) handling styles (due to differences in social media platforms and text entry rules/restrictions). A lot of the previous approaches have used an ensemble model for the task. For example, some of them uses ensemble of statistical models, , ,  some used ensemble of statistical and deep learning models, ,  some used ensemble of deep learning models . There are approaches which proposed unified architecture based on deep learning, , , , ,  while some proposed unified statistical model. Additionally, there are some approaches uses data augmentation either through translation or labeling external data to make the model generalize across domains, , . Most of the above-discussed systems either shows high performance on (a) Twitter dataset or (b) Facebook dataset (given in the TRAC-2018), but not on both English code-mixed datasets.

This may be due to the text style or level of complexities of both datasets. So, we concentrated to develop a robust system for English code-mixed texts, and uni-lingual texts, which can also handle different writing styles. Our approach is based on three main ideas: Deep-Text Learning. The goal is to learn long range associations, dependencies between regions of text, N-grams, key-patterns, topical information, and sequential dependencies. Exploiting psycho-linguistic features with basic linguistic features as meta-data. The main aim is to minimize the direct dependencies on in-depth grammatical structure of the language (i.e., to support code-mixed data). We have also included emoticons, and punctuation features with it. We use the term "NLP Features" to represent it in the entire paper. Dual embedding based on FastText and Glove. This dual embedding helps in high vocabulary coverage and to capture the rare and partially incorrect words in the text (specially by FastText ). Our "Deep-text architecture" uses model averaging strategy with three different deep learning architectures.

For example, "Dear India, stop playing with the emotions of your people for votes." Non-Aggressive(NAG) - Generally these type of text lack any kind of aggression it is basically used to state facts, wishing on occasions and polite and supportive. The additional discussion on aggressiveness task can be found in Kaggle task , which just divided the task into two classes - i.e., presence or absence of aggression in tweets. The informal setting/environment of social media often encourage multilingual speakers to switch back and forth between languages when speaking or writing. These all resulted in code-mixing and code-switching. Code-mixing refers to the use of linguistic units from different languages in a single utterance or sentence, whereas code-switching refers to the co-occurrence of speech extracts belonging to two different grammatical systems. This language interchange makes the grammar more complex and thus it becomes tough to handle it by traditional algorithms. Thus the presence of high percentage of code-mixed content in social media text has increased the complexity of the aggression detection task. For example, the dataset provided by the organizers of TRAC-2018 ,  is actually a code-mixed dataset.

Introduction	The exponential increase of interactions on the various social media platforms has generated the huge amount of data on social media platforms like Facebook and Twitter, etc. These interactions resulted not only positive effect but also negative effect over billions of people owing to the fact that there are lots of aggressive comments (like hate, anger, and bullying). These cause not only mental and psychological stress but also account deactivation and even suicide. In this paper we concentrate on problems related to aggressiveness. The fine-grained definition of the aggressiveness/aggression identification is provided by the organizers of TRAC-2018 , . They have classified the aggressiveness into three labels (Overtly aggressive(OAG), Covertly aggressive(CAG), Non-aggressive(NAG)). The detailed description for each of the three labels is described as follows: Overtly Aggressive(OAG) - This type of aggression shows direct verbal attack pointing to the particular individual or group. For example, "Well said sonu..you have courage to stand against dadagiri of Muslims". Covertly Aggressive(CAG) - This type of aggression the attack is not direct but hidden, subtle and more indirect while being stated politely most of the times.

Related work	There are several works for aggression identification submitted at TRAC 2018 among them some approaches use the ensemble of multiple statistical models, , , . Similarly, some of the models like, ,  have used ensemble of statistical and deep learning models. In these models the statistical part of the model uses additional features from text analysis like parts-of-speech tags, punctuation, emotion, emoticon etc. Model like:  has used the ensemble of deep learning models based on majority voting. Some other models like: , ,  have used different models for Facebook and twitter. While approaches like:, , , , ,  have proposed unified architecture based on deep learning. Systems like, ,  have used data augmentation either through translation or labelling external data to make the model generalize across domains. While  has proposed a unified statistical model. Among approaches like extracted features from TF-IDF of character n-grams while uses LSTM with pre-trained embeddings from FastText.  have used the BiLSTM based model and the SVM metaclassifier model for the Facebook and Twitter test sets, respectively. While  tried ensembling of CNN, LSTM, and BILSTM.

Model averaging belongs to the family of ensemble learning techniques that uses multiple models for the same problem and combines their predictions to produce a more reliable and consistent prediction accuracy . This is the simplest form of weighted average ensemble based prediction where, each ensemble member contribute equally to predictions. Specifically in our case, three different models have been used. The following contains the intuition behind the selection of these three models: Deep Pyramid CNN  being deeper helps to learn long range associations between temporal regions of text using two-view embeddings. Disconnected RNN  is very helpful in encoding the sequential information with temporal key patterns in the text. Pooled BiLSTM In this architecture the last hidden state of BiLSTM is concatenated with mean and max-pooled representation of the hidden states obtained over all the time steps of Bi-LSTM. The idea of using mean and max pooling layers together is taken from  to avoid the loss of information in longer sequences of texts and max-pooling is taken to capture the topical information. NLP Features In each of the individual models, the NLP features are concatenated with last hidden state before the softmax classification layer as meta-data. The main aim is to provide additional information to the deep learning network.

Methodology ::: NLP Features	We have identified a novel combination of features which are highly effective in aggression classification when applied in addition to the features obtained from the deep learning classifier at the classification layer. We have introduced two new features in addition to the previously available features. The first one is the Emotion Sensor Feature which use a statistical model to classify the words into 7 different classes based on the sentences obtained from twitter and blogs which contain total 1,185,540 words. The second one is the collection of selected topical signal from text collected using Empath (see Table 1.). Different from previous approaches,  where  have used Emotion features in the form of frequency while  have used emotion feature vector obtained from LIWC 2007. Unlike we have used only 6 topical signals from Emapth. We have borrowed the idea of using other features like punctuation features and parts-of-speech tags from . The Table 1. lists and describes features, tools used to obtain them and the number of features resulted from each type.

Our system outperforms all the previous state of the art approaches used for aggression identification on English code-mixed TRAC data, while being trained only from Facebook comments the system outperforms other approaches on the additional Twitter test set. The remaining part of this paper is organized as follows: Section SECREF2 is an overview of related work. Section SECREF3 presents the methodology and algorithmic details. Section SECREF4 discusses the experimental evaluation of the system, and Section SECREF5 concludes this paper.

Experiment and Evaluation ::: Results and Discussion	In this paper, we have evaluated our model using weighted macro-averaged F-score. The measure is defined as in (See , ). It weights the F-score computed per class based on the class composition in the test set and then takes the average of these per-class F-score gives the final F-score. Table TABREF57, TABREF57 and TABREF57. presents the comparative experimental results for the proposed method in this paper with respect to the state-of-the-art. The top 5 models given in Table TABREF57 and TABREF57. are the best performing models for Facebook and Twitter test dataset respectively on TRAC 2018. We have followed all the experimental guidelines as discussed in TRAC contest guideline paper, . From the results given in Table TABREF57, TABREF57 and TABREF57 it is clear that our proposed model shows the best performance among all of the approaches. These results also state that all the deep learning architectures with NLP features, perform better than individual corresponding deep learning architectures. This means NLP features, adds some value to the architectures, even if it is not very high.

Conclusion and Future Work	In this paper, we have briefly described the approach we have taken to solve the aggressive identification on online social media texts which is very challenging since the dataset is noisy and code-mixed. We presented an ensemble of deep learning models which outperform previous approaches by sufficient margin while having the ability to generalize across domains. In future, we will explore other methods to increase the understanding of deep learning models on group targeted text, although the categories are well defined we will look after if we further fine-tune the categories with more data. In the future, we are planning to pay attention on a generalized language model for code-mixed texts which can also handle Hindi-code-mixed and other multi-lingual code-mixed datasets (i.e., trying to reduce the dependencies on language-specific code-mixed resources).

During implementation we use kernel size of 3(represented by $\textbf {x}$ to denote the small overlapping regions of text.), The number of filters(number of feature maps denoted by the number of rows of $\textbf {W}$) is 128 as depicted in Figure FIGREF29. With the number of filters same in each convolution layer and max-pooling with stride 2 makes the computation time halved, and doubles the net coverage of convolution kernel. Thus the deeper layers cause to learn long-range associations between regions of text. Let's say $h_{dpcnn} \in R^{p_1}$ be the hidden state obtained from DPCNN just before the classification layer and $f_{nlp} \in R^{24}$ be the NLP features computed from the text. Lets $z_1 \in R^{p_1 + 24}$ be another hidden state obtained as where, $\oplus $ denotes concatenation. The vector $z_1$ obtained, then fed to the fully connected layer with softmax activation.

Experiment and Evaluation ::: Experimental Setup	We have used Glove Embeddings concatenated with FastText Embeddings in all the three classification models presented in this paper. Specifically, we used Glove pre-trained vectors obtained from Twitter corpus containing 27 billion tokens and 1.2 million vocabulary entries where each word is represented using 100-dimensional vector. In the case of FastText the word is represented using 300-dimensional vector. Also, we have applied spatial dropout of 0.3 at embedding layer for DPCNN(in section SECREF30) and Pooled BiLSTM(in section SECREF45). For DPCNN model(in SECREF30) we have learnt 128-dimensional vector representation for unsupervised embeddings implicitly for task specific representation as in . Additionally, for DPCNN all the convolutional layers used 128 filters, kernel size of 3 and max-pooling stride 2. Additionally, in the case of DPCNN we have used kernel and bias regularizer of value 0.00001 for all convolutional kernels. The pre-activation function used in DPCNN is Parametric ReLU (PReLU) proposed in  while the activation at each of the convolutional kernel is linear.

Methodology	In this section, we describe our system architecture for aggressiveness classifier. In section SECREF23 we describe data preprocessing applied on the input text before feeding it to each of the classification models. Section SECREF26 describes the computation of NLP features. In Sections SECREF30, SECREF34 and SECREF45 we have described the architecture of different deep learning models like Deep Pyramid CNN, Disconnected RNN and Pooled BiLSTM respectively. Finally, in Section SECREF49, we describe model averaging based classification model which combines the prediction probabilities from three deep learninig architectures discussed above. (see Figure FIGREF22. for block diagram of system architecture).

The intuition behind the NLP features are the following: Emotion Sensor Dataset We have introduced to use of emotion sensor features, as a meta-data information. We have obtained the word sensor dataset from Kaggle. In this dataset each word is statistically classified into 7 distinct classes (Disgust, Surprise, Neutral, Anger, Sad, Happy and Fear) using Naive Bayes, based on sentences collected from twitter and blogs. Controlled Topical Signals from Empath. Empath can analyse the text across 200 gold standard topics and emotions. Additionally, it uses neural embedding to draw connotation among words across more than 1.8 billion words. We have used only selected categories like violence, hate, anger, aggression, social media and dispute from 200 Empath categories useful for us unlike which takes 194 categories. Emoticons frequently used on social media indicates the sense of sentence, , . Normalized frequency of POS tags According to , , ,  POS Tags provide the degree of target aggressiveness.

Like, we have used only four tags (a) adjective (JJ, JJR, JJS), (b) adverb (RB, RBR, RBS), (c) verb (VB, VBD, VBG, VBN, VBP, VBZ) and (d) noun (NN, NNS, NNP, NNPS) (See Penn-Treebank POS Tags for abbreviations and the full list). The main reason behind the selection of these four tags is to just identify words related to persons, activities, quality, etc, in the text. Sentiment polarity obtained from VADER Sentiment Analysis  (positive, negative and neutral) like used in , , , . It helps to demarcate aggressiveness with non-aggressiveness in the text. The block diagram of the proposed system is shown in Figure FIGREF22. The proposed system does not use any data augmentation techniques like , which is the top performer in TRAC (in English code-mixed Facebook data). This means the performance achieved by our system totally depends on the training dataset provided by TRAC. This also proves the effectiveness of our approach.

Methodology ::: Deep Pyramid CNN(DPCNN)	Since it has been proved that CNNs are great feature extractors for text classification, , , , ,  while deeper networks(whether RNNs or CNN's) has been proven for learning long-range association like deeper character level CNN's, , and complex combination of RNN and CNN, , , , . Deep Pyramid CNN (DPCNN) has 15 layers of word-level CNN's and contains similar pre-activation as proposed in improved Resnet. DPCNN outperforms the 32-layer character CNN and Hierarchical attention networks it has added advantage that due to its pyramid structure it does not require dimension matching in shortcut connections defined as z + h(z) as in where h(z) represents the skipped layers essentially contains two convolutional layers with pre-activation. It uses enhanced region embedding which consumes pre-trained embeddings (in our case it is FastText+Glove based dual embedding). Enhanced Region Embedding. The current DPCNN, uses two view type enhanced region embedding. For the text categorization, it defines a region of text as view-1 and its adjacent regions as view-2.

Methodology ::: Classification Model	According to deep learning literature , , , unweighted averaging might be a reasonable ensemble for similar base learners of comparable performance. Now, similar to the information discussed in , we can compute the model averaging (unweighted) by combining the softmax probabilities of three different classification models obtained from equations DISPLAY_FORM32, DISPLAY_FORM43, DISPLAY_FORM48. The averaged class probabilities are computed as: where K is the number of classes, and $\hat{y_i}$ is the predicted label for sentence $s_i$.

Some approaches like: has used emotions frequency as one of the features, while some others use sentiment emotion as feature. Also,,  have converted emoticons to their description.  have used TF-IDF of emoticons per-class as one of the features. Compared to all these approaches, we have concentrated to capture multiple linguistic/pattern based relations, key-terms and key-patters (with their association in text) through a combination of deep learning architectures with model averaging. We have also used NLP features as additional features with our deep learning architecture, obtained from psycho-linguistic and basic linguistic features.

Experiment and Evaluation ::: Dataset Description	We have used two datasets in our experimental evaluations: (1) TRAC 2018 Dataset and (2) Kaggle Dataset. TRAC 2018 Dataset: We have used the English code-mixed dataset provided by TRAC 2018. This dataset contains three labels, (a) Non-Aggressive(NAG), (b) Overtly-Aggressive (OAG) and (c) Covertly-Aggressive(CAG). The distribution of training, validation and test sets are described in Table TABREF56. Kaggle Dataset: This dataset contains 20001 tweets which are manually labeled. The labels are divided into two categories (indicating presence or absence of aggression in tweets) AGG(Aggressive) or NAG(Non-Aggressive). We have used the same test split available in the baseline code. The distribution for each of the training and test is given in Table TABREF56.

Experiment and Evaluation ::: Evaluation Strategy	To compare our experimental results we have used top-5 systems from the published results of TRAC-2018. To compare our results on Kaggle dataset, we have used the last & the best published result on Kaggle website as a baseline. We have conducted the separate experiments, to properly investigate the performance of (a) each of the classifiers (used in our model averaging based system), (b) impact of the NLP features on each of these classifiers and finally, (c) the performance of our proposed system. In Table TABREF57, TABREF57 and TABREF57, models, named as DPCNN(ref SECREF30), DRNN (ref SECREF34) and Pooled BiLSTM(ref SECREF45) are corresponding models without NLP features. Similarly, DPCNN+NLP Features, DRNN + NLP Features and Pooled BiLSTM + NLP Features are corresponding models with NLP features. The Model Averaging (A+B+C) is the ensemble of three models (i.e., model averaging of DPCNN, DRNN and Pooled BiLSTM) without NLP features.

We have used only 20 NLP features (except TF-IDF Emoticon feature and Punctuation feature as given in Table TABREF25) for Kaggle dataset (as these are not present in the Kaggle dataset).

Methodology ::: Pooled BiLSTM	The architecture has been shown in Figure FIGREF44. Given a sequence $s_i = [x_{1}, x_{2}, x_{3}, ..... x_{j}]$, where $x_j \in R^d$ is the d-dimensional word vector for word $w_j$, the hidden state obtained after BiLSTM is given as To avoid the loss of information because of modelling the entire sequence, we have concatenated the max-pooled($c_{max}$) and mean-pooled($c_{mean}$) representation of hidden states calculated over all time steps . We have also concatenated the nlp features, $f_{nlp} \in R^{24}$ the final feature vector $z_{3}$ is given as where $\oplus $ denotes concatenation. The final feature $z_3$ vector is fed to the fully connected layer with softmax activation. Let $y_{i3}^*$ be the softmax probablities, specifically for class label $k$ given as: where $K$ is the number of classes and $W_{bilstm}$ and $b_{bilstm}$ are the weight matrix and bias respectively.

For, DRNN(in section SECREF34) we have used the window size of 8 and rest of the parameters related to LSTM units are same as given in. For, Pooled BiLSTM(in section SECREF45) we have used LSTM hidden units size as 256. The maximum sequence length is 200 in all three models. In each of the classification model the classification layer contains the fully connected layer with softmax activation with output size of 3 equal to number of classes in case of TRAC 2018 dataset and its 2 in case of Kaggle dataset. Training has been done using ADAM optimizer for DPCNN and RMSPROP for DRNN and Pooled Bi-LSTM models. All the models are trained end-to-end using softmax cross entropy loss for TRAC 2018 dataset and binary cross entropy loss for Kaggle dataset. To train our model for TRAC 2018 dataset, we merged the training and validation dataset and then used 10% split from shuffled dataset to save the best model, for all classifiers.

Hence the hidden state representation at time step $t$ depends on all the previous input vectors given as Specifically we have used Bi-directional LSTM  to capture both past and future context. It provides $h_t$ from both directions(forward & backward). The forward LSTM takes the natural order of words from $x_{1}$ to $x_{n}$ to obtain $\overrightarrow{h_t}$, while backward-LSTM $x_{n}$ to $x_{1}$ to obtain $\overleftarrow{h_t}$. then $h_t$ is calculated as where $\oplus $ is the concatenation and $L$ is the size for one-directional LSTM. Therefore we denote the hidden state in equation DISPLAY_FORM37 with BiLSTM as To avoid handling of long sequence and to capture local information for each word we define the window size $k$ for each word such that the BiLSTM only sees the the previous $k-1$ words with the current word, where $k$ is a hyperparameter. We use padding <PAD> to make the slices of fixed size k(as shown in Figure FIGREF33).

Then using unlabeled data, it trains a neural network of one hidden layer with an artificial task of predicting view-2 from view-1. The obtained hidden layer, which is an embedding function that takes view-1 as input, serves as an unsupervised embedding function in the model for text categorization. The detailed architecture has been shown in Figure FIGREF29. Let each word input $x_j \in R^d$ be the d-dimensional vector for the $j^{th}$ word $w_{j}$ and the sentence $s_i$ contains sequence of $n$ words $\lbrace w_{1},w_{2},w_{3},......,w_{n}\rbrace $ as shown in Figure FIGREF29. In comparision to conventional convolution layer, DPCNN proposes to use pre-activation, thus essentially the convolutional layer of DPCNN is $\textbf {W}\sigma (\textbf {x})+\textbf {b}$, where $\textbf {W}$ and $\textbf {b}$(unique to each layer) are the weights matrix and bias respectively, we use $\sigma $ as PReLU.

Methodology ::: Data Preprocessing	We consider the text to be well formatted before applying the text to the embedding layer. First, we detect non-English text(which are few) and translate all of them to English using Google Translate. Still, there is some code mixed words like "mc", "bc" and other English abbreviations and spelling errors like "nd" in place of "and", "u" in place of "you" causes deep learning model to confuse with sentences of the same meaning. We follow the strategy of preprocessor as in to normalize the abbreviations and remove spelling errors, URLs and punctuation marks, converting emojis to their description. https://spacy.io/usage/linguistic-features#pos-tagging

Let $y_{i1}^*$ be the softmax probabilities, specifically for class label $k$ is given as: where $K$ is the number of classes, $W_{dpcnn}$ and $b_{dpcnn}$ are the weight matrix and bias respectively.

Methodology ::: Disconnected RNN(DRNN)	Given a sequence $s_i = [x_{1}, x_{2}, x_{3},....x_{n}]$ where $x_{j} \in R^d$ represents the d-dimensional word vector for word $w_{j}$ and $n$ is the length of input text applied to a variant of RNN called Long Short-Term Memory (LSTM) as shown in Figure FIGREF33. It is widely used for sequential modelling with long-term dependencies. For sequence modelling it keeps on updating the memory cell with current input using an adaptive gating mechanism. At time step $t$ the memory $c_t$ and the hidden state $h_t$ are updated as follows: where $\hat{c}_t$ is the current cell state obtained from current input $x_t$ and previous hidden state $h_{t-1}$, $i_t$, $f_t$ and $o_t$ are the activation corresponding to input gate, forget gate and output gate respectively, $\sigma $ denotes the logistic sigmoid function and $\odot $ denotes the element-wise multiplication.

It provides each hidden state $h_t$ with sequence of $k$ previous words. Since the phrase of $k$ words can lie anywhere in the text it helps to model the position invariant phrase representation due to which the it identifies key phrases important for identifying particular category. In this case, the equation of $h_t$ is given as The output hidden vectors, $H = [h_1, h_2, h_3, ...... h_n] \in R^{n \times 2L}$ are converted to fixed-length vector $h_{drnn} \in R^{2L}$ with max pooling over time: Let's say $f_{nlp} \in R^{24}$ be the NLP features computed from the text. Let's $z_2 \in R^{2L + 24}$ be another hidden state obtained as where $\oplus $ denotes concatenation. The vector $z_2$ obtained, then fed to the fully connected layer with softmax activation.

Let $y_{i2}^*$ be the softmax probabilities, specifically for class label $k$ is given as: where $K$ is the number of classes, $W_{drnn}$ is the weight matrix, and $b_{drnn}$ is the bias.

Finally, Our Proposed Method, which represents the model averaging of three models with NLP features.

Experimental results show that our proposed system outperforms all the previous approaches on English code-mixed dataset and uni-lingual English dataset.

Question: How have the differences in communication styles between Twitter and Facebook increase the complexity of the problem?
Answer: Systems do not perform well both in Facebook and Twitter texts