prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many users do they look at?
Context 1: Introduction
Over the past two decades, the emergence of social media has enabled the proliferation of traceable human behavior  The content posted by users can reflect who their friends are, what topics they are interested in, or which company they are working for  At the same time, users are listing a number of profile fields to define themselves to others  The utilization of such metadata has proven important in facilitating further developments of applications in advertising BIBREF0 , personalization BIBREF1 , and recommender systems BIBREF2   However, profile information can be limited, depending on the platform, or it is often deliberately omitted BIBREF3   To uncloak this information, a number of studies have utilized social media users' footprints to approximate their profiles 
This paper explores the potential of predicting a user's industry –the aggregate of enterprises in a particular field– by identifying industry indicative text in social media  The accurate prediction of users' industry can have a big impact on targeted advertising by minimizing wasted advertising BIBREF4 and improved personalized user experience  A number of studies in the social sciences have associated language use with social factors such as occupation, social class, education, and income BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8   An additional goal of this paper is to examine such findings, and in particular the link between language and occupational class, through a data-driven approach 
In addition, we explore how meaning changes depending on the occupational context  By leveraging word embeddings, we seek to quantify how, for example, cloud might mean a separate concept (e g , condensed water vapor) in the text written by users that work in environmental jobs while it might be used differently by users in technology occupations (e g , Internet-based computing) 
Specifically, this paper makes four main contributions  First, we build a large, industry-annotated dataset that contains over 20,000 blog users  In addition to their posted text, we also link a number of user metadata including their gender, location, occupation, introduction and interests 
Second, we build content-based classifiers for the industry prediction task and study the effect of incorporating textual features from the users' profile metadata using various meta-classification techniques, significantly improving both the overall accuracy and the average per industry accuracy 
Next, after examining which words are indicative for each industry, we build vector-space representations of word meanings and calculate one deviation for each industry, illustrating how meaning is differentiated based on the users' industries  We qualitatively examine the resulting industry-informed semantic representations of words by listing the words per industry that are most similar to job related and general interest terms 
Finally, we rank the different industries based on the normalized relative frequencies of emotionally charged words (positive and negative) and, in addition, discover that, for both genders, these frequencies do not statistically significantly correlate with an industry's gender dominance ratio 
After discussing related work in Section SECREF2 , we present the dataset used in this study in Section SECREF3   In Section SECREF4 we evaluate two feature selection methods and examine the industry inference problem using the text of the users' postings  We then augment our content-based classifier by building an ensemble that incorporates several metadata classifiers  We list the most industry indicative words and expose how each industrial semantic field varies with respect to a variety of terms in Section SECREF5   We explore how the frequencies of emotionally charged words in each gender correlate with the industries and their respective gender dominance ratio and, finally, conclude in Section SECREF6  
Related Work
Alongside the wide adoption of social media by the public, researchers have been leveraging the newly available data to create and refine models of users' behavior and profiling  There exists a myriad research that analyzes language in order to profile social media users  Some studies sought to characterize users' personality BIBREF9 , BIBREF10 , while others sequenced the expressed emotions BIBREF11 , studied mental disorders BIBREF12 , and the progression of health conditions BIBREF13   At the same time, a number of researchers sought to predict the social media users' age and/or gender BIBREF14 , BIBREF15 , BIBREF16 , while others targeted and analyzed the ethnicity, nationality, and race of the users BIBREF17 , BIBREF18 , BIBREF19   One of the profile fields that has drawn a great deal of attention is the location of a user  Among others, Hecht et al  Hecht11 predicted Twitter users' locations using machine learning on nationwide and state levels  Later, Han et al  Han14 identified location indicative words to predict the location of Twitter users down to the city level 
As a separate line of research, a number of studies have focused on discovering the political orientation of users BIBREF15 , BIBREF20 , BIBREF21   Finally, Li et al  Li14a proposed a way to model major life events such as getting married, moving to a new place, or graduating  In a subsequent study, BIBREF22 described a weakly supervised information extraction method that was used in conjunction with social network information to identify the name of a user's spouse, the college they attended, and the company where they are employed 
The line of work that is most closely related to our research is the one concerned with understanding the relation between people's language and their industry  Previous research from the fields of psychology and economics have explored the potential for predicting one's occupation from their ability to use math and verbal symbols BIBREF23 and the relationship between job-types and demographics BIBREF24   More recently, Huang et al  Huang15 used machine learning to classify Sina Weibo users to twelve different platform-defined occupational classes highlighting the effect of homophily in user interactions  This work examined only users that have been verified by the Sina Weibo platform, introducing a potential bias in the resulting dataset  Finally, Preotiuc-Pietro et al  Preoctiuc15 predicted the occupational class of Twitter users using the Standard Occupational Classification (SOC) system, which groups the different jobs based on skill requirements  In that work, the data collection process was limited to only users that specifically mentioned their occupation in their self-description in a way that could be directly mapped to a SOC occupational class  The mapping between a substring of their self-description and a SOC occupational class was done manually  Because of the manual annotation step, their method was not scalable; moreover, because they identified the occupation class inside a user self-description, only a very small fraction of the Twitter users could be included (in their case, 5,191 users) 
Both of these recent studies are based on micro-blogging platforms, which inherently restrict the number of characters that a post can have, and consequently the way that users can express themselves 
Moreover, both studies used off-the-shelf occupational taxonomies (rather than self-declared occupation categories), resulting in classes that are either too generic (e g , media, welfare and electronic are three of the twelve Sina Weibo categories), or too intermixed (e g , an assistant accountant is in a different class from an accountant in SOC)  To address these limitations, we investigate the industry prediction task in a large blog corpus consisting of over 20K American users, 40K web-blogs, and 560K blog posts 
Dataset
We compile our industry-annotated dataset by identifying blogger profiles located in the U S  on the profile finder on http://www blogger com, and scraping only those users that had the industry profile element completed 
For each of these bloggers, we retrieve all their blogs, and for each of these blogs we download the 21 most recent blog postings  We then clean these blog posts of HTML tags and tokenize them, and drop those bloggers whose cumulative textual content in their posts is less than 600 characters  Following these guidelines, we identified all the U S  bloggers with completed industry information 
Traditionally, standardized industry taxonomies organize economic activities into groups based on similar production processes, products or services, delivery systems or behavior in financial markets  Following such assumptions and regardless of their many similarities, a tomato farmer would be categorized into a distinct industry from a tobacco farmer  As demonstrated in Preotiuc-Pietro et al  Preoctiuc15 such groupings can cause unwarranted misclassifications 
The Blogger platform provides a total of 39 different industry options  Even though a completed industry value is an implicit text annotation, we acknowledge the same problem noted in previous studies: some categories are too broad, while others are very similar  To remedy this and following Guibert et al  Guibert71, who argued that the denominations used in a classification must reflect the purpose of the study, we group the different Blogger industries based on similar educational background and similar technical terminology  To do that, we exclude very general categories and merge conceptually similar ones  Examples of broad categories are the Education and the Student options: a teacher could be teaching in any concentration, while a student could be enrolled in any discipline  Examples of conceptually similar categories are the Investment Banking and the Banking options 
The final set of categories is shown in Table TABREF1 , along with the number of users in each category  The resulting dataset consists of 22,880 users, 41,094 blogs, and 561,003 posts  Table TABREF2 presents additional statistics of our dataset 
Text-based Industry Modeling
After collecting our dataset, we split it into three sets: a train set, a development set, and a test set  The sizes of these sets are 17,880, 2,500, and 2,500 users, respectively, with users randomly assigned to these sets  In all the experiments that follow, we evaluate our classifiers by training them on the train set, configure the parameters and measure performance on the development set, and finally report the prediction accuracy and results on the test set  Note that all the experiments are performed at user level, i e , all the data for one user is compiled into one instance in our data sets 
To measure the performance of our classifiers, we use the prediction accuracy  However, as shown in Table TABREF1 , the available data is skewed across categories, which could lead to somewhat distorted accuracy numbers depending on how well a model learns to predict the most populous classes  Moreover, accuracy alone does not provide a great deal of insight into the individual performance per industry, which is one of the main objectives in this study  Therefore, in our results below, we report: (1) micro-accuracy ( INLINEFORM0 ), calculated as the percentage of correctly classified instances out of all the instances in the development (test) data; and (2) macro-accuracy ( INLINEFORM1 ), calculated as the average of the per-category accuracies, where the per-category accuracy is the percentage of correctly classified instances out of the instances belonging to one category in the development (test) data 
Leveraging Blog Content
In this section, we seek the effectiveness of using solely textual features obtained from the users' postings to predict their industry 
The industry prediction baseline Majority is set by discovering the most frequently featured class in our training set and picking that class in all predictions in the respective development or testing set 
After excluding all the words that are not used by at least three separate users in our training set, we build our AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier  As seen in Figure FIGREF3 , we can far exceed the Majority baseline performance by incorporating basic language signals into machine learning algorithms (173% INLINEFORM0 improvement) 
We additionally explore the potential of improving our text classification task by applying a number of feature ranking methods and selecting varying proportions of top ranked features in an attempt to exclude noisy features  We start by ranking the different features, w, according to their Information Gain Ratio score (IGR) with respect to every industry, i, and training our classifier using different proportions of the top features  INLINEFORM0 INLINEFORM1
Even though we find that using the top 95% of all the features already exceeds the performance of the All Words model on the development data, we further experiment with ranking our features with a more aggressive formula that heavily promotes the features that are tightly associated with any industry category  Therefore, for every word in our training set, we define our newly introduced ranking method, the Aggressive Feature Ranking (AFR), as: INLINEFORM0
In Figure FIGREF3 we illustrate the performance of all four methods in our industry prediction task on the development data  Note that for each method, we provide both the accuracy ( INLINEFORM0 ) and the average per-class accuracy ( INLINEFORM1 )  The Majority and All Words methods apply to all the features; therefore, they are represented as a straight line in the figure  The IGR and AFR methods are applied to varying subsets of the features using a 5% step 
Our experiments demonstrate that the word choice that the users make in their posts correlates with their industry  The first observation in Figure FIGREF3 is that the INLINEFORM0 is proportional to INLINEFORM1 ; as INLINEFORM2 increases, so does INLINEFORM3   Secondly, the best result on the development set is achieved by using the top 90% of the features using the AFR method  Lastly, the improvements of the IGR and AFR feature selections are not substantially better in comparison to All Words (at most 5% improvement between All Words and AFR), which suggest that only a few noisy features exist and most of the words play some role in shaping the “language" of an industry 
As a final evaluation, we apply on the test data the classifier found to work best on the development data (AFR feature selection, top 90% features), for an INLINEFORM0 of 0 534 and INLINEFORM1 of 0 477 
Leveraging User Metadata
Together with the industry information and the most recent postings of each blogger, we also download a number of accompanying profile elements  Using these additional elements, we explore the potential of incorporating users' metadata in our classifiers 
Table TABREF7 shows the different user metadata we consider together with their coverage percentage (not all users provide a value for all of the profile elements)  With the exception of the gender field, the remaining metadata elements shown in Table TABREF7 are completed by the users as a freely editable text field  This introduces a considerable amount of noise in the set of possible metadata values  Examples of noise in the occupation field include values such as “Retired”, “I work ”, or “momma” which are not necessarily informative for our industry prediction task 
To examine whether the metadata fields can help in the prediction of a user's industry, we build classifiers using the different metadata elements  For each metadata element that has a textual value, we use all the words in the training set for that field as features  The only two exceptions are the state field, which is encoded as one feature that can take one out of 50 different values representing the 50 U S  states; and the gender field, which is encoded as a feature with a distinct value for each user gender option: undefined, male, or female 
As shown in Table TABREF9 , we build four different classifiers using the multinomial NB algorithm: Occu (which uses the words found in the occupation profile element), Intro (introduction), Inter (interests), and Gloc (combined gender, city, state) 
In general, all the metadata classifiers perform better than our majority baseline ( INLINEFORM0 of 18 88%)  For the Gloc classifier, this result is in alignment with previous studies BIBREF24   However, the only metadata classifier that outperforms the content classifier is the Occu classifier, which despite missing and noisy occupation values exceeds the content classifier's performance by an absolute 3 2% 
To investigate the promise of combining the five different classifiers we have built so far, we calculate their inter-prediction agreement using Fleiss's Kappa BIBREF25 , as well as the lower prediction bounds using the double fault measure BIBREF26   The Kappa values, presented in the lower left side of Table TABREF10 , express the classification agreement for categorical items, in this case the users' industry  Lower values, especially values below 30%, mean smaller agreement  Since all five classifiers have better-than-baseline accuracy, this low agreement suggests that their predictions could potentially be combined to achieve a better accumulated result 
Moreover, the double fault measure values, which are presented in the top-right hand side of Table TABREF10 , express the proportion of test cases for which both of the two respective classifiers make false predictions, essentially providing the lowest error bound for the pairwise ensemble classifier performance  The lower those numbers are, the greater the accuracy potential of any meta-classification scheme that combines those classifiers  Once again, the low double fault measure values suggest potential gain from a combination of the base classifiers into an ensemble of models 
After establishing the promise of creating an ensemble of classifiers, we implement two meta-classification approaches  First, we combine our classifiers using features concatenation (or early fusion)  Starting with our content-based classifier (Text), we successively add the features derived from each metadata element  The results, both micro- and macro-accuracy, are presented in Table TABREF12   Even though all these four feature concatenation ensembles outperform the content-based classifier in the development set, they fail to outperform the Occu classifier 
Second, we explore the potential of using stacked generalization (or late fusion) BIBREF27   The base classifiers, referred to as L0 classifiers, are trained on different folds of the training set and used to predict the class of the remaining instances  Those predictions are then used together with the true label of the training instances to train a second classifier, referred to as the L1 classifier: this L1 is used to produce the final prediction on both the development data and the test data  Traditionally, stacking uses different machine learning algorithms on the same training data  However in our case, we use the same algorithm (multinomial NB) on heterogeneous data (i e , different types of data such as content, occupation, introduction, interests, gender, city and state) in order to exploit all available sources of information 
The ensemble learning results on the development set are shown in Table TABREF12   We notice a constant improvement for both metrics when adding more classifiers to our ensemble except for the Gloc classifier, which slightly reduces the performance  The best result is achieved using an ensemble of the Text, Occu, Intro, and Inter L0 classifiers; the respective performance on the test set is an INLINEFORM0 of 0 643 and an INLINEFORM1 of 0 564  Finally, we present in Figure FIGREF11 the prediction accuracy for the final classifier for each of the different industries in our test dataset  Evidently, some industries are easier to predict than others  For example, while the Real Estate and Religion industries achieve accuracy figures above 80%, other industries, such as the Banking industry, are predicted correctly in less than 17% of the time  Anecdotal evidence drawn from the examination of the confusion matrix does not encourage any strong association of the Banking class with any other  The misclassifications are roughly uniform across all other classes, suggesting that the users in the Banking industry use language in a non-distinguishing way 
Qualitative Analysis
In this section, we provide a qualitative analysis of the language of the different industries 
Top-Ranked Words
To conduct a qualitative exploration of which words indicate the industry of a user, Table TABREF14 shows the three top-ranking content words for the different industries using the AFR method 
Not surprisingly, the top ranked words align well with what we would intuitively expect for each industry  Even though most of these words are potentially used by many users regardless of their industry in our dataset, they are still distinguished by the AFR method because of the different frequencies of these words in the text of each industry 
Industry-specific Word Similarities
Next, we examine how the meaning of a word is shaped by the context in which it is uttered  In particular, we qualitatively investigate how the speakers' industry affects meaning by learning vector-space representations of words that take into account such contextual information  To achieve this, we apply the contextualized word embeddings proposed by Bamman et al  Bamman14, which are based on an extension of the “skip-gram" language model BIBREF28  
In addition to learning a global representation for each word, these contextualized embeddings compute one deviation from the common word embedding representation for each contextual variable, in this case, an industry option  These deviations capture the terms' meaning variations (shifts in the INLINEFORM0 -dimensional space of the representations, where INLINEFORM1 in our experiments) in the text of the different industries, however all the embeddings are in the same vector space to allow for comparisons to one another 
Using the word representations learned for each industry, we present in Table TABREF16 the terms in the Technology and the Tourism industries that have the highest cosine similarity with a job-related word, customers  Similarly, Table TABREF17 shows the words in the Environment and the Tourism industries that are closest in meaning to a general interest word, food  More examples are given in the Appendix SECREF8  
The terms that rank highest in each industry are noticeably different  For example, as seen in Table TABREF17 , while food in the Environment industry is similar to nutritionally and locally, in the Tourism industry the same word relates more to terms such as delicious and pastries  These results not only emphasize the existing differences in how people in different industries perceive certain terms, but they also demonstrate that those differences can effectively be captured in the resulting word embeddings 
Emotional Orientation per Industry and Gender
As a final analysis, we explore how words that are emotionally charged relate to different industries  To quantify the emotional orientation of a text, we use the Positive Emotion and Negative Emotion categories in the Linguistic Inquiry and Word Count (LIWC) dictionary BIBREF29   The LIWC dictionary contains lists of words that have been shown to correlate with the psychological states of people that use them; for example, the Positive Emotion category contains words such as “happy,” “pretty,” and “good ”
For the text of all the users in each industry we measure the frequencies of Positive Emotion and Negative Emotion words normalized by the text's length  Table TABREF20 presents the industries' ranking for both categories of words based on their relative frequencies in the text of each industry 
We further perform a breakdown per-gender, where we once again calculate the proportion of emotionally charged words in each industry, but separately for each gender  We find that the industry rankings of the relative frequencies INLINEFORM0 of emotionally charged words for the two genders are statistically significantly correlated, which suggests that regardless of their gender, users use positive (or negative) words with a relative frequency that correlates with their industry  (In other words, even if e g , Fashion has a larger number of women users, both men and women working in Fashion will tend to use more positive words than the corresponding gender in another industry with a larger number of men users such as Automotive )
Finally, motivated by previous findings of correlations between job satisfaction and gender dominance in the workplace BIBREF30 , we explore the relationship between the usage of Positive Emotion and Negative Emotion words and the gender dominance in an industry  Although we find that there are substantial gender imbalances in each industry (Appendix SECREF9 ), we did not find any statistically significant correlation between the gender dominance ratio in the different industries and the usage of positive (or negative) emotional words in either gender in our dataset 
Conclusion
In this paper, we examined the task of predicting a social media user's industry  We introduced an annotated dataset of over 20,000 blog users and applied a content-based classifier in conjunction with two feature selection methods for an overall accuracy of up to 0 534, which represents a large improvement over the majority class baseline of 0 188 
We also demonstrated how the user metadata can be incorporated in our classifiers  Although concatenation of features drawn both from blog content and profile elements did not yield any clear improvements over the best individual classifiers, we found that stacking improves the prediction accuracy to an overall accuracy of 0 643, as measured on our test dataset  A more in-depth analysis showed that not all industries are equally easy to predict: while industries such as Real Estate and Religion are clearly distinguishable with accuracy figures over 0 80, others such as Banking are much harder to predict 
Finally, we presented a qualitative analysis to provide some insights into the language of different industries, which highlighted differences in the top-ranked words in each industry, word semantic similarities, and the relative frequency of emotionally charged words 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation 
Additional Examples of Word Similarities
Context 2: Introduction
Emotion detection has long been a topic of interest to scholars in natural language processing (NLP) domain  Researchers aim to recognize the emotion behind the text and distribute similar ones into the same group  Establishing an emotion classifier can not only understand each user's feeling but also be extended to various application, for example, the motivation behind a user's interests BIBREF0  Based on releasing of large text corpus on social media and the emotion categories proposed by BIBREF1, BIBREF2, numerous models have provided and achieved fabulous precision so far  For example, DeepMoji BIBREF3 which utilized transfer learning concept to enhance emotions and sarcasm understanding behind the target sentence  CARER BIBREF4 learned contextualized affect representations to make itself more sensitive to rare words and the scenario behind the texts 
As methods become mature, text-based emotion detecting applications can be extended from a single utterance to a dialogue contributed by a series of utterances  Table TABREF2 illustrates the difference between single utterance and dialogue emotion recognition  The same utterances in Table TABREF2, even the same person said the same sentence, the emotion it convey may be various, which may depend on different background of the conversation, tone of speaking or personality  Therefore, for emotion detection, the information from preceding utterances in a conversation is relatively critical 
In SocialNLP 2019 EmotionX, the challenge is to recognize emotions for all utterances in EmotionLines dataset, a dataset consists of dialogues  According to the needs for considering context at the same time, we develop two classification models, inspired by bidirectional encoder representations from transformers (BERT) BIBREF5, FriendsBERT and ChatBERT  In this paper, we introduce our approaches including causal utterance modeling, model pre-training, and fine-turning 
Dataset
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues  The former comes from the scripts of the Friends TV sitcom  The other is made up of Facebook messenger chats  Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances  All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content  Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral  If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral” 
For the datasets, there are properties worth additional mentioning  Although Friends and EmotionPush share the same data format, they are quite different in nature  Friends is a speech-based dataset which is annotated dialogues from the TV sitcom  It means most of the utterances are generated by the a few main characters  The personality of a character often affects the way of speaking, and therefore “who is the speaker" might provide extra clues for emotion prediction  In contrast, EmotionPush does not have this trait due to the anonymous mechanism  In addition, features such as typo, hyperlink, and emoji that only appear in chat-based data will need some domain-specific techniques to process 
Incidentally, the objective of the challenge is to predict the emotion for each utterance  Just, according to EmotionX 2019 specification, there are only four emotions be selected as our label candidates, which are Joy, Sadness, Anger, and Neutral  These emotions will be considered during performance evaluation  The technical detail will also be introduced and discussed in following Section SECREF13 and Section SECREF26 
Model Description
For this challenge, we adapt BERT which is proposed by BIBREF5 to help understand the context at the same time  Technically, BERT, designed on end-to-end architecture, is a deep pre-trained transformer encoder that dynamically provides language representation and BERT already achieved multiple state-of-the-art results on GLUE benchmark BIBREF7 and many tasks  A quick recap for BERT's architecture and its pre-training tasks will be illustrated in the following subsections 
Model Description ::: Model Architecture
BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level  Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context  The transformer and its self-attention mechanism are proposed by BIBREF8  This self-attention mechanism can be interpreted as a key-value mapping given query  By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$  The self-attention BIBREF8 is formally represented as:
The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version  Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration  Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder 
For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512  The way that BERT was designed is for giving model the sentence-level and token-level understanding  In two sentences case, a special token ([SEP]) will be inserted between two sentences  In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning  The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks  In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels  Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$ 
Model Description ::: Pre-training Tasks
In pre-training, intead of using unidirectional language models, BERT developed two pre-training tasks: (1) Masked LM (cloze test) and (2) Next Sentence Prediction  At the first pre-training task, bidirectional language modeling can be done at this cloze-like pre-training  In detail, 15% tokens of input sequence will be masked at random and model need to predict those masked tokens  The encoder will try to learn contextual representations from every given tokens due to masking tokens at random  Model will not know which part of the input is going to be masked, so that the information of each masked tokens should be inferred by remaining tokens  At Next Sentence Prediction, two sentences concatenated together will be considered as model input  In order to give model a good nature language understanding, knowing relationship between sentence is one of important abilities  When generating input sequences, 50% of time the sentence B is actually followed by sentence A, and rest 50% of the time the sentence B will be picked randomly from dataset, and model need to predict if the sentence B is next sentence of sentence A  That is, the attention information will be shared between sentences  Such sentence-level understanding may have difficulties to be learned at first pre-training task (Masked LM), therefore, the pre-training task (NSP) is developed as second training goal to capture the cross sentence relationship 
In this competition, limited by the size of dataset and the challenge in contextual emotion recognition, we consider BERT with both two pre-training tasks can give a good starting point to extract emotion changing during dialogue-like conversation  Especially the second pre-training task, it might be more important for dialogue-like conversation where the emotion may various by the context of continuous utterances  That is, given a set of continues conversations, the emotion of current utterance might be influenced by previous utterance  By this assumption and with supporting from the experiment results of BERT, we can take sentence A as one-sentence context and consider sentence B as the target sentence for emotion prediction  The detail will be described in Section SECREF4 
Methodology
The main goal of the present work is to predict the emotion of utterance within the dialogue  Following are four major difficulties we concern about:
The emotion of the utterances depends not only on the text but also on the interaction happened earlier 
The source of the two datasets are different  Friends is speech-based dialogues and EmotionPush is chat-based dialogues  It makes datasets possess different characteristics 
There are only $1,000$ dialogues in both training datasets which are not large enough for the stability of training a complex neural-based model 
The prediction targets (emotion labels) are highly unbalanced 
The proposed approach is summarized in Figure FIGREF3, which aims to overcome these challenges  The framework could be separated into three steps and described as follow:
Methodology ::: Causal Utterance Modeling
Given a dialogue $D^{(i)}$ which includes sequence of utterances denoted as $D^{(i)}=(u^{(i)}_{1}, u^{(i)}_{2},    , u^{(i)}_{n})$, where $i$ is the index in dataset and $n$ is the number of utterances in the given dialogue  In order to conserve the emotional information of both utterance and conversation, we rearrange each two consecutive utterances $u_{t}, u_{t-1}$ into a single sentence representation $x_{t}$ as
The corresponding sentence representation corpus $X^{(i)}$ are denoted as $X^{(i)}=(x^{(i)}_{1}, x^{(i)}_{2},    , x^{(i)}_{n})$  Note that the first utterance within a conversation does not have its causal utterance (previous sentence), therefore, the causal utterance will be set as [None]  A practical example of sentence representation is shown in Table TABREF11 
Since the characteristics of two datasets are not identical, we customize different causal utterance modeling strategies to refine the information in text 
For Friends, there are two specific properties  The first one is that most dialogues are surrounding with the six main characters, including Rachel, Monica, Phoebe, Joey, Chandler, and Ross
Context 3: Introduction
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person  Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0   Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1   Many high profile incidents have emphasized the prevalence of cyberbullying on social media  Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs 
Detection of cyberbullying in social media is a challenging task  Definition of what constitutes cyberbullying is quite subjective  For example, frequent use of swear words might be considered as bullying by the general population  However, for teen oriented social media platforms such as Formspring, this does not necessarily mean bullying (Table TABREF9 )  Across multiple SMPs, cyberbullies attack victims on different topics such as race, religion, and gender  Depending on the topic of cyberbullying, vocabulary and perceived meaning of words vary significantly across SMPs  For example, in our experiments we found that for word `fat', the most similar words as per Twitter dataset are `female' and `woman' (Table TABREF23 )  However, other two datasets do not show such particular bias against women  This platform specific semantic similarity between words is a key aspect of cyberbullying detection across SMPs  Style of communication varies significantly across SMPs  For example, Twitter posts are short and lack anonymity  Whereas posts on Q&A oriented SMPs are long and have option of anonymity (Table TABREF7 )  Fast evolving words and hashtags in social media make it difficult to detect cyberbullying using swear word list based simple filtering approaches  The option of anonymity in certain social networks also makes it harder to identify cyberbullying as profile and history of the bully might not be available 
Past works on cyberbullying detection have at least one of the following three bottlenecks  First (Bottleneck B1), they target only one particular social media platform  How these methods perform across other SMPs is unknown  Second (Bottleneck B2), they address only one topic of cyberbullying such as racism, and sexism  Depending on the topic, vocabulary and nature of cyberbullying changes  These models are not flexible in accommodating changes in the definition of cyberbullying  Third (Bottleneck B3), they rely on carefully handcrafted features such as swear word list and POS tagging  However, these handcrafted features are not robust against variations in writing style  In contrast to existing bottlenecks, this work targets three different types of social networks (Formspring: a Q&A forum, Twitter: microblogging, and Wikipedia: collaborative knowledge repository) for three topics of cyberbullying (personal attack, racism, and sexism) without doing any explicit feature engineering by developing deep learning based models along with transfer learning 
We experimented with diverse traditional machine learning models (logistic regression, support vector machine, random forest, naive Bayes) and deep neural network models (CNN, LSTM, BLSTM, BLSTM with Attention) using variety of representation methods for words (bag of character n-gram, bag of word unigram, GloVe embeddings, SSWE embeddings)  Summary of our findings and research contributions is as follows 
Datasets
Please refer to Table TABREF7 for summary of datasets used  We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media  We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages)  Each dataset addresses a different topic of cyberbullying  Twitter dataset contains examples of racism and sexism  Wikipedia dataset contains examples of personal attack  However, Formspring dataset is not specifically about any single topic  All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts  Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset  We measure the size of a post in terms of the number of words in the post  For each dataset, there are only a few posts with large size  We truncate such large posts to the size of post ranked at 95 percentile in that dataset  For example, in Wikipedia dataset, the largest post has 2846 words  However, size of post ranked at 95 percentile in that dataset is only 231  Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words  This truncation affects only a small minority of posts in each dataset  However, it is required for efficiently training various models in our experiments  Details of each dataset are as follows 
Formspring BIBREF2 : It was a question and answer based website where users could openly invite others to ask and answer questions  The dataset includes 12K annotated question and answer pairs  Each post is manually labeled by three workers  Among these pairs, 825 were labeled as containing cyberbullying content by at least two Amazon Mechanical turk workers 
Twitter BIBREF3 : This dataset includes 16K annotated tweets  The authors bootstrapped the corpus collection, by performing an initial manual search of common slurs and terms used pertaining to religious, sexual, gender, and ethnic minorities  Of the 16K tweets, 3117 are labeled as sexist, 1937 as racist, and the remaining are marked as neither sexist nor racist 
Wikipedia BIBREF4 : For each page in Wikipedia, a corresponding talk page maintains the history of discussion among users who participated in its editing  This data set includes over 100k labeled discussion comments from English Wikipedia's talk pages  Each comment was labeled by 10 annotators via Crowdflower on whether it contains a personal attack  There are total 13590 comments labeled as personal attack 
Use of Swear Words and Anonymity
Please refer to Table TABREF9   We use the following short forms in this section: B=Bullying, S=Swearing, A=Anonymous  Some of the values for Twitter dataset are undefined as Twitter does not allow anonymous postings  Use of swear words has been repeatedly linked to cyberbullying  However, preliminary analysis of datasets reveals that depending on swear word usage can neither lead to high precision nor high recall for cyberbullying detection  Swear word list based methods will have low precision as P(B INLINEFORM0 S) is not close to 1  In fact, for teen oriented social network Formspring, 78% of the swearing posts are non-bullying  Swear words based filtering will be irritating to the users in such SMPs where swear words are used casually  Swear word list based methods will also have a low recall as P(S INLINEFORM1 B) is not close to 1

ground_truth: 22,880 users

20,000

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the sources of the datasets?
Context 2:  The utterance ratio of given by the six roles is up to $83 4\%$  Second, the personal characteristics of the six characters are very clear  Each leading role has its own emotion undulated rule  To make use of these features, we introduce the personality tokenization which help learning the personality of the six characters  Personality tokenization concatenate the speaker and says tokens before the input utterance if the speaker is one of the six characters  The example is shown in Table TABREF12 
For EmotionPush, the text are informal chats which including like slang, acronym, typo, hyperlink, and emoji  Another characteristic is that the specific name entities are tokenized with random index  (e g  “organization_80”, “person_01”, and “time_12”)  We consider some of these informal text are related to expressing emotion such as repeated typing, purposed capitalization, and emoji (e g  “:D”, “:(”, and “<3”))  Therefore, we keep most informal expressions but only process hyperlinks, empty utterance, and name entities by unifying the tokens 
Methodology ::: Model Pre-training
Since the size of both datasets are not large enough for complex neural-based model training as well as BERT model is only pre-train on formal text datasets, the issues of overfitting and domain bias are important considerations for design the pre-training process 
To avoid our model overfitting on the training data and increase the understanding of informal text, we adapted BERT and derived two models, namely FriendsBERT and ChatBERT, with different pre-training tasks before the formal training process for Friends and EmotionPush dataset, respectively  The pre-training strategies are described below 
For pre-training FriendsBERT, we collect the completed scripts of all ten seasons of Friends TV shows from emorynlp which includes 3,107 scenes within 61,309 utterances  All the utterances are followed the preprocessing methods mentions above to compose the corpus for Masked language model pre-training task  The consequent utterances in the same scenes are considered as the consequent sentences to pre-train the Next Sentence Prediction task  In the pre-training process, the training loss is the sum of the mean likelihood of two pre-train tasks 
For pre-training ChatBERT, we pre-train our model on the Twitter dataset, since the text and writing style on Twitter are close to the chat text where both may involved with many informal words or emoticons as well  The Twitter emotion dataset, 8 basic emotions from emotion wheel BIBREF1, was collected by twitter streaming API with specific emotion-related hashtags, such as #anger, #joy, #cry, #sad and etc  The hashtags in tweets are treated as emotion label for model fine-tuning  The tweets were fine-grined processing followed the rules in BIBREF9, BIBREF4, including duplicate tweets removing, the emotion hashtags must appearing in the last position of a tweet, and etc  The statis of tweets were summarized in Table TABREF17  Each tweet and corresponding emotion label composes an emotion classification dataset for pre-training 
Methodology ::: Fine-tuning
Since our emotion recognition task is treated as a sequence-level classification task, the model would be fine-tuned on the processed training data  Following the BERT construction, we take the first embedding vector which corresponds to the special token [CLS] from the final hidden state of the Transformer encoder  This vector represents the embedding vector of the corresponding conversation utterances which is denoted as $\mathbf {C} \in \mathbb {R}^{H}$, where $H$ is the embedding size  A dense neural layer is treated as a classification layer which consists of parameters $\mathbf {W} \in \mathbb {R}^{K\times H}$ and $\mathbf {b} \in \mathbb {R}^{K}$, where $K$ is the number of emotion class  The emotion prediction probabilities $\mathbf {P} \in \mathbb {R}^{K}$ are computed by a softmax activation function as
All the parameters in BERT and the classification layer would be fine-turned together to minimize the Negative Log Likelihood (NLL) loss function, as Equation (DISPLAY_FORM22), based on the ground truth emotion label $c$ 
In order to tackle the problem of highly unbalanced emotion labels, we apply weighted balanced warming on NLL loss function, as Equation (DISPLAY_FORM23), in the first epoch of fine-tuning procedure 
where $\mathbf {w}$ are the weights of corresponding emotion label $c$ which are computed and normalize by the frequency as
By adding the weighted balanced warming on NLL loss, the model could learn to predict the minor emotions (e g  anger and sadness) earlier and make the training process more stable  Since the major evaluation metrics micro F1-score is effect by the number of each label, we only apply the weighted balanced warming in first epoch to optimize the performance 
Experiments
Since the EmotionX challenge only provided the gold labels in training data, we pick the best performance model (weights) to predict the testing data  In this section, we present the experiment and evaluation results 
Experiments ::: Experimental Setup
The EmotionX challenge consists of $1,000$ dialogues for both Friends and EmotionPush  In all of our experiments, each dataset is separated into top 800 dialogues for training and last 200 dialogues for validation  Since the EmotionX challenge considers only the four emotions (anger, joy, neutral, and sadness) in the evaluation stage, we ignore all the data point corresponding to other emotions directly  The details of emotions distribution are shown in Table TABREF18 
The hyperparameters and training setup of our models (FriendsBERT and ChatBERT) are shown in Table TABREF25  Some common and easily implemented methods are selected as the baselines embedding methods and classification models  The baseline embedding methods are including bag-of-words (BOW), term frequency–inverse document frequency (TFIDF), and neural-based word embedding  The classification models are including Logistic Regression (LR), Random Forest (RF), TextCNN BIBREF10 with initial word embedding as GloVe BIBREF11, and our proposed model  All the experiment results are based on the best performances of validation results 
Experiments ::: Performance
The experiment results of validation on Friends are shown in Table TABREF19  The proposed model and baselines are evaluated based on the Precision (P ), Recall (R ), and F1-measure (F1) 
For the traditional baselines, namely BOW and TFIDF, we observe that they achieve surprising high F1 scores around $0 81$, however, the scores for Anger and Sadness are lower  This explains that traditional approaches tend to predict the labels with large sample size, such as Joy and Neutral, but fail to take of scarce samples even when an ensemble random forest classifier is adopted  In order to prevent the unbalanced learning, we choose the weighted loss mechanism for both TextCNN and causal modeling TextCNN (C-TextCNN), these models suffer less than the traditional baselines and achieve a slightly balance performance, where there are around 15% and 7% improvement on Anger and Sadness, respectively  We following adopt the casual utterance modeling to original TextCNN, mapping previous utterance as well as target utterance into model  The causal utterance modeling improve the C-TextCNN over TextCNN for 6%, 2% and 1% on Anger, Joy and overall F1 score  Motivated from these preliminary experiments, the proposed FriendsBERT also adopt the ideas of both weighted loss and causal utterance modeling  As compared to the original BERT, single sentence BERT (FriendsBERT-base-s), the proposed FriendsBERT-base improve 1% for Joy and overall F1, and 2% for Sadness  For the final validation performance, our proposed approach achieves the highest scores, which are $0 85$ and $0 86$ for FriendsBERT-base and FriendsBERT-large, respectively 
Overall, the proposed FriendsBERT successfully captures the sentence-level context-awarded information and outperforms all the baselines, which not only achieves high performance on large sample labels, but also on small sample labels  The similar settings are also adapted to EmotionPush dataset for the final evaluation 
Experiments ::: Evaluation Results
The testing dataset consists of 240 dialogues including $3,296$ and $3,536$ utterances in Friends and EmotionPush respectively  We re-train our FriendsBERT and ChatBERT with top 920 training dialogues and predict the evaluation results using the model performing the best validation results  The results are shown in Table TABREF29 and Table TABREF30  The present method achieves $81 5\%$ and $88 5\%$ micro F1-score on the testing dataset of Friends and EmotionPush, respectively 
Conclusion and Future work
In the present work, we propose FriendsBERT and ChatBERT for the multi-utterance emotion recognition task on EmotionLines dataset  The proposed models are adapted from BERT BIBREF5 with three main improvement during the model training procedure, which are the causal utterance modeling mechanism, specific model pre-training, and adapt weighted loss  The causal utterance modeling takes the advantages of the sentence-level context information during model inference  The specific model pre-training helps to against the bias in different text domain  The weighted loss avoids our model to only predict on large size sample  The effectiveness and generalizability of the proposed methods are demonstrated from the experiments 
In future work, we consider to include the conditional probabilistic constraint $P ({\rm Emo}_{B} | \hat{\rm Emo}_{A})$  Model should predict the emotion based on a certain understanding about context emotions  This might be more reasonable for guiding model than just predicting emotion of ${\rm Sentence}_B$ directly  In addition, due to the limitation of BERT input format, ambiguous number of input sentences is now becoming an important design requirement for our future work  Also, personality embedding development will be another future work of the emotion recognition  The personality embedding will be considered as sentence embedding injected into word embedding, and it seems this additional information can contribute some improvement potentially 
Context 3:  For Twitter dataset, 82% of bullying posts do not use any swear words  Such passive-aggressive cyberbullying will go undetected with swear word list based methods  Anonymity is another clue that is used for detecting cyberbullying as bully might prefer to hide its identity  Anonymity definitely leads to increased use of swear words (P(S INLINEFORM2 A) INLINEFORM3 P(S)) and cyberbullying (P(B INLINEFORM4 A) INLINEFORM5 P(B), and P(B INLINEFORM6 A&S)) INLINEFORM7 P(B))  However, significant fraction of anonymous posts are non-bullying (P(B INLINEFORM8 A) not close to 1) and many of bullying posts are not anonymous (P(A INLINEFORM9 B) not close to 1)  Further, anonymity might not be allowed by many SMPs such as Twitter 
Related Work
Cyberbullying is recognized as a phenomenon at least since 2003 BIBREF5   Use of social media exploded with launching of multiple platforms such as Wikipedia (2001), MySpace (2003), Orkut (2004), Facebook (2004), and Twitter (2005)  By 2006, researchers had pointed that cyberbullying was as serious phenomenon as offline bullying BIBREF6   However, automatic detection of cyberbullying was addressed only since 2009 BIBREF7   As a research topic, cyberbullying detection is a text classification problem  Most of the existing works fit in the following template: get training dataset from single SMP, engineer variety of features with certain style of cyberbullying as the target, apply a few traditional machine learning methods, and evaluate success in terms of measures such as F1 score and accuracy  These works heavily rely on handcrafted features such as use of swear words  These methods tend to have low precision for cyberbullying detection as handcrafted features are not robust against variations in bullying style across SMPs and bullying topics  Only recently, deep learning has been applied for cyberbullying detection BIBREF8   Table TABREF27 summarizes important related work 
Deep Neural Network (DNN) Based Models
We experimented with four DNN based models for cyberbullying detection: CNN, LSTM, BLSTM, and BLSTM with attention  These models are listed in the increasing complexity of their neural architecture and amount of information used by these models  Please refer to Figure 1 for general architecture that we have used across four models  Various models differ only in the Neural Architecture layer while having identical rest of the layers  CNNs are providing state-of-the-results on extracting contextual feature for classification tasks in images, videos, audios, and text  Recently, CNNs were used for sentiment classification BIBREF9   Long Short Term Memory networks are a special kind of RNN, capable of learning long-term dependencies  Their ability to use their internal memory to process arbitrary sequences of inputs has been found to be effective for text classification BIBREF10   Bidirectional LSTMs BIBREF11 further increase the amount of input information available to the network by encoding information in both forward and backward direction  By using two directions, input information from both the past and future of the current time frame can be used  Attention mechanisms allow for a more direct dependence between the state of the model at different points in time  Importantly, attention mechanism lets the model learn what to attend to based on the input sentence and what it has produced so far 
The embedding layer processes a fixed size sequence of words  Each word is represented as a real-valued vector, also known as word embeddings  We have experimented with three methods for initializing word embeddings: random, GloVe BIBREF12 , and SSWE BIBREF13   During the training, model improves upon the initial word embeddings to learn task specific word embeddings  We have observed that these task specific word embeddings capture the SMP specific and topic specific style of cyberbullying  Using GloVe vectors over random vector initialization has been reported to improve performance for some NLP tasks  Most of the word embedding methods such as GloVe, consider only syntactic context of the word while ignoring the sentiment conveyed by the text  SSWE method overcomes this problem by incorporating the text sentiment as one of the parameters for word embedding generation  We experimented with various dimension size for word embeddings  Experimental results reported here are with dimension size as 50  There was no significant variation in results with dimension size ranging from 30 to 200 
To avoid overfitting, we used two dropout layers, one before the neural architecture layer and one after, with dropout rates of 0 25 and 0 5 respectively  Fully connected layer is a dense output layer with the number of neurons equal to the number of classes, followed by softmax layer that provides softmax activation  All our models are trained using backpropagation  The optimizer used for training is Adam and the loss function is categorical cross-entropy  Besides learning the network weights, these methods also learn task-specific word embeddings tuned towards the bullying labels (See Section SECREF21 )  Our code is available at: https://github com/sweta20/Detecting-Cyberbullying-Across-SMPs 
Experiments
Existing works have heavily relied on traditional machine learning models for cyberbullying detection  However, they do not study the performance of these models across multiple SMPs  We experimented with four models: logistic regression (LR), support vector machine (SVM), random forest (RF), and naive Bayes (NB), as these are used in previous works (Table TABREF27 )  We used two data representation methods: character n-gram and word unigram  Past work in the domain of detecting abusive language have showed that simple n-gram features are more powerful than linguistic and syntactic features, hand-engineered lexicons, and word and paragraph embeddings BIBREF14   As compared to DNN models, performance of all four traditional machine learning models was significantly lower  Please refer to Table TABREF11  
All DNN models reported here were implemented using Keras  We pre-process the data, subjecting it to standard operations of removal of stop words, punctuation marks and lowercasing, before annotating it to assigning respective labels to each comment  For each trained model, we report its performance after doing five-fold cross-validation  We use following short forms 
Effect of Oversampling Bullying Instances
The training datasets had a major problem of class imbalance with posts marked as bullying in the minority  As a result, all models were biased towards labeling the posts as non-bullying  To remove this bias, we oversampled the data from bullying class thrice  That is, we replicated bullying posts thrice in the training data  This significantly improved the performance of all DNN models with major leap in all three evaluation measures  Table TABREF17 shows the effect of oversampling for a variety of word embedding methods with BLSTM Attention as the detection model  Results for other models are similar BIBREF15   We can notice that oversampled datasets (F+, T+, W+) have far better performance than their counterparts (F, T, W respectively)  Oversampling particularly helps the smallest dataset Formspring where number of training instances for bullying class is quite small (825) as compared to other two datasets (about 5K and 13K)

ground_truth: Friends TV sitcom, Facebook messenger chats

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What cyberbulling topics did they address?
Context 3:  We also experimented with varying the replication rate for bullying posts BIBREF15   However, we observed that for bullying posts, replication rate of three is good enough 
Choice of Initial Word Embeddings and Model
Initial word embeddings decide data representation for DNN models  However during the training, DNN models modify these initial word embeddings to learn task specific word embeddings  We have experimented with three methods to initialize word embeddings  Please refer to Table TABREF19   This table shows the effect of varying initial word embeddings for multiple DNN models across datasets  We can notice that initial word embeddings do not have a significant effect on cyberbullying detection when oversampling of bullying posts is done (rows corresponding to F+, T+, W+)  In the absence of oversampling (rows corresponding to F, T W), there is a gap in performance of simplest (CNN) and most complex (BLSTM with attention) models  However, this gap goes on reducing with the increase in the size of datasets 
Table TABREF20 compares the performance of four DNN models for three evaluation measures while using SSWE as the initial word embeddings  We have noticed that most of the time LSTM performs weaker than other three models  However, performance gap in the other three models is not significant 
Task Specific Word Embeddings
DNN models learn word embeddings over the training data  These learned embeddings across multiple datasets show the difference in nature and style of bullying across cyberbullying topics and SMPs  Here we report results for BLSTM with attention model  Results for other models are similar  We first verify that important words for each topic of cyberbullying form clusters in the learned embeddings  To enable the visualization of grouping, we reduced dimensionality with t-SNE BIBREF16 , a well-known technique for dimensionality reduction particularly well suited for visualization of high dimensional datasets  Please refer to Table TABREF22   This table shows important clusters observed in t-SNE projection of learned word embeddings  Each cluster shows that words most relevant to a particular topic of bullying form cluster 
We also observed changes in the meanings of the words across topics of cyberbullying  Table TABREF23 shows most similar words for a given query word for two datasets  Twitter dataset which is heavy on sexism and racism, considers word slave as similar to targets of racism and sexism  However, Wikipedia dataset that is about personal attacks does not show such bias 
Transfer Learning
We used transfer learning to check if the knowledge gained by DNN models on one dataset can be used to improve cyberbullying detection performance on other datasets  We report results where BLSTM with attention is used as the DNN model  Results for other models are similar BIBREF15   We experimented with following three flavors of transfer learning 
Complete Transfer Learning (TL1): In this flavor, a model trained on one dataset was directly used to detect cyberbullying in other datasets without any extra training  TL1 resulted in significantly low recall indicating that three datasets have different nature of cyberbullying with low overlap (Table TABREF25 )  However precision was relatively higher for TL1, indicating that DNN models are cautious in labeling a post as bully (Table TABREF25 )  TL1 also helps to measure similarity in nature of cyberbullying across three datasets  We can observe that bullying nature in Formspring and Wikipedia datasets is more similar to each other than the Twitter dataset  This can be inferred from the fact that with TL1, cyberbullying detection performance for Formspring dataset is higher when base model is Wikipedia (precision =0 51 and recall=0 66)as compared to Twitter as the base model (precision=0 38 and recall=0 04)  Similarly, for Wikipedia dataset, Formspring acts as a better base model than Twitter while using TL1 flavor of transfer learning  Nature of SMP might be a factor behind this similarity in nature of cyberbullying  Both Formspring and Wikipedia are task oriented social networks (Q&A and collaborative knowledge repository respectively) that allow anonymity and larger posts  Whereas communication on Twitter is short, free of anonymity and not oriented towards a particular task 
Feature Level Transfer Learning (TL2): In this flavor, a model was trained on one dataset and only learned word embeddings were transferred to another dataset for training a new model  As compared to TL1, recall score improved dramatically with TL2 (Table TABREF25 )  Improvement in precision was also significant (Table TABREF25 )  These improvements indicate that learned word embeddings are an essential part of knowledge transfer across datasets for cyberbullying detection 
Model Level Transfer Learning (TL3): In this flavor, a model was trained on one dataset and learned word embeddings, as well as network weights, were transferred to another dataset for training a new model  TL3 does not result in any significant improvement over TL2  This lack of improvement indicates that transfer of network weights is not essential for cyberbullying detection and learned word embeddings is the key knowledge gained by the DNN models 
DNN based models coupled with transfer learning beat the best-known results for all three datasets  Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0 68 and 0 93 respectively  We achieve F1 scores of 0 94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 )  For Formspring dataset, authors have not reported F1 score  Their method has accuracy score of 78 5% BIBREF2   We achieve F1 score of 0 95 with accuracy score of 98% for the same dataset 
Conclusion and Future Work
We have shown that DNN models can be used for cyberbullying detection on various topics across multiple SMPs using three datasets and four DNN models  These models coupled with transfer learning beat state of the art results for all three datasets  These models can be further improved with extra data such as information about the profile and social graph of users  Most of the current datasets do not provide any information about the severity of bullying  If such fine-grained information is made available, then cyberbullying detection models can be further improved to take a variety of actions depending on the perceived seriousness of the posts 

ground_truth: personal attack, racism, and sexism

racism, sexism, personal attack, not specifically about any single topic