A Trolling Hierarchy in Social Media and A Conditional Random Field For Trolling Detection

Abstract: An-ever increasing number of social media websites, electronic newspapers and Internet forums allow visitors to leave comments for others to read and interact. This exchange is not free from participants with malicious intentions, which do not contribute with the written conversation. Among different communities users adopt strategies to handle such users. In this paper we present a comprehensive categorization of the trolling phenomena resource, inspired by politeness research and propose a model that jointly predicts four crucial aspects of trolling: intention, interpretation, intention disclosure and response strategy. Finally, we present a new annotated dataset containing excerpts of conversations involving trolls and the interactions with other users that we hope will be a useful resource for the research community.

In contrast to traditional content distribution channels like television, radio and newspapers, Internet opened the door for direct interaction between the content creator and its audience. One of these forms of interaction is the presence of comments sections that are found in many websites. The comments section allows visitors, authenticated in some cases and unauthenticated in others, to leave a message for others to read. This is a type of multi-party asynchronous conversation that offers interesting insights: one can learn what is the commenting community thinking about the topic being discussed, their sentiment, recommendations among many other. There are some comment sections in which commentators are allowed to directly respond to others, creating a comment hierarchy. These kind of written conversations are interesting because they bring light to the types interaction between participants with minimal supervision. This lack of supervision and in some forums, anonymity, give place to interactions that may not be necessarily related with the original topic being discussed, and as in regular conversations, there are participants with not the best intentions. Such participants are called trolls in some communities.
Even though there are some studies related to trolls in different research communities, there is a lack of attention from the NLP community. We aim to reduce this gap by presenting a comprehensive categorization of trolling and propose two models to predict trolling aspects. First, we revise the some trolling definitions: “Trolling is the activity of posting messages via communication networks that are in tended to be provocative, offensive or menacing” by BIBREF0 , this definition considers trolling from the most negative perspective where a crime might be committed. In a different tone, BIBREF1 provides a working definition for troll: “A troller in a user in a computer mediated communication who constructs the identity of sincerely wishing to be part of the group in question, including professing, or conveying pseudo-sincere intentions, but whose real intention(s) is/are to cause disruption and/or trigger or exacerbate conflict for the purpose of their own amusement”. These definitions inspire our trolling categorization, but first, we define a trolling event: a comment in a conversation whose intention is to cause conflict, trouble; be malicious, purposely seek or disseminate false information or advice; give a dishonest impression to deceive; offend, insult, cause harm, humiliation or aggravation. Also, a troll or troller is the individual that generates a trolling event, trolling is the overall phenomena that involves a troll, trolling event and generates responses from others. Any participant in a forum conversation may become a troll at any given point, as we will see, the addressee of a trolling event may choose to reply with a trolling comment or counter-trolling, effectively becoming a troll as well.
We believe our work makes four contributions. First, unlike previous computational work on trolling, which focused primarily on analyzing the narrative retrospectively by the victim (e.g., determining the trolling type and the role played by each participant), we study trolling by analyzing comments in a conversation, aiming instead to identify trollers, who, once identified, could be banned from posting. Second, while previous work has focused on analyzing trolling from the troll's perspective, we additionally model trolling from the target's perspective, with the goal understanding the psychological impact of a trolling event on the target, which we believe is equally important from a practical standpoint. Third, we propose a comprehensive categorization of trolling that covers not only the troll's intention but also the victim and other commenters' reaction to the troll's comment. We believe such categorization will provide a solid basis on which future computational approaches to trolling can be built. Finally, we make our annotated data set consisting of 1000 annotated trolling events publicly available. We believe that our data set will be a valuable resource to any researcher interested in the computational modeling of trolling.
Based on the previous definitions we identify four aspects that uniquely define a trolling event-response pair: 1) Intention: what is the author of the comment in consideration purpose, a) trolling, the comment is malicious in nature, aims to disrupt, annoy, offend, harm or spread purposely false information, b) playing the comment is playful, joking, teasing others without the malicious intentions as in a), or c) none, the comment has no malicious intentions nor is playful, it is a simple comment. 2) Intention Disclosure: this aspect is meant to indicate weather a trolling comment is trying to deceive its readers, the possible values for this aspect are a) the comment's author is a troll and is trying to hide its real intentions, and pretends to convey a different meaning, at least temporarily, b) the comment's author is a troll but is clearly exposing its malicious intentions and c) the comment's author is not a troll, therefore there are not hidden or exposed malicious or playful intentions. There are two aspects defined on the comments that direct address the comment in consideration, 3) Intentions Interpretation: this aspect refers to the responder's understanding of the parent's comment intentions. There possible interpretations are the same as the intentions aspect: trolling, playing or none. The last element, is the 4) Response strategy employed by the commentators directly replaying to a comment, which can be a trolling event. The response strategy is influenced directly by the responder's interpretation of the parent's comment intention. We identify 14 possible response strategies. Some of these strategies are tied with combinations of the three other aspects. We briefly define each of them in the appendix.
Figure FIGREF2 shows this categories as a hierarchy. Using this trolling formulation, the suspected troll event and the responses are correlated and one cannot independently name the strategy response without learning about the other three aspects. This is challenging prediction problem that we address in this work.
To illustrate this hierarchy, we present some examples. These are excerpts from original conversations; the first comment, generated by author C0, on each excerpt is given as a minimal piece of context, the second comment, by the author C1 in italics, is the comment suspected to be a trolling event. The rest of the comments, are all direct responses to the suspected trolling comment. When the response author “name” is the same as the first comment, it indicates that the that same individual also replied to the suspected troll.
Example 1.
[noitemsep,nolistsep] 
My friend who makes $20,000 a year leased a brand new Chevy Spark EV, for only $75 per month and he got a California rebate for driving an electric car. Much cheaper than buying older car which usually require heavy upkeep due to its mileage. At this point I think you're just trolling.
[noitemsep,nolistsep]
IYour friend has a good credit score, which can't be said about actual poor people. Did you grow up sheltered by any chance?
[noitemsep,nolistsep]
Judging by your post history, you're indeed a troll. Have a good one.
In this example, when C1 asks “Did you grow up sheltered by any chance?", her intention is to denigrate or be offensive, and it is not hiding it, instead he is clearly disclosing her trolling intentions. In C0's response, we see that has came to the conclusion that C1 is trolling and his response strategy is frustrate the trolling event by ignoring the malicious troll's intentions.
Example 2.
[noitemsep,nolistsep] 
What do you mean look up ?:( I don't see anything lol
[noitemsep,nolistsep]
Look up! Space is cool! :)
[noitemsep,nolistsep]
why must you troll me :(
Keep going, no matter how many times you say it, he will keep asking
In this example, we hypothesize that C0 is requesting some information and C1 is given an answer that is unfit to C0's' request. We do so based on the last C0's comment; CO is showing disappointment or grievance. Also, we deduct that C1 is trying to deceive C0, therefore, C1's comment is a trolling event. This is a trolling event whose intention is to purposely convey false information, and that hiding its intentions. As for the response, in the last C0's comment, he has finally realized or interpreted that C1's real intentions are deceiving and since his comment shows a “sad emoticon” his reply is emotionally, with aggravation, so we say that CO got engaged. C2 on the other hand, acknowledges the malicious and play along with the troll.
Given these examples, address the task of predicting the four aspects of a trolling event based on the methodology described in the next section.
We collected all available comments in the stories from Reddit from August 2015. Reddit is popular website that allows registered users (without identity verification) to participate in forums specific a post or topic. These forums are of they hierarchical type, those that allow nested conversation, where the children of a comment are its direct response. To increase recall and make the annotation process feasible we created an inverted index with Lucene and queried for comments containing the word troll with an edit distance of 1, to include close variations of this word. We do so inspired by the method by BIBREF2 to created a bullying dataset, and because we hypothesize that such comments will be related or involved in a trolling event. As we observed in the dataset, people use the word troll in many different ways, sometimes it is to point out that some used is indeed trolling him or her or is accusing someone else of being a troll. Other times, people use the term, to express their frustration or dislike about a particular user, but there is no trolling event. Other times, people simple discuss about trolling and trolls, without actually participating or observing one directly. Nonetheless, we found that this search produced a dataset in which 44.3 % of the comments directly involved a trolling event. Moreover, as we exposed our trolling definition, it is possible for commentators in a conversation to believe that they are witnessing a trolling event and respond accordingly even where there is none. Therefore, even in the comments that do not involve trolling, we are interested in learning what triggers users interpretation of trolling where it is not present and what kind of response strategies are used. We define as a suspected trolling event in our dataset a comment in which at least one of its children contains the word troll.
With the gathered comments, we reconstructed the original conversation trees, from the original post, the root, to the leaves, when they were available and selected a subset to annotated. For annotation purposes, we created snippets of conversations as the ones shown in Example 1 and Example 2 consisting of the parent of the suspected trolling event, the suspected trolling event comment, and all of the direct responses to the suspected trolling event. We added an extra constraint that the parent of the suspected trolling event should also be part of the direct responses, we hypothesize that if the suspected trolling event is indeed trolling, its parent should be the object of its trolling and would have a say about it. We recognize that this limited amount of information is not always sufficient to recover the original message conveyed by all of the participants in the snippet, and additional context would be beneficial. However, the trade off is that snippets like this allow us to make use of Amazon Mechanical Turk (AMT) to have the dataset annotated, because it is not a big burden for a “turker” to work on an individual snippet in exchange for a small pay, and expedites the annotation process by distributing it over dozens of people. Specifically, for each snippet, we requested three annotators to label the four aspects previously described. Before annotating, we set up a qualification test along with borderline examples to guide them in process and align them with our criteria. The qualification test turned out to be very selective since only 5% of all of the turkers that attempted it passed the exam. Our dataset consists of 1000 conversations with 5868 sentences and 71033 tokens. The distribution over the classes per trolling aspect is shown in the table TABREF24 in the column “Size”.
Inter-Annotator Agreement. Due to the subjective nature of the task we did not expected perfect agreement. However, we obtained substantial inter-annotator agreement as we measured the fleiss-kappa statistic BIBREF3 for each of the trolling aspects: Intention: 0.578, Intention Disclosure: 0.556, Interpretation: 0.731 and Response 0.632. After inspecting the dataset, we manually reconciled aspects of the threads that found no majority on the turkers annotation and verified and corrected consistency on the four tasks on each thread.
In this section we propose to solve the following problem: given a comment in a conversation, suspected to a trolling event, it's parent comment and all it's direct responses, we aim to predict the suspected comment I: intention, its D: intention disclosure and from the responses point of view, for each response comment the R: interpretation of the suspected troll comment's intentions, and identify its B: response strategy. This problem can be seen as a multi-task prediction. To do so, we split the dataset into training and testing sets using a 5-fold cross validation setup.
For prediction we define two sets of features, a basic and an enhanced dataset, extracted from each of the comments in the dataset. The features are described below.
N-gram features. We encode each unigram and bigram collected from the training comments a binary feature. In a similar manner, we include the unigram and bigram along with their POS tag as in BIBREF2 . To extract these features we used the most current version of the Stanford CoreNLP BIBREF4 . Each token's
Lemmas as in BIBREF5 as a binary feature.
Harmful Vocabulary. In their research on bullying BIBREF6 identified a small set of words that are highly offensive. We encode them as well as binary features.
Emotions Synsets. As in BIBREF5 we extracted all lemmas associated with each of Synsets extracted from WordNet BIBREF7 from these emotions: anger, embarrassment, empathy, fear, pride, relief and sadness. As well all the synonyms from these emotions extracted from the dictionary. Also,
Emoticons. Reddit's comments make extensive use of emoticons, we argue that some emoticons are specially used in trolling events and to express a variety of emotions, which we hypothesize would be useful to identify a comments intention, interpretation and response. For that we use the emoticon dictionary BIBREF8 and we set a binary feature for each emoticon that is found in the dictionary.
Sentiment Polarity. Using a similar idea, we hypothesize that the overall comment emotion would be useful to identify the response and intention in a trolling event. So, we apply the Vader Sentiment Polarity Analyzer BIBREF9 and include a four features, one per each measurement given by the analyzer: positive, neutral, negative and a composite metric, each as a real number value.
Subjectivity Lexicon. From the MPQA Subjective Lexicon BIBREF10 we include all tokens that are found in the lexicon as binary features. This lexicon was created from a news domains, so the words in it don't necessarily align with the informal vocabulary used in Reddit, but, there are serious Reddit users that use proper language and formal constructions. We believe that these features will allow us to discriminate formal comments from being potentially labeled as trolling events, which tend to be vulgar.
Swearing Vocabulary. We manually collected 1061 swear words and short phrases from the internet, blogs, forums and smaller repositories. The informal nature of this dictionary resembles the type of language used by flaming trolls and agitated responses, so we encode a binary feature for each word or short phrase in a comment if it appears in the swearing dictionary.
Framenet. Following BIBREF11 use of FrameNet, we apply the Semaphore Parser BIBREF12 to each sentence in every comment in the training set, and construct three different binary features: every frame name that is present in the sentence, the frame name a the target word associated with it, and the argument name along with the token or lexical unit in the sentence associated with it. We argue that some frames are especially interesting from the trolling perspective. For example, the frame “Deception_success” precisely models one of the trolling models, and we argue that these features will be particularly to identify trolling events in which semantic and not just syntactic information is necessary.
Politeness Queues. BIBREF13 identified queues that signal polite and impolite interactions among groups of people collaborating online. Based on our observations of trolling examples, it is clear that flaming troll and engaged or emotional responses would use impolite queues. On the contrary, neutralizing and frustrating responses to troll avoid falling in confrontation and their vocabulary tends to be more polite. So use these queues as binary features as they appear in the comments in consideration.
The most naïve approach is to consider each of the four tasks as an independent classification problem. Such system would be deprived from the other's tasks information that we've mentioned is strictly necessary to make a correct prediction of the response strategy. Instead, as our baseline we follow a pipeline approach, using the tasks oder: I, D, R and B, so that each of the subsequent subtasks' feature set is extended with a feature per each of previously computed subtasks. We argue that this setup is a competitive baseline, as can be verified in the results table TABREF24 . For the classifier in the pipeline approach we choose a log-linear model, a logistic regression classifier. In addition to logistic regression, we tried the generative complement of logistic regression, naïve bayes and max-margin classifier, a support vector machine, but their performance was not superior to the logistic regression. It is noteworthy to mention that the feature set used for the intention predict is the combined features sets of the suspected troll comment as well as its parent. We do so in all of our experiments the learner can take advantage of the conversation context.
The nature of this problem makes the use of a joint model a logical choice. Among different options for joint inference, we choose a (conditional) probabilistic graphical model (henceforth PGM) BIBREF15 because, in contrast to ILP formulations, has the ability to learn parameters and not just impose hard constraints. Also, compared to Markov Logic Networks BIBREF16 , a relatively recent formulation that combines logic and Markov Random Fields, PGMs in practice have proved to be more scalable, even though, inference in general models is shown to be intractable. Finally, we are also interested in choosing a PGM because it allow to directly compare the strength of joint inference with the baseline, because the our model is a collection of logistic regressors trained simultaneously.
A conditional random field factorizes the conditional probability distribution over all possible values of the query variables in the model, given a set of observations as in equation EQREF22 . In our model, the query variables are the four tasks we desire to predict, INLINEFORM0 and the observations is their combined feature sets INLINEFORM1 . Each of the factors INLINEFORM2 in this distribution is a log-linear model as in equation EQREF23 and represents the probability distribution of the clique of variables INLINEFORM3 in it, given the observation set INLINEFORM4 . This is identical to the independent logistic regression model described in the baseline, except for the fact that all variables or tasks are consider a the same time. To do so, we add additional factors that connect task variables among them, permitting the flow of information of one task to the other.
Specifically, our model represent each task with a random variable, shown in figure FIGREF15 (left), represented by the circles. The plate notation that surrounds variables INLINEFORM0 and INLINEFORM1 indicates that there will as many variables INLINEFORM2 and INLINEFORM3 and edges connecting them to INLINEFORM4 as the number of responses in the problem snippet. The edges connecting INLINEFORM5 and INLINEFORM6 with INLINEFORM7 attempts to model influence of these two variables on the response, and how this information is passed along to the response strategy variable INLINEFORM8 . Figure FIGREF15 (right) explicitly represents the cliques in the underlying factor graph. We can see that there are unary factors, INLINEFORM9 , INLINEFORM10 , INLINEFORM11 and INLINEFORM12 , that model the influence of the observation features over their associated variables, just as the logistic regression model does. Factors INLINEFORM13 models the interaction between variables INLINEFORM14 and INLINEFORM15 , INLINEFORM16 the interaction between variables INLINEFORM17 and INLINEFORM18 and INLINEFORM19 models the interactions between variables INLINEFORM20 and INLINEFORM21 , using a log-linear model over the possible values of the pair of variables in that particular clique.
Due to the size of the model, we are able to perform exact inference at train and test time. For parameter learning we employ limited memory lbfgs optimizer BIBREF17 as we provide the cost function and gradient based on the equations described in BIBREF18 .
2 pass Model A hybrid mode that we experiment with is model that performs joint inference on three tasks: I: intention, D: intention disclosure and R: responders' intention interpretation. The remaining task B: response strategy is performed in a second step, with the input the other three tasks. We do so because we observed in our experiments that the close coupling between the first three tasks allow them to perform better independently of the response strategy, as we will elaborate in the results section. DISPLAYFORM0 DISPLAYFORM1 
We perform 5-fold cross validation on the dataset. We use the first fold to tune the parameters and the remaining four folds to report results. The system performance is measured using precision, recall and F-1 as shown in table TABREF24 . The left side of the table, reports results obtained using the basic feature set, while the right side does so on the enhanced feature set. In order to maintain consistency folds are created based on the threads or snippets and for the case of the baseline system, all instances in the particular fold for task in consideration are considered independent of each other. On the table, rows show the classes performance for each of the tasks, indicated by a heard with the task name. For the response strategy we present results for those class values that are at least 5% of the total distribution, we do so, because the number of labeled instances for this classes is statistically insignificant compared to the majority classes.
From the result table TABREF24 , we observe that hybrid model significantly outperform the baseline, by more than 20 points in intention and intention disclosure prediction. For the response strategy, it is clear that none of the systems offer satisfying results; this showcases the difficult of such a large number of classes. Nonetheless, the hybrid model outperforms the fully joint model and baseline in all but one the response strategy classes. However, the differences are far less impressive as in the other tasks. It is surprisingly; that the full joint model did not offered the best performance. One of the reasons behind this is that intention, intentions disclosure and interpretation tasks are hurt by the difficulty of learning parameters that maximize the response strategy, this last task drags the other three down in performance. Another reason is that, the features response strategy is not informative enough to learn the correct concept, and due to the joint inference process, all tasks receive a hit. Also, it is counter-intuitive that the augmented set of features did not outperform in all tasks but in intentions disclosure and interpretation, and just by a small margin. A reason explaining this unexpected behavior is that the majority of enhanced features are already represented in the basic feature set by means of the unigrams and bigrams, and the Framenet and Sentiment features are uninformative or redundant. Lastly, we observe that for interpretation category, none of systems were able to predict the “playing” class. This is because of the relative size of the number of instances annotated with that value, 1% of the entire dataset. We hypothesize those instances labeled by the annotators, of which a majority agreed on, incorrectly selected the playing category instead of the trolling class, and that, at the interpretation level, one can only expect to reliably differentiate between trolling and trolling.
In this section, we discuss related work in the areas of trolling, bullying and politeness, as they intersect in their scope and at least partially address the problem presented in this work.
 BIBREF19 address the problem of identifying manipulation trolls in news community forums. The major difference with this work is that all their predictions are based on meta-information such as number of votes, dates, number of comments and so on. There is no NLP approach to the problem and their task is limited to identifying trolls. BIBREF0 and BIBREF20 elaborate a deep description of the trolls personality, motivations, effects on the community that trolls interfere and the criminal and psychological aspects of trolls. Their main focus are flaming trolls, but have no NLP insights do not propose and automated prediction tasks as in this work. In a networks related framework BIBREF21 and BIBREF22 present a methodology to identify malicious individuals in a network based solely on the network's properties. Even though they offer present and evaluate a methodology, their focus is different from NLP. BIBREF23 proposes a method that involves NLP components, but fails to provide a evaluation of their system. Finally, BIBREF2 and BIBREF5 address bullying traces. That is self reported events of individuals describing being part of bullying events, but their focus is different from trolling event and the interactions with other participants.
In this paper we address the under-attended problem of trolling in Internet forums. We presented a comprehensive categorization of trolling events and defined a prediction tasks that does not only considers trolling from the troll's perspective but includes the responders to the trolls comment. Also, we evaluated three different models and analyzed their successes and shortcomings. Finally we provide an annotated dataset which we hope will be useful for the research community. We look forward to investigate trolling phenomena in larger conversations, formalize the concepts of changing roles among the participants in trolling events, and improve response strategy performance.

Figure and table captions:
Figure 1: Trolling categorization based on four aspects: Comment’s Intention and Intentions Disclosure, and Response’s Interpretation and Strategy
Figure 2: Trolling tasks modeled as a (conditional) probabilistic graphical model (left). Factor graph showing all cliques or direct interactions in the model (right). R is the number of direct responses to the suspected trolling comment.
Table 1: Prediction Results for the four aspects of trolling: Intention, Intentions Disclosure, Interpretation, and Response strategy. Three models are evaluated: a logistic regression classifier: Baseline, a four-tasks CRF: Full Joint, and a two steps process: three-tasks CRF followed by the Response Strategy prediction tasks given the the outcome of the CRF

Questions about this paper:
how was annotation done?

Answers:
Annotation was done with the help of annotators from Amazon Mechanical Turk on snippets of conversations