Introduction
There is an increasing need to interpret and act upon rumours spreading quickly through social media during breaking news, where new reports are released piecemeal and often have an unverified status at the time of posting. Previous research has posited the damage that the diffusion of false rumours can cause in society, and that corrections issued by news organisations or state agencies such as the police may not necessarily achieve the desired effect sufficiently quickly BIBREF0 , BIBREF1 . Being able to determine the accuracy of reports is therefore crucial in these scenarios. However, the veracity of rumours in circulation is usually hard to establish BIBREF2 , since as many views and testimonies as possible need to be assembled and examined in order to reach a final judgement. Examples of rumours that were later disproven, after being widely circulated, include a 2010 earthquake in Chile, where rumours of a volcano eruption and a tsunami warning in Valparaiso spawned on Twitter BIBREF3 . Another example is the England riots in 2011, where false rumours claimed that rioters were going to attack Birmingham's Children's Hospital and that animals had escaped from London Zoo BIBREF4 .
Previous work by ourselves and others has argued that looking at how users in social media orient to rumours is a crucial first step towards making an informed judgement on the veracity of a rumourous report BIBREF5 , BIBREF6 , BIBREF3 . For example, in the case of the riots in England in August 2011, Procter et al. manually analysed the stance expressed by users in social media towards rumours BIBREF4 . Each tweet discussing a rumour was manually categorised as supporting, denying or questioning it. It is obvious that manual methods have their disadvantages in that they do not scale well; the ability to perform stance categorisation of tweets in an automated way would be of great use in tracking rumours, flagging those that are largely denied or questioned as being more likely to be false.
Determining the stance of social media posts automatically has been attracting increasing interest in the scientific community in recent years, as this is a useful first step towards more in-depth rumour analysis:
Work on automatic rumour stance classification, however, is still in its infancy, with some methods ignoring temporal ordering and rumour identities (e.g. BIBREF10 ), while others being rule-based and thus with unclear generalisability to new rumours BIBREF7 .
Our work advances the state-of-the-art in tweet-level stance classification through multi-task learning and Gaussian Processes. This article substantially extends our earlier short paper BIBREF11 , fistly by using a second dataset, which enables us to test the generalisability of our results. Secondly, a comparison against additional baseline classifiers and recent state-of-the-art approaches has been added to the experimental section. Lastly, we carried out a more thorough analysis of the results, including now per-class performance scores, which furthers our understanding of rumour stance classification.
In comparison to the state-of-the-art, our approach is novel in several crucial aspects:
Based on the assumption of a common underlying linguistic signal in rumours on different topics, we build a transfer learning system based on Gaussian Processes, that can classify stance in newly emerging rumours. The paper reports results on two different rumour datasets and explores two different experimental settings – without any training data and with very limited training data. We refer to these as:
Our results demonstrate that Gaussian Process-based, multi-task learning leads to significantly improved performance over state-of-the-art methods and competitive baselines, as demonstrated on two very different datasets. The classifier relying on Gaussian Processes performs particularly well over the rest of the baseline classifiers in the Leave Part Out setting, proving that it does particularly well in determining the distribution of supporting, denying and questioning tweets associated with a rumour. Estimating the distribution of stances is the key aspect for which our classifier performs especially well compared to the baseline classifiers.
Related Work
This section provides a more in-depth motivation of the rumour stance detection task and an overview of the state-of-the-art methods and their limitations. First, however, let us start by introducing the formal definition of a rumour.
Rumour Definition
There have been multiple attempts at defining rumours in the literature. Most of them are complementary to one another, with slight variations depending on the context of their analyses. The core concept that most researchers agree on matches the definition that major dictionaries provide, such as the Oxford English Dictionary defining a rumour as “a currently circulating story or report of uncertain or doubtful truth”. For instance, DiFonzo and Bordia BIBREF12 defined rumours as “unverified and instrumentally relevant information statements in circulation.”
Researchers have long looked at the properties of rumours to understand their diffusion patterns and to distinguish them from other kinds of information that people habitually share BIBREF13 . Allport and Postman BIBREF2 claimed that rumours spread due to two factors: people want to find meaning in things and, when faced with ambiguity, people try to find meaning by telling stories. The latter factor also explains why rumours tend to change in time by becoming shorter, sharper and more coherent. This is the case, it is argued, because in this way rumours explain things more clearly. On the other hand, Rosnow BIBREF14 claimed that there are four important factors for rumour transmission. Rumours must be outcome-relevant to the listener, must increase personal anxiety, be somewhat credible and be uncertain. Furthermore, Shibutani BIBREF15 defined rumours to be “a recurrent form of communication through which men [sic] caught together in an ambiguous situation attempt to construct a meaningful interpretation of it by pooling their intellectual resources. It might be regarded as a form of collective problem-solving”.
In contrast with these three theories, Guerin and Miyazaki BIBREF16 state that a rumour is a form of relationship-enhancing talk. Building on their previous work, they recall that many ways of talking serve the purpose of forming and maintaining social relationships. Rumours, they say, can be explained by such means.
In our work, we adhere to the widely accepted fact that rumours are unverified pieces of information. More specifically, following BIBREF5 , we regard a rumour in the context of breaking news, as a “circulating story of questionable veracity, which is apparently credible but hard to verify, and produces sufficient skepticism and/or anxiety so as to motivate finding out the actual truth”.
Descriptive Analysis of Rumours in Social Media
One particularly influential piece of work in the field of rumour analysis in social media is that by Mendoza et al. BIBREF3 . By manually analysing the data from the earthquake in Chile in 2010, the authors selected 7 confirmed truths and 7 false rumours, each consisting of close to 1000 tweets or more. The veracity value of the selected stories was corroborated by using reliable sources. Each tweet from each of the news items was manually classified into one of the following classes: affirmation, denial, questioning, unknown or unrelated. In this way, each tweet was classified according to the position it showed towards the topic it was about. The study showed that a much higher percentage of tweets about false rumours are shown to deny the respective rumours (approximately 50%). This is in contrast to rumours later proven to be true, where only 0.3% of tweets were denials. Based on this, authors claimed that rumours can be detected using aggregate analysis of the stance expressed in tweets.
Recent research put together in a special issue on rumours and social media BIBREF17 also shows the increasing interest of the scientific community in the topic. BIBREF18 proposed an agenda for research that establishes an interdisciplinary methodology to explore in full the propagation and regulation of unverified content on social media. BIBREF19 described an approach for geoparsing social media posts in real-time, which can be of help to determine the veracity of rumours by tracking down the poster's location. The contribution of BIBREF20 to rumour resolution is to build an automated system that rates the level of trust of users in social media, hence enabling to get rid of users with low reputation. Complementary to these approaches, our objective is to determine the stance of tweets towards a rumour, which can then be aggregated to establish an overall veracity score for the rumour.
Another study that shows insightful conclusions with respect to stance towards rumours is that by Procter et al. BIBREF4 . The authors conducted an analysis of a large dataset of tweets related to the riots in the UK, which took place in August 2011. The dataset collected in the riots study is one of the two used in our experiments, and we describe it in more detail in section "Datasets" . After grouping the tweets into topics, where each represents a rumour, they were manually categorised into different classes, namely:
media reports, which are tweets sent by mainstream media accounts or journalists connected to media,
pictures, being tweets uploading a link to images,
rumours, being tweets claiming or counter claiming something without giving any source,
reactions, consisting of tweets being responses of users to the riots phenomenon or specific event related to the riots.
Besides categorisation of tweets by type, Procter et al. also manually categorised the accounts posting tweets into different types, such as mainstream media, only on-line media, activists, celebrities, bots, among others.
What is interesting for the purposes of our work is that the authors observed the following four-step pattern recurrently occurring across the collected rumours:
a rumour is initiated by someone claiming it may be true,
a rumour spreads together with its reformulations,
counter claims appear,
a consensus emerges about the credibility of the rumour.
This leads the authors to the conclusion that the process of 'inter-subjective sense making' by Twitter users plays a key role in exposing false rumours. This finding, together with subsequent work by Tolmie et al. into the conversational characteristics of microblogging BIBREF6 has motivated our research into automating stance classification as a methodology for accelerating this process.
Qazvinian et al. BIBREF10 conducted early work on rumour stance classification. They introduced a system that analyzes a set of tweets associated with a given topic predefined by the user. Their system would then classify each of the tweets as supporting, denying or questioning a tweet. We have adopted this scheme in terms of the different types of stance in the work we report here. However, their work ended up merging denying and questioning tweets for each rumour into a single class, converting it into a 2-way classification problem of supporting vs denying-or-questioning. Instead, we keep those classes separate and, following Procter et al., we conduct a 3-way classification BIBREF21 .
Another important characteristic that differentiates Qazvinian et al.'s work from ours is that they looked at support and denial on longstanding rumours, such as the fact that many people conjecture whether Barack Obama is a Muslim or not. By contrast, we look at rumours that emerge in the context of fast-paced, breaking news situations, where new information is released piecemeal, often with statements that employ hedging words such as “reportedly” or “according to sources” to make it clear that the information is not fully verified at the time of posting. This is a very different scenario from that in Qazvinian et al.'s work as the emergence of rumourous reports can lead to sudden changes in vocabulary, leading to situations that might not have been observed in the training data.
Another aspect that we deal with differently in our work, aiming to make it more realistically applicable to a real world scenario, is that we apply the method to each rumour separately. Ultimately, our goal is to classify new, emerging rumours, which can differ from what the classifier has observed in the training set. Previous work ignored this separation of rumours, by pooling together tweets from all the rumours in their collections, both in training and test data. By contrast, we consider the rumour stance classification problem as a form of transfer learning and seek to classify unseen rumours by training the classifier from previously labelled rumours. We argue that this makes a more realistic classification scenario towards implementing a real-world rumour-tracking system.
Following a short gap, there has been a burst of renewed interest in this task since 2015. For example, Liu et al. BIBREF9 introduce rule-based methods for stance classification, which were shown to outperform the approach by BIBREF10 . Similarly, BIBREF7 use regular expressions instead of an automated method for rumour stance classification. Hamidian and Diab BIBREF22 use Tweet Latent Vectors to assess the ability of performing 2-way classification of the stance of tweets as either supporting or denying a rumour. They study the extent to which a model trained on historical tweets can be used for classifying new tweets on the same rumour. This, however, limits the method's applicability to long-running rumours only.
The work closest to ours in terms of aims is Zeng et al. BIBREF23 , who explored the use of three different classifiers for automated rumour stance classification on unseen rumours. In their case, classifiers were set up on a 2-way classification problem dealing with tweets that support or deny rumours. In the present work, we extend this research by performing 3-way classification that also deals with tweets that question the rumours. Moreover, we adopt the three classifiers used in their work, namely Random Forest, Naive Bayes and Logistic Regression, as baselines in our work.
Lastly, researchers BIBREF7 , BIBREF24 have focused on the related task of detecting rumours in social media. While a rumour detection system could well be the step that is applied prior to our stance classification system, here we assume that rumours have already been identified to focus on the subsequent step of determining stances.
Individual tweets may discuss the same rumour in different ways, where each user expresses their own stance towards the rumour. Within this scenario, we define the tweet level rumour stance classification task as that in which a classifier has to determine the stance of each tweet towards the rumour. More specifically, given the tweet $t_i$ as input, the classifier has to determine which of the set $Y = \lbrace supporting, denying, questioning\rbrace $ applies to the tweet, $y(t_i) \in Y$ .
Here we define the task as a supervised classification problem, where the classifier is trained from a labelled set of tweets and is applied to tweets on a new, unseen set of rumours.
Let $R$ be a set of rumours, each of which consists of tweets discussing it, $\forall _{r \in R}$ $T_r$ $= \lbrace t^r_1, \cdots , t^r_{r_n}\rbrace $ . $T = \cup _{r \in R} T_r$ is the complete set of tweets from all rumours. Each tweet is classified as supporting, denying or questioning with respect to its rumour: $y(t_i) \in \lbrace s, d, q\rbrace $ .
We formulate the problem in two different settings. First, we consider the Leave One Out (LOO) setting, which means that for each rumour $r \in R$ , we construct the test set equal to $T_r$ and the training set equal to $T \setminus T_r$ . This is the most challenging scenario, where the test set contains an entirely unseen rumour.
The second setting is Leave Part Out (LPO). In this formulation, a very small number of initial tweets from the target rumour is added to the training set $\lbrace t^r_1, \cdots , t^r_{{{r_k}}}\rbrace $ . This scenario becomes applicable typically soon after a rumour breaks out and journalists have started monitoring and analysing the related tweet stream. The experimental section investigates how the number of initial training tweets influences classification performance on a fixed test set, namely: $\lbrace t^r_{{{r_l}}{}}, \cdots , t^r_{r_n}\rbrace $ , $l>k$ .
The tweet-level stance classification problem here assumes that tweets from the training set are already labelled with the rumour discussed and the attitude expressed towards that. This information can be acquired either via manual annotation as part of expert analysis, as is the case with our dataset, or automatically, e.g. using pattern-based rumour detection BIBREF7 . Our method is then used to classify the stance expressed in each new tweet from the test set.
We evaluate our work on two different datasets, which we describe below. We use two recent datasets from previous work for our study, both of which adapt to our needs. We do not use the dataset by BIBREF10 given that it uses a different annotation scheme limited to two categories of stances.
The reason why we use the two datasets separately instead of combining them is that they have very different characteristics. Our experiments, instead, enable us to assess the ability of our classifier to deal with these different characteristics.
The first dataset consists of several rumours circulating on Twitter during the England riots in 2011 (see Table 2 ). The dataset was collected by tracking a long set of keywords associated with the event. The dataset was analysed and annotated manually as supporting, questioning, or denying a rumour, by a team of social scientists studying the role of social media during the riots BIBREF4 .
As can be seen from the dataset overview in Table 2 , different rumours exhibit varying proportions of supporting, denying and questioning tweets, which was also observed in other studies of rumours BIBREF3 , BIBREF10 . These variations in the number of instances for each class across rumours posits the challenge of properly modelling a rumour stance classifier. The classifier needs to be able to deal with a test set where the distribution of classes can be very different to that observed in the training set.
Thus, we perform 7-fold cross-validation in the experiments, each fold having six rumours in the training set, and the remaining rumour in the test set.
The seven rumours were as follows BIBREF4 :
Rioters had attacked London Zoo and released the animals.
Rioters were gathering to attack Birmingham's Children's Hospital.
Rioters had set the London Eye on fire.
Police had beaten a sixteen year old girl.
The Army was being mobilised in London to deal with the rioters.
Rioters had broken into a McDonalds and set about cooking their own food.
A store belonging to the Miss Selfridge retail group had been set on fire in Manchester.
Additionally, we use another rumour dataset associated with five different events, which was collected as part of the PHEME FP7 research project and described in detail in BIBREF5 , BIBREF25 . Note that the authors released datasets for nine events, but here we remove non-English datasets, as well as small English datasets each of which includes only 1 rumour, as opposed to the 40+ rumours in each of the datasets that we are using. We summarise the details of the five events we use from this dataset in Table 3 .
In contrast to the England riots dataset, the PHEME datasets were collected by tracking conversations initiated by rumourous tweets. This was done in two steps. First, we collected tweets that contained a set of keywords associated with a story unfolding in the news. We will be referring to the latter as an event. Next, we sampled the most retweeted tweets, on the basis that rumours by definition should be “a circulation story which produces sufficient skepticism or anxiety”. This allows us to filter potentially rumourous tweets and collect conversations initiated by those. Conversations were tracked by collecting replies to tweets and, therefore, unlike the England riots, this dataset also comprises replying tweets by definition. This is an important characteristic of the dataset, as one would expect that replies are generally shorter and potentially less descriptive than the source tweets that initiated the conversation. We take this difference into consideration when performing the analysis of our results.
This dataset includes tweets associated with the following five events:
Ferguson unrest: Citizens of Ferguson in Michigan, USA, protested after the fatal shooting of an 18-year-old African American, Michael Brown, by a white police officer on August 9, 2014.
Ottawa shooting: Shootings occurred on Ottawa's Parliament Hill in Canada, resulting in the death of a Canadian soldier on October 22, 2014.
Sydney siege: A gunman held as hostages ten customers and eight employees of a Lindt chocolate café located at Martin Place in Sydney, Australia, on December 15, 2014.
Charlie Hebdo shooting: Two brothers forced their way into the offices of the French satirical weekly newspaper Charlie Hebdo in Paris, killing 11 people and wounding 11 more, on January 7, 2015.
Germanwings plane crash: A passenger plane from Barcelona to Düsseldorf crashed in the French Alps on March 24, 2015, killing all passengers and crew on board. The plane was ultimately found to have been deliberately crashed by the co-pilot of the plane.
In this case, we perform 5-fold cross-validation, having four events in the training set and the remaining event in the test set for each fold.
This section details the features and evaluation measures used in our experiments on tweet level stance classification.
We begin by describing the classifiers we use for our experimentation, including Gaussian Processes, as well as a set of competitive baseline classifiers that we use for comparison.
Gaussian Processes are a Bayesian non-parametric machine learning framework that has been shown to work well for a range of NLP problems, often beating other state-of-the-art methods BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 .
A Gaussian Process defines a prior over functions, which combined with the likelihood of data points gives rise to a posterior over functions explaining the data. The key concept is a kernel function, which specifies how outputs correlate as a function of the input. Thus, from a practitioner's point of view, a key step is to choose an appropriate kernel function capturing the similarities between inputs.
We use Gaussian Processes as this probabilistic kernelised framework avoids the need for expensive cross-validation for hyperparameter selection. Instead, the marginal likelihood of the data can be used for hyperparameter selection.
The central concept of Gaussian Process Classification (GPC; BIBREF30 ) is a latent function $f$ over inputs $\mathbf {x}$ : $f(\mathbf {x}) \sim \ \mathcal {GP}(m(\mathbf {x}), k(\mathbf {x}, \mathbf {x}^{\prime }))$ , where $m$ is the mean function, assumed to be 0 and $k$ is the kernel function, specifying the degree to which the outputs covary as a function of the inputs. We use a linear kernel, $k(\mathbf {x}, \mathbf {x}^{\prime }) = \sigma ^2 \mathbf {x}^{\top }\mathbf {x}^{\prime }$ . The latent function is then mapped by the probit function $\Phi (f)$ into the range $[0, 1]$ , such that the resulting value can be interpreted as $p(y=1 | \mathbf {x})$ .
The GPC posterior is calculated as $ p(f^* | X, \mathbf {y}, \mathbf {x_*}) = \int p(f^* | X, \mathbf {x_*}, \mathbf {f}) \frac{p(\mathbf {y} | \mathbf {f})p(\mathbf {f})}{p(\mathbf {y}|X)} d\mathbf {f} \, \!, $
where $p(\mathbf {y}|\mathbf {f}) = \displaystyle \prod _{j=1}^{n} \Phi (f_j)^{y_j} (1 - \Phi (f_j))^{1-y_j}$ is the Bernoulli likelihood of class $y$ . After calculating the above posterior from the training data, this is used in prediction, i.e., $ p(y_* \!=\! 1|X, \mathbf {y}, \mathbf {x_*}) \!=\!\! \int \Phi \left(f_*\right)p\left(f_*|X, \mathbf {y}, \mathbf {x_*}\right)df_* \, . $
The above integrals are intractable and approximation techniques are required to solve them. There exist various methods to deal with calculating the posterior; here we use Expectation Propagation (EP; BIBREF31 ). In EP, the posterior is approximated by a fully factorised distribution, where each component is assumed to be an unnormalised Gaussian.
In order to conduct multi-class classification, we perform a one-vs-all classification for each label and then assign the one with the highest likelihood, amongst the three (supporting, denying, questioning). We choose this method due to interpretability of results, similar to recent work on occupational class classification BIBREF29 .
In the Leave-Part-Out (LPO) setting initial labelled tweets from the target rumour are observed as well, as opposed to the Leave-One-Out (LOO) setting. In the case of LPO, we propose to weigh the importance of tweets from the reference rumours depending on how similar their characteristics are to the tweets from the target rumour available for training. To handle this with GPC, we use a multiple output model based on the Intrinsic Coregionalisation Model (ICM; BIBREF32 ). This model has already been applied successfully to NLP regression problems BIBREF28 and it can also be applied to classification ones. ICM parametrizes the kernel by a matrix which represents the extent of covariance between pairs of tasks. The complete kernel takes form of $ k((\mathbf {x}, d), (\mathbf {x}^{\prime }, d^{\prime })) = k_{data}(\mathbf {x}, \mathbf {x}^{\prime }) B_{d, d^{\prime }} \, , $
where B is a square coregionalisation matrix, $d$ and $d^{\prime }$ denote the tasks of the two inputs and $k_{data}$ is a kernel for comparing inputs $\mathbf {x}$ and $\mathbf {x}^{\prime }$ (here, linear). We parametrize the coregionalisation matrix $B=\kappa  I+vv^T$ , where $v$ specifies the correlation between tasks and the vector $\mathbf {\kappa }$ controls the extent of task independence. Note that in case of LOO setting this model does not provide useful information, since no target rumour data is available to estimate similarity to other rumours.
We tune hyperparameters $\mathbf {v}$ , $\kappa $ and $\sigma ^2$ by maximizing evidence of the model $p(\mathbf {y}|X)$ , thus having no need for a validation set.
We consider GPs in three different settings, varying in what data the model is trained on and what kernel it uses. The first setting (denoted GP) considers only target rumour data for training. The second (GPPooled) additionally considers tweets from reference rumours (i.e. other than the target rumour). The third setting is GPICM, where an ICM kernel is used to weight influence from tweets from reference rumours.
To assess and compare the efficiency of Gaussian Processes for rumour stance classification, we also experimented with five more baseline classifiers, all of which were implemented using the scikit Python package BIBREF33 : (1) majority classifier, which is a naive classifier that labels all the instances in the test set with the most common class in the training set, (2) logistic regression (MaxEnt), (3) support vector machines (SVM), (4) naive bayes (NB) and (5) random forest (RF). The selection of these baselines is in line with the classifiers used in recent research on stance classification BIBREF23 , who found that random forests, followed by logistic regression, performed best.
We conducted a series of preprocessing steps in order to address data sparsity. All words were converted to lowercase; stopwords have been removed; all emoticons were replaced by words; and stemming was performed. In addition, multiple occurrences of a character were replaced with a double occurrence BIBREF34 , to correct for misspellings and lengthenings, e.g., looool. All punctuation was also removed, except for ., ! and ?, which we hypothesize to be important for expressing emotion. Lastly, usernames were removed as they tend to be rumour-specific, i.e., very few users comment on more than one rumour. After preprocessing the text data, we use either the resulting bag of words (BOW) feature representation and replace all words with their Brown cluster ids (Brown). Brown clustering is a hard hierarchical clustering method BIBREF35 . It clusters words based on maximizing the probability of the words under the bigram language model, where words are generated based on their clusters. In previous work it has been shown that Brown clusters yield better performance than directly using the BOW features BIBREF11 .
In our experiments, the clusters used were obtained using 1000 clusters acquired from a large scale Twitter corpus BIBREF36 , from which we can learn Brown clusters aimed at representing a generalisable Twitter vocabulary. Retweets are removed from the training set to prevent bias BIBREF37 . More details on the Brown clusters that we used as well as the words that are part of each cluster are available online.
During the experimentation process, we also tested additional features, including the use of the bag of words instead of the Brown clusters, as well as using word embeddings trained from the training sets BIBREF38 . However, results turned out to be substantially poorer than those we obtained with the Brown clusters. We conjecture that this was due to the little data available to train the word embeddings; further exploring use of word embeddings trained from larger training datasets is left future work. In order to focus on our main objective of proving the effectiveness of a multi-task learning approach, as well as for clarity purposes, since the number of approaches to show in the figures increases if we also consider the BOW features, we only show results for the classifiers relying on Brown clusters as features.
Accuracy is often deemed a suitable evaluation measure to assess the performance of a classifier on a multi-class classification task. However, the classes are clearly imbalanced in our case, with varying tendencies towards one of the classes in each of the rumours. We argue that in these scenarios the sole evaluation based on accuracy is insufficient, and further measurement is needed to account for category imbalance. This is especially necessary in our case, as a classifier that always predicts the majority class in an imbalanced dataset will achieve high accuracy, even if the classifier is useless in practice. To tackle this, we use both micro-averaged and macro-averaged F1 scores. Note that the micro-averaged F1 score is equivalent to the well-known accuracy measure, while the macro-averaged F1 score complements it by measuring performance assigning the same weight to each category.
Both of the measures rely on precision (Equation 50 ) and recall (Equation 51 ) to compute the final F1 score.
$$\text{Precision}_k = \frac{tp_k}{tp_k+fp_k}$$   (Eq. 50)
$$\text{Recall}_k = \frac{tp_k}{tp_k+fn_k}$$   (Eq. 51)
where $tp_k$ (true positives) refer to the number of instances correctly classified in class $k$ , $fp_k$ is the number of instances incorrectly classified in class $k$ , and $fn_k$ is the number of instances that actually belong to class $k$ but were not classified as such.
The above equations can be used to compute precision and recall for a specific class. Precision and recall for all the classes in a problem with $c$ classes are computed differently if they are microaveraged (see Equations 52 and 53 ) or macroaveraged (see Equations 54 and 55 ).
$$\text{Precision}_{\text{micro}} = \frac{\sum _{k = 1}^{c} tp_k}{\sum _{k = 1}^{c} tp_k + \sum _{k = 1}^{c} fp_k}$$   (Eq. 52)
$$\text{Recall}_{\text{micro}} = \frac{\sum _{k = 1}^{c} tp_k}{\sum _{k = 1}^{c} tp_k + \sum _{k = 1}^{c} fn_k}$$   (Eq. 53)
After computing microaveraged and macroveraged precision and recall, the final F1 score is computed in the same way, i.e., calculating the harmonic mean of the precision and recall in question (see Equation 56 ).
$$\text{F1} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$   (Eq. 56)
After computing the F1 score for each fold, we compute the micro-averaged score across folds.
First, we look at the results on each dataset separately. Then we complement the analysis by aggregating the results from both datasets, which leads to further understanding the performance of our classifiers on rumour stance classification.
We show the results for the LOO and LPO settings in the same figure, distinguished by the training size displayed in the X axis. In all the cases, labelled tweets from the remainder of the rumours (rumours other than the test/targer rumour) are used for training, and hence the training size shown in the X axis is in addition to those. Note that the training size refers to the number of labelled instances that the classifier is making use of from the target rumour. Thus, a training size of 0 indicates the LOO setting, while training sizes from 10 to 50 pertain to the LPO setting.
Figure 1 and Table 4 show how micro-averaged and macro-averaged F1 scores for the England riots dataset change as the number of tweets from the target rumour used for training increases. We observe that, as initially expected, the performance of most of the methods improves as the number of labelled training instances from the target rumour increases. This increase is especially remarkable with the GP-ICM method, which gradually increases after having as few as 10 training instances. GP-ICM's performance keeps improving as the number of training instances approaches 50 Two aspects stand out from analysing GP-ICM's performance:
It performs poorly in terms of micro-averaged F1 when no labelled instances from the target rumour are used. However, it makes very effective use of the labelled training instances, overtaking the rest of the approaches and achieving the best results. This proves the ability of GP-ICM to make the most of the labelled instances from the target rumour, which the rest of the approaches struggle with.
Irrespective of the number of labelled instances, GP-ICM is robust when evaluated in terms of macro-averaged F1. This means that GP-ICM is managing to determine the distribution of classes effectively, assigning labels to instances in the test set in a way that is better distributed than the rest of the classifier.
Despite the saliency of GP-ICM, we notice that two other baseline approaches, namely MaxEnt and RF, achieve competitive results that are above the rest of the baselines, but still perform worse than GP-ICM.
The results from the PHEME dataset are shown in Figure 2 and Table 5 . Overall, we can observe that results are lower in this case than they were for the riots dataset. The reason for this can be attributed to the following two observations: on the one hand, each fold pertaining to a different event in the PHEME dataset means that the classifier encounters a new event in the classification, where it will likely find new vocabulary, which may be more difficult to classify; on the other hand, the PHEME dataset is more prominently composed of tweets that are replying to others, which are likely shorter and less descriptive on their own and hence more difficult to get meaningful features from. Despite the additional difficulty in this dataset, we are interested in exploring if the same trend holds across classifiers, from which we can generalise the analysis to different types of classifiers.
One striking difference with respect to the results from the riots dataset is that, in this case, the classifiers, including GP-ICM, are not gaining as much from the inclusion of labelled instances from the target rumour. This is likely due to the heterogeneity of each of the events in the PHEME dataset. Here a diverse set of rumourous newsworthy pieces of information are discussed pertaining to the selected events as they unfold. By contrast, each rumour in the riots dataset is more homogeneous, as each rumour focuses on a specific story.
Interestingly, when we compare the performance of different classifiers, we observe that GP-ICM again outperforms the rest of the approaches, both in terms of micro-averaged and macro-averaged F1 scores. While the micro-averaged F1 score does not increase as the number of training instances increases, we can see a slight improvement in terms of macro-averaged F1. This improvement suggests that GP-ICM does still take advantage of the labelled training instances to boost performance, in this case by better distributing the predicted labels.
Again, as we observed in the case of the riots dataset, two baselines stand out, MaxEnt and RF. They are very close to the performance of GP-ICM for the PHEME dataset, event outperforming it in a few occasions. In the following subsection we take a closer look at the differences among the three classifiers.
We delve into the results of the best-performing classifiers, namely GP-ICM, MaxEnt and RF, looking at their per-class performance. This will help us understand when they perform well and where it is that GP-ICM stands out achieving the best results.
Tables 6 and 7 show per-class F1 measures for the aforementioned three best-performing classifiers for the England riots dataset and the PHEME dataset, respectively. They also show statistics of the mis-classifications that the classifiers made, in the form of percentage of deviations towards the other classes.
Looking at the per-class performance analysis, we observe that the performance of GP-ICM varies when we look into Precision and Recall. Still, in all the dataset-class pairs, GP-ICM performs best in terms of either Precision or Recall, even though never in both. Moreover, it is generally the best in terms of F1, achieving the best Precision and Recall. The only exception is with MaxEnt classifying questioning tweets more accurately in terms of F1 for the England riots.
When we look at the deviations, we see that all the classifiers suffer from the datasets being imbalanced towards supporting tweets. This results in all classifiers classifying numerous instances as supporting, while they are actually denying or questioning. This is a known problem in rumour diffusion, as previous studies have found that people barely deny or question rumours but generally tend to support them irrespective of their actual veracity value BIBREF5 . While we have found that GP-ICM can tackle the imbalance issue quite effectively and better than other classifiers, this caveat posits the need for further research in dealing with the striking majority of supporting tweets in the context of rumours in social media.
Experimentation with two different approaches based on Gaussian Processes (GP and GP-ICM) and comparison with respect to a set of competitive baselines over two rumour datasets enables us to gain generalisable insight on rumour stance classification on Twitter. This is reinforced by the fact that the two datasets are very different from each other. The first dataset, collected during the England riots in 2011, is a single event that we have split into folds, each fold belonging to a separate rumour within the event; hence, all the rumours are part of the same event. The second dataset, collected within the PHEME project, includes tweets for a set of five newsworthy events, where each event has been assigned a separate fold; therefore, the classifier needs to learn from four events and test on a new, unknown event, which has proven more challenging.
Results are generally consistent across datasets, which enables us to generalise conclusions well. We observe that while GP itself does not suffice to achieve competitive results, GP-ICM does instead help boost the performance of the classifier substantially to even outperform the rest of the baselines in the majority of the cases.
GP-ICM has proven to consistently perform well in both datasets, despite their very different characteristics, being competitive not only in terms of micro-averaged F1, but also in terms of macro-averaged F1. GP-ICM manages to balance the varying class distributions effectively, showing that its performance is above the rest of the baselines in accurately determining the distribution of classes. This is very important in this task of rumour stance classification, owing to the fact that even if a classifier that is 100% accurate is unlikely, a classifier that accurately guesses the overall distribution of classes can be of great help. If a classifier makes a good estimation of the number of denials in an aggregated set of tweets, it can be useful to flag those potentially false rumours with high level of confidence.
Another factor that stands out from GP-ICM is its capacity to perform well when a few labelled instances of the target rumour are leveraged in the training phase. GP-ICM effectively exploits the knowledge garnered from the few instances from the target rumour, outperforming the rest of the baselines even when its performance was modest when no labelled instances were used from the target rumour.
In light of these results, we deem GP-ICM the most competitive approach to use when one can afford to get a few instances labelled from the target rumour. The labels from the target rumour can be obtained in practice in different ways: (1) having someone in-house (e.g. journalists monitoring breaking news stories) label a few instances prior to running the classifier, (2) making use of resources for human computation such as crowdsourcing platforms to outsource the labelling work, or (3) developing techniques that will attempt to classify the first few instances, incorporating in the training set those for which a classification with high level of confidence has been produced. The latter presents an ambitious avenue for future work that could help alleviate the labelling task.
On the other hand, in the absence of labelled data from the target rumour, which is the case of the LOO setting, the effectiveness of the GP-ICM classifier is not as prominent. For this scenario, other classifiers such as MaxEnt and Random Forests have proven more competitive and one could see them as better options. However, we do believe that the remarkable difference that the reliance on the LPO setting produces is worth exploiting where possible.
Social media is becoming an increasingly important tool for maintaining social resilience: individuals use it to express opinions and follow events as they unfold; news media organisations use it as a source to inform their coverage of these events; and government agencies, such as the emergency services, use it to gather intelligence to help in decision-making and in advising the public about how they should respond BIBREF1 . While previous research has suggested that mechanisms for exposing false rumours are implicit in the ways in which people use social media BIBREF4 , it is nevertheless critically important to explore if there are ways in which computational tools can help to accelerate these mechanisms so that misinformation and disinformation can be targeted more rapidly, and the benefits of social media to society maintained BIBREF8 .
As a first step to achieving this aim, this paper has investigated the problem of classifying the different types of stance expressed by individuals in tweets about rumours. First, we considered a setting where no training data from the target rumours is available (LOO). Without access to annotated examples of the target rumour the learning problem becomes very difficult. We showed that in the supervised domain adaptation setting (LPO), even annotating a small number of tweets helps to achieve better results. Moreover, we demonstrated the benefits of a multi-task learning approach, as well as that Brown cluster features are more useful for the task than simple bag of words.
Findings from previous work, such as BIBREF39 , BIBREF4 , have suggested that the aggregate stance of individual users is correlated with actual rumour veracity. Hence, the next step in our own work will be to make use of the classifier for the stance expressed in the reactions of individual Twitter users in order to predict the actual veracity of the rumour in question. Another interesting direction for future work would be the addition of non-textual features to the classifier. For example, the rumour diffusion patterns BIBREF40 may be a useful cue for stance classification.
This work is partially supported by the European Union under grant agreement No. 611233 Pheme. The work was implemented using the GPy toolkit BIBREF41 . This research utilised Queen Mary's MidPlus computational facilities, supported by QMUL Research-IT and funded by EPSRC grant EP/K000128/1.

Question: Why is a Gaussian process an especially appropriate method for this classification problem?
Answer: avoids the need for expensive cross-validation for hyperparameter selection