prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which languages are similar to each other?
Context 1: Introduction
Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines  If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required 
LID is further also an important step in harvesting scarce language resources  Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data  Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries 
Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research  However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation 
Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa  These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso)  The Nguni languages are similar to each other and harder to distinguish  The same is true of the Sotho languages 
This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long  The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks 
Section SECREF2 reviews existing works on the topic and summarises the remaining research problems  Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results 
Related Works
The focus of this section is on recently published datasets and LID research applicable to the South African context  An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0 
The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle   The DSL datasets, like other LID datasets, consists of text sentences labelled by language  The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language 
The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average  In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3  The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages  A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets  The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data 
The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8  The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language  Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9 
Existing NLP datasets, models and services BIBREF10 are available for South African languages  These include an LID algorithm BIBREF11 that uses a character level n-gram language model  Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID  The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features  The winning approach for DSL 2015 used an ensemble naive Bayes classifier  The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID  
Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0  Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20  Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data 
Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24  The latter is reported to achieve 95 12% in the DSL 2015 shared task  In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data  The neural methods seem to work well in tasks where more training data is available 
In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related  Shallow methods still seem to work well compared to deeper models for LID  Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa  Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out  Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers 
Methodology
The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26  We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier 
The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda  The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0 01 and hashed text features 
The lexicon based classifier is then used to predict the specific language within a language group  For the South African languages this is done for the Nguni and Sotho groups  If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result  The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets 
The lexicon based classifier is designed to trade higher precision for lower recall  The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place 
The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8  The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog  Character (5-6)-gram features with 16 dimensional vectors worked the best  The implementation discussed in BIBREF23 is available from https://github com/tomkocmi/LanideNN  Following the instructions for an OSX pip install of an old r0 8 release of TensorFlow, the LanideNN code could be executed in Python 3 7 4  Settings were left at their defaults and a learning rate of 0 001 was used followed by a refinement with learning rate of 0 0001  Only one code modification was applied to return the results from a method that previously just printed to screen  The LID algorithm described in BIBREF8 is also available on GitHub 
The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15  All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2 
Results and Analysis
The average classification accuracy results are summarised in Table TABREF9  The accuracies reported are for classifying a piece of text by its specific language label  Classifying text only by language group or family is a much easier task as reported in BIBREF8 
Different variations of the proposed classifier were evaluated  A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself  A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy 
From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks  Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected 
The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon  Without a good lexicon a non-stacked naive Bayesian classifier might even perform better 
The execution performance of some of the LID implementations are shown in Table TABREF10  Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2 9 GHz CPU (Turbo Boosted to 3 4 GHz) and 8GB RAM  The C++ implementation in BIBREF17 is the fastest  The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation  The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU 
Conclusion
LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms  It performed well relative to the other methods beating their results  However, the performance is dependent on the support of the lexicon 
We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning  We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks  We would also like to investigate data augmentation strategies to reduce the amount of training data that is required 
Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa  In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research 
Context 3: 10pt
1 10pt
[ Characterizing Political Fake News in Twitter by its Meta-DataJulio Amador Díaz LópezAxel Oehmichen Miguel Molina-Solana( j amador, axelfrancois oehmichen11, mmolinas@imperial ac uk ) Imperial College London This article presents a preliminary approach towards characterizing political fake news on Twitter through the analysis of their meta-data  In particular, we focus on more than 1 5M tweets collected on the day of the election of Donald Trump as 45th president of the United States of America  We use the meta-data embedded within those tweets in order to look for differences between tweets containing fake news and tweets not containing them  Specifically, we perform our analysis only on tweets that went viral, by studying proxies for users' exposure to the tweets, by characterizing accounts spreading fake news, and by looking at their polarization  We found significant differences on the distribution of followers, the number of URLs on tweets, and the verification of the users 
]
Introduction
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e g  it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0   Since then, governments and corporations alike (e g  Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3   Yet, the ability to define, identify and stop fake news from spreading is limited 
Since the Obama campaign in 2008, social media has been pervasive in the political arena in the United States  Studies report that up to 62% of American adults receive their news from social media BIBREF4   The wide use of platforms such as Twitter and Facebook has facilitated the diffusion of fake news by simplifying the process of receiving content with no significant third party filtering, fact-checking or editorial judgement  Such characteristics make these platforms suitable means for sharing news that, disguised as legit ones, try to confuse readers 
Such use and their prominent rise has been confirmed by Craig Silverman, a Canadian journalist who is a prominent figure on fake news BIBREF5 : “In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlet” 
Our current research hence departs from the assumption that social media is a conduit for fake news and asks the question of whether fake news (as spam was some years ago) can be identified, modelled and eventually blocked  In order to do so, we use a sample of more that 1 5M tweets collected on November 8th 2016 —election day in the United States— with the goal of identifying features that tweets containing fake news are likely to have  As such, our paper aims to provide a preliminary characterization of fake news in Twitter by looking into meta-data embedded in tweets  Considering meta-data as a relevant factor of analysis is in line with findings reported by Morris et al  BIBREF6   We argue that understanding differences between tweets containing fake news and regular tweets will allow researchers to design mechanisms to block fake news in Twitter 
Specifically, our goals are: 1) compare the characteristics of tweets labelled as containing fake news to tweets labelled as not containing them, 2) characterize, through their meta-data, viral tweets containing fake news and the accounts from which they originated, and 3) determine the extent to which tweets containing fake news expressed polarized political views 
For our study, we used the number of retweets to single-out those that went viral within our sample  Tweets within that subset (viral tweets hereafter) are varied and relate to different topics  We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al  BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious  The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties 
From our results, the following main observations can be made:
Our findings resonate with similar work done on fake news such as the one from Allcot and Gentzkow BIBREF9   Therefore, even if our study is a preliminary attempt at characterizing fake news on Twitter using only their meta-data, our results provide external validity to previous research  Moreover, our work not only stresses the importance of using meta-data, but also underscores which parameters may be useful to identify fake news on Twitter 
The rest of the paper is organized as follows  The next section briefly discusses where this work is located within the literature on fake news and contextualizes the type of fake news we are studying  Then, we present our hypotheses, the data, and the methodology we follow  Finally, we present our findings, conclusions of this study, and future lines of work 
Defining Fake news
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news  In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news  On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news  On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news 
With respect to social sciences, efforts from psychology, political science and sociology, have been dedicated to understand why people consume and/or believe misinformation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14   Most of these studies consistently reported that psychological biases such as priming effects and confirmation bias play an important role in people ability to discern misinformation 
In relation to the production and distribution of fake news, a recent paper in the field of Economics BIBREF9 found that most fake news sites use names that resemble those of legitimate organizations, and that sites supplying fake news tend to be short-lived

ground_truth: Nguni languages (zul, xho, nbl, ssw), Sotho languages (nso, sot, tsn)

The Nguni languages are similar to each other, The same is true of the Sotho languages

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What types of social media did they consider?
Context 2: Introduction
Explanations of happenings in one's life, causal explanations, are an important topic of study in social, psychological, economic, and behavioral sciences  For example, psychologists have analyzed people's causal explanatory style BIBREF0 and found strong negative relationships with depression, passivity, and hostility, as well as positive relationships with life satisfaction, quality of life, and length of life BIBREF1 , BIBREF2 , BIBREF0  
To help understand the significance of causal explanations, consider how they are applied to measuring optimism (and its converse, pessimism) BIBREF0   For example, in “My parser failed because I always have bugs ”, the emphasized text span is considered a causal explanation which indicates pessimistic personality – a negative event where the author believes the cause is pervasive  However, in “My parser failed because I barely worked on the code ”, the explanation would be considered a signal of optimistic personality – a negative event for which the cause is believed to be short-lived 
Language-based models which can detect causal explanations from everyday social media language can be used for more than automating optimism detection  Language-based assessments would enable other large-scale downstream tasks: tracking prevailing causal beliefs (e g , about climate change or autism), better extracting process knowledge from non-fiction (e g , gravity causes objects to move toward one another), or detecting attribution of blame or praise in product or service reviews (“I loved this restaurant because the fish was cooked to perfection”) 
In this paper, we introduce causal explanation analysis and its subtasks of detecting the presence of causality (causality prediction) and identifying explanatory phrases (causal explanation identification)  There are many challenges to achieving these task  First, the ungrammatical texts in social media incur poor syntactic parsing results which drastically affect the performance of discourse relation parsing pipelines   Many causal relations are implicit and do not contain any discourse markers (e g , `because')  Further, Explicit causal relations are also more difficult in social media due to the abundance of abbreviations and variations of discourse connectives (e g , `cuz' and `bcuz') 
Prevailing approaches for social media analyses, utilizing traditional linear models or bag of words models (e g , SVM trained with n-gram, part-of-speech (POS) tags, or lexicon-based features) alone do not seem appropriate for this task since they simply cannot segment the text into meaningful discourse units or discourse arguments such as clauses or sentences rather than random consecutive token sequences or specific word tokens  Even when the discourse units are clear, parsers may still fail to accurately identify discourse relations since the content of social media is quite different than that of newswire which is typically used for discourse parsing 
In order to overcome these difficulties of discourse relation parsing in social media, we simplify and minimize the use of syntactic parsing results and capture relations between discourse arguments, and investigate the use of a recursive neural network model (RNN)  Recent work has shown that RNNs are effective for utilizing discourse structures for their downstream tasks BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , but they have yet to be directly used for discourse relation prediction in social media  We evaluated our model by comparing it to off-the-shelf end-to-end discourse relation parsers and traditional models  We found that the SVM and random forest classifiers work better than the LSTM classifier for the causality detection, while the LSTM classifier outperforms other models for identifying causal explanation 
The contributions of this work include: (1) the proposal of models for both (a) causality prediction and (b) causal explanation identification, (2) the extensive evaluation of a variety of models from social media classification models and discourse relation parsers to RNN-based application models, demonstrating that feature-based models work best for causality prediction while RNNs are superior for the more difficult task of causal explanation identification, (3) performance analysis on architectural differences of the pipeline and the classifier structures, (4) exploration of the applications of causal explanation to downstream tasks, and (5) release of a novel, anonymized causality Facebook dataset along with our causality prediction and causal explanation identification models 
Related Work
Identifying causal explanations in documents can be viewed as discourse relation parsing  The Penn Discourse Treebank (PDTB) BIBREF7 has a `Cause' and `Pragmatic Cause' discourse type under a general `Contingency' class and Rhetorical Structure Theory (RST) BIBREF8 has a `Relations of Cause'  In most cases, the development of discourse parsers has taken place in-domain, where researchers have used the existing annotations of discourse arguments in newswire text (e g  Wall Street Journal) from the discourse treebank and focused on exploring different features and optimizing various types of models for predicting relations BIBREF9 , BIBREF10 , BIBREF11   In order to further develop automated systems, researchers have proposed end-to-end discourse relation parsers, building models which are trained and evaluated on the annotated PDTB and RST Discourse Treebank (RST DT)  These corpora consist of documents from Wall Street Journal (WSJ) which are much more well-organized and grammatical than social media texts BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15  
Only a few works have attempted to parse discourse relations for out-of-domain problems such as text categorizations on social media texts; Ji and Bhatia used models which are pretrained with RST DT for building discourse structures from movie reviews, and Son adapted the PDTB discourse relation parsing approach for capturing counterfactual conditionals from tweets BIBREF4 , BIBREF3 , BIBREF16   These works had substantial differences to what propose in this paper  First, Ji and Bhatia used a pretrained model (not fully optimal for some parts of the given task) in their pipeline; Ji's model performed worse than the baseline on the categorization of legislative bills, which is thought to be due to legislative discourse structures differing from those of the training set (WSJ corpus)  Bhatia also used a pretrained model finding that utilizing discourse relation features did not boost accuracy BIBREF4 , BIBREF3   Both Bhatia and Son used manual schemes which may limit the coverage of certain types of positive samples– Bhatia used a hand-crafted schema for weighting discourse structures for the neural network model and Son manually developed seven surface forms of counterfactual thinking for the rule-based system BIBREF4 , BIBREF16   We use social-media-specific features from pretrained models which are directly trained on tweets and we avoid any hand-crafted rules except for those included in the existing discourse argument extraction techniques 
The automated systems for discourse relation parsing involve multiple subtasks from segmenting the whole text into discourse arguments to classifying discourse relations between the arguments  Past research has found that different types of models and features yield varying performance for each subtask  Some have optimized models for discourse relation classification (i e  given a document indicating if the relation existing) without discourse argument parsing using models such as Naive-Bayes or SVMs, achieve relatively stronger accuracies but a simpler task than that associated with discourse arguments BIBREF10 , BIBREF11 , BIBREF9   Researchers who, instead, tried to build the end-to-end parsing pipelines considered a wider range of approaches including sequence models and RNNs BIBREF12 , BIBREF15 , BIBREF14 , BIBREF17   Particularly, when they tried to utilize the discourse structures for out-domain applications, they used RNN-based models and found that those models are advantageous for their downstream tasks BIBREF4 , BIBREF3  
In our case, for identifying causal explanations from social media using discourse structure, we build an RNN-based model for its structural effectiveness in this task (see details in section UID13 )  However, we also note that simpler models such as SVMs and logistic regression obtained the state-of-the-art performances for text categorization tasks in social media BIBREF18 , BIBREF19 , so we build relatively simple models with different properties for each stage of the full pipeline of our parser 
Methods
We build our model based on PDTB-style discourse relation parsing since PDTB has a relatively simpler text segmentation method; for explicit discourse relations, it finds the presence of discourse connectives within a document and extracts discourse arguments which parametrize the connective while for implicit relations, it considers all adjacent sentences as candidate discourse arguments 
Dataset
We created our own causal explanation dataset by collecting 3,268 random Facebook status update messages  Three well-trained annotators manually labeled whether or not each message contains the causal explanation and obtained 1,598 causality messages with substantial agreement ( $\kappa =0 61$ )  We used the majority vote for our gold standard  Then, on each causality message, annotators identified which text spans are causal explanations 
For each task, we used 80% of the dataset for training our model and 10% for tuning the hyperparameters of our models  Finally, we evaluated all of our models on the remaining 10% (Table 1 and Table 2 )  For causal explanation detection task, we extracted discourse arguments using our parser and selected discourse arguments which most cover the annotated causal explanation text span as our gold standard 
Model
We build two types of models  First, we develop feature-based models which utilize features of the successful models in social media analysis and causal relation discourse parsing  Then, we build a recursive neural network model which uses distributed representation of discourse arguments as this approach can even capture latent properties of causal relations which may exist between distant discourse arguments  We specifically selected bidirectional LSTM since the model with the discourse distributional structure built in this form outperformed the traditional models in similar NLP downstream tasks BIBREF3  
As the first step of our pipeline, we use Tweebo parser BIBREF20 to extract syntactic features from messages  Then, we demarcate sentences using punctuation (`,') tag and periods  Among those sentences, we find discourse connectives defined in PDTB annotation along with a Tweet POS tag for conjunction words which can also be a discourse marker  In order to decide whether these connectives are really discourse connectives (e g , I went home, but he stayed) as opposed to simple connections of two words (I like apple and banana) we see if verb phrases exist before and after the connective by using dependency parsing results  Although discourse connective disambiguation is a complicated task which can be much improved by syntactic features BIBREF21 , we try to minimize effects of syntactic parsing and simplify it since it is highly error-prone in social media  Finally, according to visual inspection, emojis (`E' tag) are crucial for discourse relation in social media so we take them as separate discourse arguments (e g ,in “My test result    :(” the sad feeling is caused by the test result, but it cannot be captured by plain word tokens) 
We trained a linear SVM, an rbf SVM, and a random forest with N-gram, charater N-gram, and tweet POS tags, sentiment tags, average word lengths and word counts from each message as they have a pivotal role in the models for many NLP downstream tasks in social media BIBREF19 , BIBREF18   In addition to these features, we also extracted First-Last, First3 features and Word Pairs from every adjacent pair of discourse arguments since these features were most helpful for causal relation prediction BIBREF9   First-Last, First3 features are first and last word and first three words of two discourse arguments of the relation, and Word Pairs are the cross product of words of those discourse arguments  These two features enable our model to capture interaction between two discourse arguments  BIBREF9 reported that these two features along with verbs, modality, context, and polarity (which can be captured by N-grams, sentiment tags and POS tags in our previous features) obtained the best performance for predicting Contingency class to which causality belongs 
We load the GLOVE word embedding BIBREF22 trained in Twitter for each token of extracted discourse arguments from messages  For the distributional representation of discourse arguments, we run a Word-level LSTM on the words' embeddings within each discourse argument and concatenate last hidden state vectors of forward LSTM ( $\overrightarrow{h}$ ) and backward LSTM ( $\overleftarrow{h}$ ) which is suggested by BIBREF3 ( $DA = [\overrightarrow{h};\overleftarrow{h}]$ )  Then, we feed the sequence of the vector representation of discourse arguments to the Discourse-argument-level LSTM (DA-level LSTM) to make a final prediction with log softmax function  With this structure, the model can learn the representation of interaction of tokens inside each discourse argument, then capture discourse relations across all of the discourse arguments in each message (Figure 2 )  In order to prevent the overfitting, we added a dropout layer between the Word-level LSTM and the DA-level LSTM layer 
We also explore subsets of the full RNN architecture, specifically with one of the two LSTM layers removed  In the first model variant, we directly input all word embeddings of a whole message to a BiLSTM layer and make prediction (Word LSTM) without the help of the distributional vector representations of discourse arguments  In the second model variant, we take the average of all word embeddings of each discourse argument ( $DA_k=\frac{1}{N_k} \sum _{i=1}^{N_k}W_{i}$ ), and use them as inputs to a BiLSTM layer (DA AVG LSTM) as the average vector of embeddings were quite effective for representing the whole sequence BIBREF3 , BIBREF5   As with the full architectures, for CP both of these variants ends with a many-to-one classification per message, while the CEI model ends with a sequence of classifications 
Experiment
We explored three types of models (RBF SVM, Linear SVM, and Random Forest Classifier) which have previously been shown empirically useful for the language analysis in social media  We filtered out low frequency Word Pairs features as they tend to be noisy and sparse BIBREF9   Then, we conducted univariate feature selection to restrict all remaining features to those showing at least a small relationship with the outcome  Specifically, we keep all features passing a family-wise error rate of $\alpha = 60$ with the given outcome  After comparing the performance of the optimized version of each model, we also conducted a feature ablation test on the best model in order to see how much each feature contributes to the causality prediction 
We used bidirectional LSTMs for causality classification and causal explanation identification since the discourse arguments for causal explanation can show up either before and after the effected events or results and we want our model to be optimized for both cases  However, there is a risk of overfitting due to the dataset which is relatively small for the high complexity of the model, so we added a dropout layer (p=0 3) between the Word-level LSTM and the DA-level LSTM 
For tuning our model, we explore the dimensionality of word vector and LSTM hidden state vectors of discourse arguments of 25, 50, 100, and 200 as pretrained GLOVE vectors were trained in this setting  For optimization, we used Stochastic Gradient Descent (SGD) and Adam BIBREF23 with learning rates 0 01 and 0 001 
We ignore missing word embeddings because our dataset is quite small for retraining new word embeddings  However, if embeddings are extracted as separate discourse arguments, we used the average of all vectors of all discourse arguments in that message  Average embeddings have performed well for representing text sequences in other tasks BIBREF5  
We first use state-of-the-art PDTB taggers for our baseline BIBREF13 , BIBREF12 for the evaluation of the causality prediction of our models ( BIBREF12 requires sentences extracted from the text as its input, so we used our parser to extract sentences from the message)  Then, we compare how models work for each task and disassembled them to inspect how each part of the models can affect their final prediction performances  We conducted McNemar's test to determine whether the performance differences are statistically significant at $p <  05$  
Results
We investigated various models for both causality detection and explanation identification  Based on their performances on the task, we analyzed the relationships between the types of models and the tasks, and scrutinized further for the best performing models  For performance analysis, we reported weighted F1 of classes 
Causality Prediction
In order to classify whether a message contains causal relation, we compared off-the-shelf PDTB parsers, linear SVM, RBF SVM, Random forest and LSTM classifiers  The off-the-shelf parsers achieved the lowest accuracies ( BIBREF12 and BIBREF13 in Table 3 )  This result can be expected since 1) these models were trained with news articles and 2) they are trained for all possible discourse relations in addition to causal relations (e g , contrast, condition, etc)  Among our suggested models, SVM and random forest classifier performed better than LSTM and, in the general trend, the more complex the models were, the worse they performed  This suggests that the models with more direct and simpler learning methods with features might classify the causality messages better than the ones more optimized for capturing distributional information or non-linear relationships of features 
Table 4 shows the results of a feature ablation test to see how each feature contributes to causality classification performance of the linear SVM classifier  POS tags caused the largest drop in F1  We suspect POS tags played a unique role because discourse connectives can have various surface forms (e g , because, cuz, bcuz, etc) but still the same POS tag `P'  Also POS tags can capture the occurrences of modal verbs, a feature previously found to be very useful for detecting similar discourse relations BIBREF9   N-gram features caused 0 022 F1 drop while sentiment tags did not affect the model when removed  Unlike the previous work where First-Last, First3 and Word pairs tended to gain a large F1 increase for multiclass discourse relation prediction, in our case, they did not affect the prediction performance compared to other feature types such as POS tags or N-grams 
Causal Explanation Identification
In this task, the model identifies causal explanations given the discourse arguments of the causality message  We explored over the same models as those we used for causality (sans the output layer), and found the almost opposite trend of performances (see Table 5 )  The Linear SVM obtained lowest F1 while the LSTM model made the best identification performance  As opposed to the simple binary classification of the causality messages, in order to detect causal explanation, it is more beneficial to consider the relation across discourse arguments of the whole message and implicit distributional representation due to the implicit causal relations between two distant arguments 
Architectural Variants
For causality prediction, we experimented with only word tokens in the whole message without help of Word-level LSTM layer (Word LSTM), and F1 dropped by 0 064 (CP in Table 6 )  Also, when we used the average of the sequence of word embeddings of each discourse argument as an input to the DA-level LSTM and it caused F1 drop of 0 073  This suggests that the information gained from both the interaction of words in and in between discourse arguments help when the model utilizes the distributional representation of the texts 
For causal explanation identification, in order to test how the LSTM classifier works without its capability of capturing the relations between discourse arguments, we removed DA-level LSTM layer and ran the LSTM directly on the word embedding sequence for each discourse argument for classifying whether the argument is causal explanation, and the model had 0 061 F1 drop (Word LSTM in CEI in Table 6 )  Also, when we ran DA-level LSTM on the average vectors of the word sequences of each discourse argument of messages, F1 decreased to 0 818  This follows the similar pattern observed from other types of models performances (i e , SVMs and Random Forest classifiers) that the models with higher complexity for capturing the interaction of discourse arguments tend to identify causal explanation with the higher accuracies 
For CEI task, we found that when the model ran on the sequence representation of discourse argument (DA AVG LSTM), its performance was higher than the plain sequence of word embeddings (Word LSTM)  Finally, in both subtasks, when the models ran on both Word-level and DA-Level (Full LSTM), they obtained the highest performance 
Complete Pipeline
Evaluations thus far zeroed-in on each subtask of causal explanation analysis (i e  CEI only focused on data already identified to contain causal explanations)  Here, we seek to evaluate the complete pipeline of CP and CEI, starting from all of test data (those or without causality) and evaluating the final accuracy of CEI predictions  This is intended to evaluate CEI performance under an applied setting where one does not already know whether a document has a causal explanation 
There are several approaches we could take to perform CEI starting from unannotated data  We could simply run CEI prediction by itself (CEI Only) or the pipeline of CP first and then only run CEI on documents predicted as causal (CP + CEI)  Further, the CEI model could be trained only on those documents annotated causal (as was done in the previous experiments) or on all training documents including many that are not causal 
Table 7 show results varying the pipeline and how CEI was trained  Though all setups performed decent ( $F1 > 0 81$ ) we see that the pipelined approach, first predicting causality (with the linear SVM) and then predicting causal explanations only for those with marked causal (CP + CEI $_{causal}$ ) yielded the strongest results  This also utilized the CEI model only trained on those annotated causal  Besides performance, an added benefit from this two step approach is that the CP step is less computational intensive of the CEI step and approximately 2/3 of documents will never need the CEI step applied 
We had an inevitable limitation on the size of our dataset, since there is no other causality dataset over social media and the annotation required an intensive iterative process  This might have limited performances of more complex models, but considering the processing time and the computation load, the combination of the linear model and the RNN-based model of our pipeline obtained both the high performance and efficiency for the practical applications to downstream tasks  In other words, it's possible the linear model will not perform as well if the training size is increased substantially  However, a linear model could still be used to do a first-pass, computationally efficient labeling, in order to shortlist social media posts for further labeling from an LSTM or more complex model 
Exploration
Here, we explore the use of causal explanation analysis for downstream tasks  First we look at the relationship between use of causal explanation and one's demographics: age and gender  Then, we consider their use in sentiment analysis for extracting the causes of polarity ratings  Research involving human subjects was approved by the University of Pennsylvania Institutional Review Board 
Conclusion
We developed a pipeline for causal explanation analysis over social media text, including both causality prediction and causal explanation identification  We examined a variety of model types and RNN architectures for each part of the pipeline, finding an SVM best for causality prediction and a hierarchy of BiLSTMs for causal explanation identification, suggesting the later task relies more heavily on sequential information  In fact, we found replacing either layer of the hierarchical LSTM architecture (the word-level or the DA-level) with a an equivalent “bag of features” approach resulted in reduced accuracy  Results of our whole pipeline of causal explanation analysis were found quite strong, achieving an $F1=0 868$ at identifying discourse arguments that are causal explanations 
Finally, we demonstrated use of our models in applications, finding associations between demographics and rate of mentioning causal explanations, as well as showing differences in the top words predictive of negative ratings in Yelp reviews  Utilization of discourse structure in social media analysis has been a largely untapped area of exploration, perhaps due to its perceived difficulty  We hope the strong results of causal explanation identification here leads to the integration of more syntax and deeper semantics into social media analyses and ultimately enables new applications beyond the current state of the art 
Acknowledgments
This work was supported, in part, by a grant from the Templeton Religion Trust (ID #TRT0048)  The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript  We also thank Laura Smith, Yiyi Chen, Greta Jawel and Vanessa Hernandez for their work in identifying causal explanations 
Context 3:  These authors also noticed that fake news items are more likely shared than legitimate articles coming from trusted sources, and they tend to exhibit a larger level of polarization 
The conceptual issue of how to define fake news is a serious and unresolved issue  As the focus of our work is not attempting to offer light on this, we will rely on work by other authors to describe what we consider as fake news  In particular, we use the categorization provided by Rubin et al  BIBREF7   The five categories they described, together with illustrative examples from our dataset, are as follows:
Research Hypotheses
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news  Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Taking those three dimensions into account, we propose the following hypotheses about the features that we believe can help to identify tweets containing fake news from those not containing them  They will be later tested over our collected dataset 
Exposure 
Characterization 
Polarization 
Data and Methodology
For this study, we collected publicly available tweets using Twitter's public API  Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data  Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets  Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016  Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton  The data collection ran for just one day (Nov 8th 2016) 
One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers  Among the reasons for retweeting, Body et al  BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others  As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news)  For our study, we consider that a tweet went viral if it was retweeted more than 1000 times 
Once we have the dataset of viral tweets, we eliminated duplicates (some of the tweets were collected several times because they had several handles) and an expert manually inspected the text field within the tweets to label them as containing fake news, or not containing them (according to the characterization presented before)  This annotated dataset BIBREF8 is publicly available and can be freely reused 
Finally, we use the following fields within tweets (from the ones returned by Twitter's API) to compare their distributions and look for differences between viral tweets containing fake news and viral tweets not containing fake news:
In the following section, we provide graphical descriptions of the distribution of each of the identified attributes for the two sets of tweets (those labelled as containing fake news and those labelled as not containing them)  Where appropriate, we normalized and/or took logarithms of the data for better representation  To gain a better understanding of the significance of those differences, we use the Kolmogorov-Smirnov test with the null hypothesis that both distributions are equal 
Results
The sample collected consisted on 1 785 855 tweets published by 848 196 different users  Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users  Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window 
The 1327 `viral' tweets were manually annotated as containing fake news or not  The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset  Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'  Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization  Because of this, we do not claim that this dataset can be considered a ground truth 
The following results detail characteristics of these tweets along the previously mentioned dimensions  Table TABREF23 reports the actual differences (together with their associated p-values) of the distributions of viral tweets containing fake news and viral tweets not containing them for every variable considered 
Exposure
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently  As such, Twitter users were exposed to fake news related to the election for a shorter period of time 
However, in terms of retweets, Figure FIGREF25 shows no apparent difference between containing fake news or not containing them  That is confirmed by the Kolmogorov-Smirnoff test, which does not discard the hypothesis that the associated distributions are equal 
In relation to the number of favourites, users that generated at least a viral tweet containing fake news appear to have, on average, less favourites than users that do not generate them  Figure FIGREF26 shows the distribution of favourites  Despite the apparent visual differences, the difference are not statistically significant 
Finally, the number of hashtags used in viral fake news appears to be larger than those in other viral tweets  Figure FIGREF27 shows the density distribution of the number of hashtags used  However, once again, we were not able to find any statistical difference between the average number of hashtags in a viral tweet and the average number of hashtags in viral fake news 
Characterization
We found that 82 users within our sample were spreading fake news (i e  they produced at least one tweet which was labelled as fake news)

ground_truth: Facebook status update messages

Facebook status update messages

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were their distribution results?
Context 3:  Out of those, 34 had verified accounts, and the rest were unverified  From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted  Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs  not containing fake news)  From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts 
Turning to friends, accounts distributing fake news appear to have, on average, the same number of friends than those distributing tweets with no fake news  However, the density distribution of friends from the accounts (Figure FIGREF29 ) shows that there is indeed a statistically significant difference in their distributions 
If we take into consideration the number of followers, accounts generating viral tweets with fake news do have a very different distribution on this dimension, compared to those accounts generating viral tweets with no fake news (see Figure FIGREF30 )  In fact, such differences are statistically significant 
A useful representation for friends and followers is the ratio between friends/followers  Figures FIGREF31 and FIGREF32 show this representation  Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers  The distribution of those accounts not generating fake news is more evenly distributed 
With respect to the number of mentions, Figure FIGREF33 shows that viral tweets labelled as containing fake news appear to use mentions to other users less frequently than viral tweets not containing fake news  In other words, tweets containing fake news mostly contain 1 mention, whereas other tweets tend to have two)  Such differences are statistically significant 
The analysis (Figure FIGREF34 ) of the presence of media in the tweets in our dataset shows that tweets labelled as not containing fake news appear to present more media elements than those labelled as fake news  However, the difference is not statistically significant 
On the other hand, Figure FIGREF35 shows that viral tweets containing fake news appear to include more URLs to other sites than viral tweets that do not contain fake news  In fact, the difference between the two distributions is statistically significant (assuming INLINEFORM0 ) 
Polarization
Finally, manual inspection of the text field of those viral tweets labelled as containing fake news shows that 117 of such tweets expressed support for Donald Trump, while only 8 supported Hillary Clinton  The remaining tweets contained fake news related to other topics, not expressing support for any of the candidates 
Discussion
As a summary, and constrained by our existing dataset, we made the following observations regarding differences between viral tweets labelled as containing fake news and viral tweets labelled as not containing them:
These findings (related to our initial hypothesis in Table TABREF44 ) clearly suggest that there are specific pieces of meta-data about tweets that may allow the identification of fake news  One such parameter is the time of exposure  Viral tweets containing fake news are shorter-lived than those containing other type of content  This notion seems to resonate with our findings showing that a number of accounts spreading fake news have already been deleted or suspended by Twitter by the time of writing  If one considers that researchers using different data have found similar results BIBREF9 , it appears that the lifetime of accounts, together with the age of the questioned viral content could be useful to identify fake news  In the light of this finding, accounts newly created should probably put under higher scrutiny than older ones  This in fact, would be a nice a-priori bias for a Bayesian classifier 
Accounts spreading fake news appear to have a larger proportion of friends/followers (i e  they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only  Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter  Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising) 
Finally, it is interesting to notice that the content of viral fake news was highly polarized  This finding is also in line with those of Alcott et al  BIBREF9   This feature suggests that textual sentiment analysis of the content of tweets (as most researchers do), together with the above mentioned parameters from meta-data, may prove useful for identifying fake news 
Conclusions
With the election of Donald Trump as President of the United States, the concept of fake news has become a broadly-known phenomenon that is getting tremendous attention from governments and media companies  We have presented a preliminary study on the meta-data of a publicly available dataset of tweets that became viral during the day of the 2016 US presidential election  Our aim is to advance the understanding of which features might be characteristic of viral tweets containing fake news in comparison with viral tweets without fake news 
We believe that the only way to automatically identify those deceitful tweets (i e  containing fake news) is by actually understanding and modelling them  Only then, the automation of the processes of tagging and blocking these tweets can be successfully performed  In the same way that spam was fought, we anticipate fake news will suffer a similar evolution, with social platforms implementing tools to deal with them  With most works so far focusing on the actual content of the tweets, ours is a novel attempt from a different, but also complementary, angle 
Within the used dataset, we found there are differences around exposure, characteristics of accounts spreading fake news and the tone of the content  Those findings suggest that it is indeed possible to model and automatically detect fake news  We plan to replicate and validate our experiments in an extended sample of tweets (until 4 months after the US election), and tests the predictive power of the features we found relevant within our sample 
Author Disclosure Statement
No competing financial interest exist 

ground_truth: Distributions of Followers, Friends and URLs are significantly different between the set of tweets containing fake news and those non containing them, but for Favourites, Mentions, Media, Retweets and Hashtags they are not significantly different