prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the political bias of different sources included in the model?
Context 1: Introduction and related work
In recent years there has been increasing interest on the issue of disinformation spreading on online social media  Global concern over false (or "fake") news as a threat to modern democracies has been frequently raised–ever since 2016 US Presidential elections–in correspondence of events of political relevance, where the proliferation of manipulated and low-credibility content attempts to drive and influence people opinions BIBREF0BIBREF1BIBREF2BIBREF3 
Researchers have highlighted several drivers for the diffusion of such malicious phenomenon, which include human factors (confirmation bias BIBREF4, naive realism BIBREF5), algorithmic biases (filter bubble effect BIBREF0), the presence of deceptive agents on social platforms (bots and trolls BIBREF6) and, lastly, the formation of echo chambers BIBREF7 where people polarize their opinions as they are insulated from contrary perspectives 
The problem of automatically detecting online disinformation news has been typically formulated as a binary classification task (i e  credible vs non-credible articles), and tackled with a variety of different techniques, based on traditional machine learning and/or deep learning, which mainly differ in the dataset and the features they employ to perform the classification  We may distinguish three approaches: those built on content-based features, those based on features extracted from the social context, and those which combine both aspects  A few main challenges hinder the task, namely the impossibility to manually verify all news items, the lack of gold-standard datasets and the adversarial setting in which malicious content is created BIBREF3BIBREF6 
In this work we follow the direction pointed out in a few recent contributions on the diffusion of disinformation compared to traditional and objective information  These have shown that false news spread faster and deeper than true news BIBREF8, and that social bots and echo chambers play an important role in the diffusion of malicious content BIBREF6, BIBREF7  Therefore we focus on the analysis of spreading patterns which naturally arise on social platforms as a consequence of multiple interactions between users, due to the increasing trend in online sharing of news BIBREF0 
A deep learning framework for detection of fake news cascades is provided in BIBREF9, where the authors refer to BIBREF8 in order to collect Twitter cascades pertaining to verified false and true rumors  They employ geometric deep learning, a novel paradigm for graph-based structures, to classify cascades based on four categories of features, such as user profile, user activity, network and spreading, and content  They also observe that a few hours of propagation are sufficient to distinguish false news from true news with high accuracy  Diffusion cascades on Weibo and Twitter are analyzed in BIBREF10, where authors focus on highlighting different topological properties, such as the number of hops from the source or the heterogeneity of the network, to show that fake news shape diffusion networks which are highly different from credible news, even at early stages of propagation 
In this work, we consider the results of BIBREF11 as our baseline  The authors use off-the-shelf machine learning classifiers to accurately classify news articles leveraging Twitter diffusion networks  To this aim, they consider a set of basic features which can be qualitatively interpreted w r t to the social behavior of users sharing credible vs non-credible information  Their methodology is overall in accordance with BIBREF12, where authors successfully detect Twitter astroturfing content, i e  political campaigns disguised as spontaneous grassroots, with a machine learning framework based on network features 
In this paper, we propose a classification framework based on a multi-layer formulation of Twitter diffusion networks  For each article we disentangle different social interactions on Twitter, namely tweets, retweets, mentions, replies and quotes, to accordingly build a diffusion network composed of multiple layers (on for each type of interaction), and we compute structural features separately for each layer  We pick a set of global network properties from the network science toolbox which can be qualitatively explained in terms of social dimensions and allow us to encode different networks with a tuple of features  These include traditional indicators, e g  network density, number of strong/weak connected components and diameter, and more elaborated ones such as main K-core number BIBREF13 and structural virality BIBREF14  Our main research question is whether the use of a multi-layer, disentangled network yields a significant advance in terms of classification accuracy over a conventional single-layer diffusion network  Additionally, we are interested in understanding which of the above features, and in which layer, are most effective in the classification task 
We perform classification experiments with an off-the-shelf Logistic Regression model on two different datasets of mainstream and disinformation news shared on Twitter respectively in the United States and in Italy during 2019  In the former case we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets  Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles)  We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains  We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries 
The outline of this paper is the following: we first formulate the problem and describe data collection, network representation and structural properties employed for the classification; then we provide experimental results–classification performances, layer and feature importance analyses and a temporal classification evaluation–and finally we draw conclusions and future directions 
Methodology ::: Disinformation and mainstream news
In this work we formulate our classification problem as follows: given two classes of news articles, respectively $D$ (disinformation) and $M$ (mainstream), a set of news articles $A_i$ and associated class labels $C_i \in \lbrace D,M\rbrace $, and a set of tweets $\Pi _i=\lbrace T_i^1, T_i^2,    \rbrace $ each of which contains an Uniform Resource Locator (URL) pointing explicitly to article $A_i$, predict the class $C_i$ of each article $A_i$  There is huge debate and controversy on a proper taxonomy of malicious and deceptive information BIBREF1BIBREF2BIBREF15BIBREF16BIBREF17BIBREF3BIBREF11  In this work we prefer the term disinformation to the more specific fake news to refer to a variety of misleading and harmful information  Therefore, we follow a source-based approach, a consolidated strategy also adopted by BIBREF6BIBREF16BIBREF2BIBREF1, in order to obtain relevant data for our analysis  We collected:
Disinformation articles, published by websites which are well-known for producing low-credibility content, false and misleading news reports as well as extreme propaganda and hoaxes and flagged as such by reputable journalists and fact-checkers;
Mainstream news, referring to traditional news outlets which deliver factual and credible information 
We believe that this is currently the most reliable classification approach, but it entails obvious limitations, as disinformation outlets may also publish true stories and likewise misinformation is sometimes reported on mainstream media  Also, given the choice of news sources, we cannot test whether our methodology is able to classify disinformation vs factual but not mainstream news which are published on niche, non-disinformation outlets 
Methodology ::: US dataset
We collected tweets associated to a dozen US mainstream news websites, i e  most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets  We filtered out articles associated to less than 50 tweets  The resulting dataset contains overall $\sim $1 7 million tweets for mainstream news, collected in a period of three weeks (February 25th, 2019-March 18th, 2019), which are associated to 6,978 news articles, and $\sim $1 6 million tweets for disinformation, collected in a period of three months (January 1st, 2019-March 18th, 2019) for sake of balance of the two classes, which hold 5,775 distinct articles  Diffusion censoring effects BIBREF14 were correctly taken into account in both collection procedures  We provide in Figure FIGREF4 the distribution of articles by source and political bias for both news domains 
As it is reported that conservatives and liberals exhibit different behaviors on online social platforms BIBREF19BIBREF20BIBREF21, we further assigned a political bias label to different US outlets (and therefore news articles) following the procedure described in BIBREF2  In order to assess the robustness of our method, we performed classification experiments by training only on left-biased (or right-biased) outlets of both disinformation and mainstream domains and testing on the entire set of sources, as well as excluding particular sources that outweigh the others in terms of samples to avoid over-fitting 
Methodology ::: Italian dataset
For what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa)  We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites  In order to get balanced classes (April 5th, 2019-May 5th, 2019), we retained data collected in a longer period w r t to mainstream news  In both cases we filtered out articles with less than 50 tweets; overall this dataset contains $\sim $160k mainstream tweets, corresponding to 227 news articles, and $\sim $100k disinformation tweets, corresponding to 237 news articles  We provide in Figure FIGREF5 the distribution of articles according to distinct sources for both news domains  As in the US dataset, we took into account censoring effects BIBREF14 by excluding tweets published before (left-censoring) or after two weeks (right-censoring) from the beginning of the collection process 
The different volumes of news shared on Twitter in the two countries are due both to the different population size of US and Italy (320 vs 60 millions) but also to the different usage of Twitter platform (and social media in general) for news consumption BIBREF24  Both datasets analyzed in this work are available from the authors on request 
A crucial aspect in our approach is the capability to fully capturing sharing cascades on Twitter associated to news articles  It has been reported BIBREF25 that the Twitter streaming endpoint filters out tweets matching a given query if they exceed 1% of the global daily volume of shared tweets, which nowadays is approximately $5\cdot 10^8$; however, as we always collected less than $10^6$ tweets per day, we did not incur in this issue and we thus gathered 100% of tweets matching our query 
Methodology ::: Building diffusion networks
We built Twitter diffusion networks following an approach widely adopted in the literature BIBREF6BIBREF17BIBREF2  We remark that there is an unavoidable limitation in Twitter Streaming API, which does not allow to retrieve true re-tweeting cascades because re-tweets always point to the original source and not to intermediate re-tweeting users BIBREF8BIBREF14; thus we adopt the only viable approach based on Twitter's public availability of data  Besides, by disentangling different interactions with multiple layers we potentially reduce the impact of this limitation on the global network properties compared to the single-layer approach used in our baseline 
Using the notation described in BIBREF26  we employ a multi-layer representation for Twitter diffusion networks  Sociologists have indeed recognized decades ago that it is crucial to study social systems by constructing multiple social networks where different types of ties among same individuals are used BIBREF27  Therefore, for each news article we built a multi-layer diffusion network composed of four different layers, one for each type of social interaction on Twitter platform, namely retweet (RT), reply (R), quote (Q) and mention (M), as shown in Figure FIGREF11  These networks are not necessarily node-aligned, i e  users might be missing in some layers  We do not insert "dummy" nodes to represent all users as it would have severe impact on the global network properties (e g  number of weakly connected components)  Alternatively one may look at each multi-layer diffusion network as an ensemble of individual graphs BIBREF26; since global network properties are computed separately for each layer, they are not affected by the presence of any inter-layer edges 
In our multi-layer representation, each layer is a directed graph where we add edges and nodes for each tweet of the layer type, e g  for the RT layer: whenever user $a$ retweets account $b$ we first add nodes $a$ and $b$ if not already present in the RT layer, then we build an edge that goes from $b$ to $a$ if it does not exists or we increment the weight by 1  Similarly for the other layers: for the R layer edges go from user $a$ (who replies) to user $b$, for the Q layer edges go from user $b$ (who is quoted by) to user $a$ and for the M layer edges go from user $a$ (who mentions) to user $b$ 
Note that, by construction, our layers do not include isolated nodes; they correspond to "pure tweets", i e  tweets which have not originated any interactions with other users  However, they are present in our dataset, and their number is exploited for classification, as described below 
Methodology ::: Global network properties
We used a set of global network indicators which allow us to encode each network layer by a tuple of features  Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector  We used the following global network properties:
Number of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\rightarrow v$, $v\rightarrow u$) 
Size of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph 
Number of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \leftrightarrow v$ ignoring edge directions 
Size of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph 
Diameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph 
Average Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique)  It is computed according to BIBREF28 
Main K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered) 
Density (d): the density for directed graphs is $d=\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph 
Structural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\frac{1}{|V||V-1|}\sum _i\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$  This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\frac{1}{|V||V-1|}$  In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$ 
We used networkx Python package BIBREF30 to compute all features  Whenever a layer is empty  we simply set to 0 all its features  In addition to computing the above nine features for each layer, we added two indicators for encoding information about pure tweets, namely the number T of pure tweets (containing URLs to a given news article) and the number U of unique users authoring those tweets  Therefore, a single diffusion network is represented by a vector with $9\cdot 4+2=38$ entries 
Methodology ::: Interpretation of network features and layers
Aforementioned network properties can be qualitatively explained in terms of social footprints as follows: SCC correlates with the size of the diffusion network, as the propagation of news occurs in a broadcast manner most of the time, i e  re-tweets dominate on other interactions, while LSCC allows to distinguish cases where such mono-directionality is somehow broken  WCC equals (approximately) the number of distinct diffusion cascades pertaining to each news article, with exceptions corresponding to those cases where some cascades merge together via Twitter interactions such as mentions, quotes and replies, and accordingly LWCC and DWCC equals the size and the depth of the largest cascade  CC corresponds to the level of connectedness of neighboring users in a given diffusion network whereas KC identifies the set of most influential users in a network and describes the efficiency of information spreading BIBREF17  Finally, d describes the proportions of potential connections between users which are actually activated and SV indicates whether a news item has gained popularity with a single and large broadcast or in a more viral fashion through multiple generations 
For what concerns different Twitter actions, users primarily interact with each other using retweets and mentions BIBREF20 
The former are the main engagement activity and act as a form of endorsement, allowing users to rebroadcast content generated by other users BIBREF31  Besides, when node B retweets node A we have an implicit confirmation that information from A appeared in B's Twitter feed BIBREF12  Quotes are simply a special case of retweets with comments 
Mentions usually include personal conversations as they allow someone to address a specific user or to refer to an individual in the third person; in the first case they are located at the beginning of a tweet and they are known as replies, otherwise they are put in the body of a tweet BIBREF20  The network of mentions is usually seen as a stronger version of interactions between Twitter users, compared to the traditional graph of follower/following relationships BIBREF32 
Experiments ::: Setup
We performed classification experiments using a basic off-the-shelf classifier, namely Logistic Regression (LR) with L2 penalty; this also allows us to compare results with our baseline  We applied a standardization of the features and we used the default configuration for parameters as described in scikit-learn package BIBREF33  We also tested other classifiers (such as K-Nearest Neighbors, Support Vector Machines and Random Forest) but we omit results as they give comparable performances  We remark that our goal is to show that a very simple machine learning framework, with no parameter tuning and optimization, allows for accurate results with our network-based approach 
We used the following evaluation metrics to assess the performances of different classifiers (TP=true positives, FP=false positives, FN=false negatives):
Precision = $\frac{TP}{TP+FP}$, the ability of a classifier not to label as positive a negative sample 
Recall = $\frac{TP}{TP+FN}$, the ability of a classifier to retrieve all positive samples 
F1-score = $2 \frac{\mbox{Precision} \cdot \mbox{Recall}}{\mbox{Precision} + \mbox{Recall}}$, the harmonic average of Precision and Recall 
Area Under the Receiver Operating Characteristic curve (AUROC); the Receiver Operating Characteristic (ROC) curve BIBREF34, which plots the TP rate versus the FP rate, shows the ability of a classifier to discriminate positive samples from negative ones as its threshold is varied; the AUROC value is in the range $[0, 1]$, with the random baseline classifier holding AUROC$=0 5$ and the ideal perfect classifier AUROC$=1$; thus larger AUROC values (and steeper ROCs) correspond to better classifiers 
In particular we computed so-called macro average–simple unweighted mean–of these metrics evaluated considering both labels (disinformation and mainstream)  We employed stratified shuffle split cross validation (with 10 folds) to evaluate performances 
Finally, we partitioned networks according to the total number of unique users involved in the sharing, i e  the number of nodes in the aggregated network represented with a single-layer representation considering together all layers and also pure tweets  A breakdown of both datasets according to size class (and political biases for the US scenario) is provided in Table 1 and Table 2 
Experiments ::: Classification performances
In Table 3 we first provide classification performances on the US dataset for the LR classifier evaluated on the size class described in Table 1  We can observe that in all instances our methodology performs better than a random classifier (50% AUROC), with AUROC values above 85% in all cases 
For what concerns political biases, as the classes of mainstream and disinformation networks are not balanced (e g , 1,292 mainstream and 4,149 disinformation networks with right bias) we employ a Balanced Random Forest with default parameters (as provided in imblearn Python package BIBREF35)  In order to test the robustness of our methodology, we trained only on left-biased networks or right-biased networks and tested on the entire set of sources (relative to the US dataset); we provide a comparison of AUROC values for both biases in Figure 4  We can notice that our multi-layer approach still entails significant results, thus showing that it can accurately distinguish mainstream news from disinformation regardless of the political bias  We further corroborated this result with additional classification experiments, that show similar performances, in which we excluded from the training/test set two specific sources (one at a time and both at the same time) that outweigh the others in terms of data samples–respectively "breitbart com" for right-biased sources and "politicususa com" for left-biased ones 
We performed classification experiments on the Italian dataset using the LR classifier and different size classes (we excluded $[1000, +\infty )$ which is empty); we show results for different evaluation metrics in Table 3  We can see that despite the limited number of samples (one order of magnitude smaller than the US dataset) the performances are overall in accordance with the US scenario  As shown in Table 4, we obtain results which are much better than our baseline in all size classes (see Table 4):
In the US dataset our multi-layer methodology performs much better in all size classes except for large networks ($[1000, +\infty )$ size class), reaching up to 13% improvement on smaller networks ($[0, 100)$ size class);
In the IT dataset our multi-layer methodology outperforms the baseline in all size classes, with the maximum performance gain (20%) on medium networks ($[100, 1000)$ size class); the baseline generally reaches bad performances compared to the US scenario 
Overall, our performances are comparable with those achieved by two state-of-the-art deep learning models for "fake news" detection BIBREF9BIBREF36 
Experiments ::: Layer importance analysis
In order to understand the impact of each layer on the performances of classifiers, we performed additional experiments considering separately each layer (we ignored T and U features relative to pure tweets)  In Table 5 we show metrics for each layer and all size classes, computed with a 10-fold stratified shuffle split cross validation, evaluated on the US dataset; in Figure 5 we show AUROC values for each layer compared with the general multi-layer approach  We can notice that both Q and M layers alone capture adequately the discrepancies of the two distinct news domains in the United States as they obtain good results with AUROC values in the range 75%-86%; these are comparable with those of the multi-layer approach which, nevertheless, outperforms them across all size classes 
We obtained similar performances for the Italian dataset, as the M layer obtains comparable performances w r t multi-layer approach with AUROC values in the range 72%-82%  We do not show these results for sake of conciseness 
Experiments ::: Feature importance analysis
We further investigated the importance of each feature by performing a $\chi ^2$ test, with 10-fold stratified shuffle split cross validation, considering the entire range of network sizes $[0, +\infty )$  We show the Top-5 most discriminative features for each country in Table 6 
We can notice the exact same set of features (with different relative orderings in the Top-3) in both countries; these correspond to two global network propertie–LWCC, which indicates the size of the largest cascade in the layer, and SCC, which correlates with the size of the network–associated to the same set of layers (Quotes, Retweets and Mentions) 
We further performed a $\chi ^2$ test to highlight the most discriminative features in the M layer of both countries, which performed equally well in the classification task as previously highlighted; also in this case we focused on the entire range of network sizes $[0, +\infty )$  Interestingly, we discovered exactly the same set of Top-3 features in both countries, namely LWCC, SCC and DWCC (which indicates the depth of the largest cascade in the layer) 
An inspection of the distributions of all aforementioned features revealed that disinformation news exhibit on average larger values than mainstream news 
We can qualitatively sum up these results as follows:
Sharing patterns in the two news domains exhibit discrepancies which might be country-independent and due to the content that is being shared 
Interactions in disinformation sharing cascades tends to be broader and deeper than in mainstream news, as widely reported in the literature BIBREF8BIBREF2BIBREF7 
Users likely make a different usage of mentions when sharing news belonging to the two domains, consequently shaping different sharing patterns 
Experiments ::: Temporal analysis
Similar to BIBREF9, we carried out additional experiments to answer the following question: how long do we need to observe a news spreading on Twitter in order to accurately classify it as disinformation or mainstream?
With this goal, we built several versions of our original dataset of multi-layer networks by considering in turn the following lifetimes: 1 hour, 6 hours, 12 hours, 1 day, 2 days, 3 days and 7 days; for each case, we computed the global network properties of the corresponding network and evaluated the LR classifier with 10-fold cross validation, separately for each lifetime (and considering always the entire set of networks)  We show corresponding AUROC values for both US and IT datasets in Figure 6 
We can see that in both countries news diffusion networks can be accurately classified after just a few hours of spreading, with AUROC values which are larger than 80% after only 6 hours of diffusion  These results are very promising and suggest that articles pertaining to the two news domains exhibit discrepancies in their sharing patterns that can be timely exploited in order to rapidly detect misleading items from factual information 
Conclusions
In this work we tackled the problem of the automatic classification of news articles in two domains, namely mainstream and disinformation news, with a language-independent approach which is based solely on the diffusion of news items on Twitter social platform  We disentangled different types of interactions on Twitter to accordingly build a multi-layer representation of news diffusion networks, and we computed a set of global network properties–separately for each layer–in order to encode each network with a tuple of features  Our goal was to investigate whether a multi-layer representation performs better than one layer BIBREF11, and to understand which of the features, observed at given layers, are most effective in the classification task 
Experiments with an off-the-shelf classifier such as Logistic Regression on datasets pertaining to two different media landscapes (US and Italy) yield very accurate classification results (AUROC up to 94%), even when accounting for the different political bias of news sources, which are far better than our baseline BIBREF11 with improvements up to 20%  Classification performances using single layers show that the layer of mentions alone entails better performance w r t other layers in both countries 
We also highlighted the most discriminative features across different layers in both countries; the results suggest that differences between the two news domains might be country-independent but rather due only to the typology of content shared, and that disinformation news shape broader and deeper cascades 
Additional experiments involving the temporal evolution of Twitter diffusion networks show that our methodology can accurate classify mainstream and disinformation news after a few hours of propagation on the platform 
Overall, our results prove that the topological features of multi-layer diffusion networks might be effectively exploited to detect online disinformation  We do not deny the presence of deceptive efforts to orchestrate the regular spread of information on social media via content amplification and manipulation BIBREF37BIBREF38  On the contrary, we postulate that such hidden forces might play to accentuate the discrepancies between the diffusion patterns of disinformation and mainstream news (and thus to make our methodology effective) 
In the future we aim to further investigate three directions: (1) employ temporal networks to represent news diffusion and apply classification techniques that take into account the sequential aspect of data (e g  recurrent neural networks); (2) carry out an extensive comparison of the diffusion of disinformation and mainstream news across countries to investigate deeper the presence of differences and similarities in sharing patterns; (3) leverage our network-based features in addition to state-of-the-art text-based approaches for "fake news" dete ction in order to deliver a real-world system to detect misleading and harmful information spreading on social media 
Context 2: Background
Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task  It is also one of the long-term goals of natural language understanding, and has important applications in e g , building intelligent agents for conversation and customer service support  In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise 
In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist  Using a multi-task learning approach (c f  BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable  The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable  The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier) 
Our model is pretty simple and intuitive, yet efficient  Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2 0 
The contribution of this work is summarized as follows  First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly  Second, our model achieves competitive results on SQuAD v2 0 
Model
The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3   We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question  Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph  Formally, the answer is formulated as INLINEFORM7   In case of unanswerable questions, INLINEFORM8 points to the last token of the passage 
Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2   The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable  Roughly, the model includes two different layers: the shared layer and task specific layer  The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer  On top of it, there are different answer modules for different tasks  We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task  It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9   We will briefly describe the model from ground up as follows  Detailed descriptions can be found in BIBREF1  
Lexicon Encoding Layer  We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features  Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters  Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages  At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension  As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5  
Contextual Encoding Layer  A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions  To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer  Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages 
Memory Generation Layer  In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1   The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2
Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively  A question-aware passage representation is computed as INLINEFORM5   After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6
where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i e , attention with itself)  At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3  
Span detector  We adopt a multi-turn answer module for the span detector BIBREF1   Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2   The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6   Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11   Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1
The final prediction is the average of each time step: INLINEFORM0   We randomly apply dropout on the step level in each time step during training, as done in BIBREF1  
Unanswerable classifier  We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0
, where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2   INLINEFORM3 denotes the probability of the question which is unanswerable 
Objective The objective function of the joint model has two parts: DISPLAYFORM0
Following BIBREF0 , the span loss function is defined: DISPLAYFORM0
The objective function of the binary classifier is defined: DISPLAYFORM0
where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable 
Setup
We evaluate our system on SQuAD 2 0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1 0 BIBREF15 and additional unanswerable question-answer pairs  The answerable pairs are around 100K; while the unanswerable questions are around 53K  This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles  All the questions and answers are obtained by crowd-sourcing  Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14  
Implementation details
We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags  The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10   A 2-layer BiLSTM is used encoding the contextual information of both questions and passages  Regarding the hidden size of our model, we search greedily among INLINEFORM0   During training, Adamax BIBREF16 is used as our optimizer  The min-batch size is set to 32  The learning rate is initialized to 0 002 and it is halved after every 10 epochs  The dropout rate is set to 0 1  To prevent overfitting, we also randomly set 0 5% words in both passages and questions as unknown words during the training  Here, we use a special token unk to indicate a word which doesn't appear in GloVe  INLINEFORM1 in Eq EQREF9 is set to 1 
Results
We would like to investigate effectiveness the proposed joint model  To do so, the same shared layer/architecture is employed in the following variants of the proposed model:
The results in terms of EM and F1 is summarized in Table TABREF20   We observe that Joint SAN outperforms the SAN baseline with a large margin, e g , 67 89 vs 69 27 (+1 38) and 70 68 vs 72 20 (+1 52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization  By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e g , 72 2 vs 72 66 (+0 46) in terms of F1 score  By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability 
Table TABREF21 reports comparison results in literature published   Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo)  Comparing with the much complicated model R M -Reader + Verifier, which includes several components, our model still outperforms by 0 7 in terms of F1 score  Furthermore, we observe that ELMo gives a great boosting on the performance, e g , 2 8 points in terms of F1 for DocQA  This encourages us to incorporate ELMo into our model in future 
Analysis  To better understand our model, we analyze the accuracy of the classifier in our joint model  We obtain 75 3 classification accuracy on the development with the threshold 0 5  By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76 8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0 2 in terms of F1 score)  It shows that it is important to make balance between these two components: the span detector and unanswerable classifier 
Conclusion
To sum up, we proposed a simple yet efficient model based on SAN  It showed that the joint learning algorithm boosted the performance on SQuAD 2 0  We also would like to incorporate ELMo into our model in future 
Acknowledgments
We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments  We also thank Robin Jia for the help on SQuAD evaluations 
Context 3: Introduction
The challenges of imbalanced classification—in which the proportion of elements in each class for a classification task significantly differ—and of the ability to generalise on dissimilar data have remained important problems in Natural Language Processing (NLP) and Machine Learning in general  Popular NLP tasks including sentiment analysis, propaganda detection, and event extraction from social media are all examples of imbalanced classification problems  In each case the number of elements in one of the classes (e g  negative sentiment, propagandistic content, or specific events discussed on social media, respectively) is significantly lower than the number of elements in the other classes 
The recently introduced BERT language model for transfer learning BIBREF0 uses a deep bidirectional transformer architecture to produce pre-trained context-dependent embeddings  It has proven to be powerful in solving many NLP tasks and, as we find, also appears to handle imbalanced classification well, thus removing the need to use standard methods of data augmentation to mitigate this problem (see Section SECREF11 for related work and Section SECREF16 for analysis) 
BERT is credited with the ability to adapt to many tasks and data with very little training BIBREF0  However, we show that BERT fails to perform well when the training and test data are significantly dissimilar, as is the case with several tasks that deal with social and news data  In these cases, the training data is necessarily a subset of past data, while the model is likely to be used on future data which deals with different topics  This work addresses this problem by incorporating cost-sensitivity (Section SECREF19) into BERT 
We test these methods by participating in the Shared Task on Fine-Grained Propaganda Detection for the 2nd Workshop on NLP for Internet Freedom, for which we achieve the second rank on sentence-level classification of propaganda, confirming the importance of cost-sensitivity when the training and test sets are dissimilar 
Introduction ::: Detecting Propaganda
The term `propaganda' derives from propagare in post-classical Latin, as in “propagation of the faith" BIBREF1, and thus has from the beginning been associated with an intentional and potentially multicast communication; only later did it become a pejorative term  It was pragmatically defined in the World War II era as “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends" BIBREF2 
For the philosopher and sociologist Jacques Ellul, however, in a society with mass communication, propaganda is inevitable and thus it is necessary to become more aware of it BIBREF3; but whether or not to classify a given strip of text as propaganda depends not just on its content but on its use on the part of both addressers and addressees BIBREF1, and this fact makes the automated detection of propaganda intrinsically challenging 
Despite this difficulty, interest in automatically detecting misinformation and/or propaganda has gained significance due to the exponential growth in online sources of information combined with the speed with which information is shared today  The sheer volume of social interactions makes it impossible to manually check the veracity of all information being shared  Automation thus remains a potentially viable method of ensuring that we continue to enjoy the benefits of a connected world without the spread of misinformation through either ignorance or malicious intent 
In the task introduced by BIBREF4, we are provided with articles tagged as propaganda at the sentence and fragment (or span) level and are tasked with making predictions on a development set followed by a final held-out test set  We note this gives us access to the articles in the development and test sets but not their labels 
We participated in this task under the team name ProperGander and were placed 2nd on the sentence level classification task where we make use of our methods of incorporating cost-sensitivity into BERT  We also participated in the fragment level task and were placed 7th  The significant contributions of this work are:
We show that common (`easy') methods of data augmentation for dealing with class imbalance do not improve base BERT performance 
We provide a statistical method of establishing the similarity of datasets 
We incorporate cost-sensitivity into BERT to enable models to adapt to dissimilar datasets 
We release all our program code on GitHub and Google Colaboratory, so that other researchers can benefit from this work 
Related work ::: Propaganda detection
Most of the existing works on propaganda detection focus on identifying propaganda at the news article level, or even at the news outlet level with the assumption that each of the articles of the suspected propagandistic outlet are propaganda BIBREF5, BIBREF6 
Here we study two tasks that are more fine-grained, specifically propaganda detection at the sentence and phrase (fragment) levels BIBREF4  This fine-grained setup aims to train models that identify linguistic propaganda techniques rather than distinguishing between the article source styles 
BIBREF4 EMNLP19DaSanMartino were the first to propose this problem setup and release it as a shared task  Along with the released dataset, BIBREF4 proposed a multi-granularity neural network, which uses the deep bidirectional transformer architecture known as BERT, which features pre-trained context-dependent embeddings BIBREF0  Their system takes a joint learning approach to the sentence- and phrase-level tasks, concatenating the output representation of the less granular (sentence-level) task with the more fine-grained task using learned weights 
In this work we also take the BERT model as the basis of our approach and focus on the class imbalance as well as the lack of similarity between training and test data inherent to the task 
Related work ::: Class imbalance
A common issue for many Natural Language Processing (NLP) classification tasks is class imbalance, the situation where one of the class categories comprises a significantly larger proportion of the dataset than the other classes  It is especially prominent in real-world datasets and complicates classification when the identification of the minority class is of specific importance 
Models trained on the basis of minimising errors for imbalanced datasets tend to more frequently predict the majority class; achieving high accuracy in such cases can be misleading  Because of this, the macro-averaged F-score, chosen for this competition, is a more suitable metric as it weights the performance on each class equally 
As class imbalance is a widespread issue, multiple techniques have been developed that help alleviate it BIBREF7, BIBREF8, by either adjusting the model (e g  changing the performance metric) or changing the data (e g  oversampling the minority class or undersampling the majority class) 
Related work ::: Class imbalance ::: Cost-sensitive learning
Cost-sensitive classification can be used when the “cost” of mislabelling one class is higher than that of mislabelling other classes BIBREF9, BIBREF10  For example, the real cost to a bank of miscategorising a large fraudulent transaction as authentic is potentially higher than miscategorising (perhaps only temporarily) a valid transaction as fraudulent  Cost-sensitive learning tackles the issue of class imbalance by changing the cost function of the model such that misclassification of training examples from the minority class carries more weight and is thus more `expensive'  This is achieved by simply multiplying the loss of each example by a certain factor  This cost-sensitive learning technique takes misclassification costs into account during model training, and does not modify the imbalanced data distribution directly 
Related work ::: Class imbalance ::: Data augmentation
Common methods that tackle the problem of class imbalance by modifying the data to create balanced datasets are undersampling and oversampling  Undersampling randomly removes instances from the majority class and is only suitable for problems with an abundance of data  Oversampling means creating more minority class instances to match the size of the majority class  Oversampling methods range from simple random oversampling, i e  repeating the training procedure on instances from the minority class, chosen at random, to the more complex, which involves constructing synthetic minority-class samples  Random oversampling is similar to cost-sensitive learning as repeating the sample several times makes the cost of its mis-classification grow proportionally  Kolomiyets et al  kolomiyets2011model, Zhang et al  zhang2015character, and Wang and Yang wang2015s perform data augmentation using synonym replacement, i e  replacing random words in sentences with their synonyms or nearest-neighbor embeddings, and show its effectiveness on multiple tasks and datasets  Wei et al  wei2019eda provide a great overview of `easy' data augmentation (EDA) techniques for NLP, including synonym replacement as described above, and random deletion, i e  removing words in the sentence at random with pre-defined probability

ground_truth: By assigning a political bias label to each news article and training only on left-biased or right-biased outlets of both disinformation and mainstream domains

we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets. Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles). We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains. We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use attention?
Context 3:  They show the effectiveness of EDA across five text classification tasks  However, they mention that EDA may not lead to substantial improvements when using pre-trained models  In this work we test this claim by comparing performance gains of using cost-sensitive learning versus two data augmentation methods, synonym replacement and random deletion, with a pre-trained BERT model 
More complex augmentation methods include back-translation BIBREF11, translational data augmentation BIBREF12, and noising BIBREF13, but these are out of the scope of this study 
Dataset
The Propaganda Techniques Corpus (PTC) dataset for the 2019 Shared Task on Fine-Grained Propaganda consists of a training set of 350 news articles, consisting of just over 16,965 total sentences, in which specifically propagandistic fragments have been manually spotted and labelled by experts  This is accompanied by a development set (or dev set) of 61 articles with 2,235 total sentences, whose labels are maintained by the shared task organisers; and two months after the release of this data, the organisers released a test set of 86 articles and 3,526 total sentences  In the training set, 4,720 ($\sim 28\%$) of the sentences have been assessed as containing propaganda, with 12,245 sentences ($\sim 72 \%$) as non-propaganda, demonstrating a clear class imbalance 
In the binary sentence-level classification (SLC) task, a model is trained to detect whether each and every sentence is either 'propaganda' or 'non-propaganda'; in the more challenging field-level classification (FLC) task, a model is trained to detect one of 18 possible propaganda technique types in spans of characters within sentences  These propaganda types are listed in BIBREF4 and range from those which might be recognisable at the lexical level (e g  Name_Calling, Repetition), and those which would likely need to incorporate semantic understanding (Red_Herring, Straw_Man) 
For several example sentences from a sample document annotated with fragment-level classifications (FLC) (Figure FIGREF13)  The corresponding sentence-level classification (SLC) labels would indicate that sentences 3, 4, and 7 are 'propaganda' while the the other sentences are `non-propaganda' 
Dataset ::: Data Distribution
One of the most interesting aspects of the data provided for this task is the notable difference between the training and the development/test sets  We emphasise that this difference is realistic and reflective of real world news data, in which major stories are often accompanied by the introduction of new terms, names, and even phrases  This is because the training data is a subset of past data while the model is to be used on future data which deals with different newsworthy topics 
We demonstrate this difference statistically by using a method for finding the similarity of corpora suggested by BIBREF14  We use the Wilcoxon signed-rank test BIBREF15 which compares the frequency counts of randomly sampled elements from different datasets to determine if those datasets have a statistically similar distribution of elements 
We implement this as follows  For each of the training, development and test sets, we extract all words (retaining the repeats) while ignoring a set of stopwords (identified through the Python Natural Language Toolkit)  We then extract 10,000 samples (with replacements) for various pairs of these datasets (training, development, and test sets along with splits of each of these datasets)  Finally, we use comparative word frequencies from the two sets to calculate the p-value using the Wilcoxon signed-rank test  Table TABREF15 provides the minimum and maximum p-values and their interpretations for ten such runs of each pair reported 
With p-value less than 0 05, we show that the train, development and test sets are self-similar and also significantly different from each other  In measuring self-similarity, we split each dataset after shuffling all sentences  While this comparison is made at the sentence level (as opposed to the article level), it is consistent with the granularity used for propaganda detection, which is also at the sentence level  We also perform measurements of self similarity after splitting the data at the article level and find that the conclusions of similarity between the sets hold with a p-value threshold of 0 001, where p-values for similarity between the training and dev/test sets are orders of magnitude lower compared to self-similarity  Since we use random sampling we run this test 10 times and present the both the maximum and minimum p-values  We include the similarity between 25% of a dataset and the remaining 75% of that set because that is the train/test ratio we use in our experiments, further described in our methodology (Section SECREF4) 
This analysis shows that while all splits of each of the datasets are statistically similar, the training set (and the split of the training set that we use for experimentation) are significantly different from the development and test sets  While our analysis does show that the development and the test sets are dissimilar, we note (based on the p-values) that they are significantly more similar to each other than they are to the training set 
Methodology
We were provided with two tasks: (1) propaganda fragment-level identification (FLC) and (2) propagandistic sentence-level identification (SLC)  While we develop systems for both tasks, our main focus is toward the latter  Given the differences between the training, development, and test sets, we focus on methods for generalising our models  We note that propaganda identification is, in general, an imbalanced binary classification problem as most sentences are not propagandistic 
Due to the non-deterministic nature of fast GPU computations, we run each of our models three times and report the average of these three runs through the rest of this section  When picking the model to use for our final submission, we pick the model that performs best on the development set 
When testing our models, we split the labelled training data into two non-overlapping parts: the first one, consisting of 75% of the training data is used to train models, whereas the other is used to test the effectiveness of the models  All models are trained and tested on the same split to ensure comparability  Similarly, to ensure that our models remain comparable, we continue to train on the same 75% of the training set even when testing on the development set 
Once the best model is found using these methods, we train that model on all of the training data available before then submitting the results on the development set to the leaderboard  These results are detailed in the section describing our results (Section SECREF5) 
Methodology ::: Class Imbalance in Sentence Level Classification
The sentence level classification task is an imbalanced binary classification problem that we address using BERT BIBREF0  We use BERTBASE, uncased, which consists of 12 self-attention layers, and returns a 768-dimension vector that representation a sentence  So as to make use of BERT for sentence classification, we include a fully connected layer on top of the BERT self-attention layers, which classifies the sentence embedding provided by BERT into the two classes of interest (propaganda or non-propaganda) 
We attempt to exploit various data augmentation techniques to address the problem of class imbalance  Table TABREF17 shows the results of our experiments for different data augmentation techniques when, after shuffling the training data, we train the model on 75% of the training data and test it on the remaining 25% of the training data and the development data 
We observe that BERT without augmentation consistently outperforms BERT with augmentation in the experiments when the model is trained on 75% of the training data and evaluated on the rest, i e trained and evaluated on similar data, coming from the same distribution  This is consistent with observations by Wei et al  wei2019eda that contextual word embeddings do not gain from data augmentation  The fact that we shuffle the training data prior to splitting it into training and testing subsets could imply that the model is learning to associate topic words, such as `Mueller', as propaganda  However, when we perform model evaluation using the development set, which is dissimilar to the training, we observe that synonym insertion and word dropping techniques also do not bring performance gains, while random oversampling increases performance over base BERT by 4%  Synonym insertion provides results very similar to base BERT, while random deletion harms model performance producing lower scores  We believe that this could be attributed to the fact that synonym insertion and random word dropping involve the introduction of noise to the data, while oversampling does not  As we are working with natural language data, this type of noise can in fact change the meaning of the sentence  Oversampling on the other hand purely increases the importance of the minority class by repeating training on the unchanged instances 
So as to better understand the aspects of oversampling that contribute to these gains, we perform a class-wise performance analysis of BERT with/without oversampling  The results of these experiments (Table TABREF18) show that oversampling increases the overall recall while maintaining precision  This is achieved by significantly improving the recall of the minority class (propaganda) at the cost of the recall of the majority class 
So far we have been able to establish that a) the training and test sets are dissimilar, thus requiring us to generalise our model, b) oversampling provides a method of generalisation, and c) oversampling does this while maintaining recall on the minority (and thus more interesting) class

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which natural language(s) are studied in this paper?
Context 3: 
Given this we explore alternative methods of increasing minority class recall without a significant drop in precision  One such method is cost-sensitive classification, which differs from random oversampling in that it provides a more continuous-valued and consistent method of weighting samples of imbalanced training data; for example, random oversampling will inevitably emphasise some training instances at the expense of others  We detail our methods of using cost-sensitive classification in the next section  Further experiments with oversampling might have provided insights into the relationships between these methods, which we leave for future exploration 
Methodology ::: Cost-sensitive Classification
As discussed in Section SECREF10, cost-sensitive classification can be performed by weighting the cost function  We increase the weight of incorrectly labelling a propagandistic sentence by altering the cost function of the training of the final fully connected layer of our model previously described in Section SECREF16  We make these changes through the use of PyTorch BIBREF16 which calculates the cross-entropy loss for a single prediction $x$, an array where the $j^{th}$ element represents the models prediction for class $j$, labelled with the class $class$ as given by Equation DISPLAY_FORM20 
The cross-entropy loss given in Equation DISPLAY_FORM20 is modified to accommodate an array $weight$, the $i^{th}$ element of which represents the weight of the $i^{th}$ class, as described in Equation DISPLAY_FORM21 
Intuitively, we increase the cost of getting the classification of an “important” class wrong and corresponding decrees the cost of getting a less important class wrong  In our case, we increase the cost of mislabelling the minority class which is “propaganda” (as opposed to “non-propaganda”) 
We expect the effect of this to be similar to that of oversampling, in that it is likely to enable us to increase the recall of the minority class thus resulting in the decrease in recall of the overall model while maintaining high precision  We reiterate that this specific change to a model results in increasing the model's ability to better identify elements belonging to the minority class in dissimilar datasets when using BERT 
We explore the validity of this by performing several experiments with different weights assigned to the minority class  We note that in our experiments use significantly higher weights than the weights proportional to class frequencies in the training data, that are common in literature BIBREF17  Rather than directly using the class proportions of the training set, we show that tuning weights based on performance on the development set is more beneficial  Figure FIGREF22 shows the results of these experiments wherein we are able to maintain the precision on the subset of the training set used for testing while reducing its recall and thus generalising the model  The fact that the model is generalising on a dissimilar dataset is confirmed by the increase in the development set F1 score  We note that the gains are not infinite and that a balance must be struck based on the amount of generalisation and the corresponding loss in accuracy  The exact weight to use for the best transfer of classification accuracy is related to the dissimilarity of that other dataset and hence is to be obtained experimentally through hyperparameter search  Our experiments showed that a value of 4 is best suited for this task 
We do not include the complete results of our experiments here due to space constraints but include them along with charts and program code on our project website  Based on this exploration we find that the best weights for this particular dataset are 1 for non-propaganda and 4 for propaganda and we use this to train the final model used to submit results to the leaderboard  We also found that adding Part of Speech tags and Named Entity information to BERT embeddings by concatenating these one-hot vectors to the BERT embeddings does not improve model performance  We describe these results in Section SECREF5 
Methodology ::: Fragment-level classification (FLC)
In addition to participating in the Sentence Level Classification task we also participate in the Fragment Level Classification task  We note that extracting fragments that are propagandistic is similar to the task of Named Entity Recognition, in that they are both span extraction tasks, and so use a BERT based model designed for this task - We build on the work by BIBREF18 which makes use of Continuous Random Field stacked on top of an LSTM to predict spans  This architecture is standard amongst state of the art models that perform span identification 
While the same span of text cannot have multiple named entity labels, it can have different propaganda labels  We get around this problem by picking one of the labels at random  Additionally, so as to speed up training, we only train our model on those sentences that contain some propagandistic fragment  In hindsight, we note that both these decisions were not ideal and discuss what we might have otherwise done in Section SECREF7 
Results
In this section, we show our rankings on the leaderboard on the test set  Unlike the previous exploratory sections, in which we trained our model on part of the training set, we train models described in this section on the complete training set 
Results ::: Results on the SLC task
Our best performing model, selected on the basis of a systematic analysis of the relationship between cost weights and recall, places us second amongst the 25 teams that submitted their results on this task  We present our score on the test set alongside those of comparable teams in Table TABREF25  We note that the task description paper BIBREF4 describes a method of achieving an F1 score of 60 98% on a similar task although this reported score is not directly comparable to the results on this task because of the differences in testing sets 
Results ::: Results on the FLC task
We train the model described in Section SECREF23 on the complete training set before submitting to the leaderboard  Our best performing model was placed 7th amongst the 13 teams that submitted results for this task  We present our score on the test set alongside those of comparable teams in Table TABREF27  We note that the task description paper BIBREF4 describes a method of achieving an F1 score of 22 58% on a similar task although, this reported score is not directly comparable to the results on this task 
One of the major setbacks to our method for identifying sentence fragments was the loss of training data as a result of randomly picking one label when the same fragment had multiple labels  This could have been avoided by training different models for each label and simply concatenating the results  Additionally, training on all sentences, including those that did not contain any fragments labelled as propagandistic would have likely improved our model performance  We intend to perform these experiments as part of our ongoing research 
Issues of Decontextualization in Automated Propaganda Detection
It is worth reflecting on the nature of the shared task dataset (PTC corpus) and its structural correspondence (or lack thereof) to some of the definitions of propaganda mentioned in the introduction  First, propaganda is a social phenomenon and takes place as an act of communication BIBREF19, and so it is more than a simple information-theoretic message of zeros and ones—it also incorporates an addresser and addressee(s), each in phatic contact (typically via broadcast media), ideally with a shared denotational code and contextual surround(s) BIBREF20 
As such, a dataset of decontextualised documents with labelled sentences, devoid of authorial or publisher metadata, has taken us at some remove from even a simple everyday definition of propaganda  Our models for this shared task cannot easily incorporate information about the addresser or addressee; are left to assume a shared denotational code between author and reader (one perhaps simulated with the use of pre-trained word embeddings); and they are unaware of when or where the act(s) of propagandistic communication took place  This slipperiness is illustrated in our example document (Fig  FIGREF13): note that while Sentences 3 and 7, labelled as propaganda, reflect a propagandistic attitude on the part of the journalist and/or publisher, Sentence 4—also labelled as propaganda in the training data—instead reflects a “flag-waving" propagandistic attitude on the part of U S  congressman Jeff Flake, via the conventions of reported speech BIBREF21  While reported speech often is signaled by specific morphosyntactic patterns (e g  the use of double-quotes and “Flake said") BIBREF22, we argue that human readers routinely distinguish propagandistic reportage from the propagandastic speech acts of its subjects, and to conflate these categories in a propaganda detection corpus may contribute to the occurrence of false positives/negatives 
Conclusions and Future Work
In this work we have presented a method of incorporating cost-sensitivity into BERT to allow for better generalisation and additionally, we provide a simple measure of corpus similarity to determine when this method is likely to be useful  We intend to extend our analysis of the ability to generalise models to less similar data by experimenting on other datasets and models  We hope that the release of program code and documentation will allow the research community to help in this experimentation while exploiting these methods 
Acknowledgements
We would like to thank Dr Leandro Minku from the University of Birmingham for his insights into and help with the statistical analysis presented in this paper 
This work was also partially supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Work by Elena Kochkina was partially supported by the Leverhulme Trust through the Bridges Programme and Warwick CDT for Urban Science & Progress under the EPSRC Grant Number EP/L016400/1 

ground_truth: Unanswerable

English