prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What labels do they create on their dataset?
Context 1: Problem Statement
Spoken conversations still remain the most natural and effortless means of human communication  Thus a lot of valuable information is conveyed and exchanged in such an unstructured form  In telehealth settings, nurses might call discharged patients who have returned home to continue to monitor their health status  Human language technology that can efficiently and effectively extract key information from such conversations is clinically useful, as it can help streamline workflow processes and digitally document patient medical information to increase staff productivity  In this work, we design and prototype a dialogue comprehension system in the question-answering manner, which is able to comprehend spoken conversations between nurses and patients to extract clinical information 
Motivation of Approach
Machine comprehension of written passages has made tremendous progress recently  Large quantities of supervised training data for reading comprehension (e g  SQuAD BIBREF0 ), the wide adoption and intense experimentation of neural modeling BIBREF1 , BIBREF2 , and the advancements in vector representations of word embeddings BIBREF3 , BIBREF4 all contribute significantly to the achievements obtained so far  The first factor, the availability of large scale datasets, empowers the latter two factors  To date, there is still very limited well-annotated large-scale data suitable for modeling human-human spoken dialogues  Therefore, it is not straightforward to directly port over the recent endeavors in reading comprehension to dialogue comprehension tasks 
In healthcare, conversation data is even scarcer due to privacy issues  Crowd-sourcing is an efficient way to annotate large quantities of data, but less suitable for healthcare scenarios, where domain knowledge is required to guarantee data quality  To demonstrate the feasibility of a dialogue comprehension system used for extracting key clinical information from symptom monitoring conversations, we developed a framework to construct a simulated human-human dialogue dataset to bootstrap such a prototype  Similar efforts have been conducted for human-machine dialogues for restaurant or movie reservations BIBREF5   To the best of our knowledge, no one to date has done so for human-human conversations in healthcare 
Human-human Spoken Conversations
Human-human spoken conversations are a dynamic and interactive flow of information exchange  While developing technology to comprehend such spoken conversations presents similar technical challenges as machine comprehension of written passages BIBREF6 , the challenges are further complicated by the interactive nature of human-human spoken conversations:
(1) Zero anaphora is more common: Co-reference resolution of spoken utterances from multiple speakers is needed  For example, in Figure FIGREF5 (a) headaches, the pain, it, head bulging all refer to the patient's headache symptom, but they were uttered by different speakers and across multiple utterances and turns  In addition, anaphors are more likely to be omitted (see Figure FIGREF5 (a) A4) as this does not affect the human listener’s understanding, but it might be challenging for computational models 
(2) Thinking aloud more commonly occurs: Since it is more effortless to speak than to type, one is more likely to reveal her running thoughts when talking  In addition, one cannot retract what has been uttered, while in text communications, one is more likely to confirm the accuracy of the information in a written response and revise if necessary before sending it out  Thinking aloud can lead to self-contradiction, requiring more context to fully understand the dialogue; e g , in A6 in Figure FIGREF5 (a), the patient at first says he has none of the symptoms asked, but later revises his response saying that he does get dizzy after running 
(3) Topic drift is more common and harder to detect in spoken conversations: An example is shown in Figure FIGREF5 (a) in A3, where No is actually referring to cough in the previous question, and then the topic is shifted to headache  In spoken conversations, utterances are often incomplete sentences so traditional linguistic features used in written passages such as punctuation marks indicating syntactic boundaries or conjunction words suggesting discourse relations might no longer exist 
Dialogue Comprehension Task
Figure FIGREF5 (b) illustrates the proposed dialogue comprehension task using a question answering (QA) model  The input are a multi-turn symptom checking dialogue INLINEFORM0 and a query INLINEFORM1 specifying a symptom with one of its attributes; the output is the extracted answer INLINEFORM2 from the given dialogue  A training or test sample is defined as INLINEFORM3   Five attributes, specifying certain details of clinical significance, are defined to characterize the answer types of INLINEFORM4 : (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom  For each symptom/attribute, it can take on different linguistic expressions, defined as entities  Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in BIBREF6  
Reading Comprehension
Large-scale reading comprehension tasks like SQuAD BIBREF0 and MARCO BIBREF7 provide question-answer pairs from a vast range of written passages, covering different kinds of factual answers involving entities such as location and numerical values  Furthermore, HotpotQA BIBREF8 requires multi-step inference and provides numerous answer types  CoQA BIBREF9 and QuAC BIBREF10 are designed to mimic multi-turn information-seeking discussions of the given material  In these tasks, contextual reasoning like coreference resolution is necessary to grasp rich linguistic patterns, encouraging semantic modeling beyond naive lexical matching  Neural networks contribute to impressive progress in semantic modeling: distributional semantic word embeddings BIBREF3 , contextual sequence encoding BIBREF11 , BIBREF12 and the attention mechanism BIBREF13 , BIBREF14 are widely adopted in state-of-the-art comprehension models BIBREF1 , BIBREF2 , BIBREF4  
While language understanding tasks in dialogue such as domain identification BIBREF15 , slot filling BIBREF16 and user intent detection BIBREF17 have attracted much research interest, work in dialogue comprehension is still limited, if any  It is labor-intensive and time-consuming to obtain a critical mass of annotated conversation data for computational modeling  Some propose to collect text data from human-machine or machine-machine dialogues BIBREF18 , BIBREF5   In such cases, as human speakers are aware of current limitations of dialogue systems or due to pre-defined assumptions of user simulators, there are fewer cases of zero anaphora, thinking aloud, and topic drift, which occur more often in human-human spoken interactions 
NLP for Healthcare
There is emerging interest in research and development activities at the intersection of machine learning and healthcare , of which much of the NLP related work are centered around social media or online forums (e g , BIBREF19 , BIBREF20 ), partially due to the world wide web as a readily available source of information  Other work in this area uses public data sources such as MIMIC in electronic health records: text classification approaches have been applied to analyze unstructured clinical notes for ICD code assignment BIBREF21 and automatic intensive emergency prediction BIBREF22   Sequence-to-sequence textual generation has been used for readable notes based on medical and demographic recordings BIBREF23   For mental health, there has been more focus on analyzing dialogues  For example, sequential modeling of audio and text have helped detect depression from human-machine interviews BIBREF24   However, few studies have examined human-human spoken conversations in healthcare settings 
Data Preparation
We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital  The clinical data was acquired by the Health Management Unit at Changi General Hospital  This research study was approved by the SingHealth Centralised Institutional Review Board (Protocol 1556561515)  The patients were recruited during 2014-2016 as part of their routine care delivery, and enrolled into the telemonitoring health management program with consent for use of anonymized versions of their data for research 
The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research  The speakers are 38 to 88 years old, equally distributed across gender, and comprise a range of ethnic groups (55% Chinese, 17% Malay, 14% Indian, 3% Eurasian, and 11% unspecified)  The conversations cover 11 topics (e g , medication compliance, symptom checking, education, greeting) and 9 symptoms (e g , chest pain, cough) and amount to 41 hours 
Data preprocessing and anonymization were performed by a data preparation team, separate from the data analysis team to maintain data confidentiality  The data preparation team followed standard speech recognition transcription guidelines, where words are transcribed verbatim to include false starts, disfluencies, mispronunciations, and private self-talk  Confidential information were marked and clipped off from the audio and transcribed with predefined tags in the annotation  Conversation topics and clinical symptoms were also annotated and clinically validated by certified telehealth nurses 
Linguistic Characterization on Seed Data
To analyze the linguistic structure of the inquiry-response pairs in the entire 41-hour dataset, we randomly sampled a seed dataset consisting of 1,200 turns and manually categorized them to different types, which are summarized in Table TABREF14 along with the corresponding occurrence frequency statistics  Note that each given utterance could be categorized to more than one type  We elaborate on each utterance type below 
Open-ended Inquiry: Inquiries about general well-being or a particular symptom; e g , “How are you feeling?” and “Do you cough?”
Detailed Inquiry: Inquiries with specific details that prompt yes/no answers or clarifications; e g , “Do you cough at night?”
Multi-Intent Inquiry: Inquiring more than one symptom in a question; e g , “Any cough, chest pain, or headache?”
Reconfirmation Inquiry: The nurse reconfirms particular details; e g , “Really? At night?” and “Serious or mild?”  This case is usually related to explicit or implicit coreferencing 
Inquiry with Transitional Clauses: During spoken conversations, one might repeat what the other party said, but it is unrelated to the main clause of the question  This is usually due to private self-talk while thinking aloud, and such utterances form a transitional clause before the speaker starts a new topic; e g , “Chest pain    no chest pain, I see    any cough?” 
Yes/No Response: Yes/No responses seem straightforward, but sometimes lead to misunderstanding if one does not interpret the context appropriately  One case is tag questions: A:“You don't cough at night, do you?” B:`Yes, yes” A:“cough at night?” B:“No, no cough”  Usually when the answer is unclear, clarifying inquiries will be asked for reconfirmation purposes 
Detailed Response: Responses that contain specific information of one symptom, like “I felt tightness in my chest” 
Response with Revision: Revision is infrequent but can affect comprehension significantly  One cause is thinking aloud so a later response overrules the previous one; e g , “No dizziness, oh wait    last week I felt a bit dizzy when biking” 
Response with Topic Drift: When a symptom/topic like headache is inquired, the response might be: “Only some chest pain at night”, not referring to the original symptom (headache) at all 
Response with Transitional Clauses: Repeating some of the previous content, but often unrelated to critical clinical information and usually followed by topic drift  For example, “Swelling    swelling    I don't cough at night” 
Simulating Symptom Monitoring Dataset for Training
We divide the construction of data simulation into two stages  In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification  In Section SECREF20 , we present our proposed framework for generating simulated training data  The templates and framework are verified for logical correctness and clinical soundness 
Template Construction
Each utterance in the seed data is categorized according to Table TABREF14 and then abstracted into templates by replacing entity phrases like cough and often with respective placeholders “#symptom#” and “#frequency#”  The templates are refined through verifying logical correctness and injecting expression diversity by linguistically trained researchers  As these replacements do not alter the syntactic structure, we interchange such placeholders with various verbal expressions to enlarge the simulated training set in Section SECREF20   Clinical validation was also conducted by certified telehealth nurses 
For the 9 symptoms (e g  chest pain, cough) and 5 attributes (e g , extent, frequency), we collect various expressions from the seed data, and expand them through synonym replacement  Some attributes are unique to a particular symptom; e g , “left leg” in #location# is only suitable to describe the symptom swelling, but not the symptom headache  Therefore, we only reuse general expressions like “slight” in #extent# across different symptoms to diversify linguistic expressions 
Two linguistically trained researchers constructed expression pools for each symptom and each attribute to account for different types of paraphrasing and descriptions  These expression pools are used in Section SECREF20 (c) 
Simulated Data Generation Framework
Figure FIGREF15 shows the five steps we use to generate multi-turn symptom monitoring dialogue samples 
(a) Topic Selection: While nurses might prefer to inquire the symptoms in different orders depending on the patient's history, our preliminary analysis shows that modeling results do not differ noticeably if topics are of equal prior probabilities  Thus we adopt this assumption for simplicity 
(b) Template Selection: For each selected topic, one inquiry template and one response template are randomly chosen to compose a turn  To minimize adverse effects of underfitting, we redistributed the frequency distribution in Table TABREF14 : For utterance types that are below 15%, we boosted them to 15%, and the overall relative distribution ranking is balanced and consistent with Table TABREF14  
(c) Enriching Linguistic Expressions: The placeholders in the selected templates are substituted with diverse expressions from the expression pools in Section UID19 to characterize the symptoms and their corresponding attributes 
(d) Multi-Turn Dialogue State Tracking: A greedy algorithm is applied to complete conversations  A “completed symptoms” list and a “to-do symptoms” list are used for symptom topic tracking  We also track the “completed attributes" and “to-do attributes"  For each symptom, all related attributes are iterated  A dialogue ends only when all possible entities are exhausted, generating a multi-turn dialogue sample, which encourages the model to learn from the entire discussion flow rather than a single turn to comprehend contextual dependency  The average length of a simulated dialogue is 184 words, which happens to be twice as long as an average dialogue from the real-world evaluation set  Moreover, to model the roles of the respondents, we set the ratio between patients and caregivers to 2:1; this statistic is inspired by the real scenarios in the seed dataset  For both the caregivers and patients, we assume equal probability of both genders  The corresponding pronouns in the conversations are thus determined by the role and gender of these settings 
(e) Multi-Turn Sample Annotation: For each multi-turn dialogue, a query is specified by a symptom and an attribute  The groundtruth output of the QA system is automatically labeled based on the template generation rules, but also manually verified to ensure annotation quality  Moreover, we adopt the unanswerable design in BIBREF6 : when the patient does not mention a particular symptom, the answer is defined as “No Answer”  This process is repeated until all logical permutations of symptoms and attributes are exhausted 
Experiments
Model Design
We implemented an established model in reading comprehension, a bi-directional attention pointer network BIBREF1 , and equipped it with an answerable classifier, as depicted in Figure FIGREF21   First, tokens in the given dialogue INLINEFORM0 and query INLINEFORM1 are converted into embedding vectors  Then the dialogue embeddings are fed to a bi-directional LSTM encoding layer, generating a sequence of contextual hidden states  Next, the hidden states and query embeddings are processed by a bi-directional attention layer, fusing attention information in both context-to-query and query-to-context directions  The following two bi-directional LSTM modeling layers read the contextual sequence with attention  Finally, two respective linear layers with softmax functions are used to estimate token INLINEFORM2 's INLINEFORM3 and INLINEFORM4 probability of the answer span INLINEFORM5  
In addition, we add a special tag “[SEQ]” at the head of INLINEFORM0 to account for the case of “No answer” BIBREF4 and adopt an answerable classifier as in BIBREF25   More specifically, when the queried symptom or attribute is not mentioned in the dialogue, the answer span should point to the tag “[SEQ]” and answerable probability should be predicted as 0 
Implementation Details
The model was trained via gradient backpropagation with the cross-entropy loss function of answer span prediction and answerable classification, optimized by Adam algorithm BIBREF26 with initial learning rate of INLINEFORM0   Pre-trained GloVe BIBREF3 embeddings (size INLINEFORM1 ) were used  We re-shuffled training samples at each epoch (batch size INLINEFORM2 )  Out-of-vocabulary words ( INLINEFORM3 ) were replaced with a fixed random vector  L2 regularization and dropout (rate INLINEFORM4 ) were used to alleviate overfitting BIBREF27  
Evaluation Setup
To evaluate the effectiveness of our linguistically-inspired simulation approach, the model is trained on the simulated data (see Section SECREF20 )  We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data  (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data)  (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples 
Results
Evaluation results are in Table TABREF25 with exact match (EM) and F1 score in BIBREF0 metrics  To distinguish the correct answer span from the plausible ones which contain the same words, we measure the scores on the position indices of tokens  Our results show that both EM and F1 score increase with training sample size growing and the optimal size in our setting is 100k  The best-trained model performs well on both the Base Set and the Augmented Set, indicating that out-of-distribution symptoms do not affect the comprehension of existing symptoms and outputs reasonable answers for both in- and out-of-distribution symptoms  On the Real-World Set, we obtained 78 23 EM score and 80 18 F1 score respectively 
Error analysis suggests the performance drop from the simulated test sets is due to the following: 1) sparsity issues resulting from the expression pools excluding various valid but sporadic expressions  2) nurses and patients occasionally chit-chat in the Real-World Set, which is not simulated in the training set  At times, these chit-chats make the conversations overly lengthy, causing the information density to be lower  These issues could potentially distract and confuse the comprehension model  3) an interesting type of infrequent error source, caused by patients elaborating on possible causal relations of two symptoms  For example, a patient might say “My giddiness may be due to all this cough”  We are currently investigating how to close this performance gap efficiently 
Ablation Analysis
To assess the effectiveness of bi-directional attention, we bypassed the bi-attention layer by directly feeding the contextual hidden states and query embeddings to the modeling layer  To evaluate the pre-trained GloVe embeddings, we randomly initialized and trained the embeddings from scratch  These two procedures lead to 10% and 18% performance degradation on the Augmented Set and Real-World Set, respectively (see Table TABREF27 ) 
Conclusion
We formulated a dialogue comprehension task motivated by the need in telehealth settings to extract key clinical information from spoken conversations between nurses and patients  We analyzed linguistic characteristics of real-world human-human symptom checking dialogues, constructed a simulated dataset based on linguistically inspired and clinically validated templates, and prototyped a QA system  The model works effectively on a simulated test set using symptoms excluded during training and on real-world conversations between nurses and patients  We are currently improving the model's dialogue comprehension capability in complex reasoning and context understanding and also applying the QA model to summarization and virtual nurse applications 
Acknowledgements
Research efforts were supported by funding for Digital Health and Deep Learning I2R (DL2 SSF Project No: A1718g0045) and the Science and Engineering Research Council (SERC Project No: A1818g0044), A*STAR, Singapore  In addition, this work was conducted using resources and infrastructure provided by the Human Language Technology unit at I2R  The telehealth data acquisition was funded by the Economic Development Board (EDB), Singapore Living Lab Fund and Philips Electronics – Hospital to Home Pilot Project (EDB grant reference number: S14-1035-RF-LLF H and W) 
We acknowledge valuable support and assistance from Yulong Lin, Yuanxin Xiang, and Ai Ti Aw at the Institute for Infocomm Research (I2R); Weiliang Huang at the Changi General Hospital (CGH) Department of Cardiology, Hong Choon Oh at CGH Health Services Research, and Edris Atikah Bte Ahmad, Chiu Yan Ang, and Mei Foon Yap of the CGH Integrated Care Department 
We also thank Eduard Hovy and Bonnie Webber for insightful discussions and the anonymous reviewers for their precious feedback to help improve and extend this piece of work 
Context 3: Introduction and related work
In recent years there has been increasing interest on the issue of disinformation spreading on online social media  Global concern over false (or "fake") news as a threat to modern democracies has been frequently raised–ever since 2016 US Presidential elections–in correspondence of events of political relevance, where the proliferation of manipulated and low-credibility content attempts to drive and influence people opinions BIBREF0BIBREF1BIBREF2BIBREF3 
Researchers have highlighted several drivers for the diffusion of such malicious phenomenon, which include human factors (confirmation bias BIBREF4, naive realism BIBREF5), algorithmic biases (filter bubble effect BIBREF0), the presence of deceptive agents on social platforms (bots and trolls BIBREF6) and, lastly, the formation of echo chambers BIBREF7 where people polarize their opinions as they are insulated from contrary perspectives 
The problem of automatically detecting online disinformation news has been typically formulated as a binary classification task (i e  credible vs non-credible articles), and tackled with a variety of different techniques, based on traditional machine learning and/or deep learning, which mainly differ in the dataset and the features they employ to perform the classification  We may distinguish three approaches: those built on content-based features, those based on features extracted from the social context, and those which combine both aspects  A few main challenges hinder the task, namely the impossibility to manually verify all news items, the lack of gold-standard datasets and the adversarial setting in which malicious content is created BIBREF3BIBREF6 
In this work we follow the direction pointed out in a few recent contributions on the diffusion of disinformation compared to traditional and objective information  These have shown that false news spread faster and deeper than true news BIBREF8, and that social bots and echo chambers play an important role in the diffusion of malicious content BIBREF6, BIBREF7  Therefore we focus on the analysis of spreading patterns which naturally arise on social platforms as a consequence of multiple interactions between users, due to the increasing trend in online sharing of news BIBREF0 
A deep learning framework for detection of fake news cascades is provided in BIBREF9, where the authors refer to BIBREF8 in order to collect Twitter cascades pertaining to verified false and true rumors  They employ geometric deep learning, a novel paradigm for graph-based structures, to classify cascades based on four categories of features, such as user profile, user activity, network and spreading, and content  They also observe that a few hours of propagation are sufficient to distinguish false news from true news with high accuracy  Diffusion cascades on Weibo and Twitter are analyzed in BIBREF10, where authors focus on highlighting different topological properties, such as the number of hops from the source or the heterogeneity of the network, to show that fake news shape diffusion networks which are highly different from credible news, even at early stages of propagation 
In this work, we consider the results of BIBREF11 as our baseline  The authors use off-the-shelf machine learning classifiers to accurately classify news articles leveraging Twitter diffusion networks  To this aim, they consider a set of basic features which can be qualitatively interpreted w r t to the social behavior of users sharing credible vs non-credible information  Their methodology is overall in accordance with BIBREF12, where authors successfully detect Twitter astroturfing content, i e  political campaigns disguised as spontaneous grassroots, with a machine learning framework based on network features 
In this paper, we propose a classification framework based on a multi-layer formulation of Twitter diffusion networks  For each article we disentangle different social interactions on Twitter, namely tweets, retweets, mentions, replies and quotes, to accordingly build a diffusion network composed of multiple layers (on for each type of interaction), and we compute structural features separately for each layer  We pick a set of global network properties from the network science toolbox which can be qualitatively explained in terms of social dimensions and allow us to encode different networks with a tuple of features  These include traditional indicators, e g  network density, number of strong/weak connected components and diameter, and more elaborated ones such as main K-core number BIBREF13 and structural virality BIBREF14  Our main research question is whether the use of a multi-layer, disentangled network yields a significant advance in terms of classification accuracy over a conventional single-layer diffusion network  Additionally, we are interested in understanding which of the above features, and in which layer, are most effective in the classification task 
We perform classification experiments with an off-the-shelf Logistic Regression model on two different datasets of mainstream and disinformation news shared on Twitter respectively in the United States and in Italy during 2019  In the former case we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets  Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles)  We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains  We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries 
The outline of this paper is the following: we first formulate the problem and describe data collection, network representation and structural properties employed for the classification; then we provide experimental results–classification performances, layer and feature importance analyses and a temporal classification evaluation–and finally we draw conclusions and future directions 
Methodology ::: Disinformation and mainstream news
In this work we formulate our classification problem as follows: given two classes of news articles, respectively $D$ (disinformation) and $M$ (mainstream), a set of news articles $A_i$ and associated class labels $C_i \in \lbrace D,M\rbrace $, and a set of tweets $\Pi _i=\lbrace T_i^1, T_i^2,    \rbrace $ each of which contains an Uniform Resource Locator (URL) pointing explicitly to article $A_i$, predict the class $C_i$ of each article $A_i$  There is huge debate and controversy on a proper taxonomy of malicious and deceptive information BIBREF1BIBREF2BIBREF15BIBREF16BIBREF17BIBREF3BIBREF11  In this work we prefer the term disinformation to the more specific fake news to refer to a variety of misleading and harmful information  Therefore, we follow a source-based approach, a consolidated strategy also adopted by BIBREF6BIBREF16BIBREF2BIBREF1, in order to obtain relevant data for our analysis  We collected:
Disinformation articles, published by websites which are well-known for producing low-credibility content, false and misleading news reports as well as extreme propaganda and hoaxes and flagged as such by reputable journalists and fact-checkers;
Mainstream news, referring to traditional news outlets which deliver factual and credible information 
We believe that this is currently the most reliable classification approach, but it entails obvious limitations, as disinformation outlets may also publish true stories and likewise misinformation is sometimes reported on mainstream media  Also, given the choice of news sources, we cannot test whether our methodology is able to classify disinformation vs factual but not mainstream news which are published on niche, non-disinformation outlets 
Methodology ::: US dataset
We collected tweets associated to a dozen US mainstream news websites, i e  most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets  We filtered out articles associated to less than 50 tweets  The resulting dataset contains overall $\sim $1 7 million tweets for mainstream news, collected in a period of three weeks (February 25th, 2019-March 18th, 2019), which are associated to 6,978 news articles, and $\sim $1 6 million tweets for disinformation, collected in a period of three months (January 1st, 2019-March 18th, 2019) for sake of balance of the two classes, which hold 5,775 distinct articles  Diffusion censoring effects BIBREF14 were correctly taken into account in both collection procedures  We provide in Figure FIGREF4 the distribution of articles by source and political bias for both news domains 
As it is reported that conservatives and liberals exhibit different behaviors on online social platforms BIBREF19BIBREF20BIBREF21, we further assigned a political bias label to different US outlets (and therefore news articles) following the procedure described in BIBREF2  In order to assess the robustness of our method, we performed classification experiments by training only on left-biased (or right-biased) outlets of both disinformation and mainstream domains and testing on the entire set of sources, as well as excluding particular sources that outweigh the others in terms of samples to avoid over-fitting 
Methodology ::: Italian dataset
For what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa)  We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites  In order to get balanced classes (April 5th, 2019-May 5th, 2019), we retained data collected in a longer period w r

ground_truth: (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom, No Answer

the time the patient has been experiencing the symptom, activities that trigger the symptom, the extent of seriousness, the frequency occurrence of the symptom, the location of symptom, 9 symptoms

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What model did they use for their system?
Context 2: Introduction
Over the past two decades, the emergence of social media has enabled the proliferation of traceable human behavior  The content posted by users can reflect who their friends are, what topics they are interested in, or which company they are working for  At the same time, users are listing a number of profile fields to define themselves to others  The utilization of such metadata has proven important in facilitating further developments of applications in advertising BIBREF0 , personalization BIBREF1 , and recommender systems BIBREF2   However, profile information can be limited, depending on the platform, or it is often deliberately omitted BIBREF3   To uncloak this information, a number of studies have utilized social media users' footprints to approximate their profiles 
This paper explores the potential of predicting a user's industry –the aggregate of enterprises in a particular field– by identifying industry indicative text in social media  The accurate prediction of users' industry can have a big impact on targeted advertising by minimizing wasted advertising BIBREF4 and improved personalized user experience  A number of studies in the social sciences have associated language use with social factors such as occupation, social class, education, and income BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8   An additional goal of this paper is to examine such findings, and in particular the link between language and occupational class, through a data-driven approach 
In addition, we explore how meaning changes depending on the occupational context  By leveraging word embeddings, we seek to quantify how, for example, cloud might mean a separate concept (e g , condensed water vapor) in the text written by users that work in environmental jobs while it might be used differently by users in technology occupations (e g , Internet-based computing) 
Specifically, this paper makes four main contributions  First, we build a large, industry-annotated dataset that contains over 20,000 blog users  In addition to their posted text, we also link a number of user metadata including their gender, location, occupation, introduction and interests 
Second, we build content-based classifiers for the industry prediction task and study the effect of incorporating textual features from the users' profile metadata using various meta-classification techniques, significantly improving both the overall accuracy and the average per industry accuracy 
Next, after examining which words are indicative for each industry, we build vector-space representations of word meanings and calculate one deviation for each industry, illustrating how meaning is differentiated based on the users' industries  We qualitatively examine the resulting industry-informed semantic representations of words by listing the words per industry that are most similar to job related and general interest terms 
Finally, we rank the different industries based on the normalized relative frequencies of emotionally charged words (positive and negative) and, in addition, discover that, for both genders, these frequencies do not statistically significantly correlate with an industry's gender dominance ratio 
After discussing related work in Section SECREF2 , we present the dataset used in this study in Section SECREF3   In Section SECREF4 we evaluate two feature selection methods and examine the industry inference problem using the text of the users' postings  We then augment our content-based classifier by building an ensemble that incorporates several metadata classifiers  We list the most industry indicative words and expose how each industrial semantic field varies with respect to a variety of terms in Section SECREF5   We explore how the frequencies of emotionally charged words in each gender correlate with the industries and their respective gender dominance ratio and, finally, conclude in Section SECREF6  
Related Work
Alongside the wide adoption of social media by the public, researchers have been leveraging the newly available data to create and refine models of users' behavior and profiling  There exists a myriad research that analyzes language in order to profile social media users  Some studies sought to characterize users' personality BIBREF9 , BIBREF10 , while others sequenced the expressed emotions BIBREF11 , studied mental disorders BIBREF12 , and the progression of health conditions BIBREF13   At the same time, a number of researchers sought to predict the social media users' age and/or gender BIBREF14 , BIBREF15 , BIBREF16 , while others targeted and analyzed the ethnicity, nationality, and race of the users BIBREF17 , BIBREF18 , BIBREF19   One of the profile fields that has drawn a great deal of attention is the location of a user  Among others, Hecht et al  Hecht11 predicted Twitter users' locations using machine learning on nationwide and state levels  Later, Han et al  Han14 identified location indicative words to predict the location of Twitter users down to the city level 
As a separate line of research, a number of studies have focused on discovering the political orientation of users BIBREF15 , BIBREF20 , BIBREF21   Finally, Li et al  Li14a proposed a way to model major life events such as getting married, moving to a new place, or graduating  In a subsequent study, BIBREF22 described a weakly supervised information extraction method that was used in conjunction with social network information to identify the name of a user's spouse, the college they attended, and the company where they are employed 
The line of work that is most closely related to our research is the one concerned with understanding the relation between people's language and their industry  Previous research from the fields of psychology and economics have explored the potential for predicting one's occupation from their ability to use math and verbal symbols BIBREF23 and the relationship between job-types and demographics BIBREF24   More recently, Huang et al  Huang15 used machine learning to classify Sina Weibo users to twelve different platform-defined occupational classes highlighting the effect of homophily in user interactions  This work examined only users that have been verified by the Sina Weibo platform, introducing a potential bias in the resulting dataset  Finally, Preotiuc-Pietro et al  Preoctiuc15 predicted the occupational class of Twitter users using the Standard Occupational Classification (SOC) system, which groups the different jobs based on skill requirements  In that work, the data collection process was limited to only users that specifically mentioned their occupation in their self-description in a way that could be directly mapped to a SOC occupational class  The mapping between a substring of their self-description and a SOC occupational class was done manually  Because of the manual annotation step, their method was not scalable; moreover, because they identified the occupation class inside a user self-description, only a very small fraction of the Twitter users could be included (in their case, 5,191 users) 
Both of these recent studies are based on micro-blogging platforms, which inherently restrict the number of characters that a post can have, and consequently the way that users can express themselves 
Moreover, both studies used off-the-shelf occupational taxonomies (rather than self-declared occupation categories), resulting in classes that are either too generic (e g , media, welfare and electronic are three of the twelve Sina Weibo categories), or too intermixed (e g , an assistant accountant is in a different class from an accountant in SOC)  To address these limitations, we investigate the industry prediction task in a large blog corpus consisting of over 20K American users, 40K web-blogs, and 560K blog posts 
Dataset
We compile our industry-annotated dataset by identifying blogger profiles located in the U S  on the profile finder on http://www blogger com, and scraping only those users that had the industry profile element completed 
For each of these bloggers, we retrieve all their blogs, and for each of these blogs we download the 21 most recent blog postings  We then clean these blog posts of HTML tags and tokenize them, and drop those bloggers whose cumulative textual content in their posts is less than 600 characters  Following these guidelines, we identified all the U S  bloggers with completed industry information 
Traditionally, standardized industry taxonomies organize economic activities into groups based on similar production processes, products or services, delivery systems or behavior in financial markets  Following such assumptions and regardless of their many similarities, a tomato farmer would be categorized into a distinct industry from a tobacco farmer  As demonstrated in Preotiuc-Pietro et al  Preoctiuc15 such groupings can cause unwarranted misclassifications 
The Blogger platform provides a total of 39 different industry options  Even though a completed industry value is an implicit text annotation, we acknowledge the same problem noted in previous studies: some categories are too broad, while others are very similar  To remedy this and following Guibert et al  Guibert71, who argued that the denominations used in a classification must reflect the purpose of the study, we group the different Blogger industries based on similar educational background and similar technical terminology  To do that, we exclude very general categories and merge conceptually similar ones  Examples of broad categories are the Education and the Student options: a teacher could be teaching in any concentration, while a student could be enrolled in any discipline  Examples of conceptually similar categories are the Investment Banking and the Banking options 
The final set of categories is shown in Table TABREF1 , along with the number of users in each category  The resulting dataset consists of 22,880 users, 41,094 blogs, and 561,003 posts  Table TABREF2 presents additional statistics of our dataset 
Text-based Industry Modeling
After collecting our dataset, we split it into three sets: a train set, a development set, and a test set  The sizes of these sets are 17,880, 2,500, and 2,500 users, respectively, with users randomly assigned to these sets  In all the experiments that follow, we evaluate our classifiers by training them on the train set, configure the parameters and measure performance on the development set, and finally report the prediction accuracy and results on the test set  Note that all the experiments are performed at user level, i e , all the data for one user is compiled into one instance in our data sets 
To measure the performance of our classifiers, we use the prediction accuracy  However, as shown in Table TABREF1 , the available data is skewed across categories, which could lead to somewhat distorted accuracy numbers depending on how well a model learns to predict the most populous classes  Moreover, accuracy alone does not provide a great deal of insight into the individual performance per industry, which is one of the main objectives in this study  Therefore, in our results below, we report: (1) micro-accuracy ( INLINEFORM0 ), calculated as the percentage of correctly classified instances out of all the instances in the development (test) data; and (2) macro-accuracy ( INLINEFORM1 ), calculated as the average of the per-category accuracies, where the per-category accuracy is the percentage of correctly classified instances out of the instances belonging to one category in the development (test) data 
Leveraging Blog Content
In this section, we seek the effectiveness of using solely textual features obtained from the users' postings to predict their industry 
The industry prediction baseline Majority is set by discovering the most frequently featured class in our training set and picking that class in all predictions in the respective development or testing set 
After excluding all the words that are not used by at least three separate users in our training set, we build our AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier  As seen in Figure FIGREF3 , we can far exceed the Majority baseline performance by incorporating basic language signals into machine learning algorithms (173% INLINEFORM0 improvement) 
We additionally explore the potential of improving our text classification task by applying a number of feature ranking methods and selecting varying proportions of top ranked features in an attempt to exclude noisy features  We start by ranking the different features, w, according to their Information Gain Ratio score (IGR) with respect to every industry, i, and training our classifier using different proportions of the top features  INLINEFORM0 INLINEFORM1
Even though we find that using the top 95% of all the features already exceeds the performance of the All Words model on the development data, we further experiment with ranking our features with a more aggressive formula that heavily promotes the features that are tightly associated with any industry category  Therefore, for every word in our training set, we define our newly introduced ranking method, the Aggressive Feature Ranking (AFR), as: INLINEFORM0
In Figure FIGREF3 we illustrate the performance of all four methods in our industry prediction task on the development data  Note that for each method, we provide both the accuracy ( INLINEFORM0 ) and the average per-class accuracy ( INLINEFORM1 )  The Majority and All Words methods apply to all the features; therefore, they are represented as a straight line in the figure  The IGR and AFR methods are applied to varying subsets of the features using a 5% step 
Our experiments demonstrate that the word choice that the users make in their posts correlates with their industry  The first observation in Figure FIGREF3 is that the INLINEFORM0 is proportional to INLINEFORM1 ; as INLINEFORM2 increases, so does INLINEFORM3   Secondly, the best result on the development set is achieved by using the top 90% of the features using the AFR method  Lastly, the improvements of the IGR and AFR feature selections are not substantially better in comparison to All Words (at most 5% improvement between All Words and AFR), which suggest that only a few noisy features exist and most of the words play some role in shaping the “language" of an industry 
As a final evaluation, we apply on the test data the classifier found to work best on the development data (AFR feature selection, top 90% features), for an INLINEFORM0 of 0 534 and INLINEFORM1 of 0 477 
Leveraging User Metadata
Together with the industry information and the most recent postings of each blogger, we also download a number of accompanying profile elements  Using these additional elements, we explore the potential of incorporating users' metadata in our classifiers 
Table TABREF7 shows the different user metadata we consider together with their coverage percentage (not all users provide a value for all of the profile elements)  With the exception of the gender field, the remaining metadata elements shown in Table TABREF7 are completed by the users as a freely editable text field  This introduces a considerable amount of noise in the set of possible metadata values  Examples of noise in the occupation field include values such as “Retired”, “I work ”, or “momma” which are not necessarily informative for our industry prediction task 
To examine whether the metadata fields can help in the prediction of a user's industry, we build classifiers using the different metadata elements  For each metadata element that has a textual value, we use all the words in the training set for that field as features  The only two exceptions are the state field, which is encoded as one feature that can take one out of 50 different values representing the 50 U S  states; and the gender field, which is encoded as a feature with a distinct value for each user gender option: undefined, male, or female 
As shown in Table TABREF9 , we build four different classifiers using the multinomial NB algorithm: Occu (which uses the words found in the occupation profile element), Intro (introduction), Inter (interests), and Gloc (combined gender, city, state) 
In general, all the metadata classifiers perform better than our majority baseline ( INLINEFORM0 of 18 88%)  For the Gloc classifier, this result is in alignment with previous studies BIBREF24   However, the only metadata classifier that outperforms the content classifier is the Occu classifier, which despite missing and noisy occupation values exceeds the content classifier's performance by an absolute 3 2% 
To investigate the promise of combining the five different classifiers we have built so far, we calculate their inter-prediction agreement using Fleiss's Kappa BIBREF25 , as well as the lower prediction bounds using the double fault measure BIBREF26   The Kappa values, presented in the lower left side of Table TABREF10 , express the classification agreement for categorical items, in this case the users' industry  Lower values, especially values below 30%, mean smaller agreement  Since all five classifiers have better-than-baseline accuracy, this low agreement suggests that their predictions could potentially be combined to achieve a better accumulated result 
Moreover, the double fault measure values, which are presented in the top-right hand side of Table TABREF10 , express the proportion of test cases for which both of the two respective classifiers make false predictions, essentially providing the lowest error bound for the pairwise ensemble classifier performance  The lower those numbers are, the greater the accuracy potential of any meta-classification scheme that combines those classifiers  Once again, the low double fault measure values suggest potential gain from a combination of the base classifiers into an ensemble of models 
After establishing the promise of creating an ensemble of classifiers, we implement two meta-classification approaches  First, we combine our classifiers using features concatenation (or early fusion)  Starting with our content-based classifier (Text), we successively add the features derived from each metadata element  The results, both micro- and macro-accuracy, are presented in Table TABREF12   Even though all these four feature concatenation ensembles outperform the content-based classifier in the development set, they fail to outperform the Occu classifier 
Second, we explore the potential of using stacked generalization (or late fusion) BIBREF27   The base classifiers, referred to as L0 classifiers, are trained on different folds of the training set and used to predict the class of the remaining instances  Those predictions are then used together with the true label of the training instances to train a second classifier, referred to as the L1 classifier: this L1 is used to produce the final prediction on both the development data and the test data  Traditionally, stacking uses different machine learning algorithms on the same training data  However in our case, we use the same algorithm (multinomial NB) on heterogeneous data (i e , different types of data such as content, occupation, introduction, interests, gender, city and state) in order to exploit all available sources of information 
The ensemble learning results on the development set are shown in Table TABREF12   We notice a constant improvement for both metrics when adding more classifiers to our ensemble except for the Gloc classifier, which slightly reduces the performance  The best result is achieved using an ensemble of the Text, Occu, Intro, and Inter L0 classifiers; the respective performance on the test set is an INLINEFORM0 of 0 643 and an INLINEFORM1 of 0 564  Finally, we present in Figure FIGREF11 the prediction accuracy for the final classifier for each of the different industries in our test dataset  Evidently, some industries are easier to predict than others  For example, while the Real Estate and Religion industries achieve accuracy figures above 80%, other industries, such as the Banking industry, are predicted correctly in less than 17% of the time  Anecdotal evidence drawn from the examination of the confusion matrix does not encourage any strong association of the Banking class with any other  The misclassifications are roughly uniform across all other classes, suggesting that the users in the Banking industry use language in a non-distinguishing way 
Qualitative Analysis
In this section, we provide a qualitative analysis of the language of the different industries 
Top-Ranked Words
To conduct a qualitative exploration of which words indicate the industry of a user, Table TABREF14 shows the three top-ranking content words for the different industries using the AFR method 
Not surprisingly, the top ranked words align well with what we would intuitively expect for each industry  Even though most of these words are potentially used by many users regardless of their industry in our dataset, they are still distinguished by the AFR method because of the different frequencies of these words in the text of each industry 
Industry-specific Word Similarities
Next, we examine how the meaning of a word is shaped by the context in which it is uttered  In particular, we qualitatively investigate how the speakers' industry affects meaning by learning vector-space representations of words that take into account such contextual information  To achieve this, we apply the contextualized word embeddings proposed by Bamman et al  Bamman14, which are based on an extension of the “skip-gram" language model BIBREF28  
In addition to learning a global representation for each word, these contextualized embeddings compute one deviation from the common word embedding representation for each contextual variable, in this case, an industry option  These deviations capture the terms' meaning variations (shifts in the INLINEFORM0 -dimensional space of the representations, where INLINEFORM1 in our experiments) in the text of the different industries, however all the embeddings are in the same vector space to allow for comparisons to one another 
Using the word representations learned for each industry, we present in Table TABREF16 the terms in the Technology and the Tourism industries that have the highest cosine similarity with a job-related word, customers  Similarly, Table TABREF17 shows the words in the Environment and the Tourism industries that are closest in meaning to a general interest word, food  More examples are given in the Appendix SECREF8  
The terms that rank highest in each industry are noticeably different  For example, as seen in Table TABREF17 , while food in the Environment industry is similar to nutritionally and locally, in the Tourism industry the same word relates more to terms such as delicious and pastries  These results not only emphasize the existing differences in how people in different industries perceive certain terms, but they also demonstrate that those differences can effectively be captured in the resulting word embeddings 
Emotional Orientation per Industry and Gender
As a final analysis, we explore how words that are emotionally charged relate to different industries  To quantify the emotional orientation of a text, we use the Positive Emotion and Negative Emotion categories in the Linguistic Inquiry and Word Count (LIWC) dictionary BIBREF29   The LIWC dictionary contains lists of words that have been shown to correlate with the psychological states of people that use them; for example, the Positive Emotion category contains words such as “happy,” “pretty,” and “good ”
For the text of all the users in each industry we measure the frequencies of Positive Emotion and Negative Emotion words normalized by the text's length  Table TABREF20 presents the industries' ranking for both categories of words based on their relative frequencies in the text of each industry 
We further perform a breakdown per-gender, where we once again calculate the proportion of emotionally charged words in each industry, but separately for each gender  We find that the industry rankings of the relative frequencies INLINEFORM0 of emotionally charged words for the two genders are statistically significantly correlated, which suggests that regardless of their gender, users use positive (or negative) words with a relative frequency that correlates with their industry  (In other words, even if e g , Fashion has a larger number of women users, both men and women working in Fashion will tend to use more positive words than the corresponding gender in another industry with a larger number of men users such as Automotive )
Finally, motivated by previous findings of correlations between job satisfaction and gender dominance in the workplace BIBREF30 , we explore the relationship between the usage of Positive Emotion and Negative Emotion words and the gender dominance in an industry  Although we find that there are substantial gender imbalances in each industry (Appendix SECREF9 ), we did not find any statistically significant correlation between the gender dominance ratio in the different industries and the usage of positive (or negative) emotional words in either gender in our dataset 
Conclusion
In this paper, we examined the task of predicting a social media user's industry  We introduced an annotated dataset of over 20,000 blog users and applied a content-based classifier in conjunction with two feature selection methods for an overall accuracy of up to 0 534, which represents a large improvement over the majority class baseline of 0 188 
We also demonstrated how the user metadata can be incorporated in our classifiers  Although concatenation of features drawn both from blog content and profile elements did not yield any clear improvements over the best individual classifiers, we found that stacking improves the prediction accuracy to an overall accuracy of 0 643, as measured on our test dataset  A more in-depth analysis showed that not all industries are equally easy to predict: while industries such as Real Estate and Religion are clearly distinguishable with accuracy figures over 0 80, others such as Banking are much harder to predict 
Finally, we presented a qualitative analysis to provide some insights into the language of different industries, which highlighted differences in the top-ranked words in each industry, word semantic similarities, and the relative frequency of emotionally charged words 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation 
Additional Examples of Word Similarities
Context 3: t to mainstream news  In both cases we filtered out articles with less than 50 tweets; overall this dataset contains $\sim $160k mainstream tweets, corresponding to 227 news articles, and $\sim $100k disinformation tweets, corresponding to 237 news articles  We provide in Figure FIGREF5 the distribution of articles according to distinct sources for both news domains  As in the US dataset, we took into account censoring effects BIBREF14 by excluding tweets published before (left-censoring) or after two weeks (right-censoring) from the beginning of the collection process 
The different volumes of news shared on Twitter in the two countries are due both to the different population size of US and Italy (320 vs 60 millions) but also to the different usage of Twitter platform (and social media in general) for news consumption BIBREF24  Both datasets analyzed in this work are available from the authors on request 
A crucial aspect in our approach is the capability to fully capturing sharing cascades on Twitter associated to news articles  It has been reported BIBREF25 that the Twitter streaming endpoint filters out tweets matching a given query if they exceed 1% of the global daily volume of shared tweets, which nowadays is approximately $5\cdot 10^8$; however, as we always collected less than $10^6$ tweets per day, we did not incur in this issue and we thus gathered 100% of tweets matching our query 
Methodology ::: Building diffusion networks
We built Twitter diffusion networks following an approach widely adopted in the literature BIBREF6BIBREF17BIBREF2  We remark that there is an unavoidable limitation in Twitter Streaming API, which does not allow to retrieve true re-tweeting cascades because re-tweets always point to the original source and not to intermediate re-tweeting users BIBREF8BIBREF14; thus we adopt the only viable approach based on Twitter's public availability of data  Besides, by disentangling different interactions with multiple layers we potentially reduce the impact of this limitation on the global network properties compared to the single-layer approach used in our baseline 
Using the notation described in BIBREF26  we employ a multi-layer representation for Twitter diffusion networks  Sociologists have indeed recognized decades ago that it is crucial to study social systems by constructing multiple social networks where different types of ties among same individuals are used BIBREF27  Therefore, for each news article we built a multi-layer diffusion network composed of four different layers, one for each type of social interaction on Twitter platform, namely retweet (RT), reply (R), quote (Q) and mention (M), as shown in Figure FIGREF11  These networks are not necessarily node-aligned, i e  users might be missing in some layers  We do not insert "dummy" nodes to represent all users as it would have severe impact on the global network properties (e g  number of weakly connected components)  Alternatively one may look at each multi-layer diffusion network as an ensemble of individual graphs BIBREF26; since global network properties are computed separately for each layer, they are not affected by the presence of any inter-layer edges 
In our multi-layer representation, each layer is a directed graph where we add edges and nodes for each tweet of the layer type, e g  for the RT layer: whenever user $a$ retweets account $b$ we first add nodes $a$ and $b$ if not already present in the RT layer, then we build an edge that goes from $b$ to $a$ if it does not exists or we increment the weight by 1  Similarly for the other layers: for the R layer edges go from user $a$ (who replies) to user $b$, for the Q layer edges go from user $b$ (who is quoted by) to user $a$ and for the M layer edges go from user $a$ (who mentions) to user $b$ 
Note that, by construction, our layers do not include isolated nodes; they correspond to "pure tweets", i e  tweets which have not originated any interactions with other users  However, they are present in our dataset, and their number is exploited for classification, as described below 
Methodology ::: Global network properties
We used a set of global network indicators which allow us to encode each network layer by a tuple of features  Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector  We used the following global network properties:
Number of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\rightarrow v$, $v\rightarrow u$) 
Size of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph 
Number of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \leftrightarrow v$ ignoring edge directions 
Size of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph 
Diameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph 
Average Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique)  It is computed according to BIBREF28 
Main K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered) 
Density (d): the density for directed graphs is $d=\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph 
Structural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\frac{1}{|V||V-1|}\sum _i\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$  This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\frac{1}{|V||V-1|}$  In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$ 
We used networkx Python package BIBREF30 to compute all features  Whenever a layer is empty  we simply set to 0 all its features  In addition to computing the above nine features for each layer, we added two indicators for encoding information about pure tweets, namely the number T of pure tweets (containing URLs to a given news article) and the number U of unique users authoring those tweets  Therefore, a single diffusion network is represented by a vector with $9\cdot 4+2=38$ entries 
Methodology ::: Interpretation of network features and layers
Aforementioned network properties can be qualitatively explained in terms of social footprints as follows: SCC correlates with the size of the diffusion network, as the propagation of news occurs in a broadcast manner most of the time, i e  re-tweets dominate on other interactions, while LSCC allows to distinguish cases where such mono-directionality is somehow broken  WCC equals (approximately) the number of distinct diffusion cascades pertaining to each news article, with exceptions corresponding to those cases where some cascades merge together via Twitter interactions such as mentions, quotes and replies, and accordingly LWCC and DWCC equals the size and the depth of the largest cascade  CC corresponds to the level of connectedness of neighboring users in a given diffusion network whereas KC identifies the set of most influential users in a network and describes the efficiency of information spreading BIBREF17  Finally, d describes the proportions of potential connections between users which are actually activated and SV indicates whether a news item has gained popularity with a single and large broadcast or in a more viral fashion through multiple generations 
For what concerns different Twitter actions, users primarily interact with each other using retweets and mentions BIBREF20 
The former are the main engagement activity and act as a form of endorsement, allowing users to rebroadcast content generated by other users BIBREF31  Besides, when node B retweets node A we have an implicit confirmation that information from A appeared in B's Twitter feed BIBREF12  Quotes are simply a special case of retweets with comments 
Mentions usually include personal conversations as they allow someone to address a specific user or to refer to an individual in the third person; in the first case they are located at the beginning of a tweet and they are known as replies, otherwise they are put in the body of a tweet BIBREF20  The network of mentions is usually seen as a stronger version of interactions between Twitter users, compared to the traditional graph of follower/following relationships BIBREF32 
Experiments ::: Setup
We performed classification experiments using a basic off-the-shelf classifier, namely Logistic Regression (LR) with L2 penalty; this also allows us to compare results with our baseline

ground_truth: AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the political bias of different sources included in the model?
Context 3:  We applied a standardization of the features and we used the default configuration for parameters as described in scikit-learn package BIBREF33  We also tested other classifiers (such as K-Nearest Neighbors, Support Vector Machines and Random Forest) but we omit results as they give comparable performances  We remark that our goal is to show that a very simple machine learning framework, with no parameter tuning and optimization, allows for accurate results with our network-based approach 
We used the following evaluation metrics to assess the performances of different classifiers (TP=true positives, FP=false positives, FN=false negatives):
Precision = $\frac{TP}{TP+FP}$, the ability of a classifier not to label as positive a negative sample 
Recall = $\frac{TP}{TP+FN}$, the ability of a classifier to retrieve all positive samples 
F1-score = $2 \frac{\mbox{Precision} \cdot \mbox{Recall}}{\mbox{Precision} + \mbox{Recall}}$, the harmonic average of Precision and Recall 
Area Under the Receiver Operating Characteristic curve (AUROC); the Receiver Operating Characteristic (ROC) curve BIBREF34, which plots the TP rate versus the FP rate, shows the ability of a classifier to discriminate positive samples from negative ones as its threshold is varied; the AUROC value is in the range $[0, 1]$, with the random baseline classifier holding AUROC$=0 5$ and the ideal perfect classifier AUROC$=1$; thus larger AUROC values (and steeper ROCs) correspond to better classifiers 
In particular we computed so-called macro average–simple unweighted mean–of these metrics evaluated considering both labels (disinformation and mainstream)  We employed stratified shuffle split cross validation (with 10 folds) to evaluate performances 
Finally, we partitioned networks according to the total number of unique users involved in the sharing, i e  the number of nodes in the aggregated network represented with a single-layer representation considering together all layers and also pure tweets  A breakdown of both datasets according to size class (and political biases for the US scenario) is provided in Table 1 and Table 2 
Experiments ::: Classification performances
In Table 3 we first provide classification performances on the US dataset for the LR classifier evaluated on the size class described in Table 1  We can observe that in all instances our methodology performs better than a random classifier (50% AUROC), with AUROC values above 85% in all cases 
For what concerns political biases, as the classes of mainstream and disinformation networks are not balanced (e g , 1,292 mainstream and 4,149 disinformation networks with right bias) we employ a Balanced Random Forest with default parameters (as provided in imblearn Python package BIBREF35)  In order to test the robustness of our methodology, we trained only on left-biased networks or right-biased networks and tested on the entire set of sources (relative to the US dataset); we provide a comparison of AUROC values for both biases in Figure 4  We can notice that our multi-layer approach still entails significant results, thus showing that it can accurately distinguish mainstream news from disinformation regardless of the political bias  We further corroborated this result with additional classification experiments, that show similar performances, in which we excluded from the training/test set two specific sources (one at a time and both at the same time) that outweigh the others in terms of data samples–respectively "breitbart com" for right-biased sources and "politicususa com" for left-biased ones 
We performed classification experiments on the Italian dataset using the LR classifier and different size classes (we excluded $[1000, +\infty )$ which is empty); we show results for different evaluation metrics in Table 3  We can see that despite the limited number of samples (one order of magnitude smaller than the US dataset) the performances are overall in accordance with the US scenario  As shown in Table 4, we obtain results which are much better than our baseline in all size classes (see Table 4):
In the US dataset our multi-layer methodology performs much better in all size classes except for large networks ($[1000, +\infty )$ size class), reaching up to 13% improvement on smaller networks ($[0, 100)$ size class);
In the IT dataset our multi-layer methodology outperforms the baseline in all size classes, with the maximum performance gain (20%) on medium networks ($[100, 1000)$ size class); the baseline generally reaches bad performances compared to the US scenario 
Overall, our performances are comparable with those achieved by two state-of-the-art deep learning models for "fake news" detection BIBREF9BIBREF36 
Experiments ::: Layer importance analysis
In order to understand the impact of each layer on the performances of classifiers, we performed additional experiments considering separately each layer (we ignored T and U features relative to pure tweets)  In Table 5 we show metrics for each layer and all size classes, computed with a 10-fold stratified shuffle split cross validation, evaluated on the US dataset; in Figure 5 we show AUROC values for each layer compared with the general multi-layer approach  We can notice that both Q and M layers alone capture adequately the discrepancies of the two distinct news domains in the United States as they obtain good results with AUROC values in the range 75%-86%; these are comparable with those of the multi-layer approach which, nevertheless, outperforms them across all size classes 
We obtained similar performances for the Italian dataset, as the M layer obtains comparable performances w r t multi-layer approach with AUROC values in the range 72%-82%  We do not show these results for sake of conciseness 
Experiments ::: Feature importance analysis
We further investigated the importance of each feature by performing a $\chi ^2$ test, with 10-fold stratified shuffle split cross validation, considering the entire range of network sizes $[0, +\infty )$  We show the Top-5 most discriminative features for each country in Table 6 
We can notice the exact same set of features (with different relative orderings in the Top-3) in both countries; these correspond to two global network propertie–LWCC, which indicates the size of the largest cascade in the layer, and SCC, which correlates with the size of the network–associated to the same set of layers (Quotes, Retweets and Mentions) 
We further performed a $\chi ^2$ test to highlight the most discriminative features in the M layer of both countries, which performed equally well in the classification task as previously highlighted; also in this case we focused on the entire range of network sizes $[0, +\infty )$  Interestingly, we discovered exactly the same set of Top-3 features in both countries, namely LWCC, SCC and DWCC (which indicates the depth of the largest cascade in the layer) 
An inspection of the distributions of all aforementioned features revealed that disinformation news exhibit on average larger values than mainstream news 
We can qualitatively sum up these results as follows:
Sharing patterns in the two news domains exhibit discrepancies which might be country-independent and due to the content that is being shared 
Interactions in disinformation sharing cascades tends to be broader and deeper than in mainstream news, as widely reported in the literature BIBREF8BIBREF2BIBREF7 
Users likely make a different usage of mentions when sharing news belonging to the two domains, consequently shaping different sharing patterns 
Experiments ::: Temporal analysis
Similar to BIBREF9, we carried out additional experiments to answer the following question: how long do we need to observe a news spreading on Twitter in order to accurately classify it as disinformation or mainstream?
With this goal, we built several versions of our original dataset of multi-layer networks by considering in turn the following lifetimes: 1 hour, 6 hours, 12 hours, 1 day, 2 days, 3 days and 7 days; for each case, we computed the global network properties of the corresponding network and evaluated the LR classifier with 10-fold cross validation, separately for each lifetime (and considering always the entire set of networks)  We show corresponding AUROC values for both US and IT datasets in Figure 6 
We can see that in both countries news diffusion networks can be accurately classified after just a few hours of spreading, with AUROC values which are larger than 80% after only 6 hours of diffusion  These results are very promising and suggest that articles pertaining to the two news domains exhibit discrepancies in their sharing patterns that can be timely exploited in order to rapidly detect misleading items from factual information 
Conclusions
In this work we tackled the problem of the automatic classification of news articles in two domains, namely mainstream and disinformation news, with a language-independent approach which is based solely on the diffusion of news items on Twitter social platform  We disentangled different types of interactions on Twitter to accordingly build a multi-layer representation of news diffusion networks, and we computed a set of global network properties–separately for each layer–in order to encode each network with a tuple of features  Our goal was to investigate whether a multi-layer representation performs better than one layer BIBREF11, and to understand which of the features, observed at given layers, are most effective in the classification task 
Experiments with an off-the-shelf classifier such as Logistic Regression on datasets pertaining to two different media landscapes (US and Italy) yield very accurate classification results (AUROC up to 94%), even when accounting for the different political bias of news sources, which are far better than our baseline BIBREF11 with improvements up to 20%  Classification performances using single layers show that the layer of mentions alone entails better performance w r t other layers in both countries 
We also highlighted the most discriminative features across different layers in both countries; the results suggest that differences between the two news domains might be country-independent but rather due only to the typology of content shared, and that disinformation news shape broader and deeper cascades 
Additional experiments involving the temporal evolution of Twitter diffusion networks show that our methodology can accurate classify mainstream and disinformation news after a few hours of propagation on the platform 
Overall, our results prove that the topological features of multi-layer diffusion networks might be effectively exploited to detect online disinformation  We do not deny the presence of deceptive efforts to orchestrate the regular spread of information on social media via content amplification and manipulation BIBREF37BIBREF38  On the contrary, we postulate that such hidden forces might play to accentuate the discrepancies between the diffusion patterns of disinformation and mainstream news (and thus to make our methodology effective) 
In the future we aim to further investigate three directions: (1) employ temporal networks to represent news diffusion and apply classification techniques that take into account the sequential aspect of data (e g  recurrent neural networks); (2) carry out an extensive comparison of the diffusion of disinformation and mainstream news across countries to investigate deeper the presence of differences and similarities in sharing patterns; (3) leverage our network-based features in addition to state-of-the-art text-based approaches for "fake news" dete ction in order to deliver a real-world system to detect misleading and harmful information spreading on social media 

ground_truth: By assigning a political bias label to each news article and training only on left-biased or right-biased outlets of both disinformation and mainstream domains

we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets. Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles). We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains. We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries.