prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What neural network modules are included in NeuronBlocks?
Context 1: Introduction
Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc  However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges 
The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks  This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches  Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas 
To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible  At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules  Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster  Table TABREF11 showed that about 87 5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc  It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on 
Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks  The basic idea is to provide two layers of support to the engineers  The upper layer targets common NLP tasks  For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration  The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture  By following the interface guidelines, users can also contribute to this gallery of components with their own modules 
The technical contributions of NeuronBlocks are summarized into the following three aspects 
Related Work
There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community  These frameworks offer huge flexibility in DNN model design and support various NLP tasks  However, building models under these frameworks requires a large overhead of mastering these framework details  Therefore, higher level abstraction to hide the framework details is favored by many engineers 
There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc  OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks  AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc  Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components 
Design
The Neuronblocks is built on PyTorch  The overall framework is illustrated in Figure FIGREF16   It consists of two layers: the Block Zoo and the Model Zoo  In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions  Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface  These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks  In Model Zoo, the most popular NLP tasks are identified  For each task, several end-to-end network templates are provided in the form of JSON configuration files  Users can simply browse these configurations and choose one or more to instantiate  The whole task can be completed without any coding efforts 
Block Zoo
We recognize the following major functional categories of neural network components  Each category covers as many commonly used modules as possible  The Block Zoo is an open framework, and more modules can be added in the future 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported 
Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc  Furthermore, attention mechanisms are widely used in neural networks  Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc  Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability 
Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6  
Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported  For sequence labeling task, F1/Accuracy are supported  For knowledge distillation task, MSE/RMSE are supported  For MRC task, ExactMatch/F1 are supported 
Model Zoo
In NeuronBlocks, we identify four types of most popular NLP tasks  For each task, we provide various end-to-end network templates 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Text Classification and Matching  Tasks such as domain/intent classification, question answer matching are supported 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 
Context 3: Introduction and related work
In recent years there has been increasing interest on the issue of disinformation spreading on online social media  Global concern over false (or "fake") news as a threat to modern democracies has been frequently raised–ever since 2016 US Presidential elections–in correspondence of events of political relevance, where the proliferation of manipulated and low-credibility content attempts to drive and influence people opinions BIBREF0BIBREF1BIBREF2BIBREF3 
Researchers have highlighted several drivers for the diffusion of such malicious phenomenon, which include human factors (confirmation bias BIBREF4, naive realism BIBREF5), algorithmic biases (filter bubble effect BIBREF0), the presence of deceptive agents on social platforms (bots and trolls BIBREF6) and, lastly, the formation of echo chambers BIBREF7 where people polarize their opinions as they are insulated from contrary perspectives 
The problem of automatically detecting online disinformation news has been typically formulated as a binary classification task (i e  credible vs non-credible articles), and tackled with a variety of different techniques, based on traditional machine learning and/or deep learning, which mainly differ in the dataset and the features they employ to perform the classification  We may distinguish three approaches: those built on content-based features, those based on features extracted from the social context, and those which combine both aspects  A few main challenges hinder the task, namely the impossibility to manually verify all news items, the lack of gold-standard datasets and the adversarial setting in which malicious content is created BIBREF3BIBREF6 
In this work we follow the direction pointed out in a few recent contributions on the diffusion of disinformation compared to traditional and objective information  These have shown that false news spread faster and deeper than true news BIBREF8, and that social bots and echo chambers play an important role in the diffusion of malicious content BIBREF6, BIBREF7  Therefore we focus on the analysis of spreading patterns which naturally arise on social platforms as a consequence of multiple interactions between users, due to the increasing trend in online sharing of news BIBREF0 
A deep learning framework for detection of fake news cascades is provided in BIBREF9, where the authors refer to BIBREF8 in order to collect Twitter cascades pertaining to verified false and true rumors  They employ geometric deep learning, a novel paradigm for graph-based structures, to classify cascades based on four categories of features, such as user profile, user activity, network and spreading, and content  They also observe that a few hours of propagation are sufficient to distinguish false news from true news with high accuracy  Diffusion cascades on Weibo and Twitter are analyzed in BIBREF10, where authors focus on highlighting different topological properties, such as the number of hops from the source or the heterogeneity of the network, to show that fake news shape diffusion networks which are highly different from credible news, even at early stages of propagation 
In this work, we consider the results of BIBREF11 as our baseline  The authors use off-the-shelf machine learning classifiers to accurately classify news articles leveraging Twitter diffusion networks  To this aim, they consider a set of basic features which can be qualitatively interpreted w r t to the social behavior of users sharing credible vs non-credible information  Their methodology is overall in accordance with BIBREF12, where authors successfully detect Twitter astroturfing content, i e  political campaigns disguised as spontaneous grassroots, with a machine learning framework based on network features 
In this paper, we propose a classification framework based on a multi-layer formulation of Twitter diffusion networks  For each article we disentangle different social interactions on Twitter, namely tweets, retweets, mentions, replies and quotes, to accordingly build a diffusion network composed of multiple layers (on for each type of interaction), and we compute structural features separately for each layer  We pick a set of global network properties from the network science toolbox which can be qualitatively explained in terms of social dimensions and allow us to encode different networks with a tuple of features  These include traditional indicators, e g  network density, number of strong/weak connected components and diameter, and more elaborated ones such as main K-core number BIBREF13 and structural virality BIBREF14  Our main research question is whether the use of a multi-layer, disentangled network yields a significant advance in terms of classification accuracy over a conventional single-layer diffusion network  Additionally, we are interested in understanding which of the above features, and in which layer, are most effective in the classification task 
We perform classification experiments with an off-the-shelf Logistic Regression model on two different datasets of mainstream and disinformation news shared on Twitter respectively in the United States and in Italy during 2019  In the former case we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets  Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles)  We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains  We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries 
The outline of this paper is the following: we first formulate the problem and describe data collection, network representation and structural properties employed for the classification; then we provide experimental results–classification performances, layer and feature importance analyses and a temporal classification evaluation–and finally we draw conclusions and future directions 
Methodology ::: Disinformation and mainstream news
In this work we formulate our classification problem as follows: given two classes of news articles, respectively $D$ (disinformation) and $M$ (mainstream), a set of news articles $A_i$ and associated class labels $C_i \in \lbrace D,M\rbrace $, and a set of tweets $\Pi _i=\lbrace T_i^1, T_i^2,    \rbrace $ each of which contains an Uniform Resource Locator (URL) pointing explicitly to article $A_i$, predict the class $C_i$ of each article $A_i$  There is huge debate and controversy on a proper taxonomy of malicious and deceptive information BIBREF1BIBREF2BIBREF15BIBREF16BIBREF17BIBREF3BIBREF11  In this work we prefer the term disinformation to the more specific fake news to refer to a variety of misleading and harmful information  Therefore, we follow a source-based approach, a consolidated strategy also adopted by BIBREF6BIBREF16BIBREF2BIBREF1, in order to obtain relevant data for our analysis  We collected:
Disinformation articles, published by websites which are well-known for producing low-credibility content, false and misleading news reports as well as extreme propaganda and hoaxes and flagged as such by reputable journalists and fact-checkers;
Mainstream news, referring to traditional news outlets which deliver factual and credible information 
We believe that this is currently the most reliable classification approach, but it entails obvious limitations, as disinformation outlets may also publish true stories and likewise misinformation is sometimes reported on mainstream media  Also, given the choice of news sources, we cannot test whether our methodology is able to classify disinformation vs factual but not mainstream news which are published on niche, non-disinformation outlets 
Methodology ::: US dataset
We collected tweets associated to a dozen US mainstream news websites, i e  most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets  We filtered out articles associated to less than 50 tweets  The resulting dataset contains overall $\sim $1 7 million tweets for mainstream news, collected in a period of three weeks (February 25th, 2019-March 18th, 2019), which are associated to 6,978 news articles, and $\sim $1 6 million tweets for disinformation, collected in a period of three months (January 1st, 2019-March 18th, 2019) for sake of balance of the two classes, which hold 5,775 distinct articles  Diffusion censoring effects BIBREF14 were correctly taken into account in both collection procedures  We provide in Figure FIGREF4 the distribution of articles by source and political bias for both news domains 
As it is reported that conservatives and liberals exhibit different behaviors on online social platforms BIBREF19BIBREF20BIBREF21, we further assigned a political bias label to different US outlets (and therefore news articles) following the procedure described in BIBREF2  In order to assess the robustness of our method, we performed classification experiments by training only on left-biased (or right-biased) outlets of both disinformation and mainstream domains and testing on the entire set of sources, as well as excluding particular sources that outweigh the others in terms of samples to avoid over-fitting 
Methodology ::: Italian dataset
For what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa)  We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites  In order to get balanced classes (April 5th, 2019-May 5th, 2019), we retained data collected in a longer period w r

ground_truth: Embedding Layer, Neural Network Layers, Loss Function, Metrics

Embedding Layer, Neural Network Layers, Loss Function, Metrics

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they employ their indexing-based method to create a sample of a QA Wikipedia dataset?
Context 2: Introduction
Question answering (QA) has been a blooming research field for the last decade  Selection-based QA implies a family of tasks that find answer contexts from large data given questions in natural language  Three tasks have been proposed for selection-based QA  Given a document, answer extraction BIBREF0 , BIBREF1 finds answer phrases whereas answer selection BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and answer triggering BIBREF6 , BIBREF7 find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks  Recently, various QA tasks that are not selection-based have been proposed BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ; however, selection-based QA remains still important because of its practical value to real applications (e g , IBM Watson, MIT Start) 
Several datasets have been released for selection-based QA  wang:07a created the QASent dataset consisting of 277 questions, which has been widely used for benchmarking the answer selection task  feng:15a presented InsuranceQA comprising 16K+ questions on insurance contexts  yang:15a introduced WikiQA for answer selection and triggering  jurczyk:16 created SelQA for large real-scale answer triggering  rajpurkar2016squad presented SQuAD for answer extraction and selection as well as for reading comprehension  Finally, morales-EtAl:2016:EMNLP2016 provided InfoboxQA for answer selection 
These corpora make it possible to evaluate the robustness of statistical question answering learning  Although all of these corpora target on selection-based QA, they are designed for different purposes such that it is important to understand the nature of these corpora so a better use of them can be made  In this paper, we make both intrinsic and extrinsic analyses of four latest corpora based on Wikipedia, WikiQA, SelQA, SQuAD, and InfoboxQA  We first give a thorough intrinsic analysis regarding contextual similarities, question types, and answer categories (Section SECREF2 )  We then map questions in all corpora to the current version of English Wikipedia and benchmark another selection-based QA task, answer retrieval (Section SECREF3 )  Finally, we present an extrinsic analysis through a set of experiments cross-testing these corpora using a convolutional neural network architecture (Section SECREF4 ) 
Intrinsic Analysis
Four publicly available corpora are selected for our analysis  These corpora are based on Wikipedia, so more comparable than the others, and have already been used for the evaluation of several QA systems 
WikiQA BIBREF6 comprises questions selected from the Bing search queries, where user click data give the questions and their corresponding Wikipedia articles  The abstracts of these articles are then extracted to create answer candidates  The assumption is made that if many queries lead to the same article, it must contain the answer context; however, this assumption fails for some occasions, which makes this dataset more challenging  Since the existence of answer contexts is not guaranteed in this task, it is called answer triggering instead of answer selection 
SelQA BIBREF7 is a product of five annotation tasks through crowdsourcing  It consists of about 8K questions where a half of the questions are paraphrased from the other half, aiming to reduce contextual similarities between questions and answers  Each question is associated with a section in Wikipedia where the answer context is guaranteed, and also with five sections selected from the entire Wikipedia where the selection is made by the Lucene search engine  This second dataset does not assume the existence of the answer context, so can be used for the evaluation of answer triggering 
SQuAD BIBREF12 presents 107K+ crowdsourced questions on 536 Wikipedia articles, where the answer contexts are guaranteed to exist within the provided paragraph  It contains annotation of answer phrases as well as the pointers to the sentences including the answer phrases; thus, it can be used for both answer extraction and selection  This corpus also provides human accuracy on those questions, setting up a reasonable upper bound for machines  To avoid overfitting, the evaluation set is not publicly available although system outputs can be evaluated by their provided script 
InfoboxQA BIBREF13 gives 15K+ questions based on the infoboxes from 150 articles in Wikipedia  Each question is crowdsourced and associated with an infobox, where each line of the infobox is considered an answer candidate  This corpus emphasizes the gravity of infoboxes, which summary arguably the most commonly asked information about those articles  Although the nature of this corpus is different from the others, it can also be used to evaluate answer selection 
Analysis
All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively  SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times  On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts  Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes 
All corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate  SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes  It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries  InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes  InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively  In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close 
Fig  FIGREF4 shows the distributions of seven question types grouped deterministically from the lexicons  Although these corpora have been independently developed, a general trend is found, where the what question type dominates, followed by how and who, followed by when and where, and so on 
Fig  FIGREF6 shows the distributions of answer categories automatically classified by our Convolutional Neural Network model trained on the data distributed by li:02a  Interestingly, each corpus focuses on different categories, Numeric for WikiQA and SelQA, Entity for SQuAD, and Person for InfoboxQA, which gives enough diversities for statistical learning to build robust models 
Answer Retrieval
This section describes another selection-based QA task, called answer retrieval, that finds the answer context from a larger dataset, the entire Wikipedia  SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia  We propose an automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda so they can be coherently used for answer retrieval 
Each paragraph in Wikipedia is first indexed by Lucene using {1,2,3}-grams, where the paragraphs are separated by WikiExtractor and segmented by NLP4J (28 7M+ paragraphs are indexed)  Each answer sentence from the corpora in Table TABREF3 is then queried to Lucene, and the top-5 ranked paragraphs are retrieved  The cosine similarity between each sentence in these paragraphs and the answer sentence is measured for INLINEFORM0 -grams, say INLINEFORM1   A weight is assigned to each INLINEFORM2 -gram score, say INLINEFORM3 , and the weighted sum is measured: INLINEFORM4   The fixed weights of INLINEFORM5 are used for our experiments, which can be improved 
If there exists a sentence whose INLINEFORM0 , the paragraph consisting of that sentence is considered the silver-standard answer passage  Table TABREF3 shows how robust these silver-standard passages are based on human judgement ( INLINEFORM1 ) and how many passages are collected ( INLINEFORM2 ) for INLINEFORM3 , where the human judgement is performed on 50 random samples for each case  For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively  Finally, each question is queried to Lucene and the top- INLINEFORM7 paragraphs are retrieved from the entire Wikipedia  If the answer sentence exists within those retrieved paragraphs according to the silver-standard, it is considered correct 
Finding a paragraph that includes the answer context out of the entire Wikipedia is an extremely difficult task (128 7M)  The last row of Table TABREF3 shows results from answer retrieval  Given INLINEFORM0 , SelQA and SQuAD show about 34% and 35% accuracy, which are reasonable  However, WikiQA shows a significantly lower accuracy of 12 47%; this is because the questions in WikiQA is about twice shorter than the questions in the other corpora such that not enough lexicons can be extracted from these questions for the Lucene search 
Answer Selection
Answer selection is evaluated by two metrics, mean average precision (MAP) and mean reciprocal rank (MRR)  The bigram CNN introduced by yu:14a is used to generate all the results in Table TABREF11 , where models are trained on either single or combined datasets  Clearly, the questions in WikiQA are the most challenging, and adding more training data from the other corpora hurts accuracy due to the uniqueness of query-based questions in this corpus  The best model is achieved by training on W+S+Q for SelQA; adding InfoboxQA hurts accuracy for SelQA although it gives a marginal gain for SQuAD  Just like WikiQA, InfoboxQA performs the best when it is trained on only itself  From our analysis, we suggest that to use models trained on WikiQA and InfoboxQA for short query-like questions, whereas to use ones trained on SelQA and SQuAD for long natural questions 
Answer Triggering
The results of INLINEFORM0 from the answer retrieval task in Section SECREF13 are used to create the datasets for answer triggering, where about 65% of the questions are not expected to find their answer contexts from the provided paragraphs for SelQA and SQuAD and 87 5% are not expected for WikiQA  Answer triggering is evaluated by the F1 scores as presented in Table TABREF11 , where three corpora are cross validated  The results on WikiQA are pretty low as expected from the poor accuracy on the answer retrieval task  Training on SelQA gives the best models for both WikiQA and SelQA  Training on SQuAD gives the best model for SQuAD although the model trained on SelQA is comparable  Since the answer triggering datasets are about 5 times larger than the answer selection datasets, it is computationally too expensive to combine all data for training  We plan to find a strong machine to perform this experiment in near future 
Related work
Lately, several deep learning approaches have been proposed for question answering  yu:14a presented a CNN model that recognizes the semantic similarity between two sentences  wang-nyberg:2015:ACL-IJCNLP presented a stacked bidirectional LSTM approach to read words in sequence, then outputs their similarity scores  feng:15a applied a general deep learning framework to non-factoid question answering  santos:16a introduced an attentive pooling mechanism that led to further improvements in selection-based QA 
Conclusion
We present a comprehensive comparison study of the existing corpora for selection-based question answering  Our intrinsic analysis provides a better understanding of the uniqueness or similarity between these corpora  Our extrinsic analysis shows the strength or weakness of combining these corpora together for statistical learning  Additionally, we create a silver-standard dataset for answer retrieval and triggering, which will be publicly available  In the future, we will explore different ways of improving the quality of our silver-standard datasets by fine-tuning the hyper-parameters 
Context 3: t to mainstream news  In both cases we filtered out articles with less than 50 tweets; overall this dataset contains $\sim $160k mainstream tweets, corresponding to 227 news articles, and $\sim $100k disinformation tweets, corresponding to 237 news articles  We provide in Figure FIGREF5 the distribution of articles according to distinct sources for both news domains  As in the US dataset, we took into account censoring effects BIBREF14 by excluding tweets published before (left-censoring) or after two weeks (right-censoring) from the beginning of the collection process 
The different volumes of news shared on Twitter in the two countries are due both to the different population size of US and Italy (320 vs 60 millions) but also to the different usage of Twitter platform (and social media in general) for news consumption BIBREF24  Both datasets analyzed in this work are available from the authors on request 
A crucial aspect in our approach is the capability to fully capturing sharing cascades on Twitter associated to news articles  It has been reported BIBREF25 that the Twitter streaming endpoint filters out tweets matching a given query if they exceed 1% of the global daily volume of shared tweets, which nowadays is approximately $5\cdot 10^8$; however, as we always collected less than $10^6$ tweets per day, we did not incur in this issue and we thus gathered 100% of tweets matching our query 
Methodology ::: Building diffusion networks
We built Twitter diffusion networks following an approach widely adopted in the literature BIBREF6BIBREF17BIBREF2  We remark that there is an unavoidable limitation in Twitter Streaming API, which does not allow to retrieve true re-tweeting cascades because re-tweets always point to the original source and not to intermediate re-tweeting users BIBREF8BIBREF14; thus we adopt the only viable approach based on Twitter's public availability of data  Besides, by disentangling different interactions with multiple layers we potentially reduce the impact of this limitation on the global network properties compared to the single-layer approach used in our baseline 
Using the notation described in BIBREF26  we employ a multi-layer representation for Twitter diffusion networks  Sociologists have indeed recognized decades ago that it is crucial to study social systems by constructing multiple social networks where different types of ties among same individuals are used BIBREF27  Therefore, for each news article we built a multi-layer diffusion network composed of four different layers, one for each type of social interaction on Twitter platform, namely retweet (RT), reply (R), quote (Q) and mention (M), as shown in Figure FIGREF11  These networks are not necessarily node-aligned, i e  users might be missing in some layers  We do not insert "dummy" nodes to represent all users as it would have severe impact on the global network properties (e g  number of weakly connected components)  Alternatively one may look at each multi-layer diffusion network as an ensemble of individual graphs BIBREF26; since global network properties are computed separately for each layer, they are not affected by the presence of any inter-layer edges 
In our multi-layer representation, each layer is a directed graph where we add edges and nodes for each tweet of the layer type, e g  for the RT layer: whenever user $a$ retweets account $b$ we first add nodes $a$ and $b$ if not already present in the RT layer, then we build an edge that goes from $b$ to $a$ if it does not exists or we increment the weight by 1  Similarly for the other layers: for the R layer edges go from user $a$ (who replies) to user $b$, for the Q layer edges go from user $b$ (who is quoted by) to user $a$ and for the M layer edges go from user $a$ (who mentions) to user $b$ 
Note that, by construction, our layers do not include isolated nodes; they correspond to "pure tweets", i e  tweets which have not originated any interactions with other users  However, they are present in our dataset, and their number is exploited for classification, as described below 
Methodology ::: Global network properties
We used a set of global network indicators which allow us to encode each network layer by a tuple of features  Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector  We used the following global network properties:
Number of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\rightarrow v$, $v\rightarrow u$) 
Size of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph 
Number of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \leftrightarrow v$ ignoring edge directions 
Size of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph 
Diameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph 
Average Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique)  It is computed according to BIBREF28 
Main K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered) 
Density (d): the density for directed graphs is $d=\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph 
Structural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\frac{1}{|V||V-1|}\sum _i\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$  This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\frac{1}{|V||V-1|}$  In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$ 
We used networkx Python package BIBREF30 to compute all features  Whenever a layer is empty  we simply set to 0 all its features  In addition to computing the above nine features for each layer, we added two indicators for encoding information about pure tweets, namely the number T of pure tweets (containing URLs to a given news article) and the number U of unique users authoring those tweets  Therefore, a single diffusion network is represented by a vector with $9\cdot 4+2=38$ entries 
Methodology ::: Interpretation of network features and layers
Aforementioned network properties can be qualitatively explained in terms of social footprints as follows: SCC correlates with the size of the diffusion network, as the propagation of news occurs in a broadcast manner most of the time, i e  re-tweets dominate on other interactions, while LSCC allows to distinguish cases where such mono-directionality is somehow broken  WCC equals (approximately) the number of distinct diffusion cascades pertaining to each news article, with exceptions corresponding to those cases where some cascades merge together via Twitter interactions such as mentions, quotes and replies, and accordingly LWCC and DWCC equals the size and the depth of the largest cascade  CC corresponds to the level of connectedness of neighboring users in a given diffusion network whereas KC identifies the set of most influential users in a network and describes the efficiency of information spreading BIBREF17  Finally, d describes the proportions of potential connections between users which are actually activated and SV indicates whether a news item has gained popularity with a single and large broadcast or in a more viral fashion through multiple generations 
For what concerns different Twitter actions, users primarily interact with each other using retweets and mentions BIBREF20 
The former are the main engagement activity and act as a form of endorsement, allowing users to rebroadcast content generated by other users BIBREF31  Besides, when node B retweets node A we have an implicit confirmation that information from A appeared in B's Twitter feed BIBREF12  Quotes are simply a special case of retweets with comments 
Mentions usually include personal conversations as they allow someone to address a specific user or to refer to an individual in the third person; in the first case they are located at the beginning of a tweet and they are known as replies, otherwise they are put in the body of a tweet BIBREF20  The network of mentions is usually seen as a stronger version of interactions between Twitter users, compared to the traditional graph of follower/following relationships BIBREF32 
Experiments ::: Setup
We performed classification experiments using a basic off-the-shelf classifier, namely Logistic Regression (LR) with L2 penalty; this also allows us to compare results with our baseline

ground_truth: Yes

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the political bias of different sources included in the model?
Context 3:  We applied a standardization of the features and we used the default configuration for parameters as described in scikit-learn package BIBREF33  We also tested other classifiers (such as K-Nearest Neighbors, Support Vector Machines and Random Forest) but we omit results as they give comparable performances  We remark that our goal is to show that a very simple machine learning framework, with no parameter tuning and optimization, allows for accurate results with our network-based approach 
We used the following evaluation metrics to assess the performances of different classifiers (TP=true positives, FP=false positives, FN=false negatives):
Precision = $\frac{TP}{TP+FP}$, the ability of a classifier not to label as positive a negative sample 
Recall = $\frac{TP}{TP+FN}$, the ability of a classifier to retrieve all positive samples 
F1-score = $2 \frac{\mbox{Precision} \cdot \mbox{Recall}}{\mbox{Precision} + \mbox{Recall}}$, the harmonic average of Precision and Recall 
Area Under the Receiver Operating Characteristic curve (AUROC); the Receiver Operating Characteristic (ROC) curve BIBREF34, which plots the TP rate versus the FP rate, shows the ability of a classifier to discriminate positive samples from negative ones as its threshold is varied; the AUROC value is in the range $[0, 1]$, with the random baseline classifier holding AUROC$=0 5$ and the ideal perfect classifier AUROC$=1$; thus larger AUROC values (and steeper ROCs) correspond to better classifiers 
In particular we computed so-called macro average–simple unweighted mean–of these metrics evaluated considering both labels (disinformation and mainstream)  We employed stratified shuffle split cross validation (with 10 folds) to evaluate performances 
Finally, we partitioned networks according to the total number of unique users involved in the sharing, i e  the number of nodes in the aggregated network represented with a single-layer representation considering together all layers and also pure tweets  A breakdown of both datasets according to size class (and political biases for the US scenario) is provided in Table 1 and Table 2 
Experiments ::: Classification performances
In Table 3 we first provide classification performances on the US dataset for the LR classifier evaluated on the size class described in Table 1  We can observe that in all instances our methodology performs better than a random classifier (50% AUROC), with AUROC values above 85% in all cases 
For what concerns political biases, as the classes of mainstream and disinformation networks are not balanced (e g , 1,292 mainstream and 4,149 disinformation networks with right bias) we employ a Balanced Random Forest with default parameters (as provided in imblearn Python package BIBREF35)  In order to test the robustness of our methodology, we trained only on left-biased networks or right-biased networks and tested on the entire set of sources (relative to the US dataset); we provide a comparison of AUROC values for both biases in Figure 4  We can notice that our multi-layer approach still entails significant results, thus showing that it can accurately distinguish mainstream news from disinformation regardless of the political bias  We further corroborated this result with additional classification experiments, that show similar performances, in which we excluded from the training/test set two specific sources (one at a time and both at the same time) that outweigh the others in terms of data samples–respectively "breitbart com" for right-biased sources and "politicususa com" for left-biased ones 
We performed classification experiments on the Italian dataset using the LR classifier and different size classes (we excluded $[1000, +\infty )$ which is empty); we show results for different evaluation metrics in Table 3  We can see that despite the limited number of samples (one order of magnitude smaller than the US dataset) the performances are overall in accordance with the US scenario  As shown in Table 4, we obtain results which are much better than our baseline in all size classes (see Table 4):
In the US dataset our multi-layer methodology performs much better in all size classes except for large networks ($[1000, +\infty )$ size class), reaching up to 13% improvement on smaller networks ($[0, 100)$ size class);
In the IT dataset our multi-layer methodology outperforms the baseline in all size classes, with the maximum performance gain (20%) on medium networks ($[100, 1000)$ size class); the baseline generally reaches bad performances compared to the US scenario 
Overall, our performances are comparable with those achieved by two state-of-the-art deep learning models for "fake news" detection BIBREF9BIBREF36 
Experiments ::: Layer importance analysis
In order to understand the impact of each layer on the performances of classifiers, we performed additional experiments considering separately each layer (we ignored T and U features relative to pure tweets)  In Table 5 we show metrics for each layer and all size classes, computed with a 10-fold stratified shuffle split cross validation, evaluated on the US dataset; in Figure 5 we show AUROC values for each layer compared with the general multi-layer approach  We can notice that both Q and M layers alone capture adequately the discrepancies of the two distinct news domains in the United States as they obtain good results with AUROC values in the range 75%-86%; these are comparable with those of the multi-layer approach which, nevertheless, outperforms them across all size classes 
We obtained similar performances for the Italian dataset, as the M layer obtains comparable performances w r t multi-layer approach with AUROC values in the range 72%-82%  We do not show these results for sake of conciseness 
Experiments ::: Feature importance analysis
We further investigated the importance of each feature by performing a $\chi ^2$ test, with 10-fold stratified shuffle split cross validation, considering the entire range of network sizes $[0, +\infty )$  We show the Top-5 most discriminative features for each country in Table 6 
We can notice the exact same set of features (with different relative orderings in the Top-3) in both countries; these correspond to two global network propertie–LWCC, which indicates the size of the largest cascade in the layer, and SCC, which correlates with the size of the network–associated to the same set of layers (Quotes, Retweets and Mentions) 
We further performed a $\chi ^2$ test to highlight the most discriminative features in the M layer of both countries, which performed equally well in the classification task as previously highlighted; also in this case we focused on the entire range of network sizes $[0, +\infty )$  Interestingly, we discovered exactly the same set of Top-3 features in both countries, namely LWCC, SCC and DWCC (which indicates the depth of the largest cascade in the layer) 
An inspection of the distributions of all aforementioned features revealed that disinformation news exhibit on average larger values than mainstream news 
We can qualitatively sum up these results as follows:
Sharing patterns in the two news domains exhibit discrepancies which might be country-independent and due to the content that is being shared 
Interactions in disinformation sharing cascades tends to be broader and deeper than in mainstream news, as widely reported in the literature BIBREF8BIBREF2BIBREF7 
Users likely make a different usage of mentions when sharing news belonging to the two domains, consequently shaping different sharing patterns 
Experiments ::: Temporal analysis
Similar to BIBREF9, we carried out additional experiments to answer the following question: how long do we need to observe a news spreading on Twitter in order to accurately classify it as disinformation or mainstream?
With this goal, we built several versions of our original dataset of multi-layer networks by considering in turn the following lifetimes: 1 hour, 6 hours, 12 hours, 1 day, 2 days, 3 days and 7 days; for each case, we computed the global network properties of the corresponding network and evaluated the LR classifier with 10-fold cross validation, separately for each lifetime (and considering always the entire set of networks)  We show corresponding AUROC values for both US and IT datasets in Figure 6 
We can see that in both countries news diffusion networks can be accurately classified after just a few hours of spreading, with AUROC values which are larger than 80% after only 6 hours of diffusion  These results are very promising and suggest that articles pertaining to the two news domains exhibit discrepancies in their sharing patterns that can be timely exploited in order to rapidly detect misleading items from factual information 
Conclusions
In this work we tackled the problem of the automatic classification of news articles in two domains, namely mainstream and disinformation news, with a language-independent approach which is based solely on the diffusion of news items on Twitter social platform  We disentangled different types of interactions on Twitter to accordingly build a multi-layer representation of news diffusion networks, and we computed a set of global network properties–separately for each layer–in order to encode each network with a tuple of features  Our goal was to investigate whether a multi-layer representation performs better than one layer BIBREF11, and to understand which of the features, observed at given layers, are most effective in the classification task 
Experiments with an off-the-shelf classifier such as Logistic Regression on datasets pertaining to two different media landscapes (US and Italy) yield very accurate classification results (AUROC up to 94%), even when accounting for the different political bias of news sources, which are far better than our baseline BIBREF11 with improvements up to 20%  Classification performances using single layers show that the layer of mentions alone entails better performance w r t other layers in both countries 
We also highlighted the most discriminative features across different layers in both countries; the results suggest that differences between the two news domains might be country-independent but rather due only to the typology of content shared, and that disinformation news shape broader and deeper cascades 
Additional experiments involving the temporal evolution of Twitter diffusion networks show that our methodology can accurate classify mainstream and disinformation news after a few hours of propagation on the platform 
Overall, our results prove that the topological features of multi-layer diffusion networks might be effectively exploited to detect online disinformation  We do not deny the presence of deceptive efforts to orchestrate the regular spread of information on social media via content amplification and manipulation BIBREF37BIBREF38  On the contrary, we postulate that such hidden forces might play to accentuate the discrepancies between the diffusion patterns of disinformation and mainstream news (and thus to make our methodology effective) 
In the future we aim to further investigate three directions: (1) employ temporal networks to represent news diffusion and apply classification techniques that take into account the sequential aspect of data (e g  recurrent neural networks); (2) carry out an extensive comparison of the diffusion of disinformation and mainstream news across countries to investigate deeper the presence of differences and similarities in sharing patterns; (3) leverage our network-based features in addition to state-of-the-art text-based approaches for "fake news" dete ction in order to deliver a real-world system to detect misleading and harmful information spreading on social media 

ground_truth: By assigning a political bias label to each news article and training only on left-biased or right-biased outlets of both disinformation and mainstream domains

we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets. Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles). We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains. We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries.