prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the political bias of different sources included in the model?
Context 1: Introduction and related work
In recent years there has been increasing interest on the issue of disinformation spreading on online social media  Global concern over false (or "fake") news as a threat to modern democracies has been frequently raised–ever since 2016 US Presidential elections–in correspondence of events of political relevance, where the proliferation of manipulated and low-credibility content attempts to drive and influence people opinions BIBREF0BIBREF1BIBREF2BIBREF3 
Researchers have highlighted several drivers for the diffusion of such malicious phenomenon, which include human factors (confirmation bias BIBREF4, naive realism BIBREF5), algorithmic biases (filter bubble effect BIBREF0), the presence of deceptive agents on social platforms (bots and trolls BIBREF6) and, lastly, the formation of echo chambers BIBREF7 where people polarize their opinions as they are insulated from contrary perspectives 
The problem of automatically detecting online disinformation news has been typically formulated as a binary classification task (i e  credible vs non-credible articles), and tackled with a variety of different techniques, based on traditional machine learning and/or deep learning, which mainly differ in the dataset and the features they employ to perform the classification  We may distinguish three approaches: those built on content-based features, those based on features extracted from the social context, and those which combine both aspects  A few main challenges hinder the task, namely the impossibility to manually verify all news items, the lack of gold-standard datasets and the adversarial setting in which malicious content is created BIBREF3BIBREF6 
In this work we follow the direction pointed out in a few recent contributions on the diffusion of disinformation compared to traditional and objective information  These have shown that false news spread faster and deeper than true news BIBREF8, and that social bots and echo chambers play an important role in the diffusion of malicious content BIBREF6, BIBREF7  Therefore we focus on the analysis of spreading patterns which naturally arise on social platforms as a consequence of multiple interactions between users, due to the increasing trend in online sharing of news BIBREF0 
A deep learning framework for detection of fake news cascades is provided in BIBREF9, where the authors refer to BIBREF8 in order to collect Twitter cascades pertaining to verified false and true rumors  They employ geometric deep learning, a novel paradigm for graph-based structures, to classify cascades based on four categories of features, such as user profile, user activity, network and spreading, and content  They also observe that a few hours of propagation are sufficient to distinguish false news from true news with high accuracy  Diffusion cascades on Weibo and Twitter are analyzed in BIBREF10, where authors focus on highlighting different topological properties, such as the number of hops from the source or the heterogeneity of the network, to show that fake news shape diffusion networks which are highly different from credible news, even at early stages of propagation 
In this work, we consider the results of BIBREF11 as our baseline  The authors use off-the-shelf machine learning classifiers to accurately classify news articles leveraging Twitter diffusion networks  To this aim, they consider a set of basic features which can be qualitatively interpreted w r t to the social behavior of users sharing credible vs non-credible information  Their methodology is overall in accordance with BIBREF12, where authors successfully detect Twitter astroturfing content, i e  political campaigns disguised as spontaneous grassroots, with a machine learning framework based on network features 
In this paper, we propose a classification framework based on a multi-layer formulation of Twitter diffusion networks  For each article we disentangle different social interactions on Twitter, namely tweets, retweets, mentions, replies and quotes, to accordingly build a diffusion network composed of multiple layers (on for each type of interaction), and we compute structural features separately for each layer  We pick a set of global network properties from the network science toolbox which can be qualitatively explained in terms of social dimensions and allow us to encode different networks with a tuple of features  These include traditional indicators, e g  network density, number of strong/weak connected components and diameter, and more elaborated ones such as main K-core number BIBREF13 and structural virality BIBREF14  Our main research question is whether the use of a multi-layer, disentangled network yields a significant advance in terms of classification accuracy over a conventional single-layer diffusion network  Additionally, we are interested in understanding which of the above features, and in which layer, are most effective in the classification task 
We perform classification experiments with an off-the-shelf Logistic Regression model on two different datasets of mainstream and disinformation news shared on Twitter respectively in the United States and in Italy during 2019  In the former case we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets  Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles)  We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains  We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries 
The outline of this paper is the following: we first formulate the problem and describe data collection, network representation and structural properties employed for the classification; then we provide experimental results–classification performances, layer and feature importance analyses and a temporal classification evaluation–and finally we draw conclusions and future directions 
Methodology ::: Disinformation and mainstream news
In this work we formulate our classification problem as follows: given two classes of news articles, respectively $D$ (disinformation) and $M$ (mainstream), a set of news articles $A_i$ and associated class labels $C_i \in \lbrace D,M\rbrace $, and a set of tweets $\Pi _i=\lbrace T_i^1, T_i^2,    \rbrace $ each of which contains an Uniform Resource Locator (URL) pointing explicitly to article $A_i$, predict the class $C_i$ of each article $A_i$  There is huge debate and controversy on a proper taxonomy of malicious and deceptive information BIBREF1BIBREF2BIBREF15BIBREF16BIBREF17BIBREF3BIBREF11  In this work we prefer the term disinformation to the more specific fake news to refer to a variety of misleading and harmful information  Therefore, we follow a source-based approach, a consolidated strategy also adopted by BIBREF6BIBREF16BIBREF2BIBREF1, in order to obtain relevant data for our analysis  We collected:
Disinformation articles, published by websites which are well-known for producing low-credibility content, false and misleading news reports as well as extreme propaganda and hoaxes and flagged as such by reputable journalists and fact-checkers;
Mainstream news, referring to traditional news outlets which deliver factual and credible information 
We believe that this is currently the most reliable classification approach, but it entails obvious limitations, as disinformation outlets may also publish true stories and likewise misinformation is sometimes reported on mainstream media  Also, given the choice of news sources, we cannot test whether our methodology is able to classify disinformation vs factual but not mainstream news which are published on niche, non-disinformation outlets 
Methodology ::: US dataset
We collected tweets associated to a dozen US mainstream news websites, i e  most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets  We filtered out articles associated to less than 50 tweets  The resulting dataset contains overall $\sim $1 7 million tweets for mainstream news, collected in a period of three weeks (February 25th, 2019-March 18th, 2019), which are associated to 6,978 news articles, and $\sim $1 6 million tweets for disinformation, collected in a period of three months (January 1st, 2019-March 18th, 2019) for sake of balance of the two classes, which hold 5,775 distinct articles  Diffusion censoring effects BIBREF14 were correctly taken into account in both collection procedures  We provide in Figure FIGREF4 the distribution of articles by source and political bias for both news domains 
As it is reported that conservatives and liberals exhibit different behaviors on online social platforms BIBREF19BIBREF20BIBREF21, we further assigned a political bias label to different US outlets (and therefore news articles) following the procedure described in BIBREF2  In order to assess the robustness of our method, we performed classification experiments by training only on left-biased (or right-biased) outlets of both disinformation and mainstream domains and testing on the entire set of sources, as well as excluding particular sources that outweigh the others in terms of samples to avoid over-fitting 
Methodology ::: Italian dataset
For what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa)  We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites  In order to get balanced classes (April 5th, 2019-May 5th, 2019), we retained data collected in a longer period w r t to mainstream news  In both cases we filtered out articles with less than 50 tweets; overall this dataset contains $\sim $160k mainstream tweets, corresponding to 227 news articles, and $\sim $100k disinformation tweets, corresponding to 237 news articles  We provide in Figure FIGREF5 the distribution of articles according to distinct sources for both news domains  As in the US dataset, we took into account censoring effects BIBREF14 by excluding tweets published before (left-censoring) or after two weeks (right-censoring) from the beginning of the collection process 
The different volumes of news shared on Twitter in the two countries are due both to the different population size of US and Italy (320 vs 60 millions) but also to the different usage of Twitter platform (and social media in general) for news consumption BIBREF24  Both datasets analyzed in this work are available from the authors on request 
A crucial aspect in our approach is the capability to fully capturing sharing cascades on Twitter associated to news articles  It has been reported BIBREF25 that the Twitter streaming endpoint filters out tweets matching a given query if they exceed 1% of the global daily volume of shared tweets, which nowadays is approximately $5\cdot 10^8$; however, as we always collected less than $10^6$ tweets per day, we did not incur in this issue and we thus gathered 100% of tweets matching our query 
Methodology ::: Building diffusion networks
We built Twitter diffusion networks following an approach widely adopted in the literature BIBREF6BIBREF17BIBREF2  We remark that there is an unavoidable limitation in Twitter Streaming API, which does not allow to retrieve true re-tweeting cascades because re-tweets always point to the original source and not to intermediate re-tweeting users BIBREF8BIBREF14; thus we adopt the only viable approach based on Twitter's public availability of data  Besides, by disentangling different interactions with multiple layers we potentially reduce the impact of this limitation on the global network properties compared to the single-layer approach used in our baseline 
Using the notation described in BIBREF26  we employ a multi-layer representation for Twitter diffusion networks  Sociologists have indeed recognized decades ago that it is crucial to study social systems by constructing multiple social networks where different types of ties among same individuals are used BIBREF27  Therefore, for each news article we built a multi-layer diffusion network composed of four different layers, one for each type of social interaction on Twitter platform, namely retweet (RT), reply (R), quote (Q) and mention (M), as shown in Figure FIGREF11  These networks are not necessarily node-aligned, i e  users might be missing in some layers  We do not insert "dummy" nodes to represent all users as it would have severe impact on the global network properties (e g  number of weakly connected components)  Alternatively one may look at each multi-layer diffusion network as an ensemble of individual graphs BIBREF26; since global network properties are computed separately for each layer, they are not affected by the presence of any inter-layer edges 
In our multi-layer representation, each layer is a directed graph where we add edges and nodes for each tweet of the layer type, e g  for the RT layer: whenever user $a$ retweets account $b$ we first add nodes $a$ and $b$ if not already present in the RT layer, then we build an edge that goes from $b$ to $a$ if it does not exists or we increment the weight by 1  Similarly for the other layers: for the R layer edges go from user $a$ (who replies) to user $b$, for the Q layer edges go from user $b$ (who is quoted by) to user $a$ and for the M layer edges go from user $a$ (who mentions) to user $b$ 
Note that, by construction, our layers do not include isolated nodes; they correspond to "pure tweets", i e  tweets which have not originated any interactions with other users  However, they are present in our dataset, and their number is exploited for classification, as described below 
Methodology ::: Global network properties
We used a set of global network indicators which allow us to encode each network layer by a tuple of features  Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector  We used the following global network properties:
Number of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\rightarrow v$, $v\rightarrow u$) 
Size of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph 
Number of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \leftrightarrow v$ ignoring edge directions 
Size of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph 
Diameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph 
Average Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique)  It is computed according to BIBREF28 
Main K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered) 
Density (d): the density for directed graphs is $d=\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph 
Structural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\frac{1}{|V||V-1|}\sum _i\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$  This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\frac{1}{|V||V-1|}$  In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$ 
We used networkx Python package BIBREF30 to compute all features  Whenever a layer is empty  we simply set to 0 all its features  In addition to computing the above nine features for each layer, we added two indicators for encoding information about pure tweets, namely the number T of pure tweets (containing URLs to a given news article) and the number U of unique users authoring those tweets  Therefore, a single diffusion network is represented by a vector with $9\cdot 4+2=38$ entries 
Methodology ::: Interpretation of network features and layers
Aforementioned network properties can be qualitatively explained in terms of social footprints as follows: SCC correlates with the size of the diffusion network, as the propagation of news occurs in a broadcast manner most of the time, i e  re-tweets dominate on other interactions, while LSCC allows to distinguish cases where such mono-directionality is somehow broken  WCC equals (approximately) the number of distinct diffusion cascades pertaining to each news article, with exceptions corresponding to those cases where some cascades merge together via Twitter interactions such as mentions, quotes and replies, and accordingly LWCC and DWCC equals the size and the depth of the largest cascade  CC corresponds to the level of connectedness of neighboring users in a given diffusion network whereas KC identifies the set of most influential users in a network and describes the efficiency of information spreading BIBREF17  Finally, d describes the proportions of potential connections between users which are actually activated and SV indicates whether a news item has gained popularity with a single and large broadcast or in a more viral fashion through multiple generations 
For what concerns different Twitter actions, users primarily interact with each other using retweets and mentions BIBREF20 
The former are the main engagement activity and act as a form of endorsement, allowing users to rebroadcast content generated by other users BIBREF31  Besides, when node B retweets node A we have an implicit confirmation that information from A appeared in B's Twitter feed BIBREF12  Quotes are simply a special case of retweets with comments 
Mentions usually include personal conversations as they allow someone to address a specific user or to refer to an individual in the third person; in the first case they are located at the beginning of a tweet and they are known as replies, otherwise they are put in the body of a tweet BIBREF20  The network of mentions is usually seen as a stronger version of interactions between Twitter users, compared to the traditional graph of follower/following relationships BIBREF32 
Experiments ::: Setup
We performed classification experiments using a basic off-the-shelf classifier, namely Logistic Regression (LR) with L2 penalty; this also allows us to compare results with our baseline  We applied a standardization of the features and we used the default configuration for parameters as described in scikit-learn package BIBREF33  We also tested other classifiers (such as K-Nearest Neighbors, Support Vector Machines and Random Forest) but we omit results as they give comparable performances  We remark that our goal is to show that a very simple machine learning framework, with no parameter tuning and optimization, allows for accurate results with our network-based approach 
We used the following evaluation metrics to assess the performances of different classifiers (TP=true positives, FP=false positives, FN=false negatives):
Precision = $\frac{TP}{TP+FP}$, the ability of a classifier not to label as positive a negative sample 
Recall = $\frac{TP}{TP+FN}$, the ability of a classifier to retrieve all positive samples 
F1-score = $2 \frac{\mbox{Precision} \cdot \mbox{Recall}}{\mbox{Precision} + \mbox{Recall}}$, the harmonic average of Precision and Recall 
Area Under the Receiver Operating Characteristic curve (AUROC); the Receiver Operating Characteristic (ROC) curve BIBREF34, which plots the TP rate versus the FP rate, shows the ability of a classifier to discriminate positive samples from negative ones as its threshold is varied; the AUROC value is in the range $[0, 1]$, with the random baseline classifier holding AUROC$=0 5$ and the ideal perfect classifier AUROC$=1$; thus larger AUROC values (and steeper ROCs) correspond to better classifiers 
In particular we computed so-called macro average–simple unweighted mean–of these metrics evaluated considering both labels (disinformation and mainstream)  We employed stratified shuffle split cross validation (with 10 folds) to evaluate performances 
Finally, we partitioned networks according to the total number of unique users involved in the sharing, i e  the number of nodes in the aggregated network represented with a single-layer representation considering together all layers and also pure tweets  A breakdown of both datasets according to size class (and political biases for the US scenario) is provided in Table 1 and Table 2 
Experiments ::: Classification performances
In Table 3 we first provide classification performances on the US dataset for the LR classifier evaluated on the size class described in Table 1  We can observe that in all instances our methodology performs better than a random classifier (50% AUROC), with AUROC values above 85% in all cases 
For what concerns political biases, as the classes of mainstream and disinformation networks are not balanced (e g , 1,292 mainstream and 4,149 disinformation networks with right bias) we employ a Balanced Random Forest with default parameters (as provided in imblearn Python package BIBREF35)  In order to test the robustness of our methodology, we trained only on left-biased networks or right-biased networks and tested on the entire set of sources (relative to the US dataset); we provide a comparison of AUROC values for both biases in Figure 4  We can notice that our multi-layer approach still entails significant results, thus showing that it can accurately distinguish mainstream news from disinformation regardless of the political bias  We further corroborated this result with additional classification experiments, that show similar performances, in which we excluded from the training/test set two specific sources (one at a time and both at the same time) that outweigh the others in terms of data samples–respectively "breitbart com" for right-biased sources and "politicususa com" for left-biased ones 
We performed classification experiments on the Italian dataset using the LR classifier and different size classes (we excluded $[1000, +\infty )$ which is empty); we show results for different evaluation metrics in Table 3  We can see that despite the limited number of samples (one order of magnitude smaller than the US dataset) the performances are overall in accordance with the US scenario  As shown in Table 4, we obtain results which are much better than our baseline in all size classes (see Table 4):
In the US dataset our multi-layer methodology performs much better in all size classes except for large networks ($[1000, +\infty )$ size class), reaching up to 13% improvement on smaller networks ($[0, 100)$ size class);
In the IT dataset our multi-layer methodology outperforms the baseline in all size classes, with the maximum performance gain (20%) on medium networks ($[100, 1000)$ size class); the baseline generally reaches bad performances compared to the US scenario 
Overall, our performances are comparable with those achieved by two state-of-the-art deep learning models for "fake news" detection BIBREF9BIBREF36 
Experiments ::: Layer importance analysis
In order to understand the impact of each layer on the performances of classifiers, we performed additional experiments considering separately each layer (we ignored T and U features relative to pure tweets)  In Table 5 we show metrics for each layer and all size classes, computed with a 10-fold stratified shuffle split cross validation, evaluated on the US dataset; in Figure 5 we show AUROC values for each layer compared with the general multi-layer approach  We can notice that both Q and M layers alone capture adequately the discrepancies of the two distinct news domains in the United States as they obtain good results with AUROC values in the range 75%-86%; these are comparable with those of the multi-layer approach which, nevertheless, outperforms them across all size classes 
We obtained similar performances for the Italian dataset, as the M layer obtains comparable performances w r t multi-layer approach with AUROC values in the range 72%-82%  We do not show these results for sake of conciseness 
Experiments ::: Feature importance analysis
We further investigated the importance of each feature by performing a $\chi ^2$ test, with 10-fold stratified shuffle split cross validation, considering the entire range of network sizes $[0, +\infty )$  We show the Top-5 most discriminative features for each country in Table 6 
We can notice the exact same set of features (with different relative orderings in the Top-3) in both countries; these correspond to two global network propertie–LWCC, which indicates the size of the largest cascade in the layer, and SCC, which correlates with the size of the network–associated to the same set of layers (Quotes, Retweets and Mentions) 
We further performed a $\chi ^2$ test to highlight the most discriminative features in the M layer of both countries, which performed equally well in the classification task as previously highlighted; also in this case we focused on the entire range of network sizes $[0, +\infty )$  Interestingly, we discovered exactly the same set of Top-3 features in both countries, namely LWCC, SCC and DWCC (which indicates the depth of the largest cascade in the layer) 
An inspection of the distributions of all aforementioned features revealed that disinformation news exhibit on average larger values than mainstream news 
We can qualitatively sum up these results as follows:
Sharing patterns in the two news domains exhibit discrepancies which might be country-independent and due to the content that is being shared 
Interactions in disinformation sharing cascades tends to be broader and deeper than in mainstream news, as widely reported in the literature BIBREF8BIBREF2BIBREF7 
Users likely make a different usage of mentions when sharing news belonging to the two domains, consequently shaping different sharing patterns 
Experiments ::: Temporal analysis
Similar to BIBREF9, we carried out additional experiments to answer the following question: how long do we need to observe a news spreading on Twitter in order to accurately classify it as disinformation or mainstream?
With this goal, we built several versions of our original dataset of multi-layer networks by considering in turn the following lifetimes: 1 hour, 6 hours, 12 hours, 1 day, 2 days, 3 days and 7 days; for each case, we computed the global network properties of the corresponding network and evaluated the LR classifier with 10-fold cross validation, separately for each lifetime (and considering always the entire set of networks)  We show corresponding AUROC values for both US and IT datasets in Figure 6 
We can see that in both countries news diffusion networks can be accurately classified after just a few hours of spreading, with AUROC values which are larger than 80% after only 6 hours of diffusion  These results are very promising and suggest that articles pertaining to the two news domains exhibit discrepancies in their sharing patterns that can be timely exploited in order to rapidly detect misleading items from factual information 
Conclusions
In this work we tackled the problem of the automatic classification of news articles in two domains, namely mainstream and disinformation news, with a language-independent approach which is based solely on the diffusion of news items on Twitter social platform  We disentangled different types of interactions on Twitter to accordingly build a multi-layer representation of news diffusion networks, and we computed a set of global network properties–separately for each layer–in order to encode each network with a tuple of features  Our goal was to investigate whether a multi-layer representation performs better than one layer BIBREF11, and to understand which of the features, observed at given layers, are most effective in the classification task 
Experiments with an off-the-shelf classifier such as Logistic Regression on datasets pertaining to two different media landscapes (US and Italy) yield very accurate classification results (AUROC up to 94%), even when accounting for the different political bias of news sources, which are far better than our baseline BIBREF11 with improvements up to 20%  Classification performances using single layers show that the layer of mentions alone entails better performance w r t other layers in both countries 
We also highlighted the most discriminative features across different layers in both countries; the results suggest that differences between the two news domains might be country-independent but rather due only to the typology of content shared, and that disinformation news shape broader and deeper cascades 
Additional experiments involving the temporal evolution of Twitter diffusion networks show that our methodology can accurate classify mainstream and disinformation news after a few hours of propagation on the platform 
Overall, our results prove that the topological features of multi-layer diffusion networks might be effectively exploited to detect online disinformation  We do not deny the presence of deceptive efforts to orchestrate the regular spread of information on social media via content amplification and manipulation BIBREF37BIBREF38  On the contrary, we postulate that such hidden forces might play to accentuate the discrepancies between the diffusion patterns of disinformation and mainstream news (and thus to make our methodology effective) 
In the future we aim to further investigate three directions: (1) employ temporal networks to represent news diffusion and apply classification techniques that take into account the sequential aspect of data (e g  recurrent neural networks); (2) carry out an extensive comparison of the diffusion of disinformation and mainstream news across countries to investigate deeper the presence of differences and similarities in sharing patterns; (3) leverage our network-based features in addition to state-of-the-art text-based approaches for "fake news" dete ction in order to deliver a real-world system to detect misleading and harmful information spreading on social media 
Context 2: Introduction
Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2   Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3   These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study  One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5  
In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6   In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics  This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems  The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i e , happy, sad, neutral, and angry)  Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons  First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement  Second, the characteristics of emotions must be learned from low-level speech signals  Feature-based models display limited skills when applied to this problem 
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree  Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill  The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day ” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input 
In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech  Extensive experiments are conducted to investigate the efficacy and properties of the proposed model  Our proposed model outperforms previous state-of-the-art methods by 68 8% to 71 8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets  Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes  Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model 
Related work
Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13   Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition  An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14   With the advancement of deep learning methods, more complex neural-based architectures have been proposed  Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17   These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset 
Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models  One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20   Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21  
As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23   However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions 
Model
This section describes the methodologies that are applied to the speech emotion recognition task  We start by introducing the recurrent encoder model for the audio and text modalities individually  We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder 
Audio Recurrent Encoder (ARE)
Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal  Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i e , gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns  This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6   After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data  This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10   The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively  Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12   For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer  The training objective then takes the following form: DISPLAYFORM0
where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1   The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters  C is the total number of classes, and N is the total number of samples used in training  The upper part of Figure shows the architecture of the ARE model 
Text Recurrent Encoder (TRE)
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7   We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal  To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27   Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words  The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2   In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input  Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function 
We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters  The lower part of Figure indicates the architecture of the TRE model 
Multimodal Dual Recurrent Encoder (MDRE)
We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches  In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively  These types of data are the same as those used in the ARE and TRE cases  The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently  The audio-RNN encodes MFCC features from the audio signal using equation EQREF2   The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A  On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2   The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T  Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively  INLINEFORM4 and the bias INLINEFORM5 are learned model parameters 
Multimodal Dual Recurrent Encoder with Attention (MDREA)
Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information  Figure shows the architecture of the MDREA model  First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2   We then consider the final audio encoding vector INLINEFORM0 as a context vector  As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2   Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z  This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters 
Dataset
We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset  This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors  We use categorical evaluations with majority agreement  We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories  The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female)  This data collection process resulted in 10 unique speakers  For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset  The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral) 
Feature extraction
To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals  The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function  According to the length of each wave file, the sequential step of the MFCC features is varied  To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing  The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours  All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26  
Implementation details
Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29   We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts  The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data  The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments  The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32  
In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity  To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts  The performance of the Google ASR system is reflected by its word error rate (WER) of 5 53% 
Performance evaluation
As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model  The data in each fold are split into training, development, and testing datasets (8:0 5:1 5, respectively)  After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset  We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation 
We examine the WAP values, which are shown in Table 1  First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures  On the other hand, the TRE model shows higher performance gain compared to the ARE  From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data  Second, the newly proposed model, MDRE, shows a substantial performance gain  It thus achieves the state-of-the-art performance with a WAP value of 0 718  This result shows that multimodal information is a key factor in affective computing  Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0 690 to 0 688) BIBREF19   However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture  We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model  Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism  We leave the implementation of this point as a future research direction 
To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table )  The label accuracy of the processed transcripts is 5 53% WER  The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models  However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0 691 to 0 688) 
Error analysis
We analyze the predictions of the ARE, TRE, and MDRE models  Figure shows the confusion matrix of each model  The ARE model (Fig  ) incorrectly classifies most instances of happy as neutral (43 51%); thus, it shows reduced accuracy (35 15%) in predicting the the happy class  Overall, most of the emotion classes are frequently confused with the neutral class  This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes 
Interestingly, the TRE model (Fig  ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35 15% to 75 73%)  This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data  On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16 20% of the time, even though these emotional states are opposites of one another 
The MDRE model (Fig  ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree  The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased  Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16 20% to 9 15% 
Conclusions
In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data  Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class  Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68 8% to 71 8% are obtained when the model is applied to the IEMOCAP dataset  In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features 
In the future work, we aim to extend the modalities to audio, text and video inputs  Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities  This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No 10073144) 

ground_truth: By assigning a political bias label to each news article and training only on left-biased or right-biased outlets of both disinformation and mainstream domains

we also account for political biases inherent to different news sources, referring to the procedure proposed in BIBREF2 to label different outlets. Overall we show that we are able to classify credible vs non-credible diffusion networks (and consequently news articles) with high accuracy (AUROC up to 94%), even when accounting for the political bias of sources (and training only on left-biased or right-biased articles). We observe that the layer of mentions alone conveys useful information for the classification, denoting a different usage of this functionality when sharing news belonging to the two news domains. We also show that most discriminative features, which are relative to the breadth and depth of largest cascades in different layers, are the same across the two countries.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. By how much does their model outperform the state of the art results?
Context 3: Introduction
Deep neural networks (DNNs), in particular convolutional and recurrent neural networks, with huge architectures have been proven successful in wide range of tasks including audio processing such as speech to text [1 - 4], emotion recognition [5 - 8], speech/non-speech (e g , of non-speech include noise, music, etc ,) classification [9 - 12], etc 
Training these deep architectures require large amount of annotated data, as a result, they cannot be used in low data resource scenarios which is common in speech-based applications [13 - 15]  Apart from collecting large data corpus, annotating the data is also very difficult, and requires manual supervision and efforts  Especially, annotation of speech for tasks like emotion recognition also suffer from lack of agreement among the annotators [16]  Hence, there is a need to build reliable systems that can work in low resource scenario 
In this work, we propose a novel approach to address the task of classification in low data resource scenarios  Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier  We call this approach as simultaneous two sample learning (s2sL)  The proposed approach is also applicable to low resource data suffering with data imbalance  The contributions of this paper are:
Proposed approach
The s2sL approach proposed to address low data resource problem is explained in this Section  In this work, we use MLP (modified to handle our data representation) as the base classifier  Here, we explain the s2sL approach by considering two-class classification task 
Data representation
Consider a two-class classification task with INLINEFORM0 denoting the set of class labels, and let INLINEFORM1 and INLINEFORM2 be the number of samples corresponding to INLINEFORM3 and INLINEFORM4 , respectively  In general, to train a classifier, the samples in the train set are provided as input-output pairs as follows  DISPLAYFORM0
where INLINEFORM0 refers to the INLINEFORM1 -dimensional feature vector representing the INLINEFORM2 sample corresponding to INLINEFORM3 class label, and INLINEFORM4 , refers to output label of INLINEFORM5 class  In the proposed data representation format, called simultaneous two sample (s2s) representation, we will simultaneously consider two samples as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 are the INLINEFORM2 -dimensional feature vectors representing the INLINEFORM3 sample in INLINEFORM4 class and INLINEFORM5 sample in the INLINEFORM6 class, respectively; and INLINEFORM7 , INLINEFORM8 refers to the output labels of INLINEFORM9 and INLINEFORM10 class, respectively 
Hence, in s2s data representation, we will have an input feature vector of length INLINEFORM0 i e , INLINEFORM1 , and output class labels as either INLINEFORM2 or INLINEFORM3   Each sample can be combined with all the samples (i e , with ( INLINEFORM4 ) samples) in the dataset  Therefore, by representing the data in the s2s format, the number of samples in the train set increases to INLINEFORM5 from INLINEFORM6 samples  We hypothesize that the s2s format is expected to help the network not only to learn the characteristics of the two classes separately, but also the difference and similarity in characteristics of the two classes 
Classifier training
MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework  Generally, MLPs are trained using the data format given by eq  INLINEFORM0   But to train the MLP on our s2s based data representation (as in eq  INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ) 
We have INLINEFORM0 units (instead of INLINEFORM1 units) in the input layer to accept the two samples i e , INLINEFORM2 and INLINEFORM3 , simultaneously 
The structure of the hidden layer in this approach is similar to that of a regular MLP  The number of hidden layers and hidden units can be varied depending upon the complexity of the problem  The number of units in the hidden layer is selected empirically by varying the number of hidden units from 2 to twice the length of the input layer (i e , 2 to INLINEFORM0 ) and the unit at which the highest performance is obtained are selected  In this paper, we considered only a single hidden layer  Rectified linear units (ReLU) are used for hidden layer 
The output layer will consist of units equal to twice the considered number of classes in the classification task i e, the output layer will have four units for two-class classification task  The sigmoid activation function (not softmax) is used on the output layer units  Unlike regular MLP, we use sigmoid activation units in the output layer, with binary cross-entropy as the cost function, because the output labels in the proposed s2s based data representation will have more than one unit active at a time (not one-hot encoded output) and this condition cannot be handled using softmax function 
As can be seen from Figure FIGREF4 , the output layer in our proposed method has outputs INLINEFORM0 and INLINEFORM1 which corresponds to the outputs associated with the input feature vectors INLINEFORM2 and INLINEFORM3 , respectively  For a two-class classification problem, there will be four units in the output layer and the possible output labels are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 corresponding to the class labels INLINEFORM8 and INLINEFORM9 , respectively  This architecture is referred to as s2s-MLP  In s2sL, s2s-MLP is trained using the s2s data representation format  Further, the s2s-MLP is trained using adam optimizer 
Classifier testing
Generally, the feature vector corresponding to the test sample is provided as input to the trained MLP in the testing phase and the class label is decided based on the obtained output 
However, in s2sL method, the feature vector corresponding to the test sample should also be converted to the s2s data representation format to test the trained s2s-MLP  We propose a testing approach, where the given test sample is combined with a set of preselected reference samples, whose class label is known a priori, to generate multiple instances of the same test sample as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 refer to the INLINEFORM2 -dimensional feature vector corresponding to the test sample and the INLINEFORM3 reference sample, respectively  INLINEFORM4 refers to the considered number of reference samples  These reference samples can be selected from any of the two classes 
For testing the s2s-MLP (as shown in Figure FIGREF4 ), each test sample INLINEFORM0 (same as ' INLINEFORM1 ' in (3)) is combined with all the INLINEFORM2 reference samples ( INLINEFORM3 ) to form INLINEFORM4 different instances of the same test sample INLINEFORM5   The corresponding outputs ( INLINEFORM6 ) obtained from s2s-MLP for the INLINEFORM7 generated instances of INLINEFORM8 are combined by voting-based decision approach to obtain the final decision INLINEFORM9   The class label that gets maximum votes is considered as the predicted output label 
Experiments
We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification  We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music  Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors  It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios 
The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification  Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]  For two class classification, we considered the two most confusing emotion pairs i e , (Neutral,Sad) and (Anger, Happy)  Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples)  Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets 
All experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold)  Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system  For this analysis, we considered 4 different proportions i e , INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier  For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data  5-fold cross validation is considered for all data proportions  Accuracy (in %) is used as a performance measure for balanced data classification tasks (i e , Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i e , Anger/Happy emotion classification) 
Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data  The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation  It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions  s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments 
Table TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem)  Here, state-of-the-art methods i e , Eusboost [22] and MWMOTE [23] are also considered for comparison  It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered)  In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems  s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used 
Conclusions
In this paper, we introduced a novel s2s framework to effectively learn the class discriminative characteristics, even from low data resources  In this framework, more than one sample (here, two samples) are simultaneously considered to train the classifier  Further, this framework allows to generate multiple instances of the same test sample, by considering preselected reference samples, to achieve a more profound decision making  We illustrated the significance of our approach by providing the experimental results for two different tasks namely, speech/music discrimination and emotion classification  Further, we showed that the s2s framework can also handle the low resourced data imbalance problem 
References
[1] Hinton, G , Deng, L , Yu, D , Dahl, G  E , Mohamed, A  R , Jaitly, N , Senior, A , Vanhoucke, V , Nguyen, P , Sainath, T  N  & Kingsbury, B  (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups  IEEE Signal Processing Magazine, pp  82–97 
[2] Vinyals, O , Ravuri, S  V  & Povey, D  (2012) Revisiting recurrent neural networks for robust ASR  In Proc  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp  4085–4088 
[3] Lu, L , Zhang, X , Cho K  & Renals, S  (2015) A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition  In Proc  INTERSPEECH, pp  3249–3253 
[4] Zhang, Y , Pezeshki, M , Brakel, P , Zhang, S , Bengio, C L Y  & Courville, A  (2017) Towards end-to-end speech recognition with deep convolutional neural networks  arXiv preprint, arXiv:1701

ground_truth: the attention model, MDREA, also outperforms the best existing research results (WAP 0.690 to 0.688)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which models/frameworks do they compare to?
Context 3: 02720 
[5] Han, K , Yu, D  & Tashev, I  (2014) Speech emotion recognition using deep neural network and extreme learning machine  In Proc  Interspeech, pp  223–227 
[6] Trigeorgis, G , Ringeval, F , Brueckner, R , Marchi, E , Nicolaou, M  A , Schuller, B , & Zafeiriou, S  (2016) Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network  In Proc  ICASSP, pp  5200–5204) 
[7] Huang, Che-Wei, & Shrikanth S  Narayanan  (2016) Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition  In Proc  INTERSPEECH, pp  1387–1391 
[8] Zhang, Z , Ringeval, F , Han, J , Deng, J , Marchi, E  & Schuller, B  (2016) Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks  In Proc  INTERSPEECH, pp  3593–3597 
[9] Scheirer, E  D  & Slaney, M  (2003) Multi-feature speech/music discrimination system  U S  Patent 6,570,991 
[10] Pikrakis, A  & Theodoridis S  (2014) Speech-music discrimination: A deep learning perspective  In Proc  European Signal Processing Conference (EUSIPCO), pp  616–620 
[11] Choi, K , Fazekas, G  & Sandler, M  (2016) Automatic tagging using deep convolutional neural networks  arXiv preprint, arXiv:1606 00298 
[12] Zazo Candil, R , Sainath, T N , Simko, G  & Parada, C  (2016) Feature learning with raw-waveform CLDNNs for Voice Activity Detection  In Proc  Interspeech, pp  3668–3672 
[13] Tian, L , Moore, JD  & Lai, C  (2015) Emotion recognition in spontaneous and acted dialogues  In Proc  International Conference on Affective Computing and Intelligent Interaction (ACII), pp  698–704 
[14] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2017) k-FFNN: A priori knowledge infused feed-forward neural networks  arXiv preprint, arXiv:1704 07055 
[15] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2018) Knowledge-driven feed-forward neural network for audio affective content analysis  To appear in AAAI-2018 workshop on Affective Content Analysis 
[16] Deng, J , Zhang, Z , Marchi, E  & Schuller, B  (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition  In Proc  Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp  511–516 
[17] George Tzanetakis, Gtzan musicspeech  Availabe on-line at http://marsyas info/download/datasets 
[18] Burkhardt, F , Paeschke, A , Rolfes, M , Sendlmeier, W F  & Weiss, B  (2005) A database of german emotional speech  In Proc  Interspeech, pp  1517–1520 
[19] Hall, M , Frank, E , Holmes, G , Pfahringer, B , Reutemann, P  & Witten, I H  (2009) The WEKA data mining software: an update  ACM SIGKDD explorations newsletter, 11(1), pp  10–18 
[20] Eyben, F , Weninger, F , Gross, F  & Schuller, B  (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor  In Proc  ACM international conference on Multimedia, pp  835–838 
[21] Maratea, A , Petrosino, A  & Manzo, M  (2014) Adjusted f-measure and kernel scaling for imbalanced data learning  Information Sciences 257:331–341 
[22] Galar, M , Fernandez, A , Barrenechea, E  & Herrera, F  (2013) Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling  Pattern Recognition 46(12):3460–3471 
[23] Barua, S , Islam, M  M , Yao, X  & Murase, K  (2014) Mwmote–majority weighted minority oversampling technique for imbalanced data set learning  IEEE Transactions on Knowledge and Data Engineering 26(2):405–425 

ground_truth: MLP

Eusboost, MWMOTE