prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What can word subspace represent?
Context 1: Introduction
Text classification has become an indispensable task due to the rapid growth in the number of texts in digital form available online  It aims to classify different texts, also called documents, into a fixed number of predefined categories, helping to organize data, and making easier for users to find the desired information  Over the past three decades, many methods based on machine learning and statistical models have been applied to perform this task, such as latent semantic analysis (LSA), support vector machines (SVM), and multinomial naive Bayes (MNB) 
The first step in utilizing such methods to categorize textual data is to convert the texts into a vector representation  One of the most popular text representation models is the bag-of-words model BIBREF0 , which represents each document in a collection as a vector in a vector space  Each dimension of the vectors represents a term (e g , a word, a sequence of words), and its value encodes a weight, which can be how many times the term occurs in the document 
Despite showing positive results in tasks such as language modeling and classification BIBREF1 , BIBREF2 , BIBREF3 , the BOW representation has limitations: first, feature vectors are commonly very high-dimensional, resulting in sparse document representations, which are hard to model due to space and time complexity  Second, BOW does not consider the proximity of words and their position in the text and consequently cannot encode the words semantic meanings 
To solve these problems, neural networks have been employed to learn vector representations of words BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7   In particular, the word2vec representation BIBREF8 has gained attention  Given a training corpus, word2vec can generate a vector for each word in the corpus that encodes its semantic information  These word vectors are distributed in such a way that words from similar contexts are represented by word vectors with high correlation, while words from different contexts are represented by word vectors with low correlation 
One crucial aspect of the word2vec representation is that arithmetic and distance calculation between two word vectors can be performed, giving information about their semantic relationship  However, rather than looking at pairs of word vectors, we are interested in studying the relationship between sets of vectors as a whole and, therefore, it is desirable to have a text representation based on a set of these word vectors 
To tackle this problem, we introduce the novel concept of word subspace  It is mathematically defined as a low dimensional linear subspace in a word vector space with high dimensionality  Given that words from texts of the same class belong to the same context, it is possible to model word vectors of each class as word subspaces and efficiently compare them in terms of similarity by using canonical angles between the word subspaces  Through this representation, most of the variability of the class is retained  Consequently, a word subspace can effectively and compactly represent the context of the corresponding text  We achieve this framework through the mutual subspace method (MSM) BIBREF9  
The word subspace of each text class is modeled by applying PCA without data centering to the set of word vectors of the class  When modeling the word subspaces, we assume only one occurrence of each word inside the class 
However, as seen in the BOW approach, the frequency of words inside a text is an informative feature that should be considered  In order to introduce this feature in the word subspace modeling and enhance its performance, we further extend the concept of word subspace to the term-frequency (TF) weighted word subspace 
In this extension, we consider a set of weights, which encodes the words frequencies, when performing the PCA  Text classification with TF weighted word subspace can also be performed under the framework of MSM  We show the validity of our modeling through experiments on the Reuters database, an established database for natural language processing tasks  We demonstrate the effectiveness of the word subspace formulation and its extension, comparing our methods' performance to various state-of-art methods 
The main contributions of our work are:
The remainder of this paper is organized as follows  In Section "Related Work" , we describe the main works related to text classification  In Section "Word subspace" , we present the formulation of our proposed word subspace  In Section "Conventional text classification methods" , we explain how text classification with word subspaces is performed under the MSM framework  Then, we present the TF weighted word subspace extension in Section "TF weighted word subspace"   Evaluation experiments and their results are described in Section "Experimental Evaluation"   Further discussion is then presented in Section "Discussion" , and our conclusions are described in Section "Conclusions and Future Work"  
Related Work
In this section, we outline relevant work towards text classification  We start by describing how text data is conventionally represented using the bag-of-words model and then follow to describe the conventional methods utilized in text classification 
Text Representation with bag-of-words
The bag-of-words representation comes from the hypothesis that frequencies of words in a document can indicate the relevance of the document to a query BIBREF0 , that is, if documents and a query have similar frequencies for the same words, they might have a similar meaning  This representation is based on the vector space model (VSM), that was developed for the SMART information retrieval system BIBREF10   In the VSM, the main idea is that documents in a collection can be represented as a vector in a vector space, where vectors close to each other represent semantically similar documents 
More formally, a document $d$ can be represented by a vector in $\mathbb {R}^{n}$ , where each dimension represents a different term  A term can be a single word, constituting the conventional bag-of-words, or combinations of $N$ words, constituting the bag-of-N-grams  If a term occurs in the document, its position in the vector will have a non-zero value, also known as term weight  Two documents in the VSM can be compared to each other by taking the cosine distance between them BIBREF1  
There are several ways to compute the term weights  Among them, we can highlight some: Binary weights, term-frequency (TF) weights, and term-frequency inverse document-frequency (TF-IDF) weights 
Consider a corpus with documents $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ and a vocabulary with all terms in the corpus $V = \lbrace w_i\rbrace _{i=1}^{|V|}$   The term weights can be defined as:
Binary weight: If a term occurs in the document, its weight is 1  Otherwise, it is zero 
Term-frequency weight (TF): The weight of a term $w$ is defined by the number of times it occurs in the document $d$  
$$TF(w,d) = n_d^w$$   (Eq  8)
Inverse document-frequency: The weight of a term $w$ , given the corpus $D$ , is defined as the total number of documents $|D|$ divided by the number of documents that have the term $w$ , $|D^w|$  
$$IDF(w | D) = \frac{|D|}{|D^w|}$$   (Eq  10)
Term-frequency inverse document-frequency (TF-IDF): The weight of a term $w$ is defined by the multiplication of its term-frequency and its inverse document-frequency  When considering only the TF weights, all terms have the same importance among the corpus  By using the IDF weight, words that are more common across all documents in $D$ receive a smaller weight, giving more importance to rare terms in the corpus 
$$TFIDF(w,d | D)=TF \times IDF$$   (Eq  12)
In very large corpus, it is common to consider the logarithm of the IDF in order to dampen its effect 
$$TFIDF(w,d | D)=TF \times log_{10}(IDF)$$   (Eq  13)
Conventional text classification methods
Multi-variate Bernoulli (MVB) and multinomial naive Bayes (MNB) are two generative models based on the naive Bayes assumption  In other words, they assume that all attributes (e g , the frequency of each word, the presence or absence of a word) of each text are independent of each other given the context of the class BIBREF11  
In the MVB model, a document is represented by a vector generated by a bag-of-words with binary weights  In this case, a document can be considered an event, and the presence or the absence of the words to be the attributes of the event  On the other hand, the MNB model represents each document as a vector generated by a bag-of-words with TF weights  Here, the individual word occurrences are considered as events and the document is a collection of word events 
Both these models use the Bayes rule to classify a document  Consider that each document should be classified into one of the classes in $C=\lbrace c_j\rbrace _{j=1}^{|C|}$   The probability of each class given the document is defined as:
$$P(c_j|d_i) = \frac{P(d_i|c_j)P(c_j)}{P(d_i)} $$   (Eq  16)
The prior $P(d_i)$ is the same for all classes, so to determine the class to which $d_i$ belongs to, the following equation can be used:
$$prediction(d_i) = argmax_{c_j}P(d_i|c_j)P(c_j)$$   (Eq  17)
The prior $P(c_j)$ can be obtained by the following equation:
$$P(c_j) = \frac{1+|D_j|}{|C|+|D|},$$   (Eq  18)
where $|D_j|$ is the number of documents in class $c_j$  
As for the posterior $P(d_i|c_j)$ , different calculations are performed for each model  For MVB, it is defined as:
$$P(d_i|c_j) = \prod _{k=1}^{|V|}P(w_k|c_j)^{t_i^k}(1-P(w_k|c_j))^{1-t_i^k},$$   (Eq  19)
where $w_k$ is the k-th word in the vocabulary $V$ , and $t_i^k$ is the value (0 or 1) of the k-th element of the vector of document $d_i$  
For the MNB, it is defined as:
$$P(d_i|c_j) = P(|d_i|)|d_i|!\prod _{k=1}^{|V|}\frac{P(w_k|c_j)^{n_i^k}}{n_i^k!},$$   (Eq  20)
where $|d_i|$ is the number of words in document $d_i$ and $n_i^k$ is the k-th element of the vector of document $d_i$ and it represents how many times word $w_k$ occurs in $d_i$  
Finally, the posterior $P(w_k|c_j)$ can be obtained by the following equation:
$$P(w_k|c_j) = \frac{1+|D_j^k|}{|C|+|D|},$$   (Eq  21)
where $|D_j^k|$ is the number of documents in class $c_j$ that contain the word $w_k$  
In general, MVB tends to perform better than MNB at small vocabulary sizes whereas MNB is more efficient on large vocabularies 
Despite being robust tools for text classification, both these models depend directly on the bag-of-words features and do not naturally work with representations such as word2vec 
Latent semantic analysis (LSA), or latent semantic indexing (LSI), was proposed in BIBREF12 , and it extends the vector space model by using singular value decomposition (SVD) to find a set of underlying latent variables which spans the meaning of texts 
It is built from a term-document matrix, in which each row represents a term, and each column represents a document  This matrix can be built by concatenating the vectors of all documents in a corpus, obtained using the bag-of-words model, that is, $ {X} = [ {v}_1, {v}_2,    , {v}_{|D|}]$ , where ${v}_i$ is the vector representation obtained using the bag-of-words model 
In this method, the term-document matrix is decomposed using the singular value decomposition,
$${X} = {U\Sigma V}^\top ,$$   (Eq  23)
where $U$ and $V$ are orthogonal matrices and correspond to the left singular vectors and right singular vectors of $X$ , respectively  $\Sigma $ is a diagonal matrix, and it contains the square roots of the eigenvalues of $X^TX$ and $XX^T$   LSA finds a low-rank approximation of $X$ by selecting only the $k$ largest singular values and its respective singular vectors,
$${X}_k = {U}_k{\Sigma }_k {V}_k^{\top } $$   (Eq  24)
To compare two documents, we project both of them into this lower dimension space and calculate the cosine distance between them  The projection ${\hat{d}}$ of document ${d}$ is obtained by the following equation:
$${\hat{d}} = {\Sigma }_k^{-1} {U}_k^\top {d} $$   (Eq  25)
Despite its extensive application on text classification BIBREF13 , BIBREF14 , BIBREF15 , this method was initially proposed for document indexing and, therefore, does not encode any class information when modeling the low-rank approximation  To perform classification, 1-nearest neighbor is usually performed, placing a query document into the class of the nearest training document 
The support vector machine (SVM) was first presented in BIBREF16 and performs the separation between samples of two different classes by projecting them onto a higher dimensionality space  It was first applied in text classification by BIBREF17 and have since been successfully applied in many tasks related to natural language processing BIBREF18 , BIBREF19  
Consider a training data set $D$ , with $n$ samples
$$D = \lbrace ({x}_i,c_i)|{x}_i\in \mathbb {R}^p, c_i \in \lbrace -1,1\rbrace  \rbrace _{i=1}^{n},$$   (Eq  27)
where $c_i$ represents the class to which ${x}_i$ belongs to  Each ${x}_i$ is a $p$ -dimensional vector  The goal is to find the hyperplane that divides the points from $c_i = 1$ from the points from $c_i = -1$   This hyperplane can be written as a set of points $x$ satisfying:
$${w} \cdot {x} - b = 0,$$   (Eq  28)
where $\cdot $ denotes the dot product  The vector ${w}$ is perpendicular to the hyperplane  The parameter $\frac{b}{\Vert {w}\Vert }$ determines the offset of the hyperplane from the origin along the normal vector ${w}$  
We wish to choose ${w}$ and $b$ , so they maximize the distance between the parallel hyperplanes that are as far apart as possible, while still separating the data 
If the training data is linearly separable, we can select two hyperplanes in a way that there are no points between them and then try to maximize the distance  In other words, minimize $\Vert {w}\Vert $ subject to $c_i({w}\cdot {x}_u-b) \ge 1, i=\lbrace 1,2,   ,n\rbrace $   If the training data is not linearly separable, the kernel trick can be applied, where every dot product is replaced by a non-linear kernel function 
Word subspace
All methods mentioned above utilize the BOW features to represent a document  Although this representation is simple and powerful, its main problem lies on disregarding the word semantics within a document, where the context and meaning could offer many benefits to the model such as identification of synonyms 
In our formulation, words are represented as vectors in a real-valued feature vector space $\mathbb {R}^{p}$ , by using word2vec BIBREF8   Through this representation, it is possible to calculate the distance between two words, where words from similar contexts are represented by vectors close to each other, while words from different contexts are represented as far apart vectors  Also, this representation brings the new concept of arithmetic operations between words, where operations such as addition and subtraction carry meaning (eg , “king”-“man”+“woman”=“queen”) BIBREF20  
Consider a set of documents which belong to the same context $D_c = \lbrace d_i\rbrace _{i=1}^{|D_c|}$   Each document $d_i$ is represented by a set of $N_i$ words, $d_i = \lbrace w_k\rbrace _{k=1}^{N_i}$   By considering that all words from documents of the same context belong to the same distribution, a set of words $W_c = \lbrace w_k\rbrace _{k=1}^{N_c}$ with the words in the context $c$ is obtained 
We then translate these words into word vectors using word2vec, resulting in a set of word vectors $X_c = \lbrace {x}^k_c\rbrace _{k=1}^{N_c} \in \mathbb {R}^p$   This set of word vectors is modeled into a word subspace, which is a compact, scalable and meaningful representation of the whole set  Such a word subspace is generated by applying PCA to the set of word vectors 
First, we compute an autocorrelation matrix, ${R}_c$ :
$${R}_c = \frac{1}{N_c}\sum _{i=1}^{N_c}{x}^{i}_c{x}_c^{i^{\top }} $$   (Eq  29)
The orthonormal basis vectors of $m_c$ -dimensional subspace ${Y}_c$ are obtained as the eigenvectors with the $m_c$ largest eigenvalues of the matrix ${R}_c$   We represent a subspace ${Y}_c$ by the matrix ${Y}_c \in \mathbb {R}^{p \times m_c}$ , which has the corresponding orthonormal basis vectors as its column vectors 
Text classification based on word subspace
We formulate our problem as a single label classification problem  Given a set of training documents, which we will refer as corpus, $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ , with known classes $C = \lbrace c_j\rbrace _{j=1}^{|C|}$ , we wish to classify a query document $d_q$ into one of the classes in $C$  
Text classification based on word subspace can be performed under the framework of mutual subspace method (MSM)  This task involves two different stages: A learning stage, where the word subspace for each class is modeled, and a classification stage, where the word subspace for a query is modeled and compared to the word subspaces of the classes 
In the learning stage, it is assumed that all documents of the same class belong to the same context, resulting in a set of words $W_c = \lbrace w_c^k\rbrace _{k=1}^{N_c}$   This set assumes that each word appears only once in each class  Each set $\lbrace W_c\rbrace _{c=1}^{|C|}$ is then modeled into a word subspace ${Y}_c$ , as explained in Section "Word subspace"   As the number of words in each class may vary largely, the dimension $m_c$ of each class word subspace is not set to the same value 
In the classification stage, for a query document $d_q$ , it is also assumed that each word occurs only once, generating a subspace ${Y}_q$  
To measure the similarity between a class word subspace ${Y}_c$ and a query word subspace ${Y}_q$ , the canonical angles between the two word subspaces are used BIBREF21   There are several methods for calculating canonical angles BIBREF22 , BIBREF23 , and BIBREF24 , among which the simplest and most practical is the singular value decomposition (SVD)  Consider, for example, two subspaces, one from the training data and another from the query, represented as matrices of bases, ${Y}_{c} = [{\Phi }_{1} \ldots {\Phi }_{m_c}] \in \mathbb {R}^{p \times m_c}$ and ${Y}_{q} = [{\Psi }_{1} \ldots {\Psi }_{m_q}] \in \mathbb {R}^{p \times m_q}$ , where ${\Phi }_{i}$ are the bases for ${Y}_c$ and ${\Psi }_{i}$ are the bases for ${Y}_q$   Let the SVD of ${Y}_c^{\top }{Y}_q \in \mathbb {R}^{m_c \times m_q}$ be ${Y}_c^{\top }{Y}_q = {U \Sigma V}^{\top }$ , where ${Y}_q$0 , ${Y}_q$1 represents the set of singular values  The canonical angles ${Y}_q$2 can be obtained as ${Y}_q$3 ${Y}_q$4   The similarity between the two subspaces is measured by ${Y}_q$5 angles as follows:
$$S_{({Y}_c,{Y}_q)}[t] = \frac{1}{t}\sum _{i = 1}^{t} \cos ^{2} \theta _{i},\; 1 \le t \le m_q, \; m_q \le m_c $$   (Eq  30)
Fig  1 shows the modeling and comparison of sets of words by MSM  This method can compare sets of different sizes, and naturally encodes proximity between sets with related words 
Finally, the class with the highest similarity with $d_q$ is assigned as the class of $d_q$ :
$$prediction(d_q) = argmax_c(S_{({Y}_c,{Y}_q)}) $$   (Eq  32)
TF weighted word subspace
The word subspace formulation presented in Section "Word subspace" is a practical and compact way to represent sets of word vectors, retaining most of the variability of features  However, as seen in the BOW features, the frequency of words is relevant information that can improve the characterization of a text  To incorporate this information into the word subspace modeling, we propose an extension of the word subspace, called the term-frequency (TF) weighted word subspace 
Like the word subspace, the TF weighted word subspace is mathematically defined as a low-dimensional linear subspace in a word vector space with high dimensionality  However, a weighted version of the PCA BIBREF25 , BIBREF26 is utilized to incorporate the information given by the frequencies of words (term-frequencies)  This TF weighted word subspace is equivalent to the word subspace if we consider all occurrences of the words 
Consider the set of word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c} \in \mathbb {R}^{p}$ , which represents each word in the context $c$ , and the set of weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$ , which represent the frequencies of the words in the context $c$  
We incorporate these frequencies into the subspace calculation by weighting the data matrix ${X}$ as follows:
$${\widetilde{X}}={X}{\Omega }^{1/2},$$   (Eq  33)
where ${X} \in \mathbb {R}^{p \times N_c}$ is a matrix containing the word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c}$ and ${\Omega }$ is a diagonal matrix containing the weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$  
We then perform PCA by solving the SVD of the matrix ${\widetilde{X}}$ :
$${\widetilde{X}}={AMB}^{\top },$$   (Eq  34)
where the columns of the orthogonal matrices ${A}$ and ${B}$ are, respectively, the left-singular vectors and right-singular vectors of the matrix ${\widetilde{X}}$ , and the diagonal matrix ${M}$ contains singular values of ${\widetilde{X}}$  
Finally, the orthonormal basis vectors of the $m_c$ -dimensional TF weighted subspace ${W}$ are the column vectors in ${A}$ corresponding to the $m_c$ largest singular values in ${M}$  
Text classification with TF weighted word subspace can also be performed under the framework of MSM  In this paper, we will refer to MSM with TF weighted word subspace as TF-MSM 
Experimental Evaluation
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension  We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578  Words in the texts were considered as they appeared, without performing stemming or typo correction  This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1  
To obtain the vector representation of words, we used a freely available word2vec model, trained by BIBREF8 , on approximately 100 billion words, which encodes the vector representation in $\mathbb {R}^{300}$ of over 3 million words from several different languages  Since we decided to focus on English words only, we filtered these vectors to about 800 thousand words, excluding all words with non-roman characters 
To show the validity of our word subspace representation for text classification and the proposed extension, we divided our experiment section into two parts: The first one aims to verify if sets of word vectors are suitable for subspace representation, and the second one puts our methods in practice in a text classification test, comparing our results with the conventional methods described in Section "Related Work"  
Evaluation of the word subspace representation
In this experiment, we modeled the word vectors from each class in the Reuters-8 database into a word subspace  The primary goal is to visualize how much of the text data can be represented by a lower dimensional subspace 
Subspace representations are very efficient in compactly represent data that is close to a normal distribution  This characteristic is due to the application of the PCA, that is optimal to find the direction with the highest variation within the data 
In PCA, the principal components give the directions of maximum variance, while their corresponding eigenvalues give the variance of the data in each of them  Therefore, by observing the distribution of the eigenvalues computed when performing PCA in the modeling of the subspaces, we can suggest if the data is suitable or not for subspace representation 
For each class, we normalized the eigenvalues by the largest one of the class  Fig  2 shows the mean of the eigenvalues and the standard deviation among classes  It is possible to see that the first largest eigenvalues retain larger variance than the smallest ones  In fact, looking at the first 150 largest eigenvalues, we can see that they retain, on average, 86 37% of the data variance  Also, by observing the standard deviation, we can understand that the eigenvalues distribution among classes follows the same pattern, that is, most of the variance is in the first dimensions  This plot indicates that text data represented by vectors generated with word2vec is suitable for subspace representation 
Text classification experiment
In this experiment, we performed text classification among the classes in the Reuters-8 database  We compared the classification using the word subspace, and its weighted extension, based on MSM (to which we will refer as MSM and TF-MSM, respectively) with the baselines presented in Section "Related Work" : MVB, MNB, LSA, and SVM  Since none of the baseline methods work with vector set classification, we also compared to a simple baseline for comparing sets of vectors, defined as the average of similarities between all vector pair combinations of two given sets  For two matrices ${A}$ and ${B}$ , containing the sets of vectors $\lbrace  {x}^{i}_a \rbrace _{i = 1}^{N_A}$ and $\lbrace  {x}^{i}_b \rbrace _{i = 1}^{N_B}$ , respectively, where $N_A$ and $N_B$ are the number of main words in each set, the similarity is defined as:
$$Sim_{(A,B)} = \frac{1}{N_A N_B}\sum _{i}^{N_A}\sum _{j}^{N_B}{{x}_a^i}^{\top }{x}_b^j $$   (Eq  41)
We refer to this baseline as similarity average (SA)  For this method, we only considered one occurrence of each word in each set 
Different features were used, depending on the method  Classification with SA, MSM, and TF-MSM was performed using word2vec features, to which we refer as w2v  For MVB, due to its nature, only bag-of-words features with binary weights were used (binBOW)  For the same reason, we only used bag-of-words features with term-frequency weights (tfBOW) with MNB  Classification with LSA and SVM is usually performed using bag-of-words features and, therefore, we tested with binBOW, tfBOW, and with the term-frequency inverse document-frequency weight, tfidfBOW  We also tested them using word2vec vectors  In this case, we considered each word vector from all documents in each class to be a single sample 
To determine the dimensions of the class subspaces and query subspace of MSM and TF-MSM, and the dimension of the approximation performed by LSA, we performed a 10-fold cross validation, wherein each fold, the data were randomly divided into train (60%), validation (20%) and test set (20%) 
The results can be seen in Table 2   The simplest baseline, SA with w2v, achieved an accuracy rate of 78 73%  This result is important because it shows the validity of the word2vec representation, performing better than more elaborate methods based on BOW, such as MVB with binBOW 
LSA with BOW features was almost 10% more accurate than SA, where the best results with binary weights were achieved with an approximation with 130 dimensions, with TF weights were achieved with 50 dimensions, and with TF-IDF weights were achieved with 30 dimensions  SVM with BOW features was about 3% more accurate than LSA, with binary weights leading to a higher accuracy rate 
It is interesting to note that despite the reasonably high accuracy rates achieved using LSA and SVM with BOW features, they poorly performed when using w2v features 
Among the baselines, the best method was MNB with tfBOW features, with an accuracy of 91 47%, being the only conventional method to outperform MSM  MSM with w2v had an accuracy rate of 90 62%, with the best results achieved with word subspace dimensions for the training classes ranging from 150 to 181, and for the query ranging from 3 to 217  Incorporating the frequency information in the subspace modeling resulted in higher accuracy, with TF-MSM achieving 92 01%, with dimensions of word subspaces for training classes ranging from 150 to 172, and for the query, ranging from 2 to 109  To confirm that TF-MSM is significantly more accurate than MNB, we performed a t-test to compare their results  It resulted in a p-value of 0 031, which shows that at a 95% significance level, TF-MSM has produced better results 
Discussion
Given the observation of the eigenvalues distribution of word vectors, we could see that word vectors that belong to the same context, i e , same class, are suitable for subspace representation  Our analysis showed that half of the word vector space dimensions suffice to represent most of the variability of the data in each class of the Reuters-8 database 
The results from the text classification experiment showed that subspace-based methods performed better than the text classification methods discussed in this work  Ultimately, our proposed TF weighted word subspace with MSM surpassed all the other methods  word2vec features are reliable tools to represent the semantic meaning of the words and when treated as sets of word vectors, they are capable of representing the content of texts  However, despite the fact that word vectors can be treated separately, conventional methods such as SVM and LSA may not be suitable for text classification using word vectors 
Among the conventional methods, LSA and SVM achieved about 86% and 89%, respectively, when using bag-of-words features  Interestingly, both methods had better performance when using binary weights  For LSA, we can see that despite the slight differences in the performance, tfidfBOW required approximations with smaller dimensions  SVM had the lowest accuracy rate when using the tfidfBOW features  One possible explanation for this is that TF-IDF weights are useful when rare words and very frequent words exist in the corpus, giving higher weights for rare words and lower weights for common words  Since we removed the stop words, the most frequent words among the training documents were not considered and, therefore, using TF-IDF weights did not improve the results 
Only MNB with tfBOW performed better than MSM  This result may be because tfBOW features encode the word frequencies, while MSM only considers a single occurrence of words  When incorporating the word frequencies with our TF weighted word subspace, we achieved a higher accuracy of 92 01%, performing better than MNB at a significance level of 95% 
Conclusions and Future Work
In this paper, we proposed a new method for text classification, based on the novel concept of word subspace under the MSM framework  We also proposed the term-frequency weighted word subspace which can incorporate the frequency of words directly in the modeling of the subspace by using a weighted version of PCA 
Most of the conventional text classification methods are based on the bag-of-words features, which are very simple to compute and had been proved to produce positive results  However, bag-of-words are commonly high dimensional models, with a sparse representation, which is computationally heavy to model  Also, bag-of-words fail to convey the semantic meaning of words inside a text  Due to these problems, neural networks started to be applied to generate a vector representation of words  Despite the fact that these representations can encode the semantic meaning of words, conventional methods do not work well when considering word vectors separately 
In our work, we focused on the word2vec representation, which can embed the semantic structure of words, rendering vector angles as a useful metric to show meaningful similarities between words  Our experiments showed that our word subspace modeling along with the MSM outperforms most of the conventional methods  Ultimately, our TF weighted subspace formulation resulted in significantly higher accuracy when compared to all conventional text classification methods discussed in this work  It is important to note that our method does not consider the order of the words in a text, resulting in a loss of context information  As a future work, we wish to extend our word subspace concept further in mainly two directions  First, we seek to encode word order, which may enrich the representation of context information  Second, we wish to model dynamic context change, enabling analysis of large documents, by having a long-short memory to interpret information using cues from different parts of a text 
Acknowledgment
This work is supported by JSPS KAKENHI Grant Number JP16H02842 and the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship 
Context 2: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2,    p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 

ground_truth: Word vectors, usually in the context of others within the same class

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 3: Introduction
The task of speculation detection and scope resolution is critical in distinguishing factual information from speculative information  This has multiple use-cases, like systems that determine the veracity of information, and those that involve requirement analysis  This task is particularly important to the biomedical domain, where patient reports and medical articles often use this feature of natural language  This task is commonly broken down into two subtasks: the first subtask, speculation cue detection, is to identify the uncertainty cue in a sentence, while the second subtask: scope resolution, is to identify the scope of that cue  For instance, consider the example:
It might rain tomorrow 
The speculation cue in the sentence above is ‘might’ and the scope of the cue ‘might’ is ‘rain tomorrow’  Thus, the speculation cue is the word that expresses the speculation, while the words affected by the speculation are in the scope of that cue 
This task was the CoNLL-2010 Shared Task (BIBREF0), which had 3 different subtasks  Task 1B was speculation cue detection on the BioScope Corpus, Task 1W was weasel identification from Wikipedia articles, and Task 2 was speculation scope resolution from the BioScope Corpus  For each task, the participants were provided the train and test set, which is henceforth referred to as Task 1B CoNLL and Task 2 CoNLL throughout this paper 
For our experimentation, we use the sub corpora of the BioScope Corpus (BIBREF1), namely the BioScope Abstracts sub corpora, which is referred to as BA, and the BioScope Full Papers sub corpora, which is referred to as BF  We also use the SFU Review Corpus (BIBREF2), which is referred to as SFU 
This subtask of natural language processing, along with another similar subtask, negation detection and scope resolution, have been the subject of a body of work over the years  The approaches used to solve them have evolved from simple rule-based systems (BIBREF3) based on linguistic information extracted from the sentences, to modern deep-learning based methods  The Machine Learning techniques used varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), while the deep learning approaches included Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)  Figures FIGREF1 and FIGREF1 contain a summary of the papers addressing speculation detection and scope resolution (BIBREF13, BIBREF5, BIBREF9, BIBREF3, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF6, BIBREF11, BIBREF18, BIBREF10, BIBREF19, BIBREF7, BIBREF4, BIBREF8) 
Inspired by the most recent approach of applying BERT to negation detection and scope resolution (BIBREF12), we take this approach one step further by performing a comparative analysis of three popular transformer-based architectures: BERT (BIBREF20), XLNet (BIBREF21) and RoBERTa (BIBREF22), applied to speculation detection and scope resolution  We evaluate the performance of each model across all datasets via the single dataset training approach, and report all scores including inter-dataset scores (i e  train on one dataset, evaluate on another) to test the generalizability of the models  This approach outperforms all existing systems on the task of speculation detection and scope resolution  Further, we jointly train on multiple datasets and obtain improvements over the single dataset training approach on most datasets 
Contrary to results observed on benchmark GLUE tasks, we observe XLNet consistently outperforming RoBERTa  To confirm this observation, we apply these models to the negation detection and scope resolution task, and observe a continuity in this trend, reporting state-of-the-art results on three of four datasets on the negation scope resolution task 
The rest of the paper is organized as follows: In Section 2, we provide a detailed description of our methodology and elaborate on the experimentation details  In Section 3, we present our results and analysis on the speculation detection and scope resolution task, using the single dataset and the multiple dataset training approach  In Section 4, we show the results of applying XLNet and RoBERTa on negation detection and scope resolution and propose a few reasons to explain why XLNet performs better than RoBERTa  Finally, the future scope and conclusion is mentioned in Section 5 
Methodology and Experimental Setup
We use the methodology by Khandelwal and Sawant (BIBREF12), and modify it to support experimentation with multiple models 
For Speculation Cue Detection:
Input Sentence: It might rain tomorrow 
True Labels: Not-A-Cue, Cue, Not-A-Cue, Not-A-Cue 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
1 – Normal Cue 2 – Multiword Cue 3 – Not a cue 4 – Pad token
Thus, the preprocessed sequence is as follows:
Input Sentence: [It, might, rain, tomorrow]
True Labels: [3,1,3,3]
Then, we preprocess the input using the tokenizer for the model being used (BERT, XLNet or RoBERTa): splitting each word into one or more tokens, and converting each token to it’s corresponding tokenID, and padding it to the maximum input length of the model  Thus,
Input Sentence: [wtt(It), wtt(might), wtt(rain), wtt(tom), wtt(## or), wtt(## row), wtt(〈pad 〉),wtt(〈pad 〉)   ]
True Labels: [3,1,3,3,3,3,4,4,4,4,   ]
The word ‘tomorrow' has been split into 3 tokens, ‘tom', ‘##or' and ‘##row'  The function to convert the word to tokenID is represented by wtt 
For Speculation Scope Resolution:
If a sentence has multiple cues, each cue's scope will be resolved individually 
Input Sentence: It might rain tomorrow 
True Labels: Out-Of-Scope, Out-Of-Scope, In-Scope, In-Scope 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
0 – Out-Of-Scope 1 – In-Scope
Thus, the preprocessed sequence is as follows:
True Scope Labels: [0,0,1,1]
As for cue detection, we preprocess the input using the tokenizer for the model being used  Additionally, we need to indicate which cue's scope we want to find in the input sentence  We do this by inserting a special token representing the token type (according to the cue detection annotation schema) before the cue word whose scope is being resolved  Here, we want to find the scope of the cue ‘might'  Thus,
Input Sentence: [wtt(It), wtt(〈token[1]〉), wtt(might), wtt(rain), wtt(tom), wtt(##or), wtt(##row), wtt(〈pad〉), wtt(〈pad〉)   ]
True Scope Labels: [0,0,1,1,1,1,0,0,0,0,   ]
Now, the preprocessed input for cue detection and similarly for scope detection is fed as input to our model as follows:
X = Model (Input)
Y = W*X + b
The W matrix is a matrix of size n_hidden x num_classes (n_hidden is the size of the representation of a token within the model)  These logits are fed to the loss function 
We use the following variants of each model:
BERT: bert-base-uncaseds3 amazonaws com/models huggingface co/bert/bert-base-uncased tar gz (The model used by BIBREF12)
RoBERTa: roberta-bases3 amazonaws com/models huggingface co/bert/roberta-base-pytorch_model bin (RoBERTa-base does not have an uncased variant)
XLNet: xlnet-base-caseds3 amazonaws com/models huggingface co/bert/xlnet-base-cased-pytorch_model bin (XLNet-base does not have an uncased variant)
The output of the model is a vector of probabilities per token  The loss is calculated for each token, by using the output vector and the true label for that token  We use class weights for the loss function, setting the weight for label 4 to 0 and all other labels to 1 (for cue detection only) to avoid training on padding token’s output 
We calculate the scores (Precision, Recall, F1) for the model per word of the input sentence, not per token that was fed to the model, as the tokens could be different for different models leading to inaccurate scores  For the above example, we calculate the output label for the word ‘tomorrow', not for each token it was split into (‘tom', ‘##or' and ‘##row')  To find the label for each word from the tokens it was split into, we experiment with 2 methods:
Average: We average the output vectors (softmax probabilities) for each token that the word was split into by the model's tokenizer  In the example above, we average the output of ‘tom', ‘##or' and ‘##row' to get the output for ‘tomorrow'  Then, we take an argmax over the resultant vector  This is then compared with the true label for the original word 
First Token: Here, we only consider the first token's probability vector (among all tokens the word was split into) as the output for that word, and get the label by an argmax over this vector  In the example above, we would consider the output vector corresponding to the token ‘tom' as the output for the word ‘tomorrow' 
For cue detection, the results are reported for the Average method only, while we report the scores for both Average and First Token for Scope Resolution 
For fair comparison, we use the same hyperparameters for the entire architecture for all 3 models  Only the tokenizer and the model are changed for each model  All other hyperparameters are kept same  We finetune the models for 60 epochs, using early stopping with a patience of 6 on the F1 score (word level) on the validation dataset  We use an initial learning rate of 3e-5, with a batch size of 8  We use the Categorical Cross Entropy loss function 
We use the Huggingface’s Pytorch Transformer library (BIBREF23) for the models and train all our models on Google Colaboratory 
Results: Speculation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset  For the speculation detection and scope resolution subtasks using single-dataset training, we report the results as an average of 5 runs of the model  For training the model on multiple datasets, we perform a 70-15-15 split of each training dataset, after which the train and validation part of the individual datasets are merged while the scores are reported for the test part of the individual datasets, which is not used for training or validation  We report the results as an average of 3 runs of the model  Figure FIGREF8 contains results for speculation cue detection and scope resolution when trained on a single dataset  All models perform the best when trained on the same dataset as they are evaluated on, except for BF, which gets the best results when trained on BA  This is because of the transfer learning capabilities of the models and the fact that BF is a smaller dataset than BA (BF: 2670 sentences, BA: 11871 sentences)  For speculation cue detection, there is lesser generalizability for models trained on BF or BA, while there is more generalizability for models trained on SFU  This could be because of the different nature of the biomedical domain 
Figure FIGREF11 contains the results for speculation detection and scope resolution for models trained jointly on multiple datasets  We observe that training on multiple datasets helps the performance of all models on each dataset, as the quantity of data available to train the model increases  We also observe that XLNet consistently outperforms BERT and RoBERTa  To confirm this observation, we apply the 2 models to the related task of negation detection and scope resolution
Negation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock)  The results for BERT are taken from BIBREF12  The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance  Figure FIGREF14 contains results for negation cue detection and scope resolution  We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets  Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks  A few possible reasons for this trend are:
Domain specificity, as both negation and speculation are closely related subtasks  Further experimentation on different tasks is needed to verify this 
Most benchmark tasks are sentence classification tasks, whereas the subtasks we experiment on are sequence labelling tasks  Given the pre-training objective of XLNet (training on permutations of the input), it may be able to capture long-term dependencies better, essential for sequence labelling tasks 
We work with the base variants of the models, while most results are reported with the large variants of the models 
Conclusion and Future Scope
In this paper, we expanded on the work of Khandelwal and Sawant (BIBREF12) by looking at alternative transfer-learning models and experimented with training on multiple datasets  On the speculation detection task, we obtained a gain of 0 42 F1 points on BF, 1 98 F1 points on BA and 0 29 F1 points on SFU, while on the scope resolution task, we obtained a gain of 8 06 F1 points on BF, 4 27 F1 points on BA and 11 87 F1 points on SFU, when trained on a single dataset  While training on multiple datasets, we observed a gain of 10 6 F1 points on BF and 1 94 F1 points on BA on the speculation detection task and 2 16 F1 points on BF and 0 25 F1 points on SFU on the scope resolution task over the single dataset training approach  We thus significantly advance the state-of-the-art for speculation detection and scope resolution  On the negation scope resolution task, we applied the XLNet and RoBERTa and obtained a gain of 3 16 F1 points on BF, 0 06 F1 points on BA and 0 3 F1 points on SFU  Thus, we demonstrated the usefulness of transformer-based architectures in the field of negation and speculation detection and scope resolution  We believe that a larger and more general dataset would go a long way in bolstering future research and would help create better systems that are not domain-specific 

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the baselines?
Context 4: 
Tasks ::: Sentence-Level Classification task (SLC) 
A sentence is considered propagandist if it contains at least one propagandist fragment  We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted 
Data
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators  More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles  We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators 
The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively  Figure FIGREF15 shows an annotated example, which contains several propaganda techniques  For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling  Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language 
Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets 
Setup
The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided 
The participants tried to achieve the best performance on the development set  A live leaderboard kept track of the submissions 
The test set was released and the participants had few days to make final predictions 
In phase 2, no immediate feedback on the submissions was provided  The winner was determined based on the performance on the test set 
Evaluation ::: FLC task 
FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task)  While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details  We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard) 
Evaluation ::: SLC task 
SLC is a binary classification task with imbalanced data  Therefore, the official evaluation measure for the task is the standard F$_1$ measure  We further report Precision and Recall 
Baselines
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence  The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34 
The baseline for the FLC task generates spans and selects one of the 18 techniques randomly  The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41 
Participants and Approaches
A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions  For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions 
Below, we give an overview of the approaches as described in the participants' papers  Tables TABREF28 and TABREF29 offer a high-level summary 
Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only
Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class  The team fed one sentence at a time in order to reduce the workload  In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia  Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters  Moreover, oversampling of the least represented classes proved to be crucial for the final performance  Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs 
Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28  The balancing of the embedding space was carried out by means of synthetic minority class over-sampling  Then, the learned representations were fed into an LSTM 
Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only
Team CAUnLP BIBREF29 used two context-aware representations based on BERT  In the first representation, the target sentence is followed by the title of the article  In the second representation, the previous sentence is also added  They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$
Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings  They also observed a boost in performance when balancing the dataset by dropping some negative examples 
Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features  These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT 
Team YMJA BIBREF33 also based their approach on fine-tuned BERT  Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation 
Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF IDF and BERT vectors, as well as vocabularies and readability measures 
Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36 
Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38 
Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e g , emotion classification 
Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus 
Participants and Approaches ::: Teams Participating in Both Tasks
Team MIC-CIS BIBREF42 participated in both tasks  For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models  Beside these representations, multiple features of readability, sentiment and emotions were considered  For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features  Finally, they applied sentence- and fragment-level models jointly 
Team CUNLP BIBREF45 considered two approaches for the sentence-level task  The first approach was based on fine-tuning BERT  The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features  Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings 
Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing  They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall)  In order to deal with the imbalance, they performed cost-sensitive classification, i e , the errors on the smaller positive class were more costly  For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM 
Evaluation Results
The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf  Section SECREF6)  The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set  Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system 
Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task  For this task, the results tend to be more stable across the two sets  Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda  Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set 
Conclusion and Further Work
We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification  We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively  Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline  The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline 
We plan to make the schema and the dataset publicly available to be used beyond NLP4IF  We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task  It features two complimentary tasks:
Given a free-text article, identify the propagandist text spans 
Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains 
This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2 
Acknowledgments
This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project  The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news ” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) 
The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations 

ground_truth: varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)