prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What accuracy does the proposed system achieve?
Context 1: Introduction
There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2   As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts  In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data  For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money 
In order to quickly obtain new training data, we can use crowdsourcing as one alternative way at lower cost in a short time  But as an exchange, crowd annotations from non-experts may be of lower quality than those from experts  It is one biggest challenge to build a powerful NER system on such a low quality annotated data  Although we can obtain high quality annotations for each input sentence by majority voting, it can be a waste of human labors to achieve such a goal, especially for some ambiguous sentences which may require a number of annotations to reach an agreement  Thus majority work directly build models on crowd annotations, trying to model the differences among annotators, for example, some of the annotators may be more trustful BIBREF3 , BIBREF4  
Here we focus mainly on the Chinese NER, which is more difficult than NER for other languages such as English for the lack of morphological variations such as capitalization and in particular the uncertainty in word segmentation  The Chinese NE taggers trained on news domain often perform poor in other domains  Although we can alleviate the problem by using character-level tagging to resolve the problem of poor word segmentation performances BIBREF5 , still there exists a large gap when the target domain changes, especially for the texts of social media  Thus, in order to get a good tagger for new domains and also for the conditions of new entity types, we require large amounts of labeled data  Therefore, crowdsourcing is a reasonable solution for these situations 
In this paper, we propose an approach to training a Chinese NER system on the crowd-annotated data  Our goal is to extract additional annotator independent features by adversarial training, alleviating the annotation noises of non-experts  The idea of adversarial training in neural networks has been used successfully in several NLP tasks, such as cross-lingual POS tagging BIBREF6 and cross-domain POS tagging BIBREF7   They use it to reduce the negative influences of the input divergences among different domains or languages, while we use adversarial training to reduce the negative influences brought by different crowd annotators  To our best knowledge, we are the first to apply adversarial training for crowd annotation learning 
In the learning framework, we perform adversarial training between the basic NER and an additional worker discriminator  We have a common Bi-LSTM for representing annotator-generic information and a private Bi-LSTM for representing annotator-specific information  We build another label Bi-LSTM by the crowd-annotated NE label sequence which reflects the mind of the crowd annotators who learn entity definitions by reading the annotation guidebook  The common and private Bi-LSTMs are used for NER, while the common and label Bi-LSTMs are used as inputs for the worker discriminator  The parameters of the common Bi-LSTM are learned by adversarial training, maximizing the worker discriminator loss and meanwhile minimizing the NER loss  Thus the resulting features of the common Bi-LSTM are worker invariant and NER sensitive 
For evaluation, we create two Chinese NER datasets in two domains: dialog and e-commerce  We require the crowd annotators to label the types of entities, including person, song, brand, product, and so on  Identifying these entities is useful for chatbot and e-commerce platforms BIBREF8   Then we conduct experiments on the newly created datasets to verify the effectiveness of the proposed adversarial neural network model  The results show that our system outperforms very strong baseline systems  In summary, we make the following contributions:
Related Work
Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing 
Sequence labeling  NER is widely treated as a sequence labeling problem, by assigning a unique label over each sentential word BIBREF9   Early studies on sequence labeling often use the models of HMM, MEMM, and CRF BIBREF10 based on manually-crafted discrete features, which can suffer the feature sparsity problem and require heavy feature engineering  Recently, neural network models have been successfully applied to sequence labeling BIBREF1 , BIBREF11 , BIBREF2   Among these work, the model which uses Bi-LSTM for feature extraction and CRF for decoding has achieved state-of-the-art performances BIBREF11 , BIBREF2 , which is exploited as the baseline model in our work 
Adversarial Training  Adversarial Networks have achieved great success in computer vision such as image generation BIBREF12 , BIBREF13   In the NLP community, the method is mainly exploited under the settings of domain adaption BIBREF14 , BIBREF7 , cross-lingual BIBREF15 , BIBREF6 and multi-task learning BIBREF16 , BIBREF17   All these settings involve the feature divergences between the training and test examples, and aim to learn invariant features across the divergences by an additional adversarial discriminator, such as domain discriminator  Our work is similar to these work but is applies on crowdsourcing learning, aiming to find invariant features among different crowdsourcing workers 
Crowdsourcing  Most NLP tasks require a massive amount of labeled training data which are annotated by experts  However, hiring experts is costly and non-scalable, both in terms of time and money  Instead, crowdsourcing is another solution to obtain labeled data at a lower cost but with relative lower quality than those from experts  BIBREF18 snow2008cheap collected labeled results for several NLP tasks from Amazon Mechanical Turk and demonstrated that non-experts annotations were quite useful for training new systems  In recent years, a series of work have focused on how to use crowdsourcing data efficiently in tasks such as classification BIBREF19 , BIBREF20 , and compare quality of crowd and expert labels BIBREF21  
In sequence labeling tasks, BIBREF22 dredze2009sequence viewed this task as a multi-label problem while BIBREF3 rodrigues2014sequence took workers identities into account by assuming that each sentential word was tagged correctly by one of the crowdsourcing workers and proposed a CRF-based model with multiple annotators  BIBREF4 nguyen2017aggregating introduced a crowd representation in which the crowd vectors were added into the LSTM-CRF model at train time, but ignored them at test time  In this paper, we apply adversarial training on crowd annotations on Chinese NER in new domains, and achieve better performances than previous studies on crowdsourcing learning 
Baseline: LSTM-CRF
We use a neural CRF model as the baseline system BIBREF9 , treating NER as a sequence labeling problem over Chinese characters, which has achieved state-of-the-art performances BIBREF5   To this end, we explore the BIEO schema to convert NER into sequence labeling, following BIBREF2 lample-EtAl:2016:N16-1, where sentential character is assigned with one unique tag  Concretely, we tag the non-entity character by label “O”, the beginning character of an entity by “B-XX”, the ending character of an entity by “E-XX” and the other character of an entity by “I-XX”, where “XX” denotes the entity type 
We build high-level neural features from the input character sequence by a bi-directional LSTM BIBREF2   The resulting features are combined and then are fed into an output CRF layer for decoding  In summary, the baseline model has three main components  First, we make vector representations for sentential characters $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , transforming the discrete inputs into low-dimensional neural inputs  Second, feature extraction is performed to obtain high-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , by using a bi-directional LSTM (Bi-LSTM) structure together with a linear transformation over $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$   Third, we apply a CRF tagging module over $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , obtaining the final output NE labels  The overall framework of the baseline model is shown by the right part of Figure 1  
Vector Representation of Characters
To represent Chinese characters, we simply exploit a neural embedding layer to map discrete characters into the low-dimensional vector representations  The goal is achieved by a looking-up table $\mathbf {E}^W$ , which is a model parameter and will be fine-tuned during training  The looking-up table can be initialized either by random or by using a pretrained embeddings from large scale raw corpus  For a given Chinese character sequence $c_1c_2\cdots c_n$ , we obtain the vector representation of each sentential character by: $ \mathbf {x}_t = \text{look-up}(c_t, \mathbf {E}^W), \text{~~~} t \in [1, n]$  
Feature Extraction
Based on the vector sequence $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , we extract higher-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by using a bidirectional LSTM module and a simple feed-forward neural layer, which are then used for CRF tagging at the next step 
LSTM is a type of recurrent neural network (RNN), which is designed for solving the exploding and diminishing gradients of basic RNNs BIBREF23   It has been widely used in a number of NLP tasks, including POS-tagging BIBREF11 , BIBREF24 , parsing BIBREF25 and machine translation BIBREF26 , because of its strong capabilities of modeling natural language sentences 
By traversing $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ by order and reversely, we obtain the output features $\mathbf {h}_1^{\text{private}}\mathbf {h}_2^{\text{private}}\cdots \mathbf {h}_n^{\text{private}}$ of the bi-LSTM, where $\mathbf {h}_t^{\text{private}} = \overrightarrow{\mathbf {h}}_t \oplus \overleftarrow{\mathbf {h}}_t $   Here we refer this Bi-LSTM as private in order to differentiate it with the common Bi-LSTM over the same character inputs which will be introduced in the next section 
Further we make an integration of the output vectors of bi-directional LSTM by a linear feed-forward neural layer, resulting in the features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by equation:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} \mathbf {h}_t^{\text{private}} + \mathbf {b},$$   (Eq  6)
where $\mathbf {W}$ and $\mathbf {b}$ are both model parameters 
CRF Tagging
Finally we feed the resulting features $\mathbf {h}_t^{\text{ner}}, t\in [1, n]$ into a CRF layer directly for NER decoding  CRF tagging is one globally normalized model, aiming to find the best output sequence considering the dependencies between successive labels  In the sequence labeling setting for NER, the output label of one position has a strong dependency on the label of the previous position  For example, the label before “I-XX” must be either “B-XX” or “I-XX”, where “XX” should be exactly the same 
CRF involves two parts for prediction  First we should compute the scores for each label based $\mathbf {h}_t^{\text{ner}}$ , resulting in $\mathbf {o}_t^{\text{ner}}$ , whose dimension is the number of output labels  The other part is a transition matrix $\mathbf {T}$ which defines the scores of two successive labels  $\mathbf {T}$ is also a model parameter  Based on $\mathbf {o}_t^{\text{ner}}$ and $\mathbf {T}$ , we use the Viterbi algorithm to find the best-scoring label sequence 
We can formalize the CRF tagging process as follows:
$$\begin{split} & \mathbf {o}_t^{\text{ner}} = \mathbf {W}^{\text{ner}} \mathbf {h}_t^{\text{ner}}, \text{~~~~} t \in [1,n] \\ & \text{score}(\mathbf {X}, \mathbf {y}) = \sum _{t = 1}^{n}(\mathbf {o}_{t,y_t} + T_{y_{t-1},y_t}) \\ & \mathbf {y}^{\text{ner}} = \mathop {arg~max}_{\mathbf {y}}\big (\text{score}(\mathbf {X}, \mathbf {y}))\big ), \\ \end{split}$$   (Eq  8)
where $\text{score}(\cdot )$ is the scoring function for a given output label sequence $\mathbf {y} = y_1y_2 \cdots y_n$ based on input $\mathbf {X}$ , $\mathbf {y}^{\text{ner}}$ is the resulting label sequence, $\mathbf {W}^{\text{ner}}$ is a model parameter 
Training
To train model parameters, we exploit a negative log-likelihood objective as the loss function  We apply softmax over all candidate output label sequences, thus the probability of the crowd-annotated label sequence is computed by:
$$p(\mathbf {\bar{y}}|\mathbf {X}) = \frac{\exp \big (\text{score}(\mathbf {X}, \mathbf {\bar{y}})\big )}{\sum _{\mathbf {y} \in \mathbf {Y}_{\mathbf {X}}} \exp \big (\text{score}(\mathbf {X}, \mathbf {y})\big )},$$   (Eq  10)
where $\mathbf {\bar{y}}$ is the crowd-annotated label sequences and $\mathbf {Y}_{\mathbf {X}}$ is all candidate label sequence of input $\mathbf {X}$  
Based on the above formula, the loss function of our baseline model is:
$$\text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) = -\log p(\mathbf {\bar{y}}|\mathbf {X}),$$   (Eq  11)
where $\Theta $ is the set of all model parameters  We use standard back-propagation method to minimize the loss function of the baseline CRF model 
Worker Adversarial
Adversarial learning has been an effective mechanism to resolve the problem of the input features between the training and test examples having large divergences BIBREF27 , BIBREF13   It has been successfully applied on domain adaption BIBREF7 , cross-lingual learning BIBREF15 and multi-task learning BIBREF17   All settings involve feature shifting between the training and testing 
In this paper, our setting is different  We are using the annotations from non-experts, which are noise and can influence the final performances if they are not properly processed  Directly learning based on the resulting corpus may adapt the neural feature extraction into the biased annotations  In this work, we assume that individual workers have their own guidelines in mind after short training  For example, a perfect worker can annotate highly consistently with an expert, while common crowdsourcing workers may be confused and have different understandings on certain contexts  Based on the assumption, we make an adaption for the original adversarial neural network to our setting 
Our adaption is very simple  Briefly speaking, the original adversarial learning adds an additional discriminator to classify the type of source inputs, for example, the domain category in the domain adaption setting, while we add a discriminator to classify the annotation workers  Solely the features from the input sentence is not enough for worker classification  The annotation result of the worker is also required  Thus the inputs of our discriminator are different  Here we exploit both the source sentences and the crowd-annotated NE labels as basic inputs for the worker discrimination 
In the following, we describe the proposed adversarial learning module, including both the submodels and the training method  As shown by the left part of Figure 1 , the submodel consists of four parts: (1) a common Bi-LSTM over input characters; (2) an additional Bi-LSTM to encode crowd-annotated NE label sequence; (3) a convolutional neural network (CNN) to extract features for worker discriminator; (4) output and prediction 
Common Bi-LSTM over Characters
To build the adversarial part, first we create a new bi-directional LSTM, named by the common Bi-LSTM:
$$\mathbf {h}_1^{\text{\tiny common}} \mathbf {h}_2^{\text{\tiny common}} \cdots \mathbf {h}_n^{\text{\tiny common}} = \text{Bi-LSTM}(\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n) $$   (Eq  13)
As shown in Figure 1 , this Bi-LSTM is constructed over the same input character representations of the private Bi-LSTM, in order to extract worker independent features 
The resulting features of the common Bi-LSTM are used for both NER and the worker discriminator, different with the features of private Bi-LSTM which are used for NER only  As shown in Figure 1 , we concatenate the outputs of the common and private Bi-LSTMs together, and then feed the results into the feed-forward combination layer of the NER part  Thus Formula 6 can be rewritten as:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} (\mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{private}}) + \mathbf {b},$$   (Eq  14)
where $\mathbf {W}$ is wider than the original combination because the newly-added $\mathbf {h}_t^{\text{common}}$  
Noticeably, although the resulting common features are used for the worker discriminator, they actually have no capability to distinguish the workers  Because this part is exploited to maximize the loss of the worker discriminator, it will be interpreted in the later training subsection  These features are invariant among different workers, thus they can have less noises for NER  This is the goal of adversarial learning, and we hope the NER being able to find useful features from these worker independent features 
Additional Bi-LSTM over Annotated NER Labels
In order to incorporate the annotated NE labels to predict the exact worker, we build another bi-directional LSTM (named by label Bi-LSTM) based on the crowd-annotated NE label sequence  This Bi-LSTM is used for worker discriminator only  During the decoding of the testing phase, we will never have this Bi-LSTM, because the worker discriminator is no longer required 
Assuming the crowd-annotated NE label sequence annotated by one worker is $\mathbf {\bar{y}} = \bar{y}_1\bar{y}_2 \cdots \bar{y}_n$ , we exploit a looking-up table $\mathbf {E}^{L}$ to obtain the corresponding sequence of their vector representations $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , similar to the method that maps characters into their neural representations  Concretely, for one NE label $\bar{y}_t$ ( $t \in [1, n]$ ), we obtain its neural vector by: $\mathbf {x^{\prime }}_t = \text{look-up}(\bar{y}_t, \mathbf {E}^L)$  
Next step we apply bi-directional LSTM over the sequence $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , which can be formalized as:
$$\mathbf {h}_1^{\text{label}} \mathbf {h}_2^{\text{label}} \cdots \mathbf {h}_n^{\text{label}} = \text{Bi-LSTM}(\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n) $$   (Eq  16)
The resulting feature sequence is concatenated with the outputs of the common Bi-LSTM, and further be used for worker classification 
CNN
Following, we add a convolutional neural network (CNN) module based on the concatenated outputs of the common Bi-LSTM and the label Bi-LSTM, to produce the final features for worker discriminator  A convolutional operator with window size 5 is used, and then max pooling strategy is applied over the convolution sequence to obtain the final fixed-dimensional feature vector  The whole process can be described by the following equations:
$$\begin{split} &\mathbf {h}_t^{\text{worker}} = \mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{label}} \\ &\mathbf {\tilde{h}}_t^{\text{worker}} = \tanh (\mathbf {W}^{\text{cnn}}[\mathbf {h}_{t-2}^{\text{worker}}, \mathbf {h}_{t-1}^{\text{worker}}, \cdots , \mathbf {h}_{t+2}^{\text{worker}}]) \\ &\mathbf {h}^{\text{worker}} = \text{max-pooling}(\mathbf {\tilde{h}}_1^{\text{worker}}\mathbf {\tilde{h}}_2^{\text{worker}} \cdots \mathbf {\tilde{h}}_n^{\text{worker}}) \\ \end{split}$$   (Eq  18)
where $t \in [1,n]$ and $\mathbf {W}^{\text{cnn}}$ is one model parameter  We exploit zero vector to paddle the out-of-index vectors 
Output and Prediction
After obtaining the final feature vector for the worker discriminator, we use it to compute the output vector, which scores all the annotation workers  The score function is defined by:
$$\mathbf {o}^{\text{worker}} = \mathbf {W}^{\text{worker}} \mathbf {h}^{\text{worker}},$$   (Eq  20)
where $\mathbf {W}^{\text{worker}}$ is one model parameter and the output dimension equals the number of total non-expert annotators  The prediction is to find the worker which is responsible for this annotation 
Adversarial Training
The training objective with adversarial neural network is different from the baseline model, as it includes the extra worker discriminator  Thus the new objective includes two parts, one being the negative log-likelihood from NER which is the same as the baseline, and the other being the negative the log-likelihood from the worker discriminator 
In order to obtain the negative log-likelihood of the worker discriminator, we use softmax to compute the probability of the actual worker $\bar{z}$ as well, which is defined by:
$$p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}) = \frac{\exp (\mathbf {o}^{\text{worker}}_{\bar{z}})}{\sum _{z} \exp (\mathbf {o}^{\text{worker}}_z)},$$   (Eq  22)
where $z$ should enumerate all workers 
Based on the above definition of probability, our new objective is defined as follows:
$$\begin{split} \text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) &= \text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) - \text{loss}(\Theta , \Theta ^{\prime }, \mathbf {X}) \\ \text{~~~~~~} &= -\log p(\mathbf {\bar{y}}|\mathbf {X}) + \log p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}), \end{split}$$   (Eq  23)
where $\Theta $ is the set of all model parameters related to NER, and $\Theta ^{\prime }$ is the set of the remaining parameters which are only related to the worker discriminator, $\mathbf {X}$ , $\mathbf {\bar{y}}$ and $\bar{z}$ are the input sentence, the crowd-annotated NE labels and the corresponding annotator for this annotation, respectively  It is worth noting that the parameters of the common Bi-LSTM are included in the set of $\Theta $ by definition 
In particular, our goal is not to simply minimize the new objective  Actually, we aim for a saddle point, finding the parameters $\Theta $ and $\Theta ^{\prime }$ satisfying the following conditions:
$$\begin{split} \hat{\Theta } &= \mathop {arg~min}_{\Theta }\text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \hat{\Theta }^{\prime } &= \mathop {arg~max}_{\Theta ^{\prime }}\text{R}(\hat{\Theta }, \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \end{split}$$   (Eq  24)
where the first equation aims to find one $\Theta $ that minimizes our new objective $\text{R}(\cdot )$ , and the second equation aims to find one $\Theta ^{\prime }$ maximizing the same objective 
Intuitively, the first equation of Formula 24 tries to minimize the NER loss, but at the same time maximize the worker discriminator loss by the shared parameters of the common Bi-LSTM  Thus the resulting features of common Bi-LSTM actually attempt to hurt the worker discriminator, which makes these features worker independent since they are unable to distinguish different workers  The second equation tries to minimize the worker discriminator loss by its own parameter $\Theta ^{\prime }$  
We use the standard back-propagation method to train the model parameters, the same as the baseline model  In order to incorporate the term of the argmax part of Formula 24 , we follow the previous work of adversarial training BIBREF13 , BIBREF15 , BIBREF17 , by introducing a gradient reverse layer between the common Bi-LSTM and the CNN module, whose forward does nothing but the backward simply negates the gradients 
Data Sets
With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain  We hire undergraduate students to annotate the sentences  They are required to identify the predefined types of entities in the sentences  Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences 
Labeled Data: DL-PS  In Dialog domain (DL), we collect raw sentences from a chatbot application  And then we randomly select 20K sentences as our pool and hire 43 students to annotate the sentences  We ask the annotators to label two types of entities: Person-Name and Song-Name  The annotators label the sentences independently  In particular, each sentence is assigned to three annotators for this data  Although the setting can be wasteful of labor, we can use the resulting dataset to test several well-known baselines such as majority voting 
After annotation, we remove some illegal sentences reported by the annotators  Finally, we have 16,948 sentences annotated by the students  Table 1 shows the information of annotated data  The average Kappa value among the annotators is 0 6033, indicating that the crowd annotators have moderate agreement on identifying entities on this data 
In order to evaluate the system performances, we create a set of corpus with gold annotations  Concretely, we randomly select 1,000 sentences from the final dataset and let two experts generate the gold annotations  Among them, we use 300 sentences as the development set and the remaining 700 as the test set  The rest sentences with only student annotations are used as the training set 
Labeled data: EC-MT and EC-UQ  In E-commerce domain (EC), we collect raw sentences from two types of texts: one is titles of merchandise entries (EC-MT) and another is user queries (EC-UQ)  The annotators label five types of entities: Brand, Product, Model, Material, and Specification  These five types of entities are very important for E-commerce platform, for example building knowledge graph of merchandises  Five students participate the annotations for this domain since the number of sentences is small  We use the similar strategy as DL-PS to annotate the sentences, except that only two annotators are assigned for each sentence, because we aim to test the system performances under very small duplicated annotations 
Finally, we obtain 2,337 sentences for EC-MT and 2,300 for EC-UQ  Table 1 shows the information of annotated results  Similarly, we produce the development and test datasets for system evaluation, by randomly selecting 400 sentences and letting two experts to generate the groundtruth annotations  Among them, we use 100 sentences as the development set and the remaining 300 as the test set  The rest sentences with only crowdsourcing annotations are used as the training set 
Unlabeled data  The vector representations of characters are basic inputs of our baseline and proposed models, which are obtained by the looking-up table $\mathbf {E}^W$   As introduced before, we can use pretrained embeddings from large-scale raw corpus to initialize the table  In order to pretrain the character embeddings, we use one large-scale unlabeled data from the user-generated content in Internet  Totally, we obtain a number of 5M sentences  Finally, we use the tool word2vec to pretrain the character embeddings based on the unlabeled dataset in our experiments 
Settings
For evaluation, we use the entity-level metrics of Precision (P), Recall (R), and their F1 value in our experiments, treating one tagged entity as correct only when it matches the gold entity exactly 
There are several hyper-parameters in the baseline LSTM-CRF and our final models  We set them empirically by the development performances  Concretely, we set the dimension size of the character embeddings by 100, the dimension size of the NE label embeddings by 50, and the dimension sizes of all the other hidden features by 200 
We exploit online training with a mini-batch size 128 to learn model parameters  The max-epoch iteration is set by 200, and the best-epoch model is chosen according to the development performances  We use RMSprop BIBREF28 with a learning rate $10^{-3}$ to update model parameters, and use $l_2$ -regularization by a parameter $10^{-5}$   We adopt the dropout technique to avoid overfitting by a drop value of $0 2$  
Comparison Systems
The proposed approach (henceforward referred to as “ALCrowd”) is compared with the following systems:
CRF: We use the Crfsuite tool to train a model on the crowdsourcing labeled data  As for the feature settings, we use the supervised version of BIBREF0 zhao2008unsupervised 
CRF-VT: We use the same settings of the CRF system, except that the training data is the voted version, whose groundtruths are produced by majority voting at the character level for each annotated sentence 
CRF-MA: The CRF model proposed by BIBREF3 rodrigues2014sequence, which uses a prior distributation to model multiple crowdsourcing annotators  We use the source code provided by the authors 
LSTM-CRF: Our baseline system trained on the crowdsourcing labeled data 
LSTM-CRF-VT: Our baseline system trained on the voted corpus, which is the same as CRF-VT 
LSTM-Crowd: The LSTM-CRF model with crowd annotation learning proposed by BIBREF4 nguyen2017aggregating  We use the source code provided by the authors 
The first three systems are based on the CRF model using traditional handcrafted features, and the last three systems are based on the neural LSTM-CRF model  Among them, CRF-MA, LSTM-Crowd and our system with adversarial learning (ALCrowd) are based on crowd annotation learning that directly trains the model on the crowd-annotations  Five systems, including CRF, CRF-MA, LSTM-CRF, LSTM-Crowd, and ALCrowd, are trained on the original version of labeled data, while CRF-VT and LSTM-CRF-VT are trained on the voted version  Since CRF-VT, CRF-MA and LSTM-CRF-VT all require ground-truth answers for each training sentence, which are difficult to be produced with only two annotations, we do not apply the three models on the two EC datasets 
Main Results
In this section, we show the model performances of our proposed crowdsourcing learning system (ALCrowd), and meanwhile compare it with the other systems mentioned above  Table 2 shows the experimental results on the DL-PS datasets and Table 3 shows the experiment results on the EC-MT and EC-UQ datasets, respectively 
The results of CRF and LSTM-CRF mean that the crowd annotation is an alternative solution with low cost for labeling data that could be used for training a NER system even there are some inconsistencies  Compared with CRF, LSTM-CRF achieves much better performances on all the three data, showing +6 12 F1 improvement on DL-PS, +4 51 on EC-MT, and +9 19 on EC-UQ  This indicates that LSTM-CRF is a very strong baseline system, demonstrating the effectiveness of neural network 
Interestingly, when compared with CRF and LSTM-CRF, CRF-VT and LSTM-CRF-VT trained on the voted version perform worse in the DL-PS dataset  This trend is also mentioned in BIBREF4 nguyen2017aggregating  This fact shows that the majority voting method might be unsuitable for our task  There are two possible reasons accounting for the observation  On the one hand, simple character-level voting based on three annotations for each sentence may be still not enough  In the DL-PS dataset, even with only two predefined entity types, one character can have nine NE labels  Thus the majority-voting may be incapable of handling some cases  While the cost by adding more annotations for each sentence would be greatly increased  On the other hand, the lost information produced by majority-voting may be important, at least the ambiguous annotations denote that the input sentence is difficult for NER  The normal CRF and LSTM-CRF models without discard any annotations can differentiate these difficult contexts through learning 
Three crowd-annotation learning systems provide better performances than their counterpart systems, (CRF-MA VS CRF) and (LSTM-Crowd/ALCrowd VS LSTM-CRF)  Compared with the strong baseline LSTM-CRF, ALCrowd shows its advantage with +1 08 F1 improvements on DL-PS, +1 24 on EC-MT, and +2 38 on EC-UQ, respectively  This indicates that adding the crowd-annotation learning is quite useful for building NER systems  In addition, ALCrowd also outperforms LSTM-Crowd on all the datasets consistently, demonstrating the high effectiveness of ALCrowd in extracting worker independent features  Among all the systems, ALCrowd performs the best, and significantly better than all the other models (the p-value is below $10^{-5}$ by using t-test)  The results indicate that with the help of adversarial training, our system can learn a better feature representation from crowd annotation 
Discussion
Impact of Character Embeddings  First, we investigate the effect of the pretrained character embeddings in our proposed crowdsourcing learning model  The comparison results are shown in Figure 2 , where Random refers to the random initialized character embeddings, and Pretrained refers to the embeddings pretrained on the unlabeled data  According to the results, we find that our model with the pretrained embeddings significantly outperforms that using the random embeddings, demonstrating that the pretrained embeddings successfully provide useful information 
Case Studies  Second, we present several case studies in order to study the differences between our baseline and the worker adversarial models  We conduct a closed test on the training set, the results of which can be regarded as modifications of the training corpus, since there exist inconsistent annotations for each training sentence among the different workers  Figure 3 shows the two examples from the DL-PS dataset, which compares the outputs of the baseline and our final models, as well as the majority-voting strategy 
In the first case, none of the annotations get the correct NER result, but our proposed model can capture it  The result of LSTM-CRF is the same as majority-voting  In the second example, the output of majority-voting is the worst, which can account for the reason why the same model trained on the voted corpus performs so badly, as shown in Table 2   The model of LSTM-CRF fails to recognize the named entity “Xiexie” because of not trusting the second annotation, treating it as one noise annotation  Our proposed model is able to recognize it, because of its ability of extracting worker independent features 
Conclusions
In this paper, we presented an approach to performing crowd annotation learning based on the idea of adversarial training for Chinese Named Entity Recognition (NER)  In our approach, we use a common and private Bi-LSTMs for representing annotator-generic and -specific information, and learn a label Bi-LSTM from the crowd-annotated NE label sequences  Finally, the proposed approach adopts a LSTM-CRF model to perform tagging  In our experiments, we create two data sets for Chinese NER tasks in the dialog and e-commerce domains  The experimental results show that the proposed approach outperforms strong baseline systems 
 Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No  61572338, 61525205, and 61602160)  This work is also partially supported by the joint research project of Alibaba and Soochow University  Wenliang is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization 
Context 2: 10pt
1 10pt
[ Characterizing Political Fake News in Twitter by its Meta-DataJulio Amador Díaz LópezAxel Oehmichen Miguel Molina-Solana( j amador, axelfrancois oehmichen11, mmolinas@imperial ac uk ) Imperial College London This article presents a preliminary approach towards characterizing political fake news on Twitter through the analysis of their meta-data  In particular, we focus on more than 1 5M tweets collected on the day of the election of Donald Trump as 45th president of the United States of America  We use the meta-data embedded within those tweets in order to look for differences between tweets containing fake news and tweets not containing them  Specifically, we perform our analysis only on tweets that went viral, by studying proxies for users' exposure to the tweets, by characterizing accounts spreading fake news, and by looking at their polarization  We found significant differences on the distribution of followers, the number of URLs on tweets, and the verification of the users 
]
Introduction
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e g  it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0   Since then, governments and corporations alike (e g  Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3   Yet, the ability to define, identify and stop fake news from spreading is limited 
Since the Obama campaign in 2008, social media has been pervasive in the political arena in the United States  Studies report that up to 62% of American adults receive their news from social media BIBREF4   The wide use of platforms such as Twitter and Facebook has facilitated the diffusion of fake news by simplifying the process of receiving content with no significant third party filtering, fact-checking or editorial judgement  Such characteristics make these platforms suitable means for sharing news that, disguised as legit ones, try to confuse readers 
Such use and their prominent rise has been confirmed by Craig Silverman, a Canadian journalist who is a prominent figure on fake news BIBREF5 : “In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlet” 
Our current research hence departs from the assumption that social media is a conduit for fake news and asks the question of whether fake news (as spam was some years ago) can be identified, modelled and eventually blocked  In order to do so, we use a sample of more that 1 5M tweets collected on November 8th 2016 —election day in the United States— with the goal of identifying features that tweets containing fake news are likely to have  As such, our paper aims to provide a preliminary characterization of fake news in Twitter by looking into meta-data embedded in tweets  Considering meta-data as a relevant factor of analysis is in line with findings reported by Morris et al  BIBREF6   We argue that understanding differences between tweets containing fake news and regular tweets will allow researchers to design mechanisms to block fake news in Twitter 
Specifically, our goals are: 1) compare the characteristics of tweets labelled as containing fake news to tweets labelled as not containing them, 2) characterize, through their meta-data, viral tweets containing fake news and the accounts from which they originated, and 3) determine the extent to which tweets containing fake news expressed polarized political views 
For our study, we used the number of retweets to single-out those that went viral within our sample  Tweets within that subset (viral tweets hereafter) are varied and relate to different topics  We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al  BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious  The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties 
From our results, the following main observations can be made:
Our findings resonate with similar work done on fake news such as the one from Allcot and Gentzkow BIBREF9   Therefore, even if our study is a preliminary attempt at characterizing fake news on Twitter using only their meta-data, our results provide external validity to previous research  Moreover, our work not only stresses the importance of using meta-data, but also underscores which parameters may be useful to identify fake news on Twitter 
The rest of the paper is organized as follows  The next section briefly discusses where this work is located within the literature on fake news and contextualizes the type of fake news we are studying  Then, we present our hypotheses, the data, and the methodology we follow  Finally, we present our findings, conclusions of this study, and future lines of work 
Defining Fake news
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news  In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news  On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news  On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news 
With respect to social sciences, efforts from psychology, political science and sociology, have been dedicated to understand why people consume and/or believe misinformation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14   Most of these studies consistently reported that psychological biases such as priming effects and confirmation bias play an important role in people ability to discern misinformation 
In relation to the production and distribution of fake news, a recent paper in the field of Economics BIBREF9 found that most fake news sites use names that resemble those of legitimate organizations, and that sites supplying fake news tend to be short-lived  These authors also noticed that fake news items are more likely shared than legitimate articles coming from trusted sources, and they tend to exhibit a larger level of polarization 
The conceptual issue of how to define fake news is a serious and unresolved issue  As the focus of our work is not attempting to offer light on this, we will rely on work by other authors to describe what we consider as fake news  In particular, we use the categorization provided by Rubin et al  BIBREF7   The five categories they described, together with illustrative examples from our dataset, are as follows:
Research Hypotheses
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news  Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Taking those three dimensions into account, we propose the following hypotheses about the features that we believe can help to identify tweets containing fake news from those not containing them  They will be later tested over our collected dataset 
Exposure 
Characterization 
Polarization 
Data and Methodology
For this study, we collected publicly available tweets using Twitter's public API  Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data  Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets  Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016  Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton  The data collection ran for just one day (Nov 8th 2016) 
One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers  Among the reasons for retweeting, Body et al  BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others  As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news)  For our study, we consider that a tweet went viral if it was retweeted more than 1000 times 
Once we have the dataset of viral tweets, we eliminated duplicates (some of the tweets were collected several times because they had several handles) and an expert manually inspected the text field within the tweets to label them as containing fake news, or not containing them (according to the characterization presented before)  This annotated dataset BIBREF8 is publicly available and can be freely reused 
Finally, we use the following fields within tweets (from the ones returned by Twitter's API) to compare their distributions and look for differences between viral tweets containing fake news and viral tweets not containing fake news:
In the following section, we provide graphical descriptions of the distribution of each of the identified attributes for the two sets of tweets (those labelled as containing fake news and those labelled as not containing them)  Where appropriate, we normalized and/or took logarithms of the data for better representation

ground_truth: F1 scores of 85.99 on the DL-PS data,  75.15 on the EC-MT data and 71.53 on the EC-UQ data 

F1 of 85.99 on the DL-PS dataset (dialog domain); 75.15 on EC-MT and 71.53 on EC-UQ (e-commerce domain)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the ground truth for fake news established?
Context 2:  To gain a better understanding of the significance of those differences, we use the Kolmogorov-Smirnov test with the null hypothesis that both distributions are equal 
Results
The sample collected consisted on 1 785 855 tweets published by 848 196 different users  Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users  Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window 
The 1327 `viral' tweets were manually annotated as containing fake news or not  The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset  Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'  Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization  Because of this, we do not claim that this dataset can be considered a ground truth 
The following results detail characteristics of these tweets along the previously mentioned dimensions  Table TABREF23 reports the actual differences (together with their associated p-values) of the distributions of viral tweets containing fake news and viral tweets not containing them for every variable considered 
Exposure
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently  As such, Twitter users were exposed to fake news related to the election for a shorter period of time 
However, in terms of retweets, Figure FIGREF25 shows no apparent difference between containing fake news or not containing them  That is confirmed by the Kolmogorov-Smirnoff test, which does not discard the hypothesis that the associated distributions are equal 
In relation to the number of favourites, users that generated at least a viral tweet containing fake news appear to have, on average, less favourites than users that do not generate them  Figure FIGREF26 shows the distribution of favourites  Despite the apparent visual differences, the difference are not statistically significant 
Finally, the number of hashtags used in viral fake news appears to be larger than those in other viral tweets  Figure FIGREF27 shows the density distribution of the number of hashtags used  However, once again, we were not able to find any statistical difference between the average number of hashtags in a viral tweet and the average number of hashtags in viral fake news 
Characterization
We found that 82 users within our sample were spreading fake news (i e  they produced at least one tweet which was labelled as fake news)  Out of those, 34 had verified accounts, and the rest were unverified  From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted  Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs  not containing fake news)  From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts 
Turning to friends, accounts distributing fake news appear to have, on average, the same number of friends than those distributing tweets with no fake news  However, the density distribution of friends from the accounts (Figure FIGREF29 ) shows that there is indeed a statistically significant difference in their distributions 
If we take into consideration the number of followers, accounts generating viral tweets with fake news do have a very different distribution on this dimension, compared to those accounts generating viral tweets with no fake news (see Figure FIGREF30 )  In fact, such differences are statistically significant 
A useful representation for friends and followers is the ratio between friends/followers  Figures FIGREF31 and FIGREF32 show this representation  Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers  The distribution of those accounts not generating fake news is more evenly distributed 
With respect to the number of mentions, Figure FIGREF33 shows that viral tweets labelled as containing fake news appear to use mentions to other users less frequently than viral tweets not containing fake news  In other words, tweets containing fake news mostly contain 1 mention, whereas other tweets tend to have two)  Such differences are statistically significant 
The analysis (Figure FIGREF34 ) of the presence of media in the tweets in our dataset shows that tweets labelled as not containing fake news appear to present more media elements than those labelled as fake news  However, the difference is not statistically significant 
On the other hand, Figure FIGREF35 shows that viral tweets containing fake news appear to include more URLs to other sites than viral tweets that do not contain fake news  In fact, the difference between the two distributions is statistically significant (assuming INLINEFORM0 ) 
Polarization
Finally, manual inspection of the text field of those viral tweets labelled as containing fake news shows that 117 of such tweets expressed support for Donald Trump, while only 8 supported Hillary Clinton  The remaining tweets contained fake news related to other topics, not expressing support for any of the candidates 
Discussion
As a summary, and constrained by our existing dataset, we made the following observations regarding differences between viral tweets labelled as containing fake news and viral tweets labelled as not containing them:
These findings (related to our initial hypothesis in Table TABREF44 ) clearly suggest that there are specific pieces of meta-data about tweets that may allow the identification of fake news  One such parameter is the time of exposure  Viral tweets containing fake news are shorter-lived than those containing other type of content  This notion seems to resonate with our findings showing that a number of accounts spreading fake news have already been deleted or suspended by Twitter by the time of writing  If one considers that researchers using different data have found similar results BIBREF9 , it appears that the lifetime of accounts, together with the age of the questioned viral content could be useful to identify fake news  In the light of this finding, accounts newly created should probably put under higher scrutiny than older ones  This in fact, would be a nice a-priori bias for a Bayesian classifier 
Accounts spreading fake news appear to have a larger proportion of friends/followers (i e  they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only  Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter  Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising) 
Finally, it is interesting to notice that the content of viral fake news was highly polarized  This finding is also in line with those of Alcott et al  BIBREF9   This feature suggests that textual sentiment analysis of the content of tweets (as most researchers do), together with the above mentioned parameters from meta-data, may prove useful for identifying fake news 
Conclusions
With the election of Donald Trump as President of the United States, the concept of fake news has become a broadly-known phenomenon that is getting tremendous attention from governments and media companies  We have presented a preliminary study on the meta-data of a publicly available dataset of tweets that became viral during the day of the 2016 US presidential election  Our aim is to advance the understanding of which features might be characteristic of viral tweets containing fake news in comparison with viral tweets without fake news 
We believe that the only way to automatically identify those deceitful tweets (i e  containing fake news) is by actually understanding and modelling them  Only then, the automation of the processes of tagging and blocking these tweets can be successfully performed  In the same way that spam was fought, we anticipate fake news will suffer a similar evolution, with social platforms implementing tools to deal with them  With most works so far focusing on the actual content of the tweets, ours is a novel attempt from a different, but also complementary, angle 
Within the used dataset, we found there are differences around exposure, characteristics of accounts spreading fake news and the tone of the content  Those findings suggest that it is indeed possible to model and automatically detect fake news  We plan to replicate and validate our experiments in an extended sample of tweets (until 4 months after the US election), and tests the predictive power of the features we found relevant within our sample 
Author Disclosure Statement
No competing financial interest exist 
Context 3: Introduction
Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP)  It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0  
Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP  Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis  Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc ) either explicitly or implicitly referred to in the text is considered  Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0  
Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled  The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2   In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates  In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments  For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4   In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues  The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5   Stance detection on a corpus of student essays is considered in BIBREF6   After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6   In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web 
Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed  The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0   In BIBREF9 , a stance-community detection approach called SCIFNET is proposed  SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks  A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing  The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10   Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described  Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0  
In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available  The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included  We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features 
To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets  Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content 
The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information  Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions  Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary 
A Stance Detection Data Set
We have decided to consider tweets about popular sports clubs as our domain for stance detection  Considerable amounts of tweets are being published for sports-related events at every instant  Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey  As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs 
In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11   We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs  Then, we have annotated the stance information in the tweets for these targets as Favor or Against  Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals 
For the purposes of the current study, we have not annotated any tweets with the Neither class  This stance class and even finer-grained classes can be considered in further annotation studies  We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised  Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly 
At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2  Hence, our data set is a balanced one although it is currently limited in size  The corresponding stance annotations are made publicly available at http://ceng

ground_truth: Ground truth is not established in the paper

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which sports clubs are the targets?
Context 3: metu edu tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset csv in Comma Separated Values (CSV) format  The file contains three columns with the corresponding headers  The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against 
To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish  Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish  Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues 
Stance Detection Experiments Using SVM Classifiers
It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0   In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features  Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets  The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13  
We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel  The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure 
The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set  The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class  This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets  The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets  Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i e , better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0   Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes  Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set  On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class  Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems  Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature 
We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0   However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results  This observation may be due to the relatively limited size of the tweet data set employed  Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set  Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work 
With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams  The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2  
When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0 5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1 8%  Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system  We leave other ways of exploiting hashtags for stance detection as a future work 
To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets 
Future Prospects
Future work based on the current study includes the following:
Conclusion
Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis  It is the detection of stance within text towards a target which may be explicitly specified in the text or not  In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey  The corresponding annotations are made publicly-available for research purposes  To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set  Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature  10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems  Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets 

ground_truth: Galatasaray, Fenerbahçe

Galatasaray , Fenerbahçe