prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What accuracy does the proposed system achieve?
Context 1: Introduction
There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2   As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts  In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data  For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money 
In order to quickly obtain new training data, we can use crowdsourcing as one alternative way at lower cost in a short time  But as an exchange, crowd annotations from non-experts may be of lower quality than those from experts  It is one biggest challenge to build a powerful NER system on such a low quality annotated data  Although we can obtain high quality annotations for each input sentence by majority voting, it can be a waste of human labors to achieve such a goal, especially for some ambiguous sentences which may require a number of annotations to reach an agreement  Thus majority work directly build models on crowd annotations, trying to model the differences among annotators, for example, some of the annotators may be more trustful BIBREF3 , BIBREF4  
Here we focus mainly on the Chinese NER, which is more difficult than NER for other languages such as English for the lack of morphological variations such as capitalization and in particular the uncertainty in word segmentation  The Chinese NE taggers trained on news domain often perform poor in other domains  Although we can alleviate the problem by using character-level tagging to resolve the problem of poor word segmentation performances BIBREF5 , still there exists a large gap when the target domain changes, especially for the texts of social media  Thus, in order to get a good tagger for new domains and also for the conditions of new entity types, we require large amounts of labeled data  Therefore, crowdsourcing is a reasonable solution for these situations 
In this paper, we propose an approach to training a Chinese NER system on the crowd-annotated data  Our goal is to extract additional annotator independent features by adversarial training, alleviating the annotation noises of non-experts  The idea of adversarial training in neural networks has been used successfully in several NLP tasks, such as cross-lingual POS tagging BIBREF6 and cross-domain POS tagging BIBREF7   They use it to reduce the negative influences of the input divergences among different domains or languages, while we use adversarial training to reduce the negative influences brought by different crowd annotators  To our best knowledge, we are the first to apply adversarial training for crowd annotation learning 
In the learning framework, we perform adversarial training between the basic NER and an additional worker discriminator  We have a common Bi-LSTM for representing annotator-generic information and a private Bi-LSTM for representing annotator-specific information  We build another label Bi-LSTM by the crowd-annotated NE label sequence which reflects the mind of the crowd annotators who learn entity definitions by reading the annotation guidebook  The common and private Bi-LSTMs are used for NER, while the common and label Bi-LSTMs are used as inputs for the worker discriminator  The parameters of the common Bi-LSTM are learned by adversarial training, maximizing the worker discriminator loss and meanwhile minimizing the NER loss  Thus the resulting features of the common Bi-LSTM are worker invariant and NER sensitive 
For evaluation, we create two Chinese NER datasets in two domains: dialog and e-commerce  We require the crowd annotators to label the types of entities, including person, song, brand, product, and so on  Identifying these entities is useful for chatbot and e-commerce platforms BIBREF8   Then we conduct experiments on the newly created datasets to verify the effectiveness of the proposed adversarial neural network model  The results show that our system outperforms very strong baseline systems  In summary, we make the following contributions:
Related Work
Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing 
Sequence labeling  NER is widely treated as a sequence labeling problem, by assigning a unique label over each sentential word BIBREF9   Early studies on sequence labeling often use the models of HMM, MEMM, and CRF BIBREF10 based on manually-crafted discrete features, which can suffer the feature sparsity problem and require heavy feature engineering  Recently, neural network models have been successfully applied to sequence labeling BIBREF1 , BIBREF11 , BIBREF2   Among these work, the model which uses Bi-LSTM for feature extraction and CRF for decoding has achieved state-of-the-art performances BIBREF11 , BIBREF2 , which is exploited as the baseline model in our work 
Adversarial Training  Adversarial Networks have achieved great success in computer vision such as image generation BIBREF12 , BIBREF13   In the NLP community, the method is mainly exploited under the settings of domain adaption BIBREF14 , BIBREF7 , cross-lingual BIBREF15 , BIBREF6 and multi-task learning BIBREF16 , BIBREF17   All these settings involve the feature divergences between the training and test examples, and aim to learn invariant features across the divergences by an additional adversarial discriminator, such as domain discriminator  Our work is similar to these work but is applies on crowdsourcing learning, aiming to find invariant features among different crowdsourcing workers 
Crowdsourcing  Most NLP tasks require a massive amount of labeled training data which are annotated by experts  However, hiring experts is costly and non-scalable, both in terms of time and money  Instead, crowdsourcing is another solution to obtain labeled data at a lower cost but with relative lower quality than those from experts  BIBREF18 snow2008cheap collected labeled results for several NLP tasks from Amazon Mechanical Turk and demonstrated that non-experts annotations were quite useful for training new systems  In recent years, a series of work have focused on how to use crowdsourcing data efficiently in tasks such as classification BIBREF19 , BIBREF20 , and compare quality of crowd and expert labels BIBREF21  
In sequence labeling tasks, BIBREF22 dredze2009sequence viewed this task as a multi-label problem while BIBREF3 rodrigues2014sequence took workers identities into account by assuming that each sentential word was tagged correctly by one of the crowdsourcing workers and proposed a CRF-based model with multiple annotators  BIBREF4 nguyen2017aggregating introduced a crowd representation in which the crowd vectors were added into the LSTM-CRF model at train time, but ignored them at test time  In this paper, we apply adversarial training on crowd annotations on Chinese NER in new domains, and achieve better performances than previous studies on crowdsourcing learning 
Baseline: LSTM-CRF
We use a neural CRF model as the baseline system BIBREF9 , treating NER as a sequence labeling problem over Chinese characters, which has achieved state-of-the-art performances BIBREF5   To this end, we explore the BIEO schema to convert NER into sequence labeling, following BIBREF2 lample-EtAl:2016:N16-1, where sentential character is assigned with one unique tag  Concretely, we tag the non-entity character by label “O”, the beginning character of an entity by “B-XX”, the ending character of an entity by “E-XX” and the other character of an entity by “I-XX”, where “XX” denotes the entity type 
We build high-level neural features from the input character sequence by a bi-directional LSTM BIBREF2   The resulting features are combined and then are fed into an output CRF layer for decoding  In summary, the baseline model has three main components  First, we make vector representations for sentential characters $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , transforming the discrete inputs into low-dimensional neural inputs  Second, feature extraction is performed to obtain high-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , by using a bi-directional LSTM (Bi-LSTM) structure together with a linear transformation over $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$   Third, we apply a CRF tagging module over $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , obtaining the final output NE labels  The overall framework of the baseline model is shown by the right part of Figure 1  
Vector Representation of Characters
To represent Chinese characters, we simply exploit a neural embedding layer to map discrete characters into the low-dimensional vector representations  The goal is achieved by a looking-up table $\mathbf {E}^W$ , which is a model parameter and will be fine-tuned during training  The looking-up table can be initialized either by random or by using a pretrained embeddings from large scale raw corpus  For a given Chinese character sequence $c_1c_2\cdots c_n$ , we obtain the vector representation of each sentential character by: $ \mathbf {x}_t = \text{look-up}(c_t, \mathbf {E}^W), \text{~~~} t \in [1, n]$  
Feature Extraction
Based on the vector sequence $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , we extract higher-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by using a bidirectional LSTM module and a simple feed-forward neural layer, which are then used for CRF tagging at the next step 
LSTM is a type of recurrent neural network (RNN), which is designed for solving the exploding and diminishing gradients of basic RNNs BIBREF23   It has been widely used in a number of NLP tasks, including POS-tagging BIBREF11 , BIBREF24 , parsing BIBREF25 and machine translation BIBREF26 , because of its strong capabilities of modeling natural language sentences 
By traversing $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ by order and reversely, we obtain the output features $\mathbf {h}_1^{\text{private}}\mathbf {h}_2^{\text{private}}\cdots \mathbf {h}_n^{\text{private}}$ of the bi-LSTM, where $\mathbf {h}_t^{\text{private}} = \overrightarrow{\mathbf {h}}_t \oplus \overleftarrow{\mathbf {h}}_t $   Here we refer this Bi-LSTM as private in order to differentiate it with the common Bi-LSTM over the same character inputs which will be introduced in the next section 
Further we make an integration of the output vectors of bi-directional LSTM by a linear feed-forward neural layer, resulting in the features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by equation:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} \mathbf {h}_t^{\text{private}} + \mathbf {b},$$   (Eq  6)
where $\mathbf {W}$ and $\mathbf {b}$ are both model parameters 
CRF Tagging
Finally we feed the resulting features $\mathbf {h}_t^{\text{ner}}, t\in [1, n]$ into a CRF layer directly for NER decoding  CRF tagging is one globally normalized model, aiming to find the best output sequence considering the dependencies between successive labels  In the sequence labeling setting for NER, the output label of one position has a strong dependency on the label of the previous position  For example, the label before “I-XX” must be either “B-XX” or “I-XX”, where “XX” should be exactly the same 
CRF involves two parts for prediction  First we should compute the scores for each label based $\mathbf {h}_t^{\text{ner}}$ , resulting in $\mathbf {o}_t^{\text{ner}}$ , whose dimension is the number of output labels  The other part is a transition matrix $\mathbf {T}$ which defines the scores of two successive labels  $\mathbf {T}$ is also a model parameter  Based on $\mathbf {o}_t^{\text{ner}}$ and $\mathbf {T}$ , we use the Viterbi algorithm to find the best-scoring label sequence 
We can formalize the CRF tagging process as follows:
$$\begin{split} & \mathbf {o}_t^{\text{ner}} = \mathbf {W}^{\text{ner}} \mathbf {h}_t^{\text{ner}}, \text{~~~~} t \in [1,n] \\ & \text{score}(\mathbf {X}, \mathbf {y}) = \sum _{t = 1}^{n}(\mathbf {o}_{t,y_t} + T_{y_{t-1},y_t}) \\ & \mathbf {y}^{\text{ner}} = \mathop {arg~max}_{\mathbf {y}}\big (\text{score}(\mathbf {X}, \mathbf {y}))\big ), \\ \end{split}$$   (Eq  8)
where $\text{score}(\cdot )$ is the scoring function for a given output label sequence $\mathbf {y} = y_1y_2 \cdots y_n$ based on input $\mathbf {X}$ , $\mathbf {y}^{\text{ner}}$ is the resulting label sequence, $\mathbf {W}^{\text{ner}}$ is a model parameter 
Training
To train model parameters, we exploit a negative log-likelihood objective as the loss function  We apply softmax over all candidate output label sequences, thus the probability of the crowd-annotated label sequence is computed by:
$$p(\mathbf {\bar{y}}|\mathbf {X}) = \frac{\exp \big (\text{score}(\mathbf {X}, \mathbf {\bar{y}})\big )}{\sum _{\mathbf {y} \in \mathbf {Y}_{\mathbf {X}}} \exp \big (\text{score}(\mathbf {X}, \mathbf {y})\big )},$$   (Eq  10)
where $\mathbf {\bar{y}}$ is the crowd-annotated label sequences and $\mathbf {Y}_{\mathbf {X}}$ is all candidate label sequence of input $\mathbf {X}$  
Based on the above formula, the loss function of our baseline model is:
$$\text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) = -\log p(\mathbf {\bar{y}}|\mathbf {X}),$$   (Eq  11)
where $\Theta $ is the set of all model parameters  We use standard back-propagation method to minimize the loss function of the baseline CRF model 
Worker Adversarial
Adversarial learning has been an effective mechanism to resolve the problem of the input features between the training and test examples having large divergences BIBREF27 , BIBREF13   It has been successfully applied on domain adaption BIBREF7 , cross-lingual learning BIBREF15 and multi-task learning BIBREF17   All settings involve feature shifting between the training and testing 
In this paper, our setting is different  We are using the annotations from non-experts, which are noise and can influence the final performances if they are not properly processed  Directly learning based on the resulting corpus may adapt the neural feature extraction into the biased annotations  In this work, we assume that individual workers have their own guidelines in mind after short training  For example, a perfect worker can annotate highly consistently with an expert, while common crowdsourcing workers may be confused and have different understandings on certain contexts  Based on the assumption, we make an adaption for the original adversarial neural network to our setting 
Our adaption is very simple  Briefly speaking, the original adversarial learning adds an additional discriminator to classify the type of source inputs, for example, the domain category in the domain adaption setting, while we add a discriminator to classify the annotation workers  Solely the features from the input sentence is not enough for worker classification  The annotation result of the worker is also required  Thus the inputs of our discriminator are different  Here we exploit both the source sentences and the crowd-annotated NE labels as basic inputs for the worker discrimination 
In the following, we describe the proposed adversarial learning module, including both the submodels and the training method  As shown by the left part of Figure 1 , the submodel consists of four parts: (1) a common Bi-LSTM over input characters; (2) an additional Bi-LSTM to encode crowd-annotated NE label sequence; (3) a convolutional neural network (CNN) to extract features for worker discriminator; (4) output and prediction 
Common Bi-LSTM over Characters
To build the adversarial part, first we create a new bi-directional LSTM, named by the common Bi-LSTM:
$$\mathbf {h}_1^{\text{\tiny common}} \mathbf {h}_2^{\text{\tiny common}} \cdots \mathbf {h}_n^{\text{\tiny common}} = \text{Bi-LSTM}(\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n) $$   (Eq  13)
As shown in Figure 1 , this Bi-LSTM is constructed over the same input character representations of the private Bi-LSTM, in order to extract worker independent features 
The resulting features of the common Bi-LSTM are used for both NER and the worker discriminator, different with the features of private Bi-LSTM which are used for NER only  As shown in Figure 1 , we concatenate the outputs of the common and private Bi-LSTMs together, and then feed the results into the feed-forward combination layer of the NER part  Thus Formula 6 can be rewritten as:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} (\mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{private}}) + \mathbf {b},$$   (Eq  14)
where $\mathbf {W}$ is wider than the original combination because the newly-added $\mathbf {h}_t^{\text{common}}$  
Noticeably, although the resulting common features are used for the worker discriminator, they actually have no capability to distinguish the workers  Because this part is exploited to maximize the loss of the worker discriminator, it will be interpreted in the later training subsection  These features are invariant among different workers, thus they can have less noises for NER  This is the goal of adversarial learning, and we hope the NER being able to find useful features from these worker independent features 
Additional Bi-LSTM over Annotated NER Labels
In order to incorporate the annotated NE labels to predict the exact worker, we build another bi-directional LSTM (named by label Bi-LSTM) based on the crowd-annotated NE label sequence  This Bi-LSTM is used for worker discriminator only  During the decoding of the testing phase, we will never have this Bi-LSTM, because the worker discriminator is no longer required 
Assuming the crowd-annotated NE label sequence annotated by one worker is $\mathbf {\bar{y}} = \bar{y}_1\bar{y}_2 \cdots \bar{y}_n$ , we exploit a looking-up table $\mathbf {E}^{L}$ to obtain the corresponding sequence of their vector representations $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , similar to the method that maps characters into their neural representations  Concretely, for one NE label $\bar{y}_t$ ( $t \in [1, n]$ ), we obtain its neural vector by: $\mathbf {x^{\prime }}_t = \text{look-up}(\bar{y}_t, \mathbf {E}^L)$  
Next step we apply bi-directional LSTM over the sequence $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , which can be formalized as:
$$\mathbf {h}_1^{\text{label}} \mathbf {h}_2^{\text{label}} \cdots \mathbf {h}_n^{\text{label}} = \text{Bi-LSTM}(\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n) $$   (Eq  16)
The resulting feature sequence is concatenated with the outputs of the common Bi-LSTM, and further be used for worker classification 
CNN
Following, we add a convolutional neural network (CNN) module based on the concatenated outputs of the common Bi-LSTM and the label Bi-LSTM, to produce the final features for worker discriminator  A convolutional operator with window size 5 is used, and then max pooling strategy is applied over the convolution sequence to obtain the final fixed-dimensional feature vector  The whole process can be described by the following equations:
$$\begin{split} &\mathbf {h}_t^{\text{worker}} = \mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{label}} \\ &\mathbf {\tilde{h}}_t^{\text{worker}} = \tanh (\mathbf {W}^{\text{cnn}}[\mathbf {h}_{t-2}^{\text{worker}}, \mathbf {h}_{t-1}^{\text{worker}}, \cdots , \mathbf {h}_{t+2}^{\text{worker}}]) \\ &\mathbf {h}^{\text{worker}} = \text{max-pooling}(\mathbf {\tilde{h}}_1^{\text{worker}}\mathbf {\tilde{h}}_2^{\text{worker}} \cdots \mathbf {\tilde{h}}_n^{\text{worker}}) \\ \end{split}$$   (Eq  18)
where $t \in [1,n]$ and $\mathbf {W}^{\text{cnn}}$ is one model parameter  We exploit zero vector to paddle the out-of-index vectors 
Output and Prediction
After obtaining the final feature vector for the worker discriminator, we use it to compute the output vector, which scores all the annotation workers  The score function is defined by:
$$\mathbf {o}^{\text{worker}} = \mathbf {W}^{\text{worker}} \mathbf {h}^{\text{worker}},$$   (Eq  20)
where $\mathbf {W}^{\text{worker}}$ is one model parameter and the output dimension equals the number of total non-expert annotators  The prediction is to find the worker which is responsible for this annotation 
Adversarial Training
The training objective with adversarial neural network is different from the baseline model, as it includes the extra worker discriminator  Thus the new objective includes two parts, one being the negative log-likelihood from NER which is the same as the baseline, and the other being the negative the log-likelihood from the worker discriminator 
In order to obtain the negative log-likelihood of the worker discriminator, we use softmax to compute the probability of the actual worker $\bar{z}$ as well, which is defined by:
$$p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}) = \frac{\exp (\mathbf {o}^{\text{worker}}_{\bar{z}})}{\sum _{z} \exp (\mathbf {o}^{\text{worker}}_z)},$$   (Eq  22)
where $z$ should enumerate all workers 
Based on the above definition of probability, our new objective is defined as follows:
$$\begin{split} \text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) &= \text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) - \text{loss}(\Theta , \Theta ^{\prime }, \mathbf {X}) \\ \text{~~~~~~} &= -\log p(\mathbf {\bar{y}}|\mathbf {X}) + \log p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}), \end{split}$$   (Eq  23)
where $\Theta $ is the set of all model parameters related to NER, and $\Theta ^{\prime }$ is the set of the remaining parameters which are only related to the worker discriminator, $\mathbf {X}$ , $\mathbf {\bar{y}}$ and $\bar{z}$ are the input sentence, the crowd-annotated NE labels and the corresponding annotator for this annotation, respectively  It is worth noting that the parameters of the common Bi-LSTM are included in the set of $\Theta $ by definition 
In particular, our goal is not to simply minimize the new objective  Actually, we aim for a saddle point, finding the parameters $\Theta $ and $\Theta ^{\prime }$ satisfying the following conditions:
$$\begin{split} \hat{\Theta } &= \mathop {arg~min}_{\Theta }\text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \hat{\Theta }^{\prime } &= \mathop {arg~max}_{\Theta ^{\prime }}\text{R}(\hat{\Theta }, \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \end{split}$$   (Eq  24)
where the first equation aims to find one $\Theta $ that minimizes our new objective $\text{R}(\cdot )$ , and the second equation aims to find one $\Theta ^{\prime }$ maximizing the same objective 
Intuitively, the first equation of Formula 24 tries to minimize the NER loss, but at the same time maximize the worker discriminator loss by the shared parameters of the common Bi-LSTM  Thus the resulting features of common Bi-LSTM actually attempt to hurt the worker discriminator, which makes these features worker independent since they are unable to distinguish different workers  The second equation tries to minimize the worker discriminator loss by its own parameter $\Theta ^{\prime }$  
We use the standard back-propagation method to train the model parameters, the same as the baseline model  In order to incorporate the term of the argmax part of Formula 24 , we follow the previous work of adversarial training BIBREF13 , BIBREF15 , BIBREF17 , by introducing a gradient reverse layer between the common Bi-LSTM and the CNN module, whose forward does nothing but the backward simply negates the gradients 
Data Sets
With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain  We hire undergraduate students to annotate the sentences  They are required to identify the predefined types of entities in the sentences  Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences 
Labeled Data: DL-PS  In Dialog domain (DL), we collect raw sentences from a chatbot application  And then we randomly select 20K sentences as our pool and hire 43 students to annotate the sentences  We ask the annotators to label two types of entities: Person-Name and Song-Name  The annotators label the sentences independently  In particular, each sentence is assigned to three annotators for this data  Although the setting can be wasteful of labor, we can use the resulting dataset to test several well-known baselines such as majority voting 
After annotation, we remove some illegal sentences reported by the annotators  Finally, we have 16,948 sentences annotated by the students  Table 1 shows the information of annotated data  The average Kappa value among the annotators is 0 6033, indicating that the crowd annotators have moderate agreement on identifying entities on this data 
In order to evaluate the system performances, we create a set of corpus with gold annotations  Concretely, we randomly select 1,000 sentences from the final dataset and let two experts generate the gold annotations  Among them, we use 300 sentences as the development set and the remaining 700 as the test set  The rest sentences with only student annotations are used as the training set 
Labeled data: EC-MT and EC-UQ  In E-commerce domain (EC), we collect raw sentences from two types of texts: one is titles of merchandise entries (EC-MT) and another is user queries (EC-UQ)  The annotators label five types of entities: Brand, Product, Model, Material, and Specification  These five types of entities are very important for E-commerce platform, for example building knowledge graph of merchandises  Five students participate the annotations for this domain since the number of sentences is small  We use the similar strategy as DL-PS to annotate the sentences, except that only two annotators are assigned for each sentence, because we aim to test the system performances under very small duplicated annotations 
Finally, we obtain 2,337 sentences for EC-MT and 2,300 for EC-UQ  Table 1 shows the information of annotated results  Similarly, we produce the development and test datasets for system evaluation, by randomly selecting 400 sentences and letting two experts to generate the groundtruth annotations  Among them, we use 100 sentences as the development set and the remaining 300 as the test set  The rest sentences with only crowdsourcing annotations are used as the training set 
Unlabeled data  The vector representations of characters are basic inputs of our baseline and proposed models, which are obtained by the looking-up table $\mathbf {E}^W$   As introduced before, we can use pretrained embeddings from large-scale raw corpus to initialize the table  In order to pretrain the character embeddings, we use one large-scale unlabeled data from the user-generated content in Internet  Totally, we obtain a number of 5M sentences  Finally, we use the tool word2vec to pretrain the character embeddings based on the unlabeled dataset in our experiments 
Settings
For evaluation, we use the entity-level metrics of Precision (P), Recall (R), and their F1 value in our experiments, treating one tagged entity as correct only when it matches the gold entity exactly 
There are several hyper-parameters in the baseline LSTM-CRF and our final models  We set them empirically by the development performances  Concretely, we set the dimension size of the character embeddings by 100, the dimension size of the NE label embeddings by 50, and the dimension sizes of all the other hidden features by 200 
We exploit online training with a mini-batch size 128 to learn model parameters  The max-epoch iteration is set by 200, and the best-epoch model is chosen according to the development performances  We use RMSprop BIBREF28 with a learning rate $10^{-3}$ to update model parameters, and use $l_2$ -regularization by a parameter $10^{-5}$   We adopt the dropout technique to avoid overfitting by a drop value of $0 2$  
Comparison Systems
The proposed approach (henceforward referred to as “ALCrowd”) is compared with the following systems:
CRF: We use the Crfsuite tool to train a model on the crowdsourcing labeled data  As for the feature settings, we use the supervised version of BIBREF0 zhao2008unsupervised 
CRF-VT: We use the same settings of the CRF system, except that the training data is the voted version, whose groundtruths are produced by majority voting at the character level for each annotated sentence 
CRF-MA: The CRF model proposed by BIBREF3 rodrigues2014sequence, which uses a prior distributation to model multiple crowdsourcing annotators  We use the source code provided by the authors 
LSTM-CRF: Our baseline system trained on the crowdsourcing labeled data 
LSTM-CRF-VT: Our baseline system trained on the voted corpus, which is the same as CRF-VT 
LSTM-Crowd: The LSTM-CRF model with crowd annotation learning proposed by BIBREF4 nguyen2017aggregating  We use the source code provided by the authors 
The first three systems are based on the CRF model using traditional handcrafted features, and the last three systems are based on the neural LSTM-CRF model  Among them, CRF-MA, LSTM-Crowd and our system with adversarial learning (ALCrowd) are based on crowd annotation learning that directly trains the model on the crowd-annotations  Five systems, including CRF, CRF-MA, LSTM-CRF, LSTM-Crowd, and ALCrowd, are trained on the original version of labeled data, while CRF-VT and LSTM-CRF-VT are trained on the voted version  Since CRF-VT, CRF-MA and LSTM-CRF-VT all require ground-truth answers for each training sentence, which are difficult to be produced with only two annotations, we do not apply the three models on the two EC datasets 
Main Results
In this section, we show the model performances of our proposed crowdsourcing learning system (ALCrowd), and meanwhile compare it with the other systems mentioned above  Table 2 shows the experimental results on the DL-PS datasets and Table 3 shows the experiment results on the EC-MT and EC-UQ datasets, respectively 
The results of CRF and LSTM-CRF mean that the crowd annotation is an alternative solution with low cost for labeling data that could be used for training a NER system even there are some inconsistencies  Compared with CRF, LSTM-CRF achieves much better performances on all the three data, showing +6 12 F1 improvement on DL-PS, +4 51 on EC-MT, and +9 19 on EC-UQ  This indicates that LSTM-CRF is a very strong baseline system, demonstrating the effectiveness of neural network 
Interestingly, when compared with CRF and LSTM-CRF, CRF-VT and LSTM-CRF-VT trained on the voted version perform worse in the DL-PS dataset  This trend is also mentioned in BIBREF4 nguyen2017aggregating  This fact shows that the majority voting method might be unsuitable for our task  There are two possible reasons accounting for the observation  On the one hand, simple character-level voting based on three annotations for each sentence may be still not enough  In the DL-PS dataset, even with only two predefined entity types, one character can have nine NE labels  Thus the majority-voting may be incapable of handling some cases  While the cost by adding more annotations for each sentence would be greatly increased  On the other hand, the lost information produced by majority-voting may be important, at least the ambiguous annotations denote that the input sentence is difficult for NER  The normal CRF and LSTM-CRF models without discard any annotations can differentiate these difficult contexts through learning 
Three crowd-annotation learning systems provide better performances than their counterpart systems, (CRF-MA VS CRF) and (LSTM-Crowd/ALCrowd VS LSTM-CRF)  Compared with the strong baseline LSTM-CRF, ALCrowd shows its advantage with +1 08 F1 improvements on DL-PS, +1 24 on EC-MT, and +2 38 on EC-UQ, respectively  This indicates that adding the crowd-annotation learning is quite useful for building NER systems  In addition, ALCrowd also outperforms LSTM-Crowd on all the datasets consistently, demonstrating the high effectiveness of ALCrowd in extracting worker independent features  Among all the systems, ALCrowd performs the best, and significantly better than all the other models (the p-value is below $10^{-5}$ by using t-test)  The results indicate that with the help of adversarial training, our system can learn a better feature representation from crowd annotation 
Discussion
Impact of Character Embeddings  First, we investigate the effect of the pretrained character embeddings in our proposed crowdsourcing learning model  The comparison results are shown in Figure 2 , where Random refers to the random initialized character embeddings, and Pretrained refers to the embeddings pretrained on the unlabeled data  According to the results, we find that our model with the pretrained embeddings significantly outperforms that using the random embeddings, demonstrating that the pretrained embeddings successfully provide useful information 
Case Studies  Second, we present several case studies in order to study the differences between our baseline and the worker adversarial models  We conduct a closed test on the training set, the results of which can be regarded as modifications of the training corpus, since there exist inconsistent annotations for each training sentence among the different workers  Figure 3 shows the two examples from the DL-PS dataset, which compares the outputs of the baseline and our final models, as well as the majority-voting strategy 
In the first case, none of the annotations get the correct NER result, but our proposed model can capture it  The result of LSTM-CRF is the same as majority-voting  In the second example, the output of majority-voting is the worst, which can account for the reason why the same model trained on the voted corpus performs so badly, as shown in Table 2   The model of LSTM-CRF fails to recognize the named entity “Xiexie” because of not trusting the second annotation, treating it as one noise annotation  Our proposed model is able to recognize it, because of its ability of extracting worker independent features 
Conclusions
In this paper, we presented an approach to performing crowd annotation learning based on the idea of adversarial training for Chinese Named Entity Recognition (NER)  In our approach, we use a common and private Bi-LSTMs for representing annotator-generic and -specific information, and learn a label Bi-LSTM from the crowd-annotated NE label sequences  Finally, the proposed approach adopts a LSTM-CRF model to perform tagging  In our experiments, we create two data sets for Chinese NER tasks in the dialog and e-commerce domains  The experimental results show that the proposed approach outperforms strong baseline systems 
 Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No  61572338, 61525205, and 61602160)  This work is also partially supported by the joint research project of Alibaba and Soochow University  Wenliang is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization 
Context 2: Introduction
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda  In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2 
Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf  Section SECREF3)  Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques  Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system 
With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop  The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level  We hope that the corpus would raise interest outside of the community of researchers studying propaganda  For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
Related Work
Propaganda has been tackled mostly at the article level  BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire  BIBREF4 experimented with a binarized version of that corpus: propaganda vs  the other three categories  BIBREF5 annotated a large binary corpus of propagandist vs  non-propagandist articles and proposed a feature-based system for discriminating between them  In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6 
A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda  BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy  BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1 3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda 
Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles  Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments 
The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task 
Propaganda Techniques
Propaganda uses psychological and rhetorical techniques to achieve its objective  Such techniques include the use of logical fallacies and appeal to emotions  For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources  We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:
Propaganda Techniques ::: 1  Loaded language 
Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11 
Propaganda Techniques ::: 2  Name calling or labeling 
Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12 
Propaganda Techniques ::: 3  Repetition 
Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12 
Propaganda Techniques ::: 4  Exaggeration or minimization 
Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e g , saying that an insult was just a joke 
Propaganda Techniques ::: 5  Doubt 
Questioning the credibility of someone or something 
Propaganda Techniques ::: 6  Appeal to fear/prejudice 
Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments 
Propaganda Techniques ::: 7  Flag-waving 
Playing on strong national feeling (or with respect to a group, e g , race, gender, political preference) to justify or promote an action or idea BIBREF15 
Propaganda Techniques ::: 8  Causal oversimplification 
Assuming one cause when there are multiple causes behind an issue  We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue 
Propaganda Techniques ::: 9  Slogans 
A brief and striking phrase that may include labeling and stereotyping  Slogans tend to act as emotional appeals BIBREF16 
Propaganda Techniques ::: 10  Appeal to authority 
Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17  We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14 
Propaganda Techniques ::: 11  Black-and-white fallacy, dictatorship 
Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13  As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship) 
Propaganda Techniques ::: 12  Thought-terminating cliché 
Words or phrases that discourage critical thought and meaningful discussion about a given topic  They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18 
Propaganda Techniques ::: 13  Whataboutism 
Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19 
Propaganda Techniques ::: 14  Reductio ad Hitlerum 
Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience  It can refer to any person or concept with a negative connotation BIBREF20 
Propaganda Techniques ::: 15  Red herring 
Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11  Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20 
Propaganda Techniques ::: 16  Bandwagon 
Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15 
Propaganda Techniques ::: 17  Obfuscation, intentional vagueness, confusion 
Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11  For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion 
Propaganda Techniques ::: 18  Straw man 
When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22 
Tasks
The shared task features two subtasks:
Tasks ::: Fragment-Level Classification task (FLC) 
Given a news article, detect all spans of the text in which a propaganda technique is used  In addition, for each span the propaganda technique applied must be identified

ground_truth: F1 scores of 85.99 on the DL-PS data,  75.15 on the EC-MT data and 71.53 on the EC-UQ data 

F1 of 85.99 on the DL-PS dataset (dialog domain); 75.15 on EC-MT and 71.53 on EC-UQ (e-commerce domain)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline for this task?
Context 2: 
Tasks ::: Sentence-Level Classification task (SLC) 
A sentence is considered propagandist if it contains at least one propagandist fragment  We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted 
Data
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators  More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles  We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators 
The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively  Figure FIGREF15 shows an annotated example, which contains several propaganda techniques  For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling  Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language 
Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets 
Setup
The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided 
The participants tried to achieve the best performance on the development set  A live leaderboard kept track of the submissions 
The test set was released and the participants had few days to make final predictions 
In phase 2, no immediate feedback on the submissions was provided  The winner was determined based on the performance on the test set 
Evaluation ::: FLC task 
FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task)  While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details  We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard) 
Evaluation ::: SLC task 
SLC is a binary classification task with imbalanced data  Therefore, the official evaluation measure for the task is the standard F$_1$ measure  We further report Precision and Recall 
Baselines
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence  The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34 
The baseline for the FLC task generates spans and selects one of the 18 techniques randomly  The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41 
Participants and Approaches
A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions  For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions 
Below, we give an overview of the approaches as described in the participants' papers  Tables TABREF28 and TABREF29 offer a high-level summary 
Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only
Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class  The team fed one sentence at a time in order to reduce the workload  In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia  Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters  Moreover, oversampling of the least represented classes proved to be crucial for the final performance  Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs 
Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28  The balancing of the embedding space was carried out by means of synthetic minority class over-sampling  Then, the learned representations were fed into an LSTM 
Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only
Team CAUnLP BIBREF29 used two context-aware representations based on BERT  In the first representation, the target sentence is followed by the title of the article  In the second representation, the previous sentence is also added  They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$
Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings  They also observed a boost in performance when balancing the dataset by dropping some negative examples 
Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features  These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT 
Team YMJA BIBREF33 also based their approach on fine-tuned BERT  Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation 
Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF IDF and BERT vectors, as well as vocabularies and readability measures 
Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36 
Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38 
Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e g , emotion classification 
Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus 
Participants and Approaches ::: Teams Participating in Both Tasks
Team MIC-CIS BIBREF42 participated in both tasks  For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models  Beside these representations, multiple features of readability, sentiment and emotions were considered  For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features  Finally, they applied sentence- and fragment-level models jointly 
Team CUNLP BIBREF45 considered two approaches for the sentence-level task  The first approach was based on fine-tuning BERT  The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features  Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings 
Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing  They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall)  In order to deal with the imbalance, they performed cost-sensitive classification, i e , the errors on the smaller positive class were more costly  For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM 
Evaluation Results
The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf  Section SECREF6)  The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set  Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system 
Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task  For this task, the results tend to be more stable across the two sets  Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda  Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set 
Conclusion and Further Work
We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification  We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively  Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline  The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline 
We plan to make the schema and the dataset publicly available to be used beyond NLP4IF  We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task  It features two complimentary tasks:
Given a free-text article, identify the propagandist text spans 
Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains 
This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2 
Acknowledgments
This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project  The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news ” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) 
The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations 

ground_truth: The baseline system for the SLC task is a very simple logistic regression classifier with default parameters. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.

SLC task is a very simple logistic regression classifier, FLC task generates spans and selects one of the 18 techniques randomly

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What embedding techniques are explored in the paper?
Context 3: Introduction
Measures of semantic similarity and relatedness quantify the degree to which two concepts are similar (e g , INLINEFORM0 – INLINEFORM1 ) or related (e g , INLINEFORM2 – INLINEFORM3 )  Semantic similarity can be viewed as a special case of semantic relatedness – to be similar is one of many ways that a pair of concepts may be related  The automated discovery of groups of semantically similar or related terms is critical to improving the retrieval BIBREF0 and clustering BIBREF1 of biomedical and clinical documents, and the development of biomedical terminologies and ontologies BIBREF2  
There is a long history in using distributional methods to discover semantic similarity and relatedness (e g , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 )  These methods are all based on the distributional hypothesis, which holds that two terms that are distributionally similar (i e , used in the same context) will also be semantically similar BIBREF7 , BIBREF8   Recently word embedding techniques such as word2vec BIBREF9 have become very popular  Despite the prominent role that neural networks play in many of these approaches, at their core they remain distributional techniques that typically start with a word by word co–occurrence matrix, much like many of the more traditional approaches 
However, despite these successes distributional methods do not perform well when data is very sparse (which is common)  One possible solution is to use second–order co–occurrence vectors BIBREF10 , BIBREF11   In this approach the similarity between two words is not strictly based on their co–occurrence frequencies, but rather on the frequencies of the other words which occur with both of them (i e , second order co–occurrences)  This approach has been shown to be successful in quantifying semantic relatedness BIBREF12 , BIBREF13   However, while more robust in the face of sparsity, second–order methods can result in significant amounts of noise, where contextual information that is overly general is included and does not contribute to quantifying the semantic relatedness between the two concepts 
Our goal then is to discover methods that automatically reduce the amount of noise in a second–order co–occurrence vector  We achieve this by incorporating pairwise semantic similarity scores derived from a taxonomy into our second–order vectors, and then using these scores to select only the most semantically similar co–occurrences (thereby reducing noise) 
We evaluate our method on two datasets that have been annotated in multiple ways  One has been annotated for both similarity and relatedness, and the other has been annotated for relatedness by two different types of experts (medical doctors and medical coders)  Our results show that integrating second order co–occurrences with measures of semantic similarity increases correlation with our human reference standards  We also compare our result to a number of other studies which have applied various word embedding methods to the same reference standards we have used  We find that our method often performs at a comparable or higher level than these approaches  These results suggest that our methods of integrating semantic similarity and relatedness values have the potential to improve performance of purely distributional methods 
Similarity and Relatedness Measures
This section describes the similarity and relatedness measures we integrate in our second–order co–occurrence vectors  We use two taxonomies in this study, SNOMED–CT and MeSH  SNOMED–CT (Systematized Nomenclature of Medicine Clinical Terms) is a comprehensive clinical terminology created for the electronic representation of clinical health information  MeSH (Medical Subject Headings) is a taxonomy of biomedical terms developed for indexing biomedical journal articles 
We obtain SNOMED–CT and MeSH via the Unified Medical Language System (UMLS) Metathesaurus (version 2016AA)  The Metathesaurus contains approximately 2 million biomedical and clinical concepts from over 150 different terminologies that have been semi–automatically integrated into a single source  Concepts in the Metathesaurus are connected largely by two types of hierarchical relations: INLINEFORM0 / INLINEFORM1 (PAR/CHD) and INLINEFORM2 / INLINEFORM3 (RB/RN) 
Similarity Measures
Measures of semantic similarity can be classified into three broad categories : path–based, feature–based and information content (IC)  Path–based similarity measures use the structure of a taxonomy to measure similarity – concepts positioned close to each other are more similar than those further apart  Feature–based methods rely on set theoretic measures of overlap between features (union and intersection)  The information content measures quantify the amount of information that a concept provides – more specific concepts have a higher amount of information content 
RadaMBB89 introduce the Conceptual Distance measure  This measure is simply the length of the shortest path between two concepts ( INLINEFORM0 and INLINEFORM1 ) in the MeSH hierarchy  Paths are based on broader than (RB) and narrower than (RN) relations  CaviedesC04 extends this measure to use parent (PAR) and child (CHD) relations  Our INLINEFORM2 measure is simply the reciprocal of this shortest path value (Equation EQREF3 ), so that larger values (approaching 1) indicate a high degree of similarity  DISPLAYFORM0
While the simplicity of INLINEFORM0 is appealing, it can be misleading when concepts are at different levels of specificity  Two very general concepts may have the same path length as two very specific concepts  WuP94 introduce a correction to INLINEFORM1 that incorporates the depth of the concepts, and the depth of their Least Common Subsumer (LCS)  This is the most specific ancestor two concepts share  In this measure, similarity is twice the depth of the two concept's LCS divided by the product of the depths of the individual concepts (Equation EQREF4 )  Note that if there are multiple LCSs for a pair of concepts, the deepest of them is used in this measure  DISPLAYFORM0
ZhongZLY02 take a very similar approach and again scale the depth of the LCS by the sum of the depths of the two concepts (Equation EQREF5 ), where INLINEFORM0   The value of INLINEFORM1 was set to 2 based on their recommendations  DISPLAYFORM0
PekarS02 offer another variation on INLINEFORM0 , where the shortest path of the two concepts to the LCS is used, in addition to the shortest bath between the LCS and the root of the taxonomy (Equation EQREF6 )  DISPLAYFORM0
Feature–based methods represent each concept as a set of features and then measure the overlap or sharing of features to measure similarity  In particular, each concept is represented as the set of their ancestors, and similarity is a ratio of the intersection and union of these features 
MaedcheS01 quantify the similarity between two concepts as the ratio of the intersection over their union as shown in Equation EQREF8   DISPLAYFORM0
BatetSV11 extend this by excluding any shared features (in the numerator) as shown in Equation EQREF9   DISPLAYFORM0
Information content is formally defined as the negative log of the probability of a concept  The effect of this is to assign rare (low probability) concepts a high measure of information content, since the underlying assumption is that more specific concepts are less frequently used than more common ones 
Resnik95 modified this notion of information content in order to use it as a similarity measure  He defines the similarity of two concepts to be the information content of their LCS (Equation EQREF11 )  DISPLAYFORM0
JiangC97, Lin98, and PirroE10 extend INLINEFORM0 by incorporating the information content of the individual concepts in various different ways  Lin98 defines the similarity between two concepts as the ratio of information content of the LCS with the sum of the individual concept's information content (Equation EQREF12 )  Note that INLINEFORM1 has the same form as INLINEFORM2 and INLINEFORM3 , and is in effect using information content as a measure of specificity (rather than depth)  If there is more than one possible LCS, the LCS with the greatest IC is chosen  DISPLAYFORM0
JiangC97 define the distance between two concepts to be the sum of the information content of the two concepts minus twice the information content of the concepts' LCS  We modify this from a distance to a similarity measure by taking the reciprocal of the distance (Equation EQREF13 )  Note that the denominator of INLINEFORM0 is very similar to the numerator of INLINEFORM1   DISPLAYFORM0
PirroE10 define the similarity between two concepts as the information content of the two concept's LCS divided by the sum of their individual information content values minus the information content of their LCS (Equation EQREF14 )  Note that INLINEFORM0 can be viewed as a set–theoretic version of INLINEFORM1   DISPLAYFORM0
Information Content
The information content of a concept may be derived from a corpus (corpus–based) or directly from a taxonomy (intrinsic–based)  In this work we focus on corpus–based techniques 
For corpus–based information content, we estimate the probability of a concept INLINEFORM0 by taking the sum of the probability of the concept INLINEFORM1 and the probability its descendants INLINEFORM2 (Equation EQREF16 )  DISPLAYFORM0
The initial probabilities of a concept ( INLINEFORM0 ) and its descendants ( INLINEFORM1 ) are obtained by dividing the number of times each concept and descendant occurs in the corpus, and dividing that by the total numbers of concepts ( INLINEFORM2 ) 
Ideally the corpus from which we are estimating the probabilities of concepts will be sense–tagged  However, sense–tagging is a challenging problem in its own right, and it is not always possible to carry out reliably on larger amounts of text  In fact in this paper we did not use any sense–tagging of the corpus we derived information content from 
Instead, we estimated the probability of a concept by using the UMLSonMedline dataset  This was created by the National Library of Medicine and consists of concepts from the 2009AB UMLS and the counts of the number of times they occurred in a snapshot of Medline taken on 12 January, 2009  These counts were obtained by using the Essie Search Engine BIBREF14 which queried Medline with normalized strings from the 2009AB MRCONSO table in the UMLS  The frequency of a CUI was obtained by aggregating the frequency counts of the terms associated with the CUI to provide a rough estimate of its frequency  The information content measures then use this information to calculate the probability of a concept 
Another alternative is the use of Intrinsic Information Content  It assess the informativeness of concept based on its placement within a taxonomy by considering the number of incoming (ancestors) relative to outgoing (descendant) links BIBREF15 (Equation EQREF17 )  DISPLAYFORM0
where INLINEFORM0 are the number of descendants of concept INLINEFORM1 that are leaf nodes, INLINEFORM2 are the number of concept INLINEFORM3 's ancestors and INLINEFORM4 are the total number of leaf nodes in the taxonomy 
Relatedness Measures
Lesk86 observed that concepts that are related should share more words in their respective definitions than concepts that are less connected  He was able to perform word sense disambiguation by identifying the senses of words in a sentence with the largest number of overlaps between their definitions  An overlap is the longest sequence of one or more consecutive words that occur in both definitions  BanerjeeP03 extended this idea to WordNet, but observed that WordNet glosses are often very short, and did not contain enough information to distinguish between multiple concepts  Therefore, they created a super–gloss for each concept by adding the glosses of related concepts to the gloss of the concept itself (and then finding overlaps) 
PatwardhanP06 adapted this measure to second–order co–occurrence vectors  In this approach, a vector is created for each word in a concept's definition that shows which words co–occur with it in a corpus  These word vectors are averaged to create a single co-occurrence vector for the concept  The similarity between the concepts is calculated by taking the cosine between the concepts second–order vectors  LiuMPMP12 modified and extended this measure to be used to quantify the relatedness between biomedical and clinical terms in the UMLS  The work in this paper can be seen as a further extension of PatwardhanP06 and LiuMPMP12 
Method
In this section, we describe our second–order similarity vector measure  This incorporates both contextual information using the term pair's definition and their pairwise semantic similarity scores derived from a taxonomy  There are two stages to our approach  First, a co–occurrence matrix must be constructed  Second, this matrix is used to construct a second–order co–occurrence vector for each concept in a pair of concepts to be measured for relatedness 
Co–occurrence Matrix Construction
We build an INLINEFORM0 similarity matrix using an external corpus where the rows and columns represent words within the corpus and the element contains the similarity score between the row word and column word using the similarity measures discussed above  If a word maps to more than one possible sense, we use the sense that returns the highest similarity score 
For this paper our external corpus was the NLM 2015 Medline baseline  Medline is a bibliographic database containing over 23 million citations to journal articles in the biomedical domain and is maintained by National Library of Medicine  The 2015 Medline Baseline encompasses approximately 5,600 journals starting from 1948 and contains 23,343,329 citations, of which 2,579,239 contain abstracts  In this work, we use Medline titles and abstracts from 1975 to present day  Prior to 1975, only 2% of the citations contained an abstract  We then calculate the similarity for each bigram in this dataset and include those that have a similarity score greater than a specified threshold on these experiments 
Measure Term Pairs for Relatedness
We obtain definitions for each of the two terms we wish to measure  Due to the sparsity and inconsistencies of the definitions in the UMLS, we not only use the definition of the term (CUI) but also include the definition of its related concepts  This follows the method proposed by PatwardhanP06 for general English and WordNet, and which was adapted for the UMLS and the medical domain by LiuMPMP12  In particular we add the definitions of any concepts connected via a parent (PAR), child (CHD), RB (broader than), RN (narrower than) or TERM (terms associated with CUI) relation  All of the definitions for a term are combined into a single super–gloss  At the end of this process we should have two super–glosses, one for each term to be measured for relatedness 
Next, we process each super–gloss as follows:
We extract a first–order co–occurrence vector for each term in the super–gloss from the co–occurrence matrix created previously 
We take the average of the first order co–occurrence vectors associated with the terms in a super–gloss and use that to represent the meaning of the term  This is a second–order co–occurrence vector 
After a second–order co–occurrence vector has been constructed for each term, then we calculate the cosine between these two vectors to measure the relatedness of the terms 
Data
We use two reference standards to evaluate the semantic similarity and relatedness measures   UMNSRS was annotated for both similarity and relatedness by medical residents  MiniMayoSRS was annotated for relatedness by medical doctors (MD) and medical coders (coder)  In this section, we describe these data sets and describe a few of their differences 
MiniMayoSRS: The MayoSRS, developed by PakhomovPMMRC10, consists of 101 clinical term pairs whose relatedness was determined by nine medical coders and three physicians from the Mayo Clinic  The relatedness of each term pair was assessed based on a four point scale: (4 0) practically synonymous, (3 0) related, (2 0) marginally related and (1 0) unrelated  MiniMayoSRS is a subset of the MayoSRS and consists of 30 term pairs on which a higher inter–annotator agreement was achieved  The average correlation between physicians is 0 68  The average correlation between medical coders is 0 78  We evaluate our method on the mean of the physician scores, and the mean of the coders scores in this subset in the same manner as reported by PedersenPPC07 
UMNSRS: The University of Minnesota Semantic Relatedness Set (UMNSRS) was developed by PakhomovMALPM10, and consists of 725 clinical term pairs whose semantic similarity and relatedness was determined independently by four medical residents from the University of Minnesota Medical School  The similarity and relatedness of each term pair was annotated based on a continuous scale by having the resident touch a bar on a touch sensitive computer screen to indicate the degree of similarity or relatedness  The Intraclass Correlation Coefficient (ICC) for the reference standard tagged for similarity was 0 47, and 0 50 for relatedness  Therefore, as suggested by Pakhomov and colleagues,we use a subset of the ratings consisting of 401 pairs for the similarity set and 430 pairs for the relatedness set which each have an ICC of 0 73 
Experimental Framework
We conducted our experiments using the freely available open source software package UMLS::Similarity BIBREF16 version 1 47  This package takes as input two terms (or UMLS concepts) and returns their similarity or relatedness using the measures discussed in Section SECREF2  
Correlation between the similarity measures and human judgments were estimated using Spearman's Rank Correlation ( INLINEFORM0 )  Spearman's measures the statistical dependence between two variables to assess how well the relationship between the rankings of the variables can be described using a monotonic function  We used Fisher's r-to-z transformation BIBREF17 to calculate the significance between the correlation results 
Results and Discussion
Table TABREF26 shows the Spearman's Rank Correlation between the human scores from the four reference standards and the scores from the various measures of similarity introduced in Section SECREF2   Each class of measure is followed by the scores obtained when integrating our second order vector approach with these measures of semantic similarity 
Results Comparison
The results for UMNSRS tagged for similarity ( INLINEFORM0 ) and MiniMayoSRS tagged by coders show that all of the second-order similarity vector measures ( INLINEFORM1 ) except for INLINEFORM2 - INLINEFORM3 obtain a higher correlation than the original measures  We found that INLINEFORM4 - INLINEFORM5 and INLINEFORM6 - INLINEFORM7 obtain the highest correlations of all these results with human judgments 
For the UMNSRS dataset tagged for relatedness and MiniMayoSRS tagged by physicians (MD), the original INLINEFORM0 measure obtains a higher correlation than our measure ( INLINEFORM1 ) although the difference is not statistically significant ( INLINEFORM2 ) 
In order to analyze and better understand these results, we filtered the bigram pairs used to create the initial similarity matrix based on the strength of their similarity using the INLINEFORM0 and the INLINEFORM1 measures  Note that the INLINEFORM2 measure holds to a 0 to 1 scale, while INLINEFORM3 ranges from 0 to an unspecified upper bound that is dependent on the size of the corpus from which information content is estimated  As such we use a different range of threshold values for each measure  We discuss the results of this filtering below 
Thresholding Experiments
Table TABREF29 shows the results of applying the threshold parameter on each of the reference standards using the INLINEFORM0 measure  For example, a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of 1 indicates that only the bigram pairs with a similarity score greater than one were included 
These results show that using a threshold cutoff of 2 obtains the highest correlation for the UMNSRS dataset, and that a threshold cutoff of 4 obtains the highest correlation for the MiniMayoSRS dataset  All of the results show an increase in correlation with human judgments when incorporating a threshold cutoff over all of the original measures  The increase in the correlation for the UMNSRS tagged for similarity is statistically significant ( INLINEFORM0 ), however this is not the case for the UMNSRS tagged for relatedness nor for the MiniMayoSRS data 
Similarly, Table TABREF30 shows the results of applying the threshold parameter (T) on each of the reference standards using the INLINEFORM0 measure  Although, unlike INLINEFORM1 whose scores are greater than or equal to 0 without an upper limit, the INLINEFORM2 measure returns scores between 0 and 1 (inclusive)  Therefore, here a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of INLINEFORM3 indicates that only the bigram pairs with a similarity score greater than INLINEFORM4 were included  The results show an increase in accuracy for all of the datasets except for the MiniMayoSRS tagged for physicians  The increase in the results for the UMNSRS tagged for similarity and the MayoSRS is statistically significant ( INLINEFORM5 )  This is not the case for the UMNSRS tagged for relatedness nor the MiniMayoSRS 
Overall, these results indicate that including only those bigrams that have a sufficiently high similarity score increases the correlation results with human judgments, but what quantifies as sufficiently high varies depending on the dataset and measure 
Comparison with Previous Work
Recently, word embeddings BIBREF9 have become a popular method for measuring semantic relatedness in the biomedical domain  This is a neural network based approach that learns a representation of a word by word co–occurrence matrix  The basic idea is that the neural network learns a series of weights (the hidden layer within the neural network) that either maximizes the probability of a word given its context, referred to as the continuous bag of words (CBOW) approach, or that maximizes the probability of the context given a word, referred to as the Skip–gram approach  These approaches have been used in numerous recent papers 
muneeb2015evalutating trained both the Skip–gram and CBOW models over the PubMed Central Open Access (PMC) corpus of approximately 1 25 million articles  They evaluated the models on a subset of the UMNSRS data, removing word pairs that did not occur in their training corpus more than ten times  chiu2016how evaluated both the the Skip–gram and CBOW models over the PMC corpus and PubMed  They also evaluated the models on a subset of the UMNSRS ignoring those words that did not appear in their training corpus  Pakhomov2016corpus trained CBOW model over three different types of corpora: clinical (clinical notes from the Fairview Health System), biomedical (PMC corpus), and general English (Wikipedia)  They evaluated their method using a subset of the UMNSRS restricting to single word term pairs and removing those not found within their training corpus  sajad2015domain trained the Skip–gram model over CUIs identified by MetaMap on the OHSUMED corpus, a collection of 348,566 biomedical research articles  They evaluated the method on the complete UMNSRS, MiniMayoSRS and the MayoSRS datasets; any subset information about the dataset was not explicitly stated therefore we believe a direct comparison may be possible 
In addition, a previous work very closely related to ours is a retrofitting vector method proposed by YuCBJW16 that incorporates ontological information into a vector representation by including semantically related words  In their measure, they first map a biomedical term to MeSH terms, and second build a word vector based on the documents assigned to the respective MeSH term  They then retrofit the vector by including semantically related words found in the Unified Medical Language System  They evaluate their method on the MiniMayoSRS dataset 
Table TABREF31 shows a comparison to the top correlation scores reported by each of these works on the respective datasets (or subsets) they evaluated their methods on  N refers to the number of term pairs in the dataset the authors report they evaluated their method  The table also includes our top scoring results: the integrated vector-res and vector-faith  The results show that integrating semantic similarity measures into second–order co–occurrence vectors obtains a higher or on–par correlation with human judgments as the previous works reported results with the exception of the UMNSRS rel dataset  The results reported by Pakhomov2016corpus and chiu2016how obtain a higher correlation although the results can not be directly compared because both works used different subsets of the term pairs from the UMNSRS dataset 
Conclusion and Future Work
We have presented a method for quantifying the similarity and relatedness between two terms that integrates pair–wise similarity scores into second–order vectors  The goal of this approach is two–fold  First, we restrict the context used by the vector measure to words that exist in the biomedical domain, and second, we apply larger weights to those word pairs that are more similar to each other  Our hypothesis was that this combination would reduce the amount of noise in the vectors and therefore increase their correlation with human judgments  We evaluated our method on datasets that have been manually annotated for relatedness and similarity and found evidence to support this hypothesis  In particular we discovered that guiding the creation of a second–order context vector by selecting term pairs from biomedical text based on their semantic similarity led to improved levels of correlation with human judgment 
We also explored using a threshold cutoff to include only those term pairs that obtained a sufficiently large level of similarity  We found that eliminating less similar pairs improved the overall results (to a point)  In the future, we plan to explore metrics to automatically determine the threshold cutoff appropriate for a given dataset and measure  We also plan to explore additional features that can be integrated with a second–order vector measure that will reduce the noise but still provide sufficient information to quantify relatedness  We are particularly interested in approaches that learn word, phrase, and sentence embeddings from structured corpora such as literature BIBREF23 and dictionary entries BIBREF24   Such embeddings could be integrated into a second–order vector or be used on their own 
Finally, we compared our proposed method to other distributional approaches, focusing on those that used word embeddings  Our results showed that integrating semantic similarity measures into second–order co–occurrence vectors obtains the same or higher correlation with human judgments as do various different word embedding approaches  However, a direct comparison was not possible due to variations in the subsets of the UMNSRS evaluation dataset used  In the future, we would not only like to conduct a direct comparison but also explore integrating semantic similarity into various kinds of word embeddings by training on pair–wise values of semantic similarity as well as co–occurrence statistics 

ground_truth: Skip–gram, CBOW

integrated vector-res, vector-faith, Skip–gram, CBOW