prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On top of BERT does the RNN layer work better or the transformer layer?
Context 1: Introduction
Bidirectional Encoder Representations from Transformers (BERT) is a novel Transformer BIBREF0 model, which recently achieved state-of-the-art performance in several language understanding tasks, such as question answering, natural language inference, semantic similarity, sentiment analysis, and others BIBREF1  While well-suited to dealing with relatively short sequences, Transformers suffer from a major issue that hinders their applicability in classification of long sequences, i e  they are able to consume only a limited context of symbols as their input BIBREF2 
There are several natural language (NLP) processing tasks that involve such long sequences  Of particular interest are topic identification of spoken conversations BIBREF3, BIBREF4, BIBREF5 and call center customer satisfaction prediction BIBREF6, BIBREF7, BIBREF8, BIBREF9  Call center conversations, while usually quite short and to the point, often involve agents trying to solve very complex issues that the customers experience, resulting in some calls taking even an hour or more  For speech analytics purposes, these calls are typically transcribed using an automatic speech recognition (ASR) system, and processed in textual representations further down the NLP pipeline  These transcripts sometimes exceed the length of 5000 words  Furthermore, temporal information might play an important role in tasks like CSAT  For example, a customer may be angry at the beginning of the call, but after her issue is resolved, she would be very satisfied with the way it was handled  Therefore, simple bag of words models, or any model that does not include temporal dependencies between the inputs, may not be well-suited to handle this category of tasks  This motivates us to employ model such as BERT in this task 
In this paper, we propose a method that builds upon BERT's architecture  We split the input text sequence into shorter segments in order to obtain a representation for each of them using BERT  Then, we use either a recurrent LSTM BIBREF10 network, or another Transformer, to perform the actual classification  We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT)  Given that these models introduce a hierarchy of representations (segment-wise and document-wise), we refer to them as Hierarchical Transformers  To the best of our knowledge, no attempt has been done before to use the Transformer architecture for classification of such long sequences 
Our novel contributions are:
Two extensions - RoBERT and ToBERT - to the BERT model, which enable its application in classification of long texts by performing segmentation and using another layer on top of the segment representations 
State-of-the-art results on the Fisher topic classification task 
Significant improvement on the CSAT prediction task over the MS-CNN model 
Related work
Several dimensionality reduction algorithms such as RBM, autoencoders, subspace multinomial models (SMM) are used to obtain a low dimensional representation of documents from a simple BOW representation and then classify it using a simple linear classifiers BIBREF11, BIBREF12, BIBREF13, BIBREF4  In BIBREF14 hierarchical attention networks are used for document classification  They evaluate their model on several datasets with average number of words around 150  Character-level CNN are explored in BIBREF15 but it is prohibitive for very long documents  In BIBREF16, dataset collected from arXiv papers is used for classification  For classification, they sample random blocks of words and use them together for classification instead of using full document which may work well as arXiv papers are usually coherent and well written on a well defined topic  Their method may not work well on spoken conversations as random block of words usually do not represent topic of full conversation 
Several researchers addressed the problem of predicting customer satisfaction BIBREF6, BIBREF7, BIBREF8, BIBREF9  In most of these works, logistic regression, SVM, CNN are applied on different kinds of representations 
In BIBREF17, authors use BERT for document classification but the average document length is less than BERT maximum length 512  TransformerXL BIBREF2 is an extension to the Transformer architecture that allows it to better deal with long inputs for the language modelling task  It relies on the auto-regressive property of the model, which is not the case in our tasks 
Method ::: BERT
Because our work builds heavily upon BERT, we provide a brief summary of its features  BERT is built upon the Transformer architecture BIBREF0, which uses self-attention, feed-forward layers, residual connections and layer normalization as the main building blocks  It has two pre-training objectives:
Masked language modelling - some of the words in a sentence are being masked and the model has to predict them based on the context (note the difference from the typical autoregressive language model training objective);
Next sentence prediction - given two input sequences, decide whether the second one is the next sentence or not 
BERT has been shown to beat the state-of-the-art performance on 11 tasks with no modifications to the model architecture, besides adding a task-specific output layer BIBREF1  We follow same procedure suggested in BIBREF1 for our tasks  Fig  FIGREF8 shows the BERT model for classification  We obtain two kinds of representation from BERT: pooled output from last transformer block, denoted by H, and posterior probabilities, denoted by P  There are two variants of BERT - BERT-Base and BERT-Large  In this work we are using BERT-Base for faster training and experimentation, however, our methods are applicable to BERT-Large as well  BERT-Base and BERT-Large are different in model parameters such as number of transformer blocks, number of self-attention heads  Total number of parameters in BERT-Base are 110M and 340M in BERT-Large 
BERT suffers from major limitations in terms of handling long sequences  Firstly, the self-attention layer has a quadratic complexity $O(n^2)$ in terms of the sequence length $n$ BIBREF0  Secondly, BERT uses a learned positional embeddings scheme BIBREF1, which means that it won't likely be able to generalize to positions beyond those seen in the training data 
To investigate the effect of fine-tuning BERT on task performance, we use either the pre-trained BERT weights, or the weights from a BERT fine-tuned on the task-specific dataset on a segment-level (i e  we preserve the original label but fine-tune on each segment separately instead of on the whole text sequence)  We compare these results to using the fine-tuned segment-level BERT predictions directly as inputs to the next layer 
Method ::: Recurrence over BERT
Given that BERT is limited to a particular input length, we split the input sequence into segments of a fixed size with overlap  For each of these segments, we obtain H or P from BERT model  We then stack these segment-level representations into a sequence, which serves as input to a small (100-dimensional) LSTM layer  Its output serves as a document embedding  Finally, we use two fully connected layers with ReLU (30-dimensional) and softmax (the same dimensionality as the number of classes) activations to obtain the final predictions 
With this approach, we overcome BERT's computational complexity, reducing it to $O(n/k * k^2) = O(nk)$ for RoBERT, with $k$ denoting the segment size (the LSTM component has negligible linear complexity $O(k)$)  The positional embeddings are also no longer an issue 
Method ::: Transformer over BERT
Given that Transformers' edge over recurrent networks is their ability to effectively capture long distance relationships between words in a sequence BIBREF0, we experiment with replacing the LSTM recurrent layer in favor of a small Transformer model (2 layers of transformer building block containing self-attention, fully connected, etc )  To investigate if preserving the information about the input sequence order is important, we also build a variant of ToBERT which learns positional embeddings at the segment-level representations (but is limited to sequences of length seen during the training) 
ToBERT's computational complexity $O(\frac{n^2}{k^2})$ is asymptotically inferior to RoBERT, as the top-level Transformer model again suffers from quadratic complexity in the number of segments  However, in practice this number is much smaller than the input sequence length (${\frac{n}{k}} << n$), so we haven't observed performance or memory issues with our datasets 
Experiments
We evaluated our models on 3 different datasets:
CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR) 
20 newsgroups for topic identification task, consisting of written text;
Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual);
Experiments ::: CSAT
CSAT dataset consists of US English telephone speech from call centers  For each call in this dataset, customers participated in that call gave a rating on his experience with agent  Originally, this dataset has labels rated on a scale 1-9 with 9 being extremely satisfied and 1 being extremely dissatisfied  Fig  FIGREF16 shows the histogram of ratings for our dataset  As the distribution is skewed towards extremes, we choose to do binary classification with ratings above 4 5 as satisfied and below 4 5 as dissatisfied  Quantization of ratings also helped us to create a balanced dataset  This dataset contains 4331 calls and we split them into 3 sets for our experiments: 2866 calls for training, 362 calls for validation and, finally, 1103 calls for testing 
We obtained the transcripts by employing an ASR system  The ASR system uses TDNN-LSTM acoustic model trained on Fisher and Switchboard datasets with lattice-free maximum mutual information criterion BIBREF18  The word error rates using four-gram language models were 9 2% and 17 3% respectively on Switchboard and CallHome portions of Eval2000 dataset 
Experiments ::: 20 newsgroups
20 newsgroups data set is one of the frequently used datasets in the text processing community for text classification and text clustering  This data set contains approximately 20,000 English documents from 20 topics to be identified, with 11314 documents for training and 7532 for testing  In this work, we used only 90% of documents for training and the remaining 10% for validation  For fair comparison with other publications, we used 53160 words vocabulary set available in the datasets website 
Experiments ::: Fisher
Fisher Phase 1 US English corpus is often used for automatic speech recognition in speech community  In this work, we used it for topic identification as in BIBREF3  The documents are 10-minute long telephone conversations between two people discussing a given topic  We used same training and test splits as BIBREF3 in which 1374 and 1372 documents are used for training and testing respectively  For validation of our model, we used 10% of training dataset and the remaining 90% was used for actual model training  The number of topics in this data set is 40 
Experiments ::: Dataset Statistics
Table TABREF22 shows statistics of our datasets  It can be observed that average length of Fisher is much higher than 20 newsgroups and CSAT  Cumulative distribution of document lengths for each dataset is shown in Fig  FIGREF21  It can be observed that almost all of the documents in Fisher dataset have length more than 1000 words  For CSAT, more than 50% of the documents have length greater than 500 and for 20newsgroups only 10% of the documents have length greater than 500  Note that, for CSAT and 20newsgroups, there are few documents with length more than 5000 
Experiments ::: Architecture and Training Details
In this work, we split document into segments of 200 tokens with a shift of 50 tokens to extract features from BERT model  For RoBERT, LSTM model is trained to minimize cross-entropy loss with Adam optimizer BIBREF19  The initial learning rate is set to $0 001$ and is reduced by a factor of $0 95$ if validation loss does not decrease for 3-epochs  For ToBERT, the Transformer is trained with the default BERT version of Adam optimizer BIBREF1 with an initial learning rate of $5e$-5  We report accuracy in all of our experiments  We chose a model with the best validation accuracy to calculate accuracy on the test set  To accomodate for non-determinism of some TensorFlow GPU operations, we report accuracy averaged over 5 runs 
Results
Table TABREF25 presents results using pre-trained BERT features  We extracted features from the pooled output of final transformer block as these were shown to be working well for most of the tasks BIBREF1  The features extracted from a pre-trained BERT model without any fine-tuning lead to a sub-par performance  However, We also notice that ToBERT model exploited the pre-trained BERT features better than RoBERT  It also converged faster than RoBERT  Table TABREF26 shows results using features extracted after fine-tuning BERT model with our datasets  Significant improvements can be observed compared to using pre-trained BERT features  Also, it can be noticed that ToBERT outperforms RoBERT on Fisher and 20newsgroups dataset by 13 63% and 0 81% respectively  On CSAT, ToBERT performs slightly worse than RoBERT but it is not statistically significant as this dataset is small 
Table TABREF27 presents results using fine-tuned BERT predictions instead of the pooled output from final transformer block  For each document, having obtained segment-wise predictions we can obtain final prediction for the whole document in three ways:
Compute the average of all segment-wise predictions and find the most probable class;
Find the most frequently predicted class;
Train a classification model 
It can be observed from Table TABREF27 that a simple averaging operation or taking most frequent predicted class works competitively for CSAT and 20newsgroups but not for the Fisher dataset  We believe the improvements from using RoBERT or ToBERT, compared to simple averaging or most frequent operations, are proportional to the fraction of long documents in the dataset  CSAT and 20newsgroups have (on average) significantly shorter documents than Fisher, as seen in Fig  FIGREF21  Also, significant improvements for Fisher could be because of less confident predictions from BERT model as this dataset has 40 classes  Fig  FIGREF31 presents the comparison of average voting and ToBERT for various document length ranges for Fisher dataset  We used fine-tuned BERT segment-level predictions (P) for this analysis  It can be observed that ToBERT outperforms average voting in every interval  To the best of our knowledge, this is a state-of-the-art result reported on the Fisher dataset 
Table TABREF32 presents the effect of position embeddings on the model performance  It can be observed that position embeddings did not significantly affect the model performance for Fisher and 20newsgroups, but they helped slightly in CSAT prediction (an absolute improvement of 0 64% F1-score)  We think that this is explained by the fact that Fisher and 20newsgroups are topic identification tasks, and the topic does not change much throughout these documents  However, CSAT may vary during the call, and in some cases a naive assumption that the sequential nature of the transcripts is irrelevant may lead to wrong conclusions 
Table TABREF33 compares our results with previous works  It can be seen that our model ToBERT outperforms CNN based experiments by significant margin on CSAT and Fisher datasets  For CSAT dataset, we used multi-scale CNN (MS-CNN) as the baseline, given its strong results on Fisher and 20newsgroups  The setup was replicated from BIBREF5 for comparison  We also see that our result on 20 newsgroups is 0 6% worse than the state-of-the-art 
Conclusions
In this paper, we presented two methods for long documents using BERT model: RoBERT and ToBERT  We evaluated our experiments on two classification tasks - customer satisfaction prediction and topic identification - using 3 datasets: CSAT, 20newsgroups and Fisher  We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks  Also, we noticed that fine-tuned BERT performs better than pre-trained BERT  We have shown that both RoBERT and ToBERT improved the simple baselines of taking an average (or the most frequent) of segment-wise predictions for long documents to obtain final prediction  Position embeddings did not significantly affect our models performance, but slightly improved the accuracy on the CSAT task  We obtained the best results on Fisher dataset and good improvements for CSAT task compared to the CNN baseline  It is interesting to note that the longer the average input in a given task, the bigger improvement we observe w r t  the baseline for that task  Our results confirm that both RoBERT and ToBERT can be used for long sequences with competitive performance and quick fine-tuning procedure  For future work, we shall focus on training models on long documents directly (i e  in an end-to-end manner) 
Context 2: Introduction
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages  Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc ) 
The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers  A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7   With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available 
As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs  Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12  
In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18   Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20  
Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results  Yet they rely on resources of different natures  The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data  They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information  Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance  Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations 
Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models  This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT  The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers)  In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models  Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations  A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs  CRF) and feature sets 
More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system  We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features  We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt  These models are trained on the Universal Dependencies (v1 2) corpus set BIBREF21 , complemented by morphosyntactic lexicons  We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons  We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23   We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains  The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations  More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English) 
MElt
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5   The basic set of features used by MElt is given in BIBREF12   It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word)  In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words 
One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features  This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features  Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12   They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time  These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words)  In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags  Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon 
Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank  Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point  With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word 
In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42   This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word 
We incorporated in MElt the best performing feature set, described in Table TABREF1   All models discussed in this paper are based on this feature set 
Corpora
We carried out our experiments on the Universal Dependencies v1 2 treebanks BIBREF21 , hereafter UD1 2, from which morphosyntactically annotated corpora can be trivially extracted  All UD1 2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43  
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1 2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results  We considered UD1 2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish  Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian)

ground_truth: Transformer over BERT (ToBERT)

The transformer layer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which languages are explored?
Context 2:  Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse 
Lexicons
We generate our external lexicons using the set of source lexicons listed in Table TABREF3   Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory  Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category  We also added entries for punctuations when the source lexicons did not contain any 
We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included  On average, results were slightly better than those presented in the paper, although not statistically significantly  Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags  This is why we only report results based on (coarse) PoS extracted from the original lexicons 
Baseline models
In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information  Table TABREF11 provides the results of MElt and MarMoT retrained on UD1 2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i e  the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings 
These results show that Plank et al 's (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian) 
Models enriched with external lexical information
Table TABREF13 provides the results of four systems enriched with lexical information  The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons  This extension takes the form of additional features, as described in Section SECREF2 for MElt  The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings  FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data  As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens ”
The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly  The best performing system in terms of macro-average is MElt (96 60%)  Both bi-LSTM systems reach the same score (96 58%), the difference with MElt's results being non significant, whereas MarMoT is only 0 14% behind (96 46%)  Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs 
Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy  The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively)  The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt  This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT  On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94 28% vs  83 59%), with 65% lower error rate 
One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words  All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot  Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high  In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio  Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set  When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 )  This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0 28, average accuracy difference: 0 32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0 15, average accuracy difference: -0 56 compared to bi-LSTM+Polyglot, -0 57 compared to FREQBIN+Polyglot) 
Conclusion
Two main conclusions can be drawn from our comparative results  First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings  Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e g  for English) 
We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20   As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18   However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question  Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models 
In that regard, further work will be required to understand which class of models perform the best  An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition 

ground_truth: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish

Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What accuracy does the proposed system achieve?
Context 3: Introduction
There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2   As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts  In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data  For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money 
In order to quickly obtain new training data, we can use crowdsourcing as one alternative way at lower cost in a short time  But as an exchange, crowd annotations from non-experts may be of lower quality than those from experts  It is one biggest challenge to build a powerful NER system on such a low quality annotated data  Although we can obtain high quality annotations for each input sentence by majority voting, it can be a waste of human labors to achieve such a goal, especially for some ambiguous sentences which may require a number of annotations to reach an agreement  Thus majority work directly build models on crowd annotations, trying to model the differences among annotators, for example, some of the annotators may be more trustful BIBREF3 , BIBREF4  
Here we focus mainly on the Chinese NER, which is more difficult than NER for other languages such as English for the lack of morphological variations such as capitalization and in particular the uncertainty in word segmentation  The Chinese NE taggers trained on news domain often perform poor in other domains  Although we can alleviate the problem by using character-level tagging to resolve the problem of poor word segmentation performances BIBREF5 , still there exists a large gap when the target domain changes, especially for the texts of social media  Thus, in order to get a good tagger for new domains and also for the conditions of new entity types, we require large amounts of labeled data  Therefore, crowdsourcing is a reasonable solution for these situations 
In this paper, we propose an approach to training a Chinese NER system on the crowd-annotated data  Our goal is to extract additional annotator independent features by adversarial training, alleviating the annotation noises of non-experts  The idea of adversarial training in neural networks has been used successfully in several NLP tasks, such as cross-lingual POS tagging BIBREF6 and cross-domain POS tagging BIBREF7   They use it to reduce the negative influences of the input divergences among different domains or languages, while we use adversarial training to reduce the negative influences brought by different crowd annotators  To our best knowledge, we are the first to apply adversarial training for crowd annotation learning 
In the learning framework, we perform adversarial training between the basic NER and an additional worker discriminator  We have a common Bi-LSTM for representing annotator-generic information and a private Bi-LSTM for representing annotator-specific information  We build another label Bi-LSTM by the crowd-annotated NE label sequence which reflects the mind of the crowd annotators who learn entity definitions by reading the annotation guidebook  The common and private Bi-LSTMs are used for NER, while the common and label Bi-LSTMs are used as inputs for the worker discriminator  The parameters of the common Bi-LSTM are learned by adversarial training, maximizing the worker discriminator loss and meanwhile minimizing the NER loss  Thus the resulting features of the common Bi-LSTM are worker invariant and NER sensitive 
For evaluation, we create two Chinese NER datasets in two domains: dialog and e-commerce  We require the crowd annotators to label the types of entities, including person, song, brand, product, and so on  Identifying these entities is useful for chatbot and e-commerce platforms BIBREF8   Then we conduct experiments on the newly created datasets to verify the effectiveness of the proposed adversarial neural network model  The results show that our system outperforms very strong baseline systems  In summary, we make the following contributions:
Related Work
Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing 
Sequence labeling  NER is widely treated as a sequence labeling problem, by assigning a unique label over each sentential word BIBREF9   Early studies on sequence labeling often use the models of HMM, MEMM, and CRF BIBREF10 based on manually-crafted discrete features, which can suffer the feature sparsity problem and require heavy feature engineering  Recently, neural network models have been successfully applied to sequence labeling BIBREF1 , BIBREF11 , BIBREF2   Among these work, the model which uses Bi-LSTM for feature extraction and CRF for decoding has achieved state-of-the-art performances BIBREF11 , BIBREF2 , which is exploited as the baseline model in our work 
Adversarial Training  Adversarial Networks have achieved great success in computer vision such as image generation BIBREF12 , BIBREF13   In the NLP community, the method is mainly exploited under the settings of domain adaption BIBREF14 , BIBREF7 , cross-lingual BIBREF15 , BIBREF6 and multi-task learning BIBREF16 , BIBREF17   All these settings involve the feature divergences between the training and test examples, and aim to learn invariant features across the divergences by an additional adversarial discriminator, such as domain discriminator  Our work is similar to these work but is applies on crowdsourcing learning, aiming to find invariant features among different crowdsourcing workers 
Crowdsourcing  Most NLP tasks require a massive amount of labeled training data which are annotated by experts  However, hiring experts is costly and non-scalable, both in terms of time and money  Instead, crowdsourcing is another solution to obtain labeled data at a lower cost but with relative lower quality than those from experts  BIBREF18 snow2008cheap collected labeled results for several NLP tasks from Amazon Mechanical Turk and demonstrated that non-experts annotations were quite useful for training new systems  In recent years, a series of work have focused on how to use crowdsourcing data efficiently in tasks such as classification BIBREF19 , BIBREF20 , and compare quality of crowd and expert labels BIBREF21  
In sequence labeling tasks, BIBREF22 dredze2009sequence viewed this task as a multi-label problem while BIBREF3 rodrigues2014sequence took workers identities into account by assuming that each sentential word was tagged correctly by one of the crowdsourcing workers and proposed a CRF-based model with multiple annotators  BIBREF4 nguyen2017aggregating introduced a crowd representation in which the crowd vectors were added into the LSTM-CRF model at train time, but ignored them at test time  In this paper, we apply adversarial training on crowd annotations on Chinese NER in new domains, and achieve better performances than previous studies on crowdsourcing learning 
Baseline: LSTM-CRF
We use a neural CRF model as the baseline system BIBREF9 , treating NER as a sequence labeling problem over Chinese characters, which has achieved state-of-the-art performances BIBREF5   To this end, we explore the BIEO schema to convert NER into sequence labeling, following BIBREF2 lample-EtAl:2016:N16-1, where sentential character is assigned with one unique tag  Concretely, we tag the non-entity character by label “O”, the beginning character of an entity by “B-XX”, the ending character of an entity by “E-XX” and the other character of an entity by “I-XX”, where “XX” denotes the entity type 
We build high-level neural features from the input character sequence by a bi-directional LSTM BIBREF2   The resulting features are combined and then are fed into an output CRF layer for decoding  In summary, the baseline model has three main components  First, we make vector representations for sentential characters $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , transforming the discrete inputs into low-dimensional neural inputs  Second, feature extraction is performed to obtain high-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , by using a bi-directional LSTM (Bi-LSTM) structure together with a linear transformation over $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$   Third, we apply a CRF tagging module over $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , obtaining the final output NE labels  The overall framework of the baseline model is shown by the right part of Figure 1  
Vector Representation of Characters
To represent Chinese characters, we simply exploit a neural embedding layer to map discrete characters into the low-dimensional vector representations  The goal is achieved by a looking-up table $\mathbf {E}^W$ , which is a model parameter and will be fine-tuned during training  The looking-up table can be initialized either by random or by using a pretrained embeddings from large scale raw corpus  For a given Chinese character sequence $c_1c_2\cdots c_n$ , we obtain the vector representation of each sentential character by: $ \mathbf {x}_t = \text{look-up}(c_t, \mathbf {E}^W), \text{~~~} t \in [1, n]$  
Feature Extraction
Based on the vector sequence $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , we extract higher-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by using a bidirectional LSTM module and a simple feed-forward neural layer, which are then used for CRF tagging at the next step 
LSTM is a type of recurrent neural network (RNN), which is designed for solving the exploding and diminishing gradients of basic RNNs BIBREF23   It has been widely used in a number of NLP tasks, including POS-tagging BIBREF11 , BIBREF24 , parsing BIBREF25 and machine translation BIBREF26 , because of its strong capabilities of modeling natural language sentences 
By traversing $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ by order and reversely, we obtain the output features $\mathbf {h}_1^{\text{private}}\mathbf {h}_2^{\text{private}}\cdots \mathbf {h}_n^{\text{private}}$ of the bi-LSTM, where $\mathbf {h}_t^{\text{private}} = \overrightarrow{\mathbf {h}}_t \oplus \overleftarrow{\mathbf {h}}_t $   Here we refer this Bi-LSTM as private in order to differentiate it with the common Bi-LSTM over the same character inputs which will be introduced in the next section 
Further we make an integration of the output vectors of bi-directional LSTM by a linear feed-forward neural layer, resulting in the features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by equation:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} \mathbf {h}_t^{\text{private}} + \mathbf {b},$$   (Eq  6)
where $\mathbf {W}$ and $\mathbf {b}$ are both model parameters 
CRF Tagging
Finally we feed the resulting features $\mathbf {h}_t^{\text{ner}}, t\in [1, n]$ into a CRF layer directly for NER decoding  CRF tagging is one globally normalized model, aiming to find the best output sequence considering the dependencies between successive labels  In the sequence labeling setting for NER, the output label of one position has a strong dependency on the label of the previous position  For example, the label before “I-XX” must be either “B-XX” or “I-XX”, where “XX” should be exactly the same 
CRF involves two parts for prediction  First we should compute the scores for each label based $\mathbf {h}_t^{\text{ner}}$ , resulting in $\mathbf {o}_t^{\text{ner}}$ , whose dimension is the number of output labels  The other part is a transition matrix $\mathbf {T}$ which defines the scores of two successive labels  $\mathbf {T}$ is also a model parameter  Based on $\mathbf {o}_t^{\text{ner}}$ and $\mathbf {T}$ , we use the Viterbi algorithm to find the best-scoring label sequence 
We can formalize the CRF tagging process as follows:
$$\begin{split} & \mathbf {o}_t^{\text{ner}} = \mathbf {W}^{\text{ner}} \mathbf {h}_t^{\text{ner}}, \text{~~~~} t \in [1,n] \\ & \text{score}(\mathbf {X}, \mathbf {y}) = \sum _{t = 1}^{n}(\mathbf {o}_{t,y_t} + T_{y_{t-1},y_t}) \\ & \mathbf {y}^{\text{ner}} = \mathop {arg~max}_{\mathbf {y}}\big (\text{score}(\mathbf {X}, \mathbf {y}))\big ), \\ \end{split}$$   (Eq  8)
where $\text{score}(\cdot )$ is the scoring function for a given output label sequence $\mathbf {y} = y_1y_2 \cdots y_n$ based on input $\mathbf {X}$ , $\mathbf {y}^{\text{ner}}$ is the resulting label sequence, $\mathbf {W}^{\text{ner}}$ is a model parameter 
Training
To train model parameters, we exploit a negative log-likelihood objective as the loss function  We apply softmax over all candidate output label sequences, thus the probability of the crowd-annotated label sequence is computed by:
$$p(\mathbf {\bar{y}}|\mathbf {X}) = \frac{\exp \big (\text{score}(\mathbf {X}, \mathbf {\bar{y}})\big )}{\sum _{\mathbf {y} \in \mathbf {Y}_{\mathbf {X}}} \exp \big (\text{score}(\mathbf {X}, \mathbf {y})\big )},$$   (Eq  10)
where $\mathbf {\bar{y}}$ is the crowd-annotated label sequences and $\mathbf {Y}_{\mathbf {X}}$ is all candidate label sequence of input $\mathbf {X}$  
Based on the above formula, the loss function of our baseline model is:
$$\text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) = -\log p(\mathbf {\bar{y}}|\mathbf {X}),$$   (Eq  11)
where $\Theta $ is the set of all model parameters  We use standard back-propagation method to minimize the loss function of the baseline CRF model 
Worker Adversarial
Adversarial learning has been an effective mechanism to resolve the problem of the input features between the training and test examples having large divergences BIBREF27 , BIBREF13   It has been successfully applied on domain adaption BIBREF7 , cross-lingual learning BIBREF15 and multi-task learning BIBREF17   All settings involve feature shifting between the training and testing 
In this paper, our setting is different  We are using the annotations from non-experts, which are noise and can influence the final performances if they are not properly processed  Directly learning based on the resulting corpus may adapt the neural feature extraction into the biased annotations  In this work, we assume that individual workers have their own guidelines in mind after short training  For example, a perfect worker can annotate highly consistently with an expert, while common crowdsourcing workers may be confused and have different understandings on certain contexts  Based on the assumption, we make an adaption for the original adversarial neural network to our setting 
Our adaption is very simple  Briefly speaking, the original adversarial learning adds an additional discriminator to classify the type of source inputs, for example, the domain category in the domain adaption setting, while we add a discriminator to classify the annotation workers  Solely the features from the input sentence is not enough for worker classification  The annotation result of the worker is also required  Thus the inputs of our discriminator are different  Here we exploit both the source sentences and the crowd-annotated NE labels as basic inputs for the worker discrimination 
In the following, we describe the proposed adversarial learning module, including both the submodels and the training method  As shown by the left part of Figure 1 , the submodel consists of four parts: (1) a common Bi-LSTM over input characters; (2) an additional Bi-LSTM to encode crowd-annotated NE label sequence; (3) a convolutional neural network (CNN) to extract features for worker discriminator; (4) output and prediction 
Common Bi-LSTM over Characters
To build the adversarial part, first we create a new bi-directional LSTM, named by the common Bi-LSTM:
$$\mathbf {h}_1^{\text{\tiny common}} \mathbf {h}_2^{\text{\tiny common}} \cdots \mathbf {h}_n^{\text{\tiny common}} = \text{Bi-LSTM}(\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n) $$   (Eq  13)
As shown in Figure 1 , this Bi-LSTM is constructed over the same input character representations of the private Bi-LSTM, in order to extract worker independent features 
The resulting features of the common Bi-LSTM are used for both NER and the worker discriminator, different with the features of private Bi-LSTM which are used for NER only  As shown in Figure 1 , we concatenate the outputs of the common and private Bi-LSTMs together, and then feed the results into the feed-forward combination layer of the NER part  Thus Formula 6 can be rewritten as:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} (\mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{private}}) + \mathbf {b},$$   (Eq  14)
where $\mathbf {W}$ is wider than the original combination because the newly-added $\mathbf {h}_t^{\text{common}}$  
Noticeably, although the resulting common features are used for the worker discriminator, they actually have no capability to distinguish the workers  Because this part is exploited to maximize the loss of the worker discriminator, it will be interpreted in the later training subsection  These features are invariant among different workers, thus they can have less noises for NER  This is the goal of adversarial learning, and we hope the NER being able to find useful features from these worker independent features 
Additional Bi-LSTM over Annotated NER Labels
In order to incorporate the annotated NE labels to predict the exact worker, we build another bi-directional LSTM (named by label Bi-LSTM) based on the crowd-annotated NE label sequence  This Bi-LSTM is used for worker discriminator only  During the decoding of the testing phase, we will never have this Bi-LSTM, because the worker discriminator is no longer required 
Assuming the crowd-annotated NE label sequence annotated by one worker is $\mathbf {\bar{y}} = \bar{y}_1\bar{y}_2 \cdots \bar{y}_n$ , we exploit a looking-up table $\mathbf {E}^{L}$ to obtain the corresponding sequence of their vector representations $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , similar to the method that maps characters into their neural representations  Concretely, for one NE label $\bar{y}_t$ ( $t \in [1, n]$ ), we obtain its neural vector by: $\mathbf {x^{\prime }}_t = \text{look-up}(\bar{y}_t, \mathbf {E}^L)$  
Next step we apply bi-directional LSTM over the sequence $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , which can be formalized as:
$$\mathbf {h}_1^{\text{label}} \mathbf {h}_2^{\text{label}} \cdots \mathbf {h}_n^{\text{label}} = \text{Bi-LSTM}(\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n) $$   (Eq  16)
The resulting feature sequence is concatenated with the outputs of the common Bi-LSTM, and further be used for worker classification 
CNN
Following, we add a convolutional neural network (CNN) module based on the concatenated outputs of the common Bi-LSTM and the label Bi-LSTM, to produce the final features for worker discriminator  A convolutional operator with window size 5 is used, and then max pooling strategy is applied over the convolution sequence to obtain the final fixed-dimensional feature vector  The whole process can be described by the following equations:
$$\begin{split} &\mathbf {h}_t^{\text{worker}} = \mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{label}} \\ &\mathbf {\tilde{h}}_t^{\text{worker}} = \tanh (\mathbf {W}^{\text{cnn}}[\mathbf {h}_{t-2}^{\text{worker}}, \mathbf {h}_{t-1}^{\text{worker}}, \cdots , \mathbf {h}_{t+2}^{\text{worker}}]) \\ &\mathbf {h}^{\text{worker}} = \text{max-pooling}(\mathbf {\tilde{h}}_1^{\text{worker}}\mathbf {\tilde{h}}_2^{\text{worker}} \cdots \mathbf {\tilde{h}}_n^{\text{worker}}) \\ \end{split}$$   (Eq  18)
where $t \in [1,n]$ and $\mathbf {W}^{\text{cnn}}$ is one model parameter  We exploit zero vector to paddle the out-of-index vectors 
Output and Prediction
After obtaining the final feature vector for the worker discriminator, we use it to compute the output vector, which scores all the annotation workers  The score function is defined by:
$$\mathbf {o}^{\text{worker}} = \mathbf {W}^{\text{worker}} \mathbf {h}^{\text{worker}},$$   (Eq  20)
where $\mathbf {W}^{\text{worker}}$ is one model parameter and the output dimension equals the number of total non-expert annotators  The prediction is to find the worker which is responsible for this annotation 
Adversarial Training
The training objective with adversarial neural network is different from the baseline model, as it includes the extra worker discriminator  Thus the new objective includes two parts, one being the negative log-likelihood from NER which is the same as the baseline, and the other being the negative the log-likelihood from the worker discriminator 
In order to obtain the negative log-likelihood of the worker discriminator, we use softmax to compute the probability of the actual worker $\bar{z}$ as well, which is defined by:
$$p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}) = \frac{\exp (\mathbf {o}^{\text{worker}}_{\bar{z}})}{\sum _{z} \exp (\mathbf {o}^{\text{worker}}_z)},$$   (Eq  22)
where $z$ should enumerate all workers 
Based on the above definition of probability, our new objective is defined as follows:
$$\begin{split} \text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) &= \text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) - \text{loss}(\Theta , \Theta ^{\prime }, \mathbf {X}) \\ \text{~~~~~~} &= -\log p(\mathbf {\bar{y}}|\mathbf {X}) + \log p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}), \end{split}$$   (Eq  23)
where $\Theta $ is the set of all model parameters related to NER, and $\Theta ^{\prime }$ is the set of the remaining parameters which are only related to the worker discriminator, $\mathbf {X}$ , $\mathbf {\bar{y}}$ and $\bar{z}$ are the input sentence, the crowd-annotated NE labels and the corresponding annotator for this annotation, respectively  It is worth noting that the parameters of the common Bi-LSTM are included in the set of $\Theta $ by definition 
In particular, our goal is not to simply minimize the new objective  Actually, we aim for a saddle point, finding the parameters $\Theta $ and $\Theta ^{\prime }$ satisfying the following conditions:
$$\begin{split} \hat{\Theta } &= \mathop {arg~min}_{\Theta }\text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \hat{\Theta }^{\prime } &= \mathop {arg~max}_{\Theta ^{\prime }}\text{R}(\hat{\Theta }, \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \end{split}$$   (Eq  24)
where the first equation aims to find one $\Theta $ that minimizes our new objective $\text{R}(\cdot )$ , and the second equation aims to find one $\Theta ^{\prime }$ maximizing the same objective 
Intuitively, the first equation of Formula 24 tries to minimize the NER loss, but at the same time maximize the worker discriminator loss by the shared parameters of the common Bi-LSTM  Thus the resulting features of common Bi-LSTM actually attempt to hurt the worker discriminator, which makes these features worker independent since they are unable to distinguish different workers  The second equation tries to minimize the worker discriminator loss by its own parameter $\Theta ^{\prime }$  
We use the standard back-propagation method to train the model parameters, the same as the baseline model  In order to incorporate the term of the argmax part of Formula 24 , we follow the previous work of adversarial training BIBREF13 , BIBREF15 , BIBREF17 , by introducing a gradient reverse layer between the common Bi-LSTM and the CNN module, whose forward does nothing but the backward simply negates the gradients 
Data Sets
With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain  We hire undergraduate students to annotate the sentences  They are required to identify the predefined types of entities in the sentences  Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences 
Labeled Data: DL-PS  In Dialog domain (DL), we collect raw sentences from a chatbot application  And then we randomly select 20K sentences as our pool and hire 43 students to annotate the sentences  We ask the annotators to label two types of entities: Person-Name and Song-Name  The annotators label the sentences independently  In particular, each sentence is assigned to three annotators for this data  Although the setting can be wasteful of labor, we can use the resulting dataset to test several well-known baselines such as majority voting 
After annotation, we remove some illegal sentences reported by the annotators  Finally, we have 16,948 sentences annotated by the students  Table 1 shows the information of annotated data  The average Kappa value among the annotators is 0 6033, indicating that the crowd annotators have moderate agreement on identifying entities on this data 
In order to evaluate the system performances, we create a set of corpus with gold annotations  Concretely, we randomly select 1,000 sentences from the final dataset and let two experts generate the gold annotations  Among them, we use 300 sentences as the development set and the remaining 700 as the test set  The rest sentences with only student annotations are used as the training set 
Labeled data: EC-MT and EC-UQ  In E-commerce domain (EC), we collect raw sentences from two types of texts: one is titles of merchandise entries (EC-MT) and another is user queries (EC-UQ)  The annotators label five types of entities: Brand, Product, Model, Material, and Specification  These five types of entities are very important for E-commerce platform, for example building knowledge graph of merchandises  Five students participate the annotations for this domain since the number of sentences is small  We use the similar strategy as DL-PS to annotate the sentences, except that only two annotators are assigned for each sentence, because we aim to test the system performances under very small duplicated annotations 
Finally, we obtain 2,337 sentences for EC-MT and 2,300 for EC-UQ  Table 1 shows the information of annotated results  Similarly, we produce the development and test datasets for system evaluation, by randomly selecting 400 sentences and letting two experts to generate the groundtruth annotations  Among them, we use 100 sentences as the development set and the remaining 300 as the test set  The rest sentences with only crowdsourcing annotations are used as the training set 
Unlabeled data  The vector representations of characters are basic inputs of our baseline and proposed models, which are obtained by the looking-up table $\mathbf {E}^W$   As introduced before, we can use pretrained embeddings from large-scale raw corpus to initialize the table  In order to pretrain the character embeddings, we use one large-scale unlabeled data from the user-generated content in Internet  Totally, we obtain a number of 5M sentences  Finally, we use the tool word2vec to pretrain the character embeddings based on the unlabeled dataset in our experiments 
Settings
For evaluation, we use the entity-level metrics of Precision (P), Recall (R), and their F1 value in our experiments, treating one tagged entity as correct only when it matches the gold entity exactly 
There are several hyper-parameters in the baseline LSTM-CRF and our final models  We set them empirically by the development performances  Concretely, we set the dimension size of the character embeddings by 100, the dimension size of the NE label embeddings by 50, and the dimension sizes of all the other hidden features by 200 
We exploit online training with a mini-batch size 128 to learn model parameters  The max-epoch iteration is set by 200, and the best-epoch model is chosen according to the development performances  We use RMSprop BIBREF28 with a learning rate $10^{-3}$ to update model parameters, and use $l_2$ -regularization by a parameter $10^{-5}$   We adopt the dropout technique to avoid overfitting by a drop value of $0 2$  
Comparison Systems
The proposed approach (henceforward referred to as “ALCrowd”) is compared with the following systems:
CRF: We use the Crfsuite tool to train a model on the crowdsourcing labeled data  As for the feature settings, we use the supervised version of BIBREF0 zhao2008unsupervised 
CRF-VT: We use the same settings of the CRF system, except that the training data is the voted version, whose groundtruths are produced by majority voting at the character level for each annotated sentence 
CRF-MA: The CRF model proposed by BIBREF3 rodrigues2014sequence, which uses a prior distributation to model multiple crowdsourcing annotators  We use the source code provided by the authors 
LSTM-CRF: Our baseline system trained on the crowdsourcing labeled data 
LSTM-CRF-VT: Our baseline system trained on the voted corpus, which is the same as CRF-VT 
LSTM-Crowd: The LSTM-CRF model with crowd annotation learning proposed by BIBREF4 nguyen2017aggregating  We use the source code provided by the authors 
The first three systems are based on the CRF model using traditional handcrafted features, and the last three systems are based on the neural LSTM-CRF model  Among them, CRF-MA, LSTM-Crowd and our system with adversarial learning (ALCrowd) are based on crowd annotation learning that directly trains the model on the crowd-annotations  Five systems, including CRF, CRF-MA, LSTM-CRF, LSTM-Crowd, and ALCrowd, are trained on the original version of labeled data, while CRF-VT and LSTM-CRF-VT are trained on the voted version  Since CRF-VT, CRF-MA and LSTM-CRF-VT all require ground-truth answers for each training sentence, which are difficult to be produced with only two annotations, we do not apply the three models on the two EC datasets 
Main Results
In this section, we show the model performances of our proposed crowdsourcing learning system (ALCrowd), and meanwhile compare it with the other systems mentioned above  Table 2 shows the experimental results on the DL-PS datasets and Table 3 shows the experiment results on the EC-MT and EC-UQ datasets, respectively 
The results of CRF and LSTM-CRF mean that the crowd annotation is an alternative solution with low cost for labeling data that could be used for training a NER system even there are some inconsistencies  Compared with CRF, LSTM-CRF achieves much better performances on all the three data, showing +6 12 F1 improvement on DL-PS, +4 51 on EC-MT, and +9 19 on EC-UQ  This indicates that LSTM-CRF is a very strong baseline system, demonstrating the effectiveness of neural network 
Interestingly, when compared with CRF and LSTM-CRF, CRF-VT and LSTM-CRF-VT trained on the voted version perform worse in the DL-PS dataset  This trend is also mentioned in BIBREF4 nguyen2017aggregating  This fact shows that the majority voting method might be unsuitable for our task  There are two possible reasons accounting for the observation  On the one hand, simple character-level voting based on three annotations for each sentence may be still not enough  In the DL-PS dataset, even with only two predefined entity types, one character can have nine NE labels  Thus the majority-voting may be incapable of handling some cases  While the cost by adding more annotations for each sentence would be greatly increased  On the other hand, the lost information produced by majority-voting may be important, at least the ambiguous annotations denote that the input sentence is difficult for NER  The normal CRF and LSTM-CRF models without discard any annotations can differentiate these difficult contexts through learning 
Three crowd-annotation learning systems provide better performances than their counterpart systems, (CRF-MA VS CRF) and (LSTM-Crowd/ALCrowd VS LSTM-CRF)  Compared with the strong baseline LSTM-CRF, ALCrowd shows its advantage with +1 08 F1 improvements on DL-PS, +1 24 on EC-MT, and +2 38 on EC-UQ, respectively  This indicates that adding the crowd-annotation learning is quite useful for building NER systems  In addition, ALCrowd also outperforms LSTM-Crowd on all the datasets consistently, demonstrating the high effectiveness of ALCrowd in extracting worker independent features  Among all the systems, ALCrowd performs the best, and significantly better than all the other models (the p-value is below $10^{-5}$ by using t-test)  The results indicate that with the help of adversarial training, our system can learn a better feature representation from crowd annotation 
Discussion
Impact of Character Embeddings  First, we investigate the effect of the pretrained character embeddings in our proposed crowdsourcing learning model  The comparison results are shown in Figure 2 , where Random refers to the random initialized character embeddings, and Pretrained refers to the embeddings pretrained on the unlabeled data  According to the results, we find that our model with the pretrained embeddings significantly outperforms that using the random embeddings, demonstrating that the pretrained embeddings successfully provide useful information 
Case Studies  Second, we present several case studies in order to study the differences between our baseline and the worker adversarial models  We conduct a closed test on the training set, the results of which can be regarded as modifications of the training corpus, since there exist inconsistent annotations for each training sentence among the different workers  Figure 3 shows the two examples from the DL-PS dataset, which compares the outputs of the baseline and our final models, as well as the majority-voting strategy 
In the first case, none of the annotations get the correct NER result, but our proposed model can capture it  The result of LSTM-CRF is the same as majority-voting  In the second example, the output of majority-voting is the worst, which can account for the reason why the same model trained on the voted corpus performs so badly, as shown in Table 2   The model of LSTM-CRF fails to recognize the named entity “Xiexie” because of not trusting the second annotation, treating it as one noise annotation  Our proposed model is able to recognize it, because of its ability of extracting worker independent features 
Conclusions
In this paper, we presented an approach to performing crowd annotation learning based on the idea of adversarial training for Chinese Named Entity Recognition (NER)  In our approach, we use a common and private Bi-LSTMs for representing annotator-generic and -specific information, and learn a label Bi-LSTM from the crowd-annotated NE label sequences  Finally, the proposed approach adopts a LSTM-CRF model to perform tagging  In our experiments, we create two data sets for Chinese NER tasks in the dialog and e-commerce domains  The experimental results show that the proposed approach outperforms strong baseline systems 
 Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No  61572338, 61525205, and 61602160)  This work is also partially supported by the joint research project of Alibaba and Soochow University  Wenliang is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization 

ground_truth: F1 scores of 85.99 on the DL-PS data,  75.15 on the EC-MT data and 71.53 on the EC-UQ data 

F1 of 85.99 on the DL-PS dataset (dialog domain); 75.15 on EC-MT and 71.53 on EC-UQ (e-commerce domain)